Home

How we ask

The method as of the November 2026 round. Changes are logged on the corrections page.

The question

Each question is a plain sentence of the kind people type into an assistant, such as "What is the best calorie tracking app?". We send it on its own: no system prompt, no follow-up, no hints about what we hope to hear, and no account history. The wording stays the same from one round to the next so that rounds can be compared.

The assistants

We ask four assistants, through each provider's API with web search turned on, using the routing service of DataForSEO, which passes the question to the provider and returns the answer with the sources the assistant consulted. The models in this round:

When a provider retires a model, we move to its successor and note the change on the round's page.

Three answers each

Assistants don't give the same answer twice. We ask each one three separate times per round, close together, so you can see how stable its answer is. Every answer is published, including the ones that are short, hedged or odd. We never pick the "best" of the three.

What we record

• The full text of each answer, exactly as returned, formatting included.
• The links the assistant cited inside its answer.
• The search queries it ran and the pages it read, when the provider reports them.
• The model name and the time the answer was produced.

All of it is downloadable as JSON from each question page, under a Creative Commons Attribution 4.0 licence.

How we read the answers

For each answer we record two things: the product it recommends first (its "first pick"), and every product it recommends or suggests. A separate model reads each answer with a fixed instruction and returns those two fields; we normalise product names (so "Things" and "Things 3" count as one) and check a sample by hand every round against the answer text. If an answer opens with "it depends" and then lists options, its first pick is the first option it recommends, and we also flag that it declined to name a single best.

What "converge" means

An assistant's pick for a question is the product it named first in at least two of its three answers. If at least three of the four assistants share the same pick, we say they converged. Two in agreement is a split, and anything less is no consensus. The "named by all four" list shows products every assistant mentioned at least once, which is often more stable than the first pick.

What this can't tell you

• The apps can differ from the API. The consumer apps add personalisation, location and their own settings. Your answer in the app may differ from ours.
• Answers change. Models are updated and the web they search changes. A round is a snapshot of one day.
• We don't check the claims inside answers. If an assistant gets a price or a feature wrong, the error stays in the published answer. These pages report what the assistants said, not whether they were right.
• Agreement isn't proof. Assistants often read the same review sites, so four matching answers can share one source. The "sources they cited" table on each page shows where the answers came from.

Schedule

New rounds go out at the start of each month. Each question page always shows the latest round, and earlier rounds stay available under Questions and in the downloadable data.