$ cat method.txt

Every benchmark is a different ruler. A model can score 90% on an easy quiz and 30% on research-level maths, so averaging percentages says little. Instead we line every test up on one shared scale and read off where each model sits. This is item response theory, the method exam boards use to put different exam papers on the same scale:

result[model, run] ≈ a[test] · score[model] + d[run]

• a is a test's sharpness: how many points of result one step of ability is worth. Every run of the same test shares it, learned from all of them together. Saturated tests get a small a and count for less.
• d is the level of one evaluator's run of a test. Runs of the same test scatter around a common level by an amount fitted from the data (about 4 points for text models), so an evaluator whose setup runs hot or cold can't move anyone's score, while a small run, such as a lab comparing its new model only with its previous one, is also read against everyone else's runs of that test. A result is read against the other models in its run, so a run needs at least two models and a test at least three across its runs. Sparser results stay visible as raw evidence. Lab-reported results are labelled.
• Percentages are fit as proportions after removing the guessing rate (25% on four-option multiple choice), on a log-odds scale. A result near 0% or 100% says little about how far beyond that edge a model is, so it counts for less. Error ratios use −log(error + 0.005), preserving errors above 100% caused by insertions. Ratings use a linear transformation. Stored raw values are unchanged.
• A failed run, such as one flagged by its source or one far below random guessing because the answers couldn't be parsed, stays visible on the model page but doesn't count.
• Models with few results are pulled toward the middle and get a wider ±; * marks scores from fewer than three results, and † scores where every result so far was reported by the model’s own maker. Makers’ numbers are used like anyone else’s, read against the other runs of the same test; the mark only says no one else has checked them yet.
• Task filters recalculate scores using only that task’s reviewed evaluations. Models need evidence for the selected task to appear. Compare scores within the same task and connected comparison set; a task score is not on the overall score scale.
• Agent evaluations measure the model with a particular harness, tools and reasoning effort. Arena Agent’s signed IPS treatment effects are stored in their original units and converted linearly for fitting (0.1 IPS per latent unit). They are not Elo or success rates; the aggregate is counted once, without its component signals.
• All model sets are listed together by default. Overall scores use the largest connected set; models without a shared scale are not yet calibrated. Under Extra, filter to a set without changing scores, or choose scores within that set. Each scale has mean 50, sd 15; scores from different scales are not comparable. Scores shift as new models arrive.
• ± is uncertainty within this statistical model, not a guarantee of real-world performance or a validated confidence interval.

$ whatmodel weights

A score is a weighted average of what each result says on its own. A result's weight is a² ÷ σ²: how sharply its test separates models, divided by how noisy the test is. The rest is the prior, a pull toward the middle that matters only for models with few results. Each bar splits one text model's score by test:

█ LMArena (text)█ Arena vision (style controlled)█ AIME 2025█ HMMT 2025 (edition unspecified)█ Humanity's Last Exam█ Arena Agent (October 2026 aggregate IPS)█ Arena WebDev (Code Arena)▒ other tests░ prior

Every model page shows its own breakdown under “how the score was worked out”.

$ whatmodel benchmarks

Share of evidence is how much of all the weight behind a category's scores comes from that run, averaged over its scored models. Sharp, quiet tests that many models took count most. For AI models, the biggest is LMArena (text) at 6.4%; the prior supplies 3.3%.