BenchMIRT: What LLM Benchmarks Actually Measure
Ai2 broke 16 benchmarks down to single questions across 100 models and 34,000-plus items. Two dimensions explain most of what the scores capture.
Illustration: Two beams of light reveal two clear planes inside an apparently unordered cloud of suspended glass tiles.
Ai2's BenchMIRT analysis of 16 benchmarks and 100 models finds that two underlying capabilities, safety and general reasoning, account for most of what benchmark scores actually track.
At a glance
- Ai2 published BenchMIRT on September 1, 2026, along with a tech report, a dataset collection, and a code repository on GitHub.
- Data: 100 models, all released by March 2025, 16 benchmarks, more than 34,000 individual questions.
- Two dominant dimensions emerged from the response patterns: safety and general reasoning.
- Keeping 10 percent of the questions largely preserved the picture of model differences; 50 percent often matched the full benchmark.
- On held-out questions BenchMIRT predicted correctly 79 percent of the time, versus 70 percent for a simpler approach.
A leaderboard hands you one number per benchmark. On September 1, 2026, the Allen Institute for AI (Ai2) published BenchMIRT, which throws that number away and works at the level of the individual question instead. Across 100 models, most of the variation collapses onto just two axes.
What the method does
BenchMIRT applies item response theory, borrowed from educational testing, in its multidimensional form (MIRT). It estimates two quantities at once: how strong a model is on a latent capability, and how difficult and how discriminating each question is. A question every model answers correctly separates nobody, yet it still carries full weight in an average score. That asymmetry is the whole point of the exercise.
Labels versus what the questions actually load on
The analysis covers 100 models released by March 2025, 16 benchmarks, and more than 34,000 questions, including MMLU-Pro, GPQA, MATH, and BBH on the reasoning side and HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest on the safety side. Two dominant dimensions came out of the response patterns: safety and general reasoning.
What a benchmark is named turns out to be a poor guide to what it measures. BBQ lines up more with general reasoning than with safety. WMDP correlates inversely with reasoning, so stronger reasoners score lower there. HarmBench splits internally: standard and contextual questions load on safety, while its copyright questions load on general reasoning. A single headline score hides all of that.
Most questions are doing little work
The efficiency result is the practical one. Keeping 10 percent of the questions largely preserved the picture of how models differ, and 50 percent often matched the full benchmark. On questions deliberately held out, BenchMIRT correctly predicted whether a model would answer them right 79 percent of the time, against 70 percent for a simpler baseline.
The limits Ai2 states
Ai2 lists three. The model pool stops at March 2025, so newer systems are absent. The dimensions that emerge depend on which benchmarks go into the analysis in the first place. And for the narrow task of ranking models on held-out items, a plain benchmark average still edges out BenchMIRT. The team released a tech report, a dataset collection, and a GitHub repository alongside the post.
What this report does not establish
This article rests on exactly one source: Ai2's own blog post. No second, independent newsroom has confirmed the figures, the tech report was not separately reviewed for this piece, and we are aware of no outside replication. Every number above is Ai2's own reporting. How the 16 benchmarks divide between the reasoning and safety groups is also not stated unambiguously in the material available to us.
FAQ
What is BenchMIRT?
A method released by Ai2 that evaluates LLM benchmarks question by question rather than as a single score. It uses multidimensional item response theory to estimate model ability alongside each question's difficulty and discrimination.
How many models and questions did Ai2 analyze?
100 language models released by March 2025, across 16 benchmarks and more than 34,000 individual questions, including MMLU-Pro, GPQA, MATH, BBH, HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest.
Does this mean benchmark scores are useless?
No. Ai2 shows that a single score blends several capabilities and that many questions add little separation. For plain model ranking, the benchmark average actually performed slightly better than BenchMIRT in their evaluation.
