Takeaways
- The thread: Karpathy wrote a long X post arguing that "the eval game has lapped the model game" — labs are training to benchmarks they themselves helped design.
- The receipt: He calls out specific benchmark improvements over the past 12 months that don't translate to real-world capability gains practitioners can feel.
- The fix: Evals that change every week, are produced by adversarial labelers, and are kept private. He thinks the public-benchmark era is functionally over.
Why it lands
Karpathy is uniquely positioned to make this argument — he was inside OpenAI for the era when benchmarks were the rallying point, and he's outside now and willing to say the loud part. The thread is less an indictment of any specific lab than a structural critique: when the same teams designing benchmarks are training models against them, you get a leaderboard that means less every quarter.
The argument that practitioners can't feel the gains is the part that should worry product leaders at the frontier labs. If your enterprise customers are reading the same SWE-bench number you're celebrating, but their actual workflows feel the same as six months ago, the narrative gap is going to widen.
What to watch
Whether the frontier labs respond with substance or with marketing. One of them committing to a private, rotating, adversarial eval — the way Karpathy describes — would be a real signal. Most likely we get another round of benchmark-improvement announcements and the gap he's pointing at gets wider.