AI Math Benchmark Shows Big Gap Between Paid and Open Models
The First Proof project released its third round of results, assessing the mathematical reasoning of publicly available AI models. The results show that proprietary frontier models are much stronger at research mathematics than open-weight models. The project was launched by mathematicians as an independent, rigorous exam to address flaws in existing AI benchmarks.
First Proof began in February as an independent test created by mathematicians. For its third round, organizers gathered problems from specialists across many mathematical subfields, chose ten, and required models to answer without human assistance. Three graders then reviewed each response in a simulated peer-review process. Because entrants had to be openly available, OpenAI's newest model could not take part.
The tested systems included GPT 6 Astra, Claude Opus 5.5, a combined Astra-based "ProofCouncil," and a Kimi K3 harness. ProofCouncil performed best, earning acceptable judgments on nine of ten problems. Earlier rounds showed rapid improvement: the February leader solved only a few, while June's winner managed six or seven.
The widening gap could shape who can use advanced mathematical AI. Researchers at well-funded labs or institutions able to pay for proprietary systems may gain faster assistance with proofs and problem solving, while users relying on open-weight models may have less capable tools. This may influence access to mathematical discovery, collaboration, and training. It could also affect how much independent scrutiny such systems receive, since closed models are harder for outside experts to examine or compare.