New secret math benchmark stumps AI models and PhDs alike

Epoch AI has introduced FrontierMath, a new mathematics benchmark that poses significant challenges for AI models and expert mathematicians alike. The benchmark consists of hundreds of expert-level problems, with AI models reportedly solving less than 2% of them. This stark contrast to their performance on simpler benchmarks highlights the limitations of current AI models, which often excel in less complex tasks. FrontierMath's design is unique as it remains unpublished to prevent AI companies from training their models on the problems, ensuring a more accurate assessment of their capabilities. Developed in collaboration with over 60 mathematicians, the problems cover various mathematical disciplines and have undergone peer review for correctness. Notably, the problems require substantial computational power and specialized knowledge, making them particularly difficult for AI systems. The organization plans to conduct regular evaluations of AI models against this benchmark and will release additional sample problems in the future to aid the research community. Feedback from renowned mathematicians indicates that solving these problems typically necessitates a combination of expertise and advanced computational tools, underscoring the benchmark's rigor.

- FrontierMath is a new benchmark that challenges AI models and mathematicians with expert-level problems.

- AI models solve less than 2% of FrontierMath problems, highlighting their limitations compared to simpler benchmarks.

- The benchmark remains unpublished to prevent AI training on its problems, ensuring a fair assessment.

- Problems span multiple mathematical disciplines and require significant computational power and specialized knowledge.

- Regular evaluations and additional sample problems are planned to support the research community.