New secret math benchmark stumps AI models and PhDs alike

Epoch AI has introduced FrontierMath, a new mathematics benchmark that poses significant challenges for AI models and expert mathematicians alike. The benchmark consists of hundreds of expert-level problems that leading AI models, including GPT-4o and Claude 3.5 Sonnet, reportedly solve less than 2% of the time. This stark contrast to their performance on simpler benchmarks, where many models score above 90%, highlights the limitations of current AI capabilities. FrontierMath's problems are designed to be unpublished to prevent AI companies from training their models on them, ensuring a more accurate assessment of their problem-solving abilities. Developed in collaboration with over 60 mathematicians, the problems span various mathematical disciplines and have undergone peer review for correctness. Notably, the problems require complex solutions that are difficult to guess, making them particularly challenging for AI systems. Mathematicians like Terence Tao have acknowledged the difficulty of these problems, suggesting that solving them may require a combination of human expertise and AI assistance. Epoch AI plans to conduct regular evaluations of AI models against this benchmark and will release additional sample problems in the future to further aid research in this area.

- FrontierMath is a new benchmark that challenges AI models and mathematicians with expert-level problems.

- Leading AI models solve less than 2% of FrontierMath problems, contrasting with their high performance on simpler benchmarks.

- The benchmark's problems are unpublished to prevent AI training on them, ensuring a fair assessment.

- Developed with input from over 60 mathematicians, the problems span multiple disciplines and require complex solutions.

- Epoch AI plans to regularly evaluate AI models and expand the problem set in the future.