Loading...
Market
MLE-bench adopted as standard benchmark by major AI conference by mid-2025?
2
Resolution / Starting Odds
Yes • 50%
No • 50%
Announcements from major AI conferences such as NeurIPS, ICML, or CVPR
Story
OpenAI Releases MLE-bench with 75 Kaggle Competitions to Evaluate AI Agents' ML Engineering Skills
Oct 10, 2024, 05:33 PM
OpenAI has announced the release of MLE-bench, a new benchmark designed to evaluate the machine learning engineering capabilities of AI agents. The benchmark comprises 75 real-life machine learning engineering competitions sourced from Kaggle. MLE-bench aims to measure how well AI agents perform tasks in machine learning engineering, bridging the gap between theoretical AI knowledge and practical applications in real-world scenarios. The release of this benchmark could accelerate the development of AI agents capable of writing machine learning code, potentially leading to self-improving AI systems. The benchmark raises the prospect of AI agents achieving Kaggle Grandmaster status in the future.
View original story
Similar markets
NeurIPS • 25%
ICML • 25%
AAAI • 25%
CVPR • 25%
Tech industry • 25%
Healthcare • 25%
Finance • 25%
Education • 25%
Yes • 50%
No • 50%
ChatGPT-4o • 25%
Google's Gemini • 25%
Another AI model • 25%
No clear leader • 25%
Yes, at NeurIPS 2024 • 25%
Yes, at ICML 2024 • 25%
Yes, at both NeurIPS and ICML • 25%
No, it will not be featured in any major AI conferences • 25%
Llama 3.1 405B • 25%
GPT-4o • 25%
Claude Sonnet 3.5 • 25%
Other • 25%
Yes, it will outperform AdamW • 25%
Yes, it will outperform SGD • 25%
Yes, it will outperform AdaGrad • 25%
No, it will not outperform any • 25%
MMLU • 25%
ARC • 25%
GSM8K • 25%
None by June 30, 2024 • 25%