Loading...
Market
MLE-bench updated with more competitions by end of 2025?
3
Resolution / Starting Odds
Yes • 50%
No • 50%
Official announcements from OpenAI or Kaggle
Story
OpenAI Releases MLE-bench with 75 Kaggle Competitions to Evaluate AI Agents' ML Engineering Skills
Oct 10, 2024, 05:33 PM
OpenAI has announced the release of MLE-bench, a new benchmark designed to evaluate the machine learning engineering capabilities of AI agents. The benchmark comprises 75 real-life machine learning engineering competitions sourced from Kaggle. MLE-bench aims to measure how well AI agents perform tasks in machine learning engineering, bridging the gap between theoretical AI knowledge and practical applications in real-world scenarios. The release of this benchmark could accelerate the development of AI agents capable of writing machine learning code, potentially leading to self-improving AI systems. The benchmark raises the prospect of AI agents achieving Kaggle Grandmaster status in the future.
View original story
Similar markets
Yes • 50%
No • 50%
Yes, it will outperform AdamW • 25%
Yes, it will outperform SGD • 25%
Yes, it will outperform AdaGrad • 25%
No, it will not outperform any • 25%
GPT-4V • 25%
Claude 3.5 Sonnet • 25%
Flash • 25%
Other • 25%
Yes • 50%
No • 50%
GPQA Diamond (CoT) • 25%
Math (CoT) • 25%
Other AI Benchmark • 25%
None • 25%
OpenAI o1 • 33%
Anthropic • 33%
Other • 33%
Less than 80% • 33%
80% to 90% • 33%
More than 90% • 33%