Official schedules and announcements from major AI conferences such as NeurIPS, ICML, or CVPR

CamelAIOrg Releases Open-Source CRAB Framework for AI Benchmarking

Aug 10, 2024, 10:44 AM

Researchers from KAUST, UTokyo, CMU, Stanford, Harvard, Tsinghua, SUSTech, and Oxford, in collaboration with CamelAIOrg, have developed the CRAB framework, an AI framework designed for building LLM agent benchmark environments in a Python-centric way. CRAB, which stands for Cross-environment Agent Benchmark, aims to become a general-purpose agent benchmark framework for Multimodal Language Model (MLM) agents. The framework includes CRAB Benchmark-v0, developed using the CRAB framework, which features 100 tasks across two environments, Ubuntu and Android. It provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them. The CRAB framework is now open-sourced, allowing agents to control devices such as mobile phones, laptops, or desktops from a single prompt.