lmsys Introduces Open-Source RouteLLM Framework to Cut LLM Costs by Up to 85%

Jul 1, 2024, 05:01 PM

The lmsys team has introduced RouteLLM, a new open-source routing framework designed to reduce the costs of using large language models (LLMs) like GPT-4. By directing simpler queries to cheaper models, RouteLLM claims to achieve cost reductions of over 85% on MT Bench and 45% on MMLU while maintaining 95% of the quality. This initiative is reportedly 40% cheaper than existing routers such as Martian. Developed in collaboration with Anyscale, RouteLLM uses human preference data to intelligently select the best model for each query, ensuring cost-effectiveness without compromising performance.