Benchmark results from SWE-bench Verified or similar coding performance benchmarks

Anthropic Launches Claude 3.5 Haiku AI, Outperforms GPT-4o in Coding Tasks

Nov 4, 2024, 06:29 PM

Anthropic has released Claude 3.5 Haiku, a new AI language model now available on their API, Amazon Bedrock, Google Cloud's Vertex AI, and supported by platforms like BoltAI and Replit. The model is particularly adept at coding, outperforming previous Claude models—including the largest prior model, Claude 3 Opus—and state-of-the-art models like OpenAI's GPT-4o on benchmarks such as SWE-bench Verified, which measures models' ability to solve real software issues. Despite being their smallest model, Claude 3.5 Haiku surpasses older larger models in performance while offering fast response times. The model is priced at $1 per million input tokens and $5 per million output tokens, which is four times more expensive than the previous Claude 3 Haiku; however, cost savings of up to 90% can be achieved with prompt caching, making it cost-effective for developers. The model offers frontier code generation capabilities, enhancing its appeal to programmers.