Anthropic Launches Prompt Caching with 90% Cost and 80% Latency Reductions

Aug 14, 2024, 04:49 PM

Anthropic has introduced a new feature called prompt caching in its API, currently available in beta. This feature significantly reduces the costs and latency associated with AI model responses. By storing and reusing context, prompt caching can cut API input costs by up to 90% and reduce latency by up to 80%. This development is particularly beneficial for applications involving long, static instructions, as it allows for more efficient processing. The prompt caching feature is designed to improve the performance of large language models (LLMs) and is expected to have a substantial impact on applications such as Retrieval-Augmented Generation (RAG). The pricing model for Anthropic's caching involves charges for cache writes, with a cache lifetime of five minutes that refreshes each time the cached content is used. The feature supports Claude 3 Haiku, Opus, and 3.5 Sonnet.