Loading...
Market
Story
Anthropic Launches Prompt Caching with 90% Cost and 80% Latency Reductions
Aug 14, 2024, 04:49 PM
Anthropic has introduced a new feature called prompt caching in its API, currently available in beta. This feature significantly reduces the costs and latency associated with AI model responses. By storing and reusing context, prompt caching can cut API input costs by up to 90% and reduce latency by up to 80%. This development is particularly beneficial for applications involving long, static instructions, as it allows for more efficient processing. The prompt caching feature is designed to improve the performance of large language models (LLMs) and is expected to have a substantial impact on applications such as Retrieval-Augmented Generation (RAG). The pricing model for Anthropic's caching involves charges for cache writes, with a cache lifetime of five minutes that refreshes each time the cached content is used. The feature supports Claude 3 Haiku, Opus, and 3.5 Sonnet.
View original story
Similar markets
Less than 50% • 25%
50% to 70% • 25%
70% to 90% • 25%
More than 90% • 25%
Latency < 100ms • 25%
Latency 100ms - 150ms • 25%
Latency 150ms - 200ms • 25%
Latency > 200ms • 25%
<300ms • 25%
300ms-400ms • 25%
400ms-500ms • 25%
>500ms • 25%
Yes • 50%
No • 50%
Less than 40% • 25%
40% to 50% • 25%
50% to 60% • 25%
More than 60% • 25%
Yes • 50%
No • 50%
Less than 30% • 25%
30% - 35% • 25%
35% - 40% • 25%
More than 40% • 25%
20% improvement • 25%
30% improvement • 25%
40% improvement • 25%
50% or more improvement • 25%
Significantly improved • 33%
No significant change • 33%
Significantly worsened • 33%
