SambaNova Launches Fastest AI Inference Platform for Llama 3.1 at 570 Tokens/Second

Sep 10, 2024, 04:34 PM

SambaNova has announced the launch of its new cloud inference platform, SambaNova Cloud, which offers unprecedented speeds for AI model inference. Notably, the Llama 3.1 405B model achieves a speed of 132 tokens per second in full precision, while the Llama 3.1 70B model reaches up to 570 tokens per second. This performance is significantly faster than traditional GPUs, with claims of up to 10 times faster inference speeds. The platform operates in real-time and serves the Llama 3.1 405B model in 16-bit precision. It is available for developers starting today, with free access via API and no waitlist. The service has been independently verified and is expected to enable advanced AI applications.