Official benchmarks published by Groq Inc., Together Inference, or third-party validators

Groq Inc. and NVIDIA Turbocharge Llama 3.1 405B Model for Record-Breaking Speeds and Cost Efficiency

Jul 23, 2024, 03:18 PM

Groq Inc. has turbocharged the Llama 3.1 model, achieving record-breaking speeds and cost efficiency. The Llama 3.1 405B model, hosted by Groq Inc., runs at speeds up to 330 tokens per second, making it 100 times faster than previous models. This advancement is expected to significantly reduce costs, with some estimates suggesting it could be 10 times cheaper. The model is also available for download on Hugging Face. Additionally, Groq Inc. has partnered with Together Inference and Fine-tuning to bring these models to a broader audience, with speeds of up to 400 tokens per second for the Llama 3.1 8B model. NVIDIA has also announced its AI Foundry service, which will allow enterprises and nations to build custom generative AI models using Llama 3.1 405B and NVIDIA Nemotron models, with comprehensive features including synthetic data generation and fine-tuning. The Llama 3.1 70B model with 128k context is also part of this offering, and NVIDIA NeMo Retriever microservices are included for accurate responses.