Global AI market performance reports

Groq Inc. Achieves Breakthrough with 30,210 Tokens/Sec and MLPerf Benchmark

May 25, 2024, 01:06 AM

Groq Inc. has achieved a significant milestone in performance engineering, showcasing their technology's ability to process data at unprecedented speeds. By implementing Groq and semantic caching, users can experience a substantial increase in the speed of generating answers to queries. The company's engineers have been working diligently to improve their stack, achieving a performance rate of 30,210 tokens per second. This advancement places Groq Inc. ahead of top GPUs in terms of data processing speed. Additionally, the new MLPerf benchmark results highlight the efficiency of 8 x H100 GPUs attached to a single VM, utilizing only a small fraction of the physical host's CPU and memory. This allows for more applications to run simultaneously on the system. Notably, Groq's LPUs have shown impressive performance, and MOEs are now twice as fast in the latest MLX, running at 60 tokens per second on an M2 Ultra.