NVIDIA's Mistral-NeMo-Minitron 8B Model Achieves High Accuracy

Aug 21, 2024, 04:08 PM

NVIDIA has announced the release of the Mistral-NeMo-Minitron 8B, a small language model derived from the Mistral NeMo 12B model through pruning and distillation techniques. This new model is designed to offer state-of-the-art accuracy while being more computationally efficient. The Mistral-NeMo-Minitron 8B achieves high accuracy in nine popular benchmarks, including those for chatbots, virtual assistants, content generation, and coding. The model was developed using 400 billion tokens for distillation, significantly reducing the computational cost compared to training from scratch. NVIDIA's innovation in pruning and distillation has enabled the creation of a commercially permissive model that maintains high performance levels. The model achieved an MMLU score of 69.5.