Mistral Releases 25.38 GB Pixtral-12B with 400M Parameters

Sep 11, 2024, 07:13 AM

Mistral has released a new multimodal model named Pixtral-12B. The model, which is a vision language model (VLM), features a 400 million parameter vision encoder. It is based on Mistral-Nemo and incorporates elements from SigLip and ViT Vision Encoder. Pixtral-12B has a size of 25.38 GB and includes specifications such as 5120 dimensions, 40 layers, 128 head dimensions, 14336 hidden dimensions, 32 heads, 8 KV heads, a rope theta of 1000000000.0, a norm epsilon of 1e-05, and a vocabulary size of 131072. The model is expected to perform well compared to other models like Lllama 3.1. A magnet link for downloading the model has also been provided.