Mistral AI Releases 25.38 GB Pixtral 12B, Its First 12-Billion Parameter Multimodal Model

Sep 11, 2024, 08:15 AM

Mistral AI has released its first multimodal model, Pixtral 12B, which integrates both language and vision processing capabilities. The model, which is approximately 25.38 GB in size, features a 12-billion parameter architecture with 40 layers and a hidden dimension of 14,336. Key specifications include a text backbone based on Mistral Nemo 12B, a vision adapter with 400 million parameters, and a larger vocabulary of 131,072 tokens. The vision encoder uses GeLU and 2D RoPE, and the model introduces three new special tokens. Pixtral 12B is available via torrent and has been uploaded to platforms like GitHub and Hugging Face. This release marks a significant advancement in multimodal AI technology.