Mistral AI Releases 25.38 GB Pixtral 12B Multimodal AI Model with 400M Parameters

Sep 11, 2024, 07:31 AM

Mistral AI has released a new multimodal AI model called Pixtral 12B. This model integrates both language and vision processing capabilities, marking a significant advancement in multimodal AI technology. The Pixtral 12B model features a text backbone based on Mistral Nemo 12B and includes a vision adapter with 400 million parameters. The model architecture consists of 40 layers, a hidden dimension of 14,336, a dimension of 5120, a head dimension of 128, 32 heads, and 8 kv-heads. It also has a vocabulary size of 131,072. Additionally, the vision adapter uses GeLU and 2D RoPE, and the tokenizer includes three new special tokens. The model is available for download via a torrent magnet link and has a size of approximately 25.38 GB.