Official announcements from MistralAI or major tech companies

MistralAI Releases New 25.38 GB Pixtral-12B Vision-Language Model

Sep 11, 2024, 07:01 AM

MistralAI has released a new vision-language multimodal model called Pixtral-12B. The model, which is available via a magnet link, has a size of 25.38 GB. Key architectural features of Pixtral-12B include a dimension of 5120, 40 layers, a head dimension of 128, a hidden dimension of 14336, 32 heads, 8 key-value heads, a rope theta of 1000000000.0, a normalization epsilon of 1e-05, and a vocabulary size of 131072. The model also incorporates GeLU and 2D RoPE for the vision adapter and includes three new tokens in its tokenizer. The vision encoder's hidden size is also notable.