ModernBERT Released: State-of-the-Art Encoder Model with 8,192 Token Support, 2 Trillion Tokens, and 2-3x Faster Processing

Dec 19, 2024, 04:51 PM

ModernBERT, a new family of state-of-the-art encoder-only models, has been released by AnswerAI and LightOn. This updated version of BERT features a sequence length of 8,192 tokens and is designed for improved performance in tasks such as classification and retrieval. Trained on 2 trillion tokens, ModernBERT is reported to be twice as fast as DeBERTaV3 on short contexts and three times faster than NomicBERT and GTE on long contexts. The model is available in two sizes: ModernBERT-base with 149 million parameters and ModernBERT-large with 395 million parameters. This release marks a significant upgrade to traditional BERT models, which have not seen a major overhaul in over six years.