Snowflake AI's Arctic-SnowCoder-1.3B Sets SOTA with 36% Higher Performance in Code Models

Sep 6, 2024, 05:05 PM

Snowflake AI Research has introduced Arctic-SnowCoder-1.3B, a new 1.3 billion parameter model that sets the state-of-the-art (SOTA) among small language models for code. The model is trained in three phases: general pretraining on 500 billion tokens of raw code data, followed by continued pretraining on high-quality data, and finally, fine-tuning on domain-specific data. Arctic-SnowCoder-1.3B outperforms larger 1 trillion token models by 36% in code generation tasks. The model uses a total of 555 billion tokens in its training process. The research was conducted by Snowflake AI Research in collaboration with the University of Illinois at Urbana-Champaign, with contributions from Y Wei, H Han, and R Samdani.