Microsoft Open-Sources BitNet Framework for 1-Bit LLMs Including BitNet b1.58

Oct 20, 2024, 01:37 AM

Microsoft has open-sourced bitnet.cpp, the official inference framework for 1-bit large language models (LLMs), such as BitNet b1.58 and BitNet Llama8B. This framework offers a suite of optimized kernels that support fast and lossless inference of 1.58-bit models on CPUs, with NPU and GPU support expected in the future. BitNet technology significantly speeds up the calculation of large-scale language models, which previously required 4 or more bits of information. This development allows 100 billion parameter models to run on local devices, quantized with BitNet b1.58, at a rate of 5-7 tokens per second using a single CPU. Exo had previously open-sourced the first ternary model implementation for Apple Silicon in March, and now BitNet b1.58 can run on Apple M2 CPUs.