Researchers Develop Scalable MatMul-Free Language Model with 61% Memory Reduction

Jun 6, 2024, 12:12 PM

Researchers have developed a scalable, MatMul-free language model that eliminates the need for matrix multiplication operations while maintaining strong performance at billion-parameter scales. This new approach, which replaces MatMul operations with addition and negation, has shown to reduce memory usage by up to 61% and improve GPU efficiency. The model processes billion-parameter scale models at 13W beyond human-readable throughput, moving large language models (LLMs) closer to brain-like efficiency. The implementation has been a collaborative effort involving researchers W Guo, J Long, Y Zeng, and Z Liu from Princeton University, Stevens Institute of Technology, and the University of Pennsylvania.