Nous Research's DisTrO Cuts AI Training Costs by 1,000 to 10,000 Times

Aug 26, 2024, 08:39 PM

Nous Research has released a preliminary report on DisTrO (Distributed Training Over-the-Internet), a new approach to AI model training that leverages decentralized computing power. This method significantly reduces communication requirements by 1,000 to 10,000 times, allowing for effective training of AI models without relying on centralized entities. The technology is architecture-agnostic and network-agnostic, enabling the use of low bandwidth connections between heterogeneous hardware. This breakthrough could revolutionize AI by making it more accessible and open-sourced, akin to a SETI@home model for AI training.