NVIDIA Unveils Normalized Transformer (nGPT) for 4-20x Faster Training and Improved Stability of Large Language Models

Oct 19, 2024, 02:20 PM

NVIDIA researchers have introduced a new architecture called the Normalized Transformer (nGPT), which utilizes a hypersphere-based design to enhance the training speed and stability of large language models (LLMs). The nGPT architecture reportedly achieves training speeds that are 4 to 20 times faster than previous models. This advancement is expected to significantly improve convergence speed in transformer models, as noted by various experts in the field. The introduction of nGPT marks a notable development in the ongoing evolution of AI and machine learning technologies.