NVIDIA Launches NVILA: Efficient Visual Language Models Handle Large Images, Long Videos, Reduce Training Costs by 4.5x

Dec 8, 2024, 10:46 PM

NVIDIA has unveiled NVILA, a new family of open Visual Language Models (VLMs) aimed at enhancing both efficiency and accuracy in processing visual data. Building on the existing VILA model, NVILA employs a 'scale-then-compress' strategy that allows it to handle large images and long videos without a decrease in performance. This innovative approach not only improves the resolution of images and videos to capture finer details but also reduces training costs by 4.5 times. The introduction of NVILA aligns with ongoing advancements in AI, particularly in optimizing the performance of Vision Language Models. Other recent developments in the field include methods for speeding up VLMs through techniques like pruning visual tokens and leveraging smaller models to guide larger ones, which further enhance processing speed and efficiency.