Google Unveils 'Infini-attention' for Unlimited Context in LLMs

Apr 20, 2024, 03:52 PM

Google has introduced a groundbreaking technique in machine learning called 'Infini-attention', which enables handling of unlimited context in language models while maintaining a fixed-size memory footprint. This advancement, detailed in their latest papers 'Mixture of Depths' and 'Efficient Infinite Context Transformers with Infini-attention', aims to enhance the processing capabilities of pre-trained language models (LLMs) by applying local attention to input chunks and sequentially retaining relevant information. The development is seen as a significant step towards achieving very long context lengths, sufficient for most industry use cases.