MIT Researchers Introduce ContextCite, LongCite-8B and LongCite-9B for Enhanced Language Model Attribution

Sep 4, 2024, 09:53 PM

Researchers from MIT, including B Cohen-Wang, H Shah, K Georgiev, and A Madry, have introduced a new model called ContextCite, aimed at improving the attribution of language model generations to specific parts of the provided context. The model learns a surrogate that approximates how a language model's response is affected by including or excluding each part of the context. This innovation addresses the challenge of fine-grained in-line citations in long-context scenarios, which current long-context language models struggle with. ContextCite synthesizes a large-scale supervised fine-tuning (SFT) dataset with off-the-shelf language models to enhance citation generation in long-context question answering (QA). The results indicate that ContextCite's models, LongCite-8B and LongCite-9B, outperform GPT-4o by 6.4% and 3.6% in citation F1 score, respectively, and offer 2x finer citation granularity compared to proprietary models. Additionally, there is a 7-9% improvement in response correctness.