← Back to archive

NOLIMA Shows LLMs Struggle With Long Context Reasoning

14 February 2025 · 355 views · 5 likes · 0 reposts · 131 blog views

AI learning product rag research

Came across an interesting paper called NOLIMA: Long-Context Evaluation Beyond Literal Matching which analyzes how well large LLMs handle long-context reasoning.

The paper critiques current LLM evaluation methods that focus on keyword matching in long texts. This doesn't truly test comprehension. The paper presents a benchmark requiring models to infer answers, demanding deeper associations instead of easy keyword matches.

What stood out to me:

1️⃣ Performance Drops as Context Length Increases. Even top models like GPT-4o, which start with near-perfect accuracy in short contexts, struggle when the text expands. At 32K tokens, most models drop below 50% of their short-context performance. That’s a huge decline! 2️⃣ The research shows that many models rely heavily on surface-level keyword matching. When those cues are removed, performance plummets. This has big implications for real-world applications like RAG, where models need to synthesize information rather than just match words. 3️⃣ The benchmark includes questions that require two-step reasoning (e.g., linking a city to a landmark and then to a person). Models that perform well in short contexts struggle here. 4️⃣ The authors tested what happens when the haystack includes "distracting" phrases that share keywords with the question but aren’t relevant. Models are often misled by superficial overlaps rather than engaging in true reasoning.

If you’ve ever worked with LLMs for summarization, document search, or chatbot applications, you probably know that longer contexts often lead to incomplete responses. This paper explains why - models aren’t always able to hold onto relevant details as context length grows.

It also challenges the idea that increasing a model’s context window (e.g., from 128K to 1M tokens) automatically improves long-context understanding. Without better ways to test reasoning, just stuffing more text into the input isn’t enough.

NOLIMA Shows LLMs Struggle With Long Context Reasoning

Originally posted on X

PS: I'm pretty active on 𝕏 if you'd like to follow along

Related posts

Archive

2026

2025

2024

2018