Lost in the Middle: How Language Models Use Long Contexts
Research paper examining how model performance changes with relevant information position in long inputs.
← Back to Compute College
Compute College
Long context is not the same as good context
Learn why adding more tokens can increase cost and distraction without improving an AI answer.
Plain-English definition
Long context means supplying a large amount of information to a model. Good context means supplying the information necessary for the current decision in a clear, authoritative, and manageable form. The two can overlap, but a large context can contain duplicates, stale passages, conflicting instructions, and irrelevant material that consumes tokens without improving the answer.
Memory trick: A bigger library is not a better brief. Context should be useful, not merely available.
Why it matters
As context grows, the application pays for more input processing and may increase latency or memory pressure. The model may also have difficulty locating the relevant evidence among many similar passages. Summarization, retrieval, filtering, and state management can produce a smaller context with better signal.
- More context can increase the opportunity for conflicting or stale information.
- Repeated history and documents create recurring token cost on every request.
- A shorter context with citations and clear source order can be easier to evaluate and trust.
Simple example
A research assistant receives 80 pages of a report to answer one question. A better pipeline retrieves the relevant sections, includes the report date and headings, and preserves a short summary of the broader finding. The answer uses fewer tokens while keeping the evidence needed to check the claim.
- Measure whether the answer changes when irrelevant material is removed.
- Preserve enough surrounding context to avoid quote-level misinterpretation.
- Treat summaries as derived context that can lose important caveats.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Long-context research reference
The Lost in the Middle research paper documents how language models can use information less effectively when relevant material appears in the middle of long contexts. Results vary by model and task, but the paper is a useful reason to test context placement rather than assuming length solves retrieval.
Source discipline: the reference explains the concept or method; it is not a substitute for measuring a production workload under its actual provider, model, and data conditions.
Common mistake
A model that can accept the whole corpus does not necessarily need it. Capacity to fit text is not evidence that the model will use every passage accurately or that the request is economically sensible.
Practical takeaway
What you can do with this
Take a long-context workflow and create three variants: full input, retrieved input, and summarized-plus-cited input. Compare grounded quality, input tokens, latency, and reviewer confidence on held-out questions.
- Builders: keep source order and citations when compressing context.
- Product teams: define when users prefer completeness versus speed.
- Operators: watch context cost as long-window features increase usage.
Decision check: can the application show that each added context block improves the accepted result enough to justify its recurring token and latency cost?
Compute College track
Prompt & Context Engineering
Step 15 of 18: Long context vs good context