Back to the research
Context Engineeringlong contextdocument evaluation

Test Context Capacity with the Work You Actually Need

RULER offers an evaluation pattern beyond finding one fact in a long document.

How much context can your model use reliably?

Test the work you need it to perform at several input lengths. A model accepting a long document does not establish that it can retrieve every relevant fact, connect related facts, or calculate a complete answer from that document. RULER provides a useful testing pattern, while explicitly warning that synthetic tests need validation on realistic work.

RULER changes the task as well as the length

RULER's August 2024 revision tests retrieval, following chains of references, aggregation, and question answering. It varies input length and task complexity. Its error analysis includes incomplete retrieval, confusion from distracting material, and failures to combine information.

The paper's effective context length is a defined benchmark quantity. It is the longest tested input that exceeds a selected baseline score. It is not a universal capacity for all tasks. The authors also state that they did not establish how well all synthetic tasks predict real long-document work.

The historical model rankings should remain historical. The useful application today is to test several operations on the model and configuration you actually use.

Turn a document workflow into checkable questions

Consider an illustrative founder reviewing customer interview notes. The desired output is a list of recurring purchasing objections, supported by the relevant interviews. Finding one mention of price does not complete that task.

A proposed evaluation could include several kinds of question. Ask for one exact detail. Ask for all interviews that express a particular objection. Ask which later note changes an earlier interpretation. Ask for a summary that preserves an important disagreement rather than averaging it away.

Write the expected answer before running the model. Link each part of that answer to a passage in the source material. This creates a way to identify omissions as well as incorrect additions.

Begin with a compact set of notes. Add related but irrelevant notes while keeping the required evidence and question the same. Then add a separate version with a relevant correction. These changes test different problems and should be recorded separately.

Keep the prompt and output allowance stable during the first comparison. If the model changes at the same time as the input, you lose a clear explanation for the difference.

Inspect the errors before reducing the input

If an answer misses a detail, check whether that detail was available in the actual input. A document-loading failure requires a different fix from a reasoning failure.

If the evidence was present, inspect whether the answer confused entities, missed one of several relevant passages, or used an outdated statement. A shorter input may help, but it should retain the material required for the task. Removing contradictory evidence can make an answer look cleaner while making it less accurate.

This evaluation is a proposed adaptation of the research pattern. It does not establish a fixed token limit that every team should adopt. A token is a unit of text processed by the model, and the useful limit depends on the task and the tested system.

Choose a working range from observed performance

  1. Write checkable questions that require different operations on your documents.
  2. Repeat them as input length and distracting material increase.
  3. Choose an input strategy from the errors, completeness, and waiting time you observe.

Repeat the comparison when the model or document workflow materially changes.

← Explore more research