Back to the research
Context Engineeringagent memoryevaluation

Test Agent Memory with Failed Attempts and Changed Conditions

LongMemEval-V2 separates useful recalled evidence from live task success.

How do you test whether an agent remembers useful experience?

Ask questions that require evidence from its earlier work, including failed attempts and changed conditions. Check whether the agent retrieves the right evidence and answers correctly. LongMemEval-V2 tests this kind of memory, but its question-answering scores do not show that an agent will complete a live business workflow.

The benchmark tests more than remembered facts

LongMemEval-V2, published in May 2026, evaluates memory using recorded browser histories. Its questions cover interface facts, changes in state, workflows, recurring problems, and false assumptions. Some questions depend on failed attempts. A memory system returns evidence to a fixed reader model, which produces the answer.

In the paper's Table 2, the strongest overall method, AgentRunbook-C, achieved 74.9% on the smaller history collection and 70.1% on the larger one. It used a coding agent to inspect files and supporting artifacts. Those gains came with much longer query times than simple retrieval.

The aggregate also hides a tradeoff. AgentRunbook-C scored below the plain coding-agent baseline on questions about recurring problems in both collections. Better overall memory was not better on every kind of question. The benchmark uses stored histories rather than an agent learning during live work.

Separate a missing memory from a misread memory

Consider an illustrative agent that updates a customer record. An earlier attempt failed because the interface required saving a related field first. A later question asks how to complete the update.

There are several distinct ways to fail. The memory system may retrieve no relevant attempt. It may retrieve a similar attempt from another interface. It may retrieve the correct attempt but omit the error message. Or the answering model may receive the full evidence and still give the wrong order of steps.

Give these failures separate labels in a proposed evaluation. Otherwise, a team may improve its search system when the actual problem is how the evidence is presented or interpreted.

For every question, keep an expected answer and the original observation that supports it. Include the time and environment. A workflow that worked in an earlier interface version should not silently become a permanent rule.

Test whether old experience applies

Add questions where the apparent premise is wrong. For example, ask where to edit a field that the current interface does not expose. The expected response should identify the mismatch and point to the evidence. Guessing a familiar menu path should count as an error.

Also include a case where the earlier failure was temporary. A remembered outage should help explain history without preventing a valid action after service returns.

These are proposed tests for your own system. They require verified examples from that system. The paper does not establish that a particular memory format will solve them in every application.

Check accuracy and waiting time together

  1. Build questions from successful work, failed attempts, and changed conditions.
  2. Inspect the retrieved evidence before assigning the cause of a wrong answer.
  3. Compare answer accuracy, failure types, and waiting time on the same questions.

Retain the simpler memory method if it meets the task's needs. Add more involved evidence inspection where the measured errors justify it.

← Explore more research