Back to the research
The Agentic Webtask evaluationworkflow completion

Evaluate Long Web Tasks by Their Delivered Results

Odysseys separates partial progress from complete delivery and exposes limits in efficiency comparisons.

How should you judge an agent doing a long web task?

Define the required deliverable and inspect whether its parts are complete. Also record the time spent. A long browsing history can contain useful progress while leaving the requested result unfinished. The Odysseys benchmark offers a way to examine partial progress without treating it as full completion.

What the benchmark adds

Odysseys, published in April 2026, evaluates long web tasks using separate completion criteria. Its tasks combine participant browsing histories with generated and author-written work. Detailed criteria agreed better with human judgments than a single overall automated judgment in the reported comparison.

Its efficiency measure divides achieved progress by model steps. A step can contain different numbers of actions, so that measure is not interchangeable with elapsed time. Live websites can also change between runs. These limits matter when adapting the evaluation to a business workflow.

Specify the result before the browsing begins

Consider an illustrative request to compare venues for a workshop. The result might require a shortlist with capacity, availability, total quoted cost, and links to the evidence. The user may also require a document saved in a particular folder.

Write each required result as a checkable statement. A venue should not pass the availability check simply because its website loads. A drafted document should not pass the delivery check until it exists in the requested location and can be opened.

Separate required results from optional improvements. An attractive cover image may improve presentation, but it should not obscure a missing cost or an unverified date. The owner of the task should decide which results are essential before the evaluation.

This is a proposed evaluation design. The venue example is invented and was not a reported experiment in the paper.

Record partial progress without hiding unfinished work

For each criterion, record the evidence and whether it passes. Keep a separate field for complete task success. That field should remain false while any required result is missing.

Also record why a result is unavailable. A venue may not publish its price. The agent may have failed to find a published price. Those are different situations. The final document should distinguish unavailable information from an unsuccessful search.

A reviewer can then decide whether the next action is further research, a narrower request, or direct contact with the venue. The report remains useful without pretending that all work is finished.

Compare completion and time together

Use the same required criteria when comparing two agent configurations. Record elapsed time and the amount of human correction. Keep the original outputs so a reviewer can check whether an apparent speed improvement came from omitted work.

If a run takes longer, inspect what it accomplished during that time. More time may produce additional verified results, or it may repeat earlier searches. Do not infer either explanation from duration alone.

  1. Define the final artifact, its destination, and each required result.
  2. Grade those results against observable evidence, retaining missing items.
  3. Compare complete delivery, elapsed time, and human correction before choosing a configuration.

The useful question is whether the agent delivered the result the user can use. Its activity record helps explain that outcome.

← Explore more research