How to Check Whether a Web Agent Finished

How can you tell whether a web agent finished?

Check the requested result, then check whether the system judging success is reliable. The 2024 WebVoyager study reported two separate measures: how often its agent completed benchmark tasks and how often its automatic evaluator agreed with people. That distinction matters when deciding whether to trust an agent with work on your website.

A benchmark result answers a bounded question

A web agent is software that interacts with websites to carry out a user's instructions. For a founder or researcher, that could mean delegating a sequence of browsing actions instead of performing each step manually.

WebVoyager used a large multimodal model—a model that can process more than one kind of input—to interact with real websites. The researchers assembled tasks from 15 popular websites and reported a 59.1% task success rate on that benchmark. The agent outperformed the study's GPT-4 (All Tools) and text-only WebVoyager setups, according to the paper's abstract.

The useful finding is bounded: this agent completed that share of tasks under the benchmark's conditions. The number does not establish the success rate on your checkout, research workflow or account dashboard. Nor does a study from 2024 establish how today's agents perform.

For a business considering automation, the practical question is narrower than whether an agent can browse: can it reliably produce the particular result the business needs?

An evaluator can disagree with people

WebVoyager also introduced an automatic evaluation protocol using GPT-4V's multimodal capabilities. Its judgments agreed with human judgments 85.3% of the time, according to the same abstract.

That figure describes agreement about outcomes. It is separate from the agent's 59.1% task success rate. An evaluator can correctly recognize that an agent failed; agreement does not mean successful execution.

The evidence here is limited to the paper's abstract, which does not describe how disagreements were distributed. We cannot tell from it whether the evaluator more often accepted failures or rejected successful attempts. That missing detail matters if an automated success label will trigger another action.

Define the result before watching the clicks

Consider an illustrative task: find three products that meet a buyer's requirements and save them to a shortlist. Opening three product pages shows activity. A completed task requires a saved shortlist containing three products that actually meet the requirements.

A useful test would inspect the saved shortlist and compare its contents with the original request. An agent's statement that it finished would be another claim to check.

This is a proposed evaluation practice, not a method whose effectiveness the WebVoyager abstract establishes. It follows from separating execution from judgment: make the intended outcome explicit enough that a person can inspect it independently.

Keep permission separate, too. Finding a product and purchasing it are different tasks. Evidence that an agent can navigate a website does not establish that the user authorized a transaction.

Test your workflow and its success judge

Before expanding a web agent's responsibilities:

1. Specify observable completion. Choose a small set of representative tasks and write down the result each must produce, including where an agent must stop.
2. Inspect the actual outcome. Compare the resulting page, saved item or other relevant record with the original request. Record completion separately from browsing activity.
3. Audit automatic judgments. Have a person review a sample of attempts, including those labeled successful. Track both accepted failures and rejected successes before relying on the evaluator to advance the workflow.

These checks turn a broad capability claim into a test of the work you intend to delegate.