Back to the research
The Agentic Webvideo evidenceweb agents

Check Whether a Web Agent Can Inspect Video Evidence

WebArena-Pro shows why media access and correct interpretation need separate checks.

Can a web agent answer a question about a video?

Only if it can access the relevant evidence and interpret it correctly. A page title, thumbnail, or transcript may not contain the visual event the question asks about. WebArena-Pro tests this distinction by giving agents different ways to inspect audio and video within browser tasks.

Media access changed the benchmark results

WebArena-Pro, released in November 2026, uses self-hosted applications with resettable data. Agents can receive tools for transcripts and extracted video frames, or direct audio and video input. The paper compares performance with and without those tools.

On a subset selected to require media content, Claude Code with Opus 5.5 rose from 14.0% task success without media tools to 48.8% with transcript and frame tools. The subset was selected after the main collection, which limits how broadly that comparison should be interpreted.

The failure analysis also describes an agent that successfully extracted frames but counted a visual event incorrectly. Access was necessary for that task, yet access alone did not ensure an accurate answer. The benchmark covers prerecorded material and particular tool configurations, not every video workflow.

Match the evidence to the question

Consider an illustrative product demonstration. A researcher asks whether a presenter physically removed a component before replacing it. A spoken explanation may describe the replacement without showing the removal. A thumbnail cannot establish the sequence.

For a proposed review process, first identify what kind of evidence the question requires. Spoken claims require the relevant audio or a reliable transcript. A visible action requires the relevant frames. A question about order requires enough of the sequence to distinguish what happened first.

Keep these evidence types distinct in the notes. A transcript can support what the presenter said. It should not silently become proof of what the presenter did on screen.

If the agent cannot access the relevant interval, its answer should state that gap. If it can access only sparse frames, it should avoid treating an unseen event as an event that never happened.

Make the review reproducible

Save the original media URL and the relevant time interval with the answer. Record whether the answer used audio, a transcript, selected frames, or continuous video. A reviewer should be able to inspect the same evidence.

For a counting task, write down what counts as one occurrence. For example, distinguish a continuous appearance from a new appearance after a cut. This is a proposed evaluation rule for that task. It prevents an ambiguous question from producing a misleading accuracy score.

Test examples where speech and visible action differ. Include an example where the evidence is insufficient. Compare whether the agent explains those limits correctly, rather than rewarding a confident answer to every question.

Verify the answer at the relevant moment

  1. Define whether the question needs spoken, visual, or sequential evidence.
  2. Confirm that the agent's tools can access the required interval.
  3. Review the supporting interval and the final answer together.

For a publisher, provide a useful transcript and clear media context where appropriate. Treat any resulting improvement in agent answers as something to test, not a guaranteed discovery or citation benefit.

← Explore more research