I Broke a Browser Agent on Objective. Recognizing the Fault Was Not the Similar as Fixing It

WebFovea, a paper by Jiangang Han, studies a disquieting browser agent failure. A mismatch between screenshot and browser coordinates positioned clicks at three quarters of their supposed place. A click on meant for a degree 400 pixels throughout landed 300 pixels throughout.

The mismatch got here from the service that runs the mannequin, which shrank every screenshot earlier than the mannequin checked out it. The paper estimates {that a} 1920 by 1080 screenshot reached the mannequin as about 1440 by 810, which is three quarters of the scale. Anthropic’s imaginative and prescient documentation describes the identical habits for its personal fashions. It says massive pictures are scaled right down to cap their price, and that the mannequin solutions in positions on the resized image. The paper’s software program used these positions as in the event that they have been measured on the complete dimension display screen, so each click on landed at three quarters of the supposed spot.

The agent within the paper is a browser agent, a program that lets a man-made intelligence mannequin examine an internet web page and select actions corresponding to clicking a button. Coordinates, the numbers within the click on instance above, are positions on the display screen, given in pixels.

In addition to the press error, WebFovea studies two extra failures within the software program between the mannequin and the web page. Hidden finish of message markers from the mannequin have been typed into search containers. A click on inside a web page embedded in one other web page labored, however was reported as having no impact.

The paper fastened failures like these throughout 4 submissions to a 2026 contest on stay web sites. The agent’s rating rose from 31.0 to 57.0 out of 100 with the identical mannequin. Every submission bundled a number of adjustments, and the stay web sites might differ, so the rating alone doesn’t present which repair mattered most. That’s the hole I attempted to have a look at on a small scale.

To have a look at that hole, I examined what occurs after the mannequin makes an accurate alternative. I constructed a small browser web site with 4 quick duties, corresponding to trying to find a file and clicking a button. I requested Claude Haiku 5.5, a small, low price mannequin from Anthropic, to finish them.

I saved 80 of its choices that labored, then replayed every via browser code that I had intentionally damaged. A verify is a small piece of code that appears on the web page after an motion and decides whether or not the motion labored. I measured whether or not checks noticed the fault and whether or not the agent, that means the mannequin plus the code that drives the browser, nonetheless completed the duty.

Recognizing a fault and getting previous it turned out to be completely different issues. The faults matter to anybody who asks an AI software to click on via an internet site, together with internet builders and individuals who construct websites by describing them to an AI software. The WebFovea paper and its code motivated my setup. The outcomes under come from my very own browser pages and saved step-by-step logs.

The code, step-by-step logs, figures, and recording are public. The first check, the place I ran the checks on 40 variations of every web page, and the remaining agent runs help the counts under.

One proper determination, two completely different clicks

One of many 80 saved choices reveals the issue clearly. The check web page reveals a grid of numbered buttons, Merchandise 0 to Merchandise 24. The agent noticed a 900 pixel broad screenshot of it and was requested to click on Merchandise 15. It answered with the picture level (150, 413), that means 150 pixels throughout and 413 down from the highest left nook. The browser’s seen space, referred to as the viewport, was 1200 pixels broad.

I made the screenshot smaller on goal, to repeat the issue within the paper. Meaning the executor, the code that sends an motion to the browser, should scale every picture coordinate as much as browser dimension. Right here the browser is 1200 pixels broad and the picture is 900, so the size is 4 thirds. The working executor multiplied 150 by 4 thirds to get 200, and 413 by 4 thirds to get 550.67. It clicked (200, 550.67) in browser pixels and chosen Merchandise 15.

The damaged executor skipped that scaling and despatched (150, 413) instantly as browser pixels. That time fell on Merchandise 10, one row above Merchandise 15, so Merchandise 10 was chosen. I saved the mannequin’s reply and ran it twice, as soon as via every executor. The screenshot, instruction, and saved reply have been similar each instances.

The next code reveals the overall conversion. On this experiment, each picture dimensions have been three quarters of the matching browser dimensions, so every coordinate was multiplied by 4 thirds.

browser_x = image_x * viewport_width / image_widthbrowser_y = image_y * viewport_height / image_heightweb page.mouse.click on(browser_x, browser_y)

The press itself comes from Playwright, a software that lets code management a browser. Its mouse documentation specifies browser viewport pixels for a click on. The conversion belongs within the executor, the step between the mannequin’s reply and the browser. My 900 by 600 picture and 1200 by 800 viewport observe the proportions of the three quarter error in WebFovea. This can be a constructed instance on my web site, not a second commentary from stay web sites.

Determine 1. Picture by creator. Each frames come from the continual recording of 1 model of the check web page, utilizing the identical saved determination. The inexperienced circle marks the working click on on Merchandise 15. The pink circle marks the defective click on on Merchandise 10.

Watch the steady browser recording or the quick GIF.

What did I break, and what might the checks see?

To decide on what to interrupt, I used the paper’s method of describing a browser motion. The WebFovea paper treats each as a spherical journey. The motion travels from the mannequin to the web page, and the consequence travels again. The paper splits that journey into 4 steps.

A parser turns the mannequin’s reply into an motion. An executor sends the motion to the web page. Suggestions tells the mannequin what occurred. Commentary is what the mannequin is proven afterwards, normally a brand new screenshot.

Three phrases within the desk want a fast rationalization. An embedded body, or iframe, is a web page inside one other web page. A tooltip is the small label that seems beside a chart mark when the pointer rests on it. The textual content [im_end] within the desk under stands in for an finish of message marker that some AI fashions use. I added one deliberate fault at every step.

Within the Suggestions row, my embedded body comes from the identical web site as its mother or father web page, which is simpler as a result of the web page can learn its personal body instantly. The paper’s body instance concerned a tougher case throughout websites.

To see whether or not the checks caught these faults, I generated 40 layouts for every process, with completely different goal positions or values. A format is one association of the identical web page. Each verify was utilized to 40 defective actions and 40 working actions. The working actions matter as a result of a verify that all the time warns would catch each fault and likewise interrupt each appropriate motion.

To know whether or not a verify was proper, the web page stored its personal file. It logged the textual content entered, the button clicked, the body row chosen, and the tooltip seen when the screenshot was taken. That file was the reply key for counting detected faults. The checks couldn’t learn it whereas deciding whether or not to warn.

I in contrast 4 insurance policies, that means 4 methods of checking an motion. Unverified accepted the message the browser software despatched again to the mannequin, corresponding to “No detectable change”, as true. Generic seemed for adjustments within the internet deal with or the principle web page’s state. Pixel in contrast screenshots. End result verify in contrast the web page with the precise end result requested, corresponding to the precise discipline textual content, the chosen button, the row contained in the body, or whether or not the chart worth appeared.

The primary desk counts faults caught out of 40 defective actions. The second counts false alarms out of 40 working actions. The next depend is healthier within the first desk and worse within the second.

Desk 1. Faults caught within the first check, which used 40 layouts. Larger counts imply extra of the injected faults have been detected. Counts got here from the web page’s personal file.

Desk 2. Warnings on 40 working actions per process. Decrease counts imply fewer pointless warnings.

The easy checks failed in several methods. Generic missed 20 of 40 fallacious clicks. In these, the press hit a special button, which turned inexperienced, so the web page seemed as if it had responded. The opposite 20 clicks hit empty house, and Generic caught these. It additionally warned on all 40 appropriate body clicks as a result of the principle web page didn’t change when the embedded web page did.

Pixel had a special weak point. It discovered solely three corrupted searches and warned on all 40 working searches, as a result of the web page appears to be like completely different after each search. A change in pixels doesn’t show that an actual question survived parsing.

Against this, End result verify discovered all 40 faults in every process and made no false alarms on the matching working actions. I wrote each the checks and the web page code used as the reply key. The checks have been constructed for these faults, so excellent detection right here proves little a few completely different web site. A mistake in my considering might due to this fact sit in each.

Determine 2. Screenshot by creator. The parser provides [im_end] to the search time period File-10. The sector adjustments visibly, however an actual search wants the unique identify.

Did recognizing a fault assist the agent end?

Recognizing a fault was solely half of the query. I used the primary 20 of the 40 layouts for a second experiment, and glued that alternative earlier than any mannequin name. Haiku first made one determination on a working web page for every of the 80 mixtures of process and format, 4 duties instances 20 layouts. Each a kind of choices labored, and I stored all 80.

I replayed every recorded determination with the fault enabled, then let the mannequin see the consequence, as every coverage would report it, and reply. An episode is one try to finish a process. The mannequin might cease, retry, or wait, that means pause so the web page might end altering. It had as much as three browser actions.

This comparability used Unverified, Generic, and End result verify. Pixel was evaluated for fault detection however not for process completion, so this experiment can not say whether or not it could have helped the agent end. Every coverage ran 3 times on the identical 20 layouts for every of 4 duties, yielding 4 × 20 × 3 × 3 = 720 remaining episodes, from 4 duties, 20 layouts, three insurance policies, and three runs. I scored an episode as profitable when the requested motion reached its goal, or when the chart reply got here from a screenshot that truly confirmed the worth.

One alternative shapes the outcomes under. I left the search parser and click on executor damaged on each retry, as a result of fixing code just isn’t one thing the mannequin can do. This reveals what higher data alone can provide it. A verify might describe their errors, however one other motion via the identical code would make the identical mistake.

Determine 3. Picture by creator. Every depend is profitable episodes out of 60 for one process and coverage. Larger means extra duties accomplished. The press and search faults continued underneath all insurance policies; both verify gained 4 chart completions over Unverified.

Desk 3. Profitable episodes from the ultimate run. Every cell comprises 20 layouts repeated 3 times. Larger counts imply extra accomplished duties.

That persistence issues when studying the zeros. The zeros have been anticipated, and so they present what a analysis does and doesn’t give the mannequin. On one search, End result verify learn File-10[im_end] again from the sphere and instructed the mannequin it differed from File-10. A retry nonetheless went via the identical parser.

The press episodes present one thing much less apparent. In all 60 End result verify episodes, the verify reported the merchandise hit and provided the goal’s appropriate browser place. The directions nonetheless instructed the mannequin to reply in picture coordinates, and it repeated its unique level all 3 times. The unchanged executor due to this fact despatched three fallacious clicks in each episode.

On the body process the consequence was the other. The browser software stated “No detectable change,” but the web page chosen the proper row. All three insurance policies accomplished all 60 episodes.

Generic instructed the mannequin nothing had modified, as a result of it watched solely the principle web page, and in 4 of 60 episodes the mannequin clicked a second time for no purpose. End result verify learn the body and wanted no additional click on. The browser software’s message and the web page disagreed, however the screenshot confirmed the row chosen, so the mannequin might end.

Determine 4. Screenshot by creator. The End result verify studies the chosen row whereas uncooked software suggestions says “No detectable change.” The embedded body comes from the identical web site.

The chart process, the place the screenshot got here earlier than the tooltip appeared, was the one one with a completion distinction. Unverified succeeded 56 instances out of 60. In 4 episodes it stopped earlier than a screenshot confirmed the worth. Both verify led the mannequin to hover once more or wait, then take a brand new screenshot, and it succeeded in all 60.

Determine 5. Screenshot by creator. These are separate coverage episodes on the identical format. Unverified stopped earlier than the worth appeared in its screenshot; Generic repeated the hover and noticed 89.

The acquire is 4 of 60 episodes, or 6.7 proportion factors. To see how a lot the acquire might differ, I redrew the 20 layouts many instances, holding every format’s three runs collectively. The center 95 p.c of the redrawn positive aspects ran from 0 to fifteen factors. Zero acquire remains to be potential, so this small consequence doesn’t set up a dependable benefit. Generic matched End result verify on completion, so the less complicated verify was sufficient on this process.

Adjustments and limits that have an effect on the comparability

These adjustments and limits matter when evaluating the ultimate counts with earlier runs or making use of them to different browser brokers.

A improvement run confirmed that some mannequin replies have been minimize off earlier than the motion textual content was full. I switched to a hard and fast reply format earlier than the ultimate outcomes. I additionally stopped the primary try, which ran one episode after one other, after 13 episodes as a result of finishing all 720 that method would take too lengthy. These 13 search episodes had zero successes and are archived, then excluded from the ultimate desk.

I reran the complete set in 4 separate teams of 180 episodes. The prompts, faults, checks, scoring guidelines, and 20 chosen layouts stayed the identical throughout that schedule change.

Past that schedule change, the examine has limits. The 20 layouts within the full run have been a subset of the 40 layouts within the first check. The examine used one easy web site, one mannequin and immediate, intentionally injected faults, and a body from the identical web site. Every remaining episode started with a call already identified to work on a clear web page. These counts measure restoration on this setup, not the frequency of such failures on public web sites or accuracy on tougher searching duties.

The experiment was low-cost to run. The ultimate loop used about 24 US cents in logged mannequin calls. All logged mannequin work for this text, together with improvement and the stopped first try, totaled about 27 US cents.

What does this imply if you happen to check or construct web sites with AI instruments?

The check was small, however the WebFovea paper studies all 4 sorts of fault on stay web sites. That makes them value understanding about if you happen to ask an AI software to click on via a web site. This is applicable to internet builders who check pages this manner, and to individuals who construct websites by describing them to an AI software, typically referred to as vibe coding. The desk under turns every consequence from the check into one thing you are able to do.

Desk 4. What every consequence means for somebody who assessments or builds with AI instruments. Every row comes from a consequence reported above.

These are ideas from one small check, so verify them towards your individual web site and instruments. One thought goes past what I measured. Should you construct a web site, exhibiting an essential worth within the web page textual content, and never solely on hover, might make it simpler for a screenshot based mostly software to learn. I didn’t check that, so deal with it as a guess.

A final suggestion is for anybody who builds or checks a web site with an AI software. If the software says it clicked a button however the web page appears to be like the identical, ask it to learn the web page again and say what modified. That’s what the End result verify did on this experiment.

What would I verify in a browser agent?

These faults all occurred between the mannequin’s reply and the web page. So I’d save the mannequin’s request, the picture and browser dimensions, and the motion truly despatched to the browser. I’d additionally file the aspect or body that acquired it and the commentary proven to the mannequin.

The press conversion above makes one step specific, the transfer from picture place to browser place. An actual search additionally wants the textual content learn again from the search field. A body click on wants the state of that body, and a chart reply wants a screenshot that comprises its worth.

Checks can reveal a mismatch and may also help when one other commentary provides lacking proof. They can’t appropriate a persistent parser or executor fault until the agent or its surrounding code adjustments how the following motion is distributed. When an agent seems to purpose badly, examine the motion’s path via the browser earlier than altering the immediate.

After a failed click on, would your logs present the coordinates the mannequin requested for, the purpose the browser truly acquired, or solely the ultimate screenshot?

Chosen sources

• WebFovea paper, for the reported stay web site failures and 4 step framing.

• Anthropic imaginative and prescient documentation, for the way massive pictures are resized earlier than the mannequin sees them, and the way the mannequin’s coordinates relate to the resized picture.

• Playwright Mouse API, for the browser pixel conference.

• Experiment code and uncooked outcomes, together with the browser pages, step-by-step logs, figures, and recording.