Give an LLM Agent a Browser

The best way to Optimize Vector Search When RAM Will get Too Costly: On-Disk vs. In-Reminiscence ANN Indexes

Tabular LLMs: An Introduction to the Basis Fashions That Predict Your Spreadsheet

of helpful work nonetheless occurs inside net interfaces.

To resolve a ticket, the assist crew wants to make use of an admin console. An operations crew depends on dashboards to maintain observe of alarms. Earlier than talking with a buyer, a gross sales rep will most likely test the CRM first.

For LLM brokers to turn into helpful in these workflows, they should work straight via the browser.

On this publish, we’ll construct a browser-using agent with OpenAI Brokers SDK and Playwright MCP. We’ll first take a look at the loop behind browser use, after which put it into apply via a concrete case examine.

1. The Psychological Mannequin

At a excessive degree, a browser-using agent is finest understood as an LLM positioned in an interplay loop with a browser.

The loop goes like this: the agent begins with a activity and the browser’s present state. The agent then interprets that state, decides what to do subsequent, and sends an motion again to the browser. That motion would produce a brand new browser state, which turns into the enter for the subsequent choice.

This loop continues till the agent believes that the duty is full.

To make the loop work, we’d like two connections between the agent and the browser:

An remark channel, in order that the agent can obtain the present browser state.
An motion channel, in order that the agent can work together with the browser.

For the remark channel, frequent decisions embody screenshots, structured web page info (e.g., Doc Object Mannequin or accessibility tree), or a mix of each.

For the motion channel, the agent might use mouse and keyboard controls, or goal particular web page parts, or subject higher-level browser instructions.

The alternatives of remark and motion are often unbiased, however in apply, two pairings are generally seen:

screenshot + coordinate-based mouse and keyboard actions
structured web page state + element-targeted browser actions

The primary pairing leans in the direction of basic laptop use, which extends past the browser to different desktop functions. The second is extra particular to browser use, because it takes benefit of the construction already out there inside an internet web page.

On this publish, we’ll deal with structured web page observations along with element-targeted browser actions.

2. Case Examine: Resolving a Buyer Help Request

For our case examine, we’ll construct a browser-using agent that resolves a buyer request via a web-based assist console.

2.1 Getting ready the Help Console

To create the browser surroundings, I vibe-coded a small buyer assist console. It’s a static net utility constructed with plain HTML, CSS, and JavaScript. There is no such thing as a utility backend or database: all the info lives within the browser.

We are able to serve the console domestically with Python’s built-in HTTP server:

cd toy_app
python -m http.server 8000 --bind 127.0.0.1

This makes the console out there at http://127.0.0.1:8000, which we later move to the agent as APP_URL:

Determine 1. A Screenshot of the shopper assist console. (Picture by creator)

The left aspect of the console works like a assist inbox. As soon as a case is chosen, the remainder of the console reveals the associated order, buyer context, and determination insurance policies. From there, the case might be resolved with an inside be aware. The console additionally has an audit log that retains observe of the replace.

The agent’s activity is straightforward: examine one incoming assist case and carry it via to decision, fully via this console interface.

2.2 Defining the Browser-Utilizing Agent

Subsequent, we configure the browser-using agent.

For the tech stack, right here we’ll use the OpenAI Brokers SDK to energy the agent runtime and Playwright MCP to attach the agent to the browser.

Our remaining, configured agent appears to be like like this:

# pip set up openai-agents
from brokers import Agent, ModelSettings
from openai.varieties.shared import Reasoning

agent = Agent(
    title="Help Console Browser Agent",
    mannequin="gpt-5.4",
    model_settings=ModelSettings(
        reasoning=Reasoning(effort="medium"),
    ),
    directions=AGENT_INSTRUCTIONS,
    mcp_servers=[playwright_server],
)

There are three items we have to unpack right here, i.e., the LLM consumer, the agent instruction, and the browser instruments.

First, we join the Brokers SDK to Azure OpenAI:

import os
from openai import AsyncAzureOpenAI
from brokers import (
    set_default_openai_api,
    set_default_openai_client,
)

azure_client = AsyncAzureOpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    api_version=os.environ["OPENAI_API_VERSION"],
    azure_endpoint=os.environ["OPENAI_API_BASE"],
)

set_default_openai_client(azure_client)
set_default_openai_api("responses")

We register the consumer with the Brokers SDK and configure it to make use of the Responses API.

Then, now we have the next instruction, which is deliberately minimal:

AGENT_INSTRUCTIONS = """
You're an agent that may work together with an internet browser.
""".strip()

We solely arrange the agent’s function. The precise activity will come later within the immediate despatched to the agent.

Subsequent, we have to arrange the Playwright MCP.

What’s Playwright MCP?
Playwright is a browser automation library. It drives an actual browser in your behalf and performs clicking, typing, navigating, and studying no matter’s on the web page.

MCP (Mannequin Context Protocol) is a typical approach to expose instruments to an LLM, so an agent can name them straight. Playwright MCP is what you get whenever you put the 2 collectively, i.e., Playwright’s browser capabilities, uncovered as instruments an agent can choose up and use.

What makes it attention-grabbing is the way it reveals the agent the web page. As a substitute of a screenshot, Playwright MCP defaults to an accessibility snapshot, which is principally a structured learn of what’s on the web page. Interactive parts like hyperlinks, buttons, and enter fields every get a reference ID the agent can goal straight. That’s precisely the structured remark and element-targeted motion pairing we talked about earlier.

Playwright MCP runs via Node.js. To put in Node.js, on Home windows:

winget set up OpenJS.NodeJS.LTS

On macOS:

brew set up node

And on Linux:

curl -o- https://uncooked.githubusercontent.com/nvm-sh/nvm/v0.40.5/set up.sh | bash
. "$HOME/.nvm/nvm.sh"
nvm set up --lts

Then confirm the set up:

node --version
npx --version

We are able to configure the Brokers SDK to begin the MCP server via the next npx command:

from brokers.mcp import MCPServerStdio

playwright_server = MCPServerStdio(
    title="Playwright MCP",
    params={
        "command": "npx",
        "args": [
            "-y",
            "@playwright/mcp@latest",
            "--browser",
            "chrome",
        ],
    },
)

Right here, npx retrieves and runs the newest Playwright MCP package deal. The -y flag routinely accepts the npx affirmation immediate, whereas --browser chrome tells Playwright which browser to launch. Chrome can also be Playwright MCP’s default browser, and whether it is already put in, no separate browser set up is required.

MCPServerStdio configures the Brokers SDK to launch Playwright MCP as an area course of. When the agent first calls a browser device, Playwright MCP opens a visual Chrome window and executes the requested browser motion.

2.3 Working the Agent

Now we may give the agent a concrete activity:

APP_URL = "http://127.0.0.1:8000"

TASK = f"""
Open {APP_URL} and resolve the assist case for order ORD-1042.

The shopper says they obtained the fallacious merchandise. Use the data out there within the
utility to find out and apply the suitable decision. Add a concise inside
be aware and ensure the decision was efficiently recorded.

Report what you probably did when the duty is full.
""".strip()

Within the activity immediate, we described methods to entry the applying and the specified final result.

We then run the agent with:

from brokers import Runner

async with playwright_server:
    consequence = await Runner.run(
        agent,
        TASK,
        max_turns=20,
    )

print(consequence.final_output)

The async with block begins the Playwright MCP course of and retains it related whereas the agent is working. We use max_turns to set an higher restrict on what number of turns the agent can take.

As soon as began, Chrome would open, and we will watch the agent work via the assist console.

The ultimate response summarized the result accurately:

Resolved CASE-4107 for order ORD-1042 with Alternative.

The case now reveals Resolved, the recorded motion is Alternative,
and the audit log incorporates the corresponding decision entry.

In case you like, you can even examine the browser-tool calls and their outputs this fashion:

for merchandise in consequence.new_items:
    print(sort(merchandise).__name__, merchandise)

In my run, the agent discovered the case related to ORD-1042 and it inspected the order, buyer request, stock standing, and related decision coverage. It then concluded {that a} substitute was applicable, added an inside be aware, and submitted the decision. Lastly, it checked the up to date case and audit log to substantiate that the motion had been recorded.

That is precisely the agentic conduct we wish.

3. From Browser Use to Laptop Use

What we constructed on this case examine is a browser-using agent. Nonetheless, the underlying sample, that’s, the observe, resolve, act, and repeat loop, naturally extends to basic laptop use.

What adjustments are the remark and motion channels.

In our case, Playwright MCP supplies the agent with structured web page info and permits it to focus on particular person net parts. A extra basic computer-use agent might as an alternative work from screenshots and management the mouse and keyboard by coordinates.