On this article, you’ll learn to construct AI brokers that may browse and work together with actual web sites utilizing Playwright, browser-use, and LangGraph.
Subjects we are going to cowl embody:
- Why Playwright is the best basis for browser automation in 2026, and the way it differs from Selenium.
- Learn how to scrape dynamic, JavaScript-rendered pages and full multi-step varieties reliably.
- Learn how to wire browser actions into LangGraph and browser-use brokers, deal with anti-bot detection, handle ready and session persistence, and deploy the lead to Docker.
Constructing Browser-Utilizing AI Brokers in Python
Introduction
Most AI agent tutorials begin with an API. They present you the best way to name OpenWeather, hit the Stripe endpoint, pull knowledge from GitHub. That could be a positive start line till you attempt to construct one thing actual and understand that the duty you really need completed doesn’t have an API.
Take into consideration what people do with browsers on daily basis: submitting authorities varieties, studying competitor pricing, extracting analysis from websites that guard their knowledge behind JavaScript rendering, logging into portals which have by no means heard of OAuth. There are roughly 1.1 billion web sites on the web. A vanishingly small fraction of them have public APIs. The remainder solely converse browser.
An agent that’s restricted to API calls handles perhaps 5% of the duties a human employee does day by day. Give that agent a browser, and the protection approaches all the pieces. That’s the hole this text closes.
The international AI brokers market stands at $10.91 billion in 2026 and is projected to succeed in $50.31 billion by 2030, with browser-capable brokers on the heart of that development. 27.7% of enterprises are already working agentic browsers in manufacturing, up from just about none two years prior. The tooling has matured quick, and the patterns are settled sufficient to show correctly.
By the tip of this text, you should have a working browser agent that navigates actual web sites, fills varieties, extracts structured knowledge, and connects to an LLM that decides what to do subsequent, all in Python.
Why Playwright, Not Selenium
When you constructed browser automation 5 years in the past, you constructed it with Selenium. Selenium continues to be extensively deployed, nonetheless works, and isn’t going wherever. However for any new venture in 2026, Playwright is the default. The explanations are sensible, not theoretical.
Selenium communicates with the browser by sending particular person HTTP requests to a WebDriver. Each motion, click on, kind, scroll, is a separate request. Playwright makes use of a persistent WebSocket connection for the whole session. Instructions circulate by means of that channel with no per-action round-trip value. Impartial benchmarks persistently present Playwright working 30-50% quicker than Selenium on the test-suite degree and averaging ~290ms per motion versus Selenium’s ~536ms. For a browser agent that may execute tons of of actions, that hole compounds.
Playwright additionally bundles its personal browser binaries. If you set up it, you get pre-configured variations of Chromium, Firefox, and WebKit which are assured to work together with your Playwright model. No driver model mismatches, no damaged CI pipelines as a result of somebody up to date Chrome. It has built-in auto-waiting earlier than it clicks a component; it verifies the aspect is seen, enabled, and never animating. You do not need to put in writing time.sleep(2) and hope for one of the best.
For AI brokers particularly, Playwright fires actual mouse and keyboard occasions that mirror how people work together with browsers. Websites designed to detect automation search for artificial DOM clicks. Playwright’s interplay mannequin is more durable to tell apart from real human enter.
There’s additionally the browser-use library, which sits one degree increased. Browser-use is a Python library that provides an LLM a working browser. Below the hood, it makes use of Playwright to drive the browser, however the LLM reads the web page state and decides what to click on, kind, and extract, no CSS selectors required. You give it a activity in plain English, and it figures out the remainder. We are going to cowl each uncooked Playwright and browser-use on this article, as a result of they serve completely different wants: Playwright once you need exact, predictable management; browser-use once you need the agent to deal with navigation choices autonomously.
Setting Up the Surroundings
You want Python 3.10 or increased, an OpenAI API key, and about 5 minutes.
Step 1: Create a digital atmosphere
|
python –m venv browser_agent_env
# macOS / Linux supply browser_agent_env/bin/activate
# Home windows browser_agent_envScriptsactivate |
Step 2: Set up dependencies
|
pip set up playwright browser–use langchain langchain–openai langgraph langchain–neighborhood python–dotenv |
Step 3: Set up the browser binaries
That is the step most individuals miss. Playwright must obtain Chromium, Firefox, and WebKit individually from the Python bundle. Run this as soon as after putting in:
|
playwright set up chromium |
If you need all three browser engines: playwright set up. Chromium alone is adequate for many agent work and is smaller to obtain.
Step 4: Retailer your API key
Create a .env file in your venture listing:
|
OPENAI_API_KEY=your_openai_api_key_here |
Add .env to your .gitignore instantly. Don’t commit API keys.
Step 5: Confirm all the pieces works
Here’s a first script that navigates to a URL, reads the heading, and saves a screenshot. Use instance.com, a publicly obtainable check area maintained by IANA that won’t block you.
Learn how to run: Save as first_run.py and run python first_run.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
# first_run.py # Navigate to a URL, take a screenshot, and extract the web page title. # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python first_run.py
import asyncio from playwright.async_api import async_playwright
async def essential(): async with async_playwright() as p: # Launch Chromium in headless mode (no seen browser window). # Set headless=False if you wish to watch it run throughout growth. browser = await p.chromium.launch(headless=True)
# A browser context is sort of a contemporary browser profile. # It isolates cookies, storage, and cache from different contexts. context = await browser.new_context( viewport={“width”: 1280, “top”: 720}, user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/120.0.0.0 Safari/537.36” ) )
web page = await context.new_page()
# Navigate to the URL and wait till the community is idle. # “networkidle” means no open community connections for 500ms. # For quicker pages, “domcontentloaded” is adequate. await web page.goto(“https://instance.com”, wait_until=“networkidle”)
# Extract the web page title title = await web page.title() print(f“Web page title: {title}”)
# Extract the textual content content material of the h1 heading h1 = await web page.text_content(“h1”) print(f“H1 heading: {h1}”)
# Take a full-page screenshot and put it aside to disk await web page.screenshot(path=“screenshot.png”, full_page=True) print(“Screenshot saved to screenshot.png”)
await browser.shut()
asyncio.run(essential()) |
What this does: async_playwright() is the entry level for the whole Playwright session. The browser_context is equal to opening a contemporary incognito window; cookies, native storage, and cache are remoted from all the pieces else. wait_until=”networkidle” tells Playwright to attend till the web page has completed all its community exercise earlier than your code continues, which is the most secure wait technique for dynamic pages.
If this runs and saves a screenshot, your atmosphere is working accurately.
Internet Navigation and Scraping
The explanation you want Playwright as a substitute of requests + BeautifulSoup is JavaScript rendering. Fashionable web sites ship a skeleton of HTML after which construct the precise content material dynamically after the web page masses: React, Vue, Angular, Subsequent.js. A plain HTTP request fetches the skeleton. Playwright runs an actual browser, so it sees precisely what a human sees in spite of everything JavaScript has executed.
The goal beneath is books.toscrape.com, a authorized scraping sandbox constructed for observe. It paginates outcomes, makes use of dynamic class names for scores, and intently mirrors the construction of actual e-commerce product pages.
Learn how to run: Save as scrape_books.py and run python scrape_books.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
# scrape_books.py # Scrape guide titles, costs, and scores from books.toscrape.com # This can be a authorized scraping sandbox website constructed for observe. # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python scrape_books.py
import asyncio import json from playwright.async_api import async_playwright
async def scrape_books(max_pages: int = 3) -> listing[dict]: “”“ Scrape guide listings from books.toscrape.com throughout a number of pages. Returns an inventory of dicts with title, value, ranking, and web page quantity. ““” outcomes = []
async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context(viewport={“width”: 1280, “top”: 720}) web page = await context.new_page()
for page_num in vary(1, max_pages + 1): url = f“https://books.toscrape.com/catalogue/page-{page_num}.html” print(f“Scraping web page {page_num}: {url}”)
await web page.goto(url, wait_until=“domcontentloaded”)
# Await the product playing cards to be seen earlier than extracting. # That is vital on JavaScript-heavy pages the place content material masses after the HTML. # timeout=10000 means wait as much as 10 seconds earlier than elevating an error. await web page.wait_for_selector(“article.product_pod”, timeout=10000)
# Get all guide playing cards on the present web page books = await web page.query_selector_all(“article.product_pod”)
for guide in books: title_el = await guide.query_selector(“h3 a”) title = await title_el.get_attribute(“title”) if title_el else “N/A”
# Extract value textual content price_el = await guide.query_selector(“.price_color”) value = await price_el.inner_text() if price_el else “N/A”
# Extract star ranking from the CSS class title. # e.g.
rating_el = await guide.query_selector(“p.star-rating”) rating_class = await rating_el.get_attribute(“class”) if rating_el else “” ranking = rating_class.exchange(“star-rating”, “”).strip()
outcomes.append({ “title”: title, “value”: value, “ranking”: ranking, “web page”: web page_num })
print(f” Extracted {len(books)} books from web page {page_num}”)
await browser.shut()
return outcomes
async def essential(): books = await scrape_books(max_pages=2) print(f“nTotal books scraped: {len(books)}”) print(json.dumps(books[:3], indent=2))
asyncio.run(essential()) |
What this does: wait_for_selector() is the important thing name right here. As an alternative of sleeping for a set time and hoping the content material has loaded, it watches the DOM and proceeds the second the goal aspect seems, or raises a TimeoutError if it doesn’t seem inside the timeout window. That’s the proper habits: fail quick and explicitly quite than silently extracting from an empty web page.
The ranking extraction deserves consideration. The star ranking is encoded as a CSS class (star-rating Three), not a quantity. The code strips “star-rating” from the category string to get the textual content worth. That is the form of factor you solely know by inspecting the precise HTML. If you hand this activity to a uncooked LLM with no browser, it has no method to know what the category construction seems to be like. With Playwright, you’ll be able to examine it immediately and extract it precisely.
Type Completion and Multi-Step Flows
Filling varieties is the place browser brokers earn their maintain and the place most automation scripts fail. The reason being that internet varieties will not be simply inputs and buttons. They hearth focus, enter, change, and blur occasions in sequence. JavaScript validation listens for these occasions. When you inject a worth into an enter subject by immediately setting worth within the DOM (as older automation instruments usually do), the validation listeners by no means hearth and the shape breaks.
Playwright’s fill() and click on() strategies hearth actual browser occasions in the best order, which is why they work on type validation that may block lower-level approaches.
The goal beneath is the-internet.herokuapp.com/login, a public check website maintained particularly for automation observe. It accepts tomsmith / SuperSecretPassword! as legitimate credentials and returns clear success/failure messages.
Learn how to run: Save as form_submit.py and run python form_submit.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
# form_submit.py # Full and submit a multi-field login type on a public demo website. # Goal: https://the-internet.herokuapp.com/login (public check website) # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python form_submit.py
import asyncio from playwright.async_api import async_playwright
async def login_and_verify(username: str, password: str) -> dict: “”“ Try to log in to a demo website and return whether or not it succeeded. Handles: enter filling, button clicking, and consequence verification. ““” async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context() web page = await context.new_page()
await web page.goto(“https://the-internet.herokuapp.com/login”)
# Await the shape to be seen earlier than interacting. # state=”seen” is the default however makes the intent specific. await web page.wait_for_selector(“#username”, state=“seen”)
# fill() clears the sphere first, then varieties the worth. # It fires the main focus, enter, and alter occasions so as. await web page.fill(“#username”, username) await web page.fill(“#password”, password)
# click on() fires actual mouse occasions — mousedown, mouseup, click on. # This triggers JavaScript listeners {that a} plain DOM click on misses. await web page.click on(“button[type=”submit”]”)
# Await the web page to settle after type submission await web page.wait_for_load_state(“networkidle”)
# Test which consequence aspect appeared success_el = await web page.query_selector(“.flash.success”) error_el = await web page.query_selector(“.flash.error”)
if success_el: message = await success_el.inner_text() consequence = {“success”: True, “message”: message.strip()} elif error_el: message = await error_el.inner_text() consequence = {“success”: False, “message”: message.strip()} else: consequence = {“success”: False, “message”: “Unknown consequence”}
await browser.shut() return consequence
async def essential(): # Legitimate credentials for the demo website consequence = await login_and_verify(“tomsmith”, “SuperSecretPassword!”) print(f“Legitimate login: {consequence}”)
# Invalid credentials to confirm error dealing with result_fail = await login_and_verify(“wronguser”, “wrongpass”) print(f“Invalid login: {result_fail}”)
asyncio.run(essential()) |
What this does: The sample right here, fill() → click on() → wait_for_load_state() → verify for consequence aspect, is the template for nearly any type interplay. The wait_for_load_state(“networkidle”) after the submit is vital: with out it, you question the DOM earlier than the web page has up to date and get the pre-submission state, not the consequence.
For extra advanced varieties with file uploads, dropdowns, and checkboxes:
|
# File add await web page.set_input_files(“#file-upload”, “/path/to/doc.pdf”)
# Choose dropdown by seen label textual content await web page.select_option(“#country-select”, label=“Nigeria”)
# Test a checkbox await web page.verify(“#agree-terms”)
# Deal with a modal dialog (affirm/alert) web page.on(“dialog”, lambda dialog: asyncio.ensure_future(dialog.settle for())) |
Instrument Orchestration with LangChain and LangGraph
Uncooked Playwright scripts are highly effective however mounted. They do precisely what you coded, no extra. The second a web page modifications its construction, or the duty requires a choice the script didn’t anticipate, it breaks.
Connecting Playwright to an LLM modifications this. Browser actions grow to be instruments the agent can name when it decides they’re wanted. The agent reads the duty, causes about what to do, calls a device, reads the consequence, and decides what to do subsequent. That loop handles variation {that a} mounted script can not.
That is the bridge from “browser automation script” to “AI agent.”
Learn how to run: Save as agent_tools.py, guarantee OPENAI_API_KEY is in your .env, then run python agent_tools.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
# agent_tools.py # LangGraph agent with three browser instruments: navigate_and_extract, fill_and_submit_form, take_screenshot # Stipulations: pip set up playwright langchain langchain-openai langgraph python-dotenv # playwright set up chromium # Learn how to run: python agent_tools.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.instruments import device from langchain_core.messages import HumanMessage from langgraph.prebuilt import create_react_agent from playwright.async_api import async_playwright
load_dotenv()
# ── SHARED BROWSER STATE ────────────────────────────────────────────────────── # We maintain a single browser occasion alive for the agent’s lifetime. # Creating and destroying a browser on each device name is sluggish and wasteful. _browser = None _page = None _playwright = None
async def get_page(): “”“Return the shared web page, launching the browser if wanted.”“” international _browser, _page, _playwright if _browser is None: _playwright = await async_playwright().begin() _browser = await _playwright.chromium.launch(headless=True) context = await _browser.new_context(viewport={“width”: 1280, “top”: 720}) _page = await context.new_page() return _page
async def close_browser(): “”“Clear up browser assets when the agent session ends.”“” international _browser, _page, _playwright if _browser: await _browser.shut() await _playwright.cease() _browser = None _page = None _playwright = None
# ── BROWSER TOOLS ───────────────────────────────────────────────────────────── # Word: these are async instruments (async def). LangChain’s @device decorator helps # async features immediately, and the agent should be invoked with ainvoke() in order that # device calls run on the identical occasion loop as a substitute of attempting to start out a second one.
@device async def navigate_and_extract(url: str) -> str: “”“ Navigate to a URL and return the seen textual content content material of the web page. Use this to go to web sites and skim their content material. Enter: a full URL string together with https:// (e.g., ‘https://instance.com’). ““” web page = await get_page() await web page.goto(url, wait_until=“domcontentloaded”, timeout=15000) await web page.wait_for_load_state(“networkidle”) content material = await web page.inner_text(“physique”) # Truncate to keep away from flooding the LLM context window return content material[:3000] if len(content material) > 3000 else content material
@device async def fill_and_submit_form(selector_value_pairs: str) -> str: “”“ Fill type fields and submit a type on the presently loaded web page. Enter: a comma-separated string of ‘selector:worth’ pairs ending with ‘submit:button_selector’. Instance: ‘#e-mail:consumer@instance.com,#password:secret,submit:button[type=submit]’ ““” web page = await get_page() attempt: pairs = selector_value_pairs.break up(“,”) submit_selector = None
for pair in pairs: key, val = pair.break up(“:”, 1) key = key.strip() val = val.strip() if key == “submit”: submit_selector = val else: await web page.fill(key, val)
if submit_selector: await web page.click on(submit_selector) await web page.wait_for_load_state(“networkidle”)
return f“Type submitted. Present URL: {web page.url}” besides Exception as e: return f“Type interplay failed: {str(e)}”
@device async def take_screenshot(filename: str) -> str: “”“ Take a screenshot of the present browser web page and put it aside to a file. Use this to visually confirm the present state of the web page. Enter: filename string (e.g., ‘consequence.png’). ““” web page = await get_page() await web page.screenshot(path=filename, full_page=False) return f“Screenshot saved to {filename}”
# ── AGENT SETUP ───────────────────────────────────────────────────────────────
llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
instruments = [navigate_and_extract, fill_and_submit_form, take_screenshot]
# create_react_agent wires collectively the LLM, the instruments, and the ReAct reasoning loop. # The agent decides which device to name, calls it, reads the consequence, and continues. agent = create_react_agent(llm, instruments)
# ── DEMO ──────────────────────────────────────────────────────────────────────
async def essential(): consequence = await agent.ainvoke({ “messages”: [HumanMessage( content=( “Go to https://example.com, read the page content, “ “then take a screenshot called example.png” ) )] }) print(consequence[“messages”][–1].content material) await close_browser()
asyncio.run(essential()) |
What this does: The three @device-decorated features are registered with the agent. Every docstring is what the LLM reads to grasp what the device does and when to make use of it. Write them like job descriptions, not code feedback. The shared _browser and _page globals imply the browser stays open throughout a number of device calls, which is crucial for duties that span a number of pages in the identical session. As a result of the instruments are outlined with async def, the agent is invoked with ainvoke() quite than invoke(), so the device calls run on the identical occasion loop that essential() is already utilizing.
A vertical circulate diagram displaying how a activity request flows by means of the agent (click on to enlarge)
Picture by Editor
The important thing design determination on this snippet is the shared browser occasion. If every device name launched and closed its personal browser, you’ll lose all session state between calls, comparable to cookies, navigation historical past, and any type state the agent had already constructed up. Protecting the browser alive for the complete agent session preserves that context.
Utilizing browser-use for Excessive-Stage Agent Duties
Uncooked Playwright with @device features provides you exact management. The trade-off is that you’re nonetheless writing selectors, nonetheless desirous about web page construction, nonetheless dealing with each edge case manually. If the positioning modifications its HTML, your selectors break.
browser-use takes a unique method. As an alternative of writing selectors, you give the agent a activity in plain English. browser-use makes use of Playwright below the hood, however the LLM reads the present web page state on every step and decides what to do subsequent: which aspect to click on, what to kind, and when the duty is full. The web page construction is just not hardcoded into your code. The agent figures it out at runtime.
browser-use is a Python library that provides an LLM a working browser. The LLM reads every web page and decides what to click on, kind, and extract. This makes it resilient to website modifications that may break a selector-based script.
When to make use of browser-use over uncooked Playwright:
- If the duty is exploratory and the web page construction is unpredictable, use browser-use.
- In case you are working a set, repeatable workflow the place each selector is understood and secure, uncooked Playwright is extra dependable and cheaper per run.
- A browser-use agent makes a number of LLM calls per activity step; a scripted Playwright run makes none.
Learn how to run: Save as browser_use_agent.py, guarantee OPENAI_API_KEY is in your .env, then run python browser_use_agent.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
# browser_use_agent.py # A browser-use agent that accepts a pure language activity and completes it # with none CSS selectors or hardcoded web page construction. # Stipulations: pip set up browser-use playwright python-dotenv # playwright set up chromium # Learn how to run: python browser_use_agent.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from browser_use import Agent
load_dotenv()
async def run_browser_task(activity: str) -> str: “”“ Hand a pure language activity to a browser-use agent. The agent handles navigation, clicks, and extraction with out selectors. ““” # temperature=0 retains choices deterministic and reduces hallucinated actions llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
# Agent wraps the browser, the LLM, and the duty loop collectively. # max_actions_per_step limits what number of actions the agent takes earlier than # re-reading the web page — prevents runaway loops on advanced pages. agent = Agent( activity=activity, llm=llm, max_actions_per_step=5 )
# run() executes the complete activity loop: # learn web page → resolve motion → take motion → learn up to date web page → repeat consequence = await agent.run()
# final_result() returns the agent’s extracted content material or conclusion return consequence.final_result() or “Job accomplished with no extracted output.”
async def essential(): activity = ( “Go to https://books.toscrape.com and discover the three costliest books “ “on the primary web page. Return their titles and costs.” ) print(f“Job: {activity}n”) output = await run_browser_task(activity) print(f“End result:n{output}”)
asyncio.run(essential()) |
What this does: All the activity, navigating to the positioning, studying the web page, figuring out the three highest costs, and extracting them, is dealt with by the agent with out a single CSS selector in your code. If books.toscrape.com redesigns its value show tomorrow, the script nonetheless works. With a selector-based scraper, it will break silently.
The max_actions_per_step=5 parameter is price explaining. On every step, the agent reads the web page and may resolve to take as much as 5 actions (click on, kind, scroll, navigate) earlier than re-reading the web page. Protecting this low forces the agent to verify its work extra ceaselessly, which catches errors earlier.
Dealing with the Laborious Components
Three issues break most browser brokers in manufacturing. Every has an answer, however none of them is apparent till you could have already been burned.
1. Anti-Bot Detection
Web sites that don’t wish to be automated detect automation in a number of methods, comparable to checking the navigator.webdriver property (which Playwright units to true by default), searching for headless browser fingerprints within the JavaScript atmosphere, and analyzing interplay patterns which are too quick or too uniform to be human.
A very powerful mitigation is eradicating the webdriver flag. Past that, a practical consumer agent string, a normal viewport dimension, and a practical locale and timezone cowl most detection strategies in need of subtle fingerprint evaluation.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
# hard_parts.py — Half 1: Anti-bot stealth launch # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python hard_parts.py
import asyncio import json from pathlib import Path from playwright.async_api import async_playwright
async def launch_stealth_browser(playwright): “”“ Launch a browser context that appears extra like an actual human session. Covers: sensible viewport, user-agent, locale, timezone, webdriver flag. Word: For severe anti-bot targets, contemplate a paid service like Browserbase. ““” browser = await playwright.chromium.launch( headless=True, args=[ “–disable-blink-features=AutomationControlled”, # Hides webdriver detection “–no-sandbox”, “–disable-dev-shm-usage”, ] )
context = await browser.new_context( viewport={“width”: 1366, “top”: 768}, # Frequent desktop decision user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/124.0.0.0 Safari/537.36” ), locale=“en-US”, timezone_id=“America/New_York”, java_script_enabled=True, )
# Take away the ‘webdriver’ property that Playwright injects by default. # Bot detection programs verify for this within the browser’s JS atmosphere. await context.add_init_script( “Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})” )
return browser, context |
What this does: The add_init_script() name runs earlier than any web page JavaScript executes, which suggests the navigator.webdriver override is in place earlier than the positioning’s detection code can verify for it. The –disable-blink-features=AutomationControlled launch argument removes a separate automation flag on the browser engine degree. Collectively, these two modifications deal with the most typical detection strategies.
For websites with aggressive fingerprinting and CAPTCHA programs, these mitigations is not going to be sufficient. Companies like Browserbase, Spidra and Brightdata’s Scraping Browser deal with CAPTCHA fixing, residential IP rotation, and browser fingerprint administration as managed infrastructure.
2. Good Ready
The second failure mode is timing. The reflex is so as to add time.sleep() calls and improve them when issues break. That is unsuitable in each instructions: too quick on sluggish connections, too lengthy on quick ones, and utterly opaque when debugging.
Playwright has 4 correct wait methods. Use the one which matches what you might be really ready for:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Half 2: Good ready methods (add to your scraper or agent instruments)
async def smart_wait_examples(web page): “”“ 4 methods to attend for the best web page state, with out arbitrary sleeps. ““” # STRATEGY 1: Await a selected aspect to look within the DOM # Use when you understand precisely what aspect alerts content material has loaded await web page.wait_for_selector(“.product-list”, state=“seen”, timeout=10000)
# STRATEGY 2: Await a selected API response # Use when the content material comes from an XHR/fetch name you’ll be able to determine async with web page.expect_response( lambda r: “/api/merchandise” in r.url and r.standing == 200 ) as response_info: await web page.click on(“#load-more”) response = await response_info.worth print(f“API responded: {response.standing}”)
# STRATEGY 3: Await the URL to vary after type submission # Use when a profitable submit redirects to a brand new web page await web page.wait_for_url(“**/dashboard**”, timeout=10000)
# STRATEGY 4: Await a JavaScript variable to be set # Use when no visible aspect reliably alerts the prepared state await web page.wait_for_function( “() => window.__dataLoaded === true”, timeout=10000 ) |
What this does: Every technique is tied to a selected observable occasion quite than an arbitrary time delay. wait_for_selector watches the DOM. expect_response hooks into the community layer. wait_for_url screens navigation. wait_for_function evaluates JavaScript within the browser context. Use whichever one most immediately alerts “the factor I want is now prepared.”
3. Session and Cookie Persistence
The third failure mode is shedding session state. In case your agent logs right into a website throughout the 1st step after which the browser context is destroyed, step two has no authentication. Recreating the login on each run is sluggish and may set off charge limiting or lockout.
The answer is saving cookies to disk after login and loading them initially of each subsequent run:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# Half 3: Session persistence throughout runs
COOKIES_FILE = Path(“session_cookies.json”)
async def save_session(context) -> None: “”“Save browser cookies to disk after a profitable login.”“” cookies = await context.cookies() COOKIES_FILE.write_text(json.dumps(cookies, indent=2)) print(f“Session saved: {len(cookies)} cookies written.”)
async def load_session(context) -> bool: “”“Load saved cookies earlier than navigating. Returns True if session was discovered.”“” if not COOKIES_FILE.exists(): print(“No saved session. Recent login required.”) return False cookies = json.masses(COOKIES_FILE.read_text()) await context.add_cookies(cookies) print(f“Session restored: {len(cookies)} cookies loaded.”) return True |
What this does: context.cookies() returns all cookies for the present browser context, together with session tokens and authentication cookies. Writing them to JSON and reloading them on the subsequent run means the browser begins in an authenticated state. Word that classes expire; add a verify that falls again to a contemporary login if the saved session returns a redirect to the login web page.
Deploying Browser Brokers
Getting a browser agent working domestically is one factor. Working it reliably in a cloud atmosphere is one other.
The primary distinction between a Python script that works in your laptop computer and one which fails in CI is system dependencies. Playwright’s Chromium browser requires a set of shared libraries which are current on most developer machines however absent from minimal cloud pictures. The cleanest resolution is Docker.
Dockerfile — construct a container that ships all the pieces Playwright wants:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
# Dockerfile for headless Playwright-based browser agent # Construct: docker construct -t browser-agent . # Run: docker run –rm -e OPENAI_API_KEY=your_key browser-agent
FROM python:3.11–slim
# Set up system dependencies required by Chromium RUN apt–get replace && apt–get set up –y libnss3 libatk1.0–0 libatk–bridge2.0–0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxrandr2 libgbm1 libasound2 libpangocairo–1.0–0 libpango–1.0–0 libcairo2 libx11–6 libxext6 libxfixes3 fonts–liberation wget ca–certificates && rm –rf /var/lib/apt/lists/*
WORKDIR /app
# Set up Python dependencies first (cached layer — solely rebuilds on necessities change) COPY necessities.txt . RUN pip set up —no–cache–dir –r necessities.txt
# Set up Playwright browser binaries into the picture RUN playwright set up chromium RUN playwright set up–deps chromium
# Copy utility code final (modifications right here do not invalidate the pip/playwright layers) COPY . .
CMD [“python”, “agent_tools.py”]
necessities.txt: playwright browser–use langchain langchain–openai langgraph python–dotenv |
For concurrent workloads working a number of browser classes in parallel, use Playwright’s async API with asyncio.collect():
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Parallel scraping with semaphore charge limiting # Runs as much as 3 browser classes concurrently
import asyncio from playwright.async_api import async_playwright
async def scrape_url(browser, url: str, semaphore: asyncio.Semaphore) -> dict: “”“Scrape a single URL, respecting the concurrency semaphore.”“” async with semaphore: context = await browser.new_context() web page = await context.new_page() await web page.goto(url, wait_until=“domcontentloaded”) title = await web page.title() await context.shut() # Shut context (not browser) to launch assets return {“url”: url, “title”: title}
async def scrape_parallel(urls: listing[str], max_concurrent: int = 3) -> listing[dict]: “”“Scrape an inventory of URLs in parallel, capped at max_concurrent classes.”“” semaphore = asyncio.Semaphore(max_concurrent) # Cap concurrent classes
async with async_playwright() as p: # One browser shared throughout all contexts — less expensive than one browser per URL browser = await p.chromium.launch(headless=True) duties = [scrape_url(browser, url, semaphore) for url in urls] outcomes = await asyncio.collect(*duties) await browser.shut()
return listing(outcomes) |
What this does: The asyncio.Semaphore(max_concurrent) caps what number of browser contexts run on the identical time. With out it, launching 50 concurrent browser contexts will exhaust reminiscence. One browser course of is shared throughout all contexts; a context is reasonable; a full browser occasion is just not.
On the managed infrastructure aspect, Amazon Nova Act launched in March 2025 as a devoted SDK for constructing browser brokers on AWS, integrating natively with Playwright for browser management. Playwright’s personal MCP server provides AI assistants full browser management by means of the Mannequin Context Protocol, utilizing structured accessibility snapshots quite than screenshots, which suggests token prices keep low whereas the agent’s understanding of the web page stays excessive.
Placing It All Collectively
Here’s a full end-to-end agent that takes a analysis query, navigates to a public knowledge supply, extracts structured outcomes, and returns a clear abstract. It makes use of the browser instruments from Part 5 orchestrated by a LangGraph agent.
Learn how to run: Save as reference_agent.py, guarantee OPENAI_API_KEY is in your .env, and run python reference_agent.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
# reference_agent.py # Full browser-using AI agent: navigates, extracts, summarizes. # Goal: books.toscrape.com (public scraping sandbox) # Stipulations: pip set up playwright langchain langchain-openai langgraph python-dotenv # playwright set up chromium # Learn how to run: python reference_agent.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.instruments import device from langchain_core.messages import HumanMessage, SystemMessage from langgraph.prebuilt import create_react_agent from playwright.async_api import async_playwright
load_dotenv()
# ── BROWSER STATE ───────────────────────────────────────────────────────────── _browser = None _context = None _page = None _playwright = None
async def get_page(): international _browser, _context, _page, _playwright if _browser is None: _playwright = await async_playwright().begin() _browser = await _playwright.chromium.launch(headless=True) _context = await _browser.new_context( viewport={“width”: 1280, “top”: 720}, user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/120.0.0.0 Safari/537.36” ) ) # Take away webdriver fingerprint await _context.add_init_script( “Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})” ) _page = await _context.new_page() return _page
async def teardown(): international _browser, _playwright if _browser: await _browser.shut() await _playwright.cease() _browser = None _playwright = None
# ── TOOLS ─────────────────────────────────────────────────────────────────────
@device async def navigate(url: str) -> str: “”“ Navigate the browser to a URL and return the web page’s textual content content material. Use when it is advisable to open a web site or transfer to a brand new web page. Enter: full URL with https:// prefix. ““” web page = await get_page() await web page.goto(url, wait_until=“domcontentloaded”, timeout=20000) await web page.wait_for_load_state(“networkidle”) content material = await web page.inner_text(“physique”) return content material[:4000]
@device async def extract_structured(css_selector: str) -> str: “”“ Extract textual content from all parts matching a CSS selector on the present web page. Use when it is advisable to pull particular parts from the loaded web page. Enter: legitimate CSS selector string (e.g., ‘h3 a’, ‘.price_color’, ‘article.product_pod’). ““” web page = await get_page() attempt: await web page.wait_for_selector(css_selector, timeout=5000) parts = await web page.query_selector_all(css_selector) texts = [] for el in parts[:20]: # Cap at 20 parts to maintain output manageable textual content = await el.inner_text() texts.append(textual content.strip()) return “n”.be a part of(texts) if texts else “No parts discovered.” besides Exception as e: return f“Extraction failed: {str(e)}”
@device async def get_current_url() -> str: “”“Return the URL the browser is presently on. No enter required.”“” web page = await get_page() return web page.url
# ── AGENT ─────────────────────────────────────────────────────────────────────
llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
instruments = [navigate, extract_structured, get_current_url] agent = create_react_agent(llm, instruments)
SYSTEM = ( “You’re a browser-based analysis agent. You’ve got entry to an actual browser. “ “Use navigate() to open pages, extract_structured() to tug particular parts, “ “and get_current_url() to verify the place you might be. “ “All the time navigate first, then extract. Be concise in your remaining reply.” )
async def run_agent(question: str) -> str: consequence = await agent.ainvoke({ “messages”: [ SystemMessage(content=SYSTEM), HumanMessage(content=query) ] }) await teardown() return consequence[“messages”][–1].content material
# ── DEMO ──────────────────────────────────────────────────────────────────────
if __name__ == “__main__”: question = ( “Go to https://books.toscrape.com and extract the titles and costs “ “of the primary 5 books listed. Return them as a structured listing.” ) print(f“Question: {question}n”) reply = asyncio.run(run_agent(question)) print(f“Reply:n{reply}”) |
What this does: This agent has three clear instruments: navigate, extract_structured, and get_current_url, plus a system immediate that tells it precisely when to make use of each. The agent calls navigate to load the web page, extract_structured to tug the guide titles and costs by CSS selector, and synthesizes a structured listing within the remaining reply. The teardown() name after the agent finishes closes the browser cleanly so no zombie Chromium processes are left working.
Conclusion
The browser is just not a specialised device for automation engineers. It’s the common interface for the online, and the online is the place many of the world’s precise work will get completed. An AI agent that may use a browser doesn’t want a associate group sustaining API integrations. It may well attain something a human can attain.
What makes this sensible now, not simply theoretically fascinating, is the maturity of the tooling. Playwright handles the exhausting components of browser interplay. browser-use removes the necessity to write selectors for exploratory duties. LangGraph provides the LLM clear device hooks and a reasoning loop that handles variable web page buildings. The patterns on this article will not be demos. They’re the identical patterns 51% of enterprises now working AI brokers in manufacturing are constructing on.
Begin with the scraping instance. Get it working towards a website you really need knowledge from. Add the agent layer once you want choices the script can not anticipate. Add browser-use when the web page construction is simply too dynamic for selectors. Deploy in Docker once you want it working someplace aside from your laptop computer.
The exhausting half is just not the code. It’s understanding which device to succeed in for at every layer. Hopefully this text made that clearer.
On this article, you’ll learn to construct AI brokers that may browse and work together with actual web sites utilizing Playwright, browser-use, and LangGraph.
Subjects we are going to cowl embody:
- Why Playwright is the best basis for browser automation in 2026, and the way it differs from Selenium.
- Learn how to scrape dynamic, JavaScript-rendered pages and full multi-step varieties reliably.
- Learn how to wire browser actions into LangGraph and browser-use brokers, deal with anti-bot detection, handle ready and session persistence, and deploy the lead to Docker.
Constructing Browser-Utilizing AI Brokers in Python
Introduction
Most AI agent tutorials begin with an API. They present you the best way to name OpenWeather, hit the Stripe endpoint, pull knowledge from GitHub. That could be a positive start line till you attempt to construct one thing actual and understand that the duty you really need completed doesn’t have an API.
Take into consideration what people do with browsers on daily basis: submitting authorities varieties, studying competitor pricing, extracting analysis from websites that guard their knowledge behind JavaScript rendering, logging into portals which have by no means heard of OAuth. There are roughly 1.1 billion web sites on the web. A vanishingly small fraction of them have public APIs. The remainder solely converse browser.
An agent that’s restricted to API calls handles perhaps 5% of the duties a human employee does day by day. Give that agent a browser, and the protection approaches all the pieces. That’s the hole this text closes.
The international AI brokers market stands at $10.91 billion in 2026 and is projected to succeed in $50.31 billion by 2030, with browser-capable brokers on the heart of that development. 27.7% of enterprises are already working agentic browsers in manufacturing, up from just about none two years prior. The tooling has matured quick, and the patterns are settled sufficient to show correctly.
By the tip of this text, you should have a working browser agent that navigates actual web sites, fills varieties, extracts structured knowledge, and connects to an LLM that decides what to do subsequent, all in Python.
Why Playwright, Not Selenium
When you constructed browser automation 5 years in the past, you constructed it with Selenium. Selenium continues to be extensively deployed, nonetheless works, and isn’t going wherever. However for any new venture in 2026, Playwright is the default. The explanations are sensible, not theoretical.
Selenium communicates with the browser by sending particular person HTTP requests to a WebDriver. Each motion, click on, kind, scroll, is a separate request. Playwright makes use of a persistent WebSocket connection for the whole session. Instructions circulate by means of that channel with no per-action round-trip value. Impartial benchmarks persistently present Playwright working 30-50% quicker than Selenium on the test-suite degree and averaging ~290ms per motion versus Selenium’s ~536ms. For a browser agent that may execute tons of of actions, that hole compounds.
Playwright additionally bundles its personal browser binaries. If you set up it, you get pre-configured variations of Chromium, Firefox, and WebKit which are assured to work together with your Playwright model. No driver model mismatches, no damaged CI pipelines as a result of somebody up to date Chrome. It has built-in auto-waiting earlier than it clicks a component; it verifies the aspect is seen, enabled, and never animating. You do not need to put in writing time.sleep(2) and hope for one of the best.
For AI brokers particularly, Playwright fires actual mouse and keyboard occasions that mirror how people work together with browsers. Websites designed to detect automation search for artificial DOM clicks. Playwright’s interplay mannequin is more durable to tell apart from real human enter.
There’s additionally the browser-use library, which sits one degree increased. Browser-use is a Python library that provides an LLM a working browser. Below the hood, it makes use of Playwright to drive the browser, however the LLM reads the web page state and decides what to click on, kind, and extract, no CSS selectors required. You give it a activity in plain English, and it figures out the remainder. We are going to cowl each uncooked Playwright and browser-use on this article, as a result of they serve completely different wants: Playwright once you need exact, predictable management; browser-use once you need the agent to deal with navigation choices autonomously.
Setting Up the Surroundings
You want Python 3.10 or increased, an OpenAI API key, and about 5 minutes.
Step 1: Create a digital atmosphere
|
python –m venv browser_agent_env
# macOS / Linux supply browser_agent_env/bin/activate
# Home windows browser_agent_envScriptsactivate |
Step 2: Set up dependencies
|
pip set up playwright browser–use langchain langchain–openai langgraph langchain–neighborhood python–dotenv |
Step 3: Set up the browser binaries
That is the step most individuals miss. Playwright must obtain Chromium, Firefox, and WebKit individually from the Python bundle. Run this as soon as after putting in:
|
playwright set up chromium |
If you need all three browser engines: playwright set up. Chromium alone is adequate for many agent work and is smaller to obtain.
Step 4: Retailer your API key
Create a .env file in your venture listing:
|
OPENAI_API_KEY=your_openai_api_key_here |
Add .env to your .gitignore instantly. Don’t commit API keys.
Step 5: Confirm all the pieces works
Here’s a first script that navigates to a URL, reads the heading, and saves a screenshot. Use instance.com, a publicly obtainable check area maintained by IANA that won’t block you.
Learn how to run: Save as first_run.py and run python first_run.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
# first_run.py # Navigate to a URL, take a screenshot, and extract the web page title. # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python first_run.py
import asyncio from playwright.async_api import async_playwright
async def essential(): async with async_playwright() as p: # Launch Chromium in headless mode (no seen browser window). # Set headless=False if you wish to watch it run throughout growth. browser = await p.chromium.launch(headless=True)
# A browser context is sort of a contemporary browser profile. # It isolates cookies, storage, and cache from different contexts. context = await browser.new_context( viewport={“width”: 1280, “top”: 720}, user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/120.0.0.0 Safari/537.36” ) )
web page = await context.new_page()
# Navigate to the URL and wait till the community is idle. # “networkidle” means no open community connections for 500ms. # For quicker pages, “domcontentloaded” is adequate. await web page.goto(“https://instance.com”, wait_until=“networkidle”)
# Extract the web page title title = await web page.title() print(f“Web page title: {title}”)
# Extract the textual content content material of the h1 heading h1 = await web page.text_content(“h1”) print(f“H1 heading: {h1}”)
# Take a full-page screenshot and put it aside to disk await web page.screenshot(path=“screenshot.png”, full_page=True) print(“Screenshot saved to screenshot.png”)
await browser.shut()
asyncio.run(essential()) |
What this does: async_playwright() is the entry level for the whole Playwright session. The browser_context is equal to opening a contemporary incognito window; cookies, native storage, and cache are remoted from all the pieces else. wait_until=”networkidle” tells Playwright to attend till the web page has completed all its community exercise earlier than your code continues, which is the most secure wait technique for dynamic pages.
If this runs and saves a screenshot, your atmosphere is working accurately.
Internet Navigation and Scraping
The explanation you want Playwright as a substitute of requests + BeautifulSoup is JavaScript rendering. Fashionable web sites ship a skeleton of HTML after which construct the precise content material dynamically after the web page masses: React, Vue, Angular, Subsequent.js. A plain HTTP request fetches the skeleton. Playwright runs an actual browser, so it sees precisely what a human sees in spite of everything JavaScript has executed.
The goal beneath is books.toscrape.com, a authorized scraping sandbox constructed for observe. It paginates outcomes, makes use of dynamic class names for scores, and intently mirrors the construction of actual e-commerce product pages.
Learn how to run: Save as scrape_books.py and run python scrape_books.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
# scrape_books.py # Scrape guide titles, costs, and scores from books.toscrape.com # This can be a authorized scraping sandbox website constructed for observe. # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python scrape_books.py
import asyncio import json from playwright.async_api import async_playwright
async def scrape_books(max_pages: int = 3) -> listing[dict]: “”“ Scrape guide listings from books.toscrape.com throughout a number of pages. Returns an inventory of dicts with title, value, ranking, and web page quantity. ““” outcomes = []
async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context(viewport={“width”: 1280, “top”: 720}) web page = await context.new_page()
for page_num in vary(1, max_pages + 1): url = f“https://books.toscrape.com/catalogue/page-{page_num}.html” print(f“Scraping web page {page_num}: {url}”)
await web page.goto(url, wait_until=“domcontentloaded”)
# Await the product playing cards to be seen earlier than extracting. # That is vital on JavaScript-heavy pages the place content material masses after the HTML. # timeout=10000 means wait as much as 10 seconds earlier than elevating an error. await web page.wait_for_selector(“article.product_pod”, timeout=10000)
# Get all guide playing cards on the present web page books = await web page.query_selector_all(“article.product_pod”)
for guide in books: title_el = await guide.query_selector(“h3 a”) title = await title_el.get_attribute(“title”) if title_el else “N/A”
# Extract value textual content price_el = await guide.query_selector(“.price_color”) value = await price_el.inner_text() if price_el else “N/A”
# Extract star ranking from the CSS class title. # e.g.
rating_el = await guide.query_selector(“p.star-rating”) rating_class = await rating_el.get_attribute(“class”) if rating_el else “” ranking = rating_class.exchange(“star-rating”, “”).strip()
outcomes.append({ “title”: title, “value”: value, “ranking”: ranking, “web page”: web page_num })
print(f” Extracted {len(books)} books from web page {page_num}”)
await browser.shut()
return outcomes
async def essential(): books = await scrape_books(max_pages=2) print(f“nTotal books scraped: {len(books)}”) print(json.dumps(books[:3], indent=2))
asyncio.run(essential()) |
What this does: wait_for_selector() is the important thing name right here. As an alternative of sleeping for a set time and hoping the content material has loaded, it watches the DOM and proceeds the second the goal aspect seems, or raises a TimeoutError if it doesn’t seem inside the timeout window. That’s the proper habits: fail quick and explicitly quite than silently extracting from an empty web page.
The ranking extraction deserves consideration. The star ranking is encoded as a CSS class (star-rating Three), not a quantity. The code strips “star-rating” from the category string to get the textual content worth. That is the form of factor you solely know by inspecting the precise HTML. If you hand this activity to a uncooked LLM with no browser, it has no method to know what the category construction seems to be like. With Playwright, you’ll be able to examine it immediately and extract it precisely.
Type Completion and Multi-Step Flows
Filling varieties is the place browser brokers earn their maintain and the place most automation scripts fail. The reason being that internet varieties will not be simply inputs and buttons. They hearth focus, enter, change, and blur occasions in sequence. JavaScript validation listens for these occasions. When you inject a worth into an enter subject by immediately setting worth within the DOM (as older automation instruments usually do), the validation listeners by no means hearth and the shape breaks.
Playwright’s fill() and click on() strategies hearth actual browser occasions in the best order, which is why they work on type validation that may block lower-level approaches.
The goal beneath is the-internet.herokuapp.com/login, a public check website maintained particularly for automation observe. It accepts tomsmith / SuperSecretPassword! as legitimate credentials and returns clear success/failure messages.
Learn how to run: Save as form_submit.py and run python form_submit.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
# form_submit.py # Full and submit a multi-field login type on a public demo website. # Goal: https://the-internet.herokuapp.com/login (public check website) # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python form_submit.py
import asyncio from playwright.async_api import async_playwright
async def login_and_verify(username: str, password: str) -> dict: “”“ Try to log in to a demo website and return whether or not it succeeded. Handles: enter filling, button clicking, and consequence verification. ““” async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context() web page = await context.new_page()
await web page.goto(“https://the-internet.herokuapp.com/login”)
# Await the shape to be seen earlier than interacting. # state=”seen” is the default however makes the intent specific. await web page.wait_for_selector(“#username”, state=“seen”)
# fill() clears the sphere first, then varieties the worth. # It fires the main focus, enter, and alter occasions so as. await web page.fill(“#username”, username) await web page.fill(“#password”, password)
# click on() fires actual mouse occasions — mousedown, mouseup, click on. # This triggers JavaScript listeners {that a} plain DOM click on misses. await web page.click on(“button[type=”submit”]”)
# Await the web page to settle after type submission await web page.wait_for_load_state(“networkidle”)
# Test which consequence aspect appeared success_el = await web page.query_selector(“.flash.success”) error_el = await web page.query_selector(“.flash.error”)
if success_el: message = await success_el.inner_text() consequence = {“success”: True, “message”: message.strip()} elif error_el: message = await error_el.inner_text() consequence = {“success”: False, “message”: message.strip()} else: consequence = {“success”: False, “message”: “Unknown consequence”}
await browser.shut() return consequence
async def essential(): # Legitimate credentials for the demo website consequence = await login_and_verify(“tomsmith”, “SuperSecretPassword!”) print(f“Legitimate login: {consequence}”)
# Invalid credentials to confirm error dealing with result_fail = await login_and_verify(“wronguser”, “wrongpass”) print(f“Invalid login: {result_fail}”)
asyncio.run(essential()) |
What this does: The sample right here, fill() → click on() → wait_for_load_state() → verify for consequence aspect, is the template for nearly any type interplay. The wait_for_load_state(“networkidle”) after the submit is vital: with out it, you question the DOM earlier than the web page has up to date and get the pre-submission state, not the consequence.
For extra advanced varieties with file uploads, dropdowns, and checkboxes:
|
# File add await web page.set_input_files(“#file-upload”, “/path/to/doc.pdf”)
# Choose dropdown by seen label textual content await web page.select_option(“#country-select”, label=“Nigeria”)
# Test a checkbox await web page.verify(“#agree-terms”)
# Deal with a modal dialog (affirm/alert) web page.on(“dialog”, lambda dialog: asyncio.ensure_future(dialog.settle for())) |
Instrument Orchestration with LangChain and LangGraph
Uncooked Playwright scripts are highly effective however mounted. They do precisely what you coded, no extra. The second a web page modifications its construction, or the duty requires a choice the script didn’t anticipate, it breaks.
Connecting Playwright to an LLM modifications this. Browser actions grow to be instruments the agent can name when it decides they’re wanted. The agent reads the duty, causes about what to do, calls a device, reads the consequence, and decides what to do subsequent. That loop handles variation {that a} mounted script can not.
That is the bridge from “browser automation script” to “AI agent.”
Learn how to run: Save as agent_tools.py, guarantee OPENAI_API_KEY is in your .env, then run python agent_tools.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
# agent_tools.py # LangGraph agent with three browser instruments: navigate_and_extract, fill_and_submit_form, take_screenshot # Stipulations: pip set up playwright langchain langchain-openai langgraph python-dotenv # playwright set up chromium # Learn how to run: python agent_tools.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.instruments import device from langchain_core.messages import HumanMessage from langgraph.prebuilt import create_react_agent from playwright.async_api import async_playwright
load_dotenv()
# ── SHARED BROWSER STATE ────────────────────────────────────────────────────── # We maintain a single browser occasion alive for the agent’s lifetime. # Creating and destroying a browser on each device name is sluggish and wasteful. _browser = None _page = None _playwright = None
async def get_page(): “”“Return the shared web page, launching the browser if wanted.”“” international _browser, _page, _playwright if _browser is None: _playwright = await async_playwright().begin() _browser = await _playwright.chromium.launch(headless=True) context = await _browser.new_context(viewport={“width”: 1280, “top”: 720}) _page = await context.new_page() return _page
async def close_browser(): “”“Clear up browser assets when the agent session ends.”“” international _browser, _page, _playwright if _browser: await _browser.shut() await _playwright.cease() _browser = None _page = None _playwright = None
# ── BROWSER TOOLS ───────────────────────────────────────────────────────────── # Word: these are async instruments (async def). LangChain’s @device decorator helps # async features immediately, and the agent should be invoked with ainvoke() in order that # device calls run on the identical occasion loop as a substitute of attempting to start out a second one.
@device async def navigate_and_extract(url: str) -> str: “”“ Navigate to a URL and return the seen textual content content material of the web page. Use this to go to web sites and skim their content material. Enter: a full URL string together with https:// (e.g., ‘https://instance.com’). ““” web page = await get_page() await web page.goto(url, wait_until=“domcontentloaded”, timeout=15000) await web page.wait_for_load_state(“networkidle”) content material = await web page.inner_text(“physique”) # Truncate to keep away from flooding the LLM context window return content material[:3000] if len(content material) > 3000 else content material
@device async def fill_and_submit_form(selector_value_pairs: str) -> str: “”“ Fill type fields and submit a type on the presently loaded web page. Enter: a comma-separated string of ‘selector:worth’ pairs ending with ‘submit:button_selector’. Instance: ‘#e-mail:consumer@instance.com,#password:secret,submit:button[type=submit]’ ““” web page = await get_page() attempt: pairs = selector_value_pairs.break up(“,”) submit_selector = None
for pair in pairs: key, val = pair.break up(“:”, 1) key = key.strip() val = val.strip() if key == “submit”: submit_selector = val else: await web page.fill(key, val)
if submit_selector: await web page.click on(submit_selector) await web page.wait_for_load_state(“networkidle”)
return f“Type submitted. Present URL: {web page.url}” besides Exception as e: return f“Type interplay failed: {str(e)}”
@device async def take_screenshot(filename: str) -> str: “”“ Take a screenshot of the present browser web page and put it aside to a file. Use this to visually confirm the present state of the web page. Enter: filename string (e.g., ‘consequence.png’). ““” web page = await get_page() await web page.screenshot(path=filename, full_page=False) return f“Screenshot saved to {filename}”
# ── AGENT SETUP ───────────────────────────────────────────────────────────────
llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
instruments = [navigate_and_extract, fill_and_submit_form, take_screenshot]
# create_react_agent wires collectively the LLM, the instruments, and the ReAct reasoning loop. # The agent decides which device to name, calls it, reads the consequence, and continues. agent = create_react_agent(llm, instruments)
# ── DEMO ──────────────────────────────────────────────────────────────────────
async def essential(): consequence = await agent.ainvoke({ “messages”: [HumanMessage( content=( “Go to https://example.com, read the page content, “ “then take a screenshot called example.png” ) )] }) print(consequence[“messages”][–1].content material) await close_browser()
asyncio.run(essential()) |
What this does: The three @device-decorated features are registered with the agent. Every docstring is what the LLM reads to grasp what the device does and when to make use of it. Write them like job descriptions, not code feedback. The shared _browser and _page globals imply the browser stays open throughout a number of device calls, which is crucial for duties that span a number of pages in the identical session. As a result of the instruments are outlined with async def, the agent is invoked with ainvoke() quite than invoke(), so the device calls run on the identical occasion loop that essential() is already utilizing.
A vertical circulate diagram displaying how a activity request flows by means of the agent (click on to enlarge)
Picture by Editor
The important thing design determination on this snippet is the shared browser occasion. If every device name launched and closed its personal browser, you’ll lose all session state between calls, comparable to cookies, navigation historical past, and any type state the agent had already constructed up. Protecting the browser alive for the complete agent session preserves that context.
Utilizing browser-use for Excessive-Stage Agent Duties
Uncooked Playwright with @device features provides you exact management. The trade-off is that you’re nonetheless writing selectors, nonetheless desirous about web page construction, nonetheless dealing with each edge case manually. If the positioning modifications its HTML, your selectors break.
browser-use takes a unique method. As an alternative of writing selectors, you give the agent a activity in plain English. browser-use makes use of Playwright below the hood, however the LLM reads the present web page state on every step and decides what to do subsequent: which aspect to click on, what to kind, and when the duty is full. The web page construction is just not hardcoded into your code. The agent figures it out at runtime.
browser-use is a Python library that provides an LLM a working browser. The LLM reads every web page and decides what to click on, kind, and extract. This makes it resilient to website modifications that may break a selector-based script.
When to make use of browser-use over uncooked Playwright:
- If the duty is exploratory and the web page construction is unpredictable, use browser-use.
- In case you are working a set, repeatable workflow the place each selector is understood and secure, uncooked Playwright is extra dependable and cheaper per run.
- A browser-use agent makes a number of LLM calls per activity step; a scripted Playwright run makes none.
Learn how to run: Save as browser_use_agent.py, guarantee OPENAI_API_KEY is in your .env, then run python browser_use_agent.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
# browser_use_agent.py # A browser-use agent that accepts a pure language activity and completes it # with none CSS selectors or hardcoded web page construction. # Stipulations: pip set up browser-use playwright python-dotenv # playwright set up chromium # Learn how to run: python browser_use_agent.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from browser_use import Agent
load_dotenv()
async def run_browser_task(activity: str) -> str: “”“ Hand a pure language activity to a browser-use agent. The agent handles navigation, clicks, and extraction with out selectors. ““” # temperature=0 retains choices deterministic and reduces hallucinated actions llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
# Agent wraps the browser, the LLM, and the duty loop collectively. # max_actions_per_step limits what number of actions the agent takes earlier than # re-reading the web page — prevents runaway loops on advanced pages. agent = Agent( activity=activity, llm=llm, max_actions_per_step=5 )
# run() executes the complete activity loop: # learn web page → resolve motion → take motion → learn up to date web page → repeat consequence = await agent.run()
# final_result() returns the agent’s extracted content material or conclusion return consequence.final_result() or “Job accomplished with no extracted output.”
async def essential(): activity = ( “Go to https://books.toscrape.com and discover the three costliest books “ “on the primary web page. Return their titles and costs.” ) print(f“Job: {activity}n”) output = await run_browser_task(activity) print(f“End result:n{output}”)
asyncio.run(essential()) |
What this does: All the activity, navigating to the positioning, studying the web page, figuring out the three highest costs, and extracting them, is dealt with by the agent with out a single CSS selector in your code. If books.toscrape.com redesigns its value show tomorrow, the script nonetheless works. With a selector-based scraper, it will break silently.
The max_actions_per_step=5 parameter is price explaining. On every step, the agent reads the web page and may resolve to take as much as 5 actions (click on, kind, scroll, navigate) earlier than re-reading the web page. Protecting this low forces the agent to verify its work extra ceaselessly, which catches errors earlier.
Dealing with the Laborious Components
Three issues break most browser brokers in manufacturing. Every has an answer, however none of them is apparent till you could have already been burned.
1. Anti-Bot Detection
Web sites that don’t wish to be automated detect automation in a number of methods, comparable to checking the navigator.webdriver property (which Playwright units to true by default), searching for headless browser fingerprints within the JavaScript atmosphere, and analyzing interplay patterns which are too quick or too uniform to be human.
A very powerful mitigation is eradicating the webdriver flag. Past that, a practical consumer agent string, a normal viewport dimension, and a practical locale and timezone cowl most detection strategies in need of subtle fingerprint evaluation.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
# hard_parts.py — Half 1: Anti-bot stealth launch # Stipulations: pip set up playwright && playwright set up chromium # Learn how to run: python hard_parts.py
import asyncio import json from pathlib import Path from playwright.async_api import async_playwright
async def launch_stealth_browser(playwright): “”“ Launch a browser context that appears extra like an actual human session. Covers: sensible viewport, user-agent, locale, timezone, webdriver flag. Word: For severe anti-bot targets, contemplate a paid service like Browserbase. ““” browser = await playwright.chromium.launch( headless=True, args=[ “–disable-blink-features=AutomationControlled”, # Hides webdriver detection “–no-sandbox”, “–disable-dev-shm-usage”, ] )
context = await browser.new_context( viewport={“width”: 1366, “top”: 768}, # Frequent desktop decision user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/124.0.0.0 Safari/537.36” ), locale=“en-US”, timezone_id=“America/New_York”, java_script_enabled=True, )
# Take away the ‘webdriver’ property that Playwright injects by default. # Bot detection programs verify for this within the browser’s JS atmosphere. await context.add_init_script( “Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})” )
return browser, context |
What this does: The add_init_script() name runs earlier than any web page JavaScript executes, which suggests the navigator.webdriver override is in place earlier than the positioning’s detection code can verify for it. The –disable-blink-features=AutomationControlled launch argument removes a separate automation flag on the browser engine degree. Collectively, these two modifications deal with the most typical detection strategies.
For websites with aggressive fingerprinting and CAPTCHA programs, these mitigations is not going to be sufficient. Companies like Browserbase, Spidra and Brightdata’s Scraping Browser deal with CAPTCHA fixing, residential IP rotation, and browser fingerprint administration as managed infrastructure.
2. Good Ready
The second failure mode is timing. The reflex is so as to add time.sleep() calls and improve them when issues break. That is unsuitable in each instructions: too quick on sluggish connections, too lengthy on quick ones, and utterly opaque when debugging.
Playwright has 4 correct wait methods. Use the one which matches what you might be really ready for:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Half 2: Good ready methods (add to your scraper or agent instruments)
async def smart_wait_examples(web page): “”“ 4 methods to attend for the best web page state, with out arbitrary sleeps. ““” # STRATEGY 1: Await a selected aspect to look within the DOM # Use when you understand precisely what aspect alerts content material has loaded await web page.wait_for_selector(“.product-list”, state=“seen”, timeout=10000)
# STRATEGY 2: Await a selected API response # Use when the content material comes from an XHR/fetch name you’ll be able to determine async with web page.expect_response( lambda r: “/api/merchandise” in r.url and r.standing == 200 ) as response_info: await web page.click on(“#load-more”) response = await response_info.worth print(f“API responded: {response.standing}”)
# STRATEGY 3: Await the URL to vary after type submission # Use when a profitable submit redirects to a brand new web page await web page.wait_for_url(“**/dashboard**”, timeout=10000)
# STRATEGY 4: Await a JavaScript variable to be set # Use when no visible aspect reliably alerts the prepared state await web page.wait_for_function( “() => window.__dataLoaded === true”, timeout=10000 ) |
What this does: Every technique is tied to a selected observable occasion quite than an arbitrary time delay. wait_for_selector watches the DOM. expect_response hooks into the community layer. wait_for_url screens navigation. wait_for_function evaluates JavaScript within the browser context. Use whichever one most immediately alerts “the factor I want is now prepared.”
3. Session and Cookie Persistence
The third failure mode is shedding session state. In case your agent logs right into a website throughout the 1st step after which the browser context is destroyed, step two has no authentication. Recreating the login on each run is sluggish and may set off charge limiting or lockout.
The answer is saving cookies to disk after login and loading them initially of each subsequent run:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# Half 3: Session persistence throughout runs
COOKIES_FILE = Path(“session_cookies.json”)
async def save_session(context) -> None: “”“Save browser cookies to disk after a profitable login.”“” cookies = await context.cookies() COOKIES_FILE.write_text(json.dumps(cookies, indent=2)) print(f“Session saved: {len(cookies)} cookies written.”)
async def load_session(context) -> bool: “”“Load saved cookies earlier than navigating. Returns True if session was discovered.”“” if not COOKIES_FILE.exists(): print(“No saved session. Recent login required.”) return False cookies = json.masses(COOKIES_FILE.read_text()) await context.add_cookies(cookies) print(f“Session restored: {len(cookies)} cookies loaded.”) return True |
What this does: context.cookies() returns all cookies for the present browser context, together with session tokens and authentication cookies. Writing them to JSON and reloading them on the subsequent run means the browser begins in an authenticated state. Word that classes expire; add a verify that falls again to a contemporary login if the saved session returns a redirect to the login web page.
Deploying Browser Brokers
Getting a browser agent working domestically is one factor. Working it reliably in a cloud atmosphere is one other.
The primary distinction between a Python script that works in your laptop computer and one which fails in CI is system dependencies. Playwright’s Chromium browser requires a set of shared libraries which are current on most developer machines however absent from minimal cloud pictures. The cleanest resolution is Docker.
Dockerfile — construct a container that ships all the pieces Playwright wants:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
# Dockerfile for headless Playwright-based browser agent # Construct: docker construct -t browser-agent . # Run: docker run –rm -e OPENAI_API_KEY=your_key browser-agent
FROM python:3.11–slim
# Set up system dependencies required by Chromium RUN apt–get replace && apt–get set up –y libnss3 libatk1.0–0 libatk–bridge2.0–0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxrandr2 libgbm1 libasound2 libpangocairo–1.0–0 libpango–1.0–0 libcairo2 libx11–6 libxext6 libxfixes3 fonts–liberation wget ca–certificates && rm –rf /var/lib/apt/lists/*
WORKDIR /app
# Set up Python dependencies first (cached layer — solely rebuilds on necessities change) COPY necessities.txt . RUN pip set up —no–cache–dir –r necessities.txt
# Set up Playwright browser binaries into the picture RUN playwright set up chromium RUN playwright set up–deps chromium
# Copy utility code final (modifications right here do not invalidate the pip/playwright layers) COPY . .
CMD [“python”, “agent_tools.py”]
necessities.txt: playwright browser–use langchain langchain–openai langgraph python–dotenv |
For concurrent workloads working a number of browser classes in parallel, use Playwright’s async API with asyncio.collect():
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Parallel scraping with semaphore charge limiting # Runs as much as 3 browser classes concurrently
import asyncio from playwright.async_api import async_playwright
async def scrape_url(browser, url: str, semaphore: asyncio.Semaphore) -> dict: “”“Scrape a single URL, respecting the concurrency semaphore.”“” async with semaphore: context = await browser.new_context() web page = await context.new_page() await web page.goto(url, wait_until=“domcontentloaded”) title = await web page.title() await context.shut() # Shut context (not browser) to launch assets return {“url”: url, “title”: title}
async def scrape_parallel(urls: listing[str], max_concurrent: int = 3) -> listing[dict]: “”“Scrape an inventory of URLs in parallel, capped at max_concurrent classes.”“” semaphore = asyncio.Semaphore(max_concurrent) # Cap concurrent classes
async with async_playwright() as p: # One browser shared throughout all contexts — less expensive than one browser per URL browser = await p.chromium.launch(headless=True) duties = [scrape_url(browser, url, semaphore) for url in urls] outcomes = await asyncio.collect(*duties) await browser.shut()
return listing(outcomes) |
What this does: The asyncio.Semaphore(max_concurrent) caps what number of browser contexts run on the identical time. With out it, launching 50 concurrent browser contexts will exhaust reminiscence. One browser course of is shared throughout all contexts; a context is reasonable; a full browser occasion is just not.
On the managed infrastructure aspect, Amazon Nova Act launched in March 2025 as a devoted SDK for constructing browser brokers on AWS, integrating natively with Playwright for browser management. Playwright’s personal MCP server provides AI assistants full browser management by means of the Mannequin Context Protocol, utilizing structured accessibility snapshots quite than screenshots, which suggests token prices keep low whereas the agent’s understanding of the web page stays excessive.
Placing It All Collectively
Here’s a full end-to-end agent that takes a analysis query, navigates to a public knowledge supply, extracts structured outcomes, and returns a clear abstract. It makes use of the browser instruments from Part 5 orchestrated by a LangGraph agent.
Learn how to run: Save as reference_agent.py, guarantee OPENAI_API_KEY is in your .env, and run python reference_agent.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
# reference_agent.py # Full browser-using AI agent: navigates, extracts, summarizes. # Goal: books.toscrape.com (public scraping sandbox) # Stipulations: pip set up playwright langchain langchain-openai langgraph python-dotenv # playwright set up chromium # Learn how to run: python reference_agent.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.instruments import device from langchain_core.messages import HumanMessage, SystemMessage from langgraph.prebuilt import create_react_agent from playwright.async_api import async_playwright
load_dotenv()
# ── BROWSER STATE ───────────────────────────────────────────────────────────── _browser = None _context = None _page = None _playwright = None
async def get_page(): international _browser, _context, _page, _playwright if _browser is None: _playwright = await async_playwright().begin() _browser = await _playwright.chromium.launch(headless=True) _context = await _browser.new_context( viewport={“width”: 1280, “top”: 720}, user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/120.0.0.0 Safari/537.36” ) ) # Take away webdriver fingerprint await _context.add_init_script( “Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})” ) _page = await _context.new_page() return _page
async def teardown(): international _browser, _playwright if _browser: await _browser.shut() await _playwright.cease() _browser = None _playwright = None
# ── TOOLS ─────────────────────────────────────────────────────────────────────
@device async def navigate(url: str) -> str: “”“ Navigate the browser to a URL and return the web page’s textual content content material. Use when it is advisable to open a web site or transfer to a brand new web page. Enter: full URL with https:// prefix. ““” web page = await get_page() await web page.goto(url, wait_until=“domcontentloaded”, timeout=20000) await web page.wait_for_load_state(“networkidle”) content material = await web page.inner_text(“physique”) return content material[:4000]
@device async def extract_structured(css_selector: str) -> str: “”“ Extract textual content from all parts matching a CSS selector on the present web page. Use when it is advisable to pull particular parts from the loaded web page. Enter: legitimate CSS selector string (e.g., ‘h3 a’, ‘.price_color’, ‘article.product_pod’). ““” web page = await get_page() attempt: await web page.wait_for_selector(css_selector, timeout=5000) parts = await web page.query_selector_all(css_selector) texts = [] for el in parts[:20]: # Cap at 20 parts to maintain output manageable textual content = await el.inner_text() texts.append(textual content.strip()) return “n”.be a part of(texts) if texts else “No parts discovered.” besides Exception as e: return f“Extraction failed: {str(e)}”
@device async def get_current_url() -> str: “”“Return the URL the browser is presently on. No enter required.”“” web page = await get_page() return web page.url
# ── AGENT ─────────────────────────────────────────────────────────────────────
llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
instruments = [navigate, extract_structured, get_current_url] agent = create_react_agent(llm, instruments)
SYSTEM = ( “You’re a browser-based analysis agent. You’ve got entry to an actual browser. “ “Use navigate() to open pages, extract_structured() to tug particular parts, “ “and get_current_url() to verify the place you might be. “ “All the time navigate first, then extract. Be concise in your remaining reply.” )
async def run_agent(question: str) -> str: consequence = await agent.ainvoke({ “messages”: [ SystemMessage(content=SYSTEM), HumanMessage(content=query) ] }) await teardown() return consequence[“messages”][–1].content material
# ── DEMO ──────────────────────────────────────────────────────────────────────
if __name__ == “__main__”: question = ( “Go to https://books.toscrape.com and extract the titles and costs “ “of the primary 5 books listed. Return them as a structured listing.” ) print(f“Question: {question}n”) reply = asyncio.run(run_agent(question)) print(f“Reply:n{reply}”) |
What this does: This agent has three clear instruments: navigate, extract_structured, and get_current_url, plus a system immediate that tells it precisely when to make use of each. The agent calls navigate to load the web page, extract_structured to tug the guide titles and costs by CSS selector, and synthesizes a structured listing within the remaining reply. The teardown() name after the agent finishes closes the browser cleanly so no zombie Chromium processes are left working.
Conclusion
The browser is just not a specialised device for automation engineers. It’s the common interface for the online, and the online is the place many of the world’s precise work will get completed. An AI agent that may use a browser doesn’t want a associate group sustaining API integrations. It may well attain something a human can attain.
What makes this sensible now, not simply theoretically fascinating, is the maturity of the tooling. Playwright handles the exhausting components of browser interplay. browser-use removes the necessity to write selectors for exploratory duties. LangGraph provides the LLM clear device hooks and a reasoning loop that handles variable web page buildings. The patterns on this article will not be demos. They’re the identical patterns 51% of enterprises now working AI brokers in manufacturing are constructing on.
Begin with the scraping instance. Get it working towards a website you really need knowledge from. Add the agent layer once you want choices the script can not anticipate. Add browser-use when the web page construction is simply too dynamic for selectors. Deploy in Docker once you want it working someplace aside from your laptop computer.
The exhausting half is just not the code. It’s understanding which device to succeed in for at every layer. Hopefully this text made that clearer.
