SoM What Set of Marks actually does

Your agent sees the page.
Then clicks the number.

BrowserControl is a local MCP server that hands any AI agent a real Chromium browser it can see, click, type into, and debug — through numbered marks drawn over every interactive element. No selectors. No cloud. No API bill.

Get started View on GitHub
pip install browsercontrol
  • Runs 100% on your machine
  • No LLM API key
  • MIT licensed
  • Python 3.11+
39 MCP tools, every one returning a fresh annotated screenshot
7 Categories: navigation, interaction, tabs, forms, content, devtools, recording
$0 Marginal cost per 1,000 browser actions
0 Bytes of your browsing sent anywhere off-device

The loop

See the page. Pick a number. Act.

Every tool that touches the page returns the same two things: an annotated screenshot and a textual element map. The model reads pixels and labels, then names a number. Clicks resolve through document.elementFromPoint(), so sticky headers, overlays, and cookie banners never steal the hit.

1

See

A screenshot comes back with numbered red boxes over every interactive element — inputs, links, buttons, open shadow-DOM descendants, and same-origin iframes with their coordinate offsets already applied.

2

Choose

Alongside it, a compact element map lists each mark with its tag and accessible label. The model picks a number — never a brittle div.flex > button:nth-child(3) it hallucinated from stale HTML.

3

Act

The agent calls click(5). BrowserControl resolves the live DOM node under that box, drives Chromium, and re-marks the new page. Cookies and login state survive restarts.

agent session 
agent  › navigate_to("https://app.example.com/sign-in")
       ‹ screenshot + element_map — 5 elements marked
agent  › type_text(3, "ada@example.com")
agent  › type_text(4, "••••••••••")
agent  › click(5)
       ‹ ✓ clicked <button> "Sign in" → /dashboard — 12 new elements
agent  › get_console_logs()
       ‹ 0 errors, 2 warnings

Why it works

Shaped around how agents fail, not how humans test.

Selector-driven automation assumes you already know the DOM. Agents don’t — they guess, and they guess wrong. Every decision below exists to remove one of those guesses.

Vision-first, selector-free

Numbered red boxes over the live page. The agent names a number. There is no CSS selector to get wrong, no XPath to invent, no class name to hallucinate.

Shadow DOM & iframe aware

The element collector descends recursively into open shadow roots and same-origin iframes, compensating coordinate offsets as it goes. Component-framework apps just work.

Sessions that actually persist

Chromium runs against a real profile directory via launch_persistent_context. Cookies, localStorage, and login state survive server restarts. Log in once, not once per run.

DevTools in the same server

Console logs, network requests with timing, uncaught JS errors, performance metrics, computed styles, cookies. Debugging a page needs no second MCP server.

Local, private, uncapped

No cloud relay, no telemetry, no vendor rate limit. Screenshots are captured, annotated, and consumed on your machine — including the ones of your logged-in accounts.

Zero marginal cost

Everything runs on hardware you already pay for. A thousand clicks cost exactly the same as one: nothing. No per-action pricing, no surprise invoice.

How it compares

Three ways to give an agent a browser.

Selector-based drivers are precise but blind. Hosted browser agents can see, but they meter every action and route your session through someone else’s infrastructure.

  BrowserControl Selector-driven MCP Hosted browser agent
How the agent targets Numbered marks on a screenshot CSS / XPath it has to guess Natural language, resolved remotely
Sees the rendered page Yes, on every single call No — text and DOM only Yes
Where it runs Your machine Your machine Vendor cloud
Cost per 1,000 actions $0 $0 Metered per action
Logged-in sessions Persistent local profile Usually per-run, ephemeral Credentials leave your machine
Console & network access Built in Rarely Rarely exposed

Quickstart

Installed and wired up in about a minute.

Chromium installs itself on first run, so there is no system browser binary to manage and nothing to keep in sync.

  1. Install the package

    Use whichever package manager you already have.

    pip install browsercontrol
    uv add browsercontrol
    pipx install browsercontrol

  2. Point your client at it

    Drop the server block on the right into your MCP client config — Claude Desktop, Claude Code, Cursor, or anything else that speaks MCP.

  3. Ask for something visual

    “Open my dashboard, screenshot it, and tell me which network requests failed.” The first call boots Chromium and marks the page. See the first session walkthrough.

claude_desktop_config.jsonjson
{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol"
    }
  }
}

If Chromium ever fails to install itself, run python -m playwright install chromium once. Every configuration option is an environment variable — see Configuration.

Documentation

Where to go next.

Four entry points, depending on what you’re trying to do right now.

Start here

Getting started

Install, connect your client, and run a first session end to end.

 Reference

Tool reference

All 39 MCP tools with parameters and return shapes, by category.

 Recipes

Guides

Research, debugging, form filling, multi-tab work, recorded test runs.

 Under the hood

Concepts

How marks are collected, why IDs are ephemeral, and the architecture.

Give your agent a browser that actually sees.

One install, no cloud, no key. The shortest path from “ask the model” to watching the browser do the thing.

Install BrowserControl Star on GitHub