Drive native macOS apps via the AX tree with MCP: the 9 tools and the one chaining trick

Two halves of one round trip. Every tool except refresh_traversal is 'disruptive' (defined at main.swift:1800): the server first acts on the app (CGEvent post, AX attribute write, AX action) and then re-traverses the tree, writes the new state to /tmp/macos-use/<ts>_<tool>.txt with a diff against the previous traversal, and returns the file path to the LLM. Reading is free (refresh_traversal). Anything that mutates state goes through one of the eight other tools and pays the read cost on the way out.

Because the LLM has to know in its tool schema which actuation primitive it is asking for. click_and_traverse posts a synthetic CGEvent mouse down/up; type_and_traverse posts CGEvent keystrokes through the system source; press_ax_and_traverse calls AXUIElementPerformAction with kAXPressAction; set_value_and_traverse calls AXUIElementSetAttributeValue with kAXValueAttribute; set_selected_and_traverse writes kAXSelectedAttribute. They have different fallback semantics and different failure modes (Catalyst right-pane buttons silently drop synthetic clicks but accept press_ax). A single generic tool would force the model to encode that decision in a string parameter the schema cannot constrain. Nine narrow tools let the model pick the right primitive from the description and the schema validates the call before the server runs.

click_and_traverse accepts three optional parameters that turn a multi-tool dance into one call: element (case-insensitive partial text match across visible AX nodes), text (typed after the click lands), and pressKey (key name pressed after the text is in). One MCP round trip becomes click + type + return. The reason it is the default is in the server's own instructions block at main.swift:1488-1507: 'CRITICAL — Minimize tool calls by chaining actions. Example: to type into a Slack message box and send it, use ONE click_and_traverse call with element="Message to X", text="hello", pressKey="Return" — do NOT split into separate click, type, and press calls.' The server tells the model this on every connect; the model defaults to the chained form on first contact.

Two paths. Path one: pass element="Send" and let the server scan the previous traversal for a visible AX node whose text contains that substring; it clicks the first match. Path two: read the .txt file at /tmp/macos-use/<ts>_<tool>.txt, parse one of the lines that looks like [AXButton] "Send" x:820 y:612 w:60 h:28 visible, and pass (x, y, w, h) explicitly so the tool centers the click at (x + w/2, y + h/2). The server instructions block explicitly forbids estimating coordinates from screenshots ('NEVER estimate coordinates visually from screenshots') because pixel positions in the .png and screen coordinates differ by the window origin offset. The .txt is the ground truth.

Synthetic CGEvent clicks land cleanly on AppKit and most SwiftUI apps but are silently dropped by Mac Catalyst right-pane controls (Messages, Maps), some sandboxed apps, and any field in a secure-input context. The tree shows the element, the click coordinates are right, the action just does not register. The escalation lives in three other tools: press_ax_and_traverse (kAXPressAction on the element) for buttons, set_value_and_traverse (kAXValueAttribute) for text fields that ignore typed events, and set_selected_and_traverse (kAXSelectedAttribute) for list and outline rows that expose AXSelected but no AXPress action. The fallback ladder is documented in the side guide on the three-rung escalation.

Yes, briefly, for every disruptive call. The handler at main.swift:1835 fires InputGuard.shared.engage which installs a CGEventTap at .cghidEventTap headInsert with a mask covering keyDown, keyUp, both mouse buttons, mouseMoved, dragged, scrollWheel, and flagsChanged. The tap callback drops hardware events (eventSourceStateID == 0) and forwards programmatic events (non-zero stateID), which is how the agent's synthesized clicks pass through while your typing does not. The overlay shows a pulsing orange dot and a one-line label of what the agent is doing. Plain Esc with no modifiers is the kill switch. A 30-second watchdog auto-disengages if anything hangs.

Any client that speaks MCP: Claude Code, Cursor, Claude Desktop, Cline, VS Code (with an MCP extension), Windsurf, and Zed. The server is stdio-based, registered via npx, and known to work with every client that follows the standard MCP transport. Claude Code is the lowest-friction install: claude mcp add macos-use -- npx -y mcp-server-macos-use. For the JSON-config clients (Cursor, Claude Desktop, VS Code, Windsurf, Cline, Zed) the snippet is in the install block on this page.

A compact text summary, not the full tree. The summary contains: a file: field pointing at /tmp/macos-use/<ts>_<tool>.txt with the full element list, a screenshot: field pointing at a sibling .png of the captured window, and a visible_elements: sample showing up to a few dozen on-screen elements with coordinates. The full tree (often 1000+ lines for a real app) stays on disk; the model greps the file when it needs to find an element by text. Smaller summary on the wire, lower token cost per turn, full state available when the model asks.

Every disruptive call writes both an absolute traversal and a diff against the prior traversal. The diff is line-prefixed: + for new elements, - for removed elements, ~ for elements whose attributes changed. Coordinate-only churn (a window shifted one pixel) is filtered out before the diff is built. Scroll-bar and structural-only noise is dropped at main.swift:591-607. The point is to give the model a small, semantic delta after each action: 'this button became enabled', 'this row was selected', 'this text field's AXValue changed from empty to your typed string', not the entire tree again.

Fazm's screen-control feature is built on the same Swift core. The MCP server is the standalone surface around that core, packaged so any MCP client can drive it without touching the Fazm app. Open source under MIT, repo at github.com/mediar-ai/mcp-server-macos-use. The maintainers (Mediar) ship a consumer macOS app on top of it, so behavior is exercised in production every day, not only in the README.