MCP server + macOS accessibility APIs: what discovery actually returns, and where it stops being enough

Every node of every running app's AXUIElement tree, for any app the host process has Accessibility permission for in System Settings. For each node you get: role (AXButton, AXTextField, AXStaticText, AXRow, AXWindow, AXMenu, etc.), the element's text where present (AXTitle, AXValue, AXDescription, AXLabel), top-left x/y in screen coordinates, width and height, the list of AX actions the element advertises (kAXPressAction, kAXShowMenuAction, kAXCancelAction), and the AX attributes (AXEnabled, AXFocused, AXSelected, AXValue). The macos-use server flattens that tree to one-element-per-line text and writes it to /tmp/macos-use/<ts>_<tool>.txt every call. Discovery is total. You are not crawling pixels.

Discovery itself does not break down. What breaks is the assumption that a discovered element is actuatable through the obvious path. Three known classes of failure show up. (1) Mac Catalyst right-pane controls (Messages, Maps, parts of System Settings) frequently expose AXButton with sensible text and coordinates, but a CGEvent left-mouse-down/up at the element's center is silently dropped. The element does not press, no error fires, the AX tree does not change. (2) Sandboxed apps with secure-input contexts ignore synthetic CGEvent typing. (3) Some Catalyst list rows and outline rows expose AXSelected but no AXPress action, so calling kAXPressAction on them returns kAXErrorActionUnsupported. None of these are discovery problems. The tree is correct. The default actuation path just is not the one that works.

Because of the actuation gaps above. The aggregate list at Sources/MCPServer/main.swift:1482 is open_application_and_traverse, click_and_traverse, type_and_traverse, press_key_and_traverse, scroll_and_traverse, set_value_and_traverse, press_ax_and_traverse, set_selected_and_traverse, refresh_traversal. Six of those are the standard surface: discover, click, type, press key, scroll, refresh. The other three are alternate actuators. set_value writes a string via kAXValueAttribute, bypassing the input event tap entirely (main.swift:1440-1444). press_ax performs kAXPressAction on the element under (x,y) instead of synthesizing a click (main.swift:1457-1461). set_selected sets kAXSelectedAttribute on table rows and sidebar entries that have no AXPress action (main.swift:1475-1479). All three carry the same hint verbatim: 'Use when a synthetic mouse click is dropped (Catalyst right-pane buttons, sandboxed apps).' The agent's plan is: try click first, fall back to press_ax, fall back to set_selected for selection-bearing controls.

Two named classes, and only from the diff response, not from the full text dump. isScrollBarNoise at main.swift:591-597 drops any role containing 'scrollbar', 'scroll bar', 'value indicator', 'page button', or 'arrow button'. isStructuralNoise at main.swift:599-607 drops empty (no text) AXRow, AXOutline row, AXCell, AXColumn, and AXMenu containers. Coordinate-only attribute changes are also stripped (main.swift:681-682) so that a diff between two traversals does not flag every pixel-level repaint. The full traversal .txt file under /tmp/macos-use still contains everything. The filter exists so an LLM consuming the post-action diff sees only changes worth acting on, not 'AXValueIndicator moved 2 pixels.'

The tree is always there; the quality differs. AppKit (Mail, Finder, the legacy Notes) emits a tight tree: AXButton with a real title, AXTextField with kAXValueAttribute that you can both read and write, AXMenuItem with a working kAXPressAction. SwiftUI emits a similar shape but more verbose because every modifier becomes a wrapper node; element counts are 1.5x to 3x what AppKit would produce for the same view. Mac Catalyst (UIKit on Mac, e.g. Messages, Maps, parts of newer System Settings) emits a tree that looks correct but is the one where synthetic CGEvent clicks fail most often; the fallback tools were written specifically for this. Electron (Slack, Discord, VS Code) emits thousands of AXGroup and AXStaticText nodes from the Chromium accessibility shim. The tree is huge, semantic roles are washed out, but text is still extractable and CGEvent clicks usually work.

Two grants, in System Settings > Privacy & Security. (1) Accessibility, granted to the process that loads the MCP server. On Claude Code that is the terminal app (iTerm, Terminal, Ghostty), not the MCP binary; the binary inherits the grant from its parent. Grant it to the wrong process and AXUIElementCreateApplication returns -25204 on every call. (2) Screen Recording, granted to the same process. CGWindowListCreateImage on Sequoia returns blank or menu-bar-only frames without it, so the per-action screenshot the server attaches is empty. Both grants are checked once per macOS login session and cached.

Yes. Three categories. (1) Apps that opt out: some Metal-only game engines and DRM video players publish a single AXWindow with no children, because they bypass AppKit entirely. (2) Custom-drawn canvas that does not implement accessibility: a CAD tool drawing in a single NSView without exposing per-shape AXGroups produces one giant node. (3) Cross-process IPC state: the AX tree shows the visible UI of the focused process, not the data the app is fetching, not its network calls, not what is in another app's window unless you walk that app's AX root by PID. Anything that is not visible UI is not in the tree. For those cases the agent has to fall back to coordinate-driven clicks against the screenshot, or to a different control surface entirely (CLI, file, API).

Neither, exactly. Every tool call writes the full tree to /tmp/macos-use/<timestamp>_<tool>.txt and returns a compact text summary to the model: a status line, the PID and app name, the file path, the file size and element count, a grep hint, the screenshot path, and a small sample of visible elements. The model uses Read on the .txt with offset/limit or Grep on it to find specific roles or text. This pattern exists because a naive full-tree return on a typical Slack window is 8000+ elements and 200-800 KB of JSON, which every model on the market gets lost on inside a tool loop. The summary plus file pattern keeps token cost flat per action.

Install the server with `claude mcp add macos-use -- npx -y mcp-server-macos-use`. Restart Claude Code. Grant your terminal Accessibility and Screen Recording. Then in a Claude Code session say: 'open Calculator and dump the AX tree.' The agent calls macos-use_open_application_and_traverse({identifier: 'Calculator'}). The response includes a `file:` field with the .txt path and a `screenshot:` field with the .png. Open the .txt yourself with `bat` or `less`; every line is `[Role] "text" x:N y:N w:W h:H visible`. Grep for AXButton: you see the 0-9, +, -, *, /, =, AC, +/-, %, . keys with their coordinates. That is discovery. Asking the agent to click the 7 button next exercises the actuation half.