macOS Accessibility Tree Automation: The Control-Arbitration Problem Every Other Guide Skips

Six things happen in order around every disruptive tool call (open, click, type, press, scroll). One: the handler captures the cursor location with NSEvent.mouseLocation and the frontmost app with NSWorkspace.shared.frontmostApplication at main.swift:1671-1675. Two: InputGuard.shared.engage installs a CGEventTap on .cghidEventTap at the head-insert position with a mask covering keyDown, keyUp, both mouse buttons, mouseMoved, dragged, scrollWheel, and flagsChanged at InputGuard.swift:113-153. Three: a 30-second DispatchSource timer is armed as a watchdog at InputGuard.swift:172-181. Four: a fullscreen NSWindow at .screenSaver level is shown with a centered dark pill, a 16-point pulsing orange dot, and a 20-point semibold white label that says what the AI is doing. Five: the action runs and posts CGEvents that pass through because their eventSourceStateID is non-zero. Six: the overlay disengages, the cursor is restored to its saved point, and the previously frontmost app is reactivated at main.swift:1767-1781.

Source state ID. Every CGEvent carries an eventSourceStateID field. Programmatic events posted via CGEvent.post with the .hidSystemState source carry a non-zero stateID. Hardware events that originate from the keyboard or trackpad always carry stateID = 0. The tap callback at InputGuard.swift:329-332 reads that field first and short-circuits with Unmanaged.passUnretained(event) when the value is non-zero. That is how the agent can synthesize a keystroke through the same tap that is currently swallowing your typing: the bit is set by the OS, not inferable from key code alone.

InputGuard.swift:340-351 checks for keyCode == 53 and zero intersection with the modifier mask {.maskCommand, .maskControl, .maskAlternate, .maskShift}. Cmd-Esc, Opt-Esc, and Shift-Esc are all blocked along with the rest of your input. Only the unmodified Esc reaches handleEscPressed, which sets the cancelled flag, disengages the tap, and calls the optional onUserCancelled callback. The reason the bar is plain Esc and not a chord is reachability under panic: if the agent has done something visibly wrong, you want the dumbest possible single-key to interrupt, not a combo your hand has to find under pressure.

The watchdog. InputGuard.swift:24 declares watchdogTimeout: TimeInterval = 30 and startWatchdog at InputGuard.swift:172-181 schedules a DispatchSource timer on a global queue that calls disengage() unconditionally after 30 seconds. Even if the action never returns, the tap is torn down, the run loop source is removed from CFRunLoopGetMain, and the overlay is hidden. The 30-second budget is also why click-then-type-then-press is exposed as a single composed tool call rather than three separate ones: a chained call costs the same one budget window as a single click.

buildAndShowOverlay at InputGuard.swift:202-276 creates a borderless NSWindow at .screenSaver level (above almost everything except the menu bar), tinted black at 0.15 alpha so the desktop bleeds through, with collectionBehavior = [.canJoinAllSpaces, .fullScreenAuxiliary] so it follows you across Spaces and into fullscreen apps. Centered: a 720pt-or-50%-wide pill at 92% opacity, rounded to half its 80pt height, holding a 16pt orange dot that pulses between 1.0 and 0.3 opacity every 0.8 seconds (CABasicAnimation, autoreversing, infinite repeat) and a 20pt semibold label. The intrusiveness is the point. If the model is driving your computer, the screen should look different from when you are.

At main.swift:1672-1675 the handler reads NSEvent.mouseLocation (which uses bottom-left AppKit coordinates), flips the y axis using primaryScreen.frame.height, and stashes the resulting CGPoint. After the action, main.swift:1767-1771 builds a CGEvent with type .mouseMoved at that saved point and posts it via cghidEventTap. The cursor visually snaps back to where it was when the call started, even if the agent dragged it across three monitors during a click. This works because backingScaleFactor on these screens is 1.0 (1pt == 1px), as documented in the project CLAUDE.md.

No. main.swift:1671 saves NSWorkspace.shared.frontmostApplication at the start, and main.swift:1775-1781 calls .activate(options: []) on that saved app after the action, but only if the previous app is still alive (prevApp.isTerminated == false). If the action handed focus off to a different app and that handoff is the user's intent, the diff already records it: main.swift:1788-1808 detects cross-app frontmost changes and traverses the new frontmost app, attaching its tree as appSwitchTraversal in the response. Focus restoration only kicks in for incidental focus theft, not for intentional handoffs.

Yes. macOS will disable a CGEventTap if it takes too long to return from the callback (.tapDisabledByTimeout) or if user input subverted it (.tapDisabledByUserInput). InputGuard.swift:298-306 catches both cases in the callback's preamble and calls CGEvent.tapEnable(tap: tap, enable: true) to re-arm without touching the run loop source. Because the tap was inserted at the head-insert placement, it remains the first listener after re-enable. This is the difference between 'I wrote a CGEventTap once' and 'a CGEventTap that survives a long automation run'.

Same shape, different host. macos-use uses AXUIElement APIs for the tree and CGEventTap for input arbitration. Terminator uses UI Automation for the tree and a Windows raw-input hook for the equivalent block-and-pass-through. Both speak MCP, so an agent that wants to run on a mixed fleet holds the same mental model: the OS gives you a structured tree, the server filters noise, and the server protects the human's input boundary while the action happens. The specific code (.cghidEventTap vs RegisterRawInputDevices) differs; the contract does not.

Banners are advisory; a tap is enforced. If you only show an overlay, the user's keystrokes still race the agent's into the focused app. A real example: agent is mid-way through typing a Slack message, you bring up Spotlight by reflex, your Cmd-Space lands inside Slack as Cmd-Space, and the message text fragments. With a head-insert CGEventTap, your Cmd-Space is intercepted before any app sees it; the agent's CGEvent.post calls flow through because their stateID is non-zero. The tap is what makes 'AI is using the computer' a hard fact, not a polite request.

Clone the repo. xcrun --toolchain com.apple.dt.toolchain.XcodeDefault swift build. Point an MCP client at .build/debug/MCPServer. Call open_application_and_traverse with a small target like Calculator. Then call type_and_traverse and start mashing your keyboard during the 800ms the call takes. Your keys land nowhere (the tap swallows them). The orange-dot pill is centered on screen. /tmp/macos-use/tap_status.txt is rewritten with tap_created: enabled=true at <timestamp>. Press Esc; /tmp/macos-use/esc_pressed.txt appears with the timestamp and the call returns an InputGuardCancelled error. Two minutes from clone to verified.

It is most acute for AI because LLM-driven actions are slower, less predictable, and more frequent than scripted ones. A 50ms AppleScript click finishes before you can interrupt it; a model-issued click_and_traverse can take 700ms because of the before-and-after traversals. That window is exactly long enough for a human to start typing. But the same primitives (event tap, programmatic stateID filtering, watchdog auto-release, cursor restore) apply to any long-running GUI automation: long-running build wizards, recorded macros that cross app boundaries, and accessibility-tree fuzzers all benefit from a hard input boundary while the script runs.