AI Agent UI State Checkpointing: The Three Snapshots macOS-Use Takes Around Every Click

It is not graph-state checkpointing in the LangGraph sense. It is a per-tool-call transaction at the OS layer. Before every disruptive action (click, type, press, scroll, open) the server captures three snapshots: NSWorkspace.shared.frontmostApplication at main.swift:1671, NSEvent.mouseLocation flipped into top-left CGEvent coordinates at main.swift:1672-1676, and the full AX tree of the target app via showDiff = true at main.swift:1600/1617/1633/1651. The action runs. Two of those snapshots are then restored: the cursor via a CGEvent mouseMoved at main.swift:1767-1772, and the previous frontmost app via prevApp.activate at main.swift:1775-1781. The third (the AX tree) is differenced against a fresh post-action traversal and written to /tmp/macos-use/ as the artifact the agent reads.

The agent does not. The human does. macOS automation that fights the user for foreground focus or strands the cursor in the wrong app produces a workflow nobody can supervise. The contract this server encodes is: when the tool call returns, the human's interrupted state is back where they left it. That is what makes the loop usable as something the user runs in the foreground while still doing their own work, instead of a batch script they kick off and walk away from.

InputGuard.swift sets watchdogTimeout = 30 seconds at line 24. The engage() path schedules a one-shot timer at InputGuard.swift:174 that fires regardless of whether the action ever returns. When the timer fires it logs 'watchdog fired after 30s — auto-disengaging' and tears down the event tap. Without that, a crashed Swift process holding an active CGEventTap could lock keyboard and mouse for the user. The 30-second ceiling is the lockout safety net.

InputGuard installs a CGEventTap that intercepts every keyboard and mouse event during automation. When it sees an Escape keydown with no modifiers, it sets _cancelled = true at InputGuard.swift:292 and invokes onUserCancelled. The MCP handler checks InputGuard.shared.throwIfCancelled() between every primary and additional action (main.swift:1708, 1721, 1728, 1734) and again after a 200ms grace period at main.swift:1757-1763. If cancelled, it throws InputGuardCancelled, which the catch block at main.swift:1847 traps. Inside that catch block: disengage the guard, restore the cursor, reactivate the previous frontmost app, return an isError response. The cancellation path runs the same restore code as the success path.

AppKit gives mouse coordinates with origin at the bottom-left of the primary screen. CGEvent posts mouse events with origin at the top-left. main.swift:1673-1675 flips by computing primaryScreen.frame.height - nsPos.y so the saved CGPoint can be passed directly back to CGEvent(mouseEventSource:mouseType:.mouseMoved, mouseCursorPosition:) on restore. Skip that flip and the cursor restores to the wrong y on multi-monitor setups, sometimes off-screen entirely.

Every tool call writes two files to /tmp/macos-use/ named with millisecond-precision timestamps: <ts>_<tool>.txt and <ts>_<tool>.png. The .txt holds the flat-text diff produced at main.swift:1007-1028 with prefixes + (added), - (removed), and ~ (modified, with attribute transitions). The .png is captured at main.swift:1832-1840 with a red crosshair drawn at the click point if applicable. Both files share the same timestamp so an agent can correlate the visual receipt with the symbolic diff.

The handler detects this at main.swift:1788-1808. After the action it compares the current NSWorkspace.shared.frontmostApplication processIdentifier against the original PID the tool was called with. If they differ, it traverses the new frontmost app, populates appSwitchPid, appSwitchAppName, and appSwitchTraversal on the response, and appends an 'app_switch:' header to the .txt file. The agent gets one tool call, one .txt, but two traversals when focus escapes. This is also the only case where the frontmost-app restore is intentionally relaxed: if the previous frontmost was the launching app and the launched app is now what the user expects to see, restoring would be wrong. The restore at main.swift:1775-1781 only fires when isDisruptive and the previous app is still alive.

No, it is orthogonal. LangGraph checkpointers persist agent graph state (the AgentState dict, message history, the next-node pointer) into a SQLite or Postgres backend so the workflow can resume after a process exit. AG-UI synchronizes UI state between a running agent and a web frontend. Neither addresses what mcp-server-macos-use addresses: the OS-level state of a real human's desktop while an agent is poking at it. You can run macos-use under a LangGraph agent and stack both layers — the LangGraph checkpointer covers conversation resume, the macOS checkpoint-restore covers per-action atomicity on the user's desktop.

Think of each tool call as a database transaction. traverseBefore captures the read snapshot. The action is the write. traverseAfter captures the post-write read. The diff at main.swift:612-718 is the changed-rows result set. Filters at main.swift:591-607 strip non-actionable noise before persisting. The /tmp/macos-use/<ts>_<tool>.txt file is the commit log. If you ran the agent for a thousand actions, /tmp/macos-use/ would hold a thousand .txt + .png pairs, one per transaction, replayable from disk for post-mortem.

No. The CGEventTap installed by InputGuard at the .cghidEventTap layer filters events by source. Agent input is generated via CGEvent.post(tap: .cghidEventTap) inside the same process; those events are emitted from the macos-use binary and are not blocked. Human input from the physical keyboard and mouse is blocked because it originates from outside the process. The Esc keydown is the one human keystroke the tap forwards: it captures, sets _cancelled, and lets the rest of the event drop. Other keystrokes are swallowed silently.

The pattern generalizes; the APIs do not. macOS uses NSWorkspace + NSEvent + CGEvent and exposes the foreground app via processIdentifier. The Windows equivalent (the Terminator project, also MCP-speaking) uses GetForegroundWindow + UI Automation and would need GetCursorPos / SetCursorPos instead of CGEvent mouseMoved, and would diff against UIA tree snapshots instead of AX tree snapshots. The contract — three snapshots, two restored, one diffed and persisted — ports cleanly. The implementation is per-OS.

Build the binary with xcrun --toolchain com.apple.dt.toolchain.XcodeDefault swift build, point an MCP client at it, and call click_and_traverse on any Mac app while you have a different app frontmost. Watch /tmp/macos-use/ for the new <ts>_click_and_traverse.txt + .png pair. Move your cursor to the corner of the screen before triggering the call; when the call returns, the cursor lands back at that corner and the originally-frontmost app is foreground again. The .txt holds the AX diff. That is all three checkpoints visible in one round trip.