MCP Agent Plan Execution On A Real Desktop: Why macos-use Refuses To Have A Plan Primitive

No. The server exposes six tools (open, click, type, press_key, scroll, refresh) and nothing else. There is no plan object, no sequence handle, no agent-side state the server persists. The agent is the planner. What the server does offer is optional parameter chaining: click_and_traverse accepts optional text and pressKey params, so click + type + press fires inside one tool call. That is as close as the server gets to a plan primitive, and it is intentional. See Sources/MCPServer/main.swift:1300-1408 for the tool definitions.

One. Calling click_and_traverse with { x, y, text: 'hello world', pressKey: 'Return' } runs traverseBefore → click → type → press Return → traverseAfter inside the server, returns one response with one diff, and writes one /tmp/macos-use/<ts>_click_and_traverse.txt file. A naive agent that issues three separate MCP calls (click, then type, then press) pays for 3x the LLM round-trips, 3x the traversal noise, and 3x the chance of the plan getting interrupted between tool calls. The composed path is main.swift:1709-1751.

Press Esc. InputGuard installs a CGEventTap at .cghidEventTap that sees every keystroke while the tool call is running. An Escape keydown with no modifiers sets _cancelled = true at InputGuard.swift:292. The handler polls InputGuard.shared.throwIfCancelled() between every action in the chain: main.swift:1708, 1721, 1728, 1734. After the chain finishes there is a 200ms grace window at main.swift:1757 during which a late Esc still cancels. On cancel the handler throws InputGuardCancelled, runs the cursor + foreground restore code, and returns an isError response. The plan stops at the current boundary, not at the final step.

A diff. After every action the server traverses the target app twice (before and after), subtracts them, and returns the delta as a flat-text list prefixed with + added, - removed, and ~ modified. The diff gets written to /tmp/macos-use/<timestamp>_<tool>.txt alongside a PNG screenshot with a red crosshair at the click point. The agent reads that file (grep by role, by text, by coordinates) to choose the next tool call. There is no JSON blob of 'plan progress' the server maintains between calls. The diff is the state. See main.swift:1007-1028 for the diff format and main.swift:1828-1840 for the file write.

The server catches it without the agent asking. After the action, main.swift:1788-1808 compares NSWorkspace.shared.frontmostApplication processIdentifier against the PID the tool call was addressed to. If the frontmost app changed, the handler traverses the NEW frontmost app, writes its tree into toolResponse.appSwitchTraversal, and sets appSwitchPid + appSwitchAppName. The agent's next planning step gets two trees in one response: the diff of the original app, plus the full tree of whatever is now in front. No retry loop, no 'app not found' error.

scroll_into_view kicks in automatically. main.swift:1159-1285 computes the direction from the click point relative to the window bounds, then scrolls up to 30 steps (main.swift:1189, maxSteps = 30) with line-scaled deltas: 1 line/step if the distance is under 80px, 2 lines if under 250px, 3 lines otherwise. After every scroll step it probes the AX tree by text match (when the target has text) or by watching the viewport edge for newly-revealed elements. If the target appears, it clicks. If it doesn't appear after 30 steps, it logs a warning and clicks at the original coordinates anyway. The agent never learns the element was off-screen; it just gets a diff that shows the click landed.

Accessibility-driven scrolling is cheap but not free (each step posts a CGEvent scrollWheelEvent2 at main.swift:1196 and waits 100ms before the next tree probe). 30 steps at 2 lines each ≈ 60 scroll lines ≈ ~1500px of travel, which covers most practical cases (long tables, sidebar lists, chat scrollback). Beyond that the likely explanation is that the AX coordinate the agent computed is stale, the window scrolled back, or the element is inside an unscrollable container. An unbounded loop would turn a planning error into a silent hang. The cap turns it into a 3-second log line.

Those operate above MCP: they describe how a planner LLM breaks a user goal into sub-tasks, dispatches them to worker agents, and synthesizes results. Plan execution in that world is an orchestration-layer problem solved with Temporal-style durable workflows, code-compilation of plans, or multi-agent message passing. macos-use operates below MCP: it is one of the tools those planners call. Its job is to make a single call on a real desktop atomic, cancellable, and self-repairing, so the outer orchestration does not have to retry for reasons like 'the user was typing' or 'the window scrolled.' You can and should run both. See the SERP results at the top of the page for links.

Not yet. The composed path at main.swift:1710-1751 special-cases a primary action followed by a series of input actions (type or press). You can issue multiple presses in the pressKey param (the exact parsing is at main.swift:1349-1383 for type_and_traverse and 1384-1408 for press_key_and_traverse). If you need a richer sequence, issue it as separate MCP calls; the diff-as-state model means each call self-plans off the previous result.

The guard auto-releases the event tap after 30 seconds regardless of whether the tool call returned (InputGuard.swift:24 sets watchdogTimeout = 30). If your chained plan is still running when the watchdog fires, the input tap drops, the user regains full keyboard/mouse control, but the agent's OS events are still posted (they come from inside the macos-use process and do not need the tap). The tradeoff is safety over automation duration: a crashed Swift process holding an engaged tap would lock the Mac, so the ceiling is non-negotiable. If you need longer plans, split them into multiple tool calls and let the agent re-engage the guard per call.