When the AI and your hand reach for the same mouse

Native computer use shares one set of hands

There are two broad ways to give an AI computer use. The first is isolation: spin up a virtual machine or a remote desktop, screenshot it, and let the model click around inside a sandbox that is not your working session. The second is native: drive the machine you are actually sitting in front of. Browser-only servers are a narrow version of the first. macos-use is the second, because the entire reason it exists is to reach the apps a browser cannot: Finder, Mail, Calendar, Xcode, and whatever third-party Mac GUI you live in.

Native is more powerful and also more dangerous in a very specific, physical way. The model posts a left mouse click and the real cursor jumps. It types a string and the characters land in whatever app holds focus. If you happen to click somewhere else at the same instant, focus moves, and the model’s next keystroke goes into the wrong window. The failure mode is not abstract. It is you and a language model both trying to steer one cursor a few times a second.

Most write-ups on macOS computer use stop at “use the accessibility API, not screenshots.” True, and covered well elsewhere. The unglamorous engineering that makes native computer use actually usable is the handoff: a clear, enforced boundary around the moments the AI is holding the controls.

The handoff, in order

When a tool call that moves the mouse or types arrives, the server wraps the action in an engage / disengage pair. Your hardware is locked out for exactly that window, and only that window.

Actor 0 is the macOS HID layer the events flow through; actor 1 is the server; the “You” column is where your physical input and your Esc both originate.

How the guard tells your hand from the model’s

A tap that blocked all input would block the server’s own clicks too, and nothing would happen. The trick is that the events the server synthesizes and the events your hand generates are not identical at the system level. Synthetic CGEvents posted by the server carry a non-zero eventSourceStateID; genuine hardware events carry zero. The callback reads that field on every event and uses it as the gate.

That single check is what makes the whole thing work. Pass the server’s own events through untouched, swallow everything that came from a human hand, and special-case one key, Esc, as the escape hatch.

// inputGuardCallback — runs for every hardware event during automation
let sourceStateID = event.getIntegerValueField(.eventSourceStateID)
if sourceStateID != 0 {
    return Unmanaged.passUnretained(event)   // our own synthetic CGEvent, let it through
}

// keycode 53 == Esc. No Command/Control/Option/Shift held.
if type == .keyDown {
    let keyCode = event.getIntegerValueField(.keyboardEventKeycode)
    let modifierMask: CGEventFlags = [.maskCommand, .maskControl, .maskAlternate, .maskShift]
    if keyCode == 53 && event.flags.intersection(modifierMask).isEmpty {
        guard_.handleEscPressed()
        return nil                            // swallow Esc, cancel the run
    }
}

return nil                                    // block every other hardware event

Returning nil from a CGEventTap callback drops the event; returning the event passes it on. So return nil on the last line is what actually freezes your trackpad.

The lifecycle of one click

Reads do not engage the guard, only actions that touch your hardware do. Here is what happens around a single disruptive tool call.

What the tap blocks, and what slips through

The event mask covers the full set of input you could use to disrupt a run. Everything below is intercepted while the guard is engaged, with two deliberate exceptions.

The two unchecked rows are not failures. They are the intentional gaps that let the model work and let you bail out.

Why this is the part that matters

A sandboxed, screenshot-based computer-use setup does not need any of this. The model is poking at a VM you are not touching, so there is no contention and no need for an escape key. The price is that it can only do what lives inside the sandbox, which on macOS means you give up the native apps that are the whole reason to automate a Mac in the first place.

The moment you choose native control, the shared-hardware problem is yours to solve, and a guard like this is not a nice-to-have. It is the thing that makes you willing to hand the model your real cursor at all, because you know one keypress takes it back and a stuck run releases on its own in 30 seconds. If you are evaluating any macOS computer-use tool, ask where its Esc is. If the answer is “force-quit the client,” that tells you how seriously the handoff was designed.

macos-use is open source under BSL 1.1, runs on macOS 13 and up, and drops into Claude Code, Cursor, VS Code, or Claude Desktop as a standard stdio MCP server. The input guard is the same code path whether you run it standalone or through a product built on top of it.

Frequently asked questions