How To Control Someone's Screen On FaceTime When The Viewer Cannot See Your Cursor: The Accessibility-Tree Diff That Narrates Every Click

Three blocks under a header. The header is 'diff: +N added, -N removed, ~N modified' written at main.swift:1008. Added elements print with a plus prefix at main.swift:1014 ('+ [AXButton (button)] "Send" x:820 y:612 w:60 h:28'). Removed elements print with a minus prefix at main.swift:1017. Modified elements print with a tilde prefix at main.swift:1026 in the shape '~ [AXButton] "Send" | AXEnabled: \'false\' -> \'true\''. The full response is written to /tmp/macos-use/<timestamp>_<tool>.txt so you can grep it later.

SharePlay encodes at roughly 30fps and compresses text aggressively. Small UI state changes, like a disabled button going to enabled or a label swap from 'Send' to 'Sending…', are routinely lost to compression blur. The diff is unambiguous: the exact element role, the exact before and after text, the AXEnabled change. The remote viewer's AI reads 'AXButton Send changed AXEnabled false -> true' and narrates 'the Send button is enabled now' without ever inspecting a video frame.

The switch is inside buildToolResponse at main.swift:612 on the hasDiff flag. hasDiff is true for click, type, press, scroll — the four that mutate UI state. open_application and refresh_traversal return a full traversal instead, written out by the branch at main.swift:720-722. So the diff format is specific to mutation calls, which is the useful case during a FaceTime session. You do not need a full dump of the accessibility tree after every click, just what changed.

ScreenshotHelper/main.swift:70-85. After CGWindowListCreateImage captures the frontmost window, ScreenshotHelper draws a 2pt red stroke crosshair with 15pt arms centered at lastClickPoint, plus a 10pt radius circle around it. The click coordinates are passed from main.swift:1839 via the --click-point flag on the helper subprocess. lastClickPoint is set per-call at the click_and_traverse handler site, so the PNG shows where the cursor landed even though the cursor itself has already snapped back.

main.swift:1825-1829. The handler builds a timestamp in milliseconds ('Int(Date().timeIntervalSince1970 * 1000)'), strips the 'macos-use_' prefix from the tool name, and writes the response to '/tmp/macos-use/<ts>_<toolname>.txt'. The screenshot at main.swift:1834-1839 reuses the same timestamp so the .txt and .png names match. If you collect five clicks in one call they will be 1713456789012_click_and_traverse.txt through 1713456792512_click_and_traverse.txt, each paired with its own PNG.

Filtering happens in buildToolResponse at main.swift:648-718. Scroll-bar elements are dropped by isScrollBarNoise (main.swift:591). Structural containers like AXRow, AXCell, AXColumn, AXMenu without text are dropped by isStructuralNoise at main.swift:600-607. Coordinate-only changes (x, y, width, height attributes) are filtered out of modified entries at main.swift:681-682. What you are left with is role + text + the semantic attribute that flipped, which is exactly what narrates well.

The tool returns a short summary to the MCP client, with the full diff written to the .txt file. The summary at main.swift:838-857 collects up to three modified elements whose changed attribute is 'text' or 'AXValue' and prints them as 'text_changes:' followed by 'old' -> 'new' lines. That is the terse signal the AI reads first. If it wants more, the 'file:' line tells it where to grep. The hint line at main.swift:761 even shows the grep command: 'hint: grep -n AXButton <filepath>'.

Only the host's AI can. The .txt and .png live in /tmp/macos-use/ on the host machine. The MCP client (running on the host) sees the summary, then can shell out to read the full file if it decides to. The remote viewer sees neither; they see the host AI's narration and the SharePlay video feed. The receipt pair is for the host: it is what they hand a teammate, an auditor, or a bug report after the call to say 'this is exactly what happened'.

Yes, via the cross-app handoff section at main.swift:1788-1808. If hasDiff is true and the frontmost app PID changed from the one passed to the tool, the handler sets toolResponse.appSwitchPid and re-traverses the new frontmost app. The .txt file then appends a second 'app_switch:' header followed by the new app's element list (main.swift:1031-1036). The summary includes 'app_switch: <App> (PID: N) is now frontmost'. So the AI narrates 'that click launched Mail, here is its new window'.

buildDiffSummary at main.swift:888-894 returns 'No changes.' when all three arrays are empty, and that string is appended to the one-line summary. So a click that landed on a non-interactive element, or an AXButton that did not change state, produces a response like 'Clicked at (420, 300). No changes.' and the .txt file has the header '# diff: +0 added, -0 removed, ~0 modified' followed by a blank element section. The AI can read that and narrate 'nothing happened, try a different spot'.

The diff describes the post-click world in accessibility terms. The PNG describes where the click physically landed in pixel terms, with the red crosshair showing the exact coordinate. Most of the time you only need the diff. But when an action does nothing, the PNG is the tiebreaker: you can see the crosshair fell on a disabled area, or missed the target, or landed on an overlay you did not know was there. Two formats, two angles on the same event.

Nothing in the server prunes them. /tmp is cleared by macOS on reboot and by periodic launchd tasks (typically anything untouched for 3 days). For a single FaceTime session you will accumulate on the order of tens to low-hundreds of file pairs. If you need to keep them, copy /tmp/macos-use/ somewhere persistent before rebooting. If you want them gone sooner, 'rm -rf /tmp/macos-use/*' between calls is safe — the directory is recreated by main.swift:1823 before the next write.