macOS Accessibility Tree Agent: The Three-Rung Fallback Ladder For When The Click Silently Drops

The agent reads the AX tree, finds an AXButton at known coordinates, posts a CGEvent mouse-down/mouse-up at the centered point, and the app does not respond. No exception is raised, no error is returned. The traversal-after looks identical to traversal-before. macos-use makes this easy to spot because every action returns a diff: when the diff is empty (zero added, zero removed, zero modified) on what should have been a state-changing click, the input event tap was bypassed by the target app. This happens predictably on Mac Catalyst right-pane controls, sandboxed apps that opt out of synthetic events, and any app where a secure-input field has captured the event tap.

set_value_and_traverse writes a string into the element under (x,y) via kAXValueAttribute (declared at main.swift:1441). press_ax_and_traverse calls kAXPressAction on the element (main.swift:1458). set_selected_and_traverse sets kAXSelectedAttribute (main.swift:1476). All three skip CGEvent entirely. They get the AXUIElement at the hit-tested point and call AXUIElementSetAttributeValue or AXUIElementPerformAction directly. The input event tap is never engaged, so apps that drop synthetic events still receive the action because the action is happening on the accessibility object, not on the keyboard or mouse hardware path.

Catalyst is UIKit-on-AppKit. The right pane of a Catalyst window often hosts UIKit-rendered text fields and buttons that participate in the AX tree (so the agent can see them) but ignore CGEvent posts targeted at their screen coordinates. The macos-use source names this case explicitly at main.swift:1442: 'Use when typing fails (Catalyst right-pane fields, sandboxed/secure-input contexts).' For those fields, the only thing that lands is AXUIElementSetAttributeValue with kAXValueAttribute. Same for Catalyst right-pane buttons at main.swift:1459: 'Use when a synthetic mouse click is dropped (Catalyst right-pane buttons, sandboxed apps). Often the only path that actuates buttons in those apps.'

When the element exposes the AXSelected attribute but does not implement AXPressAction. The source comment at main.swift:1477 spells this out: 'where regular click is dropped and press_ax errors with kAXErrorActionUnsupported.' Common offenders are Catalyst table rows, sidebar list entries, and outline rows. Pressing them does nothing because there is no press action to perform; selecting them is the operation. set_selected_and_traverse calls AXUIElementSetAttributeValue with kAXSelectedAttribute and the boolean payload. The traversal-after typically shows the row's AXSelected flipping from false to true and the detail pane updating with the new content.

Try click_and_traverse first. It is a single call that posts a CGEvent and chains optional type/press, which is what works for ~95 percent of native AppKit and SwiftUI apps. If the diff comes back empty for a click that should have caused a visible state change, escalate to press_ax_and_traverse with the same x, y, width, height. press_ax errors with kAXErrorActionUnsupported on rows. When that error appears, escalate to set_selected_and_traverse for rows or set_value_and_traverse for text fields. The diff returned by each rung tells the agent immediately whether the rung worked: empty diff means try the next rung.

findAXElementAtPoint at main.swift:1078-1104 walks the AX tree depth-first from the application root. For each element it reads AXPosition and AXSize, builds a CGRect, and checks whether the point falls inside. It returns the deepest match. Fallback tools pass the centered point (x + width/2, y + height/2) when width and height are present, which is why the schema for set_value, press_ax, and set_selected accept optional width/height parameters at main.swift:1432-1435 and 1452-1455. The hit-tested element is the one the AX action is performed on, not whatever was visually under the cursor.

Same diff contract as click. All three escalation tools set options.showDiff = true at main.swift:1745, 1764, and 1784. That triggers a traverseBefore, executes the AX action, traverses again, subtracts, and returns the +/-/~ diff. So the agent sees exactly what changed: the AXValue of a text field flipping from empty to the new string, an AXButton's AXEnabled going from true to false, a row's AXSelected flipping. Same /tmp/macos-use/<ts>_<tool>.txt + .png pair, same flat format, same grep workflow.

Because not every AX element implements every action. AXButton usually does. AXMenuItem usually does. AXTextField and most controls in webviews, Electron apps, and games do not implement AXPressAction. CGEvent works against any window the OS knows about because it is firing real mouse and keyboard events the OS routes through the input event tap. The fallback ladder exists because neither path covers everything alone: CGEvent loses to Catalyst-and-sandboxed; AX actions lose to webviews-and-games. So the server exposes both, and the agent picks based on what the previous rung did.

Secure input is a system-level mode where macOS blocks synthetic keyboard events from reaching the foreground app while a secure field has focus. Apps like Terminal opt into it; banking sites and 1Password trigger it. type_and_traverse will silently fail. set_value_and_traverse at main.swift:1441 is the recovery. It writes into the field via kAXValueAttribute, which is not blocked by secure input because no synthetic key event is fired. Whether the target app then accepts the written value depends on whether the field calls a controller that reads kAXValue, but the input is at least delivered. For a true password field on the macOS lock screen, neither path works, by design.

The agent calls click_and_traverse with element="Save" and reads the diff. If the diff has changes, the click landed and the agent moves on. If the diff is empty, the agent grabs the x, y, width, height for the same element from the prior traversal file and calls press_ax_and_traverse. If press_ax returns kAXErrorActionUnsupported, the agent flips to set_selected_and_traverse for a row or set_value_and_traverse for a field. Each rung writes its own /tmp/macos-use/<ts>_<tool>.txt with diff or error, so the model has a complete trace of which rungs it attempted and which one actuated the UI.