Local LLM, macOS backoffice automation: the context budget problem nobody talks about.

Why the model size is not the bottleneck

People assume that running a backoffice agent locally needs a 70B model. It does not. A clean Llama 3.1 8B Instruct on Ollama, given an MCP server with a sane response shape, will close out a vendor invoice triage in twelve to fifteen tool calls without losing the plot. A 14B model is comfortable. A 30B is overkill for almost every backoffice task that is not legal review.

The bottleneck is the context budget. A 7B model loaded with something like Llama 3.1 has, depending on quantisation and how you launch the runtime, somewhere between 8K and 32K tokens of context. Once the system prompt is loaded and the MCP tool schemas are parsed in (those are not free), you have maybe 6,000 to 28,000 tokens left to actually do work. A naive computer-use server, trying to be helpful, dumps the full accessibility tree of the app it just touched. Mail with a healthy inbox is around 8,200 elements, roughly 200 KB of JSON, roughly 50,000 tokens. Two tool calls in, you are out of room. The model produces gibberish. You blame the model.

The model was fine. The response was wrong shape.

Same model, two response shapes

Toggle between the two. Both are after one click on Mail's compose button. The naive shape is what most computer-use code on GitHub does today. The macos-use shape is what fits inside a small local model.

The bytes, side by side

The naive payload below is illustrative shape, not literal output from a specific server. The macos-use payload is the literal shape that buildCompactSummary produces at Sources/MCPServer/main.swift:731. The byte count is roughly two orders of magnitude apart.

# What a naive computer-use MCP server hands back to the model.
# This is what kills a 7B local model.
# After ONE click on Mail's compose button:

{
  "tree": [
    { "role": "AXApplication", "text": "Mail", "children": [ ... ] },
    { "role": "AXWindow", "text": "Inbox", "children": [
      { "role": "AXSplitGroup", "children": [
        { "role": "AXScrollArea", "children": [
          { "role": "AXOutline", "children": [ /* 1,200 mailbox rows */ ] }
        ] },
        { "role": "AXScrollArea", "children": [
          { "role": "AXTable", "children": [ /* 8,000 message rows */ ] }
        ] }
      ] }
    ] },
    /* ...hundreds more nodes... */
  ]
}

# Total payload: ~210 KB of JSON, ~52,000 tokens.
# A 7B model with 8K usable context: dead on arrival.
# A 14B with 32K context: alive for two more turns.

# What macos-use returns instead. The bytes a 7B can actually read.
# Same click on Mail's compose button.
# Source: buildCompactSummary at Sources/MCPServer/main.swift:731

status: success
pid: 4821
app: Mail
file: /tmp/macos-use/1746619200000_click_and_traverse.txt
file_size: 2104 bytes (6 elements)
hint: grep -n 'AXButton' /tmp/macos-use/1746619200000_click_and_traverse.txt
screenshot: /tmp/macos-use/1746619200000_mail.png
summary: Clicked element 'New Message'. 4 added, 2 modified.
text_changes:
  'Inbox - 1,247 messages' -> 'New Message'
visible_elements:
  [AXTextField] "To:"        x:280 y:140 w:520 h:24  visible
  [AXTextField] "Subject:"   x:280 y:172 w:520 h:24  visible
  [AXTextArea]  ""           x:280 y:208 w:520 h:380 visible
  [AXButton]    "Send"       x:760 y:108 w:60  h:28  visible
  [AXButton]    "Attach"     x:680 y:108 w:60  h:28  visible
  [AXPopUpButton] "From:"    x:280 y:108 w:200 h:24  visible
  ...

# Total payload: ~640 bytes, ~190 tokens. 270x smaller.
# A 7B model: still has 7,800 tokens to think with.

The interactive-role allowlist that decides which roles get inlined into visible_elements lives at main.swift:937-941. The output directory /tmp/macos-use is set at main.swift:1961. The grep hint is emitted at main.swift:761.

Backoffice tasks worth running locally

Not every workflow benefits. The shape of a good local-only task is small in steps, repetitive, confined to one or two apps, and handles data the user is not eager to send to a vendor. These six are where I have seen real wins.

Why local at all, given the cloud is faster

A cloud model running a backoffice agent is faster per token, has stronger reasoning, and gets new capabilities monthly. Those are real advantages. The argument for local is not that local is better. It is that for one specific class of work, the bytes cannot leave the laptop.

Vendor invoices have counterparty bank details. CRM notes have things a human wrote in confidence. Statement exports from a bank have transaction-level history that is not yours to upload. The policy at most companies handling this kind of data is informal but firm: do not paste into ChatGPT, do not pipe through a third-party agent. A local model, on a Mac, with an MCP server that drives apps natively and writes its scratch state under /tmp/macos-use, is the boring compliant version of computer use. Nothing is sent anywhere.

The other reason is per-call cost. A backoffice automation that takes thirty tool calls to finish, run twenty times a day, on a cloud model with vision turned on, runs into real money over a month. Local is free at the margin once the laptop is paid for.

Cloud computer-use vs local + macos-use

Where the design choices actually differ.

What the agent does to your keyboard while it works

The first time a local agent drives a Mac it is uncanny. The cursor jumps, a window comes forward, text appears in a field, a button highlights and clicks. If you are still typing into another app, that is a problem. CGEvent, which is what macos-use uses for input, posts events into the same global event stream as your keyboard. They mix.

macos-use wraps every disruptive action in an InputGuard. It engages before the action, suppresses the user's keyboard and mouse so the synthetic events go through cleanly, disengages after, and ships a 30-second watchdog that breaks the guard if anything hangs (Sources/MCPServer/InputGuard.swift:24). Pressing Esc at any point cancels the in-flight action and returns control. The watchdog is the part that matters: a stuck agent that locked your keyboard for an hour would be the end of the experiment.

In practice, the lock is short enough to be invisible most of the time. You will know it is on, because there is a small floating overlay that says what the AI is doing.

What the local stack looks like end to end

Eight pieces. Three for the model, three for the agent loop, two for the operator. Nothing here costs money to run.

The setup, condensed

1. Install the runtime. Ollama (brew install ollama) or LM Studio. Pull a 7B+ instruct model, e.g. ollama pull llama3.1:8b-instruct-q4_K_M.
2. Install macos-use. npm install -g mcp-server-macos-use. The postinstall step runs swift build -c release and writes the binary into the npm prefix.
3. Pick an MCP-aware client. Claude Desktop is the easiest to point at a local model via a proxy; Cline is the most ergonomic inside VS Code; Goose is the most opinionated for autonomous loops. All three accept an MCP server in the standard config.
4. Add macos-use to the client config. "macos-use": { "command": "mcp-server-macos-use" }. Restart the client.
5. Grant permissions to the host. System Settings, Privacy and Security. Accessibility for the client app. Screen Recording for the client app. Both prompts fire on first call.
6. Sanity check. Ask the model to open Mail. If you see a tool call go through and a response with a file: line and a screenshot: line, the wiring is right.
7. Tail the scratch dir. ls -lt /tmp/macos-use/ | head. You will see one .txt and one .png per tool call. The .txt is the full accessibility tree on disk. That is what makes the compact summary affordable.

The honest counterargument

There are tasks a local 7B will fumble that a frontier cloud model handles cleanly. Anything that requires holding a long, fuzzy plan in working memory across a dozen apps. Anything that needs deep reading of a 40-page contract. Anything where the success criteria are ambiguous and the model has to negotiate them with you. For those, the cloud is the right tool, and macos-use works fine against a cloud model too.

What this page is about is the other 80 percent. The form filling, the file moving, the cross-tab reconciliation, the mostly-mechanical work that is too dull for a person and too sensitive for the cloud. That is where local pays back.

Frequently asked