agent-desktop lets AI agents operate desktop apps through the accessibility tree
A Rust CLI and C library from vercel-labs returns a desktop app's accessibility tree as JSON with a ref per element, so an agent can act without screenshots.
What it is
agent-desktop is a command-line tool and C library that gives an AI agent computer use on macOS, Windows and Linux without screenshots or pixel matching. Instead of looking at the screen, it reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on. The agent clicks or types on a ref, reads the UI again to check the result, and repeats.
The README is explicit that the loop lives in the agent: agent-desktop has no model inside. The repository sits under the vercel-labs organisation, is written in Rust and is licensed under Apache-2.0. It was created on 2026-02-19. In the GitHub trending snapshot it had 12389 stars, 3792 of them gained in the last day.
How the snapshot and ref loop works
Installation is through npm, and a status command reports the platform, permissions and supported surfaces:
npm install -g agent-desktop
agent-desktop status # platform, permissions, supported surfaces
On macOS the tool needs the Accessibility grant, and agent-desktop permissions --guide walks through it. On Windows no grant is needed for apps at the terminal’s integrity level. On Linux it has to run in the user’s graphical session with at-spi2-core.
A typical session takes a snapshot of an app’s interactive elements, acts on one ref and takes a new snapshot to verify the result:
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i # observe again to verify
Each command prints one JSON envelope. Refs carry the id of the snapshot they came from, so a bare @e8 fails with INVALID_ARGS. If the UI has changed, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent is expected to take a new snapshot and use its refs. Actions are headless by default, with no pointer movement and no focus change; --headed switches to physical input. The README says a failed command should be retried only when error.disposition.retry is safe.
Commands and platforms
The README counts 60 command names, of which 56 are operational. The other four (key-down, key-up, mouse-down and mouse-up) refuse to run until a daemon can release a held key or button. The commands are grouped into observing (snapshot, find, screenshot and others), acting on a ref (click, type, set-value, scroll and more), raw keyboard and mouse input, apps and windows, notifications, clipboard, waiting, sessions, and system commands such as status, permissions and batch.
Supported platforms and the accessibility API used on each:
- macOS 13 or later, Intel or Apple Silicon: AX API
- Windows 10 1809 or later, x64 or ARM64: UI Automation
- Ubuntu 24.04 or later, x86_64 or aarch64, X11 or Wayland: AT-SPI
Each binary is built for one OS and reaches it only through PlatformAdapter. Besides the CLI, every release ships libagent_desktop_ffi, so a program can call the same core in process.
Notes for agent harnesses
The binary carries guides that match its version, printed with agent-desktop skills get desktop and agent-desktop skills get platform, and the skill can also be installed into an agent harness from skills.sh. The README warns that snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content and should be kept out of public logs and uploads.