Agents won't hog your mouse: Vercel Labs open-sources 'agent-desktop' for desktop control

Vercel Labs has released an open-source tool, agent-desktop, that gives AI agents the ability to operate desktop apps (computer use).

The GitHub repository describes it as a command-line tool and C library written in Rust. Instead of capturing the screen it reads each app's UI structure through the operating system's accessibility APIs, returns it as JSON and attaches a ref to every element an agent can act on. Actions run headless by default, without moving the pointer or taking focus; a `--headed` flag allows physical input. It supports macOS, Windows and Linux, is licensed Apache-2.0, has about 6,700 stars and installs with `npm install -g agent-desktop`.

Chinese developer Muyang (@yyyole) introduced it in an X post that drew more than 1,900 likes, saying it reads the accessibility tree without screenshots, runs in the background and lets several agents each use their own cursor at once. The repository does not describe multi-cursor behavior beyond listing a `cursor-overlay` command under sessions.

There are limits. A reply noted that on macOS, accessibility permission must be enabled separately even after a successful install. The docs warn that snapshots, screenshots and clipboard reads can contain private data and should be kept out of public logs. Four commands that hold keys or buttons down refuse to run until a daemon can release them. Another reply asked why people get excited when you could have your bot program its own interface.