--- title: Changelog description: Release history of agent-desktop, from the GitHub releases and CHANGELOG.md. url: "https://agent-browser.dev/desktop/changelog" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Changelog" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Every release of agent-desktop, newest first. The same history is in [CHANGELOG.md](https://github.com/vercel-labs/agent-desktop/blob/main/CHANGELOG.md) and on [GitHub Releases](https://github.com/vercel-labs/agent-desktop/releases). ## [1.0.1](https://github.com/vercel-labs/agent-desktop/compare/v1.0.0...v1.0.1) (2026-10-27) ### Bug Fixes - Remove the `version` line from the skills. In 1.0.0 it still said 0.11.1; `agent-desktop version` reports the version. - Publish npm and GitHub releases automatically when the Cargo workspace version changes on main. - Remove the custom Homebrew tap, the Scoop bucket, and their package-update pull requests. ## [1.0.0](https://github.com/vercel-labs/agent-desktop/compare/v0.11.1...v1.0.0) (2026-10-27) ### Release - Add Linux support: the CLI, the C library, and the npm package now run on x86_64 and aarch64 Linux (glibc 2.35 or later) through AT-SPI. - Release the integrated macOS, Windows, and Linux CLI and C library as 1.0.0. - Bundle native CLI executables and platform helpers in npm installations, including installations with lifecycle scripts disabled. - Verify release archives, checksums, and build provenance through the manual release workflow. ## [0.10.1](https://github.com/vercel-labs/agent-desktop/compare/v0.10.0...v0.10.1) (2026-10-18) ### Bug Fixes - **windows:** let launch take a bare name without .exe ([#276](https://github.com/vercel-labs/agent-desktop/issues/276)) ([483427e](https://github.com/vercel-labs/agent-desktop/commit/483427eb84821a9d0ebc79eb56dd689c04cb15be)) ## [0.10.0](https://github.com/vercel-labs/agent-desktop/compare/v0.9.4...v0.10.0) (2026-10-18) ### ⚠ BREAKING CHANGES - the release carries these contract changes. - The C ABI major version is 5. The legacy entrypoints that each have an \*\_exact replacement are removed: ad_list_surfaces, ad_surface_list_count, ad_surface_list_get, ad_surface_list_free, ad_list_windows, ad_window_list_count, ad_window_list_get, ad_window_list_free, ad_launch_app, ad_release_window_fields, ad_get_tree, ad_find, ad_is, ad_focus_window, ad_window_op, ad_resolve_element and ad_execute_ref_action_with_policy, with the AdSurfaceList and AdWindowList handles. AdScreenshotKind loses WINDOW (FULL_SCREEN keeps 2 and 1 is rejected); AdScreenshotTarget loses pid; AdExactSurfaceInfo is version 2, 48 bytes, with a trailing unclassified field; the snapshot_id argument of ad_execute_by_ref and ad_execute_by_ref_timeout and the AdWaitPredicate.snapshot_id field are removed, so AdWaitPredicate is 40 bytes and AdWaitArgs 104. Consumers must rebuild against the regenerated header. - Bare @eN ref input is removed from every ref-taking CLI command, wait \--element, snapshot --root, find --root, drag --from/--to, batch entries and the FFI. Only the snapshot-qualified @\:eN is accepted, and a bare ref fails with INVALID_ARGS. The --snapshot flag that only qualified a bare ref is removed from every command, and batch entries carrying a snapshot or snapshot_id field are rejected as unknown fields. - get --property text no longer returns the same value as --property value. It prefers the value for textfield, combobox, listbox, datefield and timefield and falls back to the accessible name; every other role prefers the accessible name and falls back to the value. Blank strings count as absent. Ask for --property value explicitly to keep the previous answer. - An io error whose kind is PermissionDenied answers PERM_DENIED instead of INTERNAL, in the CLI envelope and in the FFI envelope, return code and last error alike. - A ref action whose element is gone no longer ends in TIMEOUT. A non-retryable resolution failure answers at once; a retryable one keeps polling, and if no later resolution succeeds the action answers the recorded STALE_REF or AMBIGUOUS_TARGET at the deadline. ### Features - support Windows 10 1809+ and Windows Server 2019+ on x64 and ARM64 ([483a31d](https://github.com/vercel-labs/agent-desktop/commit/483a31d036175f54c34ce1fbd2cea762560ab78c)) ## [0.9.4](https://github.com/vercel-labs/agent-desktop/compare/v0.9.3...v0.9.4) (2026-09-23) ### Bug Fixes - sync cargo lock with the 0.9.3 release ([#217](https://github.com/vercel-labs/agent-desktop/issues/217)) ([8c1502e](https://github.com/vercel-labs/agent-desktop/commit/8c1502e43c6c78fe856980f0bb7b3585f877d347)) ## [0.9.3](https://github.com/vercel-labs/agent-desktop/compare/v0.9.2...v0.9.3) (2026-09-22) ### Bug Fixes - harden native accessibility targeting and delivery ([#210](https://github.com/vercel-labs/agent-desktop/issues/210)) ([882c4c4](https://github.com/vercel-labs/agent-desktop/commit/882c4c4695c0db7d75b83f2655ece178591deb6c)) ## [0.9.2](https://github.com/vercel-labs/agent-desktop/compare/v0.9.1...v0.9.2) (2026-09-17) ### Bug Fixes - make headless desktop actions report what actually happened, and add jev-desktop ([#206](https://github.com/vercel-labs/agent-desktop/issues/206)) ([0691eba](https://github.com/vercel-labs/agent-desktop/commit/0691eba25f6a7082b197bc73ad5daa0e3315c976)) ## [0.9.1](https://github.com/vercel-labs/agent-desktop/compare/v0.9.0...v0.9.1) (2026-09-13) ### Bug Fixes - stop the inspector shutdown test failing on an expected reset ([22cad9e](https://github.com/vercel-labs/agent-desktop/commit/22cad9ec21136562ed191a8cfb827dc22378c554)) - **drag:** scope `--wait-for` to the drop target by default, with `--wait-for-scope from|to` ([cefc80a](https://github.com/vercel-labs/agent-desktop/commit/cefc80a89c95b21061573e366e3832bff6baa231)) - preserve `wait_timeout` envelope when an inherited deadline expires ([07913a7](https://github.com/vercel-labs/agent-desktop/commit/07913a7cdb85d79e1708063fd398c9f8571894e9)) - prevent `wait --count` false positives on incomplete snapshots ([319e9c6](https://github.com/vercel-labs/agent-desktop/commit/319e9c6c2e7c175fdec42d19b2f1614b9238a909)) - split acronym boundaries in trace key redaction ([3042e0c](https://github.com/vercel-labs/agent-desktop/commit/3042e0c7a823508fd5417584f0459d428832c8a0)) - **cursor:** gate `agent_id` stamping on multi-agent mode ([58612ea](https://github.com/vercel-labs/agent-desktop/commit/58612eac673a45634e00a651872da043297e68d3)) - skip ended-session batch entries before wait baseline pre-capture ([899ce27](https://github.com/vercel-labs/agent-desktop/commit/899ce2702a68716a7704b7403050dd7f54161208)) - restore cursor overlay lifecycle for keyboard and clipboard actions ([c827cf9](https://github.com/vercel-labs/agent-desktop/commit/c827cf96560629265ea34451a1a66fe839adab8f)) - clear remembered cursor landing when the overlay is hidden ([6adff7a](https://github.com/vercel-labs/agent-desktop/commit/6adff7af0f50344924f8e8874be72ded1b72a8b3)) - reject percent-encoded NUL in `file_url_to_path` ([de5df38](https://github.com/vercel-labs/agent-desktop/commit/de5df38ede3d45ed7e4d438e732724972351dbe4)) ## [0.9.0](https://github.com/vercel-labs/agent-desktop/compare/v0.8.5...v0.9.0) (2026-09-12) ### ⚠ BREAKING CHANGES - the response envelope moves to 2.4. Stateful ref actions can now return ACTION_FAILED for an observed postcondition contradiction, and ActionResult::from_execution takes only action and steps. ### Features - verify stateful ref actions against fresh platform state ([9bc91bc](https://github.com/vercel-labs/agent-desktop/commit/9bc91bc0ab53139d85e5c10a0ae0b3c3b4bc18b9)) ### Bug Fixes - guard the debug viewer image sink and unpin a flaky test deadline ([fca24f2](https://github.com/vercel-labs/agent-desktop/commit/fca24f2b0259bac83ff17a490052e1f2a9a9496f)) - report unread expand state as unverified rather than failed ([a7131c2](https://github.com/vercel-labs/agent-desktop/commit/a7131c241cce06687b15c53386fd7787d8fbfd72)) ## [0.8.5](https://github.com/vercel-labs/agent-desktop/compare/v0.8.4...v0.8.5) (2026-09-06) ### Features - combine multi-agent cursors with core and macOS optimizations ([#171](https://github.com/vercel-labs/agent-desktop/issues/171)) ([922bb7d](https://github.com/vercel-labs/agent-desktop/commit/922bb7d5b59f2cc8cfaad9896c81fb8fed545bec)) ## [0.8.4](https://github.com/vercel-labs/agent-desktop/compare/v0.8.3...v0.8.4) (2026-08-28) ### Features - rebuild the agent cursor overlay and fix seen-but-unactionable elements ([#145](https://github.com/vercel-labs/agent-desktop/issues/145)) ([5ec2504](https://github.com/vercel-labs/agent-desktop/commit/5ec2504bec0e9face5a69ef25c58bb1273772f2f)) ## [0.8.3](https://github.com/vercel-labs/agent-desktop/compare/v0.8.2...v0.8.3) (2026-08-22) ### Features - harden native automation and add cursor overlay ([#137](https://github.com/vercel-labs/agent-desktop/issues/137)) ([25a0087](https://github.com/vercel-labs/agent-desktop/commit/25a00877726d324c0ee64d84c8d989a5808a66d6)) ## [0.8.2](https://github.com/vercel-labs/agent-desktop/compare/v0.8.1...v0.8.2) (2026-08-20) ### Features - relocate the state root with AGENT_DESKTOP_HOME ([#135](https://github.com/vercel-labs/agent-desktop/issues/135)) ([a336b01](https://github.com/vercel-labs/agent-desktop/commit/a336b01e728893f379918b40157ab25f1c41fa80)) ## [0.8.1](https://github.com/vercel-labs/agent-desktop/compare/v0.8.0...v0.8.1) (2026-08-14) ### Features - drive Chromium apps through a verified DevTools endpoint from launch --cdp ([#129](https://github.com/vercel-labs/agent-desktop/issues/129)) ([366d348](https://github.com/vercel-labs/agent-desktop/commit/366d34803992d0595f31b19c8347bf7b26f5f277)) ## [0.8.0](https://github.com/vercel-labs/agent-desktop/compare/v0.7.0...v0.8.0) (2026-08-13) ### ⚠ BREAKING CHANGES - the response envelope is version 2.3. launch returns \{ app, pid, process_instance, window? } instead of a bare window object, and an application that presents no window is ok:true with window omitted rather than WINDOW_NOT_FOUND. The C ABI is unchanged: it still writes one window and reports WINDOW_NOT_FOUND when there is none. ### Features - decide macOS delivery by observation and stop launch waiting on an uncaused event ([#125](https://github.com/vercel-labs/agent-desktop/issues/125)) ([298f1ff](https://github.com/vercel-labs/agent-desktop/commit/298f1ff21530958d765adf3834cdadccd4816a7a)) ## [0.7.0](https://github.com/vercel-labs/agent-desktop/compare/v0.6.0...v0.7.0) (2026-08-02) ### ⚠ BREAKING CHANGES - ENVELOPE_VERSION is now 2.2. `data.complete` is present on every successful snapshot, and a snapshot that exhausts its budget returns `ok: true` with `complete: false` where it previously returned a TIMEOUT error. Callers that branched on TIMEOUT to detect an oversized tree must read `complete` instead. ### Bug Fixes - return observed trees and stop demanding renderer activation from shallow walks ([#117](https://github.com/vercel-labs/agent-desktop/issues/117)) ([32175e4](https://github.com/vercel-labs/agent-desktop/commit/32175e44c553b350c90c311560ac4d341be71632)) ## [0.6.0](https://github.com/vercel-labs/agent-desktop/compare/v0.5.0...v0.6.0) (2026-07-26) ### ⚠ BREAKING CHANGES - private artifacts on Windows are no longer written through ACL-hardened handles. That hardening guarded refmap, trace, and session files on a platform where no command can produce them, and it had never executed. It returns in Phase 2.1, built on Windows against a CI lane that runs it, under the constraints recorded in `docs/solutions/best-practices/never-ship-platform-code-that-ci-cannot-execute.md`. ### Refactoring - remove speculative Win32 private-file layer from core, add real Windows/Linux test lanes ([#106](https://github.com/vercel-labs/agent-desktop/issues/106)) ([8ad66b8](https://github.com/vercel-labs/agent-desktop/commit/8ad66b8f2115704eed56e59e2709c4eddf3cffac)) - deleted 1,062 LOC of Windows-only `unsafe` Win32 file I/O from `agent-desktop-core` and dropped the `windows-sys` dependency; Windows now uses the same portable `std::fs` path as every other non-unix target - this is a structural fix, not a downgrade: `std::fs::OpenOptions` defaults `share_mode` to `FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE`, precisely the flag the deleted code omitted and the cause of its sharing-violation failures - `cargo test` now runs on windows-latest and ubuntu-latest, so every platform-conditional branch in core is executed on each PR instead of only compiled ### Bug Fixes - the new platform lanes exposed four pre-existing defects that were unreachable while only macOS ran tests: - trace files were opened append-only while `trace.rs` locks that handle; Windows `LockFileEx` rejects append-only handles, so all trace writing failed with `Access is denied` - `refs_lock` tests placed lock files directly in `/tmp`, which is root-owned and world-writable, so the private-parent check correctly refused them - a trace test interpolated a raw path into a JSON string literal, so Windows paths produced invalid escapes and the event was silently dropped instead of skipped - the default `SystemOps::permission_report` returned `Denied`, so CLI preflight reported `PERM_DENIED` on Windows and Linux when the adapter is simply unimplemented; it now reports `Unknown` and surfaces the honest `PLATFORM_NOT_SUPPORTED`, including over the FFI - the pinned toolchain gained `clippy` and `rustfmt`, which `profile = "minimal"` had omitted ## [0.5.0](https://github.com/vercel-labs/agent-desktop/compare/v0.4.7...v0.5.0) (2026-07-20) ### ⚠ BREAKING CHANGES - default-on auto-wait changes the timing of every previously-untouched ref-action call (bounded 5000 ms default; `--timeout-ms 0` restores single-shot). `ENVELOPE_VERSION` is now `2.1` (adds the `APP_UNRESPONSIVE` code and process state in error details). FFI ABI major is `3` (append-only struct evolution; `wait --event` is intentionally not exposed over FFI). The legacy string clipboard API is removed in favor of typed content. `key-down`/`key-up` fail closed until daemon-owned held input exists. `close-app` verifies termination and the osascript fallback path is removed. `--text` matching is subtree containment: `find --text X --first` returns the outermost matching container. ### Features - implement Playwright-grade foundation contract ([3f32272](https://github.com/vercel-labs/agent-desktop/commit/3f322728b44548d2e22f4ee6ef4e6853af4e4550)) - default-on auto-wait: every ref action waits for actionability (visible, enabled, stable, unoccluded) before dispatching, under a bounded budget - live `find` locator plus a serializable `LocatorQuery`, and honest `is --property visible` - three-way `hit_test` occlusion gate and core `scroll_into_view` before element actions - `list-displays` and honest `--screen` with per-display scale factor; `native_id` identity spine; window-id-first resolution - `ProcessState` classification with the `APP_UNRESPONSIVE` code (envelope `2.1`); typed `ActionStep` delivery tier - `LaunchOptions` (`--arg`/`--env`/`--cwd`/`--no-attach`); baseline-diff desktop signals via `wait --event` - typed clipboard (`Text`/`Image`/`FileUrls`); mouse modifier chords and a `mouse-wheel` primitive - capability-supertrait `PlatformAdapter` split with `not_supported()` defaults, so Windows and Linux inherit the contract ## [0.4.7](https://github.com/vercel-labs/agent-desktop/compare/v0.4.6...v0.4.7) (2026-07-02) ### Features - add trace viewer and replay artifacts ([e3e1872](https://github.com/vercel-labs/agent-desktop/commit/e3e1872ff3088f03f3e50e65ebc9e2b068a17b5f)) ## [0.4.6](https://github.com/vercel-labs/agent-desktop/compare/v0.4.5...v0.4.6) (2026-07-02) ### Features - make sessions the first-class trace container ([35fa914](https://github.com/vercel-labs/agent-desktop/commit/35fa914b52bb0dffb0e630f9d9085789c3f81542)) ## [0.4.5](https://github.com/vercel-labs/agent-desktop/compare/v0.4.4...v0.4.5) (2026-06-30) ### Features - add --wait-for selector polling flags ([#86](https://github.com/vercel-labs/agent-desktop/issues/86)) ([ce23278](https://github.com/vercel-labs/agent-desktop/commit/ce232787b50270d6f590e11bd2b16c7354de623d)) ## [0.4.4](https://github.com/vercel-labs/agent-desktop/compare/v0.4.3...v0.4.4) (2026-06-29) ### Features - **macos,core:** harden adapter and core foundation with caller-controllable guardrails ([#82](https://github.com/vercel-labs/agent-desktop/issues/82)) ([94ce6c5](https://github.com/vercel-labs/agent-desktop/commit/94ce6c551fff4ffd2afc44ab6a9f655b4486da11)) ## [0.4.3](https://github.com/vercel-labs/agent-desktop/compare/v0.4.2...v0.4.3) (2026-06-28) ### Bug Fixes - **macos:** harden retained_handle null guard against release-only CFRetain(null) ([#80](https://github.com/vercel-labs/agent-desktop/issues/80)) ([a708fa0](https://github.com/vercel-labs/agent-desktop/commit/a708fa03326a03dbc82acc7e41bd3ec262a05248)) ## [0.4.2](https://github.com/vercel-labs/agent-desktop/compare/v0.4.1...v0.4.2) (2026-06-27) ### Features - **ffi:** Phase B and C: Python smoke harness, parity gates, build.rs codegen ([#77](https://github.com/vercel-labs/agent-desktop/issues/77)) ([9023f33](https://github.com/vercel-labs/agent-desktop/commit/9023f331b3981a41414ea0f92e2e5120a212e357)) ## [0.4.1](https://github.com/vercel-labs/agent-desktop/compare/v0.4.0...v0.4.1) (2026-06-26) ### Features - complete FFI C-ABI surface (Phase A): load-time ABI handshake, session-scoped adapter, JSON-envelope command entrypoints (version, status, snapshot, wait, execute-by-ref), an optional tracing log callback, and a unified error-envelope contract ([#67](https://github.com/vercel-labs/agent-desktop/issues/67)) ## [0.4.0](https://github.com/vercel-labs/agent-desktop/compare/v0.3.1...v0.4.0) (2026-06-24) ### ⚠ BREAKING CHANGES - the version command no longer accepts --json; it always emits the standard JSON envelope. ### Refactoring - over-engineering audit cleanup ([#64](https://github.com/vercel-labs/agent-desktop/issues/64)) ([dbb2be6](https://github.com/vercel-labs/agent-desktop/commit/dbb2be639ecc1f979031818259f587f951086b3c)) ## [0.3.1](https://github.com/vercel-labs/agent-desktop/compare/v0.3.0...v0.3.1) (2026-06-21) ### Bug Fixes - harden macos stale ref resolution ([#62](https://github.com/vercel-labs/agent-desktop/issues/62)) ([9f144c2](https://github.com/vercel-labs/agent-desktop/commit/9f144c2cafe3ba0ade479c1b87ecac6cd88adcef)) ## [0.3.0](https://github.com/vercel-labs/agent-desktop/compare/v0.2.3...v0.3.0) (2026-06-20) ### ⚠ BREAKING CHANGES - **ffi:** `AD_POLICY_KIND_PHYSICAL` is now `AD_POLICY_KIND_HEADED` (discriminant `2` unchanged). - **ffi:** `AdRefEntry` and `AdDragParams` include additional reliability metadata; C ABI consumers must zero-initialize and validate with `AD_REF_ENTRY_SIZE`, `AD_DRAG_PARAMS_SIZE`, and the `ad_*_size()` accessors. - `close-app` graceful responses now return `{ "method": "graceful", "requested": true }` instead of claiming `closed: true` before the app has exited. ### Features - add Playwright-grade ref reliability with strict late resolution, session-scoped snapshots, deterministic stale/ambiguous target handling, and skeleton drill-down preservation ([#54](https://github.com/vercel-labs/agent-desktop/pull/54)). - add Playwright-style headed/headless interaction policy: accessibility-first dispatch remains default, while `--headed` enables focused physical fallbacks only when needed ([#54](https://github.com/vercel-labs/agent-desktop/pull/54)). - add retrying wait predicates, JSONL traces with secret redaction, richer actionability reports, and shared CLI/FFI execution through the same resolver/actionability/dispatch ladder ([#54](https://github.com/vercel-labs/agent-desktop/pull/54)). ### Bug Fixes - harden macOS ref actions across focus/window raising, disclosure expansion, scroll, drag, numeric set-value, menu detection, stale-ref recovery, protected-process errors, and close-app confirmation ([#54](https://github.com/vercel-labs/agent-desktop/pull/54)). - fail closed on unsafe fallback ambiguity and stale refs, including symlink-safe refstore/latest-snapshot reads and deadline-bound resolver work ([#54](https://github.com/vercel-labs/agent-desktop/pull/54)). ### Performance - reduce repeated live actionability reads and add deadline-bound resolver behavior, with large-app snapshot and per-command CLI wall-clock gates in the E2E suite ([#54](https://github.com/vercel-labs/agent-desktop/pull/54)). ### Tests - add real-app E2E coverage for headless/headed ref actions, sessions, snapshots, traces, waits, skeleton drill-down, menus, surfaces, drag, expand/collapse, and performance checks ([#54](https://github.com/vercel-labs/agent-desktop/pull/54)). ## [0.2.3](https://github.com/vercel-labs/agent-desktop/compare/v0.2.2...v0.2.3) (2026-06-06) ### Bug Fixes - harden macos ax window fallback ([3b266fd](https://github.com/vercel-labs/agent-desktop/commit/3b266fdf040bf83438f69a400b032fd12b8715c6)) - resolve fullscreen AX tree retrieval returning ref_count: 0 ([a52b7c7](https://github.com/vercel-labs/agent-desktop/commit/a52b7c704a4d13fdc0d6b72f4080bb0fc118be64)) ## [0.2.2](https://github.com/vercel-labs/agent-desktop/compare/v0.2.1...v0.2.2) (2026-06-02) ### Bug Fixes - **macos:** guard CFArray casts with type-ID check (fixes Mail.app crash) ([#50](https://github.com/vercel-labs/agent-desktop/issues/50)) ([c02cb5e](https://github.com/vercel-labs/agent-desktop/commit/c02cb5ecb7314f053d63937437e5f5ba48de3209)) ## [0.2.1](https://github.com/vercel-labs/agent-desktop/compare/v0.2.0...v0.2.1) (2026-05-23) ### Bug Fixes - stabilize empty accessibility identity refs ([1fb5a7d](https://github.com/vercel-labs/agent-desktop/commit/1fb5a7d51eb798100b4d597c755fee1161e298bf)) ## [0.2.0](https://github.com/vercel-labs/agent-desktop/compare/v0.1.14...v0.2.0) (2026-05-20) ### ⚠ BREAKING CHANGES - chain execution deadlines now return TIMEOUT instead of ACTION_FAILED when the target app does not respond before the chain deadline. ### Refactoring - unify command execution contracts ([1291a9c](https://github.com/vercel-labs/agent-desktop/commit/1291a9cdbf0566424d38da1eab397d6d4091c06c)) ## [0.1.14](https://github.com/vercel-labs/agent-desktop/compare/v0.1.13...v0.1.14) (2026-05-04) ### Features - bundle skill docs and refactor --help for AI agents ([#36](https://github.com/vercel-labs/agent-desktop/issues/36)) ([b04d6f9](https://github.com/vercel-labs/agent-desktop/commit/b04d6f97317af67648890d2d3b5ead0d27c466c9)) ## [0.1.13](https://github.com/vercel-labs/agent-desktop/compare/v0.1.12...v0.1.13) (2026-04-17) ### Features - **ffi:** ship C-ABI cdylib with review fixes and release pipeline ([#26](https://github.com/vercel-labs/agent-desktop/issues/26)) ([3cffbd6](https://github.com/vercel-labs/agent-desktop/commit/3cffbd67f6b27f42001643bef9fd2530cb7f9003)) ## [0.1.12](https://github.com/vercel-labs/agent-desktop/compare/v0.1.11...v0.1.12) (2026-04-16) ### Features - progressive skeleton traversal with ref-rooted drill-down ([#20](https://github.com/vercel-labs/agent-desktop/issues/20)) ([c17f2fa](https://github.com/vercel-labs/agent-desktop/commit/c17f2fae7abbbe2c914a050fa9e9be5fca9c6af0)) ## [0.1.11](https://github.com/vercel-labs/agent-desktop/compare/v0.1.10...v0.1.11) (2026-03-03) ### Bug Fixes - show skill install prompt on all success paths ([39b2bc6](https://github.com/vercel-labs/agent-desktop/commit/39b2bc63480890f7ed417b2c040eecf80c4628a0)) ## [0.1.10](https://github.com/vercel-labs/agent-desktop/compare/v0.1.9...v0.1.10) (2026-03-03) ### Bug Fixes - fix skill publication authentication in CI ([208af12](https://github.com/vercel-labs/agent-desktop/commit/208af12459fea2255e1c80b8cdc9ac420316d769)) ## [0.1.9](https://github.com/vercel-labs/agent-desktop/compare/v0.1.8...v0.1.9) (2026-03-03) ### Features - scalable skill architecture with publication automation ([#14](https://github.com/vercel-labs/agent-desktop/issues/14)) ([9766520](https://github.com/vercel-labs/agent-desktop/commit/97665203a464e605bc9b156ec90029c5909399be)) ## [0.1.8](https://github.com/vercel-labs/agent-desktop/compare/v0.1.7...v0.1.8) (2026-03-01) ### Features - add electron/web app compatibility for accessibility tree traversal ([a19c1b5](https://github.com/vercel-labs/agent-desktop/commit/a19c1b5132d3b71c5de58886ba51357ffc9bd1e8)) - implement --compact flag to collapse single-child unnamed nodes ([4a300c8](https://github.com/vercel-labs/agent-desktop/commit/4a300c8cb054462ca95fb5160e89e8fce661ec3b)) ## [0.1.7](https://github.com/vercel-labs/agent-desktop/compare/v0.1.6...v0.1.7) (2026-02-28) ### Features - add notification command types, adapter trait, and CLI wiring ([c5b05ba](https://github.com/vercel-labs/agent-desktop/commit/c5b05bab600aafa36c642f21837c44583b36459c)) - add notification management commands (macOS) ([b1fd368](https://github.com/vercel-labs/agent-desktop/commit/b1fd368f195640642adf011b75cca6ecb9e5acc3)) - **macos:** add NC session RAII guard and notification adapter wiring ([0d55c21](https://github.com/vercel-labs/agent-desktop/commit/0d55c21de0f0ea6ea08ccb836e5527c9da513620)) - **macos:** implement dismiss and notification action commands ([53d697d](https://github.com/vercel-labs/agent-desktop/commit/53d697d52248c3fa06797787b1eb549ac2766533)) - **macos:** implement notification list via AX tree traversal ([53549a3](https://github.com/vercel-labs/agent-desktop/commit/53549a384eec67572ee05c31621b8b4174425ab3)) ### Bug Fixes - **macos:** remove AXPress from dismiss action list ([27ef4f3](https://github.com/vercel-labs/agent-desktop/commit/27ef4f34c038c26f5852bf1f6026762a98d0df0a)) - **macos:** restore frontmost app after notification center interaction ([3881bc8](https://github.com/vercel-labs/agent-desktop/commit/3881bc82bdb5a4bb7689f1e1e2237bb745e60c21)) - **macos:** use pgrep and async osascript for NC lifecycle ([9797585](https://github.com/vercel-labs/agent-desktop/commit/979758538fca0145e9245641fb03a0769eed68de)) ## [0.1.6](https://github.com/vercel-labs/agent-desktop/compare/v0.1.5...v0.1.6) (2026-02-24) ### Bug Fixes - handle null bounds in refmap and improve sidebar click resolution ([d4197e8](https://github.com/vercel-labs/agent-desktop/commit/d4197e8f6f6700f2f672d3e1e436ecf24cf82e01)) ## [0.1.5](https://github.com/vercel-labs/agent-desktop/compare/v0.1.4...v0.1.5) (2026-02-23) ### Features - add fallback chains for set-value, clear, focus, scroll-to, type and post-action state hints ([11f8da0](https://github.com/vercel-labs/agent-desktop/commit/11f8da06e84ed67b0e26dbc1946f7a7542e89dcd)) - add structured verbose logging across all layers ([c7316e8](https://github.com/vercel-labs/agent-desktop/commit/c7316e8b5160ab0e6ba554bcd502d4c47adf8b1a)) ### Bug Fixes - add dwell time before drag release for drop target recognition ([2a52d62](https://github.com/vercel-labs/agent-desktop/commit/2a52d62106699b36f62f9af83895f2264b80efb1)) ## [0.1.4](https://github.com/vercel-labs/agent-desktop/compare/v0.1.3...v0.1.4) (2026-02-23) ### Features - add agent-desktop skill for universal AI agent support ([ef45135](https://github.com/vercel-labs/agent-desktop/commit/ef45135087d09a7e065f65d9a0558d1e710cb8bf)) - add Claude Code skills for agent-desktop automation ([ad91cd3](https://github.com/vercel-labs/agent-desktop/commit/ad91cd32cf2de1c1c8dcda4c0dcae37f0022b4c6)) ## [0.1.3](https://github.com/vercel-labs/agent-desktop/compare/v0.1.2...v0.1.3) (2026-02-23) ### Bug Fixes - correct GitHub Release download URL and simplify tag format ([8f66a93](https://github.com/vercel-labs/agent-desktop/commit/8f66a9346e02a751a83ac02313dcec2d9c81bde8)) - include README and CHANGELOG in npm package ([084fc8c](https://github.com/vercel-labs/agent-desktop/commit/084fc8c960527c0d0654028794ec3c4fd2d970c4)) ### Performance - use curl for binary download in postinstall ([ebafb71](https://github.com/vercel-labs/agent-desktop/commit/ebafb71603f5f2b32af8ac5bf6c88df3d6012f70)) ## [0.1.2](https://github.com/vercel-labs/agent-desktop/compare/agent-desktop-v0.1.1...agent-desktop-v0.1.2) (2026-02-23) ### Bug Fixes - use macos-latest for both build targets ([91c7677](https://github.com/vercel-labs/agent-desktop/commit/91c76777cb7ee864b45e14d123c79c08f0c2d5b9)) ## [0.1.1](https://github.com/vercel-labs/agent-desktop/compare/agent-desktop-v0.1.0...agent-desktop-v0.1.1) (2026-02-23) ### Features - 10-step scroll chain, focus guards, enhanced click chain, bounds fix ([595ccb6](https://github.com/vercel-labs/agent-desktop/commit/595ccb6cc45554351ea3e30b95e4ca47bdf4e16b)) - add 19 new commands, AX-first rewrites, LOC compliance ([d3f7e03](https://github.com/vercel-labs/agent-desktop/commit/d3f7e03c67832c652a6125f61fbb7ab2f0801939)) - add 19 new commands, AX-first rewrites, LOC compliance ([eca04e8](https://github.com/vercel-labs/agent-desktop/commit/eca04e839288b121f6f41c6de525a8396d10654c)) - add release automation with GitHub Releases and npm distribution ([18fc50c](https://github.com/vercel-labs/agent-desktop/commit/18fc50cca51f2ed10b6dfb5576602b6ce344bc95)) - add structural hints to splitter columns in snapshots ([48f8470](https://github.com/vercel-labs/agent-desktop/commit/48f8470948b4f636dfa6f4489e4cb6d9f520722c)) - AX-first right-click chain with inline context menu capture ([cddc5d3](https://github.com/vercel-labs/agent-desktop/commit/cddc5d3547f058a78f8b398fa982e39a1fcbf6b1)) - Phase 1 foundation: workspace scaffold, core engine, macOS adapter, 31 commands ([a346f24](https://github.com/vercel-labs/agent-desktop/commit/a346f242c25dfad1c849e6d50f9ab25a42b462d9)) - smart AX-first click chain + macOS crate restructure ([4616c8f](https://github.com/vercel-labs/agent-desktop/commit/4616c8f65f974505b0eedb5485c865d3b905342b)) - surface-targeted snapshot, menu wait, list-surfaces command ([39178b2](https://github.com/vercel-labs/agent-desktop/commit/39178b291602d192de97aa0150c261db1dcc7ca6)) ### Bug Fixes - add menubar surface, fix press --app crash and modifier mapping ([a231962](https://github.com/vercel-labs/agent-desktop/commit/a2319623b4d1d2b6b2f6e1a4ab9a8b8cbbfd02eb)) - address code review findings (double-free, CF leaks, injection) ([2f495ff](https://github.com/vercel-labs/agent-desktop/commit/2f495ffb69be67f3136b076534e078cc31b005c2)) - align error codes with spec (APP_NOT_FOUND, PERM_DENIED) and add -i shorthand ([6dc567a](https://github.com/vercel-labs/agent-desktop/commit/6dc567a4aedff15cf82a82601089cb0b87da4e26)) - ancestor-path cycle detection + CGEvent click fallback ([198d7d7](https://github.com/vercel-labs/agent-desktop/commit/198d7d7d27167044a448b6616fa5c9c0554321bf)) - detect open menus via AXMenuBarItem.AXSelected, not AXMenus attribute ([7f0d610](https://github.com/vercel-labs/agent-desktop/commit/7f0d6103d16969a0abfa84a62b6819dbd0d1cc8e)) - make all 30 commands work end-to-end on macOS ([1d98ab8](https://github.com/vercel-labs/agent-desktop/commit/1d98ab828ce5bcb39e212548ae2f2a052e67aac9)) - remove AXShowDefaultUI from activation chain, fix child walk ([74242f5](https://github.com/vercel-labs/agent-desktop/commit/74242f5040af9c46c98a3f5232dc7567538c28e1)) - resolve all 47 code review findings from Phase 1 audit ([218503a](https://github.com/vercel-labs/agent-desktop/commit/218503a7ebacacd4fbc6b388a6cf5e3bb86af039)) - right-click uses AXShowMenu; context menus detected via focused element ([2c9aee3](https://github.com/vercel-labs/agent-desktop/commit/2c9aee397912d6a903d9ef1e26c786697383ae95)) - suppress dead_code lint on BatchCommand deserializer struct ([608d4aa](https://github.com/vercel-labs/agent-desktop/commit/608d4aaaa195b95626f17aa4bbca2d69609f14cc)) - use simple release strategy for workspace version bumps ([0ab78dd](https://github.com/vercel-labs/agent-desktop/commit/0ab78dde0e1ff702db6c8b667784fa456245b26b)) --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Configuration description: Global flags, post-action wait flags, environment variables, and the state root where agent-desktop stores snapshots, sessions, and traces. url: "https://agent-browser.dev/desktop/configuration" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Configuration" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop has no configuration file. Behavior comes from global flags on each command, a few environment variables, and the state root directory where snapshots, sessions, and traces live. ## Global flags Global flags work before or after the command name, so `agent-desktop --headed click REF` and `agent-desktop click REF --headed` are the same. `agent-desktop --help` lists them.
Flag Default Meaning
--headed off Prefer physical delivery for natural input commands and allow focus and cursor side effects. The default is strict headless semantic delivery. See Headed mode.
--session ID none Select the snapshot namespace and trace. Overrides AGENT_DESKTOP_SESSION. See Sessions.
--agent-id ID none Subagent id within the session. Overrides AGENT_DESKTOP_AGENT_ID. Required for desktop actions in a multi-agent session.
--trace PATH off Append trace events as JSONL to this file instead of the session's trace. See Traces.
--trace-strict off Fail when trace setup or a pre-action write fails. Requires --trace or a tracing session.
--wait-for SELECTOR, -w none After the command, poll until an element matches. See Post-action wait flags.
--wait-for-gone SELECTOR none After the command, poll until no element matches.
--wait-timeout MS 30000 Longest wait for the two flags above.
--verbose, -v off Write debug logs to stderr. Stdout stays one JSON envelope.
--debug, --screenshot PATH.html off Write a visual debug HTML file for snapshot or click. Each requires the other. See Traces.
--help, -h; --version, -V Print help or the version as text. The version command prints it as JSON.
## Post-action wait flags `--wait-for`, `--wait-for-gone`, and `--wait-timeout` make a command wait for the UI to show its effect. The command runs first; then the flag polls the accessibility tree until the selector is satisfied or the timeout ends. [Waiting without sleeps](/desktop/guides/waiting) compares them with the [`wait`](/desktop/commands/wait) command. **Selector.** `role:text`, split at the first `:`. `button` matches by role, `:Saved` by text, and `button:Save` by both. Text is a case-insensitive substring of an element's name, value, or description. An empty selector returns `INVALID_ARGS` before the command runs. **Commands that accept them.** `snapshot`, `press`, `hover`, `drag`, and every ref action (`click`, `double-click`, `triple-click`, `right-click`, `type`, `set-value`, `clear`, `focus`, `select`, `toggle`, `check`, `uncheck`, `expand`, `collapse`, `scroll`, `scroll-to`). Every other command returns `INVALID_ARGS`. `snapshot --root` cannot combine with them, and a `batch` entry never inherits them. **Which owner is polled.** For `snapshot`, the selected surface owner. For a ref action, the original process. Application-owned refs keep their source surface; a macOS `menu` or `menubar` ref is observed through its application without requiring a document window. Window-owned refs observe their original pinned window, even when captured from `sheet`, `alert`, `popover`, `focused`, or a window-owned `menu` such as a Linux menu. The wait reads the whole window, so a selector elsewhere in the parent can satisfy `--wait-for` early or keep `--wait-for-gone` from succeeding. On Linux, GTK3 context menus and Chromium popups can be exposed outside that window tree, so a gone wait cannot establish that a popup outside the observed scope was dismissed. To observe the overlay itself, run a separate `snapshot` with the desired `--surface` and wait selector. For `press`, the `--app` or `--window-id` window, else the focused one. For `drag`, the destination window, or the source with `--wait-for-scope from`. **Results.** - `--wait-for` succeeds with snapshot data (`snapshot_id`, `ref_count`, `tree`, and so on) plus `elapsed_ms` and `matched_selector`. For an action, the action's own report moves under `data.after_action`. - `--wait-for-gone` succeeds with `matched_selector`, `gone: true`, `target_absent: true`, and `elapsed_ms`, with the action's report under `after_action`. - On expiry the command fails with `TIMEOUT`, `details.kind: "wait_timeout"`, `details.selector`, and `details.timeout_ms`. After an action, `details.after_action` holds its report, and the error keeps the action's own delivery disposition, so a timeout does not make the action safe to repeat. - If a ref action returns `delivery_uncertain`, `error.details.post_action_observation` records read-only evidence from the original process and its application-owned source surface or pinned window. Its [fields and outcomes](/desktop/json-output#post-action-observation) do not change the original error or unsafe retry disposition. The wait uses `--wait-timeout` separately from the action timeout. - A temporary `ACTION_NOT_SUPPORTED` error with `details.kind: "window_without_accessibility_element"` is retried during wait polling. For `--wait-for-gone`, that error does not count as absence; the wait succeeds only after a complete selector read finds no match or the app/window lookup confirms the target is gone. ```bash agent-desktop click @s8f3k2p9:e5 --wait-for ":Saved" --wait-timeout 5000 agent-desktop click @s8f3k2p9:e5 --wait-for-gone "progressindicator" agent-desktop snapshot --app gnome-calculator -w "button:7" ``` ## Environment variables
Variable Meaning
AGENT_DESKTOP_HOME Relocates the state root. Must be an absolute path.
AGENT_DESKTOP_SESSION Session id used when --session is not passed. session start prints the line that sets it. An empty value is INVALID_ARGS.
AGENT_DESKTOP_AGENT_ID Subagent id used when --agent-id is not passed.
AGENT_DESKTOP_MACOS_HELPER_PATH macOS only. Absolute path to the packaged agent-desktop-macos-helper. The npm launcher sets it.
RUST_LOG Log filter for stderr logs. A valid value replaces the default level.
An invalid value returns `INVALID_ARGS` with `not_delivered`, and nothing runs. In PowerShell, set a variable with `$env:AGENT_DESKTOP_HOME = "$env:TEMP\ad-scratch"`. On Linux the adapter also reads the desktop session's `DISPLAY`, `WAYLAND_DISPLAY`, `XDG_RUNTIME_DIR`, and `DBUS_SESSION_BUS_ADDRESS`; see [Linux](/desktop/platforms/linux). ## State root agent-desktop keeps its state under one directory: `.agent-desktop` in your home directory by default. `agent-desktop status` reports the resolved path as `state_root`. `AGENT_DESKTOP_HOME` replaces the default: - It must be an absolute path. If it exists, it must be a directory you own and not a symlink. - It does not need to exist yet; it is created on first use with mode `0700` on macOS and Linux. On macOS and Linux the state root must stay private. A root that is group-writable or readable by other users makes commands that write there fail with `INTERNAL`, "private file parent is accessible by group or other users". Run `chmod 700` on it, or let agent-desktop create it. A fresh root per run isolates refs: ```bash export AGENT_DESKTOP_HOME="$(mktemp -d)" # absolute and private agent-desktop status # data.state_root shows this path ``` ### Layout ```text ~/.agent-desktop/ state root latest_snapshot_id newest snapshot in the global namespace snapshots/ s8f3k2p9/refmap.json one refmap per snapshot tmp/ clipboard-get images outside a session sessions/ run-1793352720537-17009-0/ one directory per session session.json manifest latest_snapshot_id snapshots/s8f3k2p9/refmap.json the session's own namespace trace/ JSONL segments, one per process clipboard/ clipboard-get images inside the session ``` - **Snapshots.** Each snapshot writes `snapshots//refmap.json`. Size and retention limits are in [Where refs are stored](/desktop/snapshots-and-refs#where-refs-are-stored). - **Sessions.** `session start` creates `sessions//` with its own `snapshots/`, so one session's refs are invisible to another. `session gc` deletes ended sessions with their traces. See [Sessions](/desktop/sessions). - **Traces.** A tracing session writes one JSONL segment per process under `trace/`, plus `screens/` and `refmaps/` with `--screenshots`. See [Traces](/desktop/guides/traces). The cross-process interaction lock lives outside the state root (under `/tmp` on macOS and Linux), so every agent of one user shares it whatever their `AGENT_DESKTOP_HOME`. Files you name yourself, such as a `--trace` path or a screenshot path, are also outside it. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Errors and recovery description: The error object, all 16 error codes, delivery and retry dispositions, recovery strategies, and when it is safe to retry. url: "https://agent-browser.dev/desktop/errors-and-recovery" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Errors and recovery" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). A failed command prints an envelope with `ok: false` and exits `1` (or `2` for a parse error). Branch on `error.code` and `error.disposition`, never on `message` or `suggestion`, whose wording can change. Before you retry or substitute an action, read `error.disposition.retry`. ```json { "version": "2.6", "ok": false, "command": "click", "error": { "code": "SNAPSHOT_NOT_FOUND", "message": "Snapshot 's3tufs86hddhyl' not found", "suggestion": "Re-run a snapshot and retry with a ref from the returned snapshot (CLI: snapshot, then use the printed @:eN ref; ...)", "recovery": { "strategy": "refresh_snapshot_then_retry_original", "retryable": true, "requires_fresh_snapshot": true }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` ## The error object
Field Present Meaning
code always One of the 16 codes below.
message always Human-readable text. Not a stable interface.
disposition always What the failure says about delivery and retry. See Disposition.
suggestion when set A hint for a person or an agent. Check any command it names before you run it.
recovery only when retry is safe A declared recovery plan. See Recovery strategies.
platform_detail when set The native platform's own error text.
details when set Structured context. details.kind names the cause for many codes, for example actionability_timeout, focus_verification, surface_absent, wait_timeout, session_ended, or post_action_verification.
## Error codes
Code What it means What to do
PERM_DENIED A required operating system permission is missing, such as Accessibility or Screen Recording. Run agent-desktop permissions and grant it. See the macOS, Windows, and Linux pages.
ELEMENT_NOT_FOUND No element matched the ref or query, or a requested surface is absent. Read details.kind and details.surface. For an absent menu, open the menu first; otherwise check the UI before refreshing a ref.
APP_NOT_FOUND No matching application process exists. Check the name with list-apps. Run launch if it is not running.
ACTION_FAILED The action was rejected or blocked, or its observed result contradicted the request. Read details and disposition.retry. It never carries recovery.
ACTION_NOT_SUPPORTED The target, or this platform for this input, does not expose the requested action. Use a different command or element. Repeating it fails the same way.
STALE_REF The ref no longer matches a live element. Take a fresh snapshot and use the new ref. See Snapshots and refs.
AMBIGUOUS_TARGET Several live elements, windows, or app instances match, or the ref was not unique when it was captured, and the command refuses to guess. Take a fresh snapshot. When the matches are different elements, narrow the target with find, --root, or --window-id; off macOS, identical twins have no ref path. It never carries recovery.
WINDOW_NOT_FOUND The requested window does not exist, including a --window-id that matches no window or a closed shell surface. Run list-windows --app \ and use an id it reports.
PLATFORM_NOT_SUPPORTED This platform's adapter does not implement the method. A surface kind this OS does not have is INVALID_ARGS. Use another approach. The platform pages list what each OS supports.
TIMEOUT A wait expired, the target never became actionable, or a read stayed incomplete until the deadline. Read details.kind. Retry only when disposition.retry is safe.
INVALID_ARGS An argument is invalid. Also returned for a bare @eN ref and an ended session. Fix the input. Repeating the same arguments fails the same way.
NOTIFICATION_NOT_FOUND The notification index or identity no longer matches. Run list-notifications again.
SNAPSHOT_NOT_FOUND The snapshot id is missing, pruned, or belongs to another session namespace. Take a snapshot in the same session scope you act in, then use its refs.
POLICY_DENIED Headless mode blocked a physical or headed path. Do not switch modes to recover. See Headed mode.
APP_UNRESPONSIVE The application stopped responding. Wait, take a fresh snapshot, then retry only if disposition.retry is safe.
INTERNAL An unexpected error. Read message. Retry once only if disposition.retry is safe.
For ELEMENT_NOT_FOUND, read details.kind, details.surface, and details.complete. A macOS lookup with kind: "surface_absent" and complete: true has confirmed that the requested surface is absent. An absent menubar can mean the app exposes no menu bar. For surface: "menu", open the application menu first: snapshot menubar when it exists, click its menu-bar item without --headed, then snapshot --surface menu. Missing complete, complete: false, a timeout, or an incomplete read is unknown. If the command followed an action with disposition.retry: unsafe, inspect the current UI before another action; do not repeat the action because the snapshot failed. A macOS TIMEOUT with details.kind: "focus_verification" means the app never reported AXFrontmost, or the target window never reported AXMain, before the bounded focus readback ended. details.physical_delivery_started: false means target input had not started; it does not mean a prior focus or window-raise request was never dispatched. Read disposition, which can still be delivered_unverified. ## Disposition `disposition` says whether input reached the application, and from that, whether repeating the command is safe. Every error carries it. Commands that act on a target also carry it on success, as `data.disposition`.
delivery retry Meaning
not_delivered safe The input never reached the application. Repeating the command cannot duplicate an effect.
delivery_uncertain unsafe The input may have landed, and nothing proves either way.
delivered_unverified unsafe The input was dispatched, and its effect was not confirmed.
delivered_verified unsafe The input was dispatched and the effect was read back. Repeating it would apply it twice.
unknown unknown The command established nothing about delivery, for example a TIMEOUT on an operation that may already have landed.
`retry` follows from `delivery`: only `not_delivered` is `safe`, `unknown` stays `unknown`, and every other value is `unsafe`. A refusal that cannot have delivered anything is stamped `not_delivered`: an argument error, a failure before dispatch, and any error from a read-only command such as `snapshot` or `find`. **Retry or substitute an action only when `disposition.retry` is `safe`.** When it is `unsafe` or `unknown`, observe first: take a snapshot, read the element with `get` or `is`, or check the app, and decide from what you see. A repeated `click` can submit twice, and a repeated `toggle` undoes itself. ## Recovery strategies `recovery` appears only when `disposition.retry` is `safe`, the code has a declared plan, and the adapter has not marked the failure non-retryable with `details.retryable: false`. It has `strategy`, `retryable`, `requires_fresh_snapshot`, and, for two codes, `retry_after_ms`.
strategy Codes What the caller does
refresh_snapshot_then_retry_original STALE_REF, SNAPSHOT_NOT_FOUND Take a fresh snapshot (or find), locate the same element, and run the original command again with the new qualified ref.
inspect_state_then_retry_original TIMEOUT, APP_UNRESPONSIVE, and STALE_REF when details.process_state is exited Inspect the app first (list-apps, a snapshot, or a wait), wait retry_after_ms if present, then run the original command. For an exited process, launch the app again and take a new snapshot first.
request_explicit_policy_then_retry_original POLICY_DENIED Retry only after someone has explicitly chosen the policy the command needs, such as --headed when physical input is intended. Never switch to headed mode on your own to get past the error.
`STALE_REF` uses `inspect_state_then_retry_original` only when `details.process_state` is `exited`, and `refresh_snapshot_then_retry_original` otherwise. Every other code, such as `ACTION_FAILED`, `INVALID_ARGS`, or `AMBIGUOUS_TARGET`, never carries `recovery`. Follow the "What to do" column instead. `retry_after_ms` is set only for `TIMEOUT` and `APP_UNRESPONSIVE` (250 ms in this release). `details.retryable` is the adapter's own judgment: `false` means waiting cannot change the outcome, so try a different approach, and it suppresses `recovery` even for a code that would otherwise carry one. ## Decide whether to retry 1. Read `error.code`. For `INVALID_ARGS`, `ACTION_NOT_SUPPORTED`, `PLATFORM_NOT_SUPPORTED`, or `PERM_DENIED`, change the input or the environment. A retry will not help. 2. Read `error.disposition.retry`. If it is `unsafe` or `unknown`, observe the application before you act again. 3. If it is `safe` and `recovery` is present, follow the strategy. 4. If it is `safe` and `recovery` is absent, fix the cause from the code table, then repeat the command, for example with a new ref from `find`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: agent-desktop description: A CLI and C library that lets AI agents read and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree. url: "https://agent-browser.dev/desktop" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Introduction" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop reads the accessibility tree of a desktop app and returns it as JSON. Every element an agent can act on gets a ref. The agent acts on a ref, reads the UI again to check the result, and repeats. It ships as a CLI and as a C library for macOS, Windows, and Linux. ```bash npm install -g agent-desktop # macOS, Windows, and Linux agent-desktop status # platform, permissions, supported surfaces agent-desktop skills get desktop # the guide your agent should load first ``` ## What agent-desktop is and is not - **A tool that agents call.** It is not an agent. The observe, act, and verify loop lives in the calling agent or script. - **No model inside.** It never embeds an LLM, has no GUI, and has no interactive prompt. Every command prints one JSON envelope and exits. - **Desktop apps only.** It does not automate web pages; use [agent-browser](https://agent-browser.dev) for those. For the web contents of a Chromium or Electron app you launch, agent-desktop opens a DevTools port that agent-browser connects to. See [Chromium and Electron apps](/desktop/guides/chromium-and-electron). - **Limited to what apps expose.** It reads accessibility trees and does not match pixels, so it cannot see a UI that publishes no accessibility tree. ## The loop in four commands An agent maps an app, drills into the region it needs, acts on a ref, and checks the result. ```bash agent-desktop snapshot --app gnome-calculator --skeleton -i --compact # 1. map: a shallow skeleton agent-desktop snapshot --root @s2txcl95uvubmn:e1 -i --compact # 2. drill: expand one region agent-desktop click @s2txcl95uvubmn:e11 # 3. act: press the 7 button agent-desktop wait --element @s2txcl95uvubmn:e6 --predicate value --value 7 # 4. verify: the display reads 7 ``` Refs carry the id of the snapshot that produced them (`@s2txcl95uvubmn:e11`). [Snapshots and refs](/desktop/snapshots-and-refs) defines them, and [Skeleton and drill-down](/desktop/skeleton-and-drill-down) explains the first two commands. A successful action does not prove the UI changed, so the loop ends with a read. The [quick start](/desktop/quick-start) runs these commands with their output. ## How it works The same chain in text, from the caller to the app: 1. The calling agent: your code or an LLM loop. It runs a command and reads the JSON envelope it prints. 2. The `agent-desktop` CLI, or the [C library](/desktop/ffi) when you call it in process. 3. `agent-desktop-core`: the commands, `RefStore`, and snapshot building. Core defines the `PlatformAdapter` trait and every shared type. 4. `PlatformAdapter`, the only way core reaches an operating system. 5. One platform crate: `agent-desktop-macos`, `agent-desktop-windows`, or `agent-desktop-linux`. 6. The OS accessibility API: the AX API on macOS, UI Automation on Windows, AT-SPI on Linux. 7. The app, which exposes its UI through that API. Core never imports a platform crate, and platform crates never import each other. Only the CLI and the C library connect a platform crate to core, and each picks it at compile time, so a binary drives the OS it was built for. ## Platform support
Platform Requirement Accessibility API
macOS macOS 13 or later, Intel or Apple Silicon. Accessibility permission for the app that runs agent-desktop. AX API
Windows Windows 10 1809 or later, or Windows Server 2019 or later, on x64 or ARM64, in the signed-in user's interactive desktop. UI Automation
Linux Ubuntu 24.04 or later on x86_64 or aarch64 (glibc 2.35 floor), in the user's graphical session with a running AT-SPI bus. AT-SPI
The CLI has the same 60 command names on every platform. Four of them (`key-down`, `key-up`, `mouse-down`, `mouse-up`) fail closed everywhere. Each platform page lists what works headless, what needs `--headed`, and what refuses. ## Next steps - [Installation](/desktop/installation): install the CLI, verify it, and grant permissions. - [Quick start](/desktop/quick-start): one complete run against a real app. - [Skills](/desktop/skills): the guides your agent reads before it acts. - [Commands](/desktop/commands): all 60 commands. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Installation description: Install agent-desktop with npm, pnpm, Yarn, Bun, or Homebrew, from a release archive, or from source, and verify the install on macOS, Windows, or Linux. url: "https://agent-browser.dev/desktop/installation" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Installation" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Install the `agent-desktop` npm package globally with npm, pnpm, Yarn, or Bun. It needs Node 18 or later and runs on macOS, Windows, and Linux. ```bash npm install -g agent-desktop # or: pnpm add -g, yarn global add, bun add -g agent-desktop version # confirm the command resolves ``` Without `-g`, the command lands in `./node_modules/.bin` instead of on your PATH. Run it with `npx agent-desktop`, or install globally. ## What the package contains The package carries native CLI executables for six targets, plus the helper each platform needs:
Platform Targets Helper installed beside the CLI
macOS arm64, x64 agent-desktop-macos-helper
Linux (glibc) x64, arm64 agent-desktop-linux-clipboard-owner
Windows x64, arm64 none
The [C library](/desktop/ffi) is not in the npm package. It ships as a separate archive on each release. ## Release archives For a native executable without a JavaScript launcher, download the archive for your OS and architecture from [GitHub Releases](https://github.com/vercel-labs/agent-desktop/releases). Extract the executable and its helper together into a directory on your PATH. To upgrade, replace both with files from the same release. Checksums catch corruption. Verify origin with the build attestation: ```bash gh attestation verify --repo vercel-labs/agent-desktop ``` ## Blocked install scripts The package's `postinstall` script replaces the Node launcher with a link to the native binary. If your package manager blocks install scripts, the Node launcher runs instead. It works, but every call starts Node first, which adds more than 100 ms. The launcher prints a one-time notice to stderr with the fix. Allow the script and reinstall: ```bash npm install -g agent-desktop --allow-scripts=agent-desktop ``` ## Homebrew On macOS and Linux, install from Homebrew's default catalog. Homebrew builds from the tagged source and installs the platform helper beside the CLI. ```bash brew install agent-desktop # macOS and Linux brew upgrade agent-desktop # new versions arrive shortly after each GitHub release ``` ## Scoop and older installs There is no Scoop package. The custom Homebrew tap and Scoop bucket were removed in 1.0.1 (see the [changelog](/desktop/changelog)). An older tap or bucket install stays pinned to its version and can shadow the npm or Homebrew command. Check which executable runs, and uninstall the old one with the package manager that installed it: ```bash command -v agent-desktop # macOS and Linux; in PowerShell: Get-Command agent-desktop ``` ## Build from source You need the Rust toolchain pinned in `rust-toolchain.toml` (1.89.0), which `rustup` installs on first use, and a C toolchain: the Xcode command-line tools on macOS, the MSVC build tools on Windows, or `build-essential` on Linux. ```bash git clone https://github.com/vercel-labs/agent-desktop && cd agent-desktop cargo build --release -p agent-desktop # writes target/release/agent-desktop ``` Build the helper for your platform and keep it beside the CLI. A platform crate compiles only on its own OS: ```bash cargo build --release -p agent-desktop-macos --bin agent-desktop-macos-helper # macOS cargo build --release -p agent-desktop-linux --bins # Linux ``` ## Verify the install ```bash agent-desktop version agent-desktop status ``` `version` prints the CLI version, the architecture, and the OS: ```json { "version": "2.6", "ok": true, "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" } } ``` `status` reports the platform, the permission state, the surfaces this OS can read, and the [state root](/desktop/configuration#state-root): ```json { "version": "2.6", "ok": true, "command": "status", "data": { "permissions": { "accessibility": { "state": "granted" }, "automation": { "state": "not_required" }, "screen_recording": { "state": "not_required" } }, "platform": "linux", "ref_count": null, "session_id": null, "snapshot_id": null, "state_root": "/home/user/.agent-desktop", "supported_surfaces": ["window", "sheet", "alert", "menu"], "tracing": false, "version": "1.0.1" } } ``` ## Grant permissions - **macOS:** grant Accessibility permission with `agent-desktop permissions --guide`. See [macOS](/desktop/platforms/macos). - **Windows:** no grant is needed for apps at your terminal's integrity level. See [Windows](/desktop/platforms/windows). - **Linux:** run inside the user's graphical session with a running AT-SPI bus, then check `agent-desktop status`. An empty `supported_surfaces` means the bus did not answer. See [Linux](/desktop/platforms/linux). Next, run the [quick start](/desktop/quick-start). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: JSON output description: The response envelope every agent-desktop command prints, its omission rules, the OperationReport shape, find results, exit codes, and the contract versions from 2.1 to 2.6. url: "https://agent-browser.dev/desktop/json-output" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "JSON output" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Every command prints exactly one JSON envelope on a single line of standard output. Only `--help`, `--version`, and a bare `agent-desktop` print text instead. The contract version is `2.6`. Log output goes to standard error, so a parser can read standard output as one JSON value. ## Envelope ```json { "version": "2.6", "ok": true, "command": "click", "data": { "action": "click", "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" } } } ```
Field Type Meaning
version string Contract version. Currently "2.6".
ok boolean true when the command returned a result, false when it returned an error.
command string The command name, such as click or session.
data object The result. Present only when ok is true.
error object The failure. Present only when ok is false. Its fields are in Errors and recovery.
An error envelope has `error` and no `data`: ```json { "version": "2.6", "ok": false, "command": "click", "error": { "code": "INVALID_ARGS", "message": "Invalid ref_id '@e1': expected @:e", "suggestion": "Use the snapshot-qualified ref that snapshot prints, for example @s8f3k2p9:e1. A bare @eN ref is not accepted.", "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` `ok: true` means the command returned a delivered or already-satisfied result. It does not mean the application effect happened. Read the result, or observe again, to verify. See [the observe, act, verify loop](/desktop/guides/observe-act-verify). ## Omission rules - A typed field that is `None` is omitted, and so is an empty array. An error without a suggestion has no `suggestion` key, and a node without children has no `children` key. - Snapshot node bounds are omitted unless you pass `--include-bounds`. - Some commands build their `data` by hand and write `null` or `[]` for an absent value. `session list` writes `"ended_at": null` for an open session, and `find` matches write `"value": null` and `"states": []`. Treat a missing key, `null`, and an empty array the same way. - Ignore keys you do not know. New optional fields can appear without a version change. - Snapshot `data.window` appears when a window owns the observation. On macOS, application-owned `menu` and `menubar` snapshots omit it. ## Application-owned snapshots On macOS, `menu` and `menubar` snapshots belong to the application, so `data.app` identifies the owner and `data.window` is absent. This remains true when a `--window-id` selects which app to observe or when `--root` drills into a ref from that surface. Window-owned snapshots keep `data.window`. ## Where data lives `ref_count` and `tree` sit inside `data`, never at the top level. This window-owned snapshot of GNOME Calculator is trimmed to one branch: ```json { "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "gnome-calculator", "complete": true, "ref_count": 7, "snapshot_id": "s1vylk49fulgx4", "window": { "id": "w-calculator", "title": "Calculator" }, "tree": { "role": "window", "name": "Calculator", "children": [ { "available_actions": ["Click", "SetFocus"], "name": "Undo", "native_id": { "kind": "ax_identifier", "value": "undo_button" }, "ref_id": "@s1vylk49fulgx4:e3", "role": "button", "states": ["disabled"] } ] } } } ``` A snapshot that runs out of budget still returns `ok: true`, with `"complete": false`, `"truncated": true`, `nodes_observed`, and `"subtree_truncated": true` on each cut node. Read `data.complete` before you treat a tree as whole. Node fields are in [Snapshots and refs](/desktop/snapshots-and-refs), and partial observations in [Skeleton and drill-down](/desktop/skeleton-and-drill-down#partial-observations). ## Action results and disposition A ref action such as `click`, `type`, or `toggle` returns `data.action` and `data.disposition`, and can add `steps` (each mechanism tried), `post_state`, `surfaces` (surfaces open once the action settled), and `details`. `disposition` has two fields, `delivery` and `retry`; on success, `delivery` is `delivered_verified`, `delivered_unverified`, or `not_delivered` (already satisfied). What each value proves, and when a retry is safe, is in [Errors and recovery](/desktop/errors-and-recovery#disposition). ### Post-action observation When a ref action returns `delivery_uncertain` with a post-action wait selector, `error.details.post_action_observation` records a read-only observation of the original process. Application-owned refs keep their source surface. Window-owned refs observe their original pinned window, including refs captured from `sheet`, `alert`, `popover`, `focused`, or a window-owned `menu` such as a Linux menu. This lets a closed overlay be observed without treating the parent window as absent. The action is never repeated, and its original error and unsafe retry disposition stay unchanged. Even `expected_state_observed` does not prove that the action caused the state. Observation uses `--wait-timeout` separately from the action timeout.
FieldMeaning
kind Always post_action_observation.
selector, expected Requested selector and present or absent.
surface The observed scope: the application-owned source surface, or window for a window-owned ref.
process_identity Original pid and process_instance; the instance can be null for identity_unavailable.
elapsed_ms Time spent observing after the action returned.
source_window_id Original window ID, when the ref has one.
selector_matches Observed selector result, when available; absence requires a complete read.
process_state Last observed unresponsive, exited, or crashed state. running is never emitted.
last_observation_error Last observation error's code and message, when available.
outcome One of the outcomes below.
OutcomeMeaning
expected_state_observed The expected selector state was observed, or the original window or surface was confirmed absent for an absence wait.
process_exited The original process exited; this does not establish the action's effect.
process_crashed The platform reported that the original process crashed.
owner_unavailable The original surface owner remained unavailable without confirmed process exit.
observation_failed A terminal read error stopped observation; inspect last_observation_error.
timed_out The wait budget expired without observing the expected state, including when transient read errors persisted through the last poll. last_observation_error may hold the last error.
identity_unavailable The ref has no usable process generation, so observation was skipped.
## OperationReport Thirteen commands return the adapter's own report as `data`: `mouse-click`, `mouse-move`, `mouse-wheel`, `hover`, `drag`, `clipboard-set`, `clipboard-clear`, `close-app`, `move-window`, `resize-window`, `minimize`, `maximize`, and `restore`. The report is the action result fields (`action`, `disposition`, and `steps`, `post_state`, `surfaces`, or `details` when present) plus a `target`. The command then adds the echoed request fields, such as `from` and `to` for `drag`, without overwriting any field the report already has. ```json { "action": "drag", "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [ { "label": "pointer_drag", "outcome": "succeeded", "mechanism": "physical_synthetic" } ], "target": { "kind": "drag", "from": { "x": 10, "y": 20 }, "to": { "x": 80, "y": 20 } }, "from": { "x": 10, "y": 20 }, "to": { "x": 80, "y": 20 } } ```
target.kind Fields
window id, title, process
app name, process
pointer point
drag from, to
clipboard none
The adapter fills in the identity it validated and used. A pointer target is the coordinate it dispatched to and does not prove which application received the input. Request fields such as `x`, `y`, `count`, `width`, or `app` echo your request; they are not observed state. Window reports name their readback in `details.verification_scope` (`window_size`, `window_position`, `window_minimized_state`, or `window_maximized_state`). There are no `clicked`, `moved`, `closed`, or similar flags, because a flag that every successful call carried proved nothing. A failed `--wait-for` after a delivered operation keeps the operation's own delivery in `error.disposition` and carries the report under `error.details.after_action`. ## Find results `find` returns one of three shapes, depending on the result-shaping flag. The flags are in [`find`](/desktop/commands/observation#find).
Flag data
none, or --limit N matches, total_matches, truncated
--first, --last, --nth N match (an object, or null when nothing matched)
--count count
```json { "version": "2.6", "ok": true, "command": "find", "data": { "match": { "interactive": true, "name": "7", "path": ["window:Calculator", "group", "group", "group"], "ref_id": "@s3jjre8xzrvo2l:e1", "role": "button", "states": [], "value": null }, "snapshot_id": "s3jjre8xzrvo2l" } } ``` Each match carries `ref_id`, `role`, `name`, `value`, `states`, `interactive`, and `path` (trimmed above). A match that earns a ref has a qualified `ref_id`, and the response then includes the `snapshot_id` that holds it. A match with no ref is a context match: its `ref_id` is `null` and it carries `bounds` when the platform reports them, so you can see where it is. An element without an accessible name omits `name`, and its content stays in `value`. When a query with `--role` matches nothing, `roles_present` lists the roles the search did see. ## Exit codes
Code Meaning
0 Success, and also help or version display.
1 A structured error. The envelope is on standard output with ok: false.
2 A command-line parse error. The envelope is still printed, with INVALID_ARGS.
```bash agent-desktop click @e1 # bare ref: INVALID_ARGS envelope echo $? # 1 agent-desktop click # missing argument: INVALID_ARGS envelope echo $? # 2 ``` An argument the parser accepts but the command rejects, such as a bare `@e1`, exits `1`. Exit `2` is only for arguments the parser cannot read. ## Contract versions Check `version` before you parse. Apply the migrations below when your parser supports an older contract version.
Version What changed
2.1 The error.retry_command string is gone, with no alias. Errors carry a structured error.recovery object and an error.disposition. Select a recovery strategy only when disposition.retry is safe, and never run command strings from older envelopes.
2.2 A snapshot that exhausts its budget returns ok: true with complete: false, the observed tree, truncated, nodes_observed, and subtree_truncated on cut nodes, instead of a TIMEOUT error. Snapshot data always carries complete. A --root drill-down still returns TIMEOUT for an incomplete observation.
2.3 launch returns app, pid, process_instance, and an optional window instead of a bare window object. An application that shows no window is ok: true with window omitted, not WINDOW_NOT_FOUND.
2.4 Stateful ref actions (set-value, type, clear, check, uncheck, toggle, expand, collapse) read the state back after one delivery attempt and reject an observed contradiction. A readback failure keeps the adapter's code and adds details.kind: "post_action_verification". Missing evidence stays a successful delivered_unverified with details.verification_scope: "unavailable".
2.5 Breaking for the thirteen OperationReport commands. The success flags clicked, moved, scrolled, hovered, dragged, resized, minimized, maximized, restored, closed, cleared, requested (close-app), and the inner ok of clipboard-set are removed. data is the adapter's report with disposition, steps, and target.
2.6 On Windows and Linux, and on macOS for an element without a native address, a ref resolves only when a complete window read proved its role and stable fields unique at capture and exactly one live element matches now. An identity that was not unique at capture fails with details.kind: "identity_not_unique"; a twin that appears later fails with details.kind: "saved_ref_ambiguous". A snapshot that could not prove identity can include data.identity_unproven. On macOS, application-owned menu and menubar snapshots omit data.window; data.app identifies the owner, and --window-id selects that app.
The C library returns the same envelope from its JSON functions. See the [C library overview](/desktop/ffi). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Quick start description: Run the observe, act, verify loop against a real desktop app. Check the install, map the app, drill into a region, click a ref, and verify the result. url: "https://agent-browser.dev/desktop/quick-start" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Quick start" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). This run drives GNOME Calculator on Linux: check the install, map the app, drill into one region, press a button, confirm the display changed, and put the app back. Every output below is real, trimmed where noted. Start GNOME Calculator in your graphical session, and run the commands from a terminal in the same session. On macOS or Windows, use an app name that `list-apps` prints (for example `"Calculator"`) and expect different refs. In PowerShell, quote every ref (`'@s2txcl95uvubmn:e1'`); see [Windows](/desktop/platforms/windows#use-powershell). ## 1. Check the install ```bash agent-desktop status # permissions, supported_surfaces, state_root agent-desktop skills get platform # the guide for this OS, in data.content ``` `data.permissions.accessibility.state` should be `granted`, and `data.supported_surfaces` should not be empty. Agents load the platform skill before their first action; see [Skills](/desktop/skills). ## 2. Map the app A skeleton snapshot reads three levels and marks each cut branch with `children_count`: ```bash agent-desktop snapshot --app gnome-calculator --skeleton -i --compact ``` ```json { "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "gnome-calculator", "complete": true, "ref_count": 2, "snapshot_id": "s2txcl95uvubmn", "tree": { "role": "window", "name": "Calculator", "children": [ { "children_count": 2, "ref_id": "@s2txcl95uvubmn:e1", "role": "group" }, { "children_count": 1, "ref_id": "@s2txcl95uvubmn:e2", "role": "group" } ] } } } ``` The tree is trimmed: wrapper groups and `native_id` fields are removed. `e1` is the calculator body and `e2` is the header bar. Every ref carries its snapshot id; pass it to later commands exactly as printed. ## 3. Drill into a region ```bash agent-desktop snapshot --root @s2txcl95uvubmn:e1 -i --compact ``` The drill reads `e1`'s subtree and adds 31 refs (`e3` to `e33`) to the same snapshot. Two of them: ```json [ { "ref_id": "@s2txcl95uvubmn:e6", "role": "textfield", "available_actions": ["RightClick", "SetValue", "TypeText", "SetFocus"] }, { "ref_id": "@s2txcl95uvubmn:e11", "role": "button", "name": "7", "available_actions": ["Click", "SetFocus"] } ] ``` `e6` is the display and `e11` is the 7 button. How drill targets and re-drills work is in [Skeleton and drill-down](/desktop/skeleton-and-drill-down). ## 4. Act on a ref ```bash agent-desktop click @s2txcl95uvubmn:e11 ``` ```json { "version": "2.6", "ok": true, "command": "click", "data": { "action": "click", "details": { "auto_wait": { "lease_contention_count": 0, "lease_hold_ms": 114, "read_only_preflight_attempts": 1, "read_only_resolve_attempts": 1 } }, "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "Action.DoAction", "mechanism": "semantic_api", "outcome": "succeeded" }], "surfaces": [ { "id": "w-1.11-org-gnome-Calculator-a11y-948fd89a_9bda_453b_9c10_4a9e9eed3feb", "title": "Calculator", "type": "window" } ] } } ``` The click went through the accessibility API (`semantic_api`), without moving the cursor. `delivered_unverified` means it was dispatched but nothing confirmed the effect yet, and `retry: unsafe` means a repeat could press 7 twice. See [JSON output](/desktop/json-output#action-results-and-disposition). ## 5. Verify The app updates its display asynchronously, so wait for the value instead of reading once or sleeping: ```bash agent-desktop wait --element @s2txcl95uvubmn:e6 --predicate value --value 7 --timeout 5000 ``` ```json { "version": "2.6", "ok": true, "command": "wait", "data": { "elapsed_ms": 286, "found": true, "observed": { "expected_chars": 1, "matched": true, "value_chars": 1, "value_present": true }, "predicate": "value", "ref": "@s2txcl95uvubmn:e6" } } ``` `get @s2txcl95uvubmn:e6 --property value` now returns `"7"`. A read taken immediately after the click can still show the old value, which is why the verify step waits. See [Waiting without sleeps](/desktop/guides/waiting). ## 6. Restore the state Press **C** (`e8` in the same drill) so the app is as you found it: ```bash agent-desktop click @s2txcl95uvubmn:e8 ``` ## Next steps - [The observe, act, verify loop](/desktop/guides/observe-act-verify): the loop as practice. - [Errors and recovery](/desktop/errors-and-recovery): what to do when a ref is stale or ambiguous. - [Commands](/desktop/commands): every command and flag. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Sessions description: A session is an on-disk namespace for snapshots and traces. How to start, select, and end one, how agent ids work, and how sessions relate to traces and the cursor overlay. url: "https://agent-browser.dev/desktop/sessions" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Sessions" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). A session is a directory under the state root that owns a set of snapshot refmaps and, when tracing is on, a trace. You use one when several runs, or several agents, share a machine and each needs its own snapshots, or when you want a trace of everything a run did. Without a session, every command uses one global namespace. ## What a session owns `session start` creates `sessions//` under the [state root](/desktop/configuration#layout), with a `session.json` manifest, its own `snapshots/` namespace, and, when tracing is on, a `trace/` directory. The session id has the form `run---`. An id you pass yourself with `--session` must be 1 to 64 letters, digits, `-`, or `_`. ## Start, list, end, and clean up ```bash agent-desktop session start --name docs-example # create the session and print its id agent-desktop session list # manifests, oldest first agent-desktop session end # seal the manifest agent-desktop session gc --ended # delete ended sessions nothing is using ``` `session start` returns the id and the line that activates it: ```json { "version": "2.6", "ok": true, "command": "session", "data": { "activation": { "environment": "AGENT_DESKTOP_SESSION", "value": "run-1793352720537-17009-0" }, "artifacts": "events", "created_at": 1793352720537, "name": "docs-example", "next": "export AGENT_DESKTOP_SESSION=run-1793352720537-17009-0", "session_id": "run-1793352720537-17009-0", "trace": "on" } } ``` `session list` returns `data.sessions`, one object per manifest with `session_id`, `name`, `created_at`, `ended_at` (`null` while the session is open), `trace` and `artifacts`. `session end` returns `session_id` and `ended_at`, and you can run it again on a session that already ended. `session gc` returns `data.removed`, the ids it deleted. Ending a session seals the manifest and stops its trace. After that, every other command that resolves to the ended session fails with `INVALID_ARGS` and `details.kind: "session_ended"`, except `version` and `skills`: ```json { "version": "2.6", "ok": false, "command": "snapshot", "error": { "code": "INVALID_ARGS", "message": "Session 'run-1793352720537-17009-0' has ended", "suggestion": "Unset AGENT_DESKTOP_SESSION and omit --session, then run the command or start a new session", "details": { "kind": "session_ended", "session_id": "run-1793352720537-17009-0" }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` `session gc` deletes ended sessions that nothing is using; its retention rules are in [`session`](/desktop/commands/system#session). ## Select a session A command picks its session in this order:
Order Source Notes
1 --session \ Wins over the environment variable.
2 AGENT_DESKTOP_SESSION An empty value is INVALID_ARGS.
3 none The global namespace, stored directly in the state root.
An id that is not 1 to 64 letters, digits, `-` or `_` fails with `INVALID_ARGS`. A `--session ` with no manifest still works as a snapshot namespace, but it writes no trace. `session start` does not activate the session for later commands. Each command is its own process, so pass the id to every one, or export it once in your shell: ```bash export AGENT_DESKTOP_SESSION=$(agent-desktop session start --name docs-example | jq -r '.data.session_id') agent-desktop snapshot --app gnome-calculator -i # runs inside the session agent-desktop session end "$AGENT_DESKTOP_SESSION" ``` `session end` with no id ends the session selected by `--session` or `AGENT_DESKTOP_SESSION`. On Windows, `data.next` prints the PowerShell form (`$env:AGENT_DESKTOP_SESSION = '...'`); see [Windows](/desktop/platforms/windows) for quoting rules. ## Snapshot namespaces A session is also a snapshot namespace. Snapshots taken in a session are saved under that session's directory, and every lookup is confined to the selected namespace. A qualified ref such as `@s8f3k2p9:e1` embeds its snapshot id, but the id is not a cross-session handle. Acting on that ref with a different `--session` (or none) fails with `SNAPSHOT_NOT_FOUND`. Take the snapshot and act on its refs under the same session scope. Several agents may share one session id. Each should act on refs from its own snapshot, because the session keeps one latest-snapshot pointer. Retention and pruning of refmaps are described in [Snapshots and refs](/desktop/snapshots-and-refs). ## Agent identity `--agent-id ` (or `AGENT_DESKTOP_AGENT_ID`) names the harness subagent that is running a command. The flag wins over the variable. An id is 1 to 64 letters, digits, `-` or `_`, and an invalid id fails with `INVALID_ARGS`. It is scoped by session. The identity matters in one mode: a session started with `session start --cursor --multi-agent`. There, desktop actions such as `click` and `type` require an agent id and fail with `INVALID_ARGS` without one. Observation, clipboard and session commands do not. Each distinct id gets its own cursor, and a reused id reuses its cursor. A headless action moves its cursor before it takes the interaction lease, so several agents' cursors travel at the same time while their actions take turns. Commands that draw the same cursor take turns from its travel to its click effect, so a shared cursor never clicks while showing another command's target. Snapshots stay shared by session. ## Sessions, traces, and the cursor overlay A new session records a trace by default: the manifest has `trace: "on"` and commands append segments under `trace/` until the session ends. `--no-trace` creates a namespace with no trace. `--screenshots` also stores pre-action and post-action screenshots and refmap copies, which can contain private screen content. Reading and exporting traces is covered in [Traces](/desktop/guides/traces). The agent cursor overlay is configured once per session, either with `session start --cursor` or with `cursor-overlay enable`, and action commands inherit it. Presentation settings, per-agent profiles and platform support are in [`cursor-overlay`](/desktop/commands/system#cursor-overlay). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Skeleton and drill-down description: Read a dense app in two passes. Take a shallow skeleton, drill into one region by ref, re-drill after a change, and scope searches the same way. url: "https://agent-browser.dev/desktop/skeleton-and-drill-down" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Skeleton and drill-down" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). A dense app, such as an Electron client or an IDE, can expose thousands of accessibility nodes. A full snapshot of it is slow, costs many tokens, and can run out of budget before it finishes. A skeleton snapshot reads only the top levels. You then drill into the one region you need with `snapshot --root`, so the refs you read and act on cover that region instead of the whole window. Only macOS makes the drill read less of the window. Off macOS a drill re-identifies its root over a complete read of the window, as [Drill into a region](#drill-into-a-region) explains. ## The loop 1. `snapshot --skeleton -i --compact` reads the window to depth 3. Cut containers carry `children_count`, and drill targets carry a `ref_id`. 2. Pick a node that has both `children_count` and `ref_id`. 3. `snapshot --root -i --compact` reads that region and adds its refs to the same snapshot. 4. Act with a ref from the drill, such as `click` or `set-value`. See the [interaction commands](/desktop/commands/interaction). 5. Verify with `wait` or `get` on a ref from the drill, or re-read the region with the same `snapshot --root`. If the action result lists a menu, sheet, or alert in `data.surfaces`, read that [surface](/desktop/surfaces) next. 6. If another region matters, drill it. If the layout changed, or the root ref itself is `STALE_REF`, take a new skeleton. `-i` keeps only interactive nodes, so a row's static text is dropped. To tell which row a repeated button belongs to, drill that row's container with `--root` and leave out `-i`, then match the button to the label beside it. Two shortcuts skip the loop: a small app needs only `snapshot -i`, and a known target needs only `find`. See [When to skip the skeleton](#when-to-skip-the-skeleton). ## Take a skeleton ```bash agent-desktop snapshot --app gnome-calculator --skeleton -i --compact ``` ```json { "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "gnome-calculator", "complete": true, "ref_count": 2, "snapshot_id": "s2txcl95uvubmn", "window": { "id": "w-1.11-org-gnome-Calculator-a11y-948fd89a_9bda_453b_9c10_4a9e9eed3feb", "title": "Calculator" }, "tree": { "role": "window", "name": "Calculator", "native_id": { "kind": "ax_identifier", "value": "MathWindow" }, "children": [ { "children_count": 2, "native_id": { "kind": "ax_identifier", "value": "___object_3___" }, "ref_id": "@s2txcl95uvubmn:e1", "role": "group" }, { "children_count": 1, "native_id": { "kind": "ax_identifier", "value": "headerbar" }, "ref_id": "@s2txcl95uvubmn:e2", "role": "group" } ] } } } ``` The tree above omits the unnamed wrapper groups between the window and the two drill targets. The calculator's main area is `e1`, and its header bar is `e2`. `--skeleton` changes two things: - **Depth is clamped to 3.** The effective depth is the smaller of `--max-depth` and 3, so a bare `--skeleton` reads three levels. Depth counts semantic levels: GTK and web wrappers can make the output nest deeper than three JSON levels. - **Cut containers carry `children_count`.** A container at the depth boundary lists no children. It reports how many direct children it did not expand. Interactive elements inside the first three levels get ordinary refs. `children_count` appears on any node a depth limit cuts, not only under `--skeleton`, and with `-i` agent-desktop keeps a cut container in the output even when it has no ref. ## Find the drill target A drill target is a node with both a `ref_id` and a `children_count`. Which node carries it depends on how the cut container can be recognized later: 1. **The container is ref-able.** An interactive role or a primary action gives it a normal ref, and that ref is the drill target. 2. **The container is not ref-able but has identity.** If it has a name, a description, a typed native id, or bounds with an area, it gets a drill ref of its own, as `e1` and `e2` do above. Its available actions are not stored, so use it to drill, not as an action target. 3. **The container has no usable identity.** The target moves up to the nearest ancestor that qualifies under rule 1 or 2. One ancestor ref covers every unidentified cut container beneath it. The window root is never a target. 4. **Several candidates sit on one path.** The deepest one wins. In practice, drill a node that has both `children_count` and `ref_id`. A cut container with `children_count` and no `ref_id` is covered by an ancestor that has one. If no ancestor qualifies, raise `--max-depth` or use `find` to reach what is under it. ## Drill into a region ```bash agent-desktop snapshot --root @s2txcl95uvubmn:e1 -i --compact ``` ```json { "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "gnome-calculator", "complete": true, "ref_count": 33, "snapshot_id": "s2txcl95uvubmn", "tree": { "role": "group", "native_id": { "kind": "ax_identifier", "value": "___object_3___" }, "children": [ { "available_actions": ["RightClick", "SetValue", "TypeText", "SetFocus"], "native_id": { "kind": "ax_identifier", "value": "source_view" }, "ref_id": "@s2txcl95uvubmn:e6", "role": "textfield" }, { "available_actions": ["Click", "SetFocus"], "description": "Clear Display [Escape]", "name": "C", "native_id": { "kind": "ax_identifier", "value": "calc_clear_button" }, "ref_id": "@s2txcl95uvubmn:e8", "role": "button" }, { "available_actions": ["Click", "SetFocus"], "name": "7", "native_id": { "kind": "ax_identifier", "value": "calc_7_button" }, "ref_id": "@s2txcl95uvubmn:e11", "role": "button" } ] } } } ``` The tree above is trimmed to the display and two of the 31 buttons and fields the drill found (`e3` to `e33`). `--root` takes a qualified ref from an earlier snapshot, `find`, or drill. The command then: 1. Loads that ref's snapshot and its `RefEntry`. 2. Resolves the entry strictly against the live UI, with the same rules as any [ref action](/desktop/snapshots-and-refs#how-a-ref-resolves). 3. Reads the element's subtree to `--max-depth` (default 10). `--app` and `--window-id` are not used, because the ref already names its window, and `--skeleton` has no effect. 4. Allocates refs for the subtree and writes them into the same snapshot, so `snapshot_id` does not change. Off macOS the drill re-identifies its root over a complete read of the window and counts its new refs against that same read, so a drill's refs resolve by exactly the rules the skeleton's do. The result's root is the drilled element, and `ref_count` is the whole snapshot's count: 2 skeleton refs plus 31 drill refs. New refs continue the snapshot's counter, which is why this drill starts at `e3`. The ref you drilled stays valid. ## Re-drill after a change Run the same `--root` again to re-read the region after an action. The re-drill replaces that region's refs and leaves every other ref alone. 1. Before: snapshot `s2txcl95uvubmn` holds the skeleton refs `e1` and `e2` and the refs `e3` to `e33` from the drill of `e1`. 2. You click `e11` (the `7` button) and run `snapshot --root @s2txcl95uvubmn:e1` again. 3. The re-drill removes the refs that the previous drill of `e1` created: `e3` to `e33`. It keeps `e1` itself and every ref outside its subtree, here `e2`. 4. The new read allocates `e34` to `e64`. The counter continues, so a number is never reused. 5. After: the old `e3` to `e33` return `STALE_REF`, and `e1`, `e2`, and refs from other drilled regions still resolve. `ref_count` stays 33. A re-drill removes refs created by earlier drills of that root, refs from a full snapshot that sit under the root in the same window, and refs from drills nested inside the region. Refs for other regions, other windows, and the drilled root itself survive. ```bash agent-desktop get @s2txcl95uvubmn:e2 --property role # outside the region: still resolves agent-desktop get @s2txcl95uvubmn:e11 # inside the region: replaced ``` ```json { "version": "2.6", "ok": true, "command": "get", "data": { "property": "role", "ref": "@s2txcl95uvubmn:e2", "value": "group" } } ``` ```json { "version": "2.6", "ok": false, "command": "get", "error": { "code": "STALE_REF", "message": "@s2txcl95uvubmn:e11 not found in current RefMap", "recovery": { "strategy": "refresh_snapshot_then_retry_original", "retryable": true, "requires_fresh_snapshot": true }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` The second envelope omits `suggestion`. ## What a drill refuses A drill changes a saved snapshot, so it refuses anything that could leave the snapshot half updated.
Situation Result
--root with --surface INVALID_ARGS, "--root cannot be combined with --surface". A ref already carries the surface it was captured from. An explicit --surface window, the default, is allowed.
--root with --wait-for or --wait-for-gone INVALID_ARGS, "--root cannot be combined with --wait-for or --wait-for-gone". Run snapshot without --root, or omit the wait flags.
The read of the region is incomplete or misses the deadline TIMEOUT. The read happens before any write, so the saved snapshot is untouched.
The root ref is not in its snapshot, or no live element matches it STALE_REF. A missing snapshot is SNAPSHOT_NOT_FOUND.
The ref's source window is gone WINDOW_NOT_FOUND. Nothing is saved.
A drill is all or nothing. It replaces refs inside an existing snapshot, and a partial read could not re-allocate the descendants it removed, so an incomplete read of the region fails with `TIMEOUT` instead of returning a partial tree. Raise `--timeout-ms`, lower `--max-depth`, or drill a smaller region from a deeper ref. ```json { "version": "2.6", "ok": false, "command": "snapshot", "error": { "code": "TIMEOUT", "message": "Operation exceeded its deadline", "suggestion": "The target application may be busy or unresponsive", "recovery": { "strategy": "inspect_state_then_retry_original", "retryable": true, "requires_fresh_snapshot": true, "retry_after_ms": 250 }, "details": { "elapsed_ms": 1, "kind": "deadline", "timeout_ms": 1 }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` That is `snapshot --root @s2txcl95uvubmn:e1 --timeout-ms 1`. The completeness rule applies to the drill's own read of the region, so you can drill a ref that came from a partial snapshot. ## Scope a search `find` takes the same two scopes, so you can search a region without reading it: ```bash agent-desktop find --root @s2txcl95uvubmn:e1 --role button --name C --exact --first agent-desktop find --app gtk3-widget-factory --surface menu --name "Dark Theme" --exact --first ``` - `find --root ` resolves the ref strictly, then searches only that element's subtree. It needs no `--app`. - `find --surface ` searches an overlay instead of the window. See [Surfaces](/desktop/surfaces). They cannot combine. `find --root --surface menu` returns `INVALID_ARGS`, "--root cannot be combined with --surface", with the suggestion to run `find --surface` without `--root`, or to drop `--surface`. A `find` that returns matches saves its own new snapshot, and its refs are qualified with the returned `data.snapshot_id`: ```json { "version": "2.6", "ok": true, "command": "find", "data": { "match": { "interactive": true, "name": "C", "path": ["group", "group", "group", "group", "group", "group"], "ref_id": "@s3ufwq4n1uswjj:e1", "role": "button", "states": [], "value": null }, "snapshot_id": "s3ufwq4n1uswjj" } } ``` It does not change the snapshot of the root ref you searched. `find --count` saves nothing. A search that runs out of budget fails with `TIMEOUT` instead of returning partial matches. The filters are in the [commands reference](/desktop/commands/observation#find). ## Depth with max-depth `--max-depth` limits how many levels `snapshot` reads. The default is 10, and the largest accepted value is 50: ```json { "code": "INVALID_ARGS", "message": "max depth 60 exceeds the supported maximum of 50", "suggestion": "Use a max depth of 50 or less, or drill into a subtree with snapshot --root " } ``` Depth counts meaningful levels. Under `--skeleton` the depth is the smaller of `--max-depth` and 3. For a drill, `--max-depth` is the depth below the drilled element, so to go deeper than 10 levels, drill a node near the boundary. That is what the `children_count` markers are for. ## When to skip the skeleton - **A small app.** With fewer than about 50 interactive elements, a full `snapshot -i --compact` is cheaper than two passes. GNOME Calculator, with 40 refs, is in this range; it appears above because its tree is small enough to show. - **A known target.** If you know the role or the exact name, `find --role --name --exact` returns the ref in one command. - **A surface.** Do not use `--skeleton` with `--surface menu`, `sheet`, `alert`, or `popover`. A surface is already a small, focused tree. The command does not reject the combination, but it only adds a second pass. ## Partial observations A full snapshot that runs out of budget still succeeds, and returns what it saw: ```bash agent-desktop snapshot --app gtk3-widget-factory -i --compact --timeout-ms 50 ``` ```json { "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "gtk3-widget-factory", "complete": false, "truncated": true, "nodes_observed": 73, "ref_count": 27, "snapshot_id": "s2ck1chz8ypcwk", "tree": { "role": "window", "subtree_truncated": true, "children": [] } } } ``` The `tree` above is emptied for space; the real one holds the nodes that were read, with `subtree_truncated` on every node whose descendants were cut.
Field Meaning
complete On every snapshot. false means the tree is not whole.
truncated, nodes_observed Added when complete is false. nodes_observed counts the nodes read.
subtree_truncated: true On each node whose descendants the budget cut, so you can walk from the root to every boundary.
children_count Set by a depth cut. A node cut by the budget may carry no count, so treat subtree_truncated as the signal.
Read `complete` instead of waiting for a `TIMEOUT`. Only a full snapshot can be partial; a drill never is. To finish a partial tree, raise `--timeout-ms` or drill the regions you need. Reading the same window with `--timeout-ms 150` returned a complete tree. A cold Chromium or Electron app can take 10 to 25 seconds to expose its web tree. If a fresh snapshot comes back thin, raise `--timeout-ms` and snapshot again. ## Timeouts
Command --timeout-ms default At the deadline
snapshot 3000 A partial tree with complete: false.
snapshot --root 3000, shared by resolving the root and reading it TIMEOUT. Nothing is saved.
find 5000 TIMEOUT with no partial matches.
--- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Skills description: "Load agent-desktop's skills with the skills command. They are compiled into the binary, so they always match the installed version." url: "https://agent-browser.dev/desktop/skills" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Skills" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop ships guides written for AI agents: the observe, act, and verify loop, the retry rules, and what differs on each OS. They are compiled into the binary, so the instructions an agent reads always match the version it runs. The `skills` command prints them. It needs no permission and no desktop session. ## Bootstrap an agent Run these three commands before the first action: ```bash agent-desktop status # platform, permissions, supported surfaces agent-desktop skills get desktop # the loop, refs, verification, and retry rules agent-desktop skills get platform # what differs on the OS the binary runs on ``` To make this automatic, put the bootstrap in your agent's instructions file (`AGENTS.md` or equivalent): ```markdown Use `agent-desktop` to see and operate desktop apps. Before the first use, run `agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them. ``` ## The skills command Every form prints the JSON envelope. A skill's Markdown is in `data.content`.
Command Description
agent-desktop skills List skills, the same as skills list
agent-desktop skills list List every bundled skill with its aliases and references
agent-desktop skills get \ Print one skill's main guide
agent-desktop skills get \ \ Print one reference file of that skill
agent-desktop skills get \ --full Print the main guide followed by every reference, each introduced by a --- references/\.md --- line
agent-desktop skills path Report that skills are embedded, and list the available names
`--full` is the only flag the command adds, and it has no effect when you pass a reference. To read a skill as plain Markdown, extract `data.content`: ```bash agent-desktop skills get desktop | jq -r '.data.content' ``` `skills path` confirms there is no skills directory on disk: ```json { "version": "2.6", "ok": true, "command": "skills", "data": { "available": ["agent-desktop", "agent-desktop-macos", "agent-desktop-windows", "agent-desktop-linux", "agent-desktop-ffi", "jev-desktop"], "location": "embedded", "note": "Skills are compiled into this binary and are always version-matched. Run `agent-desktop skills get ` to print a skill, or redirect into a file to extract a copy." } } ``` ## Available skills `` accepts the canonical name or an alias.
Name Alias Covers
agent-desktop desktop The core guide for every OS: the snapshot and ref loop, verification and retry rules, and when to open each reference. Load it first.
agent-desktop-macos macos What differs on macOS: permissions, Notification Center, troubleshooting.
agent-desktop-windows windows What differs on Windows: PowerShell quoting, elevation, shell surfaces, Chromium and Electron, troubleshooting.
agent-desktop-linux linux What differs on Linux: the AT-SPI bus, X11 and Wayland, window lifecycle, capture and clipboard, shell and notifications, troubleshooting.
agent-desktop-ffi ffi Embedding agent-desktop through the C library: build and link, errors, ownership, threading.
jev-desktop jev An optional helper that drives an app from a plain-language goal. It needs a clone of the repository and an API key, and it is not part of the native CLI.
`platform` resolves to the skill for the OS the binary runs on. `skills list` marks platform skills with `platform` and `current`; this is the Linux entry on a Linux machine: ```json { "aliases": ["linux", "agent-desktop-linux"], "current": true, "name": "agent-desktop-linux", "platform": "linux", "references": [ "references/bus-and-permissions.md", "references/display-servers-and-input.md", "references/windows-and-lifecycle.md", "references/capture-and-clipboard.md", "references/shell-tray-and-notifications.md", "references/troubleshooting.md" ] } ``` An unknown name returns `INVALID_ARGS` and lists every alias: ```json { "code": "INVALID_ARGS", "message": "Unknown skill 'nope'. Known: desktop, agent-desktop, macos, agent-desktop-macos, windows, agent-desktop-windows, linux, agent-desktop-linux, ffi, agent-desktop-ffi, jev, jev-desktop" } ``` ## References Load a reference only when you need it. Name it by path (`references/workflows.md`), file name (`workflows.md`), or stem (`workflows`): ```bash agent-desktop skills get desktop workflows agent-desktop skills get linux troubleshooting ```
Skill References
desktop commands-observation, commands-interaction, commands-system, workflows
macos troubleshooting
windows permissions-and-elevation, chromium-and-electron, troubleshooting, shell-and-overlay
linux bus-and-permissions, display-servers-and-input, windows-and-lifecycle, capture-and-clipboard, shell-tray-and-notifications, troubleshooting
ffi build-and-link, error-handling, ownership, threading
An unknown reference returns `INVALID_ARGS` and lists the available ones. ## Install a skill in an agent harness A harness that supports skills can also install a static copy with `npx skills add`: ```bash npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop ``` You can also add `--skill agent-desktop-ffi` or the platform skill for your OS (`agent-desktop-macos`, `agent-desktop-windows`, or `agent-desktop-linux`). An installed copy can go stale when you upgrade the CLI, so prefer `agent-desktop skills get`, which always matches the binary you run. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Snapshots and refs description: What a snapshot contains, how refs are allocated and stored, and how a ref resolves back to a live element or goes stale. url: "https://agent-browser.dev/desktop/snapshots-and-refs" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Snapshots and refs" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). A snapshot is one observation of an application's accessibility tree. Window-owned surfaces read a window; macOS `menu` and `menubar` surfaces belong to the application and need no document window. agent-desktop saves the observation with a ref map and gives every actionable element a ref that you pass to later commands. ```bash agent-desktop snapshot --app gnome-calculator -i --compact ``` ```json { "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "gnome-calculator", "complete": true, "ref_count": 40, "snapshot_id": "s26u7pv9qta5g4", "window": { "id": "w-1.11-org-gnome-Calculator-a11y-948fd89a_9bda_453b_9c10_4a9e9eed3feb", "title": "Calculator" }, "tree": { "role": "window", "name": "Calculator", "native_id": { "kind": "ax_identifier", "value": "MathWindow" }, "children": [ { "available_actions": ["Click", "SetFocus"], "description": "Clear Display [Escape]", "name": "C", "native_id": { "kind": "ax_identifier", "value": "calc_clear_button" }, "ref_id": "@s26u7pv9qta5g4:e6", "role": "button" } ] } } } ``` The tree above is trimmed to one button. The real tree nests it under several unnamed `group` nodes. ## What a snapshot contains
Field Meaning
data.app, data.window The application and, for window-owned observations, the window that was read (id, title). data.window is omitted for macOS application-owned menu surfaces and process-owned snapshots of a background app with no window.
data.snapshot_id The id of the saved snapshot. It is part of every ref.
data.ref_count The number of refs the snapshot holds. After a --root drill it counts the whole snapshot, not only the drilled region.
data.complete Present on every snapshot. false means the observation ran out of budget. See Partial observations.
data.truncated, data.nodes_observed Present only when complete is false.
data.identity_unproven Present and true only when the snapshot needed one whole read of the window to prove its refs and that read failed or was cut short. The tree is still the one presented, but on Windows and Linux, and on macOS for an element with no address, its refs fail with identity_not_unique. Retake the snapshot with a larger --timeout-ms.
data.tree The root node. Every node nests its children.
A node always carries `role` and, when they exist, `ref_id`, `name`, `value`, `description`, `native_id`, `states`, `available_actions`, `children_count`, `subtree_truncated`, and `children`. Empty fields are omitted. `bounds` appears only with `--include-bounds`: ```json { "available_actions": ["Click", "SetFocus"], "bounds": { "height": 44.0, "width": 64.0, "x": 554.0, "y": 555.0 }, "name": "7", "native_id": { "kind": "ax_identifier", "value": "calc_7_button" }, "ref_id": "@s2jqig9fk9u6gz:e9", "role": "button" } ``` `children_count` and `subtree_truncated` mark where the tree was cut; [Skeleton and drill-down](/desktop/skeleton-and-drill-down) explains both. Three flags shrink the tree, and the [commands reference](/desktop/commands/observation#snapshot) lists every flag: - `-i` (`--interactive-only`) drops leaves that have no ref. It keeps nodes that lead to refs and nodes marked with `children_count`. Static text has no ref, so a row's label is dropped: to see which row a repeated button belongs to, drill that row's container with `--root` and leave out `-i`. - `--compact` collapses a single-child wrapper that has no name, value, description, id, state, or action. - `--max-depth` limits depth. The default is 10 and the maximum is 50. ## Choose the window For window-owned surfaces, `snapshot` and `find` read one window. Without `--app` they use the focused window, and `WINDOW_NOT_FOUND` means there is none. With `--app` and no `--window-id`, agent-desktop narrows the app's windows in this order and stops at the first step that leaves exactly one: 1. If the app has only one window, use it. 2. Keep only visible windows. 3. Keep only the focused window. 4. Keep only accessible windows. If several windows are still left, the command fails with `AMBIGUOUS_TARGET`, and `details.candidates` lists up to 10 of them. Run `list-windows --app ` and pass `--window-id`. On macOS, `menu` and `menubar` surfaces are application-owned. They can be observed without a document window. If you pass `--window-id`, it selects the app that owns the surface; the returned snapshot remains application-owned and omits `data.window`. ## Refs A ref is an element id with its snapshot id inside it: ```text @:eN # for example @s26u7pv9qta5g4:e6 ``` Every command accepts only this qualified form. A bare `@e6` fails with `INVALID_ARGS`; `@eN` is the key inside the saved ref map, and agent-desktop never prints it as output. Because each ref names its snapshot, it keeps pointing at its own snapshot when you read several apps or windows in turn. ```json { "version": "2.6", "ok": false, "command": "click", "error": { "code": "INVALID_ARGS", "message": "Invalid ref_id '@e1': expected @:e", "suggestion": "Use the snapshot-qualified ref that snapshot prints, for example @s8f3k2p9:e1. A bare @eN ref is not accepted.", "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` ### Which elements get a ref Refs are allocated depth-first in document order, from `e1` upward, so a parent gets its number before its children. The counter belongs to the snapshot, and a later drill-down continues it instead of reusing numbers. An element gets a ref when either condition holds: - Its role is interactive: `button`, `cell`, `checkbox`, `colorwell`, `combobox`, `dockitem`, `incrementor`, `link`, `listbox`, `menubutton`, `menuitem`, `option`, `radiobutton`, `slider`, `switch`, `tab`, `textfield`, or `treeitem`. - It advertises a primary action. `scrollarea` (Scroll) and `disclosure` (Expand and Collapse) qualify this way, because `scroll`, `expand`, and `collapse` need a ref to target them. In the example above, the calculator's history pane is a `scrollarea` with a ref. `SetFocus`, `RightClick`, and `ScrollTo` do not count on their own. Web runtimes advertise them on nearly every node, so counting them would give every inert container a ref and overflow the [ref map size limit](#where-refs-are-stored). Static text and inert containers stay in the tree without a ref. `find` returns them as context matches with `"ref_id": null`. A container cut off by a skeleton can also receive a ref as a drill target. See [Find the drill target](/desktop/skeleton-and-drill-down#find-the-drill-target). ## What a ref stores Each ref maps to a `RefEntry`: the evidence needed to find the same element again. The ref map holds evidence, not live objects.
Group Fields Used for
Process pid, process_instance Reject a ref whose process exited or whose pid was reused.
Identity retained_object, role, name, value, description, native_id, unique_at_capture Re-identify the element. retained_object is the element's own accessibility address, recorded on macOS. unique_at_capture records whether the entry was the only element in the window its snapshot captured with this role and these stable fields. A skeleton, depth-limited or drilled capture counts against that whole window, so a shallow presentation proves the same thing a full one does. A control's value is never identity for roles whose value changes while you use them, such as textfield, checkbox, and slider.
Geometry bounds, bounds_hash, bounds_space Not identity. get --property bounds falls back to them when the live read is unavailable; re-identification never compares them.
Capabilities states, available_actions What the element offered at snapshot time.
Source source_app, source_window_id, source_window_title, source_window_bounds_hash, source_surface Tie the ref to its window and surface.
Scope path, root_ref, path_is_absolute The element's child-index position at capture, and which drill-down created the ref.
## How a ref resolves A command never reuses a live handle from the snapshot. Every command that takes a ref, including `click`, `get`, `is`, `snapshot --root`, and `find --root`, re-identifies the element from its stored evidence first. 1. The command parses `@s26u7pv9qta5g4:e6` into the snapshot id and the key `@e6`. A bare or malformed ref fails with `INVALID_ARGS`. 2. `RefStore` loads `snapshots/s26u7pv9qta5g4/refmap.json` from the selected session's namespace. A missing snapshot fails with `SNAPSHOT_NOT_FOUND`. 3. The ref map looks up `@e6` and returns its `RefEntry`. A missing key fails with `STALE_REF`. 4. If the entry carries the element's own address, `resolve_element_strict` turns it back into that exact element. If the element is alive and its process, role and recorded id still match, the command acts on it. Otherwise the ref is `STALE_REF` with `details.kind: retained_object_stale`. No search runs, and the ref never lands on a look-alike. 5. If the entry has no address, `unique_at_capture` decides. A ref that was not unique at capture fails with `AMBIGUOUS_TARGET`, `details.kind: identity_not_unique`, before any live read. 6. A ref that was unique is counted again over the live window by role, native id, name and description. The count covers the whole window, which is the scope the capture proved the ref unique in. Exactly one match: the command acts on it. None: `STALE_REF`, `details.kind: saved_ref_absent`. Two or more: `AMBIGUOUS_TARGET`, `details.kind: saved_ref_ambiguous`. A field the live read could not answer leaves the count unproven, which returns `TIMEOUT` with `details.kind: resolution_identity_unknown` instead of a match. 7. After `STALE_REF` or `AMBIGUOUS_TARGET`, take a fresh observation and use the new ref. On macOS a ref carries the element's own address, so identical twins each keep their own ref. When the address could not be taken, macOS follows the rule below. Windows and Linux refs carry no address, so a ref that was not the only element matching its role and stable fields when it was captured is refused. The comparison is the identity fields and nothing else: role, native id, name, and description. Control values are volatile and never count. Bounds never count either, so a stored rectangle or a stored path cannot pick between two candidates. A ref that fails these checks is refused rather than guessed at.
Error Meaning What to do
STALE_REF The key is not in the snapshot's ref map, no live element matches the stored identity (details.kind: saved_ref_absent), or the element the ref names is gone (retained_object_stale). The UI changed, the window closed, or a re-drill replaced the ref. Take a fresh snapshot, or re-drill the region, and use the new ref. Nothing was sent to the app.
AMBIGUOUS_TARGET Two live elements match the stored identity (saved_ref_ambiguous), the ref was not proven unique when it was captured (identity_not_unique), or several windows matched one --app. For several windows, name the window. A ref that was not proven unique came from identical twins or from a window that could not be read whole within the snapshot timeout; only the second is fixed by a new snapshot, with a larger --timeout-ms. Off macOS, identical twins cannot be picked by ref at all and every snapshot refuses them again: only an explicit coordinate click, agent-desktop --headed mouse-click --xy X,Y, can choose one. It takes the foreground and is unavailable on a Linux Wayland session.
SNAPSHOT_NOT_FOUND No saved snapshot has that id: it was pruned, it belongs to another session, or the state root differs. Take a new snapshot under the same --session and state root as the command that uses it.
The full `STALE_REF` envelope after a re-drill is in [Skeleton and drill-down](/desktop/skeleton-and-drill-down#re-drill-after-a-change), and every code with its retry rule is in [Errors and recovery](/desktop/errors-and-recovery). ## Refresh after the UI changes Refs describe one observation. Re-read after anything that can change the tree, and act only on refs from the latest read of the region: - `snapshot` returns a new snapshot id and a new ref map. - `find` with matches also saves a new snapshot, and its `data.snapshot_id` qualifies the refs it returns. - `snapshot --root` re-reads one region inside an existing snapshot. See [Skeleton and drill-down](/desktop/skeleton-and-drill-down#re-drill-after-a-change). An older snapshot stays on disk until it is pruned, and its refs keep resolving while their elements still match. Do not rely on that: after the UI changes, an old ref can fail, or match an element that is no longer the one you meant. ## Where refs are stored `RefStore` keeps one file per snapshot at `snapshots//refmap.json` under the [state root](/desktop/configuration#state-root). Inside a [session](/desktop/sessions), the same layout sits under `sessions//`, and a lookup never searches another session's snapshots.
Limit Value What happens
Size of one ref map 1 MB (1,048,576 bytes) The write fails with INTERNAL. Narrow the observation with --skeleton, --max-depth, --root, or a tighter find.
Saved snapshots per namespace 128 When a save pushes the count past 128, the oldest are removed until 96 remain. The snapshot just written is never removed.
--- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Surfaces description: Observe menus, sheets, alerts, popovers, and OS shell areas separately from a window or application with --surface, list-surfaces, and open-system-surface. url: "https://agent-browser.dev/desktop/surfaces" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Surfaces" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). A surface is a UI layer that you observe separately from a window or application's full accessibility tree: an open menu, a sheet, a popover, an alert, or an OS shell area such as the taskbar. While an overlay is open, the refs from the window behind it are the wrong target, so read the overlay itself. ```bash agent-desktop snapshot --app gtk3-widget-factory --surface menu -i --compact agent-desktop find --app gtk3-widget-factory --surface menu --name "Dark Theme" --exact --first ``` ## Surface kinds `--surface` takes these values on `snapshot` and `find`. The default is `window`. Names with hyphens also accept underscores, such as `system_tray`.
Group Kinds
The app or window and its layers window, focused, menu, menubar, sheet, popover, alert
OS shell desktop, taskbar, system-tray, system-tray-overflow, quick-settings, notification-center, toolbar, dock, spotlight, menu-bar-extras, start-menu, action-center
Each platform supports a subset:
Platform Supported surfaces
macOS window, focused, menu, menubar, sheet, popover, alert, dock, menu-bar-extras
Windows window, focused, sheet, menu, start-menu, taskbar, system-tray, system-tray-overflow, action-center
Linux window, sheet, alert, menu, plus GNOME Shell kinds the session can resolve
Per-kind limits are on the platform pages: [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), [Linux](/desktop/platforms/linux). ## Check what is supported `status` reports `supported_surfaces` for the current machine and session. On a Linux X11 session without GNOME Shell: ```json { "supported_surfaces": ["window", "sheet", "alert", "menu"] } ``` Read it before you ask for a surface. A kind this OS does not have fails with `INVALID_ARGS`, with the supported list in `details.supported_surfaces` and your request in `details.requested_surface`. On Linux, an empty `supported_surfaces` means the accessibility bus did not answer, not that the desktop is empty. ## Know when a surface opened You rarely need to poll for a surface. `click`, `double-click`, `right-click`, `triple-click`, `expand`, `select`, and `press` report the surfaces open after the action in `data.surfaces`. This is the result of clicking the **Menu** button in GTK Widget Factory: ```json { "surfaces": [ { "id": "w-1.9-org-a11y-atspi-accessible-229", "type": "window" }, { "id": "w-1.9-org-a11y-atspi-accessible-229", "item_count": 4, "type": "menu" } ] } ``` A `menu`, `sheet`, or `alert` entry means the app is waiting on that overlay. Observe it next with `snapshot --surface` or `find --surface`. The field is omitted when nothing is open or the read failed. ## Snapshot or search a surface ```bash agent-desktop snapshot --app gtk3-widget-factory --surface menu -i --compact ``` The snapshot of that menu holds six refs and nothing from the window behind it: ```json [ { "ref_id": "@s2n10v3haasex6:e1", "role": "button", "name": "Get Busy" }, { "ref_id": "@s2n10v3haasex6:e2", "role": "checkbox", "name": "Dark Theme" }, { "ref_id": "@s2n10v3haasex6:e3", "role": "checkbox", "name": "Slide Pages" }, { "ref_id": "@s2n10v3haasex6:e4", "role": "button", "name": "Inspector" }, { "ref_id": "@s2n10v3haasex6:e5", "role": "button", "name": "Keyboard Shortcuts" }, { "ref_id": "@s2n10v3haasex6:e6", "role": "button", "name": "About Widget Factory" } ] ``` To reach one item, `find --surface` is cheaper than a dump: ```json { "version": "2.6", "ok": true, "command": "find", "data": { "match": { "interactive": true, "name": "Dark Theme", "path": ["group", "group", "group", "group", "group", "group"], "ref_id": "@svj2992hf8d2k:e1", "role": "checkbox", "states": [], "value": null }, "snapshot_id": "svj2992hf8d2k" } } ``` How agent-desktop selects an owner depends on the kind and platform: - `sheet`, `popover`, and `alert`: the app's accessible windows in order, focused first and then visible, and the first that presents one wins. With no `--app`, the frontmost process. - `sheet`, `popover`, and `alert`: the app's accessible windows in order, focused first and then visible, and the first that presents one wins. With no `--app`, the frontmost process. - On macOS, `menu` and `menubar` are application-owned and do not require a document window. If you pass `--window-id`, it selects the app that owns the surface. Snapshot data includes `app` and omits `window`. A complete lookup with no open menu or menu bar returns `ELEMENT_NOT_FOUND` with `details.kind: "surface_absent"` and `details.complete: true`. To read an application menu, snapshot `menubar` when it exists, click its item without `--headed`, then snapshot `--surface menu`. An absent menu bar can mean the app exposes none. - On implementations where `menu`, `menubar`, and `focused` are window-owned, any app window identifies the process, so several open windows are not ambiguous. - On macOS, `window`, `menubar`, and `menu-bar-extras` can read a background app with no window through its process. The result omits `window`. A regular app that has not shown a window returns `WINDOW_NOT_FOUND`; `focused` requires a focused window. - A shell kind needs no `--app`, because no application owns it. On macOS, `menu-bar-extras` with `--app` reads that app's own status item. - For window-owned surfaces, `--window-id` pins the read to that window. A missing `complete`, `complete: false`, a timeout, or an incomplete accessibility read leaves menu absence unknown. After a delivery with `retry: unsafe`, inspect the current UI before taking another action. A surface that is not open returns an error. On Linux, a closed menu returns `ELEMENT_NOT_FOUND`: ```json { "code": "ELEMENT_NOT_FOUND", "message": "The window does not present an open menu", "suggestion": "Open the menu first, then snapshot the menu surface" } ``` A closed shell surface returns `WINDOW_NOT_FOUND`. `--surface` cannot combine with `--root`, because a ref already carries the surface it was captured from. Do not use `--skeleton` on a surface: it is already small and focused. See [Skeleton and drill-down](/desktop/skeleton-and-drill-down#when-to-skip-the-skeleton). ## List what is open ```bash agent-desktop list-surfaces --app gtk3-widget-factory ``` ```json { "version": "2.6", "ok": true, "command": "list-surfaces", "data": { "pid": 8258, "surfaces": [ { "id": "w-1.9-org-a11y-atspi-accessible-229", "type": "window" }, { "id": "w-1.9-org-a11y-atspi-accessible-229", "item_count": 4, "type": "menu" } ] } } ``` `list-surfaces` returns the overlays an app presents now, which are the kinds `snapshot --surface` accepts for it. Shell surfaces belong to the OS and never appear here. An entry can carry `"unclassified": ["sheet"]` or `["menu"]` when the check could not read that window, often because the app was busy. Treat the missing entry as unknown, not absent. ## Open an OS shell surface `open-system-surface` raises a shell surface. On Windows and Linux it returns the window it presents, in the `w-` form that `list-windows` uses. On macOS it returns only `{"surface": "dock"}`, because no window owns these surfaces: ```bash agent-desktop --headed open-system-surface --surface action-center agent-desktop snapshot --surface action-center -i --compact ``` It takes the foreground, so on Windows and Linux headless mode refuses it with `POLICY_DENIED`; pass `--headed` (see [Headed mode](/desktop/guides/headed-mode)). An already-present surface returns without being raised again. A kind the OS does not expose returns `INVALID_ARGS`, and the suggestion or `platform_detail` names the kinds to use instead. On macOS, `dock` and `menu-bar-extras` are always on screen, so the command returns them at once, headless, with no `--headed`; open Control Center or Notification Center by clicking its `menu-bar-extras` item. Which kinds open on Windows and Linux is on their platform pages. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: App and window commands description: Reference for launch, close-app, list-apps, list-windows, focus-window, resize-window, move-window, minimize, maximize, and restore. url: "https://agent-browser.dev/desktop/commands/app-window" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "App and window" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). These commands start and stop apps and manage their windows. `--app` takes the name that `list-apps` prints. A window id from `list-windows` is the exact selector when an app has several windows. The window commands (`resize-window`, `move-window`, `minimize`, `maximize`, `restore`) pick their target the same way: - `--window-id` selects that window. `--app` can narrow the search but is not required. - `--app` alone selects the app, then one of its windows: its only window if it has one, otherwise the visible ones, then the single focused one, then the accessible ones. If that does not settle on one window, the command fails with `AMBIGUOUS_TARGET`. - With neither flag, the target is the focused window. `close-app` and the five window commands return an [OperationReport](/desktop/json-output#operationreport): read `disposition.delivery` to learn what was proved. Lookup failures such as `APP_NOT_FOUND` and `WINDOW_NOT_FOUND` carry `not_delivered`, so they are safe to retry once the name or id is fixed. The examples on this page are real output from GNOME apps on Linux. ## launch Starts an app, or attaches to a running instance, and returns once its process is running. A running process and a visible window are separate facts, so `window` appears in the result only when the app already has one. ```bash agent-desktop launch [--timeout MS] [--arg VALUE]... [--env KEY=VALUE]... [--cwd PATH] \ [--no-attach] [--activate] [--cdp [PORT]] ```
Flag or argument Default Meaning
APP required Application name, bundle ID, or path. The accepted forms depend on the platform.
--timeout MS 30000 Upper bound for the whole launch.
--arg VALUE none Command-line argument for the app. Repeatable, order preserved. Write --arg=-x for a value that starts with a hyphen.
--env KEY=VALUE none Environment variable for the launched process. Repeatable. A repeated key is INVALID_ARGS.
--cwd PATH none Working directory for the launched process.
--no-attach off Start a fresh instance instead of attaching to a running one.
--activate off Bring the app forward so it presents a window, and wait for one up to --timeout. This takes the foreground.
--cdp \[PORT] off Launch with a loopback-only Chrome DevTools Protocol endpoint and verify that it answers. Omit PORT, or pass 0, to pick a free port.
```bash agent-desktop launch gtk3-demo ``` ```json { "version": "2.6", "ok": true, "command": "launch", "data": { "app": "gtk3-demo", "pid": 21766, "process_instance": "6d91df60:120710", "window": { "accessible": true, "app_name": "gtk3-demo", "bounds": { "height": 656.0, "width": 810.0, "x": 630.0, "y": 0.0 }, "id": "w-1.105-org-a11y-atspi-accessible-48", "is_focused": true, "minimized": false, "pid": 21766, "process_instance": "6d91df60:120710", "title": "Application Class", "visible": true } } } ``` The result also carries `cdp` when `--cdp` was verified, `renderer: "chromium"` for a detected Chromium or Electron app, and sometimes a `suggestion`. A missing `window` is not a failure. With `--cdp`, `data.cdp` holds `http_endpoint`, `port`, `product`, and `websocket_url` for a CDP client such as agent-browser; [Chromium and Electron apps](/desktop/guides/chromium-and-electron) walks through the handoff. Errors specific to `launch`:
Code and detail Cause
APP_NOT_FOUND The name does not resolve to an app.
AMBIGUOUS_TARGET Several running instances match.
APP_UNRESPONSIVE The process exited before it showed a window.
ACTION_FAILED, details.kind: cdp_requires_fresh_launch --cdp on an app that is already running. launch never quits a running app for you; run close-app, then launch again.
INVALID_ARGS, details.kind: cdp_port_in_use or cdp_switch_conflict The named port is bound, or an --arg sets a remote-debugging switch that --cdp owns.
ACTION_FAILED, details.kind: cdp_endpoint_unavailable The app started but no verified DevTools endpoint answered. The app stays running.
Platform limits: the accepted `APP` forms differ by OS. See [macOS](/desktop/platforms/macos#app-naming), [Windows](/desktop/platforms/windows#app-naming), and [Linux](/desktop/platforms/linux#app-naming). ## close-app Quits an app and all its windows, and succeeds only after the process is seen gone. No command closes a single window: click that window's Close button instead. ```bash agent-desktop close-app [--force] ```
Flag or argument Default Meaning
APP required Application name as list-apps prints it.
--force off Terminate the process instead of quitting gracefully.
```json { "version": "2.6", "ok": true, "command": "close-app", "data": { "action": "close-app", "app": "gtk3-demo", "details": { "evidence": "process_instance_exited" }, "disposition": { "delivery": "delivered_verified", "retry": "unsafe" }, "method": "graceful", "steps": [ { "label": "terminate_request", "mechanism": "semantic_api", "outcome": "succeeded" }, { "label": "process_exit", "mechanism": "semantic_api", "outcome": "succeeded", "verified": true } ], "target": { "kind": "app", "name": "gtk3-demo", "process": { "instance": "6d91df60:120710", "pid": 21766 } } } } ``` `method` is `graceful` or `force`. If the app had already exited, the report is `not_delivered` with `details.evidence: "process_instance_not_running"`. Errors specific to `close-app`: - `APP_NOT_FOUND` when no running app matches, `AMBIGUOUS_TARGET` when several instances share the name. - `INVALID_ARGS` for a protected system process, with or without `--force`. - A save prompt keeps the app open. Read it with `snapshot --surface sheet`, answer it, then close again. Platform limits: graceful close needs an X11 client on Linux, and the protected processes differ by OS. See [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). ## list-apps Lists running GUI apps. ```bash agent-desktop list-apps [--app NAME] [--timeout-ms MS] ```
Flag Default Meaning
--app NAME all apps Filter by name, case-insensitively.
--timeout-ms MS 5000 Deadline for the inventory.
```json { "version": "2.6", "ok": true, "command": "list-apps", "data": { "apps": [ { "bundle_id": "org.gnome.Calculator", "name": "gnome-calculator", "pid": 8255, "presentation": "foreground", "process_instance": "6d91df60:28340" } ] } } ``` `presentation` is `foreground` for apps with ordinary windows and `background` for apps with no Dock or taskbar entry. When the platform could not read every process in time, the result adds `complete: false` and a `skipped` array: treat a missing app as unknown, not absent. `--app` takes the `list-apps` name; Linux also accepts a desktop-entry name. See [Linux](/desktop/platforms/linux#app-naming). ## list-windows Lists windows. `data` is the array itself. ```bash agent-desktop list-windows [--app NAME] [--timeout-ms MS] ```
Flag Default Meaning
--app NAME all apps Filter to one app by exact, case-insensitive name.
--timeout-ms MS 5000 Deadline for the inventory.
```json { "version": "2.6", "ok": true, "command": "list-windows", "data": [ { "accessible": true, "app_name": "gnome-calculator", "bounds": { "height": 616.0, "width": 370.0, "x": 535.0, "y": 142.0 }, "id": "w-1.11-org-gnome-Calculator-a11y-948fd89a_9bda_453b_9c10_4a9e9eed3feb", "is_focused": false, "minimized": false, "pid": 8255, "process_instance": "6d91df60:28340", "title": "Calculator", "visible": true } ] } ``` `accessible: false` means semantic commands cannot reach that window; it still accepts screenshots and coordinate input. Errors: `APP_NOT_FOUND` for an unknown `--app`, `APP_UNRESPONSIVE` for a stopped app, and `TIMEOUT` when a busy app left every window unanswered. A timeout is not an empty list. ## focus-window Brings a window to the front and succeeds only when the OS reports that window as focused on two consecutive reads. ```bash agent-desktop focus-window [--window-id ID] [--app NAME] [--title TEXT] ```
Flag Default Meaning
--window-id ID none Window id from list-windows.
--app NAME none Application name.
--title TEXT none Window title, partial match. Combine with --app to narrow.
Pass at least one flag; with none the command fails with `INVALID_ARGS`. `data.focused` is the window that holds focus afterward, with the same fields as a `list-windows` entry. When a modal dialog holds the focus, `focused` is the dialog, so read `focused.title` before you act. ```json { "version": "2.6", "ok": true, "command": "focus-window", "data": { "focused": { "app_name": "gedit", "id": "w-1.10-org-a11y-atspi-accessible-162", "is_focused": true, "title": "Untitled Document 1 - gedit" } } } ``` The example above is trimmed to four fields. Errors: `WINDOW_NOT_FOUND` when nothing matches, `AMBIGUOUS_TARGET` when several windows still match, and `ACTION_FAILED` when the OS never reports the window focused. The command takes the foreground but does not need `--headed`. Platform limits: Linux needs X11. See [Linux](/desktop/platforms/linux) and [Windows](/desktop/platforms/windows). ## resize-window Sets a window's width and height. ```bash agent-desktop resize-window --width PX --height PX [--app NAME] [--window-id ID] ```
Flag Default Meaning
--width PX, --height PX required New window size.
--app NAME focused app Application name.
--window-id ID none Window id from list-windows.
```bash agent-desktop resize-window --app gedit --width 800 --height 600 ``` ```json { "version": "2.6", "ok": true, "command": "resize-window", "data": { "action": "resize-window", "details": { "observed": { "height": 600, "hidden": false, "maximized_horizontal": false, "maximized_vertical": false, "width": 800, "x": 0, "y": 0 }, "tolerance": 2.0, "units": "pixels", "verification_scope": "window_size" }, "disposition": { "delivery": "delivered_verified", "retry": "unsafe" }, "height": 600.0, "steps": [{ "label": "window_resize", "mechanism": "semantic_api", "outcome": "succeeded", "verified": true }], "target": { "id": "w-1.10-org-a11y-atspi-accessible-162", "kind": "window", "process": { "instance": "6d91df60:28341", "pid": 8261 }, "title": "Untitled Document 1 - gedit" }, "width": 800.0 } } ``` `delivered_verified` means the adapter read the size back within `details.tolerance`. `details.observed` is a Linux addition. Errors: `INVALID_ARGS` for a non-finite or out-of-range size, `STALE_REF` when the window identity changed before the write, and `ACTION_FAILED` when the window manager clamped or ignored the request. After a failed write, read the bounds before you retry. Platform limits: the tolerance differs by OS, and on Linux window control uses an X11 client, so Wayland-native windows are refused. See [Linux](/desktop/platforms/linux). ## move-window Moves a window to screen coordinates. ```bash agent-desktop move-window --x=X --y=Y [--app NAME] [--window-id ID] ```
Flag Default Meaning
--x X, --y Y required New position. Write a negative value with =, as in --x=-100.
--app NAME focused app Application name.
--window-id ID none Window id from list-windows.
The result has the same shape as [`resize-window`](#resize-window), with step label `window_move`, request fields `x` and `y`, and `details.verification_scope: "window_position"`. Errors and platform limits are the same as `resize-window`. ## minimize Minimizes a window. ```bash agent-desktop minimize [--app NAME] [--window-id ID] ```
Flag Default Meaning
--app NAME focused app Application name.
--window-id ID none Window id from list-windows.
```json { "version": "2.6", "ok": true, "command": "minimize", "data": { "action": "minimize", "details": { "verification_scope": "window_minimized_state" }, "disposition": { "delivery": "delivered_verified", "retry": "unsafe" }, "steps": [{ "label": "window_minimize", "mechanism": "semantic_api", "outcome": "succeeded", "verified": true }], "target": { "id": "w-1.10-org-a11y-atspi-accessible-162", "kind": "window", "process": { "instance": "6d91df60:28341", "pid": 8261 }, "title": "Untitled Document 1 - gedit" } } } ``` A minimized window reports its elements as offscreen, so ref actions on it fail until you [`restore`](#restore) it. A window manager can ignore a minimize sent within about 250 ms of a restore, so read the window state before you retry. Platform limits are the same as `resize-window`. ## maximize Maximizes a window, or zooms it on macOS. Flags are the same as [`minimize`](#minimize). The step label is `window_maximize`, and a verified report names `details.verification_scope: "window_maximized_state"`. ```bash agent-desktop maximize [--app NAME] [--window-id ID] ``` ## restore Restores a minimized or maximized window. Flags are the same as [`minimize`](#minimize). ```bash agent-desktop restore [--app NAME] [--window-id ID] ``` `restore` has no single read-back across its cases, so a successful report is `delivered_unverified`: ```json { "version": "2.6", "ok": true, "command": "restore", "data": { "action": "restore", "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "window_restore", "mechanism": "semantic_api", "outcome": "succeeded" }], "target": { "id": "w-1.10-org-a11y-atspi-accessible-162", "kind": "window", "process": { "instance": "6d91df60:28341", "pid": 8261 }, "title": "Untitled Document 1 - gedit" } } } ``` Check the window with `list-windows` afterward. Platform limits are the same as `resize-window`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: batch description: Reference for batch, with the input format, limits, validation, stop-on-error behavior, which commands are refused, and the output shape. url: "https://agent-browser.dev/desktop/commands/batch" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Batch" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). `batch` runs a JSON array of commands in order, in one process. It is bounded, sequential, and non-atomic: a failed entry does not undo earlier ones. When to batch is in [Batching commands](/desktop/guides/batch). ## batch ```bash agent-desktop batch JSON [--stop-on-error] [--timeout-ms MS] agent-desktop batch '[{"command":"version"},{"command":"wait","args":{"ms":100}}]' --stop-on-error ```
Flag or argument Default Meaning
JSON required A JSON array of entries.
--stop-on-error off Halt after the first entry that fails. Without it, later entries still run.
--timeout-ms MS 60000 Wall-clock budget for the whole batch, at most 86400000. Each entry's own deadlines are capped by what remains.
### Entry format
Key Required Meaning
command yes A command name such as click or wait.
args no The command's arguments. Omit it for a command with no arguments.
session no Session id for this entry. Without it, the entry uses the batch's own session.
`args` keys are the CLI flag names with hyphens turned into underscores (`window_id`, `timeout_ms`). A positional argument is named after its placeholder in lowercase (`ref_id`, `text`, `combo`, `app`). Repeatable flags take arrays. `wait` uses flat keys: `ms`, `element`, `predicate`, `value`, `action`, `window`, `text`, `count`, `menu`, `menu_closed`, `notification`, `event`, `window_id`, `app`, `timeout`. `skills`, `session`, and `trace` take their subcommand as `action`, as in `{"command":"session","args":{"action":"list"}}`. Unknown keys anywhere are rejected. `--headed` and the agent id apply to every entry. A global `--wait-for` does not; put a `wait` entry where you need one. ### Limits
Limit Value When exceeded
Input size 1 MiB INVALID_ARGS before parsing.
Entries 64 INVALID_ARGS.
Output size 4 MiB The entry that crosses it becomes an INVALID_ARGS entry with details.result_omitted: true, and the batch stops with stopped.reason: "output_limit".
### Validation before the first entry The whole array is checked before any entry runs, and one bad entry rejects the batch with nothing run: - The JSON must parse, and every entry must decode against its command's arguments. - Each entry gets the preflight a direct call gets: ref syntax (a bare `@e1` is rejected), flag combinations such as `snapshot --root` with `--surface`, and notification identity flags. - Each entry needs the permissions its command needs; a missing one returns `PERM_DENIED` for the whole batch. - In a multi-agent session, every mutating desktop entry needs an agent id. - `batch` (`Batch commands cannot be nested`) and `cursor-overlay` are refused inside a batch. The rejection names the entry in `details.batch_index` and `details.batch_command`, with `not_delivered`. Checks that need the live desktop, such as a stale ref or a headless physical command, happen when that entry runs and appear as that entry's own error. ### Execution and output Entries run one at a time through the same dispatch as the CLI, so each entry's `data` or `error` is what a direct call returns. Nothing is retried. The batch itself returns `ok: true` and exit code `0` even when entries fail, so read each entry's `ok`. ```json { "version": "2.6", "ok": true, "command": "batch", "data": { "completed_entries": 2, "elapsed_ms": 100, "limits": { "max_entries": 64, "max_input_bytes": 1048576, "max_output_bytes": 4194304 }, "not_started_entries": 0, "results": [ { "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" }, "execution": "completed", "index": 0, "ok": true, "version": "2.6" }, { "command": "wait", "data": { "waited_ms": 100 }, "execution": "completed", "index": 1, "ok": true, "version": "2.6" } ], "semantics": { "atomic": false, "batch_retries": false, "command_retry_contracts": "preserved", "error_disposition": "error.disposition", "order": "sequential", "successful_action_disposition": "data.disposition" }, "timeout_ms": 60000, "total_entries": 2 } } ```
Field Meaning
results\[] One envelope per entry, with index and execution (completed or not_started). A not_started entry has not_started_reason and a not_delivered error.
total_entries, completed_entries, not_started_entries Counts. Entries after a stop are counted in not_started_entries without a row.
stopped Present when the batch ended early: reason is stop_on_error, deadline, output_limit, or pre_action_baseline_failed, with the index.
- **Deadline.** Before each entry the batch checks `--timeout-ms`. If it expired, that entry is `not_started` with reason `deadline` and the batch stops. - **Event waits.** When a `wait` entry with `event` follows another entry, the batch captures the event baseline before that earlier entry runs. If the capture fails, the earlier entry does not start (`pre_action_baseline_failed`). - **Sessions.** An entry whose session ended before it starts is `not_started` with reason `session_ended`. Refs in `args` must already exist when you write the array. A `snapshot` entry inside the batch creates refs you cannot name in later entries, so take the snapshot in a separate call. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Clipboard commands description: Reference for clipboard-get, clipboard-set, and clipboard-clear, including the text, image, and file-list formats and the result shapes. url: "https://agent-browser.dev/desktop/commands/clipboard" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Clipboard" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). The clipboard commands read and write the system clipboard. They need no accessibility permission and no `--headed`. The examples are real output from Linux X11. ## clipboard-get Reads the clipboard as text, an image, or a list of files. ```bash agent-desktop clipboard-get [--format text|auto|image|file-urls] [--out PATH] ```
Flag Default Meaning
--format text text, auto (file references first, then an image, then text), image, or file-urls.
--out PATH a private file in the state root Where to write image content. Without it the image goes to the active session's clipboard/ directory, or to tmp/ under the state root.
```json { "version": "2.6", "ok": true, "command": "clipboard-get", "data": { "text": "hello from agent-desktop", "type": "text" } } ``` An image result is `{ "type": "image", "path", "width", "height" }`, and a file list is `{ "type": "file_urls", "file_urls": [...] }`. Image bytes are never inlined. When the clipboard holds nothing in the requested form, the command still succeeds: ```json { "version": "2.6", "ok": true, "command": "clipboard-get", "data": { "found": false, "type": "text" } } ``` Check `found` before you read `text`. Errors: `INVALID_ARGS` for an unknown format or an unwritable `--out`, and `PLATFORM_NOT_SUPPORTED` when a platform helper is missing. ## clipboard-set Writes text, a PNG image, or a list of file paths. ```bash agent-desktop clipboard-set TEXT printf '%s' 'Hello' | agent-desktop clipboard-set --stdin agent-desktop clipboard-set --image /tmp/a.png agent-desktop clipboard-set --file-url /tmp/a.txt --file-url /tmp/b.txt ```
Flag or argument Default Meaning
TEXT empty text Text to write.
--stdin none Read up to 128 KiB from a pipe or redirected file. Terminal input is rejected. Cannot be combined with TEXT, --image, or --file-url.
--image PATH none A regular PNG file of at most 64 MiB.
--file-url PATH none A file path to write as a file reference. Repeatable. Every path must exist.
With `--stdin`, the command waits until the input ends. Empty input and all line breaks are kept, including a final newline. Use `printf '%s'` to avoid adding one. Input must be valid UTF-8 without NUL bytes (`\0`). `--stdin` keeps text out of process arguments. Batch uses the JSON `text` field, which puts the text in process arguments. Run secret steps as separate `--stdin` commands. To copy the text `--stdin`, use `agent-desktop clipboard-set -- --stdin`. Without `--stdin`, one call writes one representation: `--file-url` wins over `--image`, which wins over `TEXT`. The result is an [OperationReport](/desktop/json-output#operationreport) plus the written `type`: ```json { "version": "2.6", "ok": true, "command": "clipboard-set", "data": { "action": "clipboard-set", "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "clipboard_write", "mechanism": "semantic_api", "outcome": "succeeded" }], "target": { "kind": "clipboard" }, "type": "text" } } ``` macOS reads the content back and reports `delivered_verified`; Windows and Linux report `delivered_unverified`, so confirm with `clipboard-get`. On Linux X11, a helper process (`agent-desktop-linux-clipboard-owner`) keeps serving the content after the command exits; without it beside the binary, the command returns `PLATFORM_NOT_SUPPORTED`. Errors: `INVALID_ARGS` for a bad `--image` or a missing `--file-url` path, and `TIMEOUT` with `lock_timeout` when another agent holds the input lock. ## clipboard-clear Empties the clipboard. It takes no arguments. ```bash agent-desktop clipboard-clear ``` ```json { "version": "2.6", "ok": true, "command": "clipboard-clear", "data": { "action": "clipboard-clear", "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "clipboard_write", "mechanism": "semantic_api", "outcome": "succeeded" }], "target": { "kind": "clipboard" } } } ``` Confirm with `clipboard-get`, which then returns `found: false`. Platform limits: Wayland needs `wl-clipboard`. See [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Commands description: "All 60 agent-desktop commands by group, with a one-line description and a link to each command's reference entry." url: "https://agent-browser.dev/desktop/commands" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "All commands" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop has 60 commands in 11 groups. Every command prints one JSON envelope with `version`, `ok`, `command`, and either `data` or `error`; [JSON output](/desktop/json-output) defines it. ```bash agent-desktop --help # list every command agent-desktop --help # flags and arguments for one command agent-desktop skills get desktop # the agent usage guide compiled into the binary ``` `--help` prints the flags of the binary you installed. Flags that apply to every command, such as `--headed`, `--session`, `--agent-id`, `--trace`, and the post-action `--wait-for` flags, are documented once in [Configuration](/desktop/configuration).
Command What it does
App and window
launch Start an app, or attach to a running one, and return once its process is running.
close-app Quit an app and all its windows, or end its process with --force.
list-apps List running GUI apps.
list-windows List windows with ids, titles, bounds, and focus state.
focus-window Bring a window to the front and confirm the OS reports it focused.
resize-window Set a window's width and height.
move-window Move a window to screen coordinates.
minimize Minimize a window.
maximize Maximize or zoom a window.
restore Restore a minimized or maximized window.
Observation
snapshot Capture the accessibility tree as JSON with qualified refs.
find Search for elements by role, name, value, text, description, native id, or state.
get Read one property of a ref: text, value, title, bounds, role, or states.
is Check one state of a ref: visible, enabled, checked, focused, expanded, or selected.
screenshot Capture an app window or a display as an image.
list-displays List displays with bounds, scale factor, and primary flag.
list-surfaces List the overlays an app presents now, such as sheets and menus.
open-system-surface Raise an OS shell surface and return the window it presents.
Interaction
click Activate an element with its primary action.
double-click Double-click an element. Usually needs --headed.
triple-click Triple-click an element. Needs --headed.
right-click Open an element's context menu.
type Insert text into a text target.
set-value Write a value directly through the accessibility value path.
clear Empty a text target.
focus Move keyboard focus to an element without clicking it.
select Pick an option in a list, dropdown, or combo box.
toggle Flip a checkbox or switch.
check Set a checkbox or switch to checked, idempotently.
uncheck Set a checkbox or switch to unchecked, idempotently.
expand Expand a disclosure triangle, tree item, or accordion.
collapse Collapse a disclosure triangle, tree item, or accordion.
Scroll
scroll Scroll a scroll area in a direction by a number of units.
scroll-to Bring an element into the visible area.
Keyboard
press Send a key combo such as return or ctrl+s.
key-down Reserved. Fails closed instead of holding a key.
key-up Reserved. Fails closed instead of releasing a key.
Mouse
hover Move the cursor to an element's center or to coordinates. Needs --headed.
drag Drag from one element or point to another. Needs --headed.
mouse-move Move the cursor to screen coordinates. Needs --headed.
mouse-click Click at screen coordinates. Needs --headed.
mouse-down Reserved. Fails closed instead of pressing a button.
mouse-up Reserved. Fails closed instead of releasing a button.
mouse-wheel Scroll the mouse wheel at screen coordinates. Needs --headed.
Notifications
list-notifications List entries in the OS notification surface.
dismiss-notification Dismiss one notification by index.
dismiss-all-notifications Dismiss every notification, optionally for one app.
notification-action Press an action button on a notification.
Clipboard
clipboard-get Read text, an image, or file references from the clipboard.
clipboard-set Write text, an image, or file references to the clipboard.
clipboard-clear Clear the clipboard.
Wait
wait Block until a time, element, window, text, menu, event, or notification condition holds.
System
status Report adapter health, platform, permissions, and supported surfaces.
permissions Check, request, or guide the grant of OS permissions.
version Print the version, target architecture, and OS.
skills List and read the skills compiled into the binary.
session Start, end, list, and clean up sessions.
cursor-overlay Configure the presentation-only agent cursor.
trace Read merged session trace timelines.
Batch
batch Run a bounded, sequential, non-atomic list of commands in one process.
--- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Interaction commands description: Reference for click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, and collapse. url: "https://agent-browser.dev/desktop/commands/interaction" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Interaction" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Every command here acts on one element. Pass the qualified ref from [`snapshot`](/desktop/commands/observation#snapshot) or [`find`](/desktop/commands/observation#find), such as `@s8f3k2p9:e1`; a bare `@e1` is rejected with `INVALID_ARGS`. By default these commands run headless: semantic accessibility calls that do not take focus, move the cursor, synthesize keys, or use the clipboard. If no semantic path exists, the action fails closed. The global `--headed` flag permits physical input and focus changes; see [Headed mode](/desktop/guides/headed-mode). The examples on this page are real output from GNOME Calculator and GTK Widget Factory on Linux. ## Result shape Every successful ref action returns an action report in `data`. This is a `click` on a calculator button: ```json { "version": "2.6", "ok": true, "command": "click", "data": { "action": "click", "details": { "auto_wait": { "lease_contention_count": 0, "lease_hold_ms": 114, "read_only_preflight_attempts": 1, "read_only_resolve_attempts": 1 } }, "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "Action.DoAction", "mechanism": "semantic_api", "outcome": "succeeded" }], "surfaces": [ { "id": "w-1.11-org-gnome-Calculator-a11y-948fd89a_9bda_453b_9c10_4a9e9eed3feb", "title": "Calculator", "type": "window" } ] } } ```
Field Meaning
action The command that ran.
disposition delivery and retry. Repeat an action only when retry is safe. See Disposition.
steps The mechanisms tried: label, outcome (attempted, skipped, or succeeded), mechanism (semantic_api or physical_synthetic), and verified when the step read back its effect.
post_state The element's state after the action, for the commands that read it back.
surfaces Windows and overlays open once the action settled. Only click, double-click, triple-click, right-click, expand, select, and press report them. A menu, sheet, or alert means the app is waiting on it. See Surfaces.
details auto_wait counters, plus the verification fields below. Later examples on this page omit auto_wait.
When the target is already in the requested state, nothing is delivered and the report says `not_delivered`, which is safe to repeat. `set-value`, `clear`, `type`, `check`, `uncheck`, `toggle`, `expand`, and `collapse` run once and then re-read the element: - A match returns `delivered_verified` with `post_state` and `details.verification_scope`: `element_state` for `check`, `uncheck`, `toggle`, `expand`, and `collapse`, or `element_value` with `application_commit: "not_verified"` for `set-value`, `clear`, and `type`. A verified value proves the element holds it, not that the app committed it. - A readable contradiction returns `ACTION_FAILED` with `details.kind: "post_action_verification"`, the observed `post_state`, and the report under `details.after_action`. The action landed, so read `post_state` instead of retrying. - An unreadable result still succeeds, as `delivered_unverified` with `details.verification_scope: "unavailable"` and a `verification_reason`: `insufficient_evidence`, `secure_field`, or `postcondition_mismatch`. ## Wait budget Every command on this page takes `--timeout-ms MS` (default `5000`; `type` adds 30 ms per character), the budget for resolving the ref and waiting for the element to become actionable. Core polls the transient checks (`visible`, `stable`, `enabled`, and, for pointer delivery, `receives_events`) until they pass. The terminal checks (`supported_action`, `policy`, `editable`) cannot be fixed by waiting and fail at once. `--timeout-ms 0` tries once without polling. A failed check list appears in `error.details.checks`, each with a `status` of `pass`, `fail`, or `unknown` and a `reason`. This is `double-click` on a text entry without `--headed`: ```json { "code": "POLICY_DENIED", "message": "Target is not actionable: supported_action (semantic gesture is unavailable and physical fallback is denied)", "details": { "actionable": false, "checks": [ { "check": "visible", "status": "pass" }, { "check": "stable", "stability": { "samples": 1, "span_ms": 0 }, "status": "pass" }, { "check": "enabled", "status": "pass" }, { "check": "supported_action", "reason": "semantic gesture is unavailable and physical fallback is denied", "status": "fail" }, { "check": "policy", "status": "pass" } ] }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } ``` `receives_events` fails with an `occluder` (`role`, `name`, `bounds`) when another element covers the target. To wait for readiness on its own, use [`wait --element REF --predicate actionable`](/desktop/commands/wait#wait). ## Errors shared by ref commands
Code Cause
INVALID_ARGS A bare or malformed ref, or an argument the command rejects.
SNAPSHOT_NOT_FOUND The snapshot that owns the ref is missing or pruned.
STALE_REF No live element matches the ref's saved identity.
AMBIGUOUS_TARGET Several live elements match. A ref action never guesses.
ACTION_NOT_SUPPORTED The element does not expose the action.
POLICY_DENIED A physical or focus-taking path was needed and headless policy refused it.
TIMEOUT, details.kind: actionability_timeout The target never became actionable within --timeout-ms. details.last_report names the failing check.
ACTION_FAILED The action was rejected, blocked, or contradicted by the readback.
APP_UNRESPONSIVE The app stopped responding.
PERM_DENIED Accessibility permission is not granted.
What each code means and when to retry is in [Errors and recovery](/desktop/errors-and-recovery). ## click Activates an element with its primary action, such as a press or invoke. ```bash agent-desktop click [--timeout-ms MS] ``` The result is the one in [Result shape](#result-shape). A click usually ends `delivered_unverified`, because a delivered activation does not prove the app acted on it: verify by reading the UI, and check `data.surfaces`. Platform limits: semantic everywhere headless; `--headed` tries physical input first on macOS and Linux. See [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). ## double-click Double-clicks an element. It is a physical gesture, so it needs `--headed` unless the element exposes a semantic open action. ```bash agent-desktop --headed double-click [--timeout-ms MS] ``` Without `--headed` and a semantic route, it returns `POLICY_DENIED` (see [Wait budget](#wait-budget)). For raw coordinates use `--headed mouse-click --xy X,Y --count 2` ([`mouse-click`](/desktop/commands/mouse#mouse-click)). Platform limits: macOS tries `AXOpen` headless; Windows and Linux need `--headed`, and Linux needs X11. See [Linux](/desktop/platforms/linux). ## triple-click Triple-clicks an element, for example to select a line of text. Arguments and errors are the same as [`double-click`](#double-click). For raw coordinates use `--headed mouse-click --xy X,Y --count 3`. ```bash agent-desktop --headed triple-click [--timeout-ms MS] ``` ## right-click Opens an element's context menu. Afterward, `data.surfaces` lists the open `menu`; read it with `snapshot --surface menu`. For combo boxes and menu buttons, use [`select`](#select) instead. ```bash agent-desktop right-click [--timeout-ms MS] ``` Headless delivery needs a context-menu action on the element. On a GTK entry without one, it returns `POLICY_DENIED` with `supported_action` failing. On macOS, a right-click can return `APP_UNRESPONSIVE` with an unsafe disposition after a modal menu opened: look before you retry. Platform limits: Windows always needs `--headed`. See [Windows](/desktop/platforms/windows). ## type Inserts text into a text target at its insertion point, then reads the field back. ```bash agent-desktop type [--timeout-ms MS] [--interval-ms MS] agent-desktop type --stdin [--timeout-ms MS] [--interval-ms MS] ```
Flag or argument Default Meaning
TEXT required unless --stdin The text, up to 10,000 bytes. A value that starts with a hyphen is accepted.
--timeout-ms MS 5000 plus 30 per character Covers resolution, actionability, and delivery. An explicit value wins. With --interval-ms, it applies to each keystroke, and 0 makes each keystroke try once.
--interval-ms MS off Types one keystroke every MS milliseconds, 1 to 2000, instead of inserting the text at once.
Use `--stdin` with `type` or `set-value` to read text from a pipe or redirected file. Terminal input is rejected. Replace the example ref with one from your snapshot: ```bash printf '%s' 'hello' | agent-desktop type @s8f3k2p9:e2 --stdin ``` The action timeout starts after the input ends; it doesn't limit this wait. Empty input and all line breaks are kept, including a final newline. Use `printf '%s'` to avoid adding one. Input must be valid UTF-8 without NUL bytes (`\0`). The limits are 10,000 bytes for `type` and 128 KiB for `set-value`. Provide either `--stdin` or a command-line value. To enter the text `--stdin`, put it after `--`: `agent-desktop type @s8f3k2p9:e2 -- --stdin`. Batch uses the JSON `text` or `value` field, which puts the text in process arguments. Run secret steps as separate `--stdin` commands. `--stdin` keeps text out of process arguments and omits observed `value` fields from JSON results, including `post_state.value`. Other app text and screenshot artifacts aren't redacted. `type` inserts; it does not replace. After `set-value hello`, `type ' world'` into the same entry produced this, because the insertion point was at the start: ```json { "action": "type", "details": { "application_commit": "not_verified", "verification_scope": "element_value" }, "disposition": { "delivery": "delivered_verified", "retry": "unsafe" }, "post_state": { "enabled": true, "role": "textfield", "value": " worldhello" }, "steps": [{ "label": "EditableText.InsertText", "mechanism": "semantic_api", "outcome": "succeeded", "verified": true }] } ``` `--interval-ms` types at a steady pace inside one command. A keystroke is one character together with the combining marks, emoji modifiers, and joined characters drawn with it, and CR LF counts as one. The first keystroke runs as a full ref action, and a headed one also focuses the target's window. The rest go to the same element, each taking the interaction lease only while it is delivered, so other agents can act between keystrokes. A headed keystroke is posted only while the target's process and focused element are unchanged, so a burst that loses focus stops with an error. The result adds `keystrokes` and `interval_ms`, and its `post_state` reads the field after the last keystroke. It stays `delivered_verified` only when that read shows the whole text; otherwise it is `delivered_unverified` with `verification_reason` set to `postcondition_mismatch`, `insufficient_evidence` (the text went in part way through the field), or `final_read_failed` (with no `post_state`). A `--wait-for` condition is waited on once, after the last keystroke. A failure after the first keystroke carries `details.typed_keystrokes` and `details.keystrokes` with a `delivered_unverified` disposition, because part of the text landed. ```bash agent-desktop type @s8f3k2p9:e2 "hello world" --interval-ms 60 ``` To set a field's whole content, use [`set-value`](#set-value). Keystroke delivery that settles on other text returns `delivered_unverified` with `verification_reason: "postcondition_mismatch"`; read the field with `get` before you write again. Platform limits: Windows needs `--headed` and returns `POLICY_DENIED` headless; use `set-value` there. See [Windows](/desktop/platforms/windows). ## set-value Writes a value directly through the accessibility value path: text fields, text areas, and sliders. ```bash agent-desktop set-value [--timeout-ms MS] agent-desktop set-value --stdin [--timeout-ms MS] ``` For piped text, use `--stdin` with the [input rules above](#type). ```json { "action": "set-value", "details": { "application_commit": "not_verified", "verification_scope": "element_value" }, "disposition": { "delivery": "delivered_verified", "retry": "unsafe" }, "post_state": { "enabled": true, "role": "textfield", "value": "hello" }, "steps": [{ "label": "EditableText.SetTextContents", "mechanism": "semantic_api", "outcome": "succeeded", "verified": true }] } ``` On a slider, the step is `Value.CurrentValue` and `post_state.value` is the new position. A value the app reformats reads as a contradiction and returns `ACTION_FAILED`. An element that exposes no value write returns `ACTION_NOT_SUPPORTED`; for a table cell, activate it and target the editor that appears. A write to a secure field ends `delivered_unverified` with `verification_reason: "secure_field"`. ## clear Empties a text target and expects an empty value on readback. The result looks like `set-value` with `post_state.value: ""`. ```bash agent-desktop clear [--timeout-ms MS] ``` ## focus Sets keyboard focus on an element without clicking it. ```bash agent-desktop --headed focus [--timeout-ms MS] ``` Moving focus can change the foreground, so headless it returns `POLICY_DENIED` on Windows and Linux: ```json { "code": "POLICY_DENIED", "message": "Focusing an element moves the keyboard focus of the application", "details": { "foreground_effect": true }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } ``` On Linux, a GTK4 app rejects the call with cause `grab_focus_rejected`; use [`focus-window`](/desktop/commands/app-window#focus-window) for its window. ## select Picks an option by display text in a list, dropdown, or combo box. ```bash agent-desktop select [--timeout-ms MS] ``` ```bash agent-desktop select @s30sxlstqzdq8x:e21 Middle ``` ```json { "action": "select", "details": {}, "disposition": { "delivery": "delivered_verified", "retry": "unsafe" }, "steps": [{ "label": "Action.DoAction", "mechanism": "semantic_api", "outcome": "succeeded", "verified": true }], "surfaces": [{ "id": "w-1.9-org-a11y-atspi-accessible-229", "type": "window" }] } ``` If the control already shows the value, nothing is delivered. A combo box whose accessible name is its current value gets a new identity when the value changes, so the old ref returns `STALE_REF` afterward: find the control again before the next action. Platform limits: on macOS, dismiss any open menu first. On Linux, a Qt combo refuses headless with `details.kind: pointer_required`, and a GTK4 list refuses with `selection_index_unproven`. See the platform pages. ## toggle Flips a checkbox or switch. Prefer [`check`](#check) and [`uncheck`](#uncheck), which are idempotent; use `toggle` only when you want the opposite of whatever the state is. ```bash agent-desktop toggle [--timeout-ms MS] ``` The readback requires the checked state to have changed, and `post_state.states` shows the result. An unreadable or indeterminate state returns `ACTION_FAILED`, and a role that cannot be checked returns `ACTION_NOT_SUPPORTED`. ## check Sets a checkbox or switch to checked. An element that is already checked delivers nothing. ```bash agent-desktop check [--timeout-ms MS] ``` ```json { "action": "check", "details": { "verification_scope": "element_state" }, "disposition": { "delivery": "delivered_verified", "retry": "unsafe" }, "post_state": { "enabled": true, "role": "checkbox", "states": ["checked"] }, "steps": [{ "label": "Action.DoAction", "mechanism": "semantic_api", "outcome": "succeeded", "verified": true }] } ``` Run it again on the same checkbox and the step is skipped: ```json { "action": "check", "details": { "verification_scope": "element_state" }, "disposition": { "delivery": "not_delivered", "retry": "safe" }, "post_state": { "enabled": true, "role": "checkbox", "states": ["checked"] }, "steps": [{ "label": "Action.DoAction", "mechanism": "semantic_api", "outcome": "skipped", "verified": true }] } ``` Errors are the same as [`toggle`](#toggle). On Linux, two clicks less than about 250 ms apart on one GTK4 button can act once. ## uncheck Sets a checkbox or switch to unchecked, idempotently. The result matches `check`, with `checked` absent from `post_state.states`. ```bash agent-desktop uncheck [--timeout-ms MS] ``` ## expand Expands a disclosure triangle, tree item, or accordion, and verifies that the element reads as expanded (`post_state.states` contains `expanded`). `data.surfaces` can list a menu or popover the expansion opened. A non-expandable element returns `ACTION_NOT_SUPPORTED`. ```bash agent-desktop expand [--timeout-ms MS] ``` ## collapse Collapses a disclosure triangle, tree item, or accordion, and verifies that `expanded` is gone. Errors are the same as [`expand`](#expand). ```bash agent-desktop collapse [--timeout-ms MS] ``` --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Keyboard commands description: Reference for press, key-down, and key-up. press sends key combos; the held-input commands fail closed on every platform. url: "https://agent-browser.dev/desktop/commands/keyboard" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Keyboard" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). `press` is the only command that delivers keys. `key-down` and `key-up` are reserved names: they validate their argument and then refuse. To insert text into a field, use [`type`](/desktop/commands/interaction#type) or [`set-value`](/desktop/commands/interaction#set-value). ## press Sends a key combo to the focused window, or to one app. ```bash agent-desktop press COMBO [--app NAME] [--window-id ID] [--force] agent-desktop press ctrl+shift+z agent-desktop press return --app gnome-calculator ```
Flag or argument Default Meaning
COMBO required Keys joined with +. The last part is the key; the parts before it are modifiers.
--app NAME focused window Send the keys to this application. With --headed, its window is focused first. Several instances with the same name return AMBIGUOUS_TARGET.
--window-id ID none Target the instance that owns this window.
--force off Send a combo the platform blocks as dangerous, such as a quit or lock shortcut.
The combo is lowercased and stripped of spaces, so `Ctrl + A` is `ctrl+a`. Modifiers are `meta` (aliases `cmd`, `command`, `super`, `win`: Command on macOS, the Windows key on Windows, Super on Linux), `ctrl` (`control`), `alt` (`option`), and `shift`. A bare uppercase letter adds Shift. Key names that work everywhere are `return`, `escape`, `tab`, `space`, `delete`, `up`, `down`, `left`, `right`, and `f1` to `f12`. Copy is `cmd+c` on macOS and `ctrl+c` elsewhere. Keys are physical keyboard input, and `press` does not need `--headed` for an unscoped press. The result is always `delivered_unverified`, because delivery does not prove the shortcut did anything: ```json { "version": "2.6", "ok": true, "command": "press", "data": { "action": "press", "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "XTEST.press_key", "mechanism": "physical_synthetic", "outcome": "succeeded", "verified": false }] } } ``` The step label names the platform mechanism; this one is Linux X11. With [post-action wait flags](/desktop/configuration#post-action-wait-flags), the press report moves under `data.after_action`. Errors specific to `press`: - `INVALID_ARGS`: an empty combo, an unknown modifier, or an unknown key (`Unknown key name 'foo'`). Nothing is sent. - `POLICY_DENIED`: the platform blocks the combo. The disposition is `not_delivered`. Pass `--force` only when you mean it. - `ACTION_FAILED` with `details.cause: "frame_not_active"`: a headless scoped press to an app whose window is not active. Nothing is sent. Bring the window forward, or pass `--headed` if focus changes are permitted. - `TIMEOUT` with `details.kind: "wait_timeout"`: a `--wait-for` selector never matched. The keys were already delivered, so do not repeat the press blindly. Platform limits: each platform has its own key names, blocked combos, and scoped-press rules. See [macOS](/desktop/platforms/macos#keys-and-notifications), [Windows](/desktop/platforms/windows#keys), and [Linux](/desktop/platforms/linux). ## key-down Reserved name for holding a key down. It never sends input. ```bash agent-desktop key-down COMBO [--force] ``` A held key needs a process that can guarantee its release, and every agent-desktop command is a short-lived process. So on every platform, headless or headed, the command parses the combo (`INVALID_ARGS` if bad), applies the blocked-combo check (`POLICY_DENIED` unless `--force`), and then refuses: ```json { "version": "2.6", "ok": false, "command": "key-down", "error": { "code": "ACTION_NOT_SUPPORTED", "message": "key-down is unavailable in stateless mode", "suggestion": "Use the atomic press command; held input requires a daemon-owned transaction that can guarantee release", "disposition": { "delivery": "unknown", "retry": "unknown" } } } ``` Nothing was sent, and the command fails the same way every time. Send the whole chord with [`press`](#press), for example `press shift+tab`. ## key-up Reserved name for releasing a held key. It follows the same steps as [`key-down`](#key-down) and ends with `ACTION_NOT_SUPPORTED` (`key-up is unavailable in stateless mode`). ```bash agent-desktop key-up COMBO [--force] ``` --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Mouse commands description: Reference for hover, drag, mouse-move, mouse-click, mouse-down, mouse-up, and mouse-wheel. Cursor movement needs --headed, and the held-button commands fail closed. url: "https://agent-browser.dev/desktop/commands/mouse" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Mouse" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Every mouse command moves the real cursor, so each one needs the global `--headed` flag. Without it the command returns `POLICY_DENIED` with `not_delivered` and sends nothing: ```json { "code": "POLICY_DENIED", "message": "hover moves the cursor and is disabled in headless mode", "recovery": { "strategy": "request_explicit_policy_then_retry_original", "retryable": true, "requires_fresh_snapshot": false }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } ``` Read [Headed mode](/desktop/guides/headed-mode) before you add the flag. Coordinates are absolute screen points written `x,y`; write a negative value with `=` so the shell does not read it as a flag (`--xy=-100,50`). Raw coordinates carry no window identity and never focus an app; a ref target focuses its exact window first. The pointer commands return an [OperationReport](/desktop/json-output#operationreport) with a `pointer` target. The examples are real output from Linux X11. ## hover Moves the cursor to an element's center or to a point. ```bash agent-desktop --headed hover @s8f3k2p9:e5 agent-desktop --headed hover --xy 640,400 ```
Flag or argument Default Meaning
REF none Qualified ref. Wins over --xy.
--xy X,Y none Absolute coordinates. One of REF or --xy is required.
--duration MS none Reserved. A positive value is rejected; run hover, then wait MS.
--timeout-ms MS 5000 Budget for the ref to resolve and become actionable.
```json { "version": "2.6", "ok": true, "command": "hover", "data": { "action": "mouse-move", "details": { "delivered_events": 1 }, "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "pointer_move", "mechanism": "physical_synthetic", "outcome": "succeeded" }], "target": { "kind": "pointer", "point": { "x": 640.0, "y": 400.0 } }, "x": 640.0, "y": 400.0 } } ``` For a ref, the cursor goes to the center of the live bounds after visibility, stability, and hit-test checks. On macOS the position is read back and the result is `delivered_verified`. Errors: `INVALID_ARGS` for a missing target or bad coordinates; `ACTION_NOT_SUPPORTED` with cause `wayland_session` on Linux Wayland; `ACTION_FAILED` when another element covers the target; the ref errors `STALE_REF`, `AMBIGUOUS_TARGET`, and `SNAPSHOT_NOT_FOUND`. ## drag Presses at one point, moves to another, and releases. ```bash agent-desktop --headed drag --from @s8f3k2p9:e1 --to @s8f3k2p9:e5 agent-desktop --headed drag --from-xy 100,200 --to-xy 400,500 --drop-delay 800 ```
Flag Default Meaning
--from REF, --from-xy X,Y required, one of Source element or point.
--to REF, --to-xy X,Y required, one of Destination element or point. The two kinds can be mixed.
--duration MS platform default Length of the movement, at most 60000.
--drop-delay MS 500 (150 on Linux) Hold over the destination before releasing, so the drop target activates. At most 10000.
--timeout-ms MS 5000 Budget for the whole gesture. --duration plus --drop-delay must be below it.
--wait-for-scope to|from to Which endpoint's window a post-action --wait-for polls.
The destination app is never focused first, because raising it could cover the source. The report describes the gesture (`target.kind: "drag"` with `from` and `to`), not its outcome, so verify the moved item with a fresh snapshot. Native cross-app drag and drop can ignore synthetic events. Errors: `INVALID_ARGS` for a missing endpoint or timing that does not fit `--timeout-ms`, `TIMEOUT` when preparation used up the budget, and the errors listed under [`hover`](#hover). ## mouse-move Moves the cursor to absolute coordinates. ```bash agent-desktop --headed mouse-move --xy 700,450 ``` The report matches the [`hover`](#hover) example. On macOS the position is read back (`details.verification_scope: "pointer_position"`). ## mouse-click Clicks at absolute coordinates. ```bash agent-desktop --headed mouse-click --xy 500,300 [--button left|right|middle] [--count N] [--modifiers NAME]... ```
Flag Default Meaning
--xy X,Y required Absolute coordinates.
--button left left, right, or middle.
--count N 1 Clicks, 1 to 100. Use 2 for a double click and 3 for a triple click.
--modifiers NAME none A modifier held during the click: shift, meta, ctrl, alt. Repeat the flag for several.
```json { "version": "2.6", "ok": true, "command": "mouse-click", "data": { "action": "mouse-click", "count": 1, "details": { "delivered_events": 3 }, "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "steps": [{ "label": "pointer_click", "mechanism": "physical_synthetic", "outcome": "succeeded" }], "target": { "kind": "pointer", "point": { "x": 1300.0, "y": 880.0 } }, "x": 1300.0, "y": 880.0 } } ``` `target.point` is where the click was dispatched; it does not prove which app received it. Before you click at bounds read from a snapshot, check `get --property bounds` and its `live` flag. ## mouse-down Reserved name for pressing and holding a button. It never sends input. ```bash agent-desktop --headed mouse-down --xy 500,300 [--button NAME] [--modifiers NAME]... ``` The first failing check decides the error: a bad argument is `INVALID_ARGS`; headless is `POLICY_DENIED` (`not_delivered`); Linux Wayland is `ACTION_NOT_SUPPORTED` with cause `wayland_session`; everywhere else, `--headed` ends in `ACTION_NOT_SUPPORTED`, `mouse-down is unavailable in stateless mode`, with a disposition of `unknown` and no event sent. Use [`mouse-click`](#mouse-click) or [`drag`](#drag). ## mouse-up Reserved name for releasing a held button. It fails closed through the same checks as [`mouse-down`](#mouse-down). ```bash agent-desktop --headed mouse-up --xy 500,300 ``` ## mouse-wheel Scrolls the mouse wheel at absolute coordinates. Use it for canvases and custom surfaces with no accessibility scroll action; for a scroll area with a ref, use [`scroll`](/desktop/commands/scroll#scroll). ```bash agent-desktop --headed mouse-wheel --x 700 --y 400 --dy -2 ```
Flag Default Meaning
--x X, --y Y required Absolute coordinates.
--dy LINES -3 when --dx is 0, else 0 Vertical lines. Positive scrolls up, negative scrolls down.
--dx LINES 0 Horizontal lines. Positive scrolls right.
--modifiers NAME none Modifiers held during the scroll; repeatable.
```json { "version": "2.6", "ok": true, "command": "mouse-wheel", "data": { "action": "mouse-wheel", "details": { "delivered_events": 5 }, "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }, "dx": 0.0, "dy": -2.0, "steps": [{ "label": "pointer_wheel", "mechanism": "physical_synthetic", "outcome": "succeeded" }], "target": { "kind": "pointer", "point": { "x": 700.0, "y": 400.0 } } } } ``` Platform limits for all mouse commands: Linux needs X11. See [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Notification commands description: Reference for list-notifications, dismiss-notification, dismiss-all-notifications, and notification-action on the operating system notification surface. url: "https://agent-browser.dev/desktop/commands/notifications" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Notifications" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). These commands drive the OS notification surface: Notification Center on macOS, Action Center on Windows, and the GNOME Shell message list on Linux. Command names, arguments, and JSON fields are the same everywhere. Titles and bodies are returned verbatim and are not redacted, so treat them as sensitive. - **Listing** is read-only. Linux lists without `--headed`, even when the list is closed. On macOS and Windows a headless call works only while the surface is already open; otherwise it returns `POLICY_DENIED`, and `--headed` may open the surface. - **Mutations** (`dismiss-notification`, `dismiss-all-notifications`, `notification-action`) always need `--headed`, because they open and focus the notification surface. - **Identity guard.** Notifications reorder between listings, so `dismiss-notification` and `notification-action` need `--expected-app` or `--expected-title` copied from the same `list-notifications` result. If the row at `INDEX` no longer matches, nothing is pressed and the command returns `NOTIFICATION_NOT_FOUND`. To block until a new notification arrives, use [`wait --notification`](/desktop/commands/wait#notification). The JSON examples on this page are built from the command code and tests; notifications could not be exercised in the Linux session the other examples came from, which had no notification service: ```json { "code": "PLATFORM_NOT_SUPPORTED", "message": "No notification service is running in this session", "details": { "daemon": "absent", "detail": "no service owns org.freedesktop.Notifications" } } ``` ## list-notifications Lists the notifications on the OS surface. ```bash agent-desktop list-notifications [--app NAME] [--text TEXT] [--limit N] ```
Flag Default Meaning
--app NAME all apps Only notifications from this app, ignoring case.
--text TEXT none Only notifications that contain this text.
--limit N no limit Maximum number returned.
```json { "version": "2.6", "ok": true, "command": "list-notifications", "data": { "count": 1, "notifications": [ { "index": 1, "app_name": "Calendar", "title": "Standup in 10 minutes", "body": "Room 4", "actions": ["Snooze", "Open"] } ] } } ``` `index` is 1-based and counts the unfiltered list, so a filtered result can start above 1. `body` and `actions` are omitted when empty. Errors: `POLICY_DENIED` on macOS or Windows with the surface closed, and `PLATFORM_NOT_SUPPORTED` when there is no notification service (on Linux, an owner other than GNOME Shell). ## dismiss-notification Dismisses one notification by index. ```bash agent-desktop --headed dismiss-notification INDEX --expected-app NAME agent-desktop --headed dismiss-notification 2 --expected-title "Standup in 10 minutes" ```
Flag or argument Default Meaning
INDEX required 1-based index from list-notifications.
--expected-app APP, --expected-title TITLE at least one The app or title the notification at INDEX must have.
--app NAME none Require the notification to come from this app.
`data.dismissed` is the removed notification, in the `list-notifications` row shape. Errors: `INVALID_ARGS` without an `--expected-*` flag, `POLICY_DENIED` without `--headed`, and `NOTIFICATION_NOT_FOUND` when the row no longer matches. ## dismiss-all-notifications Dismisses every notification, or every one from one app. ```bash agent-desktop --headed dismiss-all-notifications [--app NAME] ``` `data.dismissed_count` counts the removed notifications. When some could not be closed, `data.failures` and `data.failed_count` are added. ## notification-action Presses a named action button on a notification. ```bash agent-desktop --headed notification-action INDEX ACTION --expected-app NAME ``` `ACTION` is a name from the notification's `actions`. The result is an action report, usually `delivered_unverified`: check the effect in the app. Errors: as for `dismiss-notification`, plus `ACTION_NOT_SUPPORTED` when the notification has no button with that name. Platform limits: GNOME Shell only on Linux, with mutations on X11; see [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Observation commands description: Reference for snapshot, find, get, is, screenshot, list-displays, list-surfaces, and open-system-surface. url: "https://agent-browser.dev/desktop/commands/observation" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Observation" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Observation commands read UI state. All of them except `open-system-surface` leave the UI as it was, and their errors carry `not_delivered` because nothing was sent to the app. Concepts behind the flags are explained in [Skeleton and drill-down](/desktop/skeleton-and-drill-down) (`--skeleton`, `--root`, `--max-depth`, partial observations, timeouts, scoped search), [Snapshots and refs](/desktop/snapshots-and-refs) (refs, `-i`, `--compact`, `--include-bounds`, window selection), and [Surfaces](/desktop/surfaces). ## snapshot Captures an application's accessibility tree as JSON. Window-owned surfaces are read from a window; on macOS, `menu` and `menubar` are application-owned and need no document window. Interactive elements get qualified refs that later commands accept. ```bash agent-desktop snapshot [--app NAME] [--window-id ID] [-i] [--compact] [--skeleton] \ [--root REF] [--surface KIND] [--max-depth N] [--include-bounds] \ [--timeout-ms MS] [--force-electron-a11y] ```
Flag Default Meaning
--app NAME focused app or window Application to read. The selected surface determines whether a window is required. See Choose the window.
--window-id ID none Read one window by id from list-windows. For macOS menu and menubar, it selects the owning app.
-i, --interactive-only off Drop leaves without a ref. See What a snapshot contains.
--compact off Collapse single-child unnamed wrappers.
--include-bounds off Add bounds to each node.
--skeleton off Clamp depth to 3 and mark cut containers with children_count. See Take a skeleton.
--root REF none Drill into one ref's subtree and replace that region's refs. Cannot combine with --surface or --wait-for. See Drill into a region.
--max-depth N 10 Maximum depth, up to 50. See Depth with max-depth.
--timeout-ms MS 3000 Observation deadline. A full snapshot that runs out succeeds with complete: false. See Partial observations.
--surface KIND window Read the selected UI surface. On macOS, menu and menubar are application-owned. See Surface kinds.
--force-electron-a11y off Assume Chromium renderer accessibility is already forced, which skips the activation guidance.
`data` carries `app`, `snapshot_id`, `ref_count`, `complete`, and `tree`. Window-owned snapshots also carry `window`; macOS application-owned `menu` and `menubar` snapshots, and process-owned snapshots of a background app with no window, omit it. [Snapshots and refs](/desktop/snapshots-and-refs) shows the window-owned shape. With a post-action `--wait-for` ([Configuration](/desktop/configuration#post-action-wait-flags)), a successful snapshot adds `elapsed_ms` and `matched_selector`. Errors specific to `snapshot`: `INVALID_ARGS` for `--root` with `--surface` or `--wait-for`, a bare ref, `--max-depth` above 50, or an unsupported `--surface` when the platform supports other surfaces; `PLATFORM_NOT_SUPPORTED` when the adapter supports no surfaces; `WINDOW_NOT_FOUND` when a window-owned surface has no window; `APP_NOT_FOUND` when an application-owned surface has no matching app; `AMBIGUOUS_TARGET` when a window-owned lookup has several candidate windows; `ELEMENT_NOT_FOUND` when a macOS application-owned menu surface is absent; `STALE_REF`, `SNAPSHOT_NOT_FOUND`, or `TIMEOUT` for a `--root` drill. See [What a drill refuses](/desktop/skeleton-and-drill-down#what-a-drill-refuses). Platform limits: supported surfaces and Chromium settle time differ by OS. See [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). ## find Searches by role, name, value, text, description, native id, or state, and returns refs you can act on. Use it when you know the target. ```bash agent-desktop find [--app NAME] [--window-id ID] [--role ROLE] [--name TEXT] [--value TEXT] \ [--text TEXT] [--description TEXT] [--native-id ID] [--exact] \ [--state TOKEN[=BOOL]]... [--root REF] [--surface KIND] [--timeout-ms MS] \ [--count | --first | --last | --nth N | --limit N] ```
Flag Default Meaning
--app NAME, --window-id ID focused app or window Which owner to search, with the same rules as snapshot.
--role ROLE none Accessibility role, case-insensitive. textarea, textbox, and searchfield fold to textfield.
--name TEXT none Accessible name. Fuzzy unless --exact.
--value TEXT none Current value. Unnamed elements carry their content here.
--text TEXT none Text in the name, value, or description. Also matches ancestors whose subtree contains it; use --name to match the node itself.
--description TEXT none Accessible description.
--native-id ID none Native automation id, such as an AXIdentifier or AutomationId.
--exact off Case-insensitive equality instead of a fuzzy match.
--state TOKEN\[=BOOL] none Filter by a state, repeatable. A bare token means true. Tokens: focused, disabled, secure, expanded, checked, selected, hidden, busy, modal, required, indeterminate, pressed, readonly, offscreen, invalid, multiselectable, haspopup.
--root REF none Search one ref's subtree. See Scope a search.
--surface KIND window Search an overlay. Cannot combine with --root.
--timeout-ms MS 5000 Traversal deadline.
--count off Return the match count only.
--first, --last, --nth N off Return one match: the first, the last, or zero-based index N.
--limit N 50 Return at most N matches. 0 returns all. Use --limit 2 before a mutation to check the match is unique.
`--count`, `--first`, `--last`, `--nth`, and `--limit` are mutually exclusive. The result shapes and match fields are in [JSON output](/desktop/json-output#find-results). ```bash agent-desktop find --app gnome-calculator --role button --name 7 --exact --first ``` ```json { "version": "2.6", "ok": true, "command": "find", "data": { "match": { "interactive": true, "name": "7", "path": ["window:Calculator", "group", "group", "group", "group", "group", "group", "group", "group", "group", "group", "group"], "ref_id": "@s3jjre8xzrvo2l:e1", "role": "button", "states": [], "value": null }, "snapshot_id": "s3jjre8xzrvo2l" } } ``` When a `--role` filter matches nothing, `roles_present` lists the roles in the searched tree, which tells a wrong role name from an empty screen. Errors specific to `find`: `INVALID_ARGS` for conflicting shaping flags, `--root` with `--surface`, or an unknown `--state` token; `TIMEOUT` with `details.kind: "locator_incomplete"` when the traversal ran out of budget, which does not mean the app hangs. Raise `--timeout-ms` or narrow the search. ## get Reads one property of a ref. ```bash agent-desktop get [--property text|value|title|bounds|role|states] ```
Flag or argument Default Meaning
REF required Qualified ref.
--property text text is what a person reads: the value for textfield, combobox, listbox, datefield, and timefield, and the name for every other role, each falling back to the other. value is the raw value. title is the accessible name. bounds, role, and states are as named.
```json { "version": "2.6", "ok": true, "command": "get", "data": { "property": "value", "ref": "@s30sxlstqzdq8x:e19", "value": "entry" } } ``` `text`, `value`, `bounds`, and `states` read live state when the adapter can and fall back to the snapshot's saved values otherwise; a missing value is `null`. Only `bounds` reports whether it was live, in `data.live`. When `live` is `false`, the rectangle came from the snapshot and may be out of date, so check it before you pass bounds to `mouse-click`. Errors specific to `get`: `INVALID_ARGS` for a bare ref or an unknown property, and the ref errors `SNAPSHOT_NOT_FOUND`, `STALE_REF`, and `AMBIGUOUS_TARGET`. See [Snapshots and refs](/desktop/snapshots-and-refs#how-a-ref-resolves). ## is Checks one boolean state of a ref. ```bash agent-desktop is [--property visible|enabled|checked|focused|expanded|selected] ```
Flag or argument Default Meaning
REF required Qualified ref.
--property visible The state to test.
```bash agent-desktop is @s30sxlstqzdq8x:e45 --property enabled ``` ```json { "version": "2.6", "ok": true, "command": "is", "data": { "applicable": true, "property": "enabled", "ref": "@s30sxlstqzdq8x:e45", "result": false } } ``` `applicable: false` means the property does not apply or could not be read, so `result: false` then means unknown, not disabled or unchecked. Errors are the same as [`get`](#get). ## screenshot Captures an app window or a display as a PNG. A screenshot is evidence for you, not proof that an action worked. ```bash agent-desktop screenshot [PATH] [--app NAME] [--window-id ID] [--screen N] ```
Flag or argument Default Meaning
PATH none Save the image to this file. Without it, the image returns base64-encoded in data.data.
--app NAME none Capture this app's window.
--window-id ID none Capture one window by id.
--screen N none Capture the display at index N of list-displays, where 0 is the primary. Cannot combine with --app or --window-id.
```bash agent-desktop screenshot --screen 0 screen.png ``` ```json { "version": "2.6", "ok": true, "command": "screenshot", "data": { "format": "png", "height": 900, "path": "/tmp/screen.png", "scale_factor": 1.0, "width": 1440 } } ``` Errors specific to `screenshot`: `PERM_DENIED` without screen-capture permission; `INVALID_ARGS` for `--screen` with a window selector, an out-of-range index, or an unwritable `PATH`; `WINDOW_NOT_FOUND` or `APP_NOT_FOUND` for an unknown target. On Linux, a covered window on a non-compositing window manager fails with `ACTION_FAILED` and `details.reason: "covered"`, and a display capture under a compositing one returns `PLATFORM_NOT_SUPPORTED`. Platform limits: macOS needs Screen Recording permission, and Wayland asks for portal consent. See [macOS](/desktop/platforms/macos) and [Linux](/desktop/platforms/linux). ## list-displays Lists connected displays, primary first. It takes no flags. ```bash agent-desktop list-displays ``` ```json { "version": "2.6", "ok": true, "command": "list-displays", "data": [ { "bounds": { "height": 900.0, "width": 1440.0, "x": 0.0, "y": 0.0 }, "id": "screen", "is_primary": true, "scale": 1.0 } ] } ``` Use the array index, not `id`, with `screenshot --screen`. ## list-surfaces Lists the overlays an app presents now: the values `snapshot --surface` and `find --surface` accept for it. ```bash agent-desktop list-surfaces [--app NAME] ```
Flag Default Meaning
--app NAME focused app Application to inspect.
`data` is `pid` and `surfaces`, each with `id`, `type`, and, when known, `title`, `item_count`, and `unclassified`. [Surfaces](/desktop/surfaces#list-what-is-open) shows an example and explains `unclassified`. Errors: `APP_NOT_FOUND`, `AMBIGUOUS_TARGET` when several instances match, and `TIMEOUT`. ## open-system-surface Raises an OS shell surface, such as the Start menu or Action Center, and returns the window it presents. A surface that is already present returns without being raised again. ```bash agent-desktop --headed open-system-surface --surface KIND ```
Flag Default Meaning
--surface KIND required The shell surface: start-menu, taskbar, system-tray, system-tray-overflow, action-center, dock, or menu-bar-extras. dock and menu-bar-extras are the macOS kinds.
`data` is `surface` and, on Windows and Linux, `window`, with the same fields as a [`list-windows`](/desktop/commands/app-window#list-windows) entry. On macOS no window owns the surface, so `data` is only `{"surface": "dock"}` and `window` is left out. This shape comes from the command's unit test: ```json { "version": "2.6", "ok": true, "command": "open-system-surface", "data": { "surface": "action-center", "window": { "app_name": "ShellExperienceHost.exe", "id": "w-4242", "is_focused": true, "pid": 99, "title": "Action center" } } } ``` The command takes the foreground, so on Windows and Linux headless mode refuses it with `POLICY_DENIED` before anything is raised; pass `--headed`. A kind the OS does not expose returns `INVALID_ARGS`, with the kinds to use in `suggestion` and the alternative in `platform_detail` where one exists. On macOS, `dock` and `menu-bar-extras` are always on screen, so the command returns them at once, headless, with no `--headed`; open Control Center or Notification Center by clicking its `menu-bar-extras` item. Which kinds open on Windows and Linux is on their platform pages. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Scroll commands description: Reference for scroll and scroll-to, which move a scroll area or bring an element into view. url: "https://agent-browser.dev/desktop/commands/scroll" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Scroll" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Both commands act on a qualified ref and share the [result shape](/desktop/commands/interaction#result-shape), [wait budget](/desktop/commands/interaction#wait-budget), and [shared errors](/desktop/commands/interaction#errors-shared-by-ref-commands) of the interaction commands. `scrollarea` elements get refs so that `scroll` can target them. ## scroll Scrolls a scroll area in one direction by a number of units. ```bash agent-desktop scroll [--direction up|down|left|right] [--amount N] [--timeout-ms MS] ```
Flag or argument Default Meaning
REF required Qualified ref of the scroll area.
--direction down up, down, left, or right.
--amount N 3 Number of scroll units.
--timeout-ms MS 5000 Wait budget.
macOS and Windows scroll through native accessibility paths headless. On Linux, `scroll` is a physical wheel event, so headless it refuses before sending anything: ```json { "version": "2.6", "ok": false, "command": "scroll", "error": { "code": "POLICY_DENIED", "message": "Physical pointer input is disabled by the current interaction policy", "recovery": { "strategy": "request_explicit_policy_then_retry_original", "retryable": true, "requires_fresh_snapshot": false }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` A delivered scroll is usually `delivered_unverified`. Take a fresh snapshot before you act on elements that were offscreen, because scrolling changes positions and a repeated scroll moves the view again. For a canvas or custom surface with no scroll action, use [`mouse-wheel`](/desktop/commands/mouse#mouse-wheel). Platform limits: Linux needs `--headed` and X11. See [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). ## scroll-to Brings an element into the visible area. Success means a re-read found the element visible. ```bash agent-desktop scroll-to [--timeout-ms MS] ``` Most ref actions already scroll their target into view first, so you need `scroll-to` mainly to reveal an element before you read it or act on a neighbor. An element without a ref cannot be targeted: scroll its container with `scroll`. When the element's bounds cannot be read, the visibility check never passes and the command ends with `TIMEOUT` (`details.kind: "actionability_timeout"`, a `stable` check reading `live bounds unavailable`). On Windows, an element with no scrollable ancestor fails with `ACTION_FAILED` and `details.kind: "scroll_into_view_unsupported"`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: System commands description: Reference for status, permissions, version, skills, session, cursor-overlay, and trace. url: "https://agent-browser.dev/desktop/commands/system" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "System" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). These commands report adapter and OS state, manage sessions and their traces, and configure the cursor overlay. None of them acts on an application. The examples are real output from Linux. ## status Shows adapter health, platform, permissions, and the active snapshot and session. It takes no flags. ```bash agent-desktop status ```
Field Meaning
platform, version The OS and the agent-desktop version.
permissions The object permissions returns.
snapshot_id, ref_count The latest snapshot in the selected namespace, or null.
session_id, tracing The active session, or null, and whether a trace writer is attached.
artifacts events or full, for an active session only.
supported_surfaces Surface kinds this adapter can read. Empty means the accessibility service did not answer. See Surfaces.
state_root The resolved state root.
[Installation](/desktop/installation#verify-the-install) shows a full result. A missing accessibility service does not fail `status`; it shows as a denied permission and an empty `supported_surfaces`. An invalid `AGENT_DESKTOP_HOME` or `AGENT_DESKTOP_SESSION` returns `INVALID_ARGS`. ## permissions Reports the OS permission states agent-desktop needs. ```bash agent-desktop permissions [--request] [--guide] ```
Flag Default Meaning
--request off Ask the OS to prompt for missing permissions, through a bounded helper so a stalled prompt cannot hang the command. This is the only path that prompts.
--guide off macOS only. Open System Settings with an on-screen guide and return without waiting. data.guide is started, already_running, or not_needed.
```json { "version": "2.6", "ok": true, "command": "permissions", "data": { "accessibility": { "state": "granted" }, "automation": { "state": "not_required" }, "screen_recording": { "state": "not_required" } } } ``` Each state is `granted`, `denied` (with a `suggestion`), `not_required`, or `unknown`. On Windows and Linux, `--guide` returns `PLATFORM_NOT_SUPPORTED`. ## version Prints the version, CPU architecture, and OS. It takes no flags. ```json { "version": "2.6", "ok": true, "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" } } ``` The envelope's `version` is the JSON contract version; `data.version` is the release. ## skills Prints the guides compiled into the binary. Usage, names, and references are on [Skills](/desktop/skills). ```bash agent-desktop skills [list] agent-desktop skills get NAME [REFERENCE] [--full] agent-desktop skills path ``` ## session Creates, lists, ends, and cleans up sessions. What a session is and how commands select one are on [Sessions](/desktop/sessions). ```bash agent-desktop session start [--name LABEL] [--no-trace] [--screenshots] [--cursor [--multi-agent]] agent-desktop session end [ID] agent-desktop session list agent-desktop session gc [--older-than SECONDS] [--ended] ```
Subcommand and flag Default Meaning
start --name LABEL none A label of 1 to 128 bytes.
start --no-trace off Create the session without tracing.
start --screenshots off Capture pre-action and post-action screenshots and refmap copies. Unredacted. Requires tracing.
start --cursor off Enable the cursor overlay with its default style.
start --multi-agent off Give each harness subagent its own cursor. Requires --cursor; desktop actions then need an agent id.
end \[ID] the active session Seal the manifest; running it again on an ended session succeeds. Also stops every cursor overlay in the session.
gc --older-than SECONDS none Remove only sessions older than this; without --ended, sessions that never ended qualify once stale.
gc --ended off Remove ended sessions even if they wrote in the last five minutes.
`start` returns `session_id`, `name`, `trace`, `artifacts`, `created_at`, `next` (the shell line that exports the id), and `activation`; [Sessions](/desktop/sessions#start-list-end-and-clean-up) shows a full result. `end` returns `session_id` and `ended_at`, `list` returns `data.sessions`, and `gc` returns `data.removed`. `--screenshots` with `--no-trace` is `INVALID_ARGS`, and `--multi-agent` without `--cursor` is a parse error. ## cursor-overlay Configures the presentation-only cursor overlay for a session. The overlay draws an agent cursor, a click ripple, and an element highlight, and never changes what an action does. On macOS and Windows, after `type` or `set-value` lands, it also draws a caret at the field's insertion point, and moves it after a `press` that moves the insertion point. It needs an active session, and it cannot run inside a [batch](/desktop/commands/batch). ```bash agent-desktop --session "$AGENT_DESKTOP_SESSION" cursor-overlay enable [STYLE FLAGS] agent-desktop --session "$AGENT_DESKTOP_SESSION" cursor-overlay disable ```
enable flag Default Meaning
--multi-agent off Require an agent id for desktop actions. Rejected together with --agent-id.
--label TEXT, --max-words N none, 6 Intent text beside the cursor, limited to 1 to 12 words, drawn as a black pill sized to the text.
--accent HEX #4299FF Cursor body, ripple, element outline and caret color.
--fill HEX, --rim HEX the accent, #111318 Cursor body and outline colors. --fill sets a body color other than the accent.
--size N 1.0 Size multiplier, 0.5 to 4.0.
--no-ripple, --no-highlight off Turn off the click ripple or the element outline.
--image PATH, --hotspot X,Y built-in arrow Draw the cursor from a PNG (at most 2 MiB and 1024 pixels per side) with its click point.
--pointer-image PATH, --pointer-hotspot X,Y none PNG shown over pressable controls, with its click point.
--travel-ms MIN,MAX 90,320 Travel time range for short to long moves.
--bow N, --overshoot N, --tremor PX 1, 0.035, 1.1 Path curvature (0 to 3), overshoot fraction (0 to 0.15), and tremor in points (0 to 4).
--dwell-ms N 0 Pause on the target before the action, 0 to 300. Maximum travel plus dwell must not exceed 700 ms.
--motion-seed N none Vary each move deterministically.
```json { "version": "2.6", "ok": true, "command": "cursor-overlay", "data": { "activation": { "environment": "AGENT_DESKTOP_SESSION", "value": "run-1793353372413-23159-0" }, "cursor_overlay": { "enabled": true, "max_words": 6, "style": { "accent": "#4299FF", "highlight": true, "rim": "#111318", "ripple": true, "size": 1.0 } }, "next": "export AGENT_DESKTOP_SESSION=run-1793353372413-23159-0", "rendered": true, "session_id": "run-1793353372413-23159-0" } } ``` `rendered: false` means the setting was saved but nothing was drawn, as on Linux Wayland. With `--agent-id`, `enable` saves the style as that agent's profile. `disable` stops every cursor in the session; if teardown cannot be confirmed, the setting is still saved and the error's disposition is `delivery_uncertain`. How headed actions and the overlay interact is in [Headed mode](/desktop/guides/headed-mode#see-where-actions-land). ## trace Reads the merged trace timeline of the active session. Both actions need an active session and no OS permission. Recording and privacy are in [Traces](/desktop/guides/traces). ```bash agent-desktop --session "$AGENT_DESKTOP_SESSION" trace show [--limit N] [--event PREFIX] agent-desktop --session "$AGENT_DESKTOP_SESSION" trace export [--out PATH] [--limit N] ```
Subcommand and flag Default Meaning
show --limit N 500 Return the last N merged events. 0 returns all.
show --event PREFIX none Keep only events whose name starts with this prefix, before the limit applies.
export --out PATH trace-SESSION.html in the session directory Output path for a self-contained HTML viewer.
export --limit N 5000 Embed the last N events. 0 embeds all.
`show` returns `session_id`, `segments`, `total_events`, `returned_events`, `truncated`, and `events`, plus `matched_events` with `--event` and `warnings` for skipped lines. `export` returns: ```json { "version": "2.6", "ok": true, "command": "trace", "data": { "bytes": 26611, "event_count": 9, "path": "/tmp/run.html", "returned_events": 9, "screenshots_embedded": 0, "screenshots_skipped": 0, "total_events": 9, "truncated": false } } ``` Without an active session, or with a session that recorded nothing, both return `INVALID_ARGS`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: wait description: Reference for the wait command, with every mode, the element predicates, the event kinds, timeouts and their defaults, and the timeout error details. url: "https://agent-browser.dev/desktop/commands/wait" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Wait" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). `wait` blocks until a condition holds, so you do not need a fixed sleep. Each call uses exactly one mode. When to use `wait` and when to use the post-action `--wait-for` flags is in [Waiting without sleeps](/desktop/guides/waiting); those flags are in [Configuration](/desktop/configuration#post-action-wait-flags). The examples are real output from Linux. ## wait ```bash agent-desktop wait MS agent-desktop wait --element REF [--predicate NAME] [--value TEXT] [--action NAME] agent-desktop wait --window TITLE [--app NAME] agent-desktop wait --text TEXT [--count N] [--app NAME] agent-desktop wait --menu [--app NAME] agent-desktop wait --menu-closed [--app NAME] agent-desktop wait --notification [--text TEXT] [--app NAME] agent-desktop wait --event KIND [--app NAME] [--window TITLE] [--window-id ID] ```
Flag or argument Default Meaning
MS none Pause for this many milliseconds.
--element REF none Wait until the element behind this qualified ref satisfies --predicate.
--predicate NAME exists exists, enabled, visible, actionable, or value. Requires --element.
--value TEXT none Expected value for --predicate value. Required there and rejected otherwise.
--action NAME click For --predicate actionable: click, type, set-value, or clear.
--window TITLE none Wait until a window whose title contains this text exists. With --event, it narrows the event to an exact title instead.
--text TEXT none Wait until this text appears in the app's tree. With --notification, it filters notification text instead.
--count N none For --text: wait until exactly N nodes match.
--menu, --menu-closed off Wait until a menu surface is open, or dismissed.
--notification off Wait until a new notification arrives.
--event KIND none Wait for a lifecycle signal: window-opened, window-closed, app-launched, app-terminated, focus-changed, surface-appeared, or surface-dismissed.
--window-id ID none Narrow window-opened, window-closed, or focus-changed to one window.
--app NAME none Scope a window, text, menu, notification, or event wait to one application. Element and time waits ignore it.
--timeout MS 30000 Timeout for every mode except MS, at most 86400000.
Flags from two modes, or no mode at all, return `INVALID_ARGS`. The wait polls about every 100 ms and wakes early when the platform signals a change. `wait` never changes the application, so a failed wait leaves nothing half-applied; a successful `--text` wait saves a snapshot. ### Time `wait MS` sleeps and returns `{ "waited_ms": MS }`. Use it only to space two actions apart when nothing observable marks readiness. ### Element `--element REF` polls the live element behind a ref until the predicate holds. The ref names its snapshot, so no `--app` is needed. This waits for the calculator display to read `7`: ```bash agent-desktop wait --element @s2txcl95uvubmn:e6 --predicate value --value 7 --timeout 5000 ``` ```json { "version": "2.6", "ok": true, "command": "wait", "data": { "elapsed_ms": 286, "found": true, "observed": { "expected_chars": 1, "matched": true, "value_chars": 1, "value_present": true }, "predicate": "value", "ref": "@s2txcl95uvubmn:e6" } } ``` `observed` depends on the predicate: `exists`; `enabled` or `visible` with `applicable`; `actionable` with `checks`; or the value counts above, which never echo the value itself. `actionable` runs the checks the action would run; use `--action type` before typing, because the editability check runs only for editing actions. A ref that does not resolve yet keeps being retried until the timeout. ### Window `--window TITLE` waits for any window whose title contains `TITLE`, focused or not. ```bash agent-desktop wait --window Calculator --app gnome-calculator --timeout 3000 ``` The result is `{ "elapsed_ms", "found": true, "window": WINDOW }`, where `WINDOW` has the fields `list-windows` returns. ### Text `--text TEXT` snapshots the app's window each poll and looks for the text in node names, values, and descriptions, as a case-insensitive substring. ```bash agent-desktop wait --text Calculator --app gnome-calculator --timeout 3000 ``` ```json { "version": "2.6", "ok": true, "command": "wait", "data": { "elapsed_ms": 112, "found": true, "ref": null, "role": "window", "snapshot_id": "s2uiek94hz3vms", "text": "Calculator" } } ``` `ref` and `role` describe the first match, and `ref` is `null` when that node has no ref. The wait saves its snapshot, so `snapshot_id` is fresh and acts like any snapshot. When an app has several eligible windows, such as a document and its save dialog, the wait returns `AMBIGUOUS_TARGET` before it checks any text: wait for the dialog with `--window` instead. ### Menu `--menu` waits until a menu surface is open, and `--menu-closed` until it is gone. Both return `{ "elapsed_ms", "found": true }`. On Linux, `--menu-closed` on a menu that never opened is true at once, so wait for `--menu` first. ### Notification `--notification` takes a baseline of the notification list when it starts and returns the first notification not in it. `--text` and `--app` filter which notifications count. The result is `{ "condition": "notification", "matched": true, "notification": {...}, "elapsed_ms" }`. Headless limits on macOS and Windows are on the [notification commands](/desktop/commands/notifications) page. ### Event `--event KIND` compares a baseline taken at the start with fresh reads and returns the first matching change, so you need no window id or title in advance. Put the action that causes the event and the wait in one [batch](/desktop/commands/batch): the batch captures the baseline before the action runs. This entry waited for the window of an app the previous entry launched: ```json { "command": "wait", "data": { "elapsed_ms": 144, "event": { "app": "gtk3-demo", "kind": "window_opened", "pid": 23270, "title": "Application Class", "window_id": "w-1.167-org-a11y-atspi-accessible-44" }, "found": true }, "execution": "completed", "index": 1, "ok": true, "version": "2.6" } ``` `event.kind` uses snake_case. Surface events add `surface` and require `--app`; `--window` and `--window-id` apply only to the three window kinds. `--app` resolves once, at the start, to one process. When no app matches, `app-launched`, `window-opened`, and `surface-appeared` keep polling, so you can race a launch. `app-terminated`, `window-closed`, and `surface-dismissed` return at once with `found: true` and `target_unresolved: true`, which a misspelled `--app` also produces. `focus-changed` returns `APP_NOT_FOUND`. ### Errors A wait that runs out of time returns `TIMEOUT` with `details.kind: "wait_timeout"`, `details.predicate`, `details.timeout_ms`, and what the wait last saw: ```json { "version": "2.6", "ok": false, "command": "wait", "error": { "code": "TIMEOUT", "message": "Window title (7 chars) not found within 1000ms", "recovery": { "strategy": "inspect_state_then_retry_original", "retryable": true, "requires_fresh_snapshot": true, "retry_after_ms": 250 }, "details": { "app": "gedit", "kind": "wait_timeout", "last_observed": { "count": 1, "titles": ["Untitled Document 1 - gedit"], "truncated": false }, "predicate": "window", "timeout_ms": 1000, "title": "Save As" }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` Element waits carry `ref` and `last_observed`; text waits carry `text_chars` and `expected_count` but not the text; event waits carry `baseline_counts`; any mode can carry `last_error`. Other errors: `INVALID_ARGS` for a bad mode combination or a bare ref, the ref errors for `--element`, `APP_UNRESPONSIVE` when the app stops answering, and `PERM_DENIED`. Platform limits: see [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), and [Linux](/desktop/platforms/linux). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: C library API description: Every function, struct, and enum in agent_desktop.h, grouped by area, with exact C signatures. url: "https://agent-browser.dev/desktop/ffi/api" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "API" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Signatures on this page are extracted from `crates/ffi/include/agent_desktop.h`. Every `AdResult` function returns `AD_RESULT_OK` (0) or a negative code; read the details with the [last-error accessors](/desktop/ffi/ownership-and-threading#errors). Which call frees which pointer is in [Ownership](/desktop/ffi/ownership-and-threading#ownership). ## Enums Enum fields inside structs are stored as `int32_t`. The library validates each one and returns `AD_RESULT_ERR_INVALID_ARGS` for an out-of-range value. New `AdResult` codes may be appended.
Enum Values
AdResult AD_RESULT_OK 0, AD_RESULT_ERR_PERM_DENIED -1, AD_RESULT_ERR_ELEMENT_NOT_FOUND -2, AD_RESULT_ERR_APP_NOT_FOUND -3, AD_RESULT_ERR_ACTION_FAILED -4, AD_RESULT_ERR_ACTION_NOT_SUPPORTED -5, AD_RESULT_ERR_STALE_REF -6, AD_RESULT_ERR_WINDOW_NOT_FOUND -7, AD_RESULT_ERR_PLATFORM_NOT_SUPPORTED -8, AD_RESULT_ERR_TIMEOUT -9, AD_RESULT_ERR_INVALID_ARGS -10, AD_RESULT_ERR_NOTIFICATION_NOT_FOUND -11, AD_RESULT_ERR_INTERNAL -12, AD_RESULT_ERR_SNAPSHOT_NOT_FOUND -13, AD_RESULT_ERR_POLICY_DENIED -14, AD_RESULT_ERR_AMBIGUOUS_TARGET -15, AD_RESULT_ERR_APP_UNRESPONSIVE -16
AdImageFormat AD_IMAGE_FORMAT_PNG 0, AD_IMAGE_FORMAT_JPG 1
AdActionKind AD_ACTION_KIND_CLICK 0, AD_ACTION_KIND_DOUBLE_CLICK 1, AD_ACTION_KIND_RIGHT_CLICK 2, AD_ACTION_KIND_TRIPLE_CLICK 3, AD_ACTION_KIND_SET_VALUE 4, AD_ACTION_KIND_SET_FOCUS 5, AD_ACTION_KIND_EXPAND 6, AD_ACTION_KIND_COLLAPSE 7, AD_ACTION_KIND_SELECT 8, AD_ACTION_KIND_TOGGLE 9, AD_ACTION_KIND_CHECK 10, AD_ACTION_KIND_UNCHECK 11, AD_ACTION_KIND_SCROLL 12, AD_ACTION_KIND_SCROLL_TO 13, AD_ACTION_KIND_PRESS_KEY 14, AD_ACTION_KIND_KEY_DOWN 15, AD_ACTION_KIND_KEY_UP 16, AD_ACTION_KIND_TYPE_TEXT 17, AD_ACTION_KIND_CLEAR 18, AD_ACTION_KIND_HOVER 19, AD_ACTION_KIND_DRAG 20
AdDirection AD_DIRECTION_UP 0, AD_DIRECTION_DOWN 1, AD_DIRECTION_LEFT 2, AD_DIRECTION_RIGHT 3
AdDeliveryDisposition AD_DELIVERY_DISPOSITION_UNKNOWN 0, AD_DELIVERY_DISPOSITION_NOT_DELIVERED 1, AD_DELIVERY_DISPOSITION_DELIVERY_UNCERTAIN 2, AD_DELIVERY_DISPOSITION_DELIVERED_UNVERIFIED 3, AD_DELIVERY_DISPOSITION_DELIVERED_VERIFIED 4
AdFindSelectionKind AD_FIND_SELECTION_KIND_STRICT 0, AD_FIND_SELECTION_KIND_FIRST 1, AD_FIND_SELECTION_KIND_LAST 2, AD_FIND_SELECTION_KIND_NTH 3
AdIdentifierKind AD_IDENTIFIER_KIND_AX_IDENTIFIER 0, AD_IDENTIFIER_KIND_AX_DOM_IDENTIFIER 1, AD_IDENTIFIER_KIND_AUTOMATION_ID 2, AD_IDENTIFIER_KIND_RUNTIME_ID 3, AD_IDENTIFIER_KIND_ATSPI_OBJECT_PATH 4
AdModifier AD_MODIFIER_META 0, AD_MODIFIER_CTRL 1, AD_MODIFIER_ALT 2, AD_MODIFIER_SHIFT 3
AdMouseButton AD_MOUSE_BUTTON_LEFT 0, AD_MOUSE_BUTTON_RIGHT 1, AD_MOUSE_BUTTON_MIDDLE 2
AdMouseEventKind AD_MOUSE_EVENT_KIND_MOVE 0, AD_MOUSE_EVENT_KIND_DOWN 1, AD_MOUSE_EVENT_KIND_UP 2, AD_MOUSE_EVENT_KIND_CLICK 3
AdPolicyKind AD_POLICY_KIND_HEADLESS 0, AD_POLICY_KIND_FOCUS_FALLBACK 1, AD_POLICY_KIND_HEADED 2
AdRetryDisposition AD_RETRY_DISPOSITION_UNKNOWN 0, AD_RETRY_DISPOSITION_SAFE 1, AD_RETRY_DISPOSITION_UNSAFE 2
AdScreenshotKind AD_SCREENSHOT_KIND_SCREEN 0, AD_SCREENSHOT_KIND_FULL_SCREEN 2
AdSnapshotSurface AD_SNAPSHOT_SURFACE_WINDOW 0, AD_SNAPSHOT_SURFACE_FOCUSED 1, AD_SNAPSHOT_SURFACE_MENU 2, AD_SNAPSHOT_SURFACE_MENUBAR 3, AD_SNAPSHOT_SURFACE_SHEET 4, AD_SNAPSHOT_SURFACE_POPOVER 5, AD_SNAPSHOT_SURFACE_ALERT 6, AD_SNAPSHOT_SURFACE_DESKTOP 7, AD_SNAPSHOT_SURFACE_TASKBAR 8, AD_SNAPSHOT_SURFACE_SYSTEM_TRAY 9, AD_SNAPSHOT_SURFACE_QUICK_SETTINGS 10, AD_SNAPSHOT_SURFACE_NOTIFICATION_CENTER 11, AD_SNAPSHOT_SURFACE_TOOLBAR 12, AD_SNAPSHOT_SURFACE_DOCK 13, AD_SNAPSHOT_SURFACE_SPOTLIGHT 14, AD_SNAPSHOT_SURFACE_MENU_BAR_EXTRAS 15, AD_SNAPSHOT_SURFACE_SYSTEM_TRAY_OVERFLOW 16, AD_SNAPSHOT_SURFACE_START_MENU 17, AD_SNAPSHOT_SURFACE_ACTION_CENTER 18
AdStepMechanism AD_STEP_MECHANISM_SEMANTIC_API 1, AD_STEP_MECHANISM_PHYSICAL_SYNTHETIC 2
AdWindowOpKind AD_WINDOW_OP_KIND_RESIZE 0, AD_WINDOW_OP_KIND_MOVE 1, AD_WINDOW_OP_KIND_MINIMIZE 2, AD_WINDOW_OP_KIND_MAXIMIZE 3, AD_WINDOW_OP_KIND_RESTORE 4
Sizes and versions of versioned structs are the `AD_*_SIZE` and `AD_*_VERSION` macros in the header. `AD_MAX_STRING_BYTES` (1 MiB) bounds every C string the library reads. ## ABI handshake ```c // Returns the major ABI version of the loaded library. uint32_t ad_abi_version(void); // Checks the header major against the library; call once after loading. AdResult ad_init(uint32_t expected_major); ``` ## Adapter lifecycle An `AdAdapter` is an opaque token. Create one per process, or one per session. ```c // Creates a platform adapter with no session; null on failure. struct AdAdapter *ad_adapter_create(void); // Creates an adapter scoped to a session id; a null session means none. struct AdAdapter *ad_adapter_create_with_session(const char *session); // Destroys an adapter; calls in flight finish, later calls fail with invalid args. void ad_adapter_destroy(struct AdAdapter *adapter); // Returns OK when accessibility permission is granted or not required. AdResult ad_check_permissions(const struct AdAdapter *adapter); ``` ## Command-backed JSON entrypoints Each writes an owned JSON envelope into `*out`; free it with `ad_free_string`, including an `"ok": false` envelope. `*out` is null only on an argument or infrastructure failure. ```c // Snapshots the app surface selected by AdSnapshotSurface. AdResult ad_snapshot(const struct AdAdapter *adapter, const char *app, int32_t surface, uint8_t max_depth, bool interactive_only, bool compact, char **out); // Runs a ref action with the default 5000 ms auto-wait; policy is an AdPolicyKind. AdResult ad_execute_by_ref(const struct AdAdapter *adapter, const char *ref_id, const struct AdAction *action, int32_t policy, char **out); // Same, with an explicit auto-wait budget (-1 default, 0 single attempt). AdResult ad_execute_by_ref_timeout(const struct AdAdapter *adapter, const char *ref_id, const struct AdAction *action, int32_t policy, int64_t timeout_ms, char **out); // Runs the wait command until the condition holds or scope.timeout_ms elapses. AdResult ad_wait(const struct AdAdapter *adapter, const struct AdWaitArgs *args, char **out); // Returns the status envelope. AdResult ad_status(const struct AdAdapter *adapter, char **out); // Returns the version envelope. AdResult ad_version(char **out); // Returns the merged trace timeline of the adapter's session. AdResult ad_trace_show(const struct AdAdapter *adapter, int32_t limit, const char *event_prefix, char **out); // Exports the session trace as a self-contained HTML file. AdResult ad_trace_export(const struct AdAdapter *adapter, int32_t limit, const char *out_path, char **out); ``` ## Actions, refs, and native handles `ad_execute_by_ref` is the primary action entrypoint. `ad_execute_action` and `ad_execute_action_with_policy` act on a native handle and skip strict re-identification and the actionability checks. The two exact-entry calls, `ad_execute_ref_action_exact_with_policy` and `ad_resolve_element_exact`, fail closed with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED` on every platform. An `AdExactRefEntry` carries no retained element token and no capture proof, so the element it names cannot be re-identified. Act through a snapshot ref with `ad_execute_by_ref`, and get a native handle with `ad_find_exact`. ```c typedef struct AdNativeHandle { const void *ptr; } AdNativeHandle; typedef struct AdScrollParams { int32_t direction; uint32_t amount; } AdScrollParams; typedef struct AdKeyCombo { const char *key; const int32_t *modifiers; uint32_t modifier_count; } AdKeyCombo; typedef struct AdPoint { double x; double y; } AdPoint; typedef struct AdDragParams { struct AdPoint from; struct AdPoint to; uint64_t duration_ms; uint64_t drop_delay_ms; } AdDragParams; typedef struct AdAction { int32_t kind; const char *text; struct AdScrollParams scroll; struct AdKeyCombo key; struct AdDragParams drag; } AdAction; typedef struct AdElementState { const char *role; char **states; uint32_t state_count; const char *value; } AdElementState; typedef struct AdActionStep { const char *label; const char *outcome; int32_t mechanism; bool has_mechanism; bool verified; bool has_verified; uint64_t _reserved; } AdActionStep; typedef struct AdDeliverySemantics { int32_t delivery; int32_t retry; } AdDeliverySemantics; typedef struct AdActionResult { const char *action; const char *ref_id; struct AdElementState *post_state; struct AdActionStep *steps; uint32_t step_count; const char *details_json; struct AdDeliverySemantics disposition; } AdActionResult; ``` ```c typedef struct AdRefProcess { uint32_t pid; } AdRefProcess; typedef struct AdRefIdentity { const char *role; const char *name; const char *value; const char *description; const char *native_id; } AdRefIdentity; typedef struct AdRect { double x; double y; double width; double height; } AdRect; typedef struct AdRefGeometry { struct AdRect bounds; uint64_t bounds_hash; bool has_bounds; bool has_bounds_hash; int32_t bounds_space; } AdRefGeometry; typedef struct AdStringSlice { const char *const *items; size_t count; } AdStringSlice; typedef struct AdRefCapabilities { struct AdStringSlice states; struct AdStringSlice available_actions; } AdRefCapabilities; typedef struct AdRefSource { const char *app; const char *window_id; const char *window_title; uint64_t window_bounds_hash; int32_t surface; bool has_window_bounds_hash; } AdRefSource; typedef struct AdRefScope { const char *root_ref; const uint32_t *path; size_t path_count; bool path_is_absolute; } AdRefScope; typedef struct AdRefEntry { struct AdRefProcess process; struct AdRefIdentity identity; struct AdRefGeometry geometry; struct AdRefCapabilities capabilities; struct AdRefSource source; struct AdRefScope scope; } AdRefEntry; typedef struct AdExactRefEntry { uint32_t version; uint32_t size; struct AdRefEntry entry; const char *process_instance; int32_t identifier_kind; } AdExactRefEntry; ``` ```c // Acts on a native handle under the default policy, with no preflight. AdResult ad_execute_action(const struct AdAdapter *adapter, const struct AdNativeHandle *handle, const struct AdAction *action, struct AdActionResult *out); // Acts on a native handle with an explicit AdPolicyKind. AdResult ad_execute_action_with_policy(const struct AdAdapter *adapter, const struct AdNativeHandle *handle, const struct AdAction *action, int32_t policy, struct AdActionResult *out); // Always fails with ACTION_NOT_SUPPORTED: an exact ref entry carries no // retained element token and no capture proof, so it cannot be // re-identified. Act through a snapshot ref with ad_execute_by_ref. AdResult ad_execute_ref_action_exact_with_policy(const struct AdAdapter *adapter, const struct AdExactRefEntry *entry, const struct AdAction *action, int32_t policy, struct AdActionResult *out); // Always fails with ACTION_NOT_SUPPORTED for the same reason. To get a native // handle, use ad_find_exact. AdResult ad_resolve_element_exact(const struct AdAdapter *adapter, const struct AdExactRefEntry *entry, struct AdNativeHandle *out); // Releases a native handle on the resolving thread and zeroes it. AdResult ad_free_handle(const struct AdAdapter *adapter, struct AdNativeHandle *handle); // Frees every string and array owned by an AdActionResult. void ad_free_action_result(struct AdActionResult *result); ``` ## Find, get, and is ```c typedef struct AdFindSelection { int32_t kind; uint32_t nth; } AdFindSelection; typedef struct AdFindControl { uint32_t version; struct AdFindSelection selection; uint64_t timeout_ms; } AdFindControl; typedef struct AdFindIdentity { const char *role; const char *name; const char *description; const char *native_id; const char *value; } AdFindIdentity; typedef struct AdFindStatePredicate { const char *token; int32_t expected; } AdFindStatePredicate; typedef struct AdFindStateSlice { const struct AdFindStatePredicate *items; size_t count; } AdFindStateSlice; typedef struct AdFindFilter { struct AdFindIdentity identity; const char *has_text; struct AdFindStateSlice states; const struct AdFindQuery *has; const struct AdFindQuery *has_not; bool exact; } AdFindFilter; typedef struct AdFindQuery { struct AdFindControl control; struct AdFindFilter filter; } AdFindQuery; ``` ```c // Finds and strictly resolves one element in an exact window; duplicates need a first, last, or nth selection. // The handle is published only when the landed element still matches the selected entry's recorded identity. AdResult ad_find_exact(const struct AdAdapter *adapter, const struct AdExactWindowInfo *win, const struct AdFindQuery *query, struct AdNativeHandle *out_handle); // Reads "value" or "bounds" from a resolved handle into an owned string, or null. AdResult ad_get(const struct AdAdapter *adapter, const struct AdNativeHandle *handle, const char *property, char **out); // Checks a boolean state of the element a query matches in an exact window. AdResult ad_is_exact(const struct AdAdapter *adapter, const struct AdExactWindowInfo *win, const struct AdFindQuery *query, const char *property, bool *out); ``` ## Raw tree `ad_get_tree_exact` returns a flat, breadth-first tree with no refs, refmap, or envelope. Use `ad_snapshot` for the observe and act loop. ```c typedef struct AdNodeContent { const char *ref_id; const char *role; const char *name; const char *value; const char *description; const char *hint; } AdNodeContent; typedef struct AdNodePresentation { char **states; struct AdRect bounds; uint32_t state_count; bool has_bounds; } AdNodePresentation; typedef struct AdNodeRelation { int32_t parent_index; uint32_t child_start; uint32_t child_count; } AdNodeRelation; typedef struct AdNode { struct AdNodeContent content; struct AdNodePresentation presentation; struct AdNodeRelation relation; } AdNode; typedef struct AdNodeTree { struct AdNode *nodes; uint32_t count; } AdNodeTree; typedef struct AdTreeOptions { uint8_t max_depth; bool include_bounds; bool interactive_only; bool compact; int32_t surface; } AdTreeOptions; ``` ```c // Reads an exact window into a flat tree; children are contiguous at nodes[child_start..]. AdResult ad_get_tree_exact(const struct AdAdapter *adapter, const struct AdExactWindowInfo *win, const struct AdTreeOptions *opts, struct AdNodeTree *out); // Frees a tree and zeroes it. void ad_free_tree(struct AdNodeTree *tree); ``` ## Apps and windows Window functions take an `AdExactWindowInfo`, which carries the process-generation token that targeting needs. ```c typedef struct AdWindowInfo { const char *id; const char *title; const char *app_name; uint32_t pid; struct AdRect bounds; bool has_bounds; bool is_focused; } AdWindowInfo; typedef struct AdExactWindowInfo { uint32_t version; uint32_t size; struct AdWindowInfo window; const char *process_instance; bool accessible; } AdExactWindowInfo; typedef struct AdAppInfo { const char *name; uint32_t pid; const char *bundle_id; } AdAppInfo; typedef struct AdWindowOp { int32_t kind; double width; double height; double x; double y; } AdWindowOp; ``` ```c // Launches an app and writes its first window. AdResult ad_launch_app_exact(const struct AdAdapter *adapter, const char *id, uint64_t timeout_ms, struct AdExactWindowInfo *out); // Closes an app; force skips graceful shutdown. AdResult ad_close_app(const struct AdAdapter *adapter, const char *id, bool force); // Lists running apps into an opaque list. AdResult ad_list_apps(const struct AdAdapter *adapter, struct AdAppList **out); // Returns the number of apps in the list. uint32_t ad_app_list_count(const struct AdAppList *list); // Returns a borrowed entry, or null when out of range. const struct AdAppInfo *ad_app_list_get(const struct AdAppList *list, uint32_t index); // Frees the list. void ad_app_list_free(struct AdAppList *list); // Lists windows with process-generation evidence, optionally filtered. AdResult ad_list_windows_exact(const struct AdAdapter *adapter, const char *app_filter, bool focused_only, struct AdExactWindowList **out); // Returns the number of windows in the list. uint32_t ad_exact_window_list_count(const struct AdExactWindowList *list); // Returns a borrowed window entry, or null when out of range. const struct AdExactWindowInfo *ad_exact_window_list_get(const struct AdExactWindowList *list, uint32_t index); // Frees the window list. void ad_exact_window_list_free(struct AdExactWindowList *list); // Focuses an exact window. AdResult ad_focus_window_exact(const struct AdAdapter *adapter, const struct AdExactWindowInfo *win); // Resizes, moves, minimizes, maximizes, or restores an exact window. AdResult ad_window_op_exact(const struct AdAdapter *adapter, const struct AdExactWindowInfo *win, struct AdWindowOp op); // Releases the strings inside an AdExactWindowInfo written by ad_launch_app_exact. void ad_release_exact_window_fields(struct AdExactWindowInfo *win); ``` ## Surfaces ```c typedef struct AdSurfaceInfo { const char *kind; const char *title; int64_t item_count; } AdSurfaceInfo; typedef struct AdExactSurfaceInfo { uint32_t version; uint32_t size; const char *id; struct AdSurfaceInfo surface; const char *unclassified; } AdExactSurfaceInfo; ``` ```c // Lists the overlay surfaces a process presents. AdResult ad_list_surfaces_exact(const struct AdAdapter *adapter, uint32_t pid, struct AdExactSurfaceList **out); // Returns the number of surfaces in the list. uint32_t ad_exact_surface_list_count(const struct AdExactSurfaceList *list); // Returns a borrowed surface entry, or null when out of range. const struct AdExactSurfaceInfo *ad_exact_surface_list_get(const struct AdExactSurfaceList *list, uint32_t index); // Frees the surface list. void ad_exact_surface_list_free(struct AdExactSurfaceList *list); ``` ## Wait Zero-initialize `AdWaitArgs` and check its size before calling [`ad_wait`](#command-backed-json-entrypoints). ```c typedef struct AdOptionalU64 { uint64_t value; bool present; } AdOptionalU64; typedef struct AdWaitSurfaceModes { bool menu; bool menu_closed; bool notification; } AdWaitSurfaceModes; typedef struct AdWaitMode { struct AdOptionalU64 pause; const char *element; const char *window; const char *text; struct AdWaitSurfaceModes surfaces; } AdWaitMode; typedef struct AdOptionalUsize { size_t value; bool present; } AdOptionalUsize; typedef struct AdWaitPredicate { const char *predicate; const char *value; const char *action; struct AdOptionalUsize count; } AdWaitPredicate; typedef struct AdWaitScope { uint64_t timeout_ms; const char *app; } AdWaitScope; typedef struct AdWaitArgs { struct AdWaitMode mode; struct AdWaitPredicate predicate; struct AdWaitScope scope; } AdWaitArgs; ``` ## Displays and screenshots ```c typedef struct AdDisplayInfo { uint32_t version; uint32_t size; const char *id; struct AdRect bounds; bool is_primary; double scale; } AdDisplayInfo; typedef struct AdScreenshotTarget { int32_t kind; uint64_t screen_index; } AdScreenshotTarget; ``` ```c // Lists displays in screenshot screen-index order. AdResult ad_list_displays(const struct AdAdapter *adapter, struct AdDisplayList **out); // Returns the number of displays. uint32_t ad_display_list_count(const struct AdDisplayList *list); // Returns a borrowed display entry, or null when out of range. const struct AdDisplayInfo *ad_display_list_get(const struct AdDisplayList *list, uint32_t index); // Frees the display list. void ad_display_list_free(struct AdDisplayList *list); // Captures a display into an opaque image buffer. AdResult ad_screenshot(const struct AdAdapter *adapter, const struct AdScreenshotTarget *target, struct AdImageBuffer **out); // Captures one exact window into an opaque image buffer. AdResult ad_screenshot_window_exact(const struct AdAdapter *adapter, const struct AdExactWindowInfo *window, struct AdImageBuffer **out); // Returns a borrowed pointer to the image bytes. const uint8_t *ad_image_buffer_data(const struct AdImageBuffer *buf); // Returns the byte length of the image. uint64_t ad_image_buffer_size(const struct AdImageBuffer *buf); // Returns the pixel width. uint32_t ad_image_buffer_width(const struct AdImageBuffer *buf); // Returns the pixel height. uint32_t ad_image_buffer_height(const struct AdImageBuffer *buf); // Returns the encoding. AdImageFormat ad_image_buffer_format(const struct AdImageBuffer *buf); // Returns the point-to-pixel scale factor. double ad_image_buffer_scale_factor(const struct AdImageBuffer *buf); // Frees the image buffer; freeing twice is undefined behavior. void ad_image_buffer_free(struct AdImageBuffer *buf); ``` ## Clipboard ```c // Reads clipboard text into an owned string. AdResult ad_get_clipboard(const struct AdAdapter *adapter, char **out); // Writes UTF-8 text to the clipboard. AdResult ad_set_clipboard(const struct AdAdapter *adapter, const char *text); // Clears the clipboard. AdResult ad_clear_clipboard(const struct AdAdapter *adapter); ``` ## Mouse These send physical input and carry no headless policy. For policy-checked clicks, use ref actions. ```c typedef struct AdMouseEvent { int32_t kind; struct AdPoint point; int32_t button; uint32_t click_count; } AdMouseEvent; ``` ```c // Drags the pointer from params.from to params.to. AdResult ad_drag(const struct AdAdapter *adapter, const struct AdDragParams *params); // Sends a move, down, up, or click event at a screen point. AdResult ad_mouse_event(const struct AdAdapter *adapter, const struct AdMouseEvent *event); // Same, while holding a modifier chord. AdResult ad_mouse_event_with_modifiers(const struct AdAdapter *adapter, const struct AdMouseEvent *event, const int32_t *modifiers, uint32_t modifier_count); // Sends a wheel event in line deltas; modifier_mask bits 0 to 3 are meta, ctrl, alt, shift. AdResult ad_mouse_wheel(const struct AdAdapter *adapter, struct AdPoint point, double delta_x, double delta_y, uint32_t modifier_mask); ``` ## Notifications ```c typedef struct AdNotificationIdentity { const char *app; const char *title; } AdNotificationIdentity; typedef struct AdNotificationActionRequest { uint32_t index; int32_t policy; const char *action_name; struct AdNotificationIdentity identity; } AdNotificationActionRequest; typedef struct AdNotificationFilter { const char *app; const char *text; uint32_t limit; bool has_limit; } AdNotificationFilter; typedef struct AdNotificationInfo { uint32_t index; const char *app_name; const char *title; const char *body; char **actions; uint32_t action_count; } AdNotificationInfo; ``` ```c // Lists notifications; indexes are stable only within one list. AdResult ad_list_notifications(const struct AdAdapter *adapter, const struct AdNotificationFilter *filter, struct AdNotificationList **out); // Returns the number of notifications. uint32_t ad_notification_list_count(const struct AdNotificationList *list); // Returns a borrowed notification, or null when out of range. const struct AdNotificationInfo *ad_notification_list_get(const struct AdNotificationList *list, uint32_t index); // Frees the list. void ad_notification_list_free(struct AdNotificationList *list); // Presses a named action, failing closed when the identity no longer matches. AdResult ad_notification_action(const struct AdAdapter *adapter, const struct AdNotificationActionRequest *request, struct AdActionResult *out); // Dismisses one notification when it matches the expected app or title. AdResult ad_dismiss_notification(const struct AdAdapter *adapter, uint32_t index, const char *app_filter, const char *expected_app, const char *expected_title); // Dismisses every matching notification and returns the dismissed and failed lists. AdResult ad_dismiss_all_notifications(const struct AdAdapter *adapter, const char *app_filter, struct AdNotificationList **dismissed_out, struct AdNotificationList **failed_out); // Frees both lists returned by ad_dismiss_all_notifications. void ad_dismiss_all_notifications_free(struct AdNotificationList *dismissed, struct AdNotificationList *failed); ``` ## Errors These read the calling thread's last-error slot. See [Errors](/desktop/ffi/ownership-and-threading#errors). ```c // Returns the code of the last failure on this thread. AdResult ad_last_error_code(void); // Returns a borrowed description, or null. const char *ad_last_error_message(void); // Returns a borrowed recovery hint, or null. const char *ad_last_error_suggestion(void); // Returns a borrowed OS diagnostic, or null. const char *ad_last_error_platform_detail(void); // Returns borrowed structured JSON, or null; it may contain on-screen text. const char *ad_last_error_details(void); // Writes the delivery and retry semantics of the last failure. AdResult ad_last_error_delivery_semantics(struct AdDeliverySemantics *out); ``` ## Logging and memory ```c // Registers or clears one process-wide callback for log events. AdResult ad_set_log_callback(void (*callback)(int32_t level, const char *msg)); // Frees a string the library returned; null-tolerant. void ad_free_string(char *s); ``` ## Size getters Each returns a struct size as compiled in the library. Compare it with your binding's size before you pass the struct. ```c size_t ad_action_size(void); size_t ad_action_result_size(void); size_t ad_action_step_size(void); size_t ad_display_info_size(void); size_t ad_drag_params_size(void); size_t ad_element_state_size(void); size_t ad_exact_ref_entry_size(void); size_t ad_exact_surface_info_size(void); size_t ad_exact_window_info_size(void); size_t ad_ref_entry_size(void); size_t ad_wait_args_size(void); ``` `AdAdapter`, `AdAppList`, `AdDisplayList`, `AdExactSurfaceList`, `AdExactWindowList`, `AdImageBuffer`, and `AdNotificationList` are opaque: use them only through the functions above. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Build and link description: Get libagent_desktop_ffi from a release or build it from source, find the header, link a C program against it, and run the ABI handshake. url: "https://agent-browser.dev/desktop/ffi/build-and-link" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Build and link" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). Use a prebuilt archive from a release, or build the library from source with the `release-ffi` profile. ## Prebuilt archives Every GitHub release ships `agent-desktop-ffi-v-.tar.gz` (`.zip` on Windows) for macOS arm64 and x86_64, Linux x64 and arm64 (glibc), and Windows x64 and ARM64. Each archive holds:
Path Contents
lib/ The library and the helper that must stay beside it: agent-desktop-macos-helper on macOS, agent-desktop-linux-clipboard-owner on Linux. On Windows, also agent_desktop_ffi.dll.lib, the import library for MSVC.
include/agent_desktop.h The generated C header
`checksums.txt` in the release catches corruption; verify origin with `gh attestation verify --repo vercel-labs/agent-desktop`. ## Build from source ```bash cargo build --profile release-ffi -p agent-desktop-ffi # writes target/release-ffi/libagent_desktop_ffi.so on Linux ``` The output is `libagent_desktop_ffi.dylib` on macOS, `libagent_desktop_ffi.so` on Linux, and `agent_desktop_ffi.dll` on Windows, in `target/release-ffi/`. The `release-ffi` profile keeps `panic = "unwind"`, so every entrypoint can catch a panic and report it as `AD_RESULT_ERR_INTERNAL`. The workspace `release` profile uses `panic = "abort"`, so `cargo build --release` produces a library whose panics abort the host process. Do not ship it. A source build also needs the platform helper beside the library; see [Installation](/desktop/installation#build-from-source). ## The header `crates/ffi/include/agent_desktop.h` is generated by cbindgen; do not edit it. When the Rust ABI or `crates/ffi/cbindgen.toml` changes, regenerate it with `scripts/update-ffi-header.sh`, and CI fails on drift. The header ends with C11 `_Static_assert` guards that pin the size and field offsets of every size-pinned struct; define `AGENT_DESKTOP_ABI_ASSERTS` to skip them. ## Link and run a program This program checks the ABI, snapshots one app, and prints the envelope: ```c #include #include "agent_desktop.h" int main(int argc, char **argv) { const char *app = argc > 1 ? argv[1] : NULL; /* NULL reads the focused window */ if (ad_init(AD_ABI_VERSION_MAJOR) != AD_RESULT_OK) { fprintf(stderr, "ABI mismatch: %s\n", ad_last_error_message()); return 1; } AdAdapter *adapter = ad_adapter_create(); if (!adapter) { fprintf(stderr, "adapter_create failed: %s\n", ad_last_error_message()); return 1; } char *json = NULL; AdResult rc = ad_snapshot(adapter, app, AD_SNAPSHOT_SURFACE_WINDOW, 3, true, true, &json); /* max_depth, interactive_only, compact */ if (json) { /* an ok:true or ok:false envelope: free either */ printf("%s\n", json); ad_free_string(json); } else { fprintf(stderr, "snapshot failed (%d): %s\n", (int)rc, ad_last_error_message()); } ad_adapter_destroy(adapter); return rc == AD_RESULT_OK ? 0 : 1; } ``` Compile and run it from the repository root on Linux or macOS: ```bash cc -std=c11 -I crates/ffi/include main.c \ -L target/release-ffi -lagent_desktop_ffi \ -Wl,-rpath,"$PWD/target/release-ffi" \ -o snapshot_demo ./snapshot_demo gnome-calculator ``` On Linux against GNOME Calculator it printed this envelope, shown here with the window metadata and `tree` elided: ````json {"version":"2.6","ok":true,"command":"snapshot","data":{"app":"gnome-calculator","complete":true,"ref_count":2,"snapshot_id":"s18ekeexjf5trb","window":{"id":"...","title":"..."},"tree":{"...":"..."}}}``` The macOS install name is `@rpath/libagent_desktop_ffi.dylib`, so a program that links the library needs an rpath to its directory; `dlopen` and `ctypes` callers load by path and need nothing. On Windows, an MSVC program links `agent_desktop_ffi.dll.lib` and loads the DLL at run time. On Linux the program must run inside the user's graphical session to see apps; see [Linux](/desktop/platforms/linux). ## Validate struct sizes A language whose struct layout can differ from C's, such as Python `ctypes` or Go `cgo`, must check each size-pinned struct before passing one to the library. Each has an `AD_*_SIZE` macro and a runtime getter that returns the size compiled on the Rust side: `AdAction` (`ad_action_size`), `AdActionResult`, `AdActionStep`, `AdDisplayInfo`, `AdDragParams`, `AdElementState`, `AdExactRefEntry`, `AdExactSurfaceInfo`, `AdExactWindowInfo`, `AdRefEntry`, and `AdWaitArgs`, each with a matching `ad__size()` getter. Zero-initialize every caller-allocated struct before you set fields, and set the `version` and `size` fields of versioned structs to the header's constants. ## Smoke test `tests/ffi-python/smoke.py` is the reference for non-C hosts and uses only the Python standard library. It checks the ABI version, the size getters, `ad_version`, and the create, snapshot, free, and destroy cycle: ```bash python3 tests/ffi-python/smoke.py target/release-ffi/libagent_desktop_ffi.so crates/ffi/include/agent_desktop.h ```` On the Linux test desktop it ended with `All legs passed.` --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: C library description: Call agent-desktop in process through libagent_desktop_ffi, the C ABI over the same core and platform adapters as the CLI. url: "https://agent-browser.dev/desktop/ffi" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Overview" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). The C library exposes agent-desktop as a shared library: `libagent_desktop_ffi.dylib` on macOS, `libagent_desktop_ffi.so` on Linux, and `agent_desktop_ffi.dll` on Windows. Your program calls `ad_*` functions instead of starting the CLI for each command. Any language with a C FFI can load it; `tests/ffi-python/smoke.py` is the tested reference. The header is `crates/ffi/include/agent_desktop.h`. ## Relationship to the CLI The library and the CLI run the same core and the same platform adapter; the `agent-desktop` binary and the `agent-desktop-ffi` crate are the only two places that connect a platform adapter to core. - **Same envelope.** The command-backed functions return the CLI's JSON envelope as an owned string. See [JSON output](/desktop/json-output). - **Same refs.** `ad_snapshot` writes a refmap under the same [state root](/desktop/configuration#state-root) and returns qualified refs, which `ad_execute_by_ref` resolves strictly. A bare `@e5` returns `AD_RESULT_ERR_INVALID_ARGS`. - **Same safety rules.** Ref actions run the same resolution, actionability checks, and interaction policy as the CLI, and mutations take the same cross-process input lock, so CLI and library calls from one user serialize. - **Same sessions.** `ad_adapter_create_with_session` scopes refmaps to a [session](/desktop/sessions) like `--session`. ## Two layers
Layer Functions Use when
JSON entrypoints ad_snapshot, ad_execute_by_ref, ad_execute_by_ref_timeout, ad_wait, ad_status, ad_version, ad_trace_show, ad_trace_export You run the observe, act, and verify loop: parse the envelope and pass refs back.
Struct-based exact APIs ad_list_windows_exact, ad_launch_app_exact, ad_find_exact, ad_get_tree_exact, and the rest of the \*\_exact family, plus clipboard, mouse, display, notification, and screenshot calls You need typed results without JSON. These carry process-generation evidence in versioned, size-pinned structs.
## What the library does not expose - `ad_snapshot` reads the owner selected by `surface`. On macOS, `menu` and `menubar` are application-owned and need no document window; window-owned snapshots include `data.window`. The ABI has no window id, skeleton mode, or `--root` drill-down; use the CLI for those. - `ad_wait` does not expose `wait --event` or `--window-id`. - The library carries no skills. Read the FFI guide with `agent-desktop skills get ffi` from the CLI. ## ABI version `AD_ABI_VERSION_MAJOR` is 7 in this header. The major increments when a symbol is removed or changes incompatibly: a struct layout change, or a call that now fails where it used to succeed. ABI 7 makes `ad_execute_ref_action_exact_with_policy` and `ad_resolve_element_exact` fail closed with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED`. New functions and new result codes are additive. Call `ad_init(AD_ABI_VERSION_MAJOR)` once after loading the library: a mismatch returns `AD_RESULT_ERR_INVALID_ARGS`, and you should call nothing else. ## Next steps - [Build and link](/desktop/ffi/build-and-link): build the library, link it, and run a minimal program. - [API](/desktop/ffi/api): every function, struct, and enum, grouped by area. - [Ownership, threading, errors](/desktop/ffi/ownership-and-threading): who frees what, which threads may call, and how to read failures. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Ownership, threading, and errors description: Who frees every pointer the C library returns, which threads may call it, and how to read failures. url: "https://agent-browser.dev/desktop/ffi/ownership-and-threading" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Ownership, threading, errors" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). ## Ownership Every pointer the library hands you was allocated by Rust, so C's `free()` cannot release it. Each one has a matching free function. Borrowed pointers, such as the `ad_last_error_*` strings and entries returned by `*_list_get`, are never freed.
Allocated by Free with
The JSON entrypoints (ad_snapshot, ad_execute_by_ref, ad_execute_by_ref_timeout, ad_wait, ad_status, ad_version, ad_trace_show, ad_trace_export) ad_free_string(out)
ad_get_clipboard, ad_get (which may write null) ad_free_string(text)
ad_adapter_create, ad_adapter_create_with_session ad_adapter_destroy(adapter)
ad_list_apps ad_app_list_free(list)
ad_list_displays ad_display_list_free(list)
ad_list_windows_exact ad_exact_window_list_free(list)
ad_list_surfaces_exact ad_exact_surface_list_free(list)
ad_list_notifications ad_notification_list_free(list)
ad_dismiss_all_notifications ad_dismiss_all_notifications_free(dismissed, failed)
ad_launch_app_exact (the window struct's strings) ad_release_exact_window_fields(\&out)
ad_get_tree_exact ad_free_tree(\&out)
ad_find_exact (the native handle) ad_free_handle(adapter, \&handle)
ad_execute_action, ad_execute_action_with_policy, ad_notification_action ad_free_action_result(\&out)
ad_screenshot, ad_screenshot_window_exact ad_image_buffer_free(buf)
- Every free function accepts null and does nothing. - Freeing a list handle or an image buffer twice is undefined behavior. Set the pointer to `NULL` after you free it. - `ad_free_handle` zeroes the handle, so a second call on the same struct is a no-op. Free handles before you destroy the adapter that produced them. - Strings inside a struct belong to the struct's owner. Do not free them one by one. - Every fallible function zeroes its out-parameter before validating anything, so calling the paired free function after a failure is safe. - `ad_resolve_element_exact` and `ad_execute_ref_action_exact_with_policy` allocate nothing. They fail with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED` and leave their out-parameter null or zeroed. ## Threading Entrypoints may be called from any host thread. Native handles are the exception: resolve, use, and release a handle on the same thread, with the adapter that produced it; anything else returns `AD_RESULT_ERR_INVALID_ARGS`. To pass a target across threads, pass the qualified ref and use `ad_execute_by_ref`.
Concurrent calls Contract
read and read Both run concurrently.
read and mutation The read does not wait and may overlap the mutation; its result is a point-in-time observation.
mutation and mutation One holder of the interaction lock runs at a time for the same OS user. A waiter that cannot get it returns a structured timeout.
Snapshots, finds, listings, screenshots, clipboard reads, status, and trace reads are observations. Ref actions, native-handle actions, input synthesis, clipboard writes, app and window changes, and notification actions are mutations. The lock is shared with the CLI, so mutations serialize across threads and processes of one user. It is scoped to one command, not a transaction: the UI can change between an observation and a later action. - `ad_adapter_destroy` revokes the adapter. Calls already in flight finish; calls that start afterward return `AD_RESULT_ERR_INVALID_ARGS`. - `ad_wait` blocks its calling thread for at most `scope.timeout_ms`. - The `ad_set_log_callback` callback may run on any thread that calls an `ad_*` function, concurrently, and may still fire briefly after you unregister it. Keep it alive for the process lifetime and never let it unwind across the C ABI. - A language runtime's global lock does not give thread affinity; keep native handles in thread-confined objects, or use refs. - On macOS, `ad_check_permissions` reports Accessibility trust for the host executable (`python3`, `node`, your app), not for the library. ## Errors A failing `AdResult` function returns a negative code and fills the calling thread's last-error slot. Read it before the next call that can fail. ```c AdExactWindowInfo win = {0}; AdResult rc = ad_launch_app_exact(adapter, "gnome-calculator", 5000, &win); if (rc != AD_RESULT_OK) { const char *msg = ad_last_error_message(); const char *sug = ad_last_error_suggestion(); /* may be NULL */ fprintf(stderr, "launch failed (%d): %s\n", (int)rc, msg ? msg : "(no message)"); if (sug) fprintf(stderr, " suggestion: %s\n", sug); return -1; } ad_release_exact_window_fields(&win); ```
Accessor Returns
ad_last_error_code() The code of the last failure, or AD_RESULT_OK
ad_last_error_message() Description, or null
ad_last_error_suggestion() Recovery hint, or null
ad_last_error_platform_detail() OS diagnostic, or null
ad_last_error_details() Structured JSON, or null. It can contain text from the user's screen, so keep it out of shared logs.
ad_last_error_delivery_semantics(\&sem) Writes the delivery and retry values of the last failure; both are UNKNOWN when none is recorded.
The strings stay valid across later successful calls on the same thread; only the next failing call replaces them, like POSIX `errno`. The slot is thread-local. Retry only when the retry disposition is `AD_RETRY_DISPOSITION_SAFE`, as described in [Errors and recovery](/desktop/errors-and-recovery#disposition). `AdResult` codes mirror the CLI's [error codes](/desktop/errors-and-recovery#error-codes): `AD_RESULT_ERR_PERM_DENIED` is -1 through `AD_RESULT_ERR_APP_UNRESPONSIVE` at -16. The full list is on the [API page](/desktop/ffi/api#enums). Treat any negative value outside that list as an unknown error, because new codes can be appended. ### Two failure modes for JSON entrypoints The JSON entrypoints fail in two ways, so check `*out` before you decide whether to free: - **Argument or infrastructure failure** (a null adapter, bad UTF-8, a bad enum): `*out` is null, nothing is allocated, and only the last-error slot reports it. - **Command failure** (app not found, `STALE_REF`, `TIMEOUT`): `*out` holds an `"ok": false` envelope that you free with `ad_free_string`. The last-error slot is also set. ### Refs, enums, and panics Ref-taking functions accept only `@:e`. A bare or malformed ref returns `AD_RESULT_ERR_INVALID_ARGS` before dispatch; a missing snapshot returns `AD_RESULT_ERR_SNAPSHOT_NOT_FOUND`; a missing ref returns `AD_RESULT_ERR_STALE_REF`. Every `int32_t` enum field is validated, so an out-of-range value returns `AD_RESULT_ERR_INVALID_ARGS` instead of undefined behavior. Every entrypoint traps panics and reports them as `AD_RESULT_ERR_INTERNAL`, but only in a library built with the `release-ffi` profile; see [Build and link](/desktop/ffi/build-and-link#build-from-source). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Batching commands description: Run several agent-desktop commands in one process with batch. When to batch, the non-atomic semantics, validation before the first entry, and how to read the result. url: "https://agent-browser.dev/desktop/guides/batch" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Batch" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). `batch` runs a JSON array of commands in order, in one process. It is not a transaction. Use it to save process starts on a known sequence, and to pair an action with `wait --event` so the wait's baseline is read before the action runs. The input format, limits, and output fields are in the [`batch` reference](/desktop/commands/batch). ## When to batch Batch these: - An action followed by `wait --event`, such as a click or launch that opens a window. See [Waiting without sleeps](/desktop/guides/waiting#wait-for-an-event-your-action-causes). - A fixed sequence on refs you already hold, where you do not need to read anything between steps. Do not batch these: - Steps where the next command depends on the previous output. Observe, act, and verify each need a read; see [the observe, act, verify loop](/desktop/guides/observe-act-verify). - Steps that need a ref an earlier entry would create. A batch is validated before its first entry runs, so it cannot name refs from a `snapshot` entry inside it. ## Know what runs first Before any entry runs, agent-desktop checks every entry: the JSON, the command names, the arguments, ref syntax, and the permissions each command needs. One failure rejects the whole batch and nothing runs: ```json { "code": "INVALID_ARGS", "message": "Batch entry 0 ('click') failed validation: Invalid ref_id '@e1': expected @:e", "details": { "batch_command": "click", "batch_index": 0 }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } ``` The check does not reach the live desktop. Whether a ref still resolves, and whether a headless physical action is refused, are decided when the entry runs. ## Read every entry A batch returns `ok: true` and exit code `0` even when entries fail, so read each entry's `ok`. This batch clicked the calculator's C button, then a ref that does not exist, then C again, with `--stop-on-error` (fields abridged): ```json { "results": [ { "index": 0, "ok": true, "execution": "completed", "data": { "disposition": { "delivery": "delivered_unverified", "retry": "unsafe" } } }, { "index": 1, "ok": false, "execution": "completed", "error": { "code": "STALE_REF", "message": "@s3sl23kzkoxzvb:e999 not found in current RefMap", "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ], "total_entries": 3, "completed_entries": 2, "not_started_entries": 1, "stopped": { "index": 1, "reason": "stop_on_error" } } ``` Each entry keeps its own disposition, and a batch never retries an entry. Apply the normal rule per entry: retry only when `retry` is `safe`. ## Control failures - **Default.** A failed entry does not stop the batch. Later entries still run, so an action after a failed one can act on a UI you did not expect. - **`--stop-on-error`.** The batch halts after the first failed entry and reports `stopped`. Use it for any sequence where a later step is wrong after an earlier one fails. - **`--timeout-ms`.** The default 60000 ms covers the whole batch, waits included. When it expires, the next entry is reported `not_started` with reason `deadline`. ## Use refs in a batch Refs in `args` must be qualified and come from a snapshot or `find` taken before the batch, under the same `--session`. If an earlier entry changes the UI, a later ref can go stale and that entry fails with `STALE_REF`. Keep batched ref actions to controls that one action does not rebuild. See [Snapshots and refs](/desktop/snapshots-and-refs#refresh-after-the-ui-changes). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Chromium and Electron apps description: Drive the web contents of a Chromium or Electron app with agent-browser over a DevTools port that agent-desktop opens, and keep the native UI on agent-desktop. url: "https://agent-browser.dev/desktop/guides/chromium-and-electron" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Chromium and Electron" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop reads every app through the OS accessibility tree, and that path works for Chromium and Electron apps too. Their web contents can be dense and slow to walk, though. For an app you can launch fresh, `launch --cdp` starts it with a loopback-only Chrome DevTools Protocol (CDP) port, and [agent-browser](https://agent-browser.dev) connects to that port to drive the web contents. agent-desktop never speaks CDP itself; it opens and verifies the port, and keeps the native UI.
Part of the app Drive it with
Web contents: pages, panels, editors, forms inside the app agent-browser, over the CDP port
Native menu bar, file dialogs and sheets, window management, notifications, screenshots agent-desktop
An app that is already running, or one you did not launch agent-desktop only. The port exists only on a process launched with it
## 1. Launch the app with a DevTools port ```bash agent-desktop launch chrome --cdp --env ACCESSIBILITY_ENABLED=1 \ --arg=--force-renderer-accessibility --arg=--user-data-dir=/tmp/chrome-profile ``` `--cdp` adds the remote-debugging switches, waits until the endpoint answers, and returns it in `data.cdp`. Omit the port, as here, to let the OS pick a free one, or pass one (`--cdp 9229`). The `--env` and `--force-renderer-accessibility` arguments are for Linux, where Chromium joins the accessibility bus only when asked, so that agent-desktop can still see the native UI; see [Linux](/desktop/platforms/linux#chromium-qt-and-electron). An Electron app needs no extra arguments. This is Google Chrome on Linux, with the suggestion and the window object trimmed: ```json { "version": "2.6", "ok": true, "command": "launch", "data": { "app": "chrome", "cdp": { "http_endpoint": "http://127.0.0.1:44297", "port": 44297, "product": "Chrome/156.0.8078.79", "websocket_url": "ws://127.0.0.1:44297/devtools/browser/4a28a252-c7ea-4288-9985-50e34fc49b8f" }, "pid": 11342, "process_instance": "1e1c1e2e:168278", "renderer": "chromium", "suggestion": "Next: run `agent-browser connect 44297` (preferred; `agent-browser skills get electron` has the guide) or connect any CDP client such as Playwright or Puppeteer. ...", "window": { "id": "w-1.10-org-a11y-atspi-accessible-1", "title": "Untitled - Google Chrome" } } } ``` Pass the app name that `list-apps` prints (`chrome`, `Slack`, `Obsidian`). agent-desktop refuses `--cdp` for an app that is already running, because a running process has no port and quitting it would lose the user's state. It finds the running app by that name, so a launcher alias such as `google-chrome` on Linux skips the check and the launch times out instead. ## 2. Connect agent-browser to the port ```bash agent-browser connect 44297 # the port from data.cdp.port agent-browser snapshot -i # - heading "Settings" [level=1, ref=e1] # - textbox "Name" [ref=e3] # - button "Save" [ref=e2] agent-browser fill @e3 "Ada" agent-browser click @e2 ``` From here the web contents follow agent-browser's own snapshot, ref, and act loop; see [agent-browser](https://agent-browser.dev) and `agent-browser skills get electron`. Playwright, Puppeteer, and other CDP clients can use `data.cdp.websocket_url` instead. Do not write raw CDP or call app-internal APIs. ## 3. Keep the native UI on agent-desktop The window chrome, menus, and dialogs are not in the web contents, so they stay on agent-desktop, in the same run: ```bash agent-desktop find --app chrome --role button --name Reload --first agent-desktop find --app chrome --role textfield --first # the address bar ``` Against the session above, the second `find` returned the address bar, `"name": "Address and search bar"`, with the page agent-browser had opened as its value. Use `snapshot --surface` for a native file dialog or sheet the web contents open; see [Surfaces](/desktop/surfaces). ## 4. Close the app ```bash agent-desktop close-app chrome ``` `close-app` ends the process and with it the port; the run above returned `delivered_verified` with `details.evidence: "process_instance_exited"`, and the endpoint stopped answering. While the port is open, any local process of the same user can control the app's web contents, so request `--cdp` only for the steps that need it. ## When `--cdp` fails
Error Cause What to do
ACTION_FAILED, details.kind: cdp_requires_fresh_launch The app is already running. details.pids lists it, and nothing was launched (not_delivered). Run close-app, confirm the exit, and launch again with --cdp, or drive the running app through accessibility.
ACTION_FAILED, details.kind: cdp_endpoint_unavailable The app started, but no DevTools endpoint answered before the deadline. The app keeps running (delivered_unverified). Use the accessibility path, or close-app and check the app name and arguments.
INVALID_ARGS, details.kind: cdp_port_in_use The port you named is bound. Name another port, or omit it.
INVALID_ARGS, details.kind: cdp_switch_conflict An --arg sets --remote-debugging-\* or --remote-allow-origins, which --cdp owns. Drop that --arg.
The flags are in the [`launch` reference](/desktop/commands/app-window#launch). For a dense app you cannot relaunch, map it with a skeleton and drill into one region instead; see [Skeleton and drill-down](/desktop/skeleton-and-drill-down). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Headed mode description: What headless ref actions do by default, what the --headed flag permits, which commands need it, and how to handle POLICY_DENIED without switching modes by reflex. url: "https://agent-browser.dev/desktop/guides/headed-mode" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Headed mode" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop runs ref actions headless by default. The global `--headed` flag is a permission to use physical input: take focus, move the real cursor, and synthesize keys and pointer events. ## What headless actions do A headless ref action calls the app's accessibility actions. It does not take focus, move the cursor, synthesize keystrokes, or use the clipboard, which keeps it safe to run while a person uses the desktop. When the accessibility API cannot do what you asked, the action fails closed with a structured error such as `ACTION_NOT_SUPPORTED` or `POLICY_DENIED`. It never silently switches to physical input. A result shows which mechanism ran: in `data.steps`, `semantic_api` is an accessibility call and `physical_synthetic` is synthesized input. ## What `--headed` permits `--headed` authorizes the focus and cursor side effects an action declares: - For a ref, core focuses the exact source window before headed pointer input and verifies the target point before delivery. A process-owned ref has no window to focus, but physical input still requires a verified point. - On macOS, clicking a native menu-bar item uses `AXPress`; it works headless and does not move the pointer under `--headed`. A headless `right-click` can use `AXShowMenu` when the target provides it.- Raw coordinates (`--xy`) carry no window identity, so they never focus anything. - It applies to every entry of a [batch](/desktop/guides/batch). - It permits physical input; it does not force it. Each platform decides per action whether to use a physical or a semantic path, and `check`, `set-value`, and `expand` stay semantic under `--headed` everywhere. - It does not lift other guards. A blocked key combo still needs `--force`, and a platform that cannot deliver physical input, such as pointer input on Wayland, still refuses. ```bash agent-desktop --headed double-click @s8f3k2p9:e3 # a physical gesture ``` Physical input gets no confirmation from the OS, so a physical result is usually `delivered_unverified`. Confirm the effect with a read, as in [the observe, act, verify loop](/desktop/guides/observe-act-verify). ## Which commands need `--headed` On every platform: `triple-click`, `hover`, `drag`, `mouse-move`, `mouse-click`, and `mouse-wheel`. `double-click` needs it too, except on macOS when the element offers `AXOpen`. `key-down`, `key-up`, `mouse-down`, and `mouse-up` never deliver input in either mode; see the [keyboard](/desktop/commands/keyboard#key-down) and [mouse](/desktop/commands/mouse#mouse-down) references. Depending on the OS:
Command macOS Windows Linux
right-click Headless --headed Headless, if the element has a context-menu action
type Headless --headed Headless, except into a terminal text node
focus Headless --headed --headed
scroll Headless Headless --headed, X11 only
Listing notifications Headless while the surface is open Headless while the surface is open Headless
Notification mutations --headed --headed --headed, X11 only
open-system-surface Headless --headed --headed
`press` is keyboard input and does not need `--headed` for an unscoped press; scoped presses have platform rules. Each platform page has the full capability table: [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), [Linux](/desktop/platforms/linux). ## Handle `POLICY_DENIED` `POLICY_DENIED` means the interaction policy blocked a physical or focus path. Nothing was delivered, so the disposition is `not_delivered`. This is `hover` without `--headed`: ```json { "version": "2.6", "ok": false, "command": "hover", "error": { "code": "POLICY_DENIED", "message": "hover moves the cursor and is disabled in headless mode", "recovery": { "strategy": "request_explicit_policy_then_retry_original", "retryable": true, "requires_fresh_snapshot": false }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` Get an explicit decision from the user before you retry. Some `POLICY_DENIED` errors need something other than `--headed`: a blocked key combo needs `--force`, and a failed `supported_action` check in `error.details.checks` means the element offers no route for that action. > **Do not switch to headed to recover from a failure:** > > Use `--headed` only when the task needs physical input and the user permits it. Do not add it after a headless action fails. A headless failure usually means the semantic path is missing or the UI changed. A headed retry would move the user's cursor or steal focus to work around that, and if the first attempt may have been delivered, a retry can act twice. Instead: 1. Read `error.code`, `error.disposition`, and `error.details`. 2. Fix the cause: take a fresh snapshot, run `wait --element REF --predicate actionable`, or pick the semantic command (`set-value` for `type`, `check` for `click`). 3. If the step is physical by nature, such as a drag or a hover, and the user permits it, run that step with `--headed` from the start. ## See where actions land The agent cursor overlay draws a cursor, a ripple, and an element highlight where each action lands. It is presentation only and never changes what an action does. Enable it once per session with `session start --cursor` or `cursor-overlay enable`; a headed command hides it while it dispatches real input. See [`cursor-overlay`](/desktop/commands/system#cursor-overlay). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Using agent-desktop over MCP description: Serve every agent-desktop command as an MCP tool over stdio. How tools map to commands, how global flags apply, and what a tool returns. url: "https://agent-browser.dev/desktop/guides/mcp" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "MCP" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). `agent-desktop mcp` runs a Model Context Protocol server over stdio. Every command is a tool, so an MCP client gets the same surface as the CLI with no separate setup. ## Connect a client Add the server to your client's MCP configuration: ```json { "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } } ``` The server speaks both the stateless `2026-07-28` protocol and the `initialize` handshake of `2025-11-25` and earlier. ## How tools map to commands Each tool is one command, named `desktop_` plus the command with hyphens as underscores. Subcommands join the name the same way: | Command | Tool | | --------------- | ----------------------- | | `snapshot` | `desktop_snapshot` | | `list-windows` | `desktop_list_windows` | | `session start` | `desktop_session_start` | | `skills get` | `desktop_skills_get` | A tool's arguments are its command's arguments, with hyphens as underscores: `snapshot --app Finder -i` is `desktop_snapshot` with `{ "app": "Finder", "interactive_only": true }`. Positional arguments use their names too, such as `ref_id` for `click`. A null argument counts as omitted. The tool list is read from the CLI's own command definitions, so it always matches the installed binary. ## Apply global flags once Global flags given to `mcp` apply to every call: ```bash agent-desktop mcp --session s1 --headed ``` `AGENT_DESKTOP_SESSION`, `AGENT_DESKTOP_AGENT_ID` and `AGENT_DESKTOP_HOME` in the server's environment apply the same way. A tool cannot pass a global flag. Commands that need a session, such as `trace` and `cursor-overlay`, work only when the server has one, headed-only commands need `mcp --headed`, and `--wait-for` is not available per call. ## Read results A tool returns the command's JSON envelope as `structuredContent`, and the same JSON as text. `isError` is `true` when the envelope's `ok` is `false`, so read `error.code` and `error.disposition` exactly as you would from the CLI. A `screenshot` without an output path returns its PNG as an image block instead of base64 text. An unknown argument returns a tool error that names the accepted arguments, and a value of the wrong type returns an error that names the argument, so the model can correct the call. ## Start with the skills Call `desktop_skills_get` with `{ "name": "desktop" }` for the observe-act loop and `{ "name": "platform" }` for what differs on your OS. The server's instructions point clients there too. ## Security The server adds nothing the CLI cannot already do. Tool arguments become CLI arguments without a shell. A value is always passed as a value, never as a flag. A tool cannot set global flags such as `--session` or `--headed`; only the server's own command line can. A tool still has the CLI's full reach: it can operate any app the accessibility tree exposes, and `desktop_launch` starts any program on `PATH` with any arguments. Treat the server like a shell for the signed-in user. Keep your client's confirmation on for actions, and remember that text read from the screen is untrusted input to the model. The server runs at most 8 tool calls at once and answers more with a tool error. It refuses any message over 4 MiB. A `notifications/cancelled` stops the call and it gets no response. A call that has not started never starts. A running call gets `SIGTERM` first, so it releases any key or button it holds, and is killed if it is still running 3 seconds later; on Windows it is killed at once. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: The observe, act, verify loop description: Run the observe, act, verify loop against a desktop app. Pick the smallest observation, act on a qualified ref, and confirm the effect by reading the app again. url: "https://agent-browser.dev/desktop/guides/observe-act-verify" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Observe, act, verify" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop does not run the loop; your agent does, one command at a time. `ok: true` does not show that an action worked, so each step ends with a read. Before the first step, run `agent-desktop status`. If a permission is missing or `supported_surfaces` is empty, fix that first with the page for your OS: [macOS](/desktop/platforms/macos), [Windows](/desktop/platforms/windows), or [Linux](/desktop/platforms/linux). ## Observe Read the smallest part of the UI that holds your target. A larger read costs more time and tokens, and its refs go stale sooner.
What you know Run
The role and exact name of the element find --app APP --role button --name Save --exact --first
Nothing yet, and the app is small snapshot --app APP -i --compact
Nothing yet, and the app is dense (Electron, or more than about 50 interactive elements) snapshot --skeleton, then snapshot --root REF on one region
A menu, sheet, alert, or popover is open snapshot --surface KIND, never with --skeleton
[Skeleton and drill-down](/desktop/skeleton-and-drill-down) explains the dense-app row and [Surfaces](/desktop/surfaces) the last one. Pass every ref exactly as printed, and re-read the region after any UI change; see [Snapshots and refs](/desktop/snapshots-and-refs). ## Act Run the command that matches your intent: ```bash agent-desktop click @s8f3k2p9:e12 # activate a control agent-desktop set-value @s8f3k2p9:e3 "hello" # replace a field's value agent-desktop check @s8f3k2p9:e6 # idempotent, unlike toggle ``` Prefer the semantic, idempotent commands: `set-value` over `type`, `check` and `uncheck` over `toggle`, `select` for combo boxes. They run headless, re-read the element, and report `not_delivered` when the element was already in the requested state. Each ref action also waits up to 5000 ms for its target to become actionable, so you rarely need a `wait` before it. Physical gestures such as hover, drag, and double-click need `--headed`; read [Headed mode](/desktop/guides/headed-mode) first. Flags are in the [interaction commands](/desktop/commands/interaction). ## Verify `ok: true` means the command returned a delivered or already-satisfied result, not that the app did what you wanted. Read `data.disposition.delivery`:
Delivery What it proves What to do
delivered_verified The adapter read back the postcondition it names, such as the checked state or the field value. For values, it does not prove the app committed them. Confirm document-level effects with an independent read.
delivered_unverified The dispatch completed. The effect was not confirmed. Read the app before you act again.
not_delivered Nothing was sent, because the operation was already satisfied. Read the state if you expected a change.
Then read the app again with the smallest read that answers the question: ```bash agent-desktop is @s8f3k2p9:e6 --property checked # one state agent-desktop wait --element @s8f3k2p9:e3 --predicate value --value 7 # a value that updates asynchronously agent-desktop snapshot --root @s8f3k2p9:e2 -i --compact # the region that changed ``` Apps update asynchronously: a read taken right after a click can still show the old state, so prefer `wait` to a single read. If the result has a `menu`, `sheet`, or `alert` in `data.surfaces`, the app is waiting on that overlay; observe it with `snapshot --surface` before the window behind it. See [Waiting without sleeps](/desktop/guides/waiting). ## Recover from failures Branch on `error.code` and `error.disposition`, never on message text. `STALE_REF` and `SNAPSHOT_NOT_FOUND` mean observe again and use the new ref; `AMBIGUOUS_TARGET` means narrow the target with `find --exact` or `--root`; `POLICY_DENIED` means read [Headed mode](/desktop/guides/headed-mode) before anything else. Retry or substitute an action only when `disposition.retry` is `safe`; otherwise read the live state first. Every code and its recovery is in [Errors and recovery](/desktop/errors-and-recovery). ## Worked example This run presses a Calculator button on Linux, verifies the display, and clears it. The [quick start](/desktop/quick-start) shows the output of each step. ```bash agent-desktop find --app gnome-calculator --role button --name 7 --exact --first # ref of the 7 button agent-desktop find --app gnome-calculator --role textfield --first # ref of the display agent-desktop click @s3jjre8xzrvo2l:e1 # act agent-desktop wait --element @s3sxeejcz18ky3:e1 --predicate value --value 7 # verify agent-desktop find --app gnome-calculator --role button --name C --exact --first # then clear with C ``` 1. Each `find` returns one match in `data.match`; its `ref_id` names the snapshot that `find` saved. Use the refs from your own output. 2. `click` returns `delivered_unverified`: the accessibility call returned, which does not prove the button did its job. 3. The `wait` is the verification. If it times out, read `details.last_observed` and `data.surfaces` before you click again, because a second click could type `77`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Traces description: Record what agent-desktop did with session traces or --trace, read the timeline with trace show and trace export, and capture visual debug files without leaking private data. url: "https://agent-browser.dev/desktop/guides/traces" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Traces" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). A trace is an append-only JSONL record of what agent-desktop did: each command, its outcome and duration, and each snapshot it saved. Use one to debug a run or to audit what an agent did. There are two ways to record: a [session](/desktop/sessions), which also lets you read the trace back, and `--trace PATH`, which writes one file. ## Record a run in a session A session records a trace by default. `session start` does not activate the session for later commands, so export its id: ```bash export AGENT_DESKTOP_SESSION="$(agent-desktop session start --name trace-demo | jq -er '.data.session_id')" agent-desktop snapshot --app gnome-calculator -i --compact agent-desktop status # session_id is set and tracing is true ``` Each process that runs a command writes its own segment under `/sessions//trace/`, named `-.jsonl`, starting with a `trace.meta` line. `session start --no-trace` creates a session without one. On PowerShell, see [Windows](/desktop/platforms/windows#use-powershell) for the equivalent of `export`. ## Read the timeline `trace show` merges every segment into one ordered timeline and needs no OS permission: ```bash agent-desktop trace show --limit 20 # the last 20 events agent-desktop trace show --event command. # only command.start and command.end agent-desktop trace show --limit 0 | jq -r '.data.events[] | select(.event == "command.end") | [.command, .ok, .duration_ms] | @tsv' ``` These are three real events from the session above: ```json [ { "command": "snapshot", "duration_ms": 109, "event": "command.end", "ok": true, "segment": "23161-1793353372418", "seq": 3, "session_id": "run-1793353372413-23159-0", "ts_ms": 1793353372528, "writer_pid": 23161, "writer_proc_start_ms": 1793353372418 }, { "event": "trace.meta", "os": "linux", "pid": 23176, "proc_start_ms": 1793353372533, "schema": 1, "segment": "23176-1793353372531", "seq": 0, "session_id": "run-1793353372413-23159-0", "ts_ms": 1793353372533, "version": "1.0.1", "writer_pid": 23176, "writer_proc_start_ms": 1793353372533 }, { "command": "trace", "event": "command.start", "segment": "23176-1793353372531", "seq": 1, "session_id": "run-1793353372413-23159-0", "ts_ms": 1793353372534, "writer_pid": 23176, "writer_proc_start_ms": 1793353372533 } ] ``` `trace export` writes a self-contained HTML viewer, by default `trace-.html` in the session directory: ```bash agent-desktop trace export --out /tmp/run.html ``` Both need an active session and read session segments only, not a `--trace PATH` file. Flags and result fields are in the [`trace` reference](/desktop/commands/system#trace). ### Event names
Event Written when
trace.meta A segment opens. Carries schema, the version, os, and the writer pid.
command.start, command.end A command begins and ends. command.end has ok, duration_ms, and code on failure.
snapshot.saved, snapshot.root.saved A snapshot or a drill-down is saved.
locator.resolve A find finished its search.
ref.resolve.ok, ref.resolve.error A ref action re-identified its target, or failed to.
actionability.check.ok, actionability.check.error The pre-dispatch checks passed or failed.
action.artifacts A screenshot-enabled session captured, or skipped, the before and after images.
Other events and fields can appear; ignore the ones you do not know. ## Trace to a single file `--trace PATH` appends events to one file and needs no session: ```bash agent-desktop --trace /tmp/run.jsonl find --app gnome-calculator --role button --name 7 --exact --count jq -c 'select(.event == "command.end")' /tmp/run.jsonl ``` The file is created private. While `--trace` is set, the session's trace directory receives nothing. A trace write failure does not fail the command unless you add `--trace-strict`. ## Capture screenshots with a session `session start --screenshots` stores a before and after PNG of the target window for each ref action, and a copy of each refmap, under the session's `trace/screens/` and `trace/refmaps/`. `trace export` embeds the images. When an image cannot be taken, the `action.artifacts` event records why. Screenshots need tracing on, and on macOS the Screen Recording permission. A session's recording budget is 32 MiB, 100,000 events, and 4,096 directory entries. When it is spent, existing events stay readable and new ones are dropped; start a new session to keep recording. ## Make a visual debug file Visual debug writes one HTML file that pairs a window screenshot with labeled element bounds: ```bash agent-desktop --debug --screenshot /tmp/calc-debug.html snapshot --app gnome-calculator -i agent-desktop --debug --screenshot /tmp/click-debug.html click @s8f3k2p9:e12 ``` - `--debug` requires `--screenshot PATH.html`, a new file ending in `.html`. It is created with owner-only permissions. - It supports `snapshot` (window surface only) and `click`. - The result gains `data.debug` with `path` and `format`. If the window cannot be captured, the command still succeeds and `data.debug.warning` says why; on the openbox test desktop it read `No X11 window of the window manager matches this window`. - The before and after images of a click are two separate observations, not proof that the click was delivered. ## Keep private data private > **Traces and debug files contain desktop content:** > > Snapshot trees, window titles, clipboard reads, notifications, screenshots, and command results contain what you asked agent-desktop to inspect. Treat these files as private. - Trace lines replace sensitive fields, such as `text`, `value`, `name`, `title`, `url`, `label`, `description`, and `password`, with `{ "redacted": true }`. Command names, app names, ref and snapshot ids, error codes, and durations stay readable. - Session screenshots, the exported HTML that embeds them, and visual debug files are not redacted. - Keep these files, and the transcripts of the calling agent, out of public repositories and issue reports. `session end` seals a session, and `session gc --ended` removes ended sessions with their traces. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Waiting without sleeps description: Wait for a desktop app to reach a state without fixed sleeps. Choose between the post-action wait flags and the wait command, and read a timeout. url: "https://agent-browser.dev/desktop/guides/waiting" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "Waiting" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). A fixed sleep either wastes time or races the app. agent-desktop gives you two waits that poll for a condition instead: the post-action `--wait-for` flags, which wait inside the command you already run, and the `wait` command, which waits on its own. Every flag is in the [`wait` reference](/desktop/commands/wait) and in [Post-action wait flags](/desktop/configuration#post-action-wait-flags). ## Choose the wait
You are waiting for Use
A control or text to appear after an action --wait-for "role:text" on the action
A control to disappear after an action, such as a spinner --wait-for-gone "role" on the action
One element to reach a value or become ready wait --element REF --predicate value or actionable
A new window or dialog wait --window TITLE --app APP
Text anywhere in an app wait --text TEXT --app APP
A menu to open or close wait --menu, wait --menu-closed
A window or app lifecycle change your action causes wait --event KIND, in a batch with the action
A new notification wait --notification
Nothing observable wait MS, as a last resort
## Wait inside the action `--wait-for` polls the ref's original process until the selector matches, then returns a fresh snapshot with the action's own report under `after_action`, so you can act on the new refs at once. Application-owned refs keep their source surface. Window-owned refs stay pinned to their original window, including refs captured from a sheet, alert, popover, focused view, or window-owned Linux menu: ```bash agent-desktop click @s8f3k2p9:e5 --wait-for "button:Done" --wait-timeout 10000 agent-desktop click @s8f3k2p9:e5 --wait-for-gone "progressindicator" ``` If the wait times out, the action already ran: the error keeps the action's delivery in `error.disposition` and its report in `details.after_action`. Read the app before you retry the action. ## Wait for an element Use `wait --element` when you hold a ref and need a condition on it. The verify step of the [quick start](/desktop/quick-start#verify) waits for the calculator display to read `7`, which took 286 ms after the click: ```bash agent-desktop wait --element @s2txcl95uvubmn:e6 --predicate value --value 7 --timeout 5000 ``` Use `--predicate actionable --action type` before typing into a field that is still loading. You rarely need a wait before a plain click: ref actions already wait for their target to become actionable. ## Wait for a window, then its content A window can exist before its content does. Wait for the window, then for the content: ```bash agent-desktop wait --window "Save As" --app gedit --timeout 10000 agent-desktop wait --text "Save" --app gedit --timeout 10000 ``` `--window` matches a title that contains your text. If an app has a document and a dialog open, `wait --text` returns `AMBIGUOUS_TARGET` before it checks any text; wait for the dialog by title with `--window`, then read it with `snapshot --surface sheet` or `--window-id`. ## Wait for an event your action causes `wait --event` diffs a baseline taken when the wait starts against fresh reads, so a separate `wait` started after your action can miss the change. Put the action and the wait in one [batch](/desktop/guides/batch), which takes the baseline before the action runs: ```bash agent-desktop batch '[ {"command":"launch","args":{"app":"gtk3-demo"}}, {"command":"wait","args":{"event":"window-opened","app":"gtk3-demo","timeout":10000}} ]' --stop-on-error ``` On Linux this returned the new window 144 ms after the launch; the [`wait` reference](/desktop/commands/wait#event) shows the result. A misspelled `--app` makes closing events such as `app-terminated` return at once with `target_unresolved: true`, so check the name with `list-apps` first. ## Wait for a menu Open the menu, wait for it, and read it as a surface: ```bash agent-desktop click @s2qfke45s1966h:e1 # the Menu button agent-desktop wait --menu --app gtk3-widget-factory --timeout 5000 agent-desktop snapshot --app gtk3-widget-factory --surface menu -i --compact ``` After you choose an item or dismiss the menu, `wait --menu-closed --app gtk3-widget-factory` confirms it is gone. Wait for `--menu` before `--menu-closed`: on Linux, `--menu-closed` on a menu that never opened is true at once. See [Surfaces](/desktop/surfaces). ## Set timeouts
Wait Flag Default
wait in every mode except wait MS --timeout 30000 ms
--wait-for, --wait-for-gone --wait-timeout 30000 ms
Auto-wait before a ref action --timeout-ms 5000 ms
A whole batch, including waits inside it --timeout-ms 60000 ms
`wait` uses `--timeout`; the other commands use `--timeout-ms`. ## Read a timeout A `TIMEOUT` from a wait says what the wait last saw in `details.last_observed`. A `wait --window "Save As"` that timed out reported the only window it found, `Untitled Document 1 - gedit`, which tells you the dialog never opened rather than that the title was wrong. Use it to decide between a wrong title, a wrong `--app`, and a slow app. The fields per mode are in the [`wait` reference](/desktop/commands/wait#errors). On macOS and Windows, a headless `wait --notification` sees only an already-open notification surface; see [macOS](/desktop/platforms/macos) and [Windows](/desktop/platforms/windows). --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Linux description: Requirements, the accessibility bus, per-command capabilities on X11 and Wayland, app naming, surfaces, and troubleshooting for agent-desktop on Linux. url: "https://agent-browser.dev/desktop/platforms/linux" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Linux" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop reads Linux apps through AT-SPI over D-Bus. The loop, refs, JSON envelope, sessions, and traces match every platform, so this page covers only what differs on Linux. GNOME is the measured desktop; X11 and Wayland sessions both work and differ in input, capture, and window control. The examples on this page ran on an X11 session (Xvfb with the openbox window manager, no GNOME Shell), so shell surfaces and notifications were not exercised. ## Requirements - Ubuntu 24.04 or later on x86_64 or aarch64, with a glibc 2.35 floor. Other distributions need an `at-spi2-core` bridge with fix (commit `dc0dc33`); Ubuntu 22.04's bridge aborts on `Action.GetActions`. - A graphical session of the signed-in user, a running AT-SPI bus (`at-spi2-core`), and a window manager. Run agent-desktop from a shell inside that session: the session bus, display, and runtime directory come from its environment. A TTY login, an SSH shell without those variables, a container without a bus, or another user's session sees no apps. - X11 keys need XTEST, and Wayland keys need GNOME Mutter RemoteDesktop. The Wayland clipboard needs `wl-clipboard`. - On X11, `clipboard-set` needs `agent-desktop-linux-clipboard-owner` beside the binary. npm installs and release archives include it; from source, build it with `cargo build --release -p agent-desktop-linux --bins`. - A private state root. A group-writable or world-readable `~/.agent-desktop` makes commands that write there fail with `INTERNAL`. Run `chmod 700` on it, or let agent-desktop create it. ## Check the accessibility bus Linux has no permission dialog for accessibility; whether the bus answers gates everything. Run `status` first: ```bash agent-desktop status ``` ```json { "permissions": { "accessibility": { "state": "granted" }, "automation": { "state": "not_required" }, "screen_recording": { "state": "not_required" } }, "platform": "linux", "supported_surfaces": ["window", "sheet", "alert", "menu"] } ```
State What status and list-apps show Meaning
Answered, healthy accessibility is granted, supported_surfaces lists surfaces, list-apps returns rows Normal
Answered, empty granted, but list-apps returns no rows The bus is up and no app registered. Start an app, or restart one that started before the bus
Unavailable denied with a suggestion, empty supported_surfaces, and commands fail with PLATFORM_NOT_SUPPORTED The session bus or the accessibility bus cannot be reached
An unavailable bus has four usual causes, each reported with the raw D-Bus error in `platform_detail`: - **No session bus.** Run inside a desktop session, or start one with `dbus-run-session -- COMMAND`. - **AT-SPI not installed.** Install `at-spi2-core`. For Flatpak add `--talk-name=org.a11y.Bus`; for snap connect `desktop-legacy` or `unity7`. - **Launcher cannot start.** Read `systemctl --user status at-spi-dbus-bus.service`. - **Socket gone.** Restart `at-spi-dbus-bus.service`, or log out and back in. Judge health with `status`, not with `busctl`: on a cold session `org.a11y.Bus` has no owner until its first use. Do not restart a bus you did not start, because other clients on a shared host depend on it. ## What works headless Ref actions are semantic AT-SPI calls that never move the pointer or change the active window. Under `--headed` on X11, `click`, `right-click`, and `type` try physical input first and fall back to the semantic action only after a `not_delivered` refusal. See [Headed mode](/desktop/guides/headed-mode).
Group Works headless Needs --headed Fails closed or not supported
Observation and surfaces snapshot, find, get, is, list-surfaces, on X11 and Wayland
Ref actions click, right-click, type, set-value, clear, select, toggle, check, uncheck, expand, collapse, scroll-to, when the element exposes the AT-SPI capability focus; type into a terminal text node; some Qt and libadwaita combos for select An element with no semantic route refuses and never falls back to physical input
Wheel scrolling scroll, mouse-wheel, X11 only Headless scroll: POLICY_DENIED
Pointer gestures double-click, triple-click, hover, drag, mouse-click, mouse-move, X11 only Wayland: ACTION_NOT_SUPPORTED, cause wayland_session
Keyboard press on X11 and Mutter Wayland. A scoped headless press needs the target window already active A scoped press to a background app (X11) Blocked combos such as alt+f4 without --force
Held input key-down, key-up, mouse-down, mouse-up
App and window launch, list-apps, list-windows on X11 and Wayland; window control and graceful close-app on X11 focus-window and window control on Wayland: ACTION_NOT_SUPPORTED
Capture screenshot, list-displays X11 under a compositing window manager (GNOME): display and --screen captures return PLATFORM_NOT_SUPPORTED (details.reason: "compositing"); capture one window instead. Wayland: portal consent on first use, and window targets are PLATFORM_NOT_SUPPORTED
Clipboard clipboard-get, clipboard-set, clipboard-clear (text, images, file lists) Missing helper or wl-clipboard: PLATFORM_NOT_SUPPORTED
Wait All modes; --notification on GNOME
Notifications list-notifications, even with the list closed Every dismiss and action, on X11 GNOME Shell only; otherwise PLATFORM_NOT_SUPPORTED
Shell surfaces snapshot --surface system-tray with a StatusNotifierWatcher open-system-surface for action-center, and for system-tray when a StatusNotifierWatcher is present quick-settings, taskbar, start-menu, and system-tray-overflow refuse
System status, permissions, version, batch, skills, session, trace, cursor-overlay Wayland: the overlay setting saves with rendered: false
## App naming `--app` takes the name `list-apps` prints, such as `gnome-calculator`, or a desktop-entry name or id backed by a verified executable, such as `Files`. It never matches a window title. ```json { "bundle_id": "org.gnome.Calculator", "name": "gnome-calculator", "pid": 8255, "presentation": "foreground", "process_instance": "6d91df60:28340" } ``` `list-windows --app` is an exact, case-insensitive match; `list-apps --app` also matches a substring. Window ids are opaque strings such as `w-1.11-org-gnome-Calculator-a11y-...`, valid for the life of the window's process. Several instances of one app share a name; pick one with `--window-id`. `launch` accepts an absolute path, a desktop id, a bare command, or a desktop entry name. ## Surfaces `supported_surfaces` lists `window`, `sheet`, `alert`, and `menu`, plus GNOME Shell kinds the session can resolve. After an opener, read `data.surfaces`. A GTK popover appears asynchronously, so if it is absent, run `wait --menu` before you assume the opener failed. A list-backed GTK4 dropdown is part of the window tree, not a menu surface. See [Surfaces](/desktop/surfaces). ## Chromium, Qt, and Electron An app appears in `list-apps` only when its toolkit registered on the accessibility bus.
App Needs
GTK3, GTK4 Nothing. An app that started before the bus never registers: restart it
Qt QT_ACCESSIBILITY=1 in its environment
Chromium browsers --force-renderer-accessibility; in a bare session without a settings daemon, also --env ACCESSIBILITY_ENABLED=1 on launch
Electron No flag. --no-sandbox where chrome-sandbox is not setuid
```bash agent-desktop launch google-chrome --arg=--force-renderer-accessibility ``` To drive the web contents with agent-browser instead, launch the app with `--cdp`; see [Chromium and Electron apps](/desktop/guides/chromium-and-electron). GTK apps nest deeply with unnamed `group` wrappers that `-i --compact` keeps; read past them, or let a skeleton find the drill targets. Electron native dialogs are not on the AT-SPI tree. ## Troubleshooting
Symptom Cause Fix
Empty supported_surfaces, or PLATFORM_NOT_SUPPORTED about the AT-SPI bus The session bus or accessibility bus is unreachable Run inside the desktop session; see the four causes above
list-apps or snapshot returns ACTION_NOT_SUPPORTED, "The native window type of accessibility frame owned by pid N could not be proven" No window manager is running, so windows cannot be matched to their X11 clients Run inside a real desktop session, or start a window manager
The app runs but --app finds nothing The app is not on the bus, or the name is wrong Use the list-apps name, and see the toolkit table
The window appears but snapshot is thin The content is not ready wait --window, then wait --text, and raise --timeout-ms
ACTION_FAILED, cause frame_not_active, on a scoped press The target window is not active and focus changes are not allowed Bring the window forward, or pass --headed if focus changes are permitted
ACTION_FAILED, cause blocked_by_modal_dialog A modal dialog or popover blocks the window Read it with snapshot --surface and answer or dismiss it
ACTION_FAILED, cause grab_focus_rejected, on focus A GTK4 app rejects Component.GrabFocus Focus its window with focus-window
Two quick click calls on one GTK4 button act once GTK4 reads clicks less than about 250 ms apart as one Wait for the effect between clicks
--menu-closed returns at once The menu never opened Wait for --menu first
screenshot returns ACTION_FAILED with details.reason: "covered" A non-compositing window manager cannot capture a covered window Raise it with focus-window; on a non-compositing window manager, capture the display with --screen 0
INTERNAL, "private file parent is accessible by group or other users" The state root is not private chmod 700 the state root
More cases are in `agent-desktop skills get linux troubleshooting`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: macOS description: Requirements, permissions, per-command capabilities, app naming, surfaces, and troubleshooting for agent-desktop on macOS. url: "https://agent-browser.dev/desktop/platforms/macos" docs_index: /desktop/llms.txt lastUpdated: 2026-11-05 navTitle: "macOS" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop reads macOS apps through the Accessibility (AX) API. The loop, refs, JSON envelope, sessions, and traces match every platform, so this page covers only what differs on macOS. The macOS behavior here is checked against the macOS adapter, its tests, and the macOS skill (`agent-desktop skills get macos`); it was not run for these docs. ## Requirements - macOS 13 or later, Intel or Apple Silicon. - Accessibility permission for the app that launches agent-desktop, such as Terminal, iTerm2, VS Code, or your agent's host app. If System Settings lists the `agent-desktop` binary separately, grant it too. - Screen Recording permission, only for `screenshot` and for `session start --screenshots`. - Automation permission for System Events, only where a command reports it missing, such as opening Notification Center. ## Grant and check permissions The grant belongs to the app that launched agent-desktop, not to the shell. A different terminal or an IDE needs its own grant. 1. Run `agent-desktop permissions`. It does not prompt. 2. If a permission is `denied`, run `agent-desktop permissions --guide`. It opens System Settings at the missing permission with a panel that shows which app to add, and returns at once with `data.guide` set to `started`, `already_running`, or `not_needed`. 3. If the guide cannot run, open **System Settings > Privacy & Security > Accessibility**, add the launching app, and turn it on. 4. Run `agent-desktop permissions` again and confirm `granted`. The app does not need a restart. ```json { "version": "2.6", "ok": true, "command": "permissions", "data": { "accessibility": { "state": "granted" }, "automation": { "state": "unknown" }, "screen_recording": { "state": "granted" } } } ``` `automation` stays `unknown` until macOS would need to prompt. `permissions --request` is the only path that prompts, and it covers Automation; `--guide` covers Accessibility and Screen Recording. Under tmux or SSH there is no launching app, so add the terminal app that started tmux by hand. ## What works headless Ref actions are headless by default. Under `--headed`, `click`, `right-click`, `type`, `clear`, and `scroll` use physical paths first, except a click on an item in the application's native menu bar hierarchy: agent-desktop activates the owning app and calls `AXPress` without moving the cursor. A headless click on these menu items calls `AXPress` without activating the app. `right-click` can use `AXShowMenu` headless when the item provides it. Under `--headed`, right-click, double-click, and triple-click support windowless native menu items by focusing the app and keeping pointer verification. `hover` and `drag` still require the ref's source window and fail with `ACTION_NOT_SUPPORTED` when it has none. Other ref-target pointer actions focus the exact owning window and verify the hit target before delivery. A window-backed popup uses its live owning window even when its snapshot is application-owned. `expand`, `collapse`, `set-value`, `select`, `toggle`, `check`, `uncheck`, `focus`, and `scroll-to` stay semantic. See [Headed mode](/desktop/guides/headed-mode).
Group Works headless Needs --headed Fails closed or not supported
Observation snapshot, find, get, is, list-surfaces, list-displays
Capture screenshot, with Screen Recording permission
Ref actions click, right-click, type, clear, scroll, expand, collapse, set-value, select, toggle, check, uncheck, focus, scroll-to None required
Multi-click and pointer double-click when the element offers AXOpen double-click, triple-click, hover, drag, mouse-move, mouse-click, mouse-wheel Headless: POLICY_DENIED
Keyboard press, always physical keyboard input; it never runs a menu item, so click the item instead Quit shortcuts such as cmd+q: POLICY_DENIED without --force
Held input key-down, key-up, mouse-down, mouse-up
App and window launch, close-app, list-apps, list-windows, focus-window, resize-window, move-window, minimize, maximize, restore
Clipboard clipboard-get, clipboard-set (read back: delivered_verified), clipboard-clear
Wait All modes. wait --notification sees only an already-open Notification Center
Notifications list-notifications while Notification Center is open Listing while it is closed, and every dismiss and action Headless listing with the center closed: POLICY_DENIED
Shell surfaces open-system-surface for dock and menu-bar-extras, returned at once because they are always on screen start-menu, taskbar, system-tray, system-tray-overflow, action-center: INVALID_ARGS
System status, permissions, version, batch, skills, session, cursor-overlay, trace
Judge delivery by observation. AX return codes do not prove an effect: an app can return success from an action that does nothing, so re-read with `get`, `is`, or a snapshot. ## App naming `--app` takes the display name `list-apps` prints, such as `"System Settings"` or `"TextEdit"`. When no running app has that display name, it also matches a bundle ID or an executable name, such as `ControlCenter` for Control Center. `launch` also accepts a bundle ID such as `com.apple.TextEdit`. Window ids such as `w-4521` stay valid for the life of the window; find them with `list-windows`. ## Surfaces `supported_surfaces` lists `window`, `focused`, `menu`, `menubar`, `sheet`, `popover`, `alert`, `dock`, and `menu-bar-extras`. Use `--surface menubar` for an app's File, Edit, and View menus. Use `--surface menu-bar-extras` for status items: without `--app` it reads system items such as Wi-Fi, Battery, and Clock; with `--app` it reads that app's own item. Use `--surface dock` for the Dock. These shell surfaces are available headless through `open-system-surface`. A background app with no window, such as a menu-bar app or Control Center, can be read through its process. This applies to `window`, `menubar`, and `menu-bar-extras`, not `focused`; a regular app that has not shown a window returns `WINDOW_NOT_FOUND`. While a menu, sheet, alert, or popover is open, refs from the window tree behind it are the wrong target. See [Surfaces](/desktop/surfaces). ## Keys and notifications - The primary modifier is `cmd`: `press cmd+c`. Punctuation keys work as the symbol or by name (`comma`, `period`, `slash`). - `cmd+q`, `cmd+shift+q`, `cmd+alt+esc`, `ctrl+cmd+q`, and `cmd+shift+delete` return `POLICY_DENIED` unless you pass `--force`. To quit an app, use `close-app`. - `press cmd+s --app "TextEdit"` sends keys to one app without focusing it. - When agent-desktop opens Notification Center, it closes it afterward and restores the previous focus. Pass `--expected-app` and `--expected-title` on a single dismiss or action, because the stack shifts. Use `wait --notification --app "App"` for banners that vanish before you can list them. ## Chromium and Electron agent-desktop turns on web accessibility in a Chromium renderer when it needs it. For a dense running app, take a skeleton and drill with `--root`. For the web contents of an Electron app you can launch fresh, `launch "App" --cdp` opens a DevTools endpoint for agent-browser. See [Chromium and Electron apps](/desktop/guides/chromium-and-electron). ## Troubleshooting
Symptom Cause Fix
PERM_DENIED, or permissions shows denied The launching app has no Accessibility permission, or no Screen Recording permission for screenshot Follow the grant steps above
Few nodes or no refs Custom-rendered UI, or a Chromium renderer still settling Check permissions, drop -i, raise --timeout-ms, and run list-surfaces in case the content is in a sheet
complete: false or a slow snapshot A deep tree ran out of budget Use find, or a skeleton and --root. See Skeleton and drill-down
APP_UNRESPONSIVE after right-click with retry: unsafe AXShowMenu entered modal menu tracking after the menu opened Do not retry. Run snapshot --surface menu; for a combo box or menu button, use select
select cannot confirm the choice It needs a closed-to-open menu change, and a menu was already open Dismiss the open menu, then run select again
WINDOW_NOT_FOUND A window-owned surface has no open window launch "App" with --activate, then list-windows --app "App"
A shortcut does nothing ctrl used where macOS uses cmd, or the keys went to another app Use cmd+..., or scope with --app
More cases are in `agent-desktop skills get macos troubleshooting`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) --- title: Windows description: Requirements, permissions and elevation, per-command capabilities, app naming, surfaces, PowerShell notes, and troubleshooting for agent-desktop on Windows. url: "https://agent-browser.dev/desktop/platforms/windows" docs_index: /desktop/llms.txt lastUpdated: 2026-11-03 navTitle: "Windows" --- > For an index of all documentation, see [/desktop/llms.txt](/desktop/llms.txt). agent-desktop reads Windows apps through UI Automation (UIA). The loop, refs, JSON envelope, sessions, and traces match every platform, so this page covers only what differs on Windows. The Windows behavior here is checked against the Windows adapter, its tests, and the Windows skill (`agent-desktop skills get windows`); it was not run for these docs. Examples elsewhere are POSIX shell; the PowerShell forms are in [Use PowerShell](#use-powershell). ## Requirements - Windows 10 1809 or later, or Windows Server 2019 or later, on x64 or ARM64. - The interactive desktop session of the signed-in user. Session 0 (services and SYSTEM tasks), Server Core, the secure desktop (UAC prompts and the sign-in screen), a locked desktop, and another user's session are not supported. - The binary is unsigned. SmartScreen can warn on a browser download, antivirus can quarantine it, and Smart App Control on a clean Windows 11 install blocks it. ## Permissions and elevation Windows has no grant dialog for UI Automation; access is controlled by integrity level. - Apps at your terminal's integrity level need no grant. `permissions` reports `accessibility: granted` when UIA answers, and `automation: not_required`. `permissions --guide` returns `PLATFORM_NOT_SUPPORTED`. - An app running as administrator can be read but not driven from a lower-integrity terminal, because User Interface Privilege Isolation (UIPI) blocks the input. agent-desktop compares the process tokens before sending anything and returns `PERM_DENIED` with `E_ACCESSDENIED` in `platform_detail`. Run agent-desktop from a terminal elevated to the same level. - Elevated and non-elevated runs cannot share a state root; the second fails with `INTERNAL`, "private file parent is owned by a foreign principal". Give the elevated run its own `AGENT_DESKTOP_HOME`. ## What works headless Ref actions are semantic UIA calls headless and under `--headed`, except the ones that are physical or take focus. See [Headed mode](/desktop/guides/headed-mode).
Group Works headless Needs --headed Fails closed or not supported
Observation and capture snapshot, find, get, is, screenshot, list-surfaces, list-displays
Ref actions click, set-value, clear, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to right-click, type, focus Headless right-click and type: POLICY_DENIED. Use set-value to write text headless
Multi-click and pointer double-click, triple-click, hover, drag, mouse-click, mouse-move, mouse-wheel Headless: POLICY_DENIED
Keyboard press A scoped press --app to an app that is not frontmost Blocked shortcuts without --force
Held input key-down, key-up, mouse-down, mouse-up
App and window launch, close-app, list-windows, list-apps, focus-window, resize-window, move-window, minimize, maximize, restore close-app on explorer.exe and other protected images: INVALID_ARGS
Clipboard clipboard-get, clipboard-set, clipboard-clear (text and images)
Wait All modes except notifications with the Action Center closed --notification when the Action Center is closed
Notifications list-notifications while the Action Center is open Listing while it is closed, and every dismiss and action Headless on a closed center: POLICY_DENIED
Shell surfaces snapshot --surface KIND on a surface that is already present open-system-surface for start-menu, taskbar, system-tray, system-tray-overflow, action-center quick-settings: INVALID_ARGS, naming action-center
System status, permissions, version, batch, skills, session, cursor-overlay, trace
## App naming - `launch` takes an absolute path, or a bare name under System32 or the Windows directory, with or without `.exe` (`notepad` or `notepad.exe`). It does not resolve a display name such as "Google Chrome". - For a running app, pass the name `list-apps` prints to `--app`, such as `notepad.exe`. - Window ids are `w-` followed by the window handle. A hosted (UWP) app is reported through its `ApplicationFrameHost` frame. - A window behind another window is drivable. A minimized one is not: every element reports `offscreen`. Run `restore --app IMAGE` and snapshot again, instead of `focus-window`, which steals the user's foreground. ## Surfaces `supported_surfaces` lists `window`, `focused`, `sheet` (a Chromium modal), `menu`, and the shell kinds `start-menu`, `taskbar`, `system-tray`, `system-tray-overflow`, and `action-center`. There is no `menubar` surface. `snapshot --surface KIND` reads a shell surface without `--app` and without `--headed`; a closed one returns `WINDOW_NOT_FOUND` with a suggestion that names `open-system-surface`. The `menu` detector covers classic Win32, WPF, and Chromium or Electron menus, but not WinUI 3 or MSIX hosts, so read "no menu" as "not detected" there. See [Surfaces](/desktop/surfaces). ## Use PowerShell PowerShell reads a bare `@token` as its splatting operator and drops the argument before agent-desktop sees it. Quote every ref: ```powershell agent-desktop set-value '@s8f3k2p9:e1' "hello" $env:AGENT_DESKTOP_HOME = "$env:TEMP\ad-scratch" $start = agent-desktop session start | ConvertFrom-Json $env:AGENT_DESKTOP_SESSION = $start.data.session_id ``` Use `$env:NAME = 'value'` instead of `export`, and `| ConvertFrom-Json` instead of `jq`. An unquoted ref fails with `INVALID_ARGS`. `cmd.exe` and bash need no quoting. ## Keys - `cmd` and `meta` are the Windows key, so `meta+c` is a shell shortcut. Write `ctrl+c` to copy. - `alt+f4`, `win+l`, `win+d`, `alt+tab`, `win+r`, `win+x`, `win+e`, `ctrl+shift+esc`, and their supersets are refused without `--force`. `ctrl+alt+delete` cannot be synthesized at all. - `type` is keystroke synthesis and needs `--headed`; use `set-value` headless. - `press --app` writes to the foreground input queue and cannot target one background process. ## Chromium and Electron Chromium 138 and later exposes a full UIA tree with no flag, but the first read starts the renderer's accessibility build and returns a small fraction of the tree. A cold launch takes 10 to 25 seconds to settle. - Pass a longer `--timeout-ms` to `snapshot` instead of concluding the tree is empty. - A settled tree with boundary nodes carrying `children_count` and no controls is a depth effect: drill into a boundary with `snapshot --root`. - For the web contents of an app you can launch fresh, `launch --cdp` opens a DevTools port for agent-browser; see [Chromium and Electron apps](/desktop/guides/chromium-and-electron). - A WPF window read before its automation peer exists can bind an empty generic provider. Take a new snapshot after a few seconds. ## Troubleshooting
Symptom Cause Fix
A Chromium or Electron app returns far fewer nodes than it shows The first read came before the renderer settled snapshot --timeout-ms 30000
PERM_DENIED with E_ACCESSDENIED The target runs at a higher integrity level Run the terminal elevated to the target's level
INVALID_ARGS on a ref in PowerShell An unquoted @ ref was eaten as splatting Quote the ref in single quotes
POLICY_DENIED from open-system-surface or a notification command The command raises shell chrome, which takes the foreground Pass --headed, or open the surface yourself first
wait --notification times out while toasts are posted The Action Center collects a toast only while open Hold the center open, or confirm with list-notifications while it is open
A hosted app reads as ApplicationFrameHost The app is suspended Activate the app so it resumes, then list again
The binary will not start SmartScreen, antivirus quarantine, or Smart App Control Choose More info, then Run anyway; restore a quarantined file after checking its SHA-256 checksum
Headed coordinates land offset on a scaled display A host process locked an incompatible DPI awareness Run from a plain terminal and prefer refs to --xy
More cases are in `agent-desktop skills get windows troubleshooting`. --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt)