agent-native

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

agent-native

macOS native app automation via the Accessibility tree. Like agent-browser, but for desktop apps.

通过无障碍（Accessibility）树实现macOS原生应用自动化。类似agent-browser，但针对桌面应用。

Prerequisites

前置条件

macOS 13+ with Accessibility permissions granted to your terminal

Binary at

agent-native

(install via

swift build -c release && cp .build/release/agent-native /usr/local/bin/

)

macOS 13及以上系统，且已为终端授予无障碍（Accessibility）权限

可执行文件

agent-native

（安装方式：

swift build -c release && cp .build/release/agent-native /usr/local/bin/

）

Core Workflow

核心工作流

apps -> pick/open target -> snapshot -> interact by ref -> re-snapshot

Always start with
agent-native apps
to see what's already running. Prefer reusing an already-open app (e.g. if a browser is already open, use it instead of opening a different one).
- Known browsers: Safari, Arc, Chrome, Firefox, Helium. Any of these can be used for web tasks.
Only call
```
agent-native open <app>
```
if the target app isn't already running.
Snapshot, interact, and re-snapshot as needed.

bash

agent-native apps                            # Check what's already running
agent-native snapshot Safari -i              # Use the already-open browser
agent-native click @n5                       # Interact using refs
agent-native snapshot Safari -i              # Re-snapshot after UI changes

Always re-snapshot after actions that change the UI. Refs are invalidated when UI structure changes.

apps -> 选择/打开目标应用 -> 快照 -> 通过引用交互 -> 重新快照

始终从
agent-native apps
开始，查看当前运行的应用。优先复用已打开的应用（例如，如果浏览器已打开，直接使用它而非打开新的实例）。
- 支持的浏览器：Safari、Arc、Chrome、Firefox、Helium。这些浏览器均可用于网页任务。
仅当目标应用未运行时，才调用
```
agent-native open <app>
```
。
根据需要进行快照、交互和重新快照。

bash

agent-native apps                            # 查看当前运行的应用
agent-native snapshot Safari -i              # 使用已打开的浏览器
agent-native click @n5                       # 通过引用进行交互
agent-native snapshot Safari -i              # UI变化后重新快照

在任何会改变UI的操作后，务必重新快照。当UI结构变化时，旧的引用会失效。

Commands

命令列表

Navigation

导航操作

bash

agent-native open <app>                          # Open/activate by name or bundle ID
agent-native open "System Settings"
agent-native open com.apple.Safari               # Already-running apps just activate

bash

agent-native open <app>                          # 通过名称或Bundle ID打开/激活应用
agent-native open "System Settings"
agent-native open com.apple.Safari               # 已运行的应用仅会被激活

Screenshot

截图

bash

agent-native screenshot <app> [path]             # Capture app's frontmost window
agent-native screenshot Slack                    # Saves to /tmp/agent-native-screenshot.png
agent-native screenshot Slack /tmp/slack.png     # Custom path
agent-native screenshot Slack --json             # {"path": "...", "width": ..., "height": ...}

bash

agent-native screenshot <app> [path]             # 捕获应用的前置窗口
agent-native screenshot Slack                    # 保存至/tmp/agent-native-screenshot.png
agent-native screenshot Slack /tmp/slack.png     # 自定义保存路径
agent-native screenshot Slack --json             # 返回格式：{"path": "...", "width": ..., "height": ...}

Keystroke Sending

发送按键

bash

agent-native key <app> <keys...>                 # Send keystrokes to an app
agent-native key Slack "Hello world"             # Type text
agent-native key Slack cmd+k                     # Send Cmd+K
agent-native key Slack escape                    # Special keys: escape, return, tab, delete, up, down, left, right, space
agent-native key Slack cmd+a delete              # Chain multiple keys
agent-native key Calculator 5 + 3 return         # Multiple keystrokes

Modifiers:

cmd

command

ctrl

control

alt

option

opt

shift

bash

agent-native key <app> <keys...>                 # 向应用发送按键
agent-native key Slack "Hello world"             # 输入文本
agent-native key Slack cmd+k                     # 发送Cmd+K组合键
agent-native key Slack escape                    # 特殊按键：escape、return、tab、delete、up、down、left、right、space
agent-native key Slack cmd+a delete              # 链式发送多个按键
agent-native key Calculator 5 + 3 return         # 发送多组按键

修饰键：

cmd

command

、

ctrl

control

、

alt

option

opt

、

shift

Paste File

粘贴文件

bash

agent-native paste <app> <path>                  # Copy file to clipboard and Cmd+V into app
agent-native paste Slack /tmp/screenshot.png     # Paste image into Slack
agent-native paste Slack ./report.pdf            # Paste any file

bash

agent-native paste <app> <path>                  # 将文件复制到剪贴板并执行Cmd+V粘贴至应用
agent-native paste Slack /tmp/screenshot.png     # 将图片粘贴到Slack
agent-native paste Slack ./report.pdf            # 粘贴任意类型文件

Snapshot (recommended for agents)

快照（推荐Agent使用）

bash

agent-native snapshot <app>                      # Full AX tree with refs
agent-native snapshot <app> -i                   # Interactive elements only (recommended)
agent-native snapshot <app> -i -c                # Interactive + compact
agent-native snapshot <app> -d 3                 # Limit depth
agent-native snapshot <app> -i --json            # JSON for parsing

Flag	Description
`-i`	Interactive elements only
`-c`	Compact -- remove empty structural elements
`-d <n>`	Limit tree depth
`--json`	JSON output

bash

agent-native snapshot <app>                      # 带引用的完整AX树
agent-native snapshot <app> -i                   # 仅显示可交互元素（推荐使用）
agent-native snapshot <app> -i -c                # 仅显示可交互元素并启用紧凑模式
agent-native snapshot <app> -d 3                 # 限制树的深度
agent-native snapshot <app> -i --json            # 输出JSON格式用于解析

参数	说明
`-i`	仅显示可交互元素
`-c`	紧凑模式 -- 移除空的结构元素
`-d <n>`	限制树的深度
`--json`	输出JSON格式

Interaction (use @refs from snapshot)

交互操作（使用快照中的@引用）

bash

agent-native click @n2                           # Click / press
agent-native fill @n3 "text"                     # Clear field and type
agent-native type @n3 "text"                     # Type without clearing
agent-native select @n5 "Option A"               # Select from dropdown
agent-native check @n4                           # Check checkbox (idempotent)
agent-native uncheck @n4                         # Uncheck checkbox (idempotent)
agent-native focus @n3                           # Focus element
agent-native hover @n2                           # Move cursor to element
agent-native action @n7 AXIncrement              # Any AX action

bash

agent-native click @n2                           # 点击/按下元素
agent-native fill @n3 "text"                     # 清空输入框后输入文本
agent-native type @n3 "text"                     # 追加输入文本（不清空原有内容）
agent-native select @n5 "Option A"               # 从下拉菜单中选择选项
agent-native check @n4                           # 勾选复选框（幂等操作）
agent-native uncheck @n4                         # 取消勾选复选框（幂等操作）
agent-native focus @n3                           # 聚焦到元素
agent-native hover @n2                           # 将光标移动到元素上
agent-native action @n7 AXIncrement              # 执行任意AX操作

Read State

读取状态

bash

agent-native get text @n1                        # Get text / title / label
agent-native get value @n3                       # Get input value
agent-native get attr @n2 AXEnabled              # Get any AX attribute
agent-native get title Safari                    # Frontmost window title
agent-native is enabled @n5                      # true / false
agent-native is focused @n3                      # true / false

bash

agent-native get text @n1                        # 获取文本/标题/标签
agent-native get value @n3                       # 获取输入框的值
agent-native get attr @n2 AXEnabled              # 获取任意AX属性
agent-native get title Safari                    # 获取前置窗口的标题
agent-native is enabled @n5                      # 返回true/false
agent-native is focused @n3                      # 返回true/false

Discovery

发现功能

bash

agent-native apps                                # List running GUI apps
agent-native apps --format json                  # JSON output
agent-native find <app> --role AXButton          # Find elements by filter
agent-native find <app> --title "Submit"         # Find by title
agent-native inspect @n3                         # All attributes and actions
agent-native tree <app> --depth 3                # Raw AX tree (no refs)

bash

agent-native apps                                # 列出当前运行的GUI应用
agent-native apps --format json                  # 输出JSON格式
agent-native find <app> --role AXButton          # 通过筛选条件查找元素
agent-native find <app> --title "Submit"         # 通过标题查找元素
agent-native inspect @n3                         # 查看元素的所有属性和可执行操作
agent-native tree <app> --depth 3                # 原始AX树（无引用）

Wait

等待操作

bash

agent-native wait <app> --title "Apply" --timeout 5
agent-native wait <app> --role AXSheet --timeout 10

bash

agent-native wait <app> --title "Apply" --timeout 5
agent-native wait <app> --role AXSheet --timeout 10

Browser & Electron Enhanced Access

浏览器与Electron应用增强支持

Chromium browsers (Arc, Chrome, Edge, Brave, Vivaldi) and Electron apps don't expose web DOM content in the macOS AX tree by default.

snapshot

now auto-detects these apps and enhances access automatically.

Priority chain: CDP read → AX-enhanced interact → keyboard fallback → screenshot

Chromium系浏览器（Arc、Chrome、Edge、Brave、Vivaldi）和Electron应用默认不会在macOS AX树中暴露网页DOM内容。现在

snapshot

命令会自动检测这些应用并自动启用增强访问。

优先级顺序：CDP读取 → AX增强交互 → 键盘操作 fallback → 截图

Automatic Detection in

snapshot

snapshot

中的自动检测

bash

agent-native snapshot Arc -i                      # Auto-detects Chromium, enables AX enhancement
agent-native snapshot "VS Code" -i                # Auto-detects Electron, enables AX enhancement

When

snapshot

detects a Chromium/Electron app:

CDP mode (richest): If browser was launched with
```
--remote-debugging-port
```
, snapshot reads the full web accessibility tree via Chrome DevTools Protocol
AX-enhanced mode (fallback): Sets
```
AXEnhancedUserInterface
```
to force the app to build its accessibility tree, then walks it normally

bash

agent-native snapshot Arc -i                      # 自动检测Chromium，启用AX增强
agent-native snapshot "VS Code" -i                # 自动检测Electron，启用AX增强

当

snapshot

检测到Chromium/Electron应用时：

CDP模式（功能最丰富）：如果浏览器是通过
```
--remote-debugging-port
```
启动的，快照会通过Chrome DevTools Protocol读取完整的网页无障碍树
AX增强模式（备选方案）：设置
```
AXEnhancedUserInterface
```
强制应用构建无障碍树，然后正常遍历

CDP Mode (richest web content)

CDP模式（支持最完整的网页内容）

Launch your browser with CDP enabled for the best results:

bash

undefined

启动浏览器时开启CDP以获得最佳效果：

bash

undefined

Launch Chrome with CDP

启动开启CDP的Chrome

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

Snapshot auto-detects CDP on ports 9222/9229

Snapshot会自动检测9222/9229端口的CDP

agent-native snapshot Chrome -i

Or specify a port explicitly

或者显式指定端口

agent-native snapshot Chrome -i --port 9222

undefined

agent-native snapshot Chrome -i --port 9222

undefined

Manual Persistent Control

手动持久化控制

bash

agent-native ax-enable Arc                        # Persistently enable AXEnhancedUserInterface
agent-native snapshot Arc -i                      # Now shows web page elements
agent-native ax-disable Arc                       # Restore when done

bash

agent-native ax-enable Arc                        # 持久化启用AXEnhancedUserInterface
agent-native snapshot Arc -i                      # 现在会显示网页元素
agent-native ax-disable Arc                       # 使用完成后恢复设置

Fallback: Keyboard Shortcuts + Screenshots

备选方案：快捷键 + 截图

When AX tree is still sparse (some Electron apps), fall back to keyboard-driven interaction:

bash

agent-native key Slack cmd+k                     # Open quick switcher
agent-native key Slack "channel name" return     # Type and confirm
agent-native screenshot Slack /tmp/slack.png     # Visual confirmation
agent-native get title Slack                     # Check navigation state

当AX树内容仍较少时（部分Electron应用），可 fallback 到键盘驱动的交互：

bash

agent-native key Slack cmd+k                     # 打开快速切换器
agent-native key Slack "channel name" return     # 输入频道名称并确认
agent-native screenshot Slack /tmp/slack.png     # 视觉确认
agent-native get title Slack                     # 检查导航状态

Pasting files into apps

向应用粘贴文件

bash

agent-native paste Slack /path/to/image.png      # Copies to clipboard and pastes in one step

bash

agent-native paste Slack /path/to/image.png      # 一步完成复制到剪贴板并粘贴的操作

Tips

小贴士

Always use
-i
with snapshot -- full trees are noisy.
Re-snapshot after navigation -- old refs may not resolve after UI changes.
check
/
uncheck
are idempotent -- they read current state first.
fill
clears first,
type
appends.
AX tree peers into browsers -- Safari/Chrome expose web content as AX nodes.
Electron apps need keyboard shortcuts -- use System Events keystrokes when AX is sparse.

始终在snapshot中使用
-i
参数 -- 完整的树内容过于繁杂。
导航后重新快照 -- UI变化后旧的引用可能无法解析。
check
/
uncheck
是幂等操作 -- 它们会先读取当前状态。
fill
会先清空内容，
type
是追加内容。
AX树可穿透浏览器 -- Safari/Chrome会将网页内容暴露为AX节点。
Electron应用需使用快捷键 -- 当AX内容较少时，使用系统事件按键操作。

agent-native

Original

Translation

agent-native

agent-native

Prerequisites

前置条件

Core Workflow

核心工作流

Commands

命令列表

Navigation

导航操作

Screenshot

截图

Keystroke Sending

发送按键

Paste File

粘贴文件

Snapshot (recommended for agents)

快照（推荐Agent使用）

Interaction (use @refs from snapshot)

交互操作（使用快照中的@引用）

Read State

读取状态

Discovery

发现功能

Wait

等待操作

Browser & Electron Enhanced Access

浏览器与Electron应用增强支持

Automatic Detection in snapshot

snapshot中的自动检测

CDP Mode (richest web content)

CDP模式（支持最完整的网页内容）

Launch Chrome with CDP

启动开启CDP的Chrome

Snapshot auto-detects CDP on ports 9222/9229

Snapshot会自动检测9222/9229端口的CDP

Or specify a port explicitly

或者显式指定端口

Manual Persistent Control

手动持久化控制

Fallback: Keyboard Shortcuts + Screenshots

备选方案：快捷键 + 截图

Pasting files into apps

向应用粘贴文件

Tips

小贴士

Automatic Detection in
`snapshot`

`snapshot`
中的自动检测