自动化操作 (Desktop & Browser Automation)
Automation Operations (Desktop & Browser Automation)
ACTION REQUIRED（读完后立刻执行）
ACTION REQUIRED (Execute immediately after reading)
- ：确认当前任务是否命中本 skill 的适用范围
- ：读取 ，校验工具可用性和实际路径
- ：缺工具时调用 bootstrap，不要猜路径
- ：进入"工作流"第一步并执行，不要停在确认状态
- : Confirm whether the current task falls within the scope of this skill
- : Read to verify tool availability and actual paths
- : Call bootstrap when tools are missing, do not guess paths
- : Enter the first step of the "workflow" and execute, do not stay in the confirmation state
Use this skill when the task belongs to the following scenarios:
浏览器场景（Playwright / agent-browser）
Browser Scenarios (Playwright / agent-browser)
- 打开网页并操作页面元素（点击、填表、提交）
- 爬取页面内容或截图
- 自动化登录流程
- 渗透测试中与 Web 页面交互（提交 payload、触发 XSS）
- 验证码页面的自动化处理
- 批量表单提交
- Open web pages and operate page elements (click, fill forms, submit)
- Crawl page content or take screenshots
- Automated login processes
- Interact with web pages during penetration testing (submit payloads, trigger XSS)
- Automated processing of captcha pages
- Batch form submission
桌面应用场景（OpenReverse）
Desktop Application Scenarios (OpenReverse)
- 操作 Windows 桌面应用（IDA Pro、x64dbg、Wireshark 等）
- 需要视觉驱动交互（CUA 模式）
- 需要结构化 UI 操作（UIA 模式）
- 桌面应用的网络流量观察（内置 mitmproxy）
- 自动化逆向工具的 GUI 操作
- 黑盒测试桌面软件
- Operate Windows desktop applications (IDA Pro, x64dbg, Wireshark, etc.)
- Require vision-driven interaction (CUA mode)
- Require structured UI operations (UIA mode)
- Observe network traffic of desktop applications (built-in mitmproxy)
- Automated GUI operations for reverse engineering tools
- Black-box testing of desktop software
与其他工具的分工
Division of Labor with Other Tools
| 场景 | 用什么 |
|---|
| 操作网页（浏览器内） | Playwright / agent-browser |
| 操作桌面应用（Windows GUI） | OpenReverse |
| 抓包分析、HTTP 请求捕获 | anything-analyzer 或 OpenReverse network lane |
| JS 断点、Hook、CDP 调试 | jshookmcp |
| 定位签名算法、补环境复现 | js-reverse |
简单判断：
- 目标是网页 → Playwright
- 目标是 Windows 桌面应用 → OpenReverse
- 两者都需要 → 组合使用
| Scenario | What to Use |
|---|
| Operate web pages (within browser) | Playwright / agent-browser |
| Operate desktop applications (Windows GUI) | OpenReverse |
| Packet capture analysis, HTTP request capture | anything-analyzer or OpenReverse network lane |
| JS breakpoints, Hook, CDP debugging | jshookmcp |
| Locate signature algorithms, reproduce environments | js-reverse |
Simple Judgment:
- Target is a web page → Playwright
- Target is a Windows desktop application → OpenReverse
- Both are needed → Use in combination
Part 1: 浏览器自动化（Playwright / agent-browser）
Part 1: Browser Automation (Playwright / agent-browser)
2. 获取可交互元素（返回 @e1, @e2... 引用）
2. Get interactive elements (returns references like @e1, @e2...)
agent-browser snapshot -i
agent-browser snapshot -i
3. 用引用操作元素
3. Operate elements using references
agent-browser click @e1
agent-browser fill @e2 "text"
agent-browser click @e1
agent-browser fill @e2 "text"
4. 完成后关闭
4. Close after completion
agent-browser open <url>
agent-browser close
agent-browser open <url>
agent-browser close
agent-browser snapshot # 完整无障碍树
agent-browser snapshot -i # 仅可交互元素（推荐）
agent-browser snapshot # Complete accessibility tree
agent-browser snapshot -i # Only interactive elements (recommended)
交互操作
Interactive operations
agent-browser click @e1
agent-browser fill @e2 "text"
agent-browser type @e2 "text"
agent-browser press Enter
agent-browser scroll down 500
agent-browser click @e1
agent-browser fill @e2 "text"
agent-browser type @e2 "text"
agent-browser press Enter
agent-browser scroll down 500
agent-browser get text @e1
agent-browser get title
agent-browser get url
agent-browser get text @e1
agent-browser get title
agent-browser get url
agent-browser wait @e1
agent-browser wait 2000
agent-browser wait --load networkidle
agent-browser wait @e1
agent-browser wait 2000
agent-browser wait --load networkidle
- 必须执行 ，否则进程泄漏
- 操作前先 snapshot，不要猜元素引用
- 提交表单后用 等页面稳定
- Must execute , otherwise process leakage will occur
- Take snapshot before operation, do not guess element references
- Use to wait for page stability after submitting forms
Part 2: 桌面应用自动化（OpenReverse）
Part 2: Desktop Application Automation (OpenReverse)
OpenReverse 是面向 AI Agent 的桌面交互与证据采集框架，支持：
- UIA 模式：Windows UI Automation，结构化桌面控件操作
- CUA 模式：视觉驱动交互（Computer Use Agent），适合复杂 GUI
- 网络观察：内置 mitmproxy 代理 + 本地抓取
OpenReverse is a desktop interaction and evidence collection framework for AI Agent, supporting:
- UIA Mode: Windows UI Automation, structured desktop control operations
- CUA Mode: Vision-driven interaction (Computer Use Agent), suitable for complex GUIs
- Network Observation: Built-in mitmproxy proxy + local capture
交互模式选择
Interaction Mode Selection
| 模式 | 适合场景 | 底层 |
|---|
| UIA | 目标应用有标准 Windows 控件（按钮、文本框、列表） | Windows UI Automation API |
| CUA | 目标应用 UI 复杂或非标准控件（IDA 的反汇编视图、自定义渲染界面） | 视觉识别 + 鼠标键盘 |
| Mode | Suitable Scenarios | Underlying Technology |
|---|
| UIA | Target application has standard Windows controls (buttons, text boxes, lists) | Windows UI Automation API |
| CUA | Target application has complex or non-standard UI (IDA's disassembly view, custom rendered interfaces) | Visual recognition + mouse and keyboard |
网络观察模式
Network Observation Mode
| 模式 | 适合场景 |
|---|
| Proxy Lane | 目标应用可以配置代理（推荐） |
| Local Lane | 目标应用无法走代理，需要本地抓取 |
| Mode | Suitable Scenarios |
|---|
| Proxy Lane | Target application can be configured with proxy (recommended) |
| Local Lane | Target application cannot use proxy, need local capture |
安装与配置
Installation and Configuration
1. Clone 项目
1. Clone the project
2. 安装依赖
2. Install dependencies
3. 接入 Agent 宿主（Claude Code / Codex / Zed）
3. Connect to Agent host (Claude Code / Codex / Zed)
npm run init:agents -- --target=all /path/to/project
npm run init:agents -- --target=all /path/to/project
4. 安装 CUA runtime（如果需要视觉驱动模式）
4. Install CUA runtime (if vision-driven mode is needed)
npm run install:cua-runtime
npm run doctor:cua-runtime
npm run install:cua-runtime
npm run doctor:cua-runtime
5. 安装网络观察依赖（如果需要抓包）
5. Install network observation dependencies (if packet capture is needed)
npm run install:mitmproxy
npm run doctor:network
npm run install:mitmproxy
npm run doctor:network
| 需求 | 配置 |
|---|
| 只操作桌面应用 | UIA 或 CUA，不接网络 lane |
| 操作桌面应用 + 抓包 | UIA/CUA + proxy lane |
| 操作桌面应用 + 本地抓取 | UIA/CUA + local lane |
| Requirement | Configuration |
|---|
| Only operate desktop applications | UIA or CUA, no network lane connected |
| Operate desktop applications + packet capture | UIA/CUA + proxy lane |
| Operate desktop applications + local capture | UIA/CUA + local lane |
逆向场景示例
Reverse Engineering Scenario Examples
text
场景：自动化操作 IDA Pro 进行批量分析
1. 用 OpenReverse CUA 模式打开 IDA Pro
2. 自动加载目标二进制
3. 等待分析完成
4. 通过 UI 操作导出函数列表
5. 同时用 network lane 观察 IDA 的网络行为（如 Lumina 请求）
text
场景：自动化操作 x64dbg 调试
1. 用 OpenReverse UIA 模式启动 x64dbg
2. 加载目标程序
3. 设置断点
4. 运行并观察寄存器/内存变化
5. 截图保存证据
text
Scenario: Automate IDA Pro for batch analysis
1. Open IDA Pro using OpenReverse CUA mode
2. Automatically load the target binary
3. Wait for analysis completion
4. Export function list through UI operations
5. Simultaneously observe IDA's network behavior (such as Lumina requests) using network lane
text
Scenario: Automate x64dbg debugging
1. Start x64dbg using OpenReverse UIA mode
2. Load the target program
3. Set breakpoints
4. Run and observe register/memory changes
5. Take screenshots to save evidence
按需自举（On-Demand Bootstrap）
On-Demand Bootstrap
自动化能力边界
Automation Capability Boundaries
| 工具 | 可自动安装 | 安装方式 | 说明 |
|---|
| Playwright | ✓ | npm + npx playwright install | 浏览器自动化引擎 |
| agent-browser CLI | ✓ | npm install -g agent-browser | 浏览器操作 CLI |
| Node.js | ✓ | winget | 前置依赖 |
| OpenReverse | ✗ | 手动 clone + npm install | 实验阶段，依赖较重 |
| mitmproxy | ✗ | 手动安装 | OpenReverse 网络观察依赖 |
| Tool | Auto-installable | Installation Method | Description |
|---|
| Playwright | ✓ | npm + npx playwright install | Browser automation engine |
| agent-browser CLI | ✓ | npm install -g agent-browser | Browser operation CLI |
| Node.js | ✓ | winget | Pre-requisite dependency |
| OpenReverse | ✗ | Manual clone + npm install | Experimental stage, heavy dependencies |
| mitmproxy | ✗ | Manual installation | OpenReverse network observation dependency |
- 浏览器操作缺 Playwright → 自动 bootstrap
- 桌面操作需要 OpenReverse → 引导用户手动安装（给出完整步骤）
- Playwright is missing for browser operations → Automatically bootstrap
- OpenReverse is needed for desktop operations → Guide users to install manually (provide complete steps)
OpenReverse 手动安装引导
OpenReverse Manual Installation Guide
如果 AI 检测到需要桌面应用自动化但 OpenReverse 未安装：
markdown
⚠️ **需要 OpenReverse 进行桌面应用自动化**
**安装步骤**：
1. `git clone https://github.com/zhexulong/openreverse.git`
2. `cd openreverse && npm install`
3. `npm run init:agents -- --target=all <你的项目路径>`
4. 如需视觉模式：`npm run install:cua-runtime`
5. 如需网络观察：`npm run install:mitmproxy`
**验证**：`npm run doctor:cua-runtime` 和 `npm run doctor:network`
If AI detects that desktop application automation is needed but OpenReverse is not installed:
markdown
⚠️ **OpenReverse is required for desktop application automation**
**Installation Steps**:
1. `git clone https://github.com/zhexulong/openreverse.git`
2. `cd openreverse && npm install`
3. `npm run init:agents -- --target=all <your project path>`
4. For vision mode: `npm run install:cua-runtime`
5. For network observation: `npm run install:mitmproxy`
**Verification**: `npm run doctor:cua-runtime` and `npm run doctor:network`
上游入口:
（总控）、
适用场景: 任何需要自动化操作浏览器或桌面应用的任务
下游出口:
- 抓到的请求需要分析 → 或
- 需要 JS 调试/Hook →
- 需要还原签名算法 →
- 桌面应用是逆向工具 →
同级关联模块:
（浏览器操作后可能需要分析 JS）、
（OpenReverse 可以自动化操作 IDA GUI）
Upstream Entries:
(master control),
Applicable Scenarios: Any task requiring automated operation of browsers or desktop applications
Downstream Exports:
- Captured requests need analysis → or
- JS debugging/Hook needed →
- Signature algorithm restoration needed →
- Desktop application is a reverse engineering tool →
Peer Related Modules:
(JS analysis may be needed after browser operations),
(OpenReverse can automate IDA GUI operations)
任务完成自检（声称完成前 MUST 通过）
Task Completion Self-Check (MUST pass before claiming completion)