Automation Operations (Desktop & Browser Automation)

ACTION REQUIRED (Execute immediately after reading)

1. NOW

   : Confirm whether the current task falls within the scope of this skill

2. NOW

   : Read

   ../tool-index.md

   to verify tool availability and actual paths

3. NEXT

   : Call bootstrap when tools are missing, do not guess paths

4. ACT

   : Enter the first step of the "workflow" and execute, do not stay in the confirmation state

Scope of Application

Browser Scenarios (Playwright / agent-browser)

• Open web pages and operate page elements (click, fill forms, submit)
• Crawl page content or take screenshots
• Automated login processes
• Interact with web pages during penetration testing (submit payloads, trigger XSS)
• Automated processing of captcha pages
• Batch form submission

Desktop Application Scenarios (OpenReverse)

• Operate Windows desktop applications (IDA Pro, x64dbg, Wireshark, etc.)
• Require vision-driven interaction (CUA mode)
• Require structured UI operations (UIA mode)
• Observe network traffic of desktop applications (built-in mitmproxy)
• Automated GUI operations for reverse engineering tools
• Black-box testing of desktop software

Division of Labor with Other Tools

• Target is a web page → Playwright
• Target is a Windows desktop application → OpenReverse
• Both are needed → Use in combination

Part 1: Browser Automation (Playwright / agent-browser)

Core Workflow

# 1. Open page
agent-browser open <url>

# 2. Get interactive elements (returns references like @e1, @e2...)
agent-browser snapshot -i

# 3. Operate elements using references
agent-browser click @e1
agent-browser fill @e2 "text"

# 4. Close after completion
agent-browser close

Command Reference

# Navigation
agent-browser open <url>
agent-browser close

# Page snapshot
agent-browser snapshot        # Complete accessibility tree
agent-browser snapshot -i     # Only interactive elements (recommended)

# Interactive operations
agent-browser click @e1
agent-browser fill @e2 "text"
agent-browser type @e2 "text"
agent-browser press Enter
agent-browser scroll down 500

# Get information
agent-browser get text @e1
agent-browser get title
agent-browser get url

# Wait
agent-browser wait @e1
agent-browser wait 2000
agent-browser wait --load networkidle

Notes

• Must execute

  agent-browser close

  , otherwise process leakage will occur
• Take snapshot before operation, do not guess element references
• Use

  wait --load networkidle

  to wait for page stability after submitting forms

Part 2: Desktop Application Automation (OpenReverse)

Overview

• UIA Mode: Windows UI Automation, structured desktop control operations
• CUA Mode: Vision-driven interaction (Computer Use Agent), suitable for complex GUIs
• Network Observation: Built-in mitmproxy proxy + local capture

Interaction Mode Selection

Network Observation Mode

Installation and Configuration

# 1. Clone the project
git clone https://github.com/zhexulong/openreverse.git
cd openreverse

# 2. Install dependencies
npm install

# 3. Connect to Agent host (Claude Code / Codex / Zed)
npm run init:agents -- --target=all /path/to/project

# 4. Install CUA runtime (if vision-driven mode is needed)
npm run install:cua-runtime
npm run doctor:cua-runtime

# 5. Install network observation dependencies (if packet capture is needed)
npm run install:mitmproxy
npm run doctor:network

Common Combinations

Reverse Engineering Scenario Examples

Scenario: Automate IDA Pro for batch analysis

1. Open IDA Pro using OpenReverse CUA mode
2. Automatically load the target binary
3. Wait for analysis completion
4. Export function list through UI operations
5. Simultaneously observe IDA's network behavior (such as Lumina requests) using network lane

Scenario: Automate x64dbg debugging

1. Start x64dbg using OpenReverse UIA mode
2. Load the target program
3. Set breakpoints
4. Run and observe register/memory changes
5. Take screenshots to save evidence

On-Demand Bootstrap

Automation Capability Boundaries

Bootstrap Trigger

• Playwright is missing for browser operations → Automatically bootstrap
• OpenReverse is needed for desktop operations → Guide users to install manually (provide complete steps)

OpenReverse Manual Installation Guide

⚠️ **OpenReverse is required for desktop application automation**

**Installation Steps**:
1. `git clone https://github.com/zhexulong/openreverse.git`
2. `cd openreverse && npm install`
3. `npm run init:agents -- --target=all <your project path>`
4. For vision mode: `npm run install:cua-runtime`
5. For network observation: `npm run install:mitmproxy`

**Verification**: `npm run doctor:cua-runtime` and `npm run doctor:network`

Routing Context

skills/SKILL.md

routing.md

• Captured requests need analysis →

  anything-analyzer

  or

  js-reverse

• JS debugging/Hook needed →

  jshookmcp

• Signature algorithm restoration needed →

  js-reverse

• Desktop application is a reverse engineering tool →

  ida-reverse/

js-reverse

ida-reverse

Task Completion Self-Check (MUST pass before claiming completion)

• Have I executed every step in the workflow (instead of just reading)?
• Have I used real tool paths based on

  tool-index

  ?
• Have I produced reproducible evidence (commands/scripts/screenshots/reports)?
• Have I completed and written back the Checklist items required by RULES?