Computer Control Skill

When To Use

• Automating GUI-based workflows that lack CLI alternatives
• Testing web applications through visual interaction
• Filling forms, navigating menus, or interacting with desktop apps
• Building automation pipelines that need visual verification

When NOT To Use

• Tasks achievable through CLI or API (no GUI needed)
• Browser automation better served by Playwright or CDP

Why this stays opt-in. Per docs/inclusive-defaults.md (TRUE-exception category 4), Computer Use takes screenshots and synthesizes keyboard/mouse input — cross-process side effects that must always be explicitly invoked, never default-on.

Architecture

1. Display Toolkit (

   phantom.display

   ) - executes OS-level actions via xdotool/scrot on the real or virtual display
2. Agent Loop (

   phantom.loop

   ) - manages the conversation cycle between Claude API and the display toolkit
3. CLI (

   phantom.cli

   ) - command-line interface for running tasks or checking environment readiness

User Task
    |
    v
Agent Loop  <---->  Claude API (beta)
    |                   |
    v                   v
Display Toolkit    tool_use responses
    |              (click, type, screenshot)
    v
OS Commands (xdotool, scrot)
    |
    v
Display (X11 / Xvfb / WSLg)

Quick Start

Check environment

cd plugins/phantom
uv run python -m phantom.cli --check

Run a task

export ANTHROPIC_API_KEY="sk-ant-..."
uv run python -m phantom.cli "Open Firefox and search for Claude AI"

Use in Python

from phantom.display import DisplayConfig, DisplayToolkit
from phantom.loop import LoopConfig, run_loop

result = run_loop(
    task="Take a screenshot of the desktop",
    api_key="sk-ant-...",
    loop_config=LoopConfig(
        model="claude-sonnet-4-6",
        max_iterations=10,
    ),
    display_config=DisplayConfig(width=1920, height=1080),
)

print(f"Done in {result.iterations} iterations")
print(result.final_text)

API Versions

computer_20251124

computer-use-2025-11-24

computer_20250124

computer-use-2025-01-24

resolve_tool_version()

Available Actions

• screenshot

  - capture display

• left_click

  - click at

  [x, y]

• type

  - type text string

• key

  - press key combo (e.g.,

  ctrl+s

  )

• mouse_move

  - move cursor

• scroll

  - scroll with direction and amount

• left_click_drag

  - drag between coordinates

• right_click

  ,

  middle_click

  ,

  double_click

  ,

  triple_click

• hold_key

  - hold key for duration

• wait

  - pause between actions

• zoom

  - inspect screen region at full resolution

Safety

1. Use a sandbox: Run in Docker or a VM, not your main OS
2. Limit access: Do not provide login credentials unless necessary, and never for banking or sensitive services
3. Set iteration caps: Always use

   max_iterations

   to prevent runaway API costs
4. Human approval: For actions with real-world consequences, add confirmation callbacks via

   on_action

5. Close sensitive apps: Claude sees the full screen via screenshots; close anything private before starting

Environment Requirements

sudo apt install xdotool scrot xclip

# Install Xvfb for virtual display
sudo apt install xvfb xdotool scrot xclip
Xvfb :1 -screen 0 1920x1080x24 &
export DISPLAY=:1

Prompting Tips

1. Be specific about each step of the task
2. Add "After each step, take a screenshot and verify" to catch mistakes early
3. Use keyboard shortcuts when UI elements are hard to click
4. Provide example screenshots for repeatable workflows
5. Set a system prompt with domain-specific instructions