screenclaw

Core Rules

1. Coordinates must come from the

   XxY

   grid intersections on ScreenClaw screenshots; internal visual coordinates or speculative coordinates are not allowed.
2. A fixed

   session_id

   is used for each session, and all public calls only use

   scripts/screenclaw.py|ps1|sh

   .
3. Read

   references/api/{endpoint}.md

   before calling an endpoint; if there are parameter errors or API errors, return to the document to correct them.
4. For first-time dynamic coordinates, high-risk coordinates, or when the target or numbers are unclear, you must first crop and zoom or verify with a marker.
5. API success only means the instruction has been sent, not that the interface has achieved the goal; you must take a screenshot to verify after the operation.
6. When receiving

   SELF_CHECK_REQUIRED

   , read and execute the self-check procedure in

   references/self_check.md

   to reload key context, then summarize the executed content with

   self_check

   and retry taking screenshots.

Mental Model

1. External facts take priority: screenshots, markers, API returns, and window lists are facts; your intuition and preset conclusions must obey these facts. If the marker is not on the target, it means the coordinates are wrong.
2. Falsify first, then operate: candidate coordinates are assumed to be wrong by default. Use crop or marker to find counterexamples, confirm where the marked point actually falls, then operate.
3. Return to screenshots when failing: when operations fail, results do not meet expectations, or continuous fine-tuning is ineffective, first re-take screenshots, crop, read documents and parameters, do not continue guessing coordinates or repeating clicks.

Fixed Work Cycle

Understand Goal -> health -> config -> get_window_list -> screenshot -> Read Coordinates -> Marker Verification -> Operation -> Screenshot Verification

1. Initialization

• Respond according to the user's language.
• Read

  references/config.md

  to get the rules for

  api_url

  ,

  token

  ,

  ai_app_type

  , and

  session_id

  .
• Read

  scripts/README.md

  to understand the unified script entry and dot path format.
• Call

  health

  to confirm the service is available.
• Search

  references/scenarios/

  for matching templates.
• For complex, multi-step, high-risk tasks, maintain a short plan of 2-5 steps first; short single-step tasks do not require creating to-dos.

2. Get Target Window

• Call

  get_window_list

  to find the main window and possible child windows.
• When a new process or window is uncertain, take screenshots of candidate windows and record window content and available

  window_id/main_window_id

  .
• When subsequent operations fail, first check if the wrong window was selected, then switch operation modes.

3. Screenshot and Read Coordinates

• Use

  screenshot coordinate_type=grid

  to locate coordinates.
• Use

  coordinate_type=no

  to analyze content or show images to users.
• By default, rely on server-side adaptive grid parameters first.
• When the target is not covered by intersections, read

  references/api/screenshot.md

  to adjust

  grid.density_x/y

  .
• When numbers or elements are unclear, use

  crop_zoom_screenshot

  or adjust number parameters.
• For first-time dynamic coordinates or high-risk coordinates, first use

  crop_zoom_screenshot

  to see details clearly, then verify in reverse with

  marker.0.x/y

  .
• For marker verification, first find where the marked point on the image actually falls, describe what is there, then judge whether it matches the target; do not assume candidate coordinates are correct first.

4. Operation and Verification

• Prioritize the

  background

  operation mode.
• Only consider

  hijack

  when

  background

  is invalid or physical input is required.
• For scenarios requiring continuous physical input such as user-initiated requests, real-time game operations, and Chinese input method candidate panels, read

  references/api/delegated.md

  before entering delegation mode.
• Use single-step calls during the exploration phase; use

  batch

  only when the process is stable and instant observation of hover/menu/operation results is needed.
• Take a screenshot to verify after each operation; if verification fails, return to screenshot and coordinate reading.
• When receiving

  SELF_CHECK_REQUIRED

  , you must update the current plan or next action.

Coordinate Concept

XxY

50x35

x

API Index

Read the corresponding document

references/api/{endpoint}.md

before executing the API

references/api/health.md

references/api/get_window_list.md

references/api/screenshot.md

references/api/crop_zoom_screenshot.md

references/api/scroll_screenshot.md

references/api/click.md

references/api/long_press.md

references/api/swipe.md

references/api/drag.md

references/api/scroll.md

references/api/right_click.md

references/api/hover.md

references/api/mouse_move.md

references/api/input_text.md

references/api/press_key.md

references/api/wait.md

references/api/batch.md

references/api/delegated.md

Script Downgrade

scripts/screenclaw.py -> scripts/screenclaw.ps1 -> scripts/screenclaw.sh -> curl

Reference Documents

• references/config.md

  - Connection configuration,

  ai_app_type

  ,

  session_id

• scripts/README.md

  - Unified script entry and dot path format

• references/self_check.md

  - Long-term self-check reload checklist

• references/api/*.md

  - Parameters and troubleshooting for each API

• references/scenarios/

  - Scenario templates and application knowledge