Crawler Skill

Playwriter exploration -> CDP evidence capture -> Documentation -> Code generation

crawler

.hypercore/crawler/<ACTION>.json

crawler

• "Scrape product cards from this shop, inspect the API first, then generate a crawler."
• "Figure out how this logged-in dashboard loads data and document the cookies and headers."
• "Analyze this Cloudflare-protected site and recommend the safest crawl approach."

• "Open this site and click through the signup flow."
• "Rewrite this crawl runbook for readability."

• "Grab three prices from this public page right now." Prefer lightweight extraction unless the user asks for a reusable crawler or site-wide strategy.

1. Start with pre-crawl-checklist.md before making crawl or code decisions.
2. Use playwriter-commands.md when you need session control, page interaction, visual inspection, or selector validation (Playwright MCP = driving).
3. Use chrome-devtools-mcp.md when you need first-party Chrome DevTools fidelity for live network requests, console errors, performance traces, Lighthouse audits, or memory snapshots (Chrome DevTools MCP = debugging).
4. Use cdp-capture.md when you need structured network, cookie, token, storage, or rate-limit evidence with lower token cost than full Playwriter snapshots.
5. Use network-crawling.md when turning Playwriter / chrome-devtools-mcp / CDP evidence into

   API.md

   ,

   NETWORK.md

   , and raw evidence files.
6. Use selector-strategies.md when DOM extraction is still on the table.
7. Use crawling-patterns.md when pagination, authentication, lazy loading, or retries shape the approach.
8. Use anti-bot-checklist.md when the target shows blocks, CAPTCHA, Cloudflare, or explicit anti-detect requirements.
9. Use action-manifest.md when the run needs a durable state file under

   .hypercore/crawler/<ACTION>.json

   .
10. Use document-templates.md when writing

    .hypercore/crawler/[site]/

    artifacts.
11. Use code-templates.md only after the method is chosen and the discovery evidence is documented.

Mandatory Sequential Thinking

• Always use the

  sequential-thinking

  tool before starting crawl design, extraction strategy, or code generation decisions.
• Run

  sequential-thinking

  for each major phase: discovery, method selection, and implementation planning.
• If

  sequential-thinking

  is unavailable, stop and report the blocker instead of continuing without structured reasoning.

• Do discovery before code generation, selector lock-in, or auth assumptions.
• Use Playwriter to reproduce the user-visible flow, then prefer CDP for structured network/auth evidence capture.
• Prefer an API-backed crawler when CDP or fallback browser-network evidence shows a stable endpoint and manageable auth.
• Keep large DOM or accessibility snapshots rare; use them for structure checks and selector validation, not as the default capture surface.
• If CDP attach fails, document the limitation in

  ANALYSIS.md

  and use Playwriter interception only when the fallback evidence is still sufficient.
• Stop and report blockers when legal constraints, repeated

  403/429/503

  , CAPTCHA, or strong anti-bot signals make automation unsafe.
• Do not promise

  CRAWLER.ts

  until the method, auth material, and rate-limit posture are documented.

playwriter session new

accessibilitySnapshot

screenshotWithAccessibilityLabels

chrome-devtools-mcp

list_network_requests

list_console_messages

performance_start_trace

Network.*

Storage.*

Runtime.evaluate

.hypercore/crawler/[site]/

fetch

httpx

fetch

curl_cffi

chrome-devtools-mcp

playwriter

Runtime.Enable

.hypercore/crawler/<ACTION>.json

• ACTION.json

  preserves intent, current status, capture mode, blockers, output pointers, and the next step.

• .hypercore/crawler/[site-name]/

  preserves detailed evidence, analysis, and generated code for that site.

.hypercore/crawler/
├── <ACTION>.json              # durable action context
└── [site-name]/
    ├── ANALYSIS.md
    ├── SELECTORS.md
    ├── API.md
    ├── NETWORK.md
    ├── raw/
    │   ├── network-summary.json
    │   ├── auth-signals.json
    │   └── endpoint-candidates.json
    └── CRAWLER.ts

.hypercore/crawler/[site-name]/
├── ANALYSIS.md      # Site structure
├── SELECTORS.md     # DOM selectors
├── API.md           # API endpoints
├── NETWORK.md       # Auth/network details
├── raw/
│   ├── network-summary.json      # normalized request/response evidence
│   ├── auth-signals.json         # cookies/storage/header evidence
│   └── endpoint-candidates.json  # deduped API candidates
└── CRAWLER.ts       # Generated crawler code

• .hypercore/crawler/<ACTION>.json

  is required for reusable, blocked, or resumable crawl work.

• ANALYSIS.md

  is always required for reusable crawl work.

• SELECTORS.md

  is required when DOM extraction is used or kept as a fallback path.

• API.md

  is required when API discovery was attempted; document discovered endpoints or the absence of a usable API.

• NETWORK.md

  is required when cookies, tokens, headers, rate limits, or bot-detection signals affect the method.

• raw/network-summary.json

  ,

  raw/auth-signals.json

  , and

  raw/endpoint-candidates.json

  are recommended when CDP capture is available, and should back the human-readable docs instead of replacing them.

• CRAWLER.ts

  is required only after discovery evidence is written and the chosen method is justified.

• write

  ANALYSIS.md

  with the blocker, the evidence that triggered the stop, and the safest next step
• write

  NETWORK.md

  when auth signals, block responses, or anti-bot findings affected the decision
• write any available raw evidence files even when the run is blocked, so the stop is auditable
• update

  ACTION.json

  so

  status

  ,

  capture_mode

  , blockers, and output pointers match the blocked state
• omit

  CRAWLER.ts

  until the blocker is resolved or the method becomes safe to automate

✅ Playwriter session created
✅ `ACTION.json` created when the run is reusable, blocked, or resumable
✅ Structure analyzed with limited Playwriter snapshots
✅ CDP capture attempted for network/auth evidence
✅ raw evidence files recorded when CDP capture is available, or the fallback limitation documented when it is not
✅ Selector extraction validated
✅ Findings documented under .hypercore/crawler/
✅ Crawler code generated
✅ sequential-thinking trace recorded for major phases
✅ legal, rate-limit, and bot-detection blockers documented before scaling
✅ blocked runs reported explicitly when crawler code is unsafe or premature
✅ `ACTION.json` status and `site_dir` match the actual run outputs
✅ completed runs leave `ACTION.json.next_step` empty or terminal and point outputs at final files

# User: /crawler crawl products from https://shop.example.com

# 1. Create durable action context
# .hypercore/crawler/extract-products.json

# 2. Session
playwriter session new  # => 1
playwriter -s 1 -e "state.page = await context.newPage(); await state.page.goto('https://shop.example.com/products')"

# 3. Structure analysis
playwriter -s 1 -e "console.log(await accessibilitySnapshot({ page: state.page }))"
# => list "Products" [ref=e5]: listitem [ref=e6]: link "Product A" [ref=e7]

# 4. CDP capture
playwriter -s 1 -e $'
const client = await state.page.context().newCDPSession(state.page);
await client.send("Network.enable");
state.cdpHits = [];
client.on("Network.responseReceived", (event) => {
  if (event.response.url.includes("/api/")) state.cdpHits.push(event.response.url);
});
'
playwriter -s 1 -e "await state.page.evaluate(() => window.scrollTo(0, 9999))"
playwriter -s 1 -e "console.log(state.cdpHits)"
# => ["/api/products?page=2"]

# 5. Update extract-products.json -> status=running, capture_mode=cdp
# 6. Documentation -> .hypercore/crawler/shop-example-com/ + raw/network-summary.json
# 7. Generate API-based crawler