Back to Details

crw-crawl

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

crw-crawl — bulk page extraction

crw-crawl — 批量页面提取

When to use

适用场景

  • You need content from many pages under a site or section, not just one.
  • Step 4 in the crw ladder: if you only need a handful of known URLs, use crw-scrape in a loop instead — it's simpler and gives you content immediately. Use crawl when the set of URLs is unknown or large.
  • Always map first (crw-map) to estimate page count before committing. A misconfigured crawl on a 50 000-page site is expensive; a map call is cheap.
  • Start conservative: 
    depth 1, limit 10
    . Scale up once you verify scope.
  • 你需要获取整个站点或指定板块下多个页面的内容，而非单个页面。
  • 属于crw工作流阶梯的第4步：如果你只需要少量已知URL的内容，建议循环使用crw-scrape——它更简单，能立即返回内容。当URL集合未知或数量庞大时，再使用crawl功能。
  • 爬取前务必先执行映射(crw-map)来预估页面数量。对一个包含50000个页面的站点进行错误配置的爬取成本很高，而映射调用成本极低。
  • 初始配置要保守：
    depth 1, limit 10
    。确认范围无误后再扩大规模。

Quick start

快速开始

CLI (synchronous streaming output):
bash
crw crawl "https://docs.example.com" -d 2 -l 50         # markdown to stdout
crw crawl "https://docs.example.com/api" -d 1 -l 20 --format json
crw crawl "https://example.com" --js --rate-limit 1.0 --concurrency 3
MCP (async — returns a job ID, poll for results):
undefined
CLI（同步流式输出）：
bash
crw crawl "https://docs.example.com" -d 2 -l 50         # markdown输出到标准输出
crw crawl "https://docs.example.com/api" -d 1 -l 20 --format json
crw crawl "https://example.com" --js --rate-limit 1.0 --concurrency 3
MCP（异步——返回任务ID，轮询获取结果）：
undefined

Start the crawl

启动爬取

crw_crawl(url="https://docs.example.com", maxDepth=2, maxPages=50) → { "id": "a1b2c3d4-..." }
crw_crawl(url="https://docs.example.com", maxDepth=2, maxPages=50) → { "id": "a1b2c3d4-..." }

Poll until status == "completed"

轮询直到status == "completed"

crw_check_crawl_status(id="a1b2c3d4-...") → { "status": "scraping|completed|failed", "data": [...] }

**REST** (async — POST to start, GET to poll, DELETE to cancel):
```bash
crw_check_crawl_status(id="a1b2c3d4-...") → { "status": "scraping|completed|failed", "data": [...] }

**REST**（异步——POST启动，GET轮询，DELETE取消）：
```bash

Start

启动

curl -X POST "$CRW_API_URL/v1/crawl" -H "Authorization: Bearer $CRW_API_KEY"
-H 'Content-Type: application/json'
-d '{"url":"https://docs.example.com","maxDepth":2,"maxPages":50}'
curl -X POST "$CRW_API_URL/v1/crawl" -H "Authorization: Bearer $CRW_API_KEY"
-H 'Content-Type: application/json'
-d '{"url":"https://docs.example.com","maxDepth":2,"maxPages":50}'

→ {"id":"a1b2c3d4-..."}

→ {"id":"a1b2c3d4-..."}

Poll

轮询

curl "$CRW_API_URL/v1/crawl/a1b2c3d4-..."
-H "Authorization: Bearer $CRW_API_KEY"
curl "$CRW_API_URL/v1/crawl/a1b2c3d4-..."
-H "Authorization: Bearer $CRW_API_KEY"

→ {"status":"completed","data":[...]}

→ {"status":"completed","data":[...]}

Cancel

取消

curl -X DELETE "$CRW_API_URL/v1/crawl/a1b2c3d4-..."
-H "Authorization: Bearer $CRW_API_KEY"
undefined
curl -X DELETE "$CRW_API_URL/v1/crawl/a1b2c3d4-..."
-H "Authorization: Bearer $CRW_API_KEY"
undefined

Options

参数选项

NeedCLI flagMCP / REST field
Max depth
-d/--depth N
(default 2)
maxDepth
(default 2)
Max pages
-l/--limit N
(default 10)
maxPages
Output format
--format markdown|json|html|rawhtml|text|links
—
Structured JSON per page—
jsonSchema: {...}
JS rendering
--js
renderJs: true
(null = auto)
Wait after load—
waitFor: 2000
(ms)
Renderer override—
renderer: "lightpanda|chrome|playwright"
Rate limit
--rate-limit N
(default 2.0 req/s)		—
Concurrency
--concurrency N
(default 5)		—
Per-page timeout
--timeout MS
(default 30 000)		—
Proxy
--proxy URL
—
Stealth mode
--stealth
—
Strip nav/footer(on by default; 
--raw
to disable)		—
需求CLI 参数MCP / REST 字段
最大深度
-d/--depth N
（默认值2）
maxDepth
（默认值2）
最大页面数
-l/--limit N
（默认值10）
maxPages
输出格式
--format markdown|json|html|rawhtml|text|links
—
每页结构化JSON—
jsonSchema: {...}
JS渲染
--js
renderJs: true
（null = 自动）
加载后等待—
waitFor: 2000
（毫秒）
渲染器覆盖—
renderer: "lightpanda|chrome|playwright"
请求速率限制
--rate-limit N
（默认值2.0 请求/秒）		—
并发数
--concurrency N
（默认值5）		—
单页超时时间
--timeout MS
（默认值30000）		—
代理
--proxy URL
—
隐身模式
--stealth
—
移除导航栏/页脚（默认开启；使用
--raw
关闭）		—

Polling loop (MCP / REST)

轮询循环（MCP / REST）

The MCP and REST crawl is async. Poll 
crw_check_crawl_status
(MCP) or 
GET /v1/crawl/{id}
(REST) every few seconds. The job expires after 1 hour.
loop:
  status = crw_check_crawl_status(id=job_id)
  if status.status == "completed":  break
  if status.status == "failed":     raise error
  wait(3s)

pages = status.data   # list of {url, markdown, html, links, metadata, ...}
MCP truncates each page's content to ~15 000 chars by default. Pass 
maxLength: 0
to opt out.
MCP和REST的爬取是异步的。每隔几秒调用
crw_check_crawl_status
（MCP）或
GET /v1/crawl/{id}
（REST）进行轮询。任务1小时后过期。
loop:
  status = crw_check_crawl_status(id=job_id)
  if status.status == "completed":  break
  if status.status == "failed":     raise error
  wait(3s)

pages = status.data   # 页面列表，每个元素包含{url, markdown, html, links, metadata, ...}
默认情况下，MCP会将每个页面的内容截断至约15000字符。传入
maxLength: 0
可取消截断。

Saving crawl output to local files

将爬取结果保存到本地文件

Never stream a whole crawl into model context. Write pages to 
.crw/
and read incrementally.
CLI (streams pages as they arrive — redirect or tee):
bash
crw crawl "https://docs.example.com" -d 2 -l 100 \
  --format json > .crw/crawl-raw.jsonl
切勿将整个爬取结果直接传入模型上下文。应将页面写入
.crw/
目录，然后增量读取。
CLI（页面爬取完成后流式输出——重定向或使用tee命令）：
bash
crw crawl "https://docs.example.com" -d 2 -l 100 \
  --format json > .crw/crawl-raw.jsonl

One markdown file per page from the JSON lines

从JSON行文件中生成每个页面对应的markdown文件

grep '^{' .crw/crawl-raw.jsonl | jq -r '"(.metadata.sourceURL)\n(.markdown)"'
| split - .crw/pages/page-

**MCP / REST** (after polling completes):
```bash
grep '^{' .crw/crawl-raw.jsonl | jq -r '"(.metadata.sourceURL)\n(.markdown)"'
| split - .crw/pages/page-

**MCP / REST**（轮询完成后）：
```bash

REST: save the full result

REST：保存完整结果

curl "$CRW_API_URL/v1/crawl/$JOB_ID" -H "Authorization: Bearer $CRW_API_KEY"
| jq -c '.data[]' > .crw/pages.jsonl
curl "$CRW_API_URL/v1/crawl/$JOB_ID" -H "Authorization: Bearer $CRW_API_KEY"
| jq -c '.data[]' > .crw/pages.jsonl

Write one .md per page

生成每个页面对应的.md文件

jq -r '.markdown' .crw/pages.jsonl | split -l 1 - .crw/pages/page-

Then `grep`, `head`, or pass individual files to the model — never the whole
blob.
jq -r '.markdown' .crw/pages.jsonl | split -l 1 - .crw/pages/page-

之后可以使用`grep`、`head`命令处理，或传入单个文件给模型——切勿传入整个文件集合。

Recommended workflow

推荐工作流

1. crw map  "https://docs.example.com" --format json > .crw/urls.json
             → see how many pages exist (check last line: "Discovered N URLs")

2. crw crawl "https://docs.example.com/api" -d 1 -l 20
             → start narrow, verify output quality

3. Scale up: -l 100, -d 2, or scope to a sub-path if needed

4. Write to .crw/, read with grep/jq
1. crw map  "https://docs.example.com" --format json > .crw/urls.json
             → 查看存在的页面数量（检查最后一行："Discovered N URLs"）

2. crw crawl "https://docs.example.com/api" -d 1 -l 20
             → 从窄范围开始，验证输出质量

3. 扩大规模：设置-l 100、-d 2，或根据需要限定子路径

4. 写入.crw/目录，使用grep/jq读取

Tips

小贴士

  • Map first. 
    crw map docs.example.com | wc -l
    in 3 seconds beats a cancelled 10-minute crawl.
  • Start at depth 1, limit 10. Confirm you're in the right section before widening scope. Most docs sets are fully reachable at depth 2-3.
  • JS auto-detects. crw's renderer fallback handles most SPAs without 
    --js
    . Add it only if you see blank pages or loading skeletons.
  • Rate-limit aggressively for production sites. Default 2 req/s is polite; drop to 0.5 on fragile targets. 
    --concurrency 2
    + 
    --rate-limit 0.5
    is a safe baseline for external sites.
  • jsonSchema
    turns every page into a typed object.     Pass a JSON schema via MCP/REST to extract structured data from every crawled page — useful for price monitoring, job listings, or any repeating schema.
  • Building a knowledge base? Load 
    crw-knowledge-base
    (coming soon) — it wraps the crawl → chunk → embed → index pipeline end-to-end.
  • 先执行映射。
    crw map docs.example.com | wc -l
    只需3秒，远胜于取消一个耗时10分钟的错误爬取。
  • 初始设置为深度1、限制10页。确认处于正确板块后再扩大范围。大多数文档集在深度2-3时即可完全覆盖。
  • JS渲染自动检测。crw的渲染器回退机制无需
    --js
    即可处理大多数单页应用（SPAs）。仅当出现空白页面或加载骨架时才添加该参数。
  • 针对生产站点严格限制请求速率。默认2请求/秒是比较友好的；对于脆弱目标，可降至0.5。
    --concurrency 2
    + 
    --rate-limit 0.5
    是外部站点的安全基准配置。
  • jsonSchema
    可将每个页面转换为类型化对象    。通过MCP/REST传入JSON schema，可从每个爬取页面提取结构化数据——适用于价格监控、职位列表或任何重复结构的场景。
  • 构建知识库？ 加载
    crw-knowledge-base
    （即将推出）——它将爬取→分块→嵌入→索引的流程端到端封装。

See also

相关链接

  • crw-map — discover URLs before crawling
  • crw-scrape — single-page extraction (faster for known URLs)
  • crw — hub skill with the full workflow ladder
  • crw-map — 爬取前发现所有URL
  • crw-scrape — 单页面提取（已知URL时速度更快）
  • crw — 包含完整工作流阶梯的核心工具