Back to Details

crw-dynamic-search

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

crw-dynamic-search — Programmatic Tool Calling for Web Research

crw-dynamic-search — 面向网页研究的程序化工具调用

Search the web and scrape pages so that raw web data never enters your context window. Only your curated 
print()
output comes back — pure signal, no noise.
搜索网页并抓取页面内容，原始网页数据绝不会进入你的上下文窗口。只有你精心筛选后的
print()
输出会返回——纯粹的有效信息，无任何冗余干扰。

Why this matters

为何这很重要

A typical 
crw search --json
returns 10 results × 300-600 chars of description each = ~5K characters. That sounds manageable — until you add 
scrapeOptions
to fetch full page markdown, which can be 20-50K chars per result. A 10-result search with full content ≈ 200-500K characters. If that floods your context, you burn tokens reading cookie banners, navigation menus, and boilerplate — and your reasoning quality degrades under the noise.
By processing results inside a Python subprocess, only your 
print()
output enters context — typically 1-3K characters of pure signal. That's a 100-200x reduction.
典型的
crw search --json
会返回10条结果，每条包含300-600字符的描述，总计约5000字符。这听起来可控——但如果添加
scrapeOptions
来获取完整页面的markdown内容，每条结果可能达到20-50000字符。10条结果的完整内容搜索≈200-500000字符。如果这些内容填满你的上下文窗口，你会浪费大量token去读取Cookie提示、导航菜单和冗余内容——推理质量也会被这些干扰信息降低。
通过在Python子进程中处理结果，只有你的
print()
输出会进入上下文窗口——通常仅为1-3000字符的纯有效信息，相比原始数据减少了100-200倍。

Background: the PTC sandbox pattern

背景：PTC沙箱模式

Anthropic's Programmatic Tool Calling lets a model write code that orchestrates tool calls inside a sandbox. Intermediate results live in the sandbox; only 
print()
output crosses into the context window.
This skill applies the same principle using local Python execution. The Python process is your sandbox. Variables in memory hold raw data. Only what you 
print()
crosses into context. You write the filtering logic — you decide what matters for each query.
Anthropic的程序化工具调用允许模型编写代码，在沙箱中协调工具调用。中间结果存储在沙箱内；只有
print()
输出会进入上下文窗口。
本技能通过本地Python执行应用了相同的原理。Python进程就是你的沙箱，原始数据存储在内存变量中，只有你
print()
的内容会进入上下文窗口。你可以编写过滤逻辑——为每个查询决定哪些信息是重要的。

Core Rule

核心规则

NEVER pipe 
crw search --json
or 
crw scrape --format json
bare into context. Always process through Python so you control what enters.
bash
undefined
绝对不要直接将
crw search --json
或
crw scrape --format json
的结果传入上下文窗口。务必通过Python处理，由你控制哪些内容进入上下文。
bash
undefined

WRONG — raw results flood context, possibly 200K+ characters

错误示例——原始结果会填满上下文，可能超过200000字符

crw search "quantum computing 2025" --json
crw search "quantum computing 2025" --json

RIGHT — only your print() enters context

正确示例——只有你的print()输出会进入上下文

crw search "quantum computing 2025" --json 2>/dev/null | python3 -c " import json, sys data = json.load(sys.stdin) for r in data: print(f'[{r["position"]}] {r["title"]}') print(f' {r["url"]}') print(f' {r["description"][:150]}') "
undefined
crw search "quantum computing 2025" --json 2>/dev/null | python3 -c " import json, sys data = json.load(sys.stdin) for r in data: print(f'[{r["position"]}] {r["title"]}') print(f' {r["url"]}') print(f' {r["description"][:150]}') "
undefined

JSON Schemas

JSON 模式

You need these to write correct filtering code.
你需要这些模式来编写正确的过滤代码。

crw search --json
output

crw search --json
输出

The CLI outputs a JSON array of result objects (not a wrapper object):
json
[
  {
    "title": "string",
    "url": "string",
    "description": "string (~200-600 chars from the search backend snippet)",
    "snippet": "string (alias of description — always same value)",
    "position": 1,
    "score": 0.85,
    "category": "general | news | images | null"
  }
]
Key notes for crw vs Tavily:
  • score
    is unreliable.     The search backend aggregates results from many engines; scores are engine-dependent and often 
    null
    . Triage by 
    position
    (rank order) and keyword density in 
    description
    , not by score.
  • description
    and 
    snippet
    are always the same value — pick either. 
    snippet
    exists as an alias for Firecrawl-compat pipelines.
  • category
    is the search backend's category: 
    "general"
    for web, 
    "news"
    , 
    "images"
    .
  • published_date
    appears on news results (ISO 8601 string or null).
CLI会输出一个JSON数组，包含多个结果对象（而非包装对象）：
json
[
  {
    "title": "string",
    "url": "string",
    "description": "string (~200-600 chars from the search backend snippet)",
    "snippet": "string (alias of description — always same value)",
    "position": 1,
    "score": 0.85,
    "category": "general | news | images | null"
  }
]
crw与Tavily的关键区别：
  • score
    不可靠    。搜索后端聚合了多个引擎的结果；分数依赖于各个引擎，且经常为
    null
    。请根据
    position
    （排名顺序）和
    description
    中的关键词密度进行筛选，而非分数    。
  • description
    和
    snippet
    始终是相同的值——任选其一即可。
    snippet
    是为了兼容Firecrawl流水线而设置的别名。
  • category
    是搜索后端的分类：
    "general"
    代表网页，
    "news"
    代表新闻，
    "images"
    代表图片。
  • 新闻结果会包含
    published_date
    （ISO 8601格式字符串或null）。

crw scrape --format json
output (ScrapeData)

crw scrape --format json
输出（ScrapeData）

The CLI outputs a single object (serialized 
ScrapeData
):
json
{
  "markdown": "string | null",
  "html": "string | null",
  "links": ["url1", "url2"],
  "renderDecision": { "kind": "autoDefault", "chosen": "http" },
  "creditCost": 1,
  "contentType": "text/html",
  "metadata": {
    "title": "string | null",
    "description": "string | null",
    "ogTitle": "string | null",
    "ogDescription": "string | null",
    "ogImage": "string | null",
    "sourceURL": "string",
    "language": "string | null",
    "statusCode": 200,
    "renderedWith": "string | null",
    "elapsedMs": 1234
  }
}
For most filtering tasks you want 
markdown
(the main content) and 
metadata.title
. 
links
is a flat array of hrefs found on the page.
CLI会输出单个对象（序列化后的
ScrapeData
）：
json
{
  "markdown": "string | null",
  "html": "string | null",
  "links": ["url1", "url2"],
  "renderDecision": { "kind": "autoDefault", "chosen": "http" },
  "creditCost": 1,
  "contentType": "text/html",
  "metadata": {
    "title": "string | null",
    "description": "string | null",
    "ogTitle": "string | null",
    "ogDescription": "string | null",
    "ogImage": "string | null",
    "sourceURL": "string",
    "language": "string | null",
    "statusCode": 200,
    "renderedWith": "string | null",
    "elapsedMs": 1234
  }
}
对于大多数过滤任务，你会用到
markdown
（主要内容）和
metadata.title
。
links
是页面中所有href链接的扁平数组。

MCP 
crw_search
output (when using MCP, not CLI)

MCP 
crw_search
输出（使用MCP而非CLI时）

The results sit in one of two places depending on which backend served the call: directly in 
data
on the hosted API (
api.fastcrw.com
), or nested as 
data.results
on a self-hosted engine. Read 
data.results
when present and fall back to 
data
. The nested form:
json
{
  "success": true,
  "data": {
    "results": [
      {
        "url": "string",
        "title": "string",
        "description": "string",
        "snippet": "string",
        "position": 1,
        "score": 0.85,
        "category": "string | null",
        "publishedDate": "string | null"
      }
    ]
  }
}
When 
scrapeOptions
is passed, each result also carries 
markdown
, 
html
, 
links
, and 
metadata
populated from the full page fetch.
结果的位置取决于调用的后端：在托管API（
api.fastcrw.com
）中直接位于
data
字段，在自托管引擎中嵌套在
data.results
字段。优先读取
data.results
，若不存在则回退到
data
。嵌套格式如下：
json
{
  "success": true,
  "data": {
    "results": [
      {
        "url": "string",
        "title": "string",
        "description": "string",
        "snippet": "string",
        "position": 1,
        "score": 0.85,
        "category": "string | null",
        "publishedDate": "string | null"
      }
    ]
  }
}
当传入
scrapeOptions
时，每个结果还会包含从完整页面抓取的
markdown
、
html
、
links
和
metadata
字段。

Execution modes

执行模式

Pipe mode — for simple filters (3-5 lines)

管道模式——适用于简单过滤（3-5行代码）

bash
crw search "Python 3.13 release date" --json 2>/dev/null | python3 -c "
import json, sys
data = json.load(sys.stdin)
for r in data[:3]:
    print(r['title'])
    print(r['description'][:300])
    print()
"
bash
crw search "Python 3.13 release date" --json 2>/dev/null | python3 -c "
import json, sys
data = json.load(sys.stdin)
for r in data[:3]:
    print(r['title'])
    print(r['description'][:300])
    print()
"

Heredoc mode — for anything more complex (default)

heredoc模式——适用于更复杂的场景（默认推荐）

Single Bash call, clean multi-line Python, no escaping, no temp files. The single-quoted 
<< 'PYEOF'
heredoc is the workhorse — nothing inside is interpolated by the shell.
bash
python3 << 'PYEOF'
import json, subprocess

raw = subprocess.check_output(
    ['crw', 'search', 'your query', '--json', '--limit', '10'],
    stderr=subprocess.DEVNULL
)
data = json.loads(raw)
for r in data:
    print(f'[{r["position"]}] {r["title"]}')
    print(f'  {r["url"]}')
    print(f'  {r["description"][:200]}')
    print()
PYEOF
Save DATA to 
/tmp/
, not CODE. Saving 
/tmp/crw_results.json
for use in the next turn = good. Writing a one-shot 
/tmp/filter.py
= wasteful; use a heredoc.
单次Bash调用，清晰的多行Python代码，无需转义，无需临时文件。单引号包裹的
<< 'PYEOF'
heredoc是核心用法——其中的内容不会被Shell插值处理。
bash
python3 << 'PYEOF'
import json, subprocess

raw = subprocess.check_output(
    ['crw', 'search', 'your query', '--json', '--limit', '10'],
    stderr=subprocess.DEVNULL
)
data = json.loads(raw)
for r in data:
    print(f'[{r["position"]}] {r["title"]}')
    print(f'  {r["url"]}')
    print(f'  {r["description"][:200]}')
    print()
PYEOF
将数据保存到
/tmp/
，而非代码。将
/tmp/crw_results.json
保存供下一轮使用是合理的。编写一次性的
/tmp/filter.py
则没必要；请使用heredoc模式。

Script mode — only for reusable pipelines

脚本模式——仅适用于可复用的流水线

Only write a real file when the same script will be called across 3+ turns or invoked repeatedly. Otherwise, use a heredoc.
只有当同一脚本会被调用3次以上或重复执行时，才需要编写实际的文件。否则，请使用heredoc模式。

Multi-turn iteration

多轮迭代

Complex research needs explore then extract — see what's available before deciding what to drill into. The key: save raw JSON to 
/tmp/
once, process in separate steps.
复杂的研究需要先探索再提取——先了解可用内容，再决定深入挖掘哪些信息。关键：将原始JSON保存到
/tmp/
一次，分步骤处理。

Turn 1: Search and triage

第一轮：搜索与筛选

bash
python3 << 'PYEOF'
import json, subprocess

raw = subprocess.check_output(
    ['crw', 'search', 'solid-state battery commercialization 2025',
     '--json', '--limit', '10'],
    stderr=subprocess.DEVNULL
)
data = json.loads(raw)
bash
python3 << 'PYEOF'
import json, subprocess

raw = subprocess.check_output(
    ['crw', 'search', 'solid-state battery commercialization 2025',
     '--json', '--limit', '10'],
    stderr=subprocess.DEVNULL
)
data = json.loads(raw)

Save raw — stays on disk, never enters context

保存原始数据——存储在磁盘上，绝不会进入上下文

with open('/tmp/crw_results.json', 'w') as f: json.dump(data, f)
with open('/tmp/crw_results.json', 'w') as f: json.dump(data, f)

Print only what you need to pick next steps

仅打印你需要用于选择下一步的内容

Sort by position (rank), not score — score is unreliable from the search backend

按position（排名）排序，而非score——搜索后端的score不可靠

print(f'{len(data)} results saved to /tmp/crw_results.json\n') for r in data: print(f'[{r["position"]}] {r["title"][:90]}') print(f' {r["url"]}') print(f' {r["description"][:150]}') print() PYEOF

Context receives: ~600-800 tokens of titles + snippets. Any full page markdown is
in `/tmp/crw_results.json`, untouched.
print(f'{len(data)}条结果已保存到/tmp/crw_results.json\n') for r in data: print(f'[{r["position"]}] {r["title"][:90]}') print(f' {r["url"]}') print(f' {r["description"][:150]}') print() PYEOF

上下文窗口会收到：约600-800token的标题和摘要。完整页面的markdown内容存储在`/tmp/crw_results.json`中，不会被处理。

Turn 2: Extract from chosen results

第二轮：从选定结果中提取信息

You saw the triage. Now write targeted extraction for the results that matter:
bash
python3 << 'PYEOF'
import json, subprocess

data = json.load(open('/tmp/crw_results.json'))
你已完成筛选。现在为重要的结果编写针对性的提取逻辑：
bash
python3 << 'PYEOF'
import json, subprocess

data = json.load(open('/tmp/crw_results.json'))

Indices you chose from the triage in turn 1

你在第一轮筛选中选择的结果索引

for r in [data[0], data[2], data[4]]: # Scrape the full page for results that looked relevant try: raw = subprocess.check_output( ['crw', 'scrape', r['url'], '--format', 'json'], stderr=subprocess.DEVNULL, timeout=30 ) page = json.loads(raw) except Exception: continue
md = page.get('markdown') or ''
if not md:
    continue

print(f'## {r["title"]}')
print(f'URL: {r["url"]}\n')

# Write filtering logic that matches the query — this is the key step
# Example: keep paragraphs about commercialization timelines
for para in md.split('\n\n'):
    para = para.strip()
    if len(para) > 80 and any(kw in para.lower() for kw in
            ['toyota', 'quantumscape', 'samsung', 'production',
             'commercializ', '2025', '2026', 'gigafactory']):
        print(para)
        print()
print('---\n')
PYEOF

Context receives: ~600-800 tokens of targeted content. You made the decision.
for r in [data[0], data[2], data[4]]: # 抓取相关结果的完整页面 try: raw = subprocess.check_output( ['crw', 'scrape', r['url'], '--format', 'json'], stderr=subprocess.DEVNULL, timeout=30 ) page = json.loads(raw) except Exception: continue
md = page.get('markdown') or ''
if not md:
    continue

print(f'## {r["title"]}')
print(f'URL: {r["url"]}\n')

# 编写与查询匹配的过滤逻辑——这是关键步骤
# 示例：保留关于商业化时间表的段落
for para in md.split('\n\n'):
    para = para.strip()
    if len(para) > 80 and any(kw in para.lower() for kw in
            ['toyota', 'quantumscape', 'samsung', 'production',
             'commercializ', '2025', '2026', 'gigafactory']):
        print(para)
        print()
print('---\n')
PYEOF

上下文窗口会收到：约600-800token的针对性内容。由你决定哪些内容保留。

Turn 3: Follow leads

第三轮：跟进线索

Turn 2 often surfaces new URLs or specific sub-topics. Keep iterating:
bash
python3 << 'PYEOF'
import json, subprocess
第二轮通常会发现新的URL或特定子主题。继续迭代：
bash
python3 << 'PYEOF'
import json, subprocess

A URL you found referenced in the content you read in turn 2

你在第二轮读取的内容中发现的URL

raw = subprocess.check_output( ['crw', 'search', 'QuantumScape QSE-5 production timeline Q4 2025', '--json', '--limit', '5'], stderr=subprocess.DEVNULL ) data = json.loads(raw)
for r in data[:3]: print(f'## {r["title"]}') print(f'URL: {r["url"]}') print(r['description']) print() PYEOF
undefined
raw = subprocess.check_output( ['crw', 'search', 'QuantumScape QSE-5 production timeline Q4 2025', '--json', '--limit', '5'], stderr=subprocess.DEVNULL ) data = json.loads(raw)
for r in data[:3]: print(f'## {r["title"]}') print(f'URL: {r["url"]}') print(r['description']) print() PYEOF
undefined

When to use single-turn vs multi-turn

何时使用单轮 vs 多轮模式

Single turn (pipe or one heredoc): when you know what you're looking for. Specific factual queries, known keywords, lookup tasks.
Multi-turn (save + explore + extract): when you need to see what's available before deciding what to extract. Open-ended research, competitive analysis, queries where you don't know the right keywords yet.
单轮模式（管道或单次heredoc）：当你明确知道要查找的内容时。例如特定事实查询、已知关键词、信息查找任务。
多轮模式（保存+探索+提取）：当你需要先了解可用内容，再决定提取哪些信息时。例如开放式研究、竞品分析、不确定正确关键词的查询。

Writing your filtering code

编写过滤代码

The Python you write IS the filtering logic. There are no fixed templates. Principles:
Triage by position, not score. The search backend's scores are engine-dependent and often absent. Result order (
position: 1, 2, 3...
) is a more reliable signal — the aggregator's RRF ranking already baked in multi-engine consensus.
Be specific. A financial query should filter for numbers and financial terms. A technical query should look for code blocks and version strings. Match your filtering to the domain.
Skip structural noise. Lines shorter than ~50 chars are usually nav elements, breadcrumbs, or button labels. Skip them. Keep headings and their following paragraphs.
Print structured output so it's easy to reason over:
python
print(f'## {title}')
print(f'URL: {url}\n')
print(relevant_content)
print('---\n')
Handle errors. Pages 404, scrapes timeout, the search backend returns partial results. Always wrap scrape calls in try/except:
python
try:
    raw = subprocess.check_output(['crw', 'scrape', url, '--format', 'json'],
                                   stderr=subprocess.DEVNULL, timeout=30)
except Exception:
    continue
Token budget. Your 
print()
output is what enters context. Target 150-600 tokens per source. If you're printing 5000+ chars from one page, you're not filtering enough. Exception: dense data tables or spec pages where every row counts.
你编写的Python代码就是过滤逻辑。没有固定模板，但需遵循以下原则：
按position筛选，而非score。搜索后端的分数依赖于各个引擎，且经常缺失。结果顺序（
position: 1, 2, 3...
）是更可靠的信号——聚合器的RRF排名已经整合了多引擎的共识。
针对性要强。财务查询应筛选数字和金融术语；技术查询应查找代码块和版本字符串。根据领域调整过滤逻辑。
跳过结构性干扰。长度小于约50字符的行通常是导航元素、面包屑或按钮标签，请跳过它们。保留标题及其后续段落。
打印结构化输出，以便于推理：
python
print(f'## {title}')
print(f'URL: {url}\n')
print(relevant_content)
print('---\n')
处理错误。页面可能404、抓取超时、搜索后端返回部分结果。务必将抓取调用包裹在try/except中：
python
try:
    raw = subprocess.check_output(['crw', 'scrape', url, '--format', 'json'],
                                   stderr=subprocess.DEVNULL, timeout=30)
except Exception:
    continue
Token预算。你的
print()
输出会进入上下文窗口。每个来源目标为150-600token。如果从单个页面打印超过5000字符，说明过滤不够充分。例外情况：密集的数据表格或规格页面，其中每一行都很重要。

Full example: multi-angle research

完整示例：多角度研究

bash
python3 << 'PYEOF'
import json, subprocess
bash
python3 << 'PYEOF'
import json, subprocess

Fan out: hit the same topic from multiple angles

发散搜索：从多个角度研究同一主题

queries = [ ('general', 'EU AI Act compliance requirements 2025'), ('specific', 'EU AI Act high-risk AI systems Article 6 obligations'), ]
all_results = [] for label, q in queries: raw = subprocess.check_output( ['crw', 'search', q, '--json', '--limit', '8'], stderr=subprocess.DEVNULL ) results = json.loads(raw) for r in results: r['_query'] = label all_results.extend(results)
queries = [ ('general', 'EU AI Act compliance requirements 2025'), ('specific', 'EU AI Act high-risk AI systems Article 6 obligations'), ]
all_results = [] for label, q in queries: raw = subprocess.check_output( ['crw', 'search', q, '--json', '--limit', '8'], stderr=subprocess.DEVNULL ) results = json.loads(raw) for r in results: r['_query'] = label all_results.extend(results)

Deduplicate by URL

按URL去重

seen = set() unique = [] for r in all_results: if r['url'] not in seen: seen.add(r['url']) unique.append(r)
seen = set() unique = [] for r in all_results: if r['url'] not in seen: seen.add(r['url']) unique.append(r)

Save everything

保存所有结果

with open('/tmp/eu_ai_results.json', 'w') as f: json.dump(unique, f)
with open('/tmp/eu_ai_results.json', 'w') as f: json.dump(unique, f)

Print triage sorted by position within each query batch

按每个查询批次内的position排序，打印筛选结果

print(f'{len(unique)} unique results from {len(queries)} queries\n') for r in unique[:12]: print(f'[{r["_query"]}][pos {r["position"]}] {r["title"][:80]}') print(f' {r["url"]}') print(f' {r["description"][:120]}') print() PYEOF
undefined
print(f'{len(unique)}条来自{len(queries)}个查询的唯一结果\n') for r in unique[:12]: print(f'[{r["_query"]}][pos {r["position"]}] {r["title"][:80]}') print(f' {r["url"]}') print(f' {r["description"][:120]}') print() PYEOF
undefined

jq fallback

jq 备选方案

When 
python3
is unavailable, use 
jq
for basic filtering:
bash
undefined
当
python3
不可用时，可使用
jq
进行基础过滤：
bash
undefined

Print titles and URLs only

仅打印标题和URL

crw search "query" --json 2>/dev/null | jq '.[] | {title, url, description: .description[:200]}'
crw search "query" --json 2>/dev/null | jq '.[] | {title, url, description: .description[:200]}'

Filter by keyword in description

按描述中的关键词过滤

crw search "query" --json 2>/dev/null | jq '[.[] | select(.description | ascii_downcase | contains("keyword"))]'

jq can't do multi-step search-then-scrape, subprocess calls, or complex filtering.
Use it only for simple single-pass lookups when Python isn't available.
crw search "query" --json 2>/dev/null | jq '[.[] | select(.description | ascii_downcase | contains("keyword"))]'

jq无法完成多步骤的搜索-抓取、子进程调用或复杂过滤。仅在Python不可用时，用于简单的单次查询。

CLI quick reference

CLI 快速参考

bash
crw search "query"                                     # text output (default)
crw search "query" --json                              # JSON array
crw search "query" --json --fields title,url,snippet   # projected fields only
crw search "query" --json --limit 5                    # cap results
crw search "query" --category news --time-range week   # news, last 7 days
crw scrape "https://example.com"                       # markdown
crw scrape "https://example.com" --format json         # full ScrapeData JSON
crw scrape "https://example.com" --format json -o /tmp/page.json
Available 
--fields
for 
crw search --json
: 
title
, 
url
, 
description
, 
snippet
, 
position
, 
score
, 
category
bash
crw search "query"                                     # 文本输出（默认）
crw search "query" --json                              # JSON数组
crw search "query" --json --fields title,url,snippet   # 仅指定字段
crw search "query" --json --limit 5                    # 限制结果数量
crw search "query" --category news --time-range week   # 新闻类，过去7天
crw scrape "https://example.com"                       # markdown格式
crw scrape "https://example.com" --format json         # 完整ScrapeData JSON
crw scrape "https://example.com" --format json -o /tmp/page.json
crw search --json
支持的
--fields
选项： 
title
, 
url
, 
description
, 
snippet
, 
position
, 
score
, 
category

See also

另请参阅

  • crw-search — full search options (time-range, categories, language)
  • crw-scrape — scrape a known URL
  • crw-best-practices — choosing the right verb, post-filtering strategies
  • crw-search — 完整搜索选项（时间范围、分类、语言）
  • crw-scrape — 抓取指定URL
  • crw-best-practices — 选择合适的动词、后期过滤策略