crw-self-host

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

crw-self-host — Stand up your own crw

crw-self-host — 搭建您自己的crw服务

One static binary, one config file. No Redis, no Postgres, no Node.js runtime in the request path. Self-host free under AGPL-3.0, or point at

api.fastcrw.com

for managed.

只需一个静态二进制文件和一个配置文件。请求路径中无需Redis、Postgres或Node.js运行时。可在AGPL-3.0协议下免费自托管，或使用托管服务

api.fastcrw.com

。

When to use

适用场景

You want full data residency (URLs and queries never leave your infra).
You want to run recurring crawls/audits at VPS cost, not per-page credits.
You need to configure proxy pools, custom renderers, or LLM extraction.
Step 0 before any self-hosted crw workflow.

您需要完全的数据驻留（URL和查询绝不会离开您的基础设施）。
您希望以VPS成本运行定期爬取/审计，而非按页面付费。
您需要配置代理池、自定义渲染器或LLM提取功能。
任何自托管crw工作流的前置准备步骤。

Install paths

安装方式

Choose one. All install paths run the same Rust binary.

选择其中一种即可。所有安装方式均运行相同的Rust二进制文件。

MCP server (

crw-mcp

) — recommended for AI agent use

MCP服务器（

crw-mcp

）—— 推荐AI Agent使用

bash

npx crw-mcp                           # zero install (npm; embedded engine, ~6 MB RAM)
brew install us/crw/crw-mcp           # Homebrew
cargo install crw-mcp                 # Cargo (~17 MB, full embedded)
docker run -i ghcr.io/us/crw crw-mcp  # Docker
pip install crw                       # Python SDK (auto-downloads binary on first use)

Lean build (~4.2 MB, no headless browser engine — proxy/cloud-only mode):

bash

cargo build --profile release-small --no-default-features -p crw-mcp

bash

npx crw-mcp                           # 零安装（npm；嵌入式引擎，约6 MB内存）
brew install us/crw/crw-mcp           # Homebrew
cargo install crw-mcp                 # Cargo（约17 MB，完整嵌入式版本）
docker run -i ghcr.io/us/crw crw-mcp  # Docker
pip install crw                       # Python SDK（首次使用时自动下载二进制文件）

精简构建（约4.2 MB，无头浏览器引擎——仅代理/云模式）：

bash

cargo build --profile release-small --no-default-features -p crw-mcp

CLI (

crw

) — scrape from the terminal

CLI（

crw

）—— 从终端进行爬取

bash

brew install us/crw/crw
curl -fsSL https://fastcrw.com/install | CRW_BINARY=crw sh
cargo install crw-cli

bash

brew install us/crw/crw
curl -fsSL https://fastcrw.com/install | CRW_BINARY=crw sh
cargo install crw-cli

APT (Debian/Ubuntu):

APT（Debian/Ubuntu）：

curl -fsSL https://apt.fastcrw.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/crw.gpg echo "deb [signed-by=/usr/share/keyrings/crw.gpg] https://apt.fastcrw.com stable main"
| sudo tee /etc/apt/sources.list.d/crw.list sudo apt update && sudo apt install crw

undefined

undefined

API server (

crw-server

) — Firecrawl-compatible REST endpoint

API服务器（

crw-server

）—— 兼容Firecrawl的REST端点

For serving multiple apps, other languages (Node.js, Go, Java), or as a shared microservice.

bash

brew install us/crw/crw-server
curl -fsSL https://fastcrw.com/install | CRW_BINARY=crw-server sh
docker run -p 3000:3000 ghcr.io/us/crw

适用于为多个应用、其他语言（Node.js、Go、Java）提供服务，或作为共享微服务。

bash

brew install us/crw/crw-server
curl -fsSL https://fastcrw.com/install | CRW_BINARY=crw-server sh
docker run -p 3000:3000 ghcr.io/us/crw

Running the API server

运行API服务器

Binary directly (
crw serve
):

bash

crw serve                             # reads config.default.toml, listens on :3000
crw serve --port 3001                 # custom port (-p short form also accepted)
crw serve --config myconfig.toml      # load specific config file
CRW_PORT=3001 crw-server              # env-var alternative for the standalone binary

Docker (single container, no search):

bash

docker run -p 3000:3000 ghcr.io/us/crw
curl http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

Docker Compose — full stack with bundled search:

bash

docker compose up -d                                          # http + LightPanda + search backend
docker compose --profile heavy up -d                          # + Chrome fallback
docker compose -f docker-compose.yml \
  -f docker-compose.stealth.yml --profile stealth up -d       # + browserless stealth tier

docker compose up

starts three services:

crw

(API server),

lightpanda

(JS renderer), and

searxng

(search backend). The

crw

service waits for

searxng

to pass its healthcheck before accepting traffic, so the first search request after startup doesn't race the cold start.

After

compose up

bash

curl http://localhost:3000/health                              # → {"status":"ok",...}
curl -X POST http://localhost:3000/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query":"fastCRW","limit":5}'                          # own backend, no API key

直接运行二进制文件（
crw serve
）：

bash

crw serve                             # 读取config.default.toml，监听端口:3000
crw serve --port 3001                 # 自定义端口（也可使用短格式-p）
crw serve --config myconfig.toml      # 加载指定配置文件
CRW_PORT=3001 crw-server              # 独立二进制文件的环境变量替代方式

Docker（单容器，无搜索功能）：

bash

docker run -p 3000:3000 ghcr.io/us/crw
curl http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

Docker Compose — 附带搜索功能的完整栈：

bash

docker compose up -d                                          # HTTP + LightPanda + 搜索后端
docker compose --profile heavy up -d                          # + Chrome fallback
docker compose -f docker-compose.yml \
  -f docker-compose.stealth.yml --profile stealth up -d       # + browserless stealth tier

docker compose up

会启动三个服务：

crw

（API服务器）、

lightpanda

（JS渲染器）和

searxng

（搜索后端）。

crw

服务会等待

searxng

通过健康检查后才接受请求，因此启动后的首个搜索请求不会因冷启动而失败。

启动compose后：

bash

curl http://localhost:3000/health                              # → {"status":"ok",...}
curl -X POST http://localhost:3000/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query":"fastCRW","limit":5}'                          # 自有后端，无需API密钥

Search backend sidecar

搜索后端边车

docker-compose.yml

bundles a search-backend container (

searxng/searxng:2026.5.9-0cba32c15

) that backs

/v1/search

with no API key and no per-query cost. It is internal-only — the port is not published to the host by default. The

crw

container points at it as

http://searxng:8080

via the Docker bridge network (set in

config.docker.toml

[search] searxng_url = "http://searxng:8080"

To debug the search backend directly, add to

docker-compose.override.yml

yaml

services:
  searxng:
    ports:
      - "127.0.0.1:8888:8080"

For a standalone binary pointing at an external search backend:

bash

CRW_SEARCH__SEARXNG_URL=http://my-searxng:8080 crw-server

When

searxng_url

is unset,

/v1/search

returns HTTP 400

search_disabled

docker-compose.yml

捆绑了一个搜索后端容器（

searxng/searxng:2026.5.9-0cba32c15

），无需API密钥和按查询付费即可支持

/v1/search

接口。它仅对内开放——默认不会将端口暴露给主机。

crw

容器通过Docker桥接网络将其指向为

http://searxng:8080

（在

config.docker.toml

中设置为

[search] searxng_url = "http://searxng:8080"

）。

如需直接调试搜索后端，可添加到

docker-compose.override.yml

：

yaml

services:
  searxng:
    ports:
      - "127.0.0.1:8888:8080"

对于指向外部搜索后端的独立二进制文件：

bash

CRW_SEARCH__SEARXNG_URL=http://my-searxng:8080 crw-server

当

searxng_url

未设置时，

/v1/search

会返回HTTP 400错误

search_disabled

。

Key config knobs (

config.default.toml

)

核心配置项（

config.default.toml

）

Config is a TOML file. The binary loads

config.default.toml

by default; override with

CRW_CONFIG=<name>

(loads

<name>.toml

). Docker uses

config.docker.toml

(set via

CRW_CONFIG=config.docker

docker-compose.yml

Every key can be overridden by environment variable using the pattern

CRW_<SECTION>__<KEY>

(double underscore between section and key):

bash

CRW_SERVER__PORT=8080
CRW_RENDERER__MODE=chrome
CRW_RENDERER__POOL_SIZE=8
CRW_SEARCH__SEARXNG_URL=http://localhost:8080
CRW_CRAWLER__PROXY=http://user:pass@proxy:8080
CRW_EXTRACTION__LLM__API_KEY=sk-...
CRW_AUTH__API_KEYS='["key-one","key-two"]'   # JSON array as string

配置文件采用TOML格式。二进制文件默认加载

config.default.toml

；可通过

CRW_CONFIG=<name>

覆盖（加载

<name>.toml

）。Docker使用

config.docker.toml

（在

docker-compose.yml

中通过

CRW_CONFIG=config.docker

设置）。

所有配置项均可通过环境变量覆盖，格式为

CRW_<SECTION>__<KEY>

（节与键之间使用双下划线）：

bash

CRW_SERVER__PORT=8080
CRW_RENDERER__MODE=chrome
CRW_RENDERER__POOL_SIZE=8
CRW_SEARCH__SEARXNG_URL=http://localhost:8080
CRW_CRAWLER__PROXY=http://user:pass@proxy:8080
CRW_EXTRACTION__LLM__API_KEY=sk-...
CRW_AUTH__API_KEYS='["key-one","key-two"]'   # JSON数组字符串形式

[renderer]

— JS rendering

[renderer]

— JS渲染

toml

[renderer]
mode = "auto"        # auto | lightpanda | chrome | playwright | none
pool_size = 4        # browser context pool size
page_timeout_ms = 30000

```
auto
```
— tries LightPanda first (fast, ~64 MB), falls back to Chrome for complex SPAs and Cloudflare challenges. Recommended for production.
```
lightpanda
```
— LightPanda only; fast p50, lower recall on heavy SPAs.
```
chrome
```
— Chromium only via CDP.
```
playwright
```
— Playwright-controlled browser.
```
none
```
— HTTP-only, no JS rendering.

Configure renderer endpoints:

toml

[renderer.lightpanda]
ws_url = "ws://127.0.0.1:9222/"

[renderer.chrome]
ws_url = "ws://127.0.0.1:9223/"

toml

[renderer]
mode = "auto"        # auto | lightpanda | chrome | playwright | none
pool_size = 4        # 浏览器上下文池大小
page_timeout_ms = 30000

```
auto
```
— 优先尝试LightPanda（速度快，约64 MB内存），针对复杂SPA和Cloudflare验证回退到Chrome。推荐生产环境使用。
```
lightpanda
```
— 仅使用LightPanda；p50速度快，但对重型SPA的召回率较低。
```
chrome
```
— 仅通过CDP使用Chromium。
```
playwright
```
— Playwright控制的浏览器。
```
none
```
— 仅HTTP，无JS渲染。

配置渲染器端点：

toml

[renderer.lightpanda]
ws_url = "ws://127.0.0.1:9222/"

[renderer.chrome]
ws_url = "ws://127.0.0.1:9223/"

[search]

— web search

[search]

— 网页搜索

toml

[search]
enabled = true

toml

[search]
enabled = true

searxng_url = "http://localhost:8080" # required for /v1/search to work

searxng_url = "http://localhost:8080" # /v1/search接口正常工作所需配置

timeout_ms = 15000 default_limit = 5 max_limit = 20

undefined

timeout_ms = 15000 default_limit = 5 max_limit = 20

undefined

[crawler]

— proxy and stealth

[crawler]

— 代理与隐身模式

toml

[crawler]

toml

[crawler]

Single proxy:

单个代理：

proxy = "http://user:pass@proxy:8080"

proxy = "socks5://user:pass@proxy:1080"

Pool with rotation:

带轮询的代理池：

proxy_list = ["http://user:pass@a:8080", "http://user:pass@b:8080"]

proxy_rotation = "sticky_per_host" # sticky_per_host | round_robin | random

Stealth mode (rotate UA + inject browser-like headers):

隐身模式（轮换UA + 注入类浏览器头）：

[crawler.stealth]

enabled = true

inject_headers = true

jitter_factor = 0.2


Proxy rotation applies to both the HTTP path and the Chrome/CDP path for
scrape, crawl, and map. LightPanda has no proxy support and is skipped
(fail-closed) when a proxy is active. SOCKS5 proxies with credentials are not
supported on the Chrome path — use http/https proxies there.


代理轮换适用于爬取、抓取和映射的HTTP路径以及Chrome/CDP路径。LightPanda不支持代理，当代理激活时会被跳过（失败关闭）。Chrome路径不支持带凭证的SOCKS5代理——请在此路径使用http/https代理。

[extraction.llm]

— structured JSON extraction + summaries

[extraction.llm]

— 结构化JSON提取 + 摘要

Required to enable

formats: ["json"]

(schema extraction),

formats: ["summary"]

, and

/v1/search

with

answer: true

toml

[extraction.llm]
provider = "anthropic"          # "anthropic" | "openai" | "openai-responses" | "deepseek" | "azure" | "openai-compatible"
api_key = "sk-..."              # or CRW_EXTRACTION__LLM__API_KEY env var
model = "claude-sonnet-4-20250514"
max_tokens = 4096

启用

formats: ["json"]

（ schema提取）、

formats: ["summary"]

和带

answer: true

的

/v1/search

接口所需配置。

toml

[extraction.llm]
provider = "anthropic"          # "anthropic" | "openai" | "openai-responses" | "deepseek" | "azure" | "openai-compatible"
api_key = "sk-..."              # 或使用CRW_EXTRACTION__LLM__API_KEY环境变量
model = "claude-sonnet-4-20250514"
max_tokens = 4096

base_url = "https://custom-endpoint.example.com" # Chat Completions or Responses endpoints

base_url = "https://custom-endpoint.example.com" # Chat Completions或Responses端点

max_concurrency = 4 max_html_bytes = 100000

DeepSeek example:

DeepSeek示例：

provider = "deepseek"

api_key = "..."

model = "deepseek-chat"

base_url = "https://api.deepseek.com/v1"

Azure OpenAI example:

Azure OpenAI示例：

provider = "azure"

api_key = "..."

model = "gpt-4o-mini" # Azure deployment name

model = "gpt-4o-mini" # Azure部署名称

base_url = "https://<resource>.openai.azure.com"

azure_api_version = "2024-05-01-preview"

undefined

undefined

[auth]

— API key auth

[auth]

— API密钥认证

By default, self-hosted crw requires no auth. Add keys to lock it down:

toml

[auth]
api_keys = ["crw-key-one", "crw-key-two"]

Empty (or absent)

api_keys

= no auth required — good for local/trusted-network deployments. On managed

api.fastcrw.com

, Bearer auth is always required.

默认情况下，自托管crw无需认证。可添加密钥进行锁定：

toml

[auth]
api_keys = ["crw-key-one", "crw-key-two"]

api_keys

为空（或不存在）表示无需认证——适用于本地/可信网络部署。在托管服务

api.fastcrw.com

上，始终需要Bearer认证。

[document]

— PDF parsing

[document]

— PDF解析

toml

[document]
enabled = true
max_pages = 0              # 0 = no limit
max_upload_bytes = 52428800   # 50 MiB cap for POST /v2/parse uploads
upload_concurrency = 4
max_concurrent_parses = 4
parse_timeout_ms = 30000
sandbox = false            # set true in Docker (untrusted uploads)

toml

[document]
enabled = true
max_pages = 0              # 0 = 无限制
max_upload_bytes = 52428800   # POST /v2/parse上传的50 MiB上限
upload_concurrency = 4
max_concurrent_parses = 4
parse_timeout_ms = 30000
sandbox = false            # 在Docker中设置为true（处理不可信上传）

MCP modes: embedded vs proxy

MCP模式：嵌入式 vs 代理

crw-mcp

runs in one of two modes determined by the

CRW_API_URL

env var:

Embedded mode (no

CRW_API_URL

): the Rust engine runs in-process inside the MCP process. No separate server needed. ~6 MB RAM. Zero setup.

bash

npx crw-mcp                            # embedded
claude mcp add crw -- npx -y crw-mcp  # Claude Code, embedded

Proxy mode (

CRW_API_URL

set): MCP forwards all calls to the REST endpoint. Use this when you want a shared

crw-server

to serve multiple agents, or to point at

api.fastcrw.com

bash

CRW_API_URL=https://api.fastcrw.com CRW_API_KEY=crw_live_... npx crw-mcp

crw-mcp

的运行模式由

CRW_API_URL

环境变量决定：

嵌入式模式（未设置

CRW_API_URL

）：Rust引擎在MCP进程内运行。无需单独服务器。约6 MB内存。零配置。

bash

npx crw-mcp                            # 嵌入式模式
claude mcp add crw -- npx -y crw-mcp  # Claude Code，嵌入式模式

代理模式（已设置

CRW_API_URL

）：MCP将所有调用转发到REST端点。当您希望共享

crw-server

为多个Agent提供服务，或指向

api.fastcrw.com

时使用此模式。

bash

CRW_API_URL=https://api.fastcrw.com CRW_API_KEY=crw_live_... npx crw-mcp

Claude Code:

Claude Code：

claude mcp add crw
-e CRW_API_URL=https://api.fastcrw.com -e CRW_API_KEY=crw_live_...
-- npx -y crw-mcp

undefined

claude mcp add crw
-e CRW_API_URL=https://api.fastcrw.com -e CRW_API_KEY=crw_live_...
-- npx -y crw-mcp

undefined

Verify the setup

验证安装

bash

undefined

bash

undefined

Health (no auth)

健康检查（无需认证）

curl http://localhost:3000/health

→ {"status":"ok","version":"..."}

Scrape (no auth on default self-host)

爬取（默认自托管无需认证）

curl -X POST http://localhost:3000/v1/scrape
-H "Content-Type: application/json"
-d '{"url":"https://example.com","formats":["markdown"]}' | jq .success

→ true

Search (requires a search-backend sidecar via docker compose up)

搜索（需要通过docker compose up启动搜索后端边车）

curl -X POST http://localhost:3000/v1/search
-H "Content-Type: application/json"
-d '{"query":"hello","limit":3}' | jq .success

→ true (or 400 search_disabled if search backend not wired up)

→ true（若未连接搜索后端则返回400 search_disabled）

undefined

undefined

Production hardening notes

生产环境加固注意事项

Set
```
SEARXNG_SECRET_KEY
```
via
```
.env
```
(
```
openssl rand -hex 32
```
) — the compose default is a placeholder acceptable only for local use.
Enable
```
[document] sandbox = true
```
if you accept untrusted PDF uploads (Docker images do this by default in
```
config.docker.toml
```
).
For public-facing deployments, set
```
[auth] api_keys
```
and put a reverse proxy (nginx/Caddy) in front for TLS.
```
rate_limit_rps = 10
```
in
```
config.default.toml
```
(global). Set to
```
0
```
to disable (Docker config does this for bench/production load).

Full production hardening guide: docs.fastcrw.com/self-hosting-hardening/

通过
```
.env
```
设置
```
SEARXNG_SECRET_KEY
```
（使用
```
openssl rand -hex 32
```
生成）——compose默认值仅适用于本地环境。
若接受不可信PDF上传，启用
```
[document] sandbox = true
```
（Docker镜像在
```
config.docker.toml
```
中默认启用此设置）。
对于面向公网的部署，设置
```
[auth] api_keys
```
并在前端放置反向代理（nginx/Caddy）以提供TLS支持。
```
config.default.toml
```
中的
```
rate_limit_rps = 10
```
（全局限制）。设置为
```
0
```
可禁用限制（Docker配置为基准/生产负载禁用此限制）。

完整生产环境加固指南：docs.fastcrw.com/self-hosting-hardening/

crw-self-host

Original

Translation

crw-self-host — Stand up your own crw

crw-self-host — 搭建您自己的crw服务

When to use

适用场景

Install paths

安装方式

MCP server (crw-mcp) — recommended for AI agent use

MCP服务器（crw-mcp）—— 推荐AI Agent使用

CLI (crw) — scrape from the terminal

CLI（crw）—— 从终端进行爬取

APT (Debian/Ubuntu):

APT（Debian/Ubuntu）：

API server (crw-server) — Firecrawl-compatible REST endpoint

API服务器（crw-server）—— 兼容Firecrawl的REST端点

Running the API server

运行API服务器

Search backend sidecar

搜索后端边车

Key config knobs (config.default.toml)

核心配置项（config.default.toml）

[renderer] — JS rendering

[renderer] — JS渲染

[search] — web search

[search] — 网页搜索

searxng_url = "http://localhost:8080" # required for /v1/search to work

searxng_url = "http://localhost:8080" # /v1/search接口正常工作所需配置

[crawler] — proxy and stealth

[crawler] — 代理与隐身模式

Single proxy:

单个代理：

proxy = "http://user:pass@proxy:8080"

proxy = "http://user:pass@proxy:8080"

proxy = "socks5://user:pass@proxy:1080"

proxy = "socks5://user:pass@proxy:1080"

Pool with rotation:

带轮询的代理池：

proxy_list = ["http://user:pass@a:8080", "http://user:pass@b:8080"]

proxy_list = ["http://user:pass@a:8080", "http://user:pass@b:8080"]

proxy_rotation = "sticky_per_host" # sticky_per_host | round_robin | random

proxy_rotation = "sticky_per_host" # sticky_per_host | round_robin | random

Stealth mode (rotate UA + inject browser-like headers):

隐身模式（轮换UA + 注入类浏览器头）：

[crawler.stealth]

[crawler.stealth]

enabled = true

enabled = true

inject_headers = true

inject_headers = true

jitter_factor = 0.2

jitter_factor = 0.2

[extraction.llm] — structured JSON extraction + summaries

[extraction.llm] — 结构化JSON提取 + 摘要

base_url = "https://custom-endpoint.example.com" # Chat Completions or Responses endpoints

base_url = "https://custom-endpoint.example.com" # Chat Completions或Responses端点

DeepSeek example:

DeepSeek示例：

provider = "deepseek"

provider = "deepseek"

api_key = "..."

api_key = "..."

model = "deepseek-chat"

model = "deepseek-chat"

base_url = "https://api.deepseek.com/v1"

base_url = "https://api.deepseek.com/v1"

Azure OpenAI example:

Azure OpenAI示例：

provider = "azure"

provider = "azure"

api_key = "..."

api_key = "..."

model = "gpt-4o-mini" # Azure deployment name

model = "gpt-4o-mini" # Azure部署名称

base_url = "https://<resource>.openai.azure.com"

base_url = "https://<resource>.openai.azure.com"

azure_api_version = "2024-05-01-preview"

azure_api_version = "2024-05-01-preview"

[auth] — API key auth

MCP server (
`crw-mcp`
) — recommended for AI agent use

MCP服务器（
`crw-mcp`
）—— 推荐AI Agent使用

CLI (
`crw`
) — scrape from the terminal

CLI（
`crw`
）—— 从终端进行爬取

API server (
`crw-server`
) — Firecrawl-compatible REST endpoint

API服务器（
`crw-server`
）—— 兼容Firecrawl的REST端点

Key config knobs (
`config.default.toml`
)

核心配置项（
`config.default.toml`
）

`[renderer]`
— JS rendering

`[renderer]`
— JS渲染

`[search]`
— web search

`[search]`
— 网页搜索

`[crawler]`
— proxy and stealth

`[crawler]`
— 代理与隐身模式

`[extraction.llm]`
— structured JSON extraction + summaries

`[extraction.llm]`
— 结构化JSON提取 + 摘要

`[auth]`
— API key auth

`[auth]`
— API密钥认证

`[document]`
— PDF parsing

`[document]`
— PDF解析