firecrawl-self-hosted

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Firecrawl Self-Hosted Operations

Firecrawl自托管运维指南

Self-hosted Firecrawl deployment, troubleshooting, and best practices.

Host: littleblack (172.25.236.1) via ZeroTier Source: https://github.com/mendableai/firecrawl

自托管Firecrawl的部署、故障排查及最佳实践。

主机：littleblack（172.25.236.1），通过ZeroTier连接源码：https://github.com/mendableai/firecrawl

When to Use This Skill

何时使用该技能

Use this skill when:

Scraping JavaScript-heavy web pages that WebFetch cannot handle
Extracting content from Gemini/ChatGPT share links
Operating the self-hosted Firecrawl instance on littleblack
Troubleshooting Docker container or ZeroTier connectivity issues
Setting up new Firecrawl deployments with proper restart policies

在以下场景使用本技能：

抓取WebFetch无法处理的重度依赖JavaScript的网页
从Gemini/ChatGPT分享链接提取内容
在littleblack上运行自托管Firecrawl实例
排查Docker容器或ZeroTier连接问题
配置带有正确重启策略的新Firecrawl部署

Architecture Overview

架构概述

┌─────────────────────────────────────────────────────────────────┐
│                    LittleBlack (172.25.236.1)                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐      │
│  │   Client     │───▶│ Scraper      │───▶│ Firecrawl    │      │
│  │   (curl)     │    │ Wrapper :3003│    │ API :3002    │      │
│  └──────────────┘    └──────────────┘    └──────────────┘      │
│         │                   │                   │               │
│         │                   │                   ▼               │
│         │                   │            ┌──────────────┐       │
│         │                   │            │ Playwright   │       │
│         │                   │            │ Service      │       │
│         │                   │            └──────────────┘       │
│         │                   │                   │               │
│         │                   ▼                   ▼               │
│         │            ┌──────────────┐    ┌──────────────┐       │
│         │            │ Caddy :8080  │    │ Redis        │       │
│         │            │ (files)      │    │ RabbitMQ     │       │
│         ▼            └──────────────┘    └──────────────┘       │
│  ┌──────────────┐                                               │
│  │ Output URL   │◀── http://172.25.236.1:8080/NAME-TS.md       │
│  └──────────────┘                                               │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                    LittleBlack (172.25.236.1)                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐      │
│  │   Client     │───▶│ Scraper      │───▶│ Firecrawl    │      │
│  │   (curl)     │    │ Wrapper :3003│    │ API :3002    │      │
│  └──────────────┘    └──────────────┘    └──────────────┘      │
│         │                   │                   │               │
│         │                   │                   ▼               │
│         │                   │            ┌──────────────┐       │
│         │                   │            │ Playwright   │       │
│         │                   │            │ Service      │       │
│         │                   │            └──────────────┘       │
│         │                   │                   │               │
│         │                   ▼                   ▼               │
│         │            ┌──────────────┐    ┌──────────────┐       │
│         │            │ Caddy :8080  │    │ Redis        │       │
│         │            │ (files)      │    │ RabbitMQ     │       │
│         ▼            └──────────────┘    └──────────────┘       │
│  ┌──────────────┐                                               │
│  │ Output URL   │◀── http://172.25.236.1:8080/NAME-TS.md       │
│  └──────────────┘                                               │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Quick Reference

快速参考

Port	Service	Type	Purpose
3002	Firecrawl API	Docker	Core scraping engine
3003	Scraper Wrapper	Bun	Saves to file, returns URL
8080	Caddy	Binary	Serves saved markdown

端口	服务名称	类型	用途
3002	Firecrawl API	Docker	核心抓取引擎
3003	Scraper Wrapper	Bun	保存到文件并返回URL
8080	Caddy	Binary	提供已保存的Markdown文件服务

Usage

使用方法

Recommended: Wrapper Endpoint

推荐方式：包装器端点

bash

curl "http://172.25.236.1:3003/scrape?url=URL&name=NAME"

Returns:

json

{
  "url": "http://172.25.236.1:8080/NAME-TIMESTAMP.md",
  "file": "NAME-TIMESTAMP.md"
}

bash

curl "http://172.25.236.1:3003/scrape?url=URL&name=NAME"

返回结果：

json

{
  "url": "http://172.25.236.1:8080/NAME-TIMESTAMP.md",
  "file": "NAME-TIMESTAMP.md"
}

Direct API (Advanced)

直接调用API（进阶）

bash

curl -s -X POST http://172.25.236.1:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"URL","formats":["markdown"],"waitFor":5000}' \
  | jq -r '.data.markdown'

bash

curl -s -X POST http://172.25.236.1:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"URL","formats":["markdown"],"waitFor":5000}' \
  | jq -r '.data.markdown'

Health Checks

健康检查

Quick Status

快速状态检查

bash

undefined

bash

undefined

All containers running?

所有容器是否在运行？

ssh littleblack 'docker ps --filter "name=firecrawl" --format "{{.Names}}: {{.Status}}"'

API responding?

API是否响应？

ssh littleblack 'curl -s -o /dev/null -w "%{http_code}" http://localhost:3002/v1/scrape'

Expected: 401 (no payload) or 200 (with payload)

预期结果：401（无请求体）或200（有请求体）

Wrapper responding?

包装器是否响应？

curl -s -o /dev/null -w "%{http_code}" "http://172.25.236.1:3003/health"

undefined

curl -s -o /dev/null -w "%{http_code}" "http://172.25.236.1:3003/health"

undefined

Detailed Status

详细状态检查

bash

undefined

bash

undefined

systemd services

systemd服务状态

ssh littleblack "systemctl --user status firecrawl firecrawl-scraper caddy-firecrawl"

Docker container details

Docker容器详情

ssh littleblack 'docker ps -a --filter "name=firecrawl" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'

Logs (live)

实时日志

ssh littleblack "journalctl --user -u firecrawl -u firecrawl-scraper -u caddy-firecrawl -f"

---

ssh littleblack "journalctl --user -u firecrawl -u firecrawl-scraper -u caddy-firecrawl -f"

---

Troubleshooting

故障排查

Symptom: API Container Stopped

症状：API容器已停止

Root Cause: Docker restart policy was

no

(default). Container received SIGINT and didn't restart.

Diagnosis:

bash

undefined

根本原因：Docker默认重启策略为

no

，容器收到SIGINT信号后不会自动重启。

诊断：

bash

undefined

Check container status

检查容器状态

ssh littleblack 'docker ps -a --filter "name=firecrawl"'

Check restart policy

检查重启策略

ssh littleblack 'docker inspect --format "{{.Name}}: {{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'


**Fix**: Add `restart: unless-stopped` to ALL services in `docker-compose.yaml`:

```yaml

ssh littleblack 'docker inspect --format "{{.Name}}: {{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'


**修复方案**：在`docker-compose.yaml`的所有服务中添加`restart: unless-stopped`：

```yaml

~/firecrawl/docker-compose.yaml

x-common-service: &common-service networks: - backend restart: unless-stopped # CRITICAL: Add this line logging: driver: "json-file" options: max-size: "1G" max-file: "4"

services: playwright-service: <<: *common-service # ... rest of config

api: <<: *common-service # ... rest of config

redis: <<: *common-service # ... rest of config

rabbitmq: <<: *common-service # ... rest of config


**Apply Fix**:

```bash
ssh littleblack 'cd ~/firecrawl && docker compose up -d --force-recreate'

Verify:

bash

ssh littleblack 'docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'

x-common-service: &common-service networks: - backend restart: unless-stopped # 关键：添加此行 logging: driver: "json-file" options: max-size: "1G" max-file: "4"

services: playwright-service: <<: *common-service # ... 其余配置

api: <<: *common-service # ... 其余配置

redis: <<: *common-service # ... 其余配置

rabbitmq: <<: *common-service # ... 其余配置


**应用修复**：

```bash
ssh littleblack 'cd ~/firecrawl && docker compose up -d --force-recreate'

验证：

bash

ssh littleblack 'docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'

All should show: RestartPolicy=unless-stopped

所有容器应显示：RestartPolicy=unless-stopped

undefined

undefined

Symptom: Scraper Wrapper Not Responding

症状：抓取器包装器无响应

Diagnosis:

bash

ssh littleblack "systemctl --user status firecrawl-scraper"

Fix:

bash

ssh littleblack "systemctl --user restart firecrawl-scraper"

诊断：

bash

ssh littleblack "systemctl --user status firecrawl-scraper"

修复方案：

bash

ssh littleblack "systemctl --user restart firecrawl-scraper"

Symptom: Caddy File Server Down

症状：Caddy文件服务中断

Diagnosis:

bash

ssh littleblack "systemctl --user status caddy-firecrawl"
curl -I http://172.25.236.1:8080/

Fix:

bash

ssh littleblack "systemctl --user restart caddy-firecrawl"

诊断：

bash

ssh littleblack "systemctl --user status caddy-firecrawl"
curl -I http://172.25.236.1:8080/

修复方案：

bash

ssh littleblack "systemctl --user restart caddy-firecrawl"

Symptom: ZeroTier Unreachable

症状：ZeroTier无法连接

Diagnosis:

bash

undefined

诊断：

bash

undefined

From local machine

在本地机器执行

ping 172.25.236.1

Check ZeroTier status

检查ZeroTier状态

zerotier-cli listnetworks


**Fix**: Re-authorize device in ZeroTier Central if needed.

---

zerotier-cli listnetworks


**修复方案**：如有需要，在ZeroTier Central中重新授权设备。

---

Bootstrap: Fresh Installation

初始化：全新安装

Prerequisites

前提条件

Debian/Ubuntu server with Docker
ZeroTier network membership
Domain or static IP (optional, for public access)

带有Docker的Debian/Ubuntu服务器
ZeroTier网络成员资格
域名或静态IP（可选，用于公网访问）

Step 1: Clone Repository

步骤1：克隆仓库

bash

cd ~
git clone https://github.com/mendableai/firecrawl.git
cd firecrawl

bash

cd ~
git clone https://github.com/mendableai/firecrawl.git
cd firecrawl

Step 2: Configure docker-compose.yaml

步骤2：配置docker-compose.yaml

CRITICAL: Add restart policy to prevent shutdown on signals:

yaml

x-common-service: &common-service
  networks:
    - backend
  restart: unless-stopped # <-- ADD THIS
  logging:
    driver: "json-file"
    options:
      max-size: "1G"
      max-file: "4"

Apply to all services using the anchor:

yaml

services:
  api:
    <<: *common-service
    # ...
  playwright-service:
    <<: *common-service
    # ...
  redis:
    <<: *common-service
    # ...
  rabbitmq:
    <<: *common-service
    # ...

关键：添加重启策略以防止容器因信号终止：

yaml

x-common-service: &common-service
  networks:
    - backend
  restart: unless-stopped # <-- 添加此行
  logging:
    driver: "json-file"
    options:
      max-size: "1G"
      max-file: "4"

通过锚点将配置应用到所有服务：

yaml

services:
  api:
    <<: *common-service
    # ...
  playwright-service:
    <<: *common-service
    # ...
  redis:
    <<: *common-service
    # ...
  rabbitmq:
    <<: *common-service
    # ...

Step 3: Environment Variables

步骤3：环境变量

Create

.env

from template:

bash

cp .env.example .env

Minimal required settings:

bash

undefined

从模板创建

.env

文件：

bash

cp .env.example .env

最小必要配置：

bash

undefined

.env

NUM_WORKERS_PER_QUEUE=2 PORT=3002 HOST=0.0.0.0 REDIS_URL=redis://redis:6379 REDIS_RATE_LIMIT_URL=redis://redis:6379

undefined

NUM_WORKERS_PER_QUEUE=2 PORT=3002 HOST=0.0.0.0 REDIS_URL=redis://redis:6379 REDIS_RATE_LIMIT_URL=redis://redis:6379

undefined

Step 4: Start Services

步骤4：启动服务

bash

docker compose up -d

bash

docker compose up -d

Step 5: Verify Restart Policies

步骤5：验证重启策略

bash

docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" \
  $(docker ps -a --filter "name=firecrawl" -q)

All should show

unless-stopped

bash

docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" \
  $(docker ps -a --filter "name=firecrawl" -q)

所有容器应显示

unless-stopped

。

Step 6: Optional - Scraper Wrapper

步骤6：可选 - 抓取器包装器

Create

~/firecrawl-scraper.ts

typescript

import { serve } from "bun";
import { $ } from "bun";

const FIRECRAWL_API = "http://localhost:3002";
const OUTPUT_DIR = "/home/kab/firecrawl-output";

serve({
  port: 3003,
  async fetch(req) {
    const url = new URL(req.url);

    if (url.pathname === "/health") {
      return new Response("OK", { status: 200 });
    }

    if (url.pathname === "/scrape") {
      const targetUrl = url.searchParams.get("url");
      const name = url.searchParams.get("name") || "scraped";

      if (!targetUrl) {
        return Response.json(
          { error: "url parameter required" },
          { status: 400 },
        );
      }

      const response = await fetch(`${FIRECRAWL_API}/v1/scrape`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          url: targetUrl,
          formats: ["markdown"],
          waitFor: 5000,
        }),
      });

      const data = await response.json();
      const markdown = data?.data?.markdown;

      if (!markdown) {
        return Response.json(
          { error: "No markdown returned" },
          { status: 500 },
        );
      }

      const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
      const filename = `${name}-${timestamp}.md`;
      const filepath = `${OUTPUT_DIR}/${filename}`;

      await Bun.write(filepath, markdown);

      return Response.json({
        url: `http://172.25.236.1:8080/${filename}`,
        file: filename,
      });
    }

    return new Response("Not Found", { status: 404 });
  },
});

Create systemd user service

~/.config/systemd/user/firecrawl-scraper.service

ini

[Unit]
Description=Firecrawl Scraper Wrapper
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/kab
ExecStart=/home/kab/.bun/bin/bun run firecrawl-scraper.ts
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

Enable:

bash

systemctl --user daemon-reload
systemctl --user enable --now firecrawl-scraper

创建

~/firecrawl-scraper.ts

：

typescript

import { serve } from "bun";
import { $ } from "bun";

const FIRECRAWL_API = "http://localhost:3002";
const OUTPUT_DIR = "/home/kab/firecrawl-output";

serve({
  port: 3003,
  async fetch(req) {
    const url = new URL(req.url);

    if (url.pathname === "/health") {
      return new Response("OK", { status: 200 });
    }

    if (url.pathname === "/scrape") {
      const targetUrl = url.searchParams.get("url");
      const name = url.searchParams.get("name") || "scraped";

      if (!targetUrl) {
        return Response.json(
          { error: "url parameter required" },
          { status: 400 },
        );
      }

      const response = await fetch(`${FIRECRAWL_API}/v1/scrape`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          url: targetUrl,
          formats: ["markdown"],
          waitFor: 5000,
        }),
      });

      const data = await response.json();
      const markdown = data?.data?.markdown;

      if (!markdown) {
        return Response.json(
          { error: "No markdown returned" },
          { status: 500 },
        );
      }

      const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
      const filename = `${name}-${timestamp}.md`;
      const filepath = `${OUTPUT_DIR}/${filename}`;

      await Bun.write(filepath, markdown);

      return Response.json({
        url: `http://172.25.236.1:8080/${filename}`,
        file: filename,
      });
    }

    return new Response("Not Found", { status: 404 });
  },
});

创建systemd用户服务

~/.config/systemd/user/firecrawl-scraper.service

：

ini

[Unit]
Description=Firecrawl Scraper Wrapper
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/kab
ExecStart=/home/kab/.bun/bin/bun run firecrawl-scraper.ts
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

启用服务：

bash

systemctl --user daemon-reload
systemctl --user enable --now firecrawl-scraper

Step 7: Optional - Caddy File Server

步骤7：可选 - Caddy文件服务

Download Caddy from GitHub releases (latest version).

bash

undefined

从GitHub releases下载Caddy（最新版本）。

bash

undefined

Download and extract (check releases for current version)

下载并解压（请检查发布页面获取当前版本）

wget https://github.com/caddyserver/caddy/releases/download/v<version>/caddy_<version>linux_amd64.tar.gz # SSoT-OK tar xzf caddy*.tar.gz chmod +x caddy


Create systemd user service `~/.config/systemd/user/caddy-firecrawl.service`:

```ini
[Unit]
Description=Caddy Firecrawl File Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/kab
ExecStart=/home/kab/caddy file-server --root /home/kab/firecrawl-output --listen :8080 --browse
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

Enable:

bash

systemctl --user daemon-reload
systemctl --user enable --now caddy-firecrawl

wget https://github.com/caddyserver/caddy/releases/download/v<version>/caddy_<version>linux_amd64.tar.gz # SSoT-OK tar xzf caddy*.tar.gz chmod +x caddy


创建systemd用户服务`~/.config/systemd/user/caddy-firecrawl.service`：

```ini
[Unit]
Description=Caddy Firecrawl File Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/kab
ExecStart=/home/kab/caddy file-server --root /home/kab/firecrawl-output --listen :8080 --browse
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

启用服务：

bash

systemctl --user daemon-reload
systemctl --user enable --now caddy-firecrawl

Best Practices (Empirically Verified)

最佳实践（经验证）

1. Always Use

restart: unless-stopped

1. 始终使用

restart: unless-stopped

Docker default is

no

restart policy. Containers WILL stop on SIGINT/SIGTERM and not recover.

Anti-pattern:

yaml

services:
  api:
    image: firecrawl/api
    # Missing restart policy = container dies and stays dead

Correct:

yaml

services:
  api:
    image: firecrawl/api
    restart: unless-stopped # Auto-restart on crash or signal

Docker默认重启策略为

no

，容器在收到SIGINT/SIGTERM信号后会停止且不会自动恢复。

反模式：

yaml

services:
  api:
    image: firecrawl/api
    # 缺少重启策略 = 容器停止后无法自动恢复

正确做法：

yaml

services:
  api:
    image: firecrawl/api
    restart: unless-stopped # 崩溃或收到信号时自动重启

2. Use YAML Anchors for Consistency

2. 使用YAML锚点保证一致性

Don't repeat

restart: unless-stopped

for each service. Use anchors:

yaml

x-common-service: &common-service
  restart: unless-stopped
  logging:
    driver: "json-file"
    options:
      max-size: "1G"
      max-file: "4"

services:
  api:
    <<: *common-service
    # ...

不要为每个服务重复

restart: unless-stopped

，使用锚点：

yaml

x-common-service: &common-service
  restart: unless-stopped
  logging:
    driver: "json-file"
    options:
      max-size: "1G"
      max-file: "4"

services:
  api:
    <<: *common-service
    # ...

3. Verify After docker compose up

3. 在docker compose up后进行验证

ALWAYS verify restart policies after

docker compose up -d

bash

docker inspect --format "{{.Name}}: {{.HostConfig.RestartPolicy.Name}}" \
  $(docker ps -a --filter "name=firecrawl" -q)

执行

docker compose up -d

后，务必验证重启策略：

bash

docker inspect --format "{{.Name}}: {{.HostConfig.RestartPolicy.Name}}" \
  $(docker ps -a --filter "name=firecrawl" -q)

4. Use systemd for Non-Docker Services

4. 为非Docker服务使用systemd

For Bun scripts and Caddy, use systemd with

Restart=always

ini

[Service]
Restart=always
RestartSec=5

对于Bun脚本和Caddy，使用带有

Restart=always

的systemd服务：

ini

[Service]
Restart=always
RestartSec=5

5. Monitor with Health Checks

5. 通过健康检查进行监控

Add periodic health check to catch silent failures:

bash

undefined

添加定期健康检查以发现静默故障：

bash

undefined

Add to crontab

添加到crontab

*/5 * * * * curl -sf http://localhost:3002/health || systemctl --user restart firecrawl

---

*/5 * * * * curl -sf http://localhost:3002/health || systemctl --user restart firecrawl

---

Files Reference

文件参考

Path on LittleBlack	Purpose
`~/firecrawl/`	Firecrawl Docker deployment
`~/firecrawl/docker-compose.yaml`	Docker orchestration (EDIT THIS)
`~/firecrawl/.env`	Environment configuration
`~/firecrawl-scraper.ts`	Bun wrapper script
`~/firecrawl-output/`	Saved markdown files (Caddy root)
`~/caddy`	Caddy binary
`~/.config/systemd/user/`	User systemd services

LittleBlack上的路径	用途
`~/firecrawl/`	Firecrawl Docker部署目录
`~/firecrawl/docker-compose.yaml`	Docker编排配置文件（需编辑）
`~/firecrawl/.env`	环境变量配置文件
`~/firecrawl-scraper.ts`	Bun包装器脚本
`~/firecrawl-output/`	已保存的Markdown文件目录（Caddy根目录）
`~/caddy`	Caddy二进制文件
`~/.config/systemd/user/`	用户级systemd服务配置目录

Recovery Commands Cheatsheet

恢复命令速查表

bash

undefined

bash

undefined

Full restart (all services)

完全重启所有服务

ssh littleblack 'cd ~/firecrawl && docker compose restart' ssh littleblack 'systemctl --user restart firecrawl-scraper caddy-firecrawl'

Check everything

检查所有状态

ssh littleblack 'docker ps --filter "name=firecrawl" && systemctl --user status firecrawl-scraper caddy-firecrawl --no-pager'

Logs (last 100 lines)

日志（最后100行）

ssh littleblack 'docker logs firecrawl-api-1 --tail 100' ssh littleblack 'journalctl --user -u firecrawl-scraper --no-pager -n 100'

Force recreate with new config

强制重新创建并应用新配置

ssh littleblack 'cd ~/firecrawl && docker compose up -d --force-recreate'

Verify restart policies

验证重启策略

ssh littleblack 'docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'

---

ssh littleblack 'docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'

---