scrapfly-crawler

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Scrapfly Crawler

Scrapfly Crawler

Use the Scrapfly Crawler API via the Python SDK to schedule and manage site-wide crawl jobs with automatic link discovery, depth control, and structured data extraction.
通过 Python SDK 使用 Scrapfly Crawler API 调度和管理全站爬取任务,支持自动发现链接、深度控制和结构化数据提取。

When to use

适用场景

  • Crawling entire websites or specific sections
  • Discovering all pages/URLs on a site
  • Collecting content from multiple pages in bulk
  • Extracting structured data at scale across a site
  • Downloading site archives (WARC/HAR format)
  • Building LLM.txt files from crawled documentation
  • 爬取整个网站或特定区域
  • 发现网站上的所有页面/URL
  • 批量收集多个页面的内容
  • 在整个网站范围内大规模提取结构化数据
  • 下载网站存档(WARC/HAR 格式)
  • 根据爬取的文档构建 LLM.txt 文件

Setup

设置

bash
pip install scrapfly-sdk
The API key must be provided via environment variable
SCRAPFLY_API_KEY
or passed directly to the client.
bash
pip install scrapfly-sdk
必须通过环境变量
SCRAPFLY_API_KEY
提供 API key,或直接传给客户端。

SDK Imports

SDK 导入

python
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl
Additional imports for webhooks:
python
from scrapfly import (
    webhook_from_payload,
    CrawlStartedWebhook,
    CrawlUrlDiscoveredWebhook,
    CrawlUrlFailedWebhook,
    CrawlCompletedWebhook,
)
python
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl
Webhook 所需的额外导入:
python
from scrapfly import (
    webhook_from_payload,
    CrawlStartedWebhook,
    CrawlUrlDiscoveredWebhook,
    CrawlUrlFailedWebhook,
    CrawlCompletedWebhook,
)

CrawlerConfig Parameters

CrawlerConfig 参数

ParameterTypeDefaultDescription
url
strrequiredStarting URL for the crawl
page_limit
intNoneMaximum number of pages to crawl
max_depth
intNoneMaximum link traversal depth from start URL
max_duration
intNoneMaximum crawl duration in seconds
exclude_paths
list[str]NoneURL path patterns to skip (supports
*
, max 100). Mutually exclusive with
include_only_paths
include_only_paths
list[str]NoneOnly crawl URLs matching these path patterns (supports
*
, max 100). Mutually exclusive with
exclude_paths
ignore_base_path_restriction
boolFalseAllow crawling any path on the same domain, not just the seed URL's base path
follow_external_links
boolFalseFollow links to external domains
allowed_external_domains
list[str]NoneWhitelist of external domains when
follow_external_links=True
(supports
*
, max 250)
headers
dictNoneCustom HTTP headers as JSON object
delay
intNoneDelay between requests in milliseconds (0-15000)
user_agent
strNoneCustom User-Agent string (ignored when
unblocker=True
)
max_concurrency
intNoneMaximum concurrent crawl requests
rendering_delay
intNoneWait time in ms after page load before extraction (0-25000).
0
disables browser rendering
use_sitemaps
boolFalseDiscover URLs from
sitemap.xml
when available
respect_robots_txt
boolFalseRespect
robots.txt
rules and
Disallow
directives
ignore_no_follow
boolFalseIgnore
rel="nofollow"
attributes on links
cache
boolFalseEnable cache layer for crawled pages
cache_ttl
intNoneCache time-to-live in seconds (0-604800)
cache_clear
boolFalseForce refresh of cached pages
content_formats
list[str]NoneContent formats to extract:
html
,
clean_html
,
markdown
,
text
extraction_rules
dictNoneCustom extraction rules for structured data
unblocker
boolFalseEnable the Unblocker anti-bot bypass with browser rendering.
asp
is the permanently supported deprecated alias; when both are supplied,
asp
wins
proxy_pool
strNoneProxy pool to use (e.g.,
public_residential_pool
)
country
strNoneProxy country selection (ISO code)
webhook_name
strNoneName of webhook configured in dashboard
webhook_events
list[str]NoneEvents to subscribe to (e.g.,
crawler_started
,
crawler_url_visited
,
crawler_finished
)
max_api_credit
intNoneMaximum API credits to spend on this crawl
参数类型默认值说明
url
str必填爬取的起始 URL
page_limit
intNone最多爬取的页面数
max_depth
intNone从起始 URL 出发遍历链接的最大深度
max_duration
intNone最长爬取时长(秒)
exclude_paths
list[str]None要跳过的 URL 路径模式(支持
*
,最多 100 个)。与
include_only_paths
互斥
include_only_paths
list[str]None仅爬取匹配这些路径模式的 URL(支持
*
,最多 100 个)。与
exclude_paths
互斥
ignore_base_path_restriction
boolFalse允许爬取同一域名下的任意路径,而不仅限于种子 URL 的基础路径
follow_external_links
boolFalse跟随指向外部域名的链接
allowed_external_domains
list[str]None当
follow_external_links=True
时允许访问的外部域名白名单(支持
*
,最多 250 个)
headers
dictNone以 JSON 对象形式提供的自定义 HTTP headers
delay
intNone请求之间的延迟(毫秒,0-15000)
user_agent
strNone自定义 User-Agent 字符串(当
unblocker=True
时忽略)
max_concurrency
intNone最大并发爬取请求数
rendering_delay
intNone页面加载后执行提取前的等待时间(毫秒,0-25000)。
0
表示禁用浏览器渲染
use_sitemaps
boolFalse在可用时从
sitemap.xml
中发现 URL
respect_robots_txt
boolFalse遵守
robots.txt
规则和
Disallow
指令
ignore_no_follow
boolFalse忽略链接上的
rel="nofollow"
属性
cache
boolFalse为已爬取的页面启用缓存层
cache_ttl
intNone缓存生存时间(秒,0-604800)
cache_clear
boolFalse强制刷新已缓存的页面
content_formats
list[str]None要提取的内容格式:
html
、
clean_html
、
markdown
、
text
extraction_rules
dictNone用于结构化数据的自定义提取规则
unblocker
boolFalse启用带浏览器渲染的 Unblocker 反机器人拦截绕过功能。
asp
是永久支持但已弃用的别名;同时提供两者时,以
asp
为准
proxy_pool
strNone要使用的代理池(例如
public_residential_pool
)
country
strNone代理国家/地区选择(ISO 代码)
webhook_name
strNoneDashboard 中配置的 Webhook 名称
webhook_events
list[str]None要订阅的事件(例如
crawler_started
、
crawler_url_visited
、
crawler_finished
)
max_api_credit
intNone本次爬取最多可消耗的 API 额度

Webhook Events

Webhook 事件

  • "crawler_started"
    - Crawler execution began
  • "crawler_url_visited"
    - A URL was successfully crawled
  • "crawler_url_skipped"
    - URLs were skipped
  • "crawler_url_discovered"
    - New URLs were discovered
  • "crawler_url_failed"
    - A URL failed to crawl
  • "crawler_stopped"
    - Crawler stopped due to failure or limits
  • "crawler_cancelled"
    - Crawl was manually cancelled
  • "crawler_finished"
    - Crawler completed successfully
  • "crawler_started"
    - Crawler 执行已开始
  • "crawler_url_visited"
    - 一个 URL 已成功爬取
  • "crawler_url_skipped"
    - URL 被跳过
  • "crawler_url_discovered"
    - 发现了新的 URL
  • "crawler_url_failed"
    - 一个 URL 爬取失败
  • "crawler_stopped"
    - Crawler 因失败或达到限制而停止
  • "crawler_cancelled"
    - 爬取任务被手动取消
  • "crawler_finished"
    - Crawler 已成功完成

Examples

示例

Quick start: crawl and get results

快速开始:爬取并获取结果

python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

Create, start, and wait for crawl in one chain

Create, start, and wait for crawl in one chain

2. Create and run crawler

2. Create and run crawler

crawl = Crawl( client, CrawlerConfig(url='https://web-scraping.dev/products', page_limit=5) ).crawl().wait()
crawl = Crawl( client, CrawlerConfig(url='https://web-scraping.dev/products', page_limit=5) ).crawl().wait()

3. Get results

3. Get results

pages = crawl.warc().get_pages()
pages = crawl.warc().get_pages()

4. Process results

4. Process results

print(f"Crawled {len(pages)} pages:") for page in pages: print(f" • {page['url']} ({page['status_code']})")
print(f"Crawled {len(pages)} pages:") for page in pages: print(f" • {page['url']} ({page['status_code']})")

5. Access specific URLs

5. Access specific URLs

html = crawl.read('https://web-scraping.dev/products') if html: print(f"\nMain page is {len(html):,} bytes")
undefined
html = crawl.read('https://web-scraping.dev/products') if html: print(f"\nMain page is {len(html):,} bytes")
undefined

Low-level workflow: start, poll, download

底层工作流:启动、轮询、下载

python
import os
import time
from scrapfly import ScrapflyClient, CrawlerConfig

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

config = CrawlerConfig(
    url='https://web-scraping.dev',
    page_limit=10,
    max_depth=2,
    content_formats=['html', 'markdown']
)
python
import os
import time
from scrapfly import ScrapflyClient, CrawlerConfig

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

config = CrawlerConfig(
    url='https://web-scraping.dev',
    page_limit=10,
    max_depth=2,
    content_formats=['html', 'markdown']
)

1. Start crawler

1. Start crawler

start_response = client.start_crawl(config) print(f"Crawler started: {start_response.uuid}")
start_response = client.start_crawl(config) print(f"Crawler started: {start_response.uuid}")

2. Poll for status

2. Poll for status

while True: status = client.get_crawl_status(start_response.uuid) print(f"Status: {status.status} - {status.progress_pct:.1f}%") print(f"Crawled: {status.urls_crawled}/{status.urls_discovered} pages")
if status.is_complete:
    break
elif status.is_failed:
    print("Crawl failed!")
    break

time.sleep(5)
while True: status = client.get_crawl_status(start_response.uuid) print(f"Status: {status.status} - {status.progress_pct:.1f}%") print(f"Crawled: {status.urls_crawled}/{status.urls_discovered} pages")
if status.is_complete:
    break
elif status.is_failed:
    print("Crawl failed!")
    break

time.sleep(5)

3. Download and process results

3. Download and process results

if status.is_complete: artifact = client.get_crawl_artifact(start_response.uuid) pages = artifact.get_pages() print(f"Downloaded {len(pages)} pages")
for page in pages:
    print(f"  {page['url']}: {page['status_code']} ({len(page['content'])} bytes)")

# Save to file
artifact.save('crawl_results.warc.gz')
undefined
if status.is_complete: artifact = client.get_crawl_artifact(start_response.uuid) pages = artifact.get_pages() print(f"Downloaded {len(pages)} pages")
for page in pages:
    print(f"  {page['url']}: {page['status_code']} ({len(page['content'])} bytes)")

# Save to file
artifact.save('crawl_results.warc.gz')
undefined

Crawl with path filtering

带路径过滤的爬取

python
config = CrawlerConfig(
    url='https://web-scraping.dev/',
    page_limit=50,
    max_depth=5,
    include_only_paths=['*/product/*'],
    content_formats=['markdown', 'text'],
)
crawl = Crawl(client, config).crawl().wait()
python
config = CrawlerConfig(
    url='https://web-scraping.dev/',
    page_limit=50,
    max_depth=5,
    include_only_paths=['*/product/*'],
    content_formats=['markdown', 'text'],
)
crawl = Crawl(client, config).crawl().wait()

Crawl with anti-bot bypass

带反机器人拦截绕过的爬取

python
config = CrawlerConfig(
    url='https://web-scraping.dev',
    page_limit=50,
    unblocker=True,
    content_formats=['html'],
)
crawl = Crawl(client, config).crawl().wait()
python
config = CrawlerConfig(
    url='https://web-scraping.dev',
    page_limit=50,
    unblocker=True,
    content_formats=['html'],
)
crawl = Crawl(client, config).crawl().wait()

Crawl with extraction rules

使用提取规则爬取

python
config = CrawlerConfig(
    url='https://web-scraping.dev/products',
    page_limit=100,
    max_depth=3,
    include_only_paths=['/products/*'],
    extraction_rules={
        "/products/*": {
            "type": "model",
            "value": "product",
        },
    },
)
crawl = Crawl(client, config).crawl().wait()
python
config = CrawlerConfig(
    url='https://web-scraping.dev/products',
    page_limit=100,
    max_depth=3,
    include_only_paths=['/products/*'],
    extraction_rules={
        "/products/*": {
            "type": "model",
            "value": "product",
        },
    },
)
crawl = Crawl(client, config).crawl().wait()

Read content for specific URLs

读取指定 URL 的内容

python
undefined
python
undefined

Read HTML (from WARC - fast)

Read HTML (from WARC - fast)

content = crawl.read('https://web-scraping.dev/product/1') if content: print(f"URL: {content.url}") print(f"Status: {content.status_code}") print(f"Duration: {content.duration}s") print(content.content)
content = crawl.read('https://web-scraping.dev/product/1') if content: print(f"URL: {content.url}") print(f"Status: {content.status_code}") print(f"Duration: {content.duration}s") print(content.content)

Read markdown (from contents API)

Read markdown (from contents API)

content = crawl.read('https://web-scraping.dev/product/1', format='markdown') if content: print(content.content)
undefined
content = crawl.read('https://web-scraping.dev/product/1', format='markdown') if content: print(content.content)
undefined

Iterate URLs matching a pattern

迭代匹配指定模式的 URL

python
for content in crawl.read_iter(pattern="*/products?page=*", format="markdown"):
    print(f"{content.url}: {len(content.content)} chars")
python
for content in crawl.read_iter(pattern="*/products?page=*", format="markdown"):
    print(f"{content.url}: {len(content.content)} chars")

Batch content retrieval

批量内容检索

python
undefined
python
undefined

Get markdown for multiple URLs in a single API call (max 100 per request)

Get markdown for multiple URLs in a single API call (max 100 per request)

urls = ['https://web-scraping.dev/product/1', 'https://web-scraping.dev/product/2'] contents = crawl.read_batch(urls, formats=['markdown'])
for url, formats_dict in contents.items(): markdown = formats_dict.get('markdown', '') print(f"{url}: {len(markdown)} chars")
undefined
urls = ['https://web-scraping.dev/product/1', 'https://web-scraping.dev/product/2'] contents = crawl.read_batch(urls, formats=['markdown'])
for url, formats_dict in contents.items(): markdown = formats_dict.get('markdown', '') print(f"{url}: {len(markdown)} chars")
undefined

Download WARC/HAR archive

下载 WARC/HAR 存档

python
undefined
python
undefined

WARC format (default)

WARC format (default)

artifact = crawl.warc() artifact.save('crawl_results.warc.gz')
artifact = crawl.warc() artifact.save('crawl_results.warc.gz')

HAR format

HAR format

artifact = crawl.har() artifact.save('crawl_results.har')
artifact = crawl.har() artifact.save('crawl_results.har')

Iterate through records

Iterate through records

for record in artifact.iter_responses(): print(f"{record.url}: {record.status_code}")
undefined
for record in artifact.iter_responses(): print(f"{record.url}: {record.status_code}")
undefined

Build LLM.txt from crawled documentation

根据爬取的文档构建 LLM.txt

python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

config = CrawlerConfig(
    url='https://web-scraping.dev/',
    include_only_paths=['/docs', '/docs/*'], # only allow doc resources
    page_limit=50,
    max_depth=5,
    follow_external_links=False,
    content_formats=['markdown'],
)

crawl = Crawl(client, config).crawl().wait(poll_interval=5, verbose=True)
python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

config = CrawlerConfig(
    url='https://web-scraping.dev/',
    include_only_paths=['/docs', '/docs/*'], # only allow doc resources
    page_limit=50,
    max_depth=5,
    follow_external_links=False,
    content_formats=['markdown'],
)

crawl = Crawl(client, config).crawl().wait(poll_interval=5, verbose=True)

Get URLs from WARC

Get URLs from WARC

warc_artifact = crawl.warc() all_urls = [ record.url for record in warc_artifact.iter_responses() if record.status_code == 200 ]
warc_artifact = crawl.warc() all_urls = [ record.url for record in warc_artifact.iter_responses() if record.status_code == 200 ]

Retrieve markdown content in batches

Retrieve markdown content in batches

all_contents = {} for i in range(0, len(all_urls), 100): batch_urls = all_urls[i:i + 100] batch_contents = crawl.read_batch(batch_urls, formats=['markdown']) all_contents.update(batch_contents)
all_contents = {} for i in range(0, len(all_urls), 100): batch_urls = all_urls[i:i + 100] batch_contents = crawl.read_batch(batch_urls, formats=['markdown']) all_contents.update(batch_contents)

Build llms-full.txt

Build llms-full.txt

lines = ["# Documentation\n"] for url, formats_dict in all_contents.items(): markdown = formats_dict.get('markdown', '').strip() if markdown: lines.append(f"---\n\n### {url}\n\n{markdown}\n")
with open("llms-full.txt", "w", encoding="utf-8") as f: f.write("\n".join(lines))
undefined
lines = ["# Documentation\n"] for url, formats_dict in all_contents.items(): markdown = formats_dict.get('markdown', '').strip() if markdown: lines.append(f"---\n\n### {url}\n\n{markdown}\n")
with open("llms-full.txt", "w", encoding="utf-8") as f: f.write("\n".join(lines))
undefined

Crawl statistics

爬取统计信息

python
stats = crawl.stats()
print(f"URLs discovered: {stats['urls_discovered']}")
print(f"URLs crawled: {stats['urls_crawled']}")
print(f"Progress: {stats['progress_pct']:.1f}%")
python
stats = crawl.stats()
print(f"URLs discovered: {stats['urls_discovered']}")
print(f"URLs crawled: {stats['urls_crawled']}")
print(f"Progress: {stats['progress_pct']:.1f}%")

Key SDK Classes

核心 SDK 类

ClassDescription
ScrapflyClient
Main client - handles auth and API calls
CrawlerConfig
Crawl job configuration
Crawl
High-level crawl lifecycle manager (start, wait, read)
CrawlerStartResponse
Response from
client.start_crawl()
CrawlerStatusResponse
Response from
client.get_crawl_status()
- has
is_complete
,
is_failed
,
progress_pct
CrawlerArtifactResponse
Response from
client.get_crawl_artifact()
- has
get_pages()
,
iter_responses()
,
save()
CrawlContent
Content for a single URL - has
url
,
content
,
status_code
,
duration
类说明
ScrapflyClient
主客户端——负责身份验证和 API 调用
CrawlerConfig
爬取任务配置
Crawl
高级爬取生命周期管理器(启动、等待、读取)
CrawlerStartResponse
client.start_crawl()
的响应
CrawlerStatusResponse
client.get_crawl_status()
的响应——包含
is_complete
、
is_failed
、
progress_pct
CrawlerArtifactResponse
client.get_crawl_artifact()
的响应——包含
get_pages()
、
iter_responses()
、
save()
CrawlContent
单个 URL 的内容——包含
url
、
content
、
status_code
、
duration

Important Notes

重要说明

  • Use
    page_limit
    to control credit spend on large sites
  • Use
    include_only_paths
    and
    exclude_paths
    to focus crawls on relevant sections
  • exclude_paths
    and
    include_only_paths
    are mutually exclusive
  • Crawl jobs are asynchronous - use
    Crawl.crawl().wait()
    for simple workflows or poll with
    client.get_crawl_status()
    for more control
  • WARC/HAR artifacts are available for offline processing via
    crawl.warc()
    /
    crawl.har()
  • read_batch()
    supports up to 100 URLs per request for efficient bulk content retrieval
  • Webhook signatures should be verified for security in production
  • follow_external_links
    should be used carefully to avoid crawling the entire web
  • 使用
    page_limit
    控制大型站点的额度消耗
  • 使用
    include_only_paths
    和
    exclude_paths
    将爬取范围限定在相关区域
  • exclude_paths
    与
    include_only_paths
    互斥
  • 爬取任务是异步的——简单工作流可使用
    Crawl.crawl().wait()
    ,需要更多控制时可使用
    client.get_crawl_status()
    进行轮询
  • WARC/HAR 存档可通过
    crawl.warc()
    /
    crawl.har()
    获取,以便离线处理
  • read_batch()
    每次请求最多支持 100 个 URL,可用于高效批量检索内容
  • 在生产环境中应验证 Webhook 签名以确保安全
  • 谨慎使用
    follow_external_links
    ,避免爬取整个互联网