scrapfly-crawler
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseScrapfly Crawler
Scrapfly Crawler
Use the Scrapfly Crawler API via the Python SDK to schedule and manage site-wide crawl jobs with automatic link discovery, depth control, and structured data extraction.
通过 Python SDK 使用 Scrapfly Crawler API 调度和管理全站爬取任务,支持自动发现链接、深度控制和结构化数据提取。
When to use
适用场景
- Crawling entire websites or specific sections
- Discovering all pages/URLs on a site
- Collecting content from multiple pages in bulk
- Extracting structured data at scale across a site
- Downloading site archives (WARC/HAR format)
- Building LLM.txt files from crawled documentation
- 爬取整个网站或特定区域
- 发现网站上的所有页面/URL
- 批量收集多个页面的内容
- 在整个网站范围内大规模提取结构化数据
- 下载网站存档(WARC/HAR 格式)
- 根据爬取的文档构建 LLM.txt 文件
Setup
设置
bash
pip install scrapfly-sdkThe API key must be provided via environment variable or passed directly to the client.
SCRAPFLY_API_KEYbash
pip install scrapfly-sdk必须通过环境变量 提供 API key,或直接传给客户端。
SCRAPFLY_API_KEYSDK Imports
SDK 导入
python
from scrapfly import ScrapflyClient, CrawlerConfig, CrawlAdditional imports for webhooks:
python
from scrapfly import (
webhook_from_payload,
CrawlStartedWebhook,
CrawlUrlDiscoveredWebhook,
CrawlUrlFailedWebhook,
CrawlCompletedWebhook,
)python
from scrapfly import ScrapflyClient, CrawlerConfig, CrawlWebhook 所需的额外导入:
python
from scrapfly import (
webhook_from_payload,
CrawlStartedWebhook,
CrawlUrlDiscoveredWebhook,
CrawlUrlFailedWebhook,
CrawlCompletedWebhook,
)CrawlerConfig Parameters
CrawlerConfig 参数
| Parameter | Type | Default | Description |
|---|---|---|---|
| str | required | Starting URL for the crawl |
| int | None | Maximum number of pages to crawl |
| int | None | Maximum link traversal depth from start URL |
| int | None | Maximum crawl duration in seconds |
| list[str] | None | URL path patterns to skip (supports |
| list[str] | None | Only crawl URLs matching these path patterns (supports |
| bool | False | Allow crawling any path on the same domain, not just the seed URL's base path |
| bool | False | Follow links to external domains |
| list[str] | None | Whitelist of external domains when |
| dict | None | Custom HTTP headers as JSON object |
| int | None | Delay between requests in milliseconds (0-15000) |
| str | None | Custom User-Agent string (ignored when |
| int | None | Maximum concurrent crawl requests |
| int | None | Wait time in ms after page load before extraction (0-25000). |
| bool | False | Discover URLs from |
| bool | False | Respect |
| bool | False | Ignore |
| bool | False | Enable cache layer for crawled pages |
| int | None | Cache time-to-live in seconds (0-604800) |
| bool | False | Force refresh of cached pages |
| list[str] | None | Content formats to extract: |
| dict | None | Custom extraction rules for structured data |
| bool | False | Enable the Unblocker anti-bot bypass with browser rendering. |
| str | None | Proxy pool to use (e.g., |
| str | None | Proxy country selection (ISO code) |
| str | None | Name of webhook configured in dashboard |
| list[str] | None | Events to subscribe to (e.g., |
| int | None | Maximum API credits to spend on this crawl |
| 参数 | 类型 | 默认值 | 说明 |
|---|---|---|---|
| str | 必填 | 爬取的起始 URL |
| int | None | 最多爬取的页面数 |
| int | None | 从起始 URL 出发遍历链接的最大深度 |
| int | None | 最长爬取时长(秒) |
| list[str] | None | 要跳过的 URL 路径模式(支持 |
| list[str] | None | 仅爬取匹配这些路径模式的 URL(支持 |
| bool | False | 允许爬取同一域名下的任意路径,而不仅限于种子 URL 的基础路径 |
| bool | False | 跟随指向外部域名的链接 |
| list[str] | None | 当 |
| dict | None | 以 JSON 对象形式提供的自定义 HTTP headers |
| int | None | 请求之间的延迟(毫秒,0-15000) |
| str | None | 自定义 User-Agent 字符串(当 |
| int | None | 最大并发爬取请求数 |
| int | None | 页面加载后执行提取前的等待时间(毫秒,0-25000)。 |
| bool | False | 在可用时从 |
| bool | False | 遵守 |
| bool | False | 忽略链接上的 |
| bool | False | 为已爬取的页面启用缓存层 |
| int | None | 缓存生存时间(秒,0-604800) |
| bool | False | 强制刷新已缓存的页面 |
| list[str] | None | 要提取的内容格式: |
| dict | None | 用于结构化数据的自定义提取规则 |
| bool | False | 启用带浏览器渲染的 Unblocker 反机器人拦截绕过功能。 |
| str | None | 要使用的代理池(例如 |
| str | None | 代理国家/地区选择(ISO 代码) |
| str | None | Dashboard 中配置的 Webhook 名称 |
| list[str] | None | 要订阅的事件(例如 |
| int | None | 本次爬取最多可消耗的 API 额度 |
Webhook Events
Webhook 事件
- - Crawler execution began
"crawler_started" - - A URL was successfully crawled
"crawler_url_visited" - - URLs were skipped
"crawler_url_skipped" - - New URLs were discovered
"crawler_url_discovered" - - A URL failed to crawl
"crawler_url_failed" - - Crawler stopped due to failure or limits
"crawler_stopped" - - Crawl was manually cancelled
"crawler_cancelled" - - Crawler completed successfully
"crawler_finished"
- - Crawler 执行已开始
"crawler_started" - - 一个 URL 已成功爬取
"crawler_url_visited" - - URL 被跳过
"crawler_url_skipped" - - 发现了新的 URL
"crawler_url_discovered" - - 一个 URL 爬取失败
"crawler_url_failed" - - Crawler 因失败或达到限制而停止
"crawler_stopped" - - 爬取任务被手动取消
"crawler_cancelled" - - Crawler 已成功完成
"crawler_finished"
Examples
示例
Quick start: crawl and get results
快速开始:爬取并获取结果
python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])Create, start, and wait for crawl in one chain
Create, start, and wait for crawl in one chain
2. Create and run crawler
2. Create and run crawler
crawl = Crawl(
client,
CrawlerConfig(url='https://web-scraping.dev/products', page_limit=5)
).crawl().wait()
crawl = Crawl(
client,
CrawlerConfig(url='https://web-scraping.dev/products', page_limit=5)
).crawl().wait()
3. Get results
3. Get results
pages = crawl.warc().get_pages()
pages = crawl.warc().get_pages()
4. Process results
4. Process results
print(f"Crawled {len(pages)} pages:")
for page in pages:
print(f" • {page['url']} ({page['status_code']})")
print(f"Crawled {len(pages)} pages:")
for page in pages:
print(f" • {page['url']} ({page['status_code']})")
5. Access specific URLs
5. Access specific URLs
html = crawl.read('https://web-scraping.dev/products')
if html:
print(f"\nMain page is {len(html):,} bytes")
undefinedhtml = crawl.read('https://web-scraping.dev/products')
if html:
print(f"\nMain page is {len(html):,} bytes")
undefinedLow-level workflow: start, poll, download
底层工作流:启动、轮询、下载
python
import os
import time
from scrapfly import ScrapflyClient, CrawlerConfig
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
config = CrawlerConfig(
url='https://web-scraping.dev',
page_limit=10,
max_depth=2,
content_formats=['html', 'markdown']
)python
import os
import time
from scrapfly import ScrapflyClient, CrawlerConfig
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
config = CrawlerConfig(
url='https://web-scraping.dev',
page_limit=10,
max_depth=2,
content_formats=['html', 'markdown']
)1. Start crawler
1. Start crawler
start_response = client.start_crawl(config)
print(f"Crawler started: {start_response.uuid}")
start_response = client.start_crawl(config)
print(f"Crawler started: {start_response.uuid}")
2. Poll for status
2. Poll for status
while True:
status = client.get_crawl_status(start_response.uuid)
print(f"Status: {status.status} - {status.progress_pct:.1f}%")
print(f"Crawled: {status.urls_crawled}/{status.urls_discovered} pages")
if status.is_complete:
break
elif status.is_failed:
print("Crawl failed!")
break
time.sleep(5)while True:
status = client.get_crawl_status(start_response.uuid)
print(f"Status: {status.status} - {status.progress_pct:.1f}%")
print(f"Crawled: {status.urls_crawled}/{status.urls_discovered} pages")
if status.is_complete:
break
elif status.is_failed:
print("Crawl failed!")
break
time.sleep(5)3. Download and process results
3. Download and process results
if status.is_complete:
artifact = client.get_crawl_artifact(start_response.uuid)
pages = artifact.get_pages()
print(f"Downloaded {len(pages)} pages")
for page in pages:
print(f" {page['url']}: {page['status_code']} ({len(page['content'])} bytes)")
# Save to file
artifact.save('crawl_results.warc.gz')undefinedif status.is_complete:
artifact = client.get_crawl_artifact(start_response.uuid)
pages = artifact.get_pages()
print(f"Downloaded {len(pages)} pages")
for page in pages:
print(f" {page['url']}: {page['status_code']} ({len(page['content'])} bytes)")
# Save to file
artifact.save('crawl_results.warc.gz')undefinedCrawl with path filtering
带路径过滤的爬取
python
config = CrawlerConfig(
url='https://web-scraping.dev/',
page_limit=50,
max_depth=5,
include_only_paths=['*/product/*'],
content_formats=['markdown', 'text'],
)
crawl = Crawl(client, config).crawl().wait()python
config = CrawlerConfig(
url='https://web-scraping.dev/',
page_limit=50,
max_depth=5,
include_only_paths=['*/product/*'],
content_formats=['markdown', 'text'],
)
crawl = Crawl(client, config).crawl().wait()Crawl with anti-bot bypass
带反机器人拦截绕过的爬取
python
config = CrawlerConfig(
url='https://web-scraping.dev',
page_limit=50,
unblocker=True,
content_formats=['html'],
)
crawl = Crawl(client, config).crawl().wait()python
config = CrawlerConfig(
url='https://web-scraping.dev',
page_limit=50,
unblocker=True,
content_formats=['html'],
)
crawl = Crawl(client, config).crawl().wait()Crawl with extraction rules
使用提取规则爬取
python
config = CrawlerConfig(
url='https://web-scraping.dev/products',
page_limit=100,
max_depth=3,
include_only_paths=['/products/*'],
extraction_rules={
"/products/*": {
"type": "model",
"value": "product",
},
},
)
crawl = Crawl(client, config).crawl().wait()python
config = CrawlerConfig(
url='https://web-scraping.dev/products',
page_limit=100,
max_depth=3,
include_only_paths=['/products/*'],
extraction_rules={
"/products/*": {
"type": "model",
"value": "product",
},
},
)
crawl = Crawl(client, config).crawl().wait()Read content for specific URLs
读取指定 URL 的内容
python
undefinedpython
undefinedRead HTML (from WARC - fast)
Read HTML (from WARC - fast)
content = crawl.read('https://web-scraping.dev/product/1')
if content:
print(f"URL: {content.url}")
print(f"Status: {content.status_code}")
print(f"Duration: {content.duration}s")
print(content.content)
content = crawl.read('https://web-scraping.dev/product/1')
if content:
print(f"URL: {content.url}")
print(f"Status: {content.status_code}")
print(f"Duration: {content.duration}s")
print(content.content)
Read markdown (from contents API)
Read markdown (from contents API)
content = crawl.read('https://web-scraping.dev/product/1', format='markdown')
if content:
print(content.content)
undefinedcontent = crawl.read('https://web-scraping.dev/product/1', format='markdown')
if content:
print(content.content)
undefinedIterate URLs matching a pattern
迭代匹配指定模式的 URL
python
for content in crawl.read_iter(pattern="*/products?page=*", format="markdown"):
print(f"{content.url}: {len(content.content)} chars")python
for content in crawl.read_iter(pattern="*/products?page=*", format="markdown"):
print(f"{content.url}: {len(content.content)} chars")Batch content retrieval
批量内容检索
python
undefinedpython
undefinedGet markdown for multiple URLs in a single API call (max 100 per request)
Get markdown for multiple URLs in a single API call (max 100 per request)
urls = ['https://web-scraping.dev/product/1', 'https://web-scraping.dev/product/2']
contents = crawl.read_batch(urls, formats=['markdown'])
for url, formats_dict in contents.items():
markdown = formats_dict.get('markdown', '')
print(f"{url}: {len(markdown)} chars")
undefinedurls = ['https://web-scraping.dev/product/1', 'https://web-scraping.dev/product/2']
contents = crawl.read_batch(urls, formats=['markdown'])
for url, formats_dict in contents.items():
markdown = formats_dict.get('markdown', '')
print(f"{url}: {len(markdown)} chars")
undefinedDownload WARC/HAR archive
下载 WARC/HAR 存档
python
undefinedpython
undefinedWARC format (default)
WARC format (default)
artifact = crawl.warc()
artifact.save('crawl_results.warc.gz')
artifact = crawl.warc()
artifact.save('crawl_results.warc.gz')
HAR format
HAR format
artifact = crawl.har()
artifact.save('crawl_results.har')
artifact = crawl.har()
artifact.save('crawl_results.har')
Iterate through records
Iterate through records
for record in artifact.iter_responses():
print(f"{record.url}: {record.status_code}")
undefinedfor record in artifact.iter_responses():
print(f"{record.url}: {record.status_code}")
undefinedBuild LLM.txt from crawled documentation
根据爬取的文档构建 LLM.txt
python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
config = CrawlerConfig(
url='https://web-scraping.dev/',
include_only_paths=['/docs', '/docs/*'], # only allow doc resources
page_limit=50,
max_depth=5,
follow_external_links=False,
content_formats=['markdown'],
)
crawl = Crawl(client, config).crawl().wait(poll_interval=5, verbose=True)python
import os
from scrapfly import ScrapflyClient, CrawlerConfig, Crawl
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
config = CrawlerConfig(
url='https://web-scraping.dev/',
include_only_paths=['/docs', '/docs/*'], # only allow doc resources
page_limit=50,
max_depth=5,
follow_external_links=False,
content_formats=['markdown'],
)
crawl = Crawl(client, config).crawl().wait(poll_interval=5, verbose=True)Get URLs from WARC
Get URLs from WARC
warc_artifact = crawl.warc()
all_urls = [
record.url for record in warc_artifact.iter_responses()
if record.status_code == 200
]
warc_artifact = crawl.warc()
all_urls = [
record.url for record in warc_artifact.iter_responses()
if record.status_code == 200
]
Retrieve markdown content in batches
Retrieve markdown content in batches
all_contents = {}
for i in range(0, len(all_urls), 100):
batch_urls = all_urls[i:i + 100]
batch_contents = crawl.read_batch(batch_urls, formats=['markdown'])
all_contents.update(batch_contents)
all_contents = {}
for i in range(0, len(all_urls), 100):
batch_urls = all_urls[i:i + 100]
batch_contents = crawl.read_batch(batch_urls, formats=['markdown'])
all_contents.update(batch_contents)
Build llms-full.txt
Build llms-full.txt
lines = ["# Documentation\n"]
for url, formats_dict in all_contents.items():
markdown = formats_dict.get('markdown', '').strip()
if markdown:
lines.append(f"---\n\n### {url}\n\n{markdown}\n")
with open("llms-full.txt", "w", encoding="utf-8") as f:
f.write("\n".join(lines))
undefinedlines = ["# Documentation\n"]
for url, formats_dict in all_contents.items():
markdown = formats_dict.get('markdown', '').strip()
if markdown:
lines.append(f"---\n\n### {url}\n\n{markdown}\n")
with open("llms-full.txt", "w", encoding="utf-8") as f:
f.write("\n".join(lines))
undefinedCrawl statistics
爬取统计信息
python
stats = crawl.stats()
print(f"URLs discovered: {stats['urls_discovered']}")
print(f"URLs crawled: {stats['urls_crawled']}")
print(f"Progress: {stats['progress_pct']:.1f}%")python
stats = crawl.stats()
print(f"URLs discovered: {stats['urls_discovered']}")
print(f"URLs crawled: {stats['urls_crawled']}")
print(f"Progress: {stats['progress_pct']:.1f}%")Key SDK Classes
核心 SDK 类
| Class | Description |
|---|---|
| Main client - handles auth and API calls |
| Crawl job configuration |
| High-level crawl lifecycle manager (start, wait, read) |
| Response from |
| Response from |
| Response from |
| Content for a single URL - has |
| 类 | 说明 |
|---|---|
| 主客户端——负责身份验证和 API 调用 |
| 爬取任务配置 |
| 高级爬取生命周期管理器(启动、等待、读取) |
| |
| |
| |
| 单个 URL 的内容——包含 |
Important Notes
重要说明
- Use to control credit spend on large sites
page_limit - Use and
include_only_pathsto focus crawls on relevant sectionsexclude_paths - and
exclude_pathsare mutually exclusiveinclude_only_paths - Crawl jobs are asynchronous - use for simple workflows or poll with
Crawl.crawl().wait()for more controlclient.get_crawl_status() - WARC/HAR artifacts are available for offline processing via /
crawl.warc()crawl.har() - supports up to 100 URLs per request for efficient bulk content retrieval
read_batch() - Webhook signatures should be verified for security in production
- should be used carefully to avoid crawling the entire web
follow_external_links
- 使用 控制大型站点的额度消耗
page_limit - 使用 和
include_only_paths将爬取范围限定在相关区域exclude_paths - 与
exclude_paths互斥include_only_paths - 爬取任务是异步的——简单工作流可使用 ,需要更多控制时可使用
Crawl.crawl().wait()进行轮询client.get_crawl_status() - WARC/HAR 存档可通过 /
crawl.warc()获取,以便离线处理crawl.har() - 每次请求最多支持 100 个 URL,可用于高效批量检索内容
read_batch() - 在生产环境中应验证 Webhook 签名以确保安全
- 谨慎使用 ,避免爬取整个互联网
follow_external_links