scrapfly-scraper
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseScrapfly Scraper
Scrapfly Scraper
Use the Scrapfly Scraper API to collect web page data with proxy rotation, anti-bot bypass, JavaScript rendering, and JavaScript scenarios for browser control.
使用 Scrapfly Scraper API 抓取网页数据,支持代理轮换、反爬绕过、JavaScript 渲染,以及通过 JavaScript scenario 控制浏览器。
When to use
使用场景
- Scraping web pages (HTML, JSON, text, markdown)
- Bypassing anti-bot protections (Cloudflare, DataDome, PerimeterX, Kasada, and more)
- Collecting data through rotating proxies with geo-targeting
- Rendering JavaScript-heavy pages with headless browsers
- Control the browser using JavaScript scenario for common actions (waiting for selectors, clicking, filling elements, etc.)
- Session-reuse for session-presited scraping
- Capturing browser XHR call data
- 抓取网页(HTML、JSON、文本、Markdown)
- 绕过反爬防护(Cloudflare、DataDome、PerimeterX、Kasada 等)
- 通过支持地理位置定向的轮换代理收集数据
- 使用 headless browser 渲染 JavaScript 密集型页面
- 使用 JavaScript scenario 控制浏览器执行常见操作(等待 selector、点击、填写元素等)
- 复用 session,进行会话保持型抓取
- 捕获浏览器 XHR 调用数据
Setup
安装设置
bash
pip install scrapfly-sdkThe API key must be provided via environment variable or passed directly to the client.
SCRAPFLY_API_KEYbash
pip install scrapfly-sdkAPI key 必须通过环境变量 提供,或直接传给 client。
SCRAPFLY_API_KEYAPI Reference
API 参考
Endpoint:
The HTTP method is forwarded to the upstream URL.
To retrieve data from a web page or an API, use the GET method, which is the default method.
If the API resource like an API requries other methods, like POST. You can use it via the parameter of the
https://api.scrapfly.io/scrapemethodScrapeConfigEndpoint:
HTTP 方法会被转发至上游 URL。
要从网页或 API 获取数据,请使用 GET 方法,这也是默认方法。
如果目标 API 资源需要 GET 以外的方法(例如 POST),可以通过 的 参数指定。
https://api.scrapfly.io/scrapeScrapeConfigmethodScrapflyClient
ScrapflyClient
python
from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])python
from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])ScrapeConfig Parameters
ScrapeConfig 参数
| Parameter | Type | Default | Description |
|---|---|---|---|
| str | required | Target URL to scrape |
| str | | HTTP method: GET, POST, PUT, PATCH, HEAD, OPTIONS |
| dict | None | Custom HTTP headers |
| dict | None | Custom cookies (merged into |
| str | None | Raw request body for POST/PUT/PATCH; used when you pre-encode payload |
| dict | None | Form/JSON data for POST/PUT/PATCH; encoded according to |
| int | None | Request timeout in ms (default ~150,000ms) |
| bool | True | Auto-retry on network failures |
| str | None | Proxy country (ISO 3166-1 alpha-2, e.g. |
| str | | Proxy pool: |
| str | None | Session ID to persist cookies/fingerprint across requests |
| bool | False | Keep the same proxy IP for a given |
| bool | False | Enable the Unblocker anti-bot bypass. |
| bool | False | Enable headless browser JavaScript rendering (+5 credits) |
| int | None | Wait time in ms after page load (requires |
| str | | Browser readiness stage: |
| str | None | CSS/XPath selector to wait for (requires |
| str | None | JavaScript code to execute in the browser (auto-encoded) |
| list | None | List of browser actions (click, fill, scroll, wait, etc.) |
| bool | None | Automatically scroll page during rendering for lazy-loaded content (requires |
| dict | None | Capture screenshots: |
| list[str] | None | Screenshot options: |
| str | | Output format: |
| list[str] | None | Format modifiers (markdown only): |
| dict | None | Raw extraction spec to apply on the response (encoded and sent as |
| str | None | Name of a saved server-side extraction template |
| dict | None | Inline JSON extraction template used once ( |
| str | None | Natural language instructions for Extraction API |
| str | None | LLM model name to use for Extraction API |
| bool | False | Enable response caching |
| int | None | Cache time-to-live in seconds |
| bool | False | Clear any existing cached response when |
| bool | False | Collect DNS records and timings in |
| bool | False | Collect SSL certificate details in |
| bool | False | Enable debug recording and extra metadata |
| bool | True | Raise exceptions for upstream 4xx/5xx HTTP responses |
| str | None | Custom ID for request tracking across systems |
| list[str] | None | Custom tags for request organization and analytics |
| list[str] | None | Accept-Language values, e.g. |
| str | None | Override browser OS fingerprint, e.g. |
| str | None | Named webhook for async scrape completion callbacks |
| int | None | Max credits to spend on unblocker retries and extra features |
| 参数 | 类型 | 默认值 | 描述 |
|---|---|---|---|
| str | 必填 | 要抓取的目标 URL |
| str | | HTTP 方法:GET、POST、PUT、PATCH、HEAD、OPTIONS |
| dict | None | 自定义 HTTP headers |
| dict | None | 自定义 cookies(合并到 |
| str | None | POST/PUT/PATCH 的原始请求体;用于已自行编码 payload 的情况 |
| dict | None | POST/PUT/PATCH 的表单/JSON 数据;未提供 |
| int | None | 请求超时时间,单位为 ms(默认约 150,000ms) |
| bool | True | 网络失败时自动重试 |
| str | None | 代理所在国家(ISO 3166-1 alpha-2,例如 |
| str | | 代理池: |
| str | None | 用于跨请求持久化 cookies/fingerprint 的 Session ID |
| bool | False | 为指定 |
| bool | False | 启用 Unblocker 反爬绕过。 |
| bool | False | 启用 headless browser JavaScript 渲染(+5 credits) |
| int | None | 页面加载后的等待时间,单位为 ms(需要 |
| str | | 浏览器就绪阶段: |
| str | None | 等待出现的 CSS/XPath selector(需要 |
| str | None | 要在浏览器中执行的 JavaScript 代码(自动编码) |
| list | None | 浏览器操作列表(点击、填写、滚动、等待等) |
| bool | None | 渲染期间自动滚动页面以加载延迟加载的内容(需要 |
| dict | None | 捕获截图: |
| list[str] | None | 截图选项: |
| str | | 输出格式: |
| list[str] | None | 格式修饰符(仅适用于 Markdown): |
| dict | None | 要应用于响应的原始 extraction spec(编码后作为 |
| str | None | 已保存的 server-side extraction template 名称 |
| dict | None | 仅使用一次的内联 JSON extraction template( |
| str | None | Extraction API 的自然语言指令 |
| str | None | Extraction API 使用的 LLM model 名称 |
| bool | False | 启用响应缓存 |
| int | None | 缓存生存时间,单位为秒 |
| bool | False | 当 |
| bool | False | 在 |
| bool | False | 在 |
| bool | False | 启用调试记录和额外的 metadata |
| bool | True | 上游返回 4xx/5xx HTTP 响应时抛出异常 |
| str | None | 用于跨系统跟踪请求的自定义 ID |
| list[str] | None | 用于请求组织和 analytics 的自定义标签 |
| list[str] | None | Accept-Language 值,例如 |
| str | None | 覆盖浏览器 OS fingerprint,例如 |
| str | None | 用于异步抓取完成回调的命名 webhook |
| int | None | unblocker 重试和额外功能可消耗的最大 credits 数量 |
ScrapeApiResponse
ScrapeApiResponse
python
response = client.scrape(ScrapeConfig(url="https://httpbin.dev"))
response.content # Page content (HTML/JSON/text)
response.scrape_result # Full result dict
response.status_code # Scrapfly API HTTP status code
response.scrape_result["status_code"] # Upsteam HTTP status code
response.headers # Response headers
response.context # Metadata about features usedpython
response = client.scrape(ScrapeConfig(url="https://httpbin.dev"))
response.content # Page content (HTML/JSON/text)
response.scrape_result # Full result dict
response.status_code # Scrapfly API HTTP status code
response.scrape_result["status_code"] # Upsteam HTTP status code
response.headers # Response headers
response.context # Metadata about features usedExamples
示例
Basic scrape
基础抓取
python
from scrapfly import ScrapflyClient, ScrapeConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
result = client.scrape(ScrapeConfig(
url="https://httpbin.dev",
))
print(result.content)python
from scrapfly import ScrapflyClient, ScrapeConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
result = client.scrape(ScrapeConfig(
url="https://httpbin.dev",
))
print(result.content)Scrape with anti-bot bypass and geo-targeting
使用反爬绕过和地理位置定向进行抓取
python
result = client.scrape(ScrapeConfig(
url="https://httpbin.dev",
unblocker=True, # enable the unblocker to bypass antibots
country="us", # match the proxy with the domain country
proxy_pool="public_residential_pool", # use the residential proxy pool to match real ISP IPs
))python
result = client.scrape(ScrapeConfig(
url="https://httpbin.dev",
unblocker=True, # enable the unblocker to bypass antibots
country="us", # match the proxy with the domain country
proxy_pool="public_residential_pool", # use the residential proxy pool to match real ISP IPs
))Scrape with JavaScript rendering
使用 JavaScript 渲染进行抓取
python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/products",
render_js=True, # enable JavaScript rendering using a cloud browser
rendering_wait=5000, # rendering wait for wait for
wait_for_selector="div.product", # wait a for a selector to be present on the page
))python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/products",
render_js=True, # enable JavaScript rendering using a cloud browser
rendering_wait=5000, # rendering wait for wait for
wait_for_selector="div.product", # wait a for a selector to be present on the page
))Scrape as markdown
以 Markdown 格式抓取
python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/products",
format="markdown", # other supported formats are: json, text, clean_html
))
print(result.content)python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/products",
format="markdown", # other supported formats are: json, text, clean_html
))
print(result.content)POST request with body
携带 body 的 POST 请求
python
result = client.scrape(ScrapeConfig(
url="https://httpbin.dev/anything",
method="POST",
headers={"Content-Type": "application/json"},
body='{"query": "search term"}',
))
print(result.content)python
result = client.scrape(ScrapeConfig(
url="https://httpbin.dev/anything",
method="POST",
headers={"Content-Type": "application/json"},
body='{"query": "search term"}',
))
print(result.content)JavaScript scenario (browser actions)
JavaScript scenario(浏览器操作)
python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/login",
render_js=True,
# browser control
js_scenario=[
{
"fill":{
"selector":"input[name='username']","clear":True,"value":"user123"
}
},{
"fill":{
"selector":"input[name='password']","clear":True,"value":"password"
}
},{
"click":{
"selector":"form > button[type='submit']"
}
},{
"wait_for_navigation":{
"timeout":5000
}
}
],
# request headers
headers={
"cookie":"cookiesAccepted=true"
}
))python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/login",
render_js=True,
# browser control
js_scenario=[
{
"fill":{
"selector":"input[name='username']","clear":True,"value":"user123"
}
},{
"fill":{
"selector":"input[name='password']","clear":True,"value":"password"
}
},{
"click":{
"selector":"form > button[type='submit']"
}
},{
"wait_for_navigation":{
"timeout":5000
}
}
],
# request headers
headers={
"cookie":"cookiesAccepted=true"
}
))access the element under login
access the element under login
print(result.selector.css("div#secret-message::text").get())
undefinedprint(result.selector.css("div#secret-message::text").get())
undefinedBuilt-in Parsel selector
内置 Parsel selector
python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/products",
render_js=True
))python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/products",
render_js=True
))access the built-in Parsel selector
access the built-in Parsel selector
selector = result.selector
for product in selector.css("div.product"):
print("product name:", product.xpath(".//a/text()").get()) # using xpath
print("product price:", product.css("div.price::text").get()) # using css
undefinedselector = result.selector
for product in selector.css("div.product"):
print("product name:", product.xpath(".//a/text()").get()) # using xpath
print("product price:", product.css("div.price::text").get()) # using css
undefinedConcurrent scraping
并发抓取
python
import asyncio
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
urls = [
"https://web-scraping.dev/products",
"https://web-scraping.dev/products?page=2",
"https://web-scraping.dev/products?page=3",
]
configs = [ScrapeConfig(url=url) for url in urls]
async def concurrent_scraping():
results = []
async for result in client.concurrent_scrape(configs):
results.append(result)
for result in results:
print(result.content[:100])
print("===========")
asyncio.run(concurrent_scraping())python
import asyncio
from scrapfly import ScrapflyClient, ScrapeConfig
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
urls = [
"https://web-scraping.dev/products",
"https://web-scraping.dev/products?page=2",
"https://web-scraping.dev/products?page=3",
]
configs = [ScrapeConfig(url=url) for url in urls]
async def concurrent_scraping():
results = []
async for result in client.concurrent_scrape(configs):
results.append(result)
for result in results:
print(result.content[:100])
print("===========")
asyncio.run(concurrent_scraping())Using sessions for stateful scraping
使用 session 进行有状态抓取
python
undefinedpython
undefinedFirst request: login
First request: login
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/login",
render_js=True,
# browser control
js_scenario=[
{
"fill":{
"selector":"input[name='username']","clear":True,"value":"user123"
}
},{
"fill":{
"selector":"input[name='password']","clear":True,"value":"password"
}
},{
"click":{
"selector":"form > button[type='submit']"
}
},{
"wait_for_navigation":{
"timeout":5000
}
}
],
# request headers
headers={
"cookie":"cookiesAccepted=true"
},
session="logged-in-session"
))
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/login",
render_js=True,
# browser control
js_scenario=[
{
"fill":{
"selector":"input[name='username']","clear":True,"value":"user123"
}
},{
"fill":{
"selector":"input[name='password']","clear":True,"value":"password"
}
},{
"click":{
"selector":"form > button[type='submit']"
}
},{
"wait_for_navigation":{
"timeout":5000
}
}
],
# request headers
headers={
"cookie":"cookiesAccepted=true"
},
session="logged-in-session"
))
Second request: access protected page with same session
Second request: access protected page with same session
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/login",
session="logged-in-session"
))
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/login",
session="logged-in-session"
))
access the built-in selector
access the built-in selector
print(result.selector.css("div#secret-message::text").get())
undefinedprint(result.selector.css("div#secret-message::text").get())
undefinedCaching responses
缓存响应
python
result = client.scrape(ScrapeConfig(
url="https://example.com",
cache=True,
cache_ttl=3600, # Cache for 1 hour
))python
result = client.scrape(ScrapeConfig(
url="https://example.com",
cache=True,
cache_ttl=3600, # Cache for 1 hour
))Error Handling
错误处理
python
from scrapfly import ScrapflyClient, ScrapeConfig
from scrapfly.errors import (
ScrapflyError,
UpstreamHttpClientError,
UpstreamHttpServerError,
ScrapflyProxyError,
ScrapflyThrottleError,
)
try:
result = client.scrape(ScrapeConfig(url="https://httpbin.dev", unblocker=True))
except ScrapflyThrottleError as e:
print(f"Rate limited, retry after {e.retry_delay}s")
except UpstreamHttpClientError as e:
print(f"Target returned 4xx: {e.message}")
except UpstreamHttpServerError as e:
print(f"Target returned 5xx: {e.message}")
except ScrapflyProxyError as e:
print(f"Proxy error: {e.message}")
except ScrapflyError as e:
print(f"Scrapfly error: {e.message}")python
from scrapfly import ScrapflyClient, ScrapeConfig
from scrapfly.errors import (
ScrapflyError,
UpstreamHttpClientError,
UpstreamHttpServerError,
ScrapflyProxyError,
ScrapflyThrottleError,
)
try:
result = client.scrape(ScrapeConfig(url="https://httpbin.dev", unblocker=True))
except ScrapflyThrottleError as e:
print(f"Rate limited, retry after {e.retry_delay}s")
except UpstreamHttpClientError as e:
print(f"Target returned 4xx: {e.message}\")
except UpstreamHttpServerError as e:
print(f"Target returned 5xx: {e.message}\")
except ScrapflyProxyError as e:
print(f"Proxy error: {e.message}\")
except ScrapflyError as e:
print(f"Scrapfly error: {e.message}\")Important Notes
重要说明
- is required for
render_js=True,rendering_wait,wait_for_selector,js, andjs_scenarioscreenshots - is recommended when using
proxy_pool="public_residential_pool"unblocker=True - is the current name for the anti-bot bypass;
unblockeris the deprecated alias and keeps working forever. Only the name changed; the SDK sendsaspon the wire either way. Do not pass bothasp - Use for clean content accessible for LLMs
format="markdown" - Use parameter to maintain state across multiple requests
session - The method handles rate limiting automatically
concurrent_scrape
- 使用 、
rendering_wait、wait_for_selector、js和js_scenario时必须设置screenshotsrender_js=True - 使用 时,建议设置
unblocker=Trueproxy_pool="public_residential_pool" - 是反爬绕过功能的当前名称;
unblocker是已弃用的别名,并且会永久保持可用。仅名称发生变化,无论使用哪个名称,SDK 在网络上发送的都是asp。不要同时传入两者asp - 使用 可获得适合 LLM 访问的干净内容
format="markdown" - 使用 参数在多个请求之间维护状态
session - 方法会自动处理速率限制",
concurrent_scrape