scrapfly-scraper

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Scrapfly Scraper

Scrapfly Scraper

Use the Scrapfly Scraper API to collect web page data with proxy rotation, anti-bot bypass, JavaScript rendering, and JavaScript scenarios for browser control.
使用 Scrapfly Scraper API 抓取网页数据,支持代理轮换、反爬绕过、JavaScript 渲染,以及通过 JavaScript scenario 控制浏览器。

When to use

使用场景

  • Scraping web pages (HTML, JSON, text, markdown)
  • Bypassing anti-bot protections (Cloudflare, DataDome, PerimeterX, Kasada, and more)
  • Collecting data through rotating proxies with geo-targeting
  • Rendering JavaScript-heavy pages with headless browsers
  • Control the browser using JavaScript scenario for common actions (waiting for selectors, clicking, filling elements, etc.)
  • Session-reuse for session-presited scraping
  • Capturing browser XHR call data
  • 抓取网页(HTML、JSON、文本、Markdown)
  • 绕过反爬防护(Cloudflare、DataDome、PerimeterX、Kasada 等)
  • 通过支持地理位置定向的轮换代理收集数据
  • 使用 headless browser 渲染 JavaScript 密集型页面
  • 使用 JavaScript scenario 控制浏览器执行常见操作(等待 selector、点击、填写元素等)
  • 复用 session,进行会话保持型抓取
  • 捕获浏览器 XHR 调用数据

Setup

安装设置

bash
pip install scrapfly-sdk
The API key must be provided via environment variable
SCRAPFLY_API_KEY
or passed directly to the client.
bash
pip install scrapfly-sdk
API key 必须通过环境变量
SCRAPFLY_API_KEY
提供,或直接传给 client。

API Reference

API 参考

Endpoint:
https://api.scrapfly.io/scrape
The HTTP method is forwarded to the upstream URL. To retrieve data from a web page or an API, use the GET method, which is the default method. If the API resource like an API requries other methods, like POST. You can use it via the
method
parameter of the
ScrapeConfig
Endpoint:
https://api.scrapfly.io/scrape
HTTP 方法会被转发至上游 URL。 要从网页或 API 获取数据,请使用 GET 方法,这也是默认方法。 如果目标 API 资源需要 GET 以外的方法(例如 POST),可以通过
ScrapeConfig
的
method
参数指定。

ScrapflyClient

ScrapflyClient

python
from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
python
from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

ScrapeConfig Parameters

ScrapeConfig 参数

ParameterTypeDefaultDescription
url
strrequiredTarget URL to scrape
method
str
"GET"
HTTP method: GET, POST, PUT, PATCH, HEAD, OPTIONS
headers
dictNoneCustom HTTP headers
cookies
dictNoneCustom cookies (merged into
headers["cookie"]
)
body
strNoneRaw request body for POST/PUT/PATCH; used when you pre-encode payload
data
dictNoneForm/JSON data for POST/PUT/PATCH; encoded according to
Content-Type
when
body
is not provided
timeout
intNoneRequest timeout in ms (default ~150,000ms)
retry
boolTrueAuto-retry on network failures
country
strNoneProxy country (ISO 3166-1 alpha-2, e.g.
"us"
,
"de"
)
proxy_pool
str
"public_datacenter_pool"
Proxy pool:
"public_datacenter_pool"
or
"public_residential_pool"
session
strNoneSession ID to persist cookies/fingerprint across requests
session_sticky_proxy
boolFalseKeep the same proxy IP for a given
session
unblocker
boolFalseEnable the Unblocker anti-bot bypass.
asp
is the permanently supported deprecated alias; when both are supplied,
asp
wins
render_js
boolFalseEnable headless browser JavaScript rendering (+5 credits)
rendering_wait
intNoneWait time in ms after page load (requires
render_js=True
)
rendering_stage
str
"complete"
Browser readiness stage:
"complete"
or
"domcontentloaded"
(requires
render_js=True
)
wait_for_selector
strNoneCSS/XPath selector to wait for (requires
render_js=True
)
js
strNoneJavaScript code to execute in the browser (auto-encoded)
js_scenario
listNoneList of browser actions (click, fill, scroll, wait, etc.)
auto_scroll
boolNoneAutomatically scroll page during rendering for lazy-loaded content (requires
render_js=True
)
screenshots
dictNoneCapture screenshots:
{"fullpage": "png"}
or
{"selector": ".element"}
(requires
render_js=True
)
screenshot_flags
list[str]NoneScreenshot options:
"load_images"
,
"dark_mode"
,
"block_banners"
,
"high_quality"
,
"print_media_format"
format
str
"raw"
Output format:
"raw"
,
"clean_html"
,
"json"
,
"markdown"
,
"text"
format_options
list[str]NoneFormat modifiers (markdown only):
"no_images"
,
"no_links"
,
"only_content"
extract
dictNoneRaw extraction spec to apply on the response (encoded and sent as
extract
)
extraction_template
strNoneName of a saved server-side extraction template
extraction_ephemeral_template
dictNoneInline JSON extraction template used once (
ephemeral:
)
extraction_prompt
strNoneNatural language instructions for Extraction API
extraction_model
strNoneLLM model name to use for Extraction API
cache
boolFalseEnable response caching
cache_ttl
intNoneCache time-to-live in seconds
cache_clear
boolFalseClear any existing cached response when
cache=True
dns
boolFalseCollect DNS records and timings in
result.dns
(slower)
ssl
boolFalseCollect SSL certificate details in
result.ssl
(slower)
debug
boolFalseEnable debug recording and extra metadata
raise_on_upstream_error
boolTrueRaise exceptions for upstream 4xx/5xx HTTP responses
correlation_id
strNoneCustom ID for request tracking across systems
tags
list[str]NoneCustom tags for request organization and analytics
lang
list[str]NoneAccept-Language values, e.g.
["en-US", "en"]
os
strNoneOverride browser OS fingerprint, e.g.
"windows"
,
"macos"
webhook
strNoneNamed webhook for async scrape completion callbacks
cost_budget
intNoneMax credits to spend on unblocker retries and extra features
参数类型默认值描述
url
str必填要抓取的目标 URL
method
str
"GET"
HTTP 方法:GET、POST、PUT、PATCH、HEAD、OPTIONS
headers
dictNone自定义 HTTP headers
cookies
dictNone自定义 cookies(合并到
headers["cookie"]
中)
body
strNonePOST/PUT/PATCH 的原始请求体;用于已自行编码 payload 的情况
data
dictNonePOST/PUT/PATCH 的表单/JSON 数据;未提供
body
时根据
Content-Type
进行编码
timeout
intNone请求超时时间,单位为 ms(默认约 150,000ms)
retry
boolTrue网络失败时自动重试
country
strNone代理所在国家(ISO 3166-1 alpha-2,例如
"us"
、
"de"
)
proxy_pool
str
"public_datacenter_pool"
代理池:
"public_datacenter_pool"
或
"public_residential_pool"
session
strNone用于跨请求持久化 cookies/fingerprint 的 Session ID
session_sticky_proxy
boolFalse为指定
session
保持相同的代理 IP
unblocker
boolFalse启用 Unblocker 反爬绕过。
asp
是永久受支持但已弃用的别名;同时提供两者时,以
asp
为准
render_js
boolFalse启用 headless browser JavaScript 渲染(+5 credits)
rendering_wait
intNone页面加载后的等待时间,单位为 ms(需要
render_js=True
)
rendering_stage
str
"complete"
浏览器就绪阶段:
"complete"
或
"domcontentloaded"
(需要
render_js=True
)
wait_for_selector
strNone等待出现的 CSS/XPath selector(需要
render_js=True
)
js
strNone要在浏览器中执行的 JavaScript 代码(自动编码)
js_scenario
listNone浏览器操作列表(点击、填写、滚动、等待等)
auto_scroll
boolNone渲染期间自动滚动页面以加载延迟加载的内容(需要
render_js=True
)
screenshots
dictNone捕获截图:
{"fullpage": "png"}
或
{"selector": ".element"}
(需要
render_js=True
)
screenshot_flags
list[str]None截图选项:
"load_images"
、
"dark_mode"
、
"block_banners"
、
"high_quality"
、
"print_media_format"
format
str
"raw"
输出格式:
"raw"
、
"clean_html"
、
"json"
、
"markdown"
、
"text"
format_options
list[str]None格式修饰符(仅适用于 Markdown):
"no_images"
、
"no_links"
、
"only_content"
extract
dictNone要应用于响应的原始 extraction spec(编码后作为
extract
发送)
extraction_template
strNone已保存的 server-side extraction template 名称
extraction_ephemeral_template
dictNone仅使用一次的内联 JSON extraction template(
ephemeral:
)
extraction_prompt
strNoneExtraction API 的自然语言指令
extraction_model
strNoneExtraction API 使用的 LLM model 名称
cache
boolFalse启用响应缓存
cache_ttl
intNone缓存生存时间,单位为秒
cache_clear
boolFalse当
cache=True
时清除任何已有的缓存响应
dns
boolFalse在
result.dns
中收集 DNS 记录和时序信息(速度较慢)
ssl
boolFalse在
result.ssl
中收集 SSL 证书详情(速度较慢)
debug
boolFalse启用调试记录和额外的 metadata
raise_on_upstream_error
boolTrue上游返回 4xx/5xx HTTP 响应时抛出异常
correlation_id
strNone用于跨系统跟踪请求的自定义 ID
tags
list[str]None用于请求组织和 analytics 的自定义标签
lang
list[str]NoneAccept-Language 值,例如
["en-US", "en"]
os
strNone覆盖浏览器 OS fingerprint,例如
"windows"
、
"macos"
webhook
strNone用于异步抓取完成回调的命名 webhook
cost_budget
intNoneunblocker 重试和额外功能可消耗的最大 credits 数量

ScrapeApiResponse

ScrapeApiResponse

python
response = client.scrape(ScrapeConfig(url="https://httpbin.dev"))

response.content                      # Page content (HTML/JSON/text)
response.scrape_result                # Full result dict
response.status_code                  # Scrapfly API HTTP status code
response.scrape_result["status_code"] # Upsteam HTTP status code
response.headers                      # Response headers
response.context                      # Metadata about features used
python
response = client.scrape(ScrapeConfig(url="https://httpbin.dev"))

response.content                      # Page content (HTML/JSON/text)
response.scrape_result                # Full result dict
response.status_code                  # Scrapfly API HTTP status code
response.scrape_result["status_code"] # Upsteam HTTP status code
response.headers                      # Response headers
response.context                      # Metadata about features used

Examples

示例

Basic scrape

基础抓取

python
from scrapfly import ScrapflyClient, ScrapeConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
))
print(result.content)
python
from scrapfly import ScrapflyClient, ScrapeConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
))
print(result.content)

Scrape with anti-bot bypass and geo-targeting

使用反爬绕过和地理位置定向进行抓取

python
result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
    unblocker=True, # enable the unblocker to bypass antibots
    country="us", # match the proxy with the domain country
    proxy_pool="public_residential_pool", # use the residential proxy pool to match real ISP IPs
))
python
result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
    unblocker=True, # enable the unblocker to bypass antibots
    country="us", # match the proxy with the domain country
    proxy_pool="public_residential_pool", # use the residential proxy pool to match real ISP IPs
))

Scrape with JavaScript rendering

使用 JavaScript 渲染进行抓取

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True, # enable JavaScript rendering using a cloud browser
    rendering_wait=5000, # rendering wait for wait for
    wait_for_selector="div.product", # wait a for a selector to be present on the page
))
python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True, # enable JavaScript rendering using a cloud browser
    rendering_wait=5000, # rendering wait for wait for
    wait_for_selector="div.product", # wait a for a selector to be present on the page
))

Scrape as markdown

以 Markdown 格式抓取

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    format="markdown", # other supported formats are: json, text, clean_html 
))
print(result.content)
python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    format="markdown", # other supported formats are: json, text, clean_html 
))
print(result.content)

POST request with body

携带 body 的 POST 请求

python
result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev/anything",
    method="POST",
    headers={"Content-Type": "application/json"},
    body='{"query": "search term"}',
))
print(result.content)
python
result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev/anything",
    method="POST",
    headers={"Content-Type": "application/json"},
    body='{"query": "search term"}',
))
print(result.content)

JavaScript scenario (browser actions)

JavaScript scenario(浏览器操作)

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    render_js=True,
    # browser control
    js_scenario=[
        {
            "fill":{
                "selector":"input[name='username']","clear":True,"value":"user123"
                }
            },{
            "fill":{
                "selector":"input[name='password']","clear":True,"value":"password"
                }
            },{
            "click":{
                "selector":"form > button[type='submit']"
                }
            },{
            "wait_for_navigation":{
                "timeout":5000
                }
            }
    ],
    # request headers
    headers={
        "cookie":"cookiesAccepted=true"
    }
))
python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    render_js=True,
    # browser control
    js_scenario=[
        {
            "fill":{
                "selector":"input[name='username']","clear":True,"value":"user123"
                }
            },{
            "fill":{
                "selector":"input[name='password']","clear":True,"value":"password"
                }
            },{
            "click":{
                "selector":"form > button[type='submit']"
                }
            },{
            "wait_for_navigation":{
                "timeout":5000
                }
            }
    ],
    # request headers
    headers={
        "cookie":"cookiesAccepted=true"
    }
))

access the element under login

access the element under login

print(result.selector.css("div#secret-message::text").get())
undefined
print(result.selector.css("div#secret-message::text").get())
undefined

Built-in Parsel selector

内置 Parsel selector

python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True
))
python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True
))

access the built-in Parsel selector

access the built-in Parsel selector

selector = result.selector for product in selector.css("div.product"): print("product name:", product.xpath(".//a/text()").get()) # using xpath print("product price:", product.css("div.price::text").get()) # using css
undefined
selector = result.selector for product in selector.css("div.product"): print("product name:", product.xpath(".//a/text()").get()) # using xpath print("product price:", product.css("div.price::text").get()) # using css
undefined

Concurrent scraping

并发抓取

python
import asyncio
from scrapfly import ScrapflyClient, ScrapeConfig

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

urls = [
    "https://web-scraping.dev/products",
    "https://web-scraping.dev/products?page=2",
    "https://web-scraping.dev/products?page=3",
]

configs = [ScrapeConfig(url=url) for url in urls]

async def concurrent_scraping():
    results = []
    async for result in client.concurrent_scrape(configs):
        results.append(result)

    for result in results:
        print(result.content[:100])
        print("===========")

asyncio.run(concurrent_scraping())
python
import asyncio
from scrapfly import ScrapflyClient, ScrapeConfig

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

urls = [
    "https://web-scraping.dev/products",
    "https://web-scraping.dev/products?page=2",
    "https://web-scraping.dev/products?page=3",
]

configs = [ScrapeConfig(url=url) for url in urls]

async def concurrent_scraping():
    results = []
    async for result in client.concurrent_scrape(configs):
        results.append(result)

    for result in results:
        print(result.content[:100])
        print("===========")

asyncio.run(concurrent_scraping())

Using sessions for stateful scraping

使用 session 进行有状态抓取

python
undefined
python
undefined

First request: login

First request: login

result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/login", render_js=True, # browser control js_scenario=[ { "fill":{ "selector":"input[name='username']","clear":True,"value":"user123" } },{ "fill":{ "selector":"input[name='password']","clear":True,"value":"password" } },{ "click":{ "selector":"form > button[type='submit']" } },{ "wait_for_navigation":{ "timeout":5000 } } ], # request headers headers={ "cookie":"cookiesAccepted=true" }, session="logged-in-session" ))
result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/login", render_js=True, # browser control js_scenario=[ { "fill":{ "selector":"input[name='username']","clear":True,"value":"user123" } },{ "fill":{ "selector":"input[name='password']","clear":True,"value":"password" } },{ "click":{ "selector":"form > button[type='submit']" } },{ "wait_for_navigation":{ "timeout":5000 } } ], # request headers headers={ "cookie":"cookiesAccepted=true" }, session="logged-in-session" ))

Second request: access protected page with same session

Second request: access protected page with same session

result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/login", session="logged-in-session" ))
result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/login", session="logged-in-session" ))

access the built-in selector

access the built-in selector

print(result.selector.css("div#secret-message::text").get())
undefined
print(result.selector.css("div#secret-message::text").get())
undefined

Caching responses

缓存响应

python
result = client.scrape(ScrapeConfig(
    url="https://example.com",
    cache=True,
    cache_ttl=3600,  # Cache for 1 hour
))
python
result = client.scrape(ScrapeConfig(
    url="https://example.com",
    cache=True,
    cache_ttl=3600,  # Cache for 1 hour
))

Error Handling

错误处理

python
from scrapfly import ScrapflyClient, ScrapeConfig
from scrapfly.errors import (
    ScrapflyError,
    UpstreamHttpClientError,
    UpstreamHttpServerError,
    ScrapflyProxyError,
    ScrapflyThrottleError,
)

try:
    result = client.scrape(ScrapeConfig(url="https://httpbin.dev", unblocker=True))
except ScrapflyThrottleError as e:
    print(f"Rate limited, retry after {e.retry_delay}s")
except UpstreamHttpClientError as e:
    print(f"Target returned 4xx: {e.message}")
except UpstreamHttpServerError as e:
    print(f"Target returned 5xx: {e.message}")
except ScrapflyProxyError as e:
    print(f"Proxy error: {e.message}")
except ScrapflyError as e:
    print(f"Scrapfly error: {e.message}")
python
from scrapfly import ScrapflyClient, ScrapeConfig
from scrapfly.errors import (
    ScrapflyError,
    UpstreamHttpClientError,
    UpstreamHttpServerError,
    ScrapflyProxyError,
    ScrapflyThrottleError,
)

try:
    result = client.scrape(ScrapeConfig(url="https://httpbin.dev", unblocker=True))
except ScrapflyThrottleError as e:
    print(f"Rate limited, retry after {e.retry_delay}s")
except UpstreamHttpClientError as e:
    print(f"Target returned 4xx: {e.message}\")
except UpstreamHttpServerError as e:
    print(f"Target returned 5xx: {e.message}\")
except ScrapflyProxyError as e:
    print(f"Proxy error: {e.message}\")
except ScrapflyError as e:
    print(f"Scrapfly error: {e.message}\")

Important Notes

重要说明

  • render_js=True
    is required for
    rendering_wait
    ,
    wait_for_selector
    ,
    js
    ,
    js_scenario
    , and
    screenshots
  • proxy_pool="public_residential_pool"
    is recommended when using
    unblocker=True
  • unblocker
    is the current name for the anti-bot bypass;
    asp
    is the deprecated alias and keeps working forever. Only the name changed; the SDK sends
    asp
    on the wire either way. Do not pass both
  • Use
    format="markdown"
    for clean content accessible for LLMs
  • Use
    session
    parameter to maintain state across multiple requests
  • The
    concurrent_scrape
    method handles rate limiting automatically
  • 使用
    rendering_wait
    、
    wait_for_selector
    、
    js
    、
    js_scenario
    和
    screenshots
    时必须设置
    render_js=True
  • 使用
    unblocker=True
    时,建议设置
    proxy_pool="public_residential_pool"
  • unblocker
    是反爬绕过功能的当前名称;
    asp
    是已弃用的别名,并且会永久保持可用。仅名称发生变化,无论使用哪个名称,SDK 在网络上发送的都是
    asp
    。不要同时传入两者
  • 使用
    format="markdown"
    可获得适合 LLM 访问的干净内容
  • 使用
    session
    参数在多个请求之间维护状态
  • concurrent_scrape
    方法会自动处理速率限制",