scrapfly-extraction

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Scrapfly Extraction

Scrapfly Extraction

Use the Scrapfly Extraction API to extract structured data from HTML, markdown, or text using LLM prompts, pre-trained AI models, or custom extraction templates.
使用 Scrapfly Extraction API,通过 LLM 提示词、预训练 AI 模型或自定义提取模板,从 HTML、Markdown 或文本中提取结构化数据。

When to use

何时使用

  • Extracting structured data from web page content
  • Using natural language prompts to pull specific information from documents
  • Extracting product, article, review, or real estate data with auto AI models
  • Parsing HTML/markdown into structured formats with custom templates
  • Asking questions about document content and getting AI-powered answers
  • 从网页内容中提取结构化数据
  • 使用自然语言提示词从文档中提取特定信息
  • 使用自动 AI 模型提取产品、文章、评论或房地产数据
  • 使用自定义模板将 HTML/Markdown 解析为结构化格式
  • 对文档内容提出问题并获取 AI 驱动的答案

Setup

设置

bash
pip install scrapfly-sdk
The API key must be provided via environment variable
SCRAPFLY_API_KEY
or passed directly to the client.
bash
pip install scrapfly-sdk
API 密钥必须通过环境变量
SCRAPFLY_API_KEY
提供,或直接传递给客户端。

API Reference

API 参考

Endpoint:
POST https://api.scrapfly.io/extraction
端点:
POST https://api.scrapfly.io/extraction

ScrapflyClient

ScrapflyClient

python
from scrapfly import ScrapflyClient, ExtractionConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
python
from scrapfly import ScrapflyClient, ExtractionConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

ExtractionConfig Parameters

ExtractionConfig 参数

ParameterTypeDefaultDescription
body
strrequiredDocument content to extract from
content_type
strrequiredInput format:
"text/html"
,
"text/markdown"
,
"text/plain"
,
"text/xml"
url
strNoneBase URL for resolving relative links in HTML
charset
strNoneDocument encoding (auto-detected if omitted)
extraction_prompt
strNoneNatural language instruction for LLM extraction
extraction_model
strNonePre-trained model:
"product"
,
"article"
,
"review_list"
,
"real_estate_listing"
extraction_template
strNoneCustom template name or inline template definition
timeout
intNoneProcessing timeout in seconds (60-155)
webhook_name
strNoneWebhook name for async processing
You must provide exactly one of:
extraction_prompt
,
extraction_model
, or
extraction_template
.
参数类型默认值说明
body
str必填要提取数据的文档内容
content_type
str必填输入格式:
"text/html"
、
"text/markdown"
、
"text/plain"
、
"text/xml"
url
strNone用于解析 HTML 中相对链接的基础 URL
charset
strNone文档编码(如果省略,将自动检测)
extraction_prompt
strNone用于 LLM 提取的自然语言指令
extraction_model
strNone预训练模型:
"product"
、
"article"
、
"review_list"
、
"real_estate_listing"
extraction_template
strNone自定义模板名称或内联模板定义
timeout
intNone处理超时时间,单位为秒(60-155)
webhook_name
strNone用于异步处理的 Webhook 名称
必须且只能提供以下选项之一:
extraction_prompt
、
extraction_model
或
extraction_template
。

Three Extraction Methods

三种提取方式

1. LLM Prompt Extraction

1. LLM 提示词提取

Use natural language to describe what data to extract. The AI interprets the content and returns structured results.
使用自然语言描述需要提取的数据。AI 会解释内容并返回结构化结果。

2. Auto AI Models

2. 自动 AI 模型

Pre-trained models for common data types. Returns standardized schemas with quality scores.
  • "article"
    - News/blog articles (title, author, date, content, etc.)
  • "event"
    - Events (name, date, location, description, etc.)
  • "food_recipe"
    - Recipes (ingredients, steps, servings, etc.)
  • "hotel"
    - Single hotel/property (name, amenities, rating, etc.)
  • "hotel_listing"
    - Hotel search/list results
  • "job_listing"
    - Job search/list results
  • "job_posting"
    - Single job (title, company, salary, description, etc.)
  • "organization"
    - Company/organization (name, contact, description, etc.)
  • "product"
    - E-commerce product (name, price, description, images, etc.)
  • "product_listing"
    - Product search/category listing
  • "real_estate_property"
    - Single property (price, address, features, etc.)
  • "real_estate_property_listing"
    - Property search/list results
  • "review_list"
    - Lists of reviews (reviewer, rating, text, date, etc.)
  • "search_engine_results"
    - SERP data (results, snippets, etc.)
  • "social_media_post"
    - Social post (author, content, engagement, etc.)
  • "software"
    - Software/app (name, description, pricing, etc.)
  • "stock"
    - Stock/market data
  • "vehicle_ad"
    - Single vehicle listing
  • "vehicle_ad_listing"
    - Vehicle search/list results
针对常见数据类型提供的预训练模型。返回标准化 Schema 和质量评分。
  • "article"
    - 新闻/博客文章(标题、作者、日期、内容等)
  • "event"
    - 活动(名称、日期、地点、描述等)
  • "food_recipe"
    - 食谱(食材、步骤、分量等)
  • "hotel"
    - 单个酒店/物业(名称、设施、评分等)
  • "hotel_listing"
    - 酒店搜索/列表结果
  • "job_listing"
    - 职位搜索/列表结果
  • "job_posting"
    - 单个职位(标题、公司、薪资、描述等)
  • "organization"
    - 公司/组织(名称、联系方式、描述等)
  • "product"
    - 电商产品(名称、价格、描述、图片等)
  • "product_listing"
    - 产品搜索/分类列表
  • "real_estate_property"
    - 单个房产(价格、地址、特征等)
  • "real_estate_property_listing"
    - 房产搜索/列表结果
  • "review_list"
    - 评论列表(评论者、评分、文本、日期等)
  • "search_engine_results"
    - SERP 数据(结果、摘要等)
  • "social_media_post"
    - 社交媒体帖子(作者、内容、互动数据等)
  • "software"
    - 软件/应用(名称、描述、定价等)
  • "stock"
    - 股票/市场数据
  • "vehicle_ad"
    - 单个车辆信息
  • "vehicle_ad_listing"
    - 车辆搜索/列表结果

3. Custom Templates

3. 自定义模板

Structured extraction rules for consistent parsing across similar pages. Can be defined inline (a JSON object), or stored on Scrapfly and referenced by slug.
通过结构化提取规则对相似页面进行一致解析。可以内联定义(JSON 对象),也可以存储在 Scrapfly 上并通过 slug 引用。
Saved Templates (reference by slug)
已保存的模板(通过 slug 引用)
If you have saved a template in the Scrapfly dashboard (Extraction API → Templates), pass its slug as a plain string. The platform resolves it to the currently-published version and applies it:
python
result = client.extract(ExtractionConfig(
    body=html_content,
    content_type="text/html",
    url="https://example.com/product/1",
    extraction_template="product-card",
))
Saved templates support:
  • A full version history with a separate publish pointer (publish a new version atomically; roll back at any time).
  • Optional URL scope (
    match_domain
    /
    match_path
    ) configured in the dashboard. Calls against non-matching URLs return
    ERR::EXTRACTION::TEMPLATE_URL_MISMATCH
    .
  • Cache-on-read with a 60-second TTL: a newly-published version becomes live within one minute, no client redeployment needed.
The slug shape is lowercase letters / digits / dashes, 3 to 128 characters.
如果已在 Scrapfly 控制面板(Extraction API → Templates)中保存模板,请将其 slug 作为纯字符串传入。平台会将它解析为当前已发布的版本并应用该版本:
python
result = client.extract(ExtractionConfig(
    body=html_content,
    content_type="text/html",
    url="https://example.com/product/1",
    extraction_template="product-card",
))
已保存的模板支持:
  • 完整的版本历史和独立的发布指针(原子化发布新版本;可随时回滚)。
  • 可选的 URL 范围(
    match_domain
    /
    match_path
    ),通过控制面板配置。对不匹配 URL 的调用会返回
    ERR::EXTRACTION::TEMPLATE_URL_MISMATCH
    。
  • 读取时缓存,TTL 为 60 秒:新发布的版本最多在一分钟内生效,无需重新部署客户端。
slug 的格式为小写字母、数字和短横线,长度为 3 到 128 个字符。
Inline Template
内联模板
Pass a Python dict to extract once without saving. Useful for one-off scripts and for templates that vary per call.
python
extraction_template = {
    "source": "html",
    "selectors": [    
        {
            "name": "title",
            "query": "h3.product-title::text",
            "type": "css",
            "formatters": [
                {
                    "name": "uppercase"
                }
            ],
        },
        {
            "name": "description",
            "query": "p.product-description::text",
            "type": "css"
        },
        {
            "extractor": {
                "name": "price"
            },
            "name": "price",
            "query": ".product-price::text",
            "type": "css"
        },
        {
            "name": "variants",
          	"query": "div.variants",
            "type": "css",
            "nested": [
                {
                    "name": "name",
                    "query": "//a[@data-variant-id]/@data-variant-id",
                    "type": "xpath",
                    "multiple": True,
                },
                {
                    "name": "link",
                    "query": "//a[@data-variant-id]/@href",
                    "type": "xpath",
                    "multiple": True,
                },
            ]
        },
        {
            "name": "reviews",
            "query": "div.review>p::text",
            "type": "css",
            "multiple": True,            
        }
    ]
}
传入 Python 字典,无需保存即可执行一次性提取。适用于临时脚本以及每次调用都不同的模板。
python
extraction_template = {
    "source": "html",
    "selectors": [    
        {
            "name": "title",
            "query": "h3.product-title::text",
            "type": "css",
            "formatters": [
                {
                    "name": "uppercase"
                }
            ],
        },
        {
            "name": "description",
            "query": "p.product-description::text",
            "type": "css"
        },
        {
            "extractor": {
                "name": "price"
            },
            "name": "price",
            "query": ".product-price::text",
            "type": "css"
        },
        {
            "name": "variants",
          	"query": "div.variants",
            "type": "css",
            "nested": [
                {
                    "name": "name",
                    "query": "//a[@data-variant-id]/@data-variant-id",
                    "type": "xpath",
                    "multiple": True,
                },
                {
                    "name": "link",
                    "query": "//a[@data-variant-id]/@href",
                    "type": "xpath",
                    "multiple": True,
                },
            ]
        },
        {
            "name": "reviews",
            "query": "div.review>p::text",
            "type": "css",
            "multiple": True,            
        }
    ]
}

Examples

示例

LLM prompt extraction from HTML

从 HTML 中使用 LLM 提示词提取

python
from scrapfly import ScrapflyClient, ExtractionConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

html_content = "<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html>"

result = client.extract(ExtractionConfig(
    body=html_content,
    content_type="text/html",
    extraction_prompt="Extract the product name, price, and description as JSON",
))
python
from scrapfly import ScrapflyClient, ExtractionConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

html_content = "<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html>"

result = client.extract(ExtractionConfig(
    body=html_content,
    content_type="text/html",
    extraction_prompt="Extract the product name, price, and description as JSON",
))

result

result

result.extraction_result['data']
result.extraction_result['data']

or

or

print(result.data)
print(result.data)

result content_type

result content_type

result.extraction_result['content_type']
result.extraction_result['content_type']

or

or

print(result.content_type)
undefined
print(result.content_type)
undefined

LLM prompt extraction from markdown

从 Markdown 中使用 LLM 提示词提取

python
markdown_content = """
python
markdown_content = """

Best Restaurants in NYC

Best Restaurants in NYC

  1. Le Bernardin - French, $$$, 4.8 stars
  2. Peter Luger - Steakhouse, $$$, 4.5 stars
  3. Di Fara Pizza - Italian, $, 4.7 stars """
result = client.extract(ExtractionConfig( body=markdown_content, content_type="text/markdown", extraction_prompt="Extract each restaurant as a JSON array with name, cuisine, price_range, and rating fields", )) print(result.data)
undefined
  1. Le Bernardin - French, $$$, 4.8 stars
  2. Peter Luger - Steakhouse, $$$, 4.5 stars
  3. Di Fara Pizza - Italian, $, 4.7 stars """
result = client.extract(ExtractionConfig( body=markdown_content, content_type="text/markdown", extraction_prompt="Extract each restaurant as a JSON array with name, cuisine, price_range, and rating fields", )) print(result.data)
undefined

Auto AI model: product extraction

自动 AI 模型:产品提取

python
from scrapfly import ScrapflyClient, ExtractionConfig, ScrapeConfig
python
from scrapfly import ScrapflyClient, ExtractionConfig, ScrapeConfig

First scrape the page, then extract

First scrape the page, then extract

scrape_result = client.scrape(ScrapeConfig(url="https://web-scraping.dev/product/1"))
result = client.extract(ExtractionConfig( body=scrape_result.content, content_type="text/html", url="https://web-scraping.dev/product/1", extraction_model="product", ))
print(result.data)
scrape_result = client.scrape(ScrapeConfig(url="https://web-scraping.dev/product/1"))
result = client.extract(ExtractionConfig( body=scrape_result.content, content_type="text/html", url="https://web-scraping.dev/product/1", extraction_model="product", ))
print(result.data)

Returns: {"name": "...", "price": "...", "currency": "...", "description": "...", ...}

Returns: {"name": "...", "price": "...", "currency": "...", "description": "...", ...}

undefined
undefined

Ask a question about content

对内容提出问题

python
result = client.extract(ExtractionConfig(
    body=html_content,
    content_type="text/html",
    extraction_prompt="What is the most expensive item on this page and how much does it cost?",
))
print(result.data)
python
result = client.extract(ExtractionConfig(
    body=html_content,
    content_type="text/html",
    extraction_prompt="What is the most expensive item on this page and how much does it cost?",
))
print(result.data)

Scrape + Extract in one flow

在一个流程中完成抓取和提取

python
from scrapfly import ScrapflyClient, ScrapeConfig, ExtractionConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
python
from scrapfly import ScrapflyClient, ScrapeConfig, ExtractionConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

Step 1: Scrape the page

Step 1: Scrape the page

scrape_result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/product/1", format="markdown", ))
scrape_result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/product/1", format="markdown", ))

Step 2: Extract structured data from the content

Step 2: Extract structured data from the content

extraction_result = client.extract(ExtractionConfig( body=scrape_result.content, content_type="text/markdown", extraction_prompt="Extract all products as a JSON array with fields: name, price, availability", ))
print(extraction_result.data)
undefined
extraction_result = client.extract(ExtractionConfig( body=scrape_result.content, content_type="text/markdown", extraction_prompt="Extract all products as a JSON array with fields: name, price, availability", ))
print(extraction_result.data)
undefined

Inline extraction with the Scrape API

在 Scrape API 请求中使用内联提取

You can also extract data directly within a scrape request:
python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/product/1",
    extraction_prompt="Extract the product name, price, and description as JSON",
))
也可以在抓取请求中直接提取数据:
python
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/product/1",
    extraction_prompt="Extract the product name, price, and description as JSON",
))

Extraction result is included in the scrape response

Extraction result is included in the scrape response

print(result.scrape_result["extracted_data"])
undefined
print(result.scrape_result["extracted_data"])
undefined

Custom extraction template (inline)

自定义提取模板(内联)

python
undefined
python
undefined

First, scrape the web page to retrieve its HTML

First, scrape the web page to retrieve its HTML

api_response = client.scrape(scrape_config=ScrapeConfig( url='https://web-scraping.dev/product/1', render_js=True ))
html = api_response.content
api_response = client.scrape(scrape_config=ScrapeConfig( url='https://web-scraping.dev/product/1', render_js=True ))
html = api_response.content

extraction template for HTML parsing instructions. It accepts the following:

extraction template for HTML parsing instructions. It accepts the following:

selectors: CSS, XPath, JMESPath, Regex, Nested (nesting multiple selector types)

selectors: CSS, XPath, JMESPath, Regex, Nested (nesting multiple selector types)

extractors: extracts commonly accessed data types: price, image, links, emails

extractors: extracts commonly accessed data types: price, image, links, emails

formatters: transforms the extracted data for common methods: lowercase, uppercase, datatime, etc.

formatters: transforms the extracted data for common methods: lowercase, uppercase, datatime, etc.

extraction_template = { "source": "html", "selectors": [
{ "name": "title", "query": "h3.product-title::text", "type": "css", "formatters": [ { "name": "uppercase" } ], }, { "name": "description", "query": "p.product-description::text", "type": "css" }, { "extractor": { "name": "price" }, "name": "price", "query": ".product-price::text", "type": "css" }, { "name": "variants", "query": "div.variants", "type": "css", "nested": [ { "name": "name", "query": "//a[@data-variant-id]/@data-variant-id", "type": "xpath", "multiple": True, }, { "name": "link", "query": "//a[@data-variant-id]/@href", "type": "xpath", "multiple": True, }, ] }, { "name": "reviews", "query": "div.review>p::text", "type": "css", "multiple": True,
} ] }
extraction_api_response = client.extract( extraction_config=ExtractionConfig( body=html, # pass the HTML content content_type='text/html', # content data type charset='utf-8', # passed content charset, use
auto
if you aren't sure extraction_ephemeral_template=extraction_template # declared template defintion or template name saved on the dashboard ) )
extraction_template = { "source": "html", "selectors": [
{ "name": "title", "query": "h3.product-title::text", "type": "css", "formatters": [ { "name": "uppercase" } ], }, { "name": "description", "query": "p.product-description::text", "type": "css" }, { "extractor": { "name": "price" }, "name": "price", "query": ".product-price::text", "type": "css" }, { "name": "variants", "query": "div.variants", "type": "css", "nested": [ { "name": "name", "query": "//a[@data-variant-id]/@data-variant-id", "type": "xpath", "multiple": True, }, { "name": "link", "query": "//a[@data-variant-id]/@href", "type": "xpath", "multiple": True, }, ] }, { "name": "reviews", "query": "div.review>p::text", "type": "css", "multiple": True,
} ] }
extraction_api_response = client.extract( extraction_config=ExtractionConfig( body=html, # pass the HTML content content_type='text/html', # content data type charset='utf-8', # passed content charset, use
auto
if you aren't sure extraction_ephemeral_template=extraction_template # declared template defintion or template name saved on the dashboard ) )

extracted data

extracted data

print(extraction_api_response.data)
print(extraction_api_response.data)

extracted data content_type

extracted data content_type

print(extraction_api_response.content_type)
undefined
print(extraction_api_response.content_type)
undefined

Error Handling

错误处理

python
from scrapfly.errors import ScrapflyError

try:
    result = client.extract(ExtractionConfig(
        body="<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html",
        content_type="text/html",
        extraction_prompt="Extract the product price",
    ))
    print(result.data)
except ScrapflyError as e:
    print(f"Extraction failed: {e.message}")
python
from scrapfly.errors import ScrapflyError

try:
    result = client.extract(ExtractionConfig(
        body="<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html",
        content_type="text/html",
        extraction_prompt="Extract the product price",
    ))
    print(result.data)
except ScrapflyError as e:
    print(f"Extraction failed: {e.message}")

Important Notes

重要说明

  • Provide exactly one extraction method per request:
    extraction_prompt
    ,
    extraction_model
    , or
    extraction_template
  • For LLM prompts, be specific about the desired output format (e.g., "as JSON")
  • The
    url
    parameter helps resolve relative links in HTML but is not required
  • For large documents, consider using
    format="markdown"
    or
    format="text"
    in the scrape step first to reduce token usage
  • Auto AI models return quality metrics alongside extracted data
  • Extraction can also be done inline during a scrape by adding
    extraction_prompt
    or
    extraction_model
    to
    ScrapeConfig
  • 每次请求只能提供一种提取方式:
    extraction_prompt
    、
    extraction_model
    或
    extraction_template
  • 对于 LLM 提示词,请明确指定所需的输出格式(例如“as JSON”)
  • url
    参数有助于解析 HTML 中的相对链接,但不是必填项
  • 对于大型文档,可以先在抓取步骤中使用
    format="markdown"
    或
    format="text"
    ,以减少 Token 用量
  • 自动 AI 模型会在返回提取数据的同时给出质量指标
  • 通过在
    ScrapeConfig
    中添加
    extraction_prompt
    或
    extraction_model
    ,也可以在抓取过程中内联执行提取