scrapfly-extraction
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseScrapfly Extraction
Scrapfly Extraction
Use the Scrapfly Extraction API to extract structured data from HTML, markdown, or text using LLM prompts, pre-trained AI models, or custom extraction templates.
使用 Scrapfly Extraction API,通过 LLM 提示词、预训练 AI 模型或自定义提取模板,从 HTML、Markdown 或文本中提取结构化数据。
When to use
何时使用
- Extracting structured data from web page content
- Using natural language prompts to pull specific information from documents
- Extracting product, article, review, or real estate data with auto AI models
- Parsing HTML/markdown into structured formats with custom templates
- Asking questions about document content and getting AI-powered answers
- 从网页内容中提取结构化数据
- 使用自然语言提示词从文档中提取特定信息
- 使用自动 AI 模型提取产品、文章、评论或房地产数据
- 使用自定义模板将 HTML/Markdown 解析为结构化格式
- 对文档内容提出问题并获取 AI 驱动的答案
Setup
设置
bash
pip install scrapfly-sdkThe API key must be provided via environment variable or passed directly to the client.
SCRAPFLY_API_KEYbash
pip install scrapfly-sdkAPI 密钥必须通过环境变量 提供,或直接传递给客户端。
SCRAPFLY_API_KEYAPI Reference
API 参考
Endpoint:
POST https://api.scrapfly.io/extraction端点:
POST https://api.scrapfly.io/extractionScrapflyClient
ScrapflyClient
python
from scrapfly import ScrapflyClient, ExtractionConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])python
from scrapfly import ScrapflyClient, ExtractionConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])ExtractionConfig Parameters
ExtractionConfig 参数
| Parameter | Type | Default | Description |
|---|---|---|---|
| str | required | Document content to extract from |
| str | required | Input format: |
| str | None | Base URL for resolving relative links in HTML |
| str | None | Document encoding (auto-detected if omitted) |
| str | None | Natural language instruction for LLM extraction |
| str | None | Pre-trained model: |
| str | None | Custom template name or inline template definition |
| int | None | Processing timeout in seconds (60-155) |
| str | None | Webhook name for async processing |
You must provide exactly one of: , , or .
extraction_promptextraction_modelextraction_template| 参数 | 类型 | 默认值 | 说明 |
|---|---|---|---|
| str | 必填 | 要提取数据的文档内容 |
| str | 必填 | 输入格式: |
| str | None | 用于解析 HTML 中相对链接的基础 URL |
| str | None | 文档编码(如果省略,将自动检测) |
| str | None | 用于 LLM 提取的自然语言指令 |
| str | None | 预训练模型: |
| str | None | 自定义模板名称或内联模板定义 |
| int | None | 处理超时时间,单位为秒(60-155) |
| str | None | 用于异步处理的 Webhook 名称 |
必须且只能提供以下选项之一: 、 或 。
extraction_promptextraction_modelextraction_templateThree Extraction Methods
三种提取方式
1. LLM Prompt Extraction
1. LLM 提示词提取
Use natural language to describe what data to extract. The AI interprets the content and returns structured results.
使用自然语言描述需要提取的数据。AI 会解释内容并返回结构化结果。
2. Auto AI Models
2. 自动 AI 模型
Pre-trained models for common data types. Returns standardized schemas with quality scores.
- - News/blog articles (title, author, date, content, etc.)
"article" - - Events (name, date, location, description, etc.)
"event" - - Recipes (ingredients, steps, servings, etc.)
"food_recipe" - - Single hotel/property (name, amenities, rating, etc.)
"hotel" - - Hotel search/list results
"hotel_listing" - - Job search/list results
"job_listing" - - Single job (title, company, salary, description, etc.)
"job_posting" - - Company/organization (name, contact, description, etc.)
"organization" - - E-commerce product (name, price, description, images, etc.)
"product" - - Product search/category listing
"product_listing" - - Single property (price, address, features, etc.)
"real_estate_property" - - Property search/list results
"real_estate_property_listing" - - Lists of reviews (reviewer, rating, text, date, etc.)
"review_list" - - SERP data (results, snippets, etc.)
"search_engine_results" - - Social post (author, content, engagement, etc.)
"social_media_post" - - Software/app (name, description, pricing, etc.)
"software" - - Stock/market data
"stock" - - Single vehicle listing
"vehicle_ad" - - Vehicle search/list results
"vehicle_ad_listing"
针对常见数据类型提供的预训练模型。返回标准化 Schema 和质量评分。
- - 新闻/博客文章(标题、作者、日期、内容等)
"article" - - 活动(名称、日期、地点、描述等)
"event" - - 食谱(食材、步骤、分量等)
"food_recipe" - - 单个酒店/物业(名称、设施、评分等)
"hotel" - - 酒店搜索/列表结果
"hotel_listing" - - 职位搜索/列表结果
"job_listing" - - 单个职位(标题、公司、薪资、描述等)
"job_posting" - - 公司/组织(名称、联系方式、描述等)
"organization" - - 电商产品(名称、价格、描述、图片等)
"product" - - 产品搜索/分类列表
"product_listing" - - 单个房产(价格、地址、特征等)
"real_estate_property" - - 房产搜索/列表结果
"real_estate_property_listing" - - 评论列表(评论者、评分、文本、日期等)
"review_list" - - SERP 数据(结果、摘要等)
"search_engine_results" - - 社交媒体帖子(作者、内容、互动数据等)
"social_media_post" - - 软件/应用(名称、描述、定价等)
"software" - - 股票/市场数据
"stock" - - 单个车辆信息
"vehicle_ad" - - 车辆搜索/列表结果
"vehicle_ad_listing"
3. Custom Templates
3. 自定义模板
Structured extraction rules for consistent parsing across similar pages. Can be defined inline (a JSON object), or stored on Scrapfly and referenced by slug.
通过结构化提取规则对相似页面进行一致解析。可以内联定义(JSON 对象),也可以存储在 Scrapfly 上并通过 slug 引用。
Saved Templates (reference by slug)
已保存的模板(通过 slug 引用)
If you have saved a template in the Scrapfly dashboard (Extraction API → Templates), pass its slug as a plain string. The platform resolves it to the currently-published version and applies it:
python
result = client.extract(ExtractionConfig(
body=html_content,
content_type="text/html",
url="https://example.com/product/1",
extraction_template="product-card",
))Saved templates support:
- A full version history with a separate publish pointer (publish a new version atomically; roll back at any time).
- Optional URL scope (/
match_domain) configured in the dashboard. Calls against non-matching URLs returnmatch_path.ERR::EXTRACTION::TEMPLATE_URL_MISMATCH - Cache-on-read with a 60-second TTL: a newly-published version becomes live within one minute, no client redeployment needed.
The slug shape is lowercase letters / digits / dashes, 3 to 128 characters.
如果已在 Scrapfly 控制面板(Extraction API → Templates)中保存模板,请将其 slug 作为纯字符串传入。平台会将它解析为当前已发布的版本并应用该版本:
python
result = client.extract(ExtractionConfig(
body=html_content,
content_type="text/html",
url="https://example.com/product/1",
extraction_template="product-card",
))已保存的模板支持:
- 完整的版本历史和独立的发布指针(原子化发布新版本;可随时回滚)。
- 可选的 URL 范围(/
match_domain),通过控制面板配置。对不匹配 URL 的调用会返回match_path。ERR::EXTRACTION::TEMPLATE_URL_MISMATCH - 读取时缓存,TTL 为 60 秒:新发布的版本最多在一分钟内生效,无需重新部署客户端。
slug 的格式为小写字母、数字和短横线,长度为 3 到 128 个字符。
Inline Template
内联模板
Pass a Python dict to extract once without saving. Useful for one-off scripts and for templates that vary per call.
python
extraction_template = {
"source": "html",
"selectors": [
{
"name": "title",
"query": "h3.product-title::text",
"type": "css",
"formatters": [
{
"name": "uppercase"
}
],
},
{
"name": "description",
"query": "p.product-description::text",
"type": "css"
},
{
"extractor": {
"name": "price"
},
"name": "price",
"query": ".product-price::text",
"type": "css"
},
{
"name": "variants",
"query": "div.variants",
"type": "css",
"nested": [
{
"name": "name",
"query": "//a[@data-variant-id]/@data-variant-id",
"type": "xpath",
"multiple": True,
},
{
"name": "link",
"query": "//a[@data-variant-id]/@href",
"type": "xpath",
"multiple": True,
},
]
},
{
"name": "reviews",
"query": "div.review>p::text",
"type": "css",
"multiple": True,
}
]
}传入 Python 字典,无需保存即可执行一次性提取。适用于临时脚本以及每次调用都不同的模板。
python
extraction_template = {
"source": "html",
"selectors": [
{
"name": "title",
"query": "h3.product-title::text",
"type": "css",
"formatters": [
{
"name": "uppercase"
}
],
},
{
"name": "description",
"query": "p.product-description::text",
"type": "css"
},
{
"extractor": {
"name": "price"
},
"name": "price",
"query": ".product-price::text",
"type": "css"
},
{
"name": "variants",
"query": "div.variants",
"type": "css",
"nested": [
{
"name": "name",
"query": "//a[@data-variant-id]/@data-variant-id",
"type": "xpath",
"multiple": True,
},
{
"name": "link",
"query": "//a[@data-variant-id]/@href",
"type": "xpath",
"multiple": True,
},
]
},
{
"name": "reviews",
"query": "div.review>p::text",
"type": "css",
"multiple": True,
}
]
}Examples
示例
LLM prompt extraction from HTML
从 HTML 中使用 LLM 提示词提取
python
from scrapfly import ScrapflyClient, ExtractionConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
html_content = "<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html>"
result = client.extract(ExtractionConfig(
body=html_content,
content_type="text/html",
extraction_prompt="Extract the product name, price, and description as JSON",
))python
from scrapfly import ScrapflyClient, ExtractionConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])
html_content = "<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html>"
result = client.extract(ExtractionConfig(
body=html_content,
content_type="text/html",
extraction_prompt="Extract the product name, price, and description as JSON",
))result
result
result.extraction_result['data']
result.extraction_result['data']
or
or
print(result.data)
print(result.data)
result content_type
result content_type
result.extraction_result['content_type']
result.extraction_result['content_type']
or
or
print(result.content_type)
undefinedprint(result.content_type)
undefinedLLM prompt extraction from markdown
从 Markdown 中使用 LLM 提示词提取
python
markdown_content = """python
markdown_content = """Best Restaurants in NYC
Best Restaurants in NYC
- Le Bernardin - French, $$$, 4.8 stars
- Peter Luger - Steakhouse, $$$, 4.5 stars
- Di Fara Pizza - Italian, $, 4.7 stars """
result = client.extract(ExtractionConfig(
body=markdown_content,
content_type="text/markdown",
extraction_prompt="Extract each restaurant as a JSON array with name, cuisine, price_range, and rating fields",
))
print(result.data)
undefined- Le Bernardin - French, $$$, 4.8 stars
- Peter Luger - Steakhouse, $$$, 4.5 stars
- Di Fara Pizza - Italian, $, 4.7 stars """
result = client.extract(ExtractionConfig(
body=markdown_content,
content_type="text/markdown",
extraction_prompt="Extract each restaurant as a JSON array with name, cuisine, price_range, and rating fields",
))
print(result.data)
undefinedAuto AI model: product extraction
自动 AI 模型:产品提取
python
from scrapfly import ScrapflyClient, ExtractionConfig, ScrapeConfigpython
from scrapfly import ScrapflyClient, ExtractionConfig, ScrapeConfigFirst scrape the page, then extract
First scrape the page, then extract
scrape_result = client.scrape(ScrapeConfig(url="https://web-scraping.dev/product/1"))
result = client.extract(ExtractionConfig(
body=scrape_result.content,
content_type="text/html",
url="https://web-scraping.dev/product/1",
extraction_model="product",
))
print(result.data)
scrape_result = client.scrape(ScrapeConfig(url="https://web-scraping.dev/product/1"))
result = client.extract(ExtractionConfig(
body=scrape_result.content,
content_type="text/html",
url="https://web-scraping.dev/product/1",
extraction_model="product",
))
print(result.data)
Returns: {"name": "...", "price": "...", "currency": "...", "description": "...", ...}
Returns: {"name": "...", "price": "...", "currency": "...", "description": "...", ...}
undefinedundefinedAsk a question about content
对内容提出问题
python
result = client.extract(ExtractionConfig(
body=html_content,
content_type="text/html",
extraction_prompt="What is the most expensive item on this page and how much does it cost?",
))
print(result.data)python
result = client.extract(ExtractionConfig(
body=html_content,
content_type="text/html",
extraction_prompt="What is the most expensive item on this page and how much does it cost?",
))
print(result.data)Scrape + Extract in one flow
在一个流程中完成抓取和提取
python
from scrapfly import ScrapflyClient, ScrapeConfig, ExtractionConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])python
from scrapfly import ScrapflyClient, ScrapeConfig, ExtractionConfig
import os
client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])Step 1: Scrape the page
Step 1: Scrape the page
scrape_result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/product/1",
format="markdown",
))
scrape_result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/product/1",
format="markdown",
))
Step 2: Extract structured data from the content
Step 2: Extract structured data from the content
extraction_result = client.extract(ExtractionConfig(
body=scrape_result.content,
content_type="text/markdown",
extraction_prompt="Extract all products as a JSON array with fields: name, price, availability",
))
print(extraction_result.data)
undefinedextraction_result = client.extract(ExtractionConfig(
body=scrape_result.content,
content_type="text/markdown",
extraction_prompt="Extract all products as a JSON array with fields: name, price, availability",
))
print(extraction_result.data)
undefinedInline extraction with the Scrape API
在 Scrape API 请求中使用内联提取
You can also extract data directly within a scrape request:
python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/product/1",
extraction_prompt="Extract the product name, price, and description as JSON",
))也可以在抓取请求中直接提取数据:
python
result = client.scrape(ScrapeConfig(
url="https://web-scraping.dev/product/1",
extraction_prompt="Extract the product name, price, and description as JSON",
))Extraction result is included in the scrape response
Extraction result is included in the scrape response
print(result.scrape_result["extracted_data"])
undefinedprint(result.scrape_result["extracted_data"])
undefinedCustom extraction template (inline)
自定义提取模板(内联)
python
undefinedpython
undefinedFirst, scrape the web page to retrieve its HTML
First, scrape the web page to retrieve its HTML
api_response = client.scrape(scrape_config=ScrapeConfig(
url='https://web-scraping.dev/product/1',
render_js=True
))
html = api_response.content
api_response = client.scrape(scrape_config=ScrapeConfig(
url='https://web-scraping.dev/product/1',
render_js=True
))
html = api_response.content
extraction template for HTML parsing instructions. It accepts the following:
extraction template for HTML parsing instructions. It accepts the following:
selectors: CSS, XPath, JMESPath, Regex, Nested (nesting multiple selector types)
selectors: CSS, XPath, JMESPath, Regex, Nested (nesting multiple selector types)
extractors: extracts commonly accessed data types: price, image, links, emails
extractors: extracts commonly accessed data types: price, image, links, emails
formatters: transforms the extracted data for common methods: lowercase, uppercase, datatime, etc.
formatters: transforms the extracted data for common methods: lowercase, uppercase, datatime, etc.
refer to the docs for more details: https://scrapfly.io/docs/extraction-api/rules-and-template#rules
refer to the docs for more details: https://scrapfly.io/docs/extraction-api/rules-and-template#rules
extraction_template = {
"source": "html",
"selectors": [
{ "name": "title", "query": "h3.product-title::text", "type": "css", "formatters": [ { "name": "uppercase" } ], }, { "name": "description", "query": "p.product-description::text", "type": "css" }, { "extractor": { "name": "price" }, "name": "price", "query": ".product-price::text", "type": "css" }, { "name": "variants", "query": "div.variants", "type": "css", "nested": [ { "name": "name", "query": "//a[@data-variant-id]/@data-variant-id", "type": "xpath", "multiple": True, }, { "name": "link", "query": "//a[@data-variant-id]/@href", "type": "xpath", "multiple": True, }, ] }, { "name": "reviews", "query": "div.review>p::text", "type": "css", "multiple": True,
} ] }
{ "name": "title", "query": "h3.product-title::text", "type": "css", "formatters": [ { "name": "uppercase" } ], }, { "name": "description", "query": "p.product-description::text", "type": "css" }, { "extractor": { "name": "price" }, "name": "price", "query": ".product-price::text", "type": "css" }, { "name": "variants", "query": "div.variants", "type": "css", "nested": [ { "name": "name", "query": "//a[@data-variant-id]/@data-variant-id", "type": "xpath", "multiple": True, }, { "name": "link", "query": "//a[@data-variant-id]/@href", "type": "xpath", "multiple": True, }, ] }, { "name": "reviews", "query": "div.review>p::text", "type": "css", "multiple": True,
} ] }
extraction_api_response = client.extract(
extraction_config=ExtractionConfig(
body=html, # pass the HTML content
content_type='text/html', # content data type
charset='utf-8', # passed content charset, use if you aren't sure
extraction_ephemeral_template=extraction_template # declared template defintion or template name saved on the dashboard
)
)
autoextraction_template = {
"source": "html",
"selectors": [
{ "name": "title", "query": "h3.product-title::text", "type": "css", "formatters": [ { "name": "uppercase" } ], }, { "name": "description", "query": "p.product-description::text", "type": "css" }, { "extractor": { "name": "price" }, "name": "price", "query": ".product-price::text", "type": "css" }, { "name": "variants", "query": "div.variants", "type": "css", "nested": [ { "name": "name", "query": "//a[@data-variant-id]/@data-variant-id", "type": "xpath", "multiple": True, }, { "name": "link", "query": "//a[@data-variant-id]/@href", "type": "xpath", "multiple": True, }, ] }, { "name": "reviews", "query": "div.review>p::text", "type": "css", "multiple": True,
} ] }
{ "name": "title", "query": "h3.product-title::text", "type": "css", "formatters": [ { "name": "uppercase" } ], }, { "name": "description", "query": "p.product-description::text", "type": "css" }, { "extractor": { "name": "price" }, "name": "price", "query": ".product-price::text", "type": "css" }, { "name": "variants", "query": "div.variants", "type": "css", "nested": [ { "name": "name", "query": "//a[@data-variant-id]/@data-variant-id", "type": "xpath", "multiple": True, }, { "name": "link", "query": "//a[@data-variant-id]/@href", "type": "xpath", "multiple": True, }, ] }, { "name": "reviews", "query": "div.review>p::text", "type": "css", "multiple": True,
} ] }
extraction_api_response = client.extract(
extraction_config=ExtractionConfig(
body=html, # pass the HTML content
content_type='text/html', # content data type
charset='utf-8', # passed content charset, use if you aren't sure
extraction_ephemeral_template=extraction_template # declared template defintion or template name saved on the dashboard
)
)
autoextracted data
extracted data
print(extraction_api_response.data)
print(extraction_api_response.data)
extracted data content_type
extracted data content_type
print(extraction_api_response.content_type)
undefinedprint(extraction_api_response.content_type)
undefinedError Handling
错误处理
python
from scrapfly.errors import ScrapflyError
try:
result = client.extract(ExtractionConfig(
body="<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html",
content_type="text/html",
extraction_prompt="Extract the product price",
))
print(result.data)
except ScrapflyError as e:
print(f"Extraction failed: {e.message}")python
from scrapfly.errors import ScrapflyError
try:
result = client.extract(ExtractionConfig(
body="<html><body><h1>iPhone 15</h1><span class='price'>$999</span><p>Latest Apple smartphone</p></body></html",
content_type="text/html",
extraction_prompt="Extract the product price",
))
print(result.data)
except ScrapflyError as e:
print(f"Extraction failed: {e.message}")Important Notes
重要说明
- Provide exactly one extraction method per request: ,
extraction_prompt, orextraction_modelextraction_template - For LLM prompts, be specific about the desired output format (e.g., "as JSON")
- The parameter helps resolve relative links in HTML but is not required
url - For large documents, consider using or
format="markdown"in the scrape step first to reduce token usageformat="text" - Auto AI models return quality metrics alongside extracted data
- Extraction can also be done inline during a scrape by adding or
extraction_prompttoextraction_modelScrapeConfig
- 每次请求只能提供一种提取方式:、
extraction_prompt或extraction_modelextraction_template - 对于 LLM 提示词,请明确指定所需的输出格式(例如“as JSON”)
- 参数有助于解析 HTML 中的相对链接,但不是必填项
url - 对于大型文档,可以先在抓取步骤中使用 或
format="markdown",以减少 Token 用量format="text" - 自动 AI 模型会在返回提取数据的同时给出质量指标
- 通过在 中添加
ScrapeConfig或extraction_prompt,也可以在抓取过程中内联执行提取extraction_model