webextrator

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

WebExtrator Web Render & Extract

WebExtrator 网页渲染与提取

Render and extract web content through AceDataCloud's WebExtrator API — real headless Chromium plus a three-tier extraction pipeline (schema.org JSON-LD mapper → LLM typed extractor → Readability/markdown fallback).

Setup: See authentication for token setup.

通过AceDataCloud的WebExtrator API渲染并提取网页内容——基于真实无头Chromium，搭配三层提取流水线（schema.org JSON-LD映射器 → LLM类型提取器 → Readability/markdown降级方案）。

配置： 请查看身份认证了解令牌配置方法。

Quick Start

快速开始

bash

curl -X POST https://api.acedata.cloud/webextrator/extract \
  -H "Authorization: Bearer $ACEDATACLOUD_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "expected_type": "general"}'

Returns synchronously in seconds — no task polling needed.

bash

curl -X POST https://api.acedata.cloud/webextrator/extract \
  -H "Authorization: Bearer $ACEDATACLOUD_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "expected_type": "general"}'

同步返回结果，耗时仅数秒——无需轮询任务状态。

Endpoints

接口列表

Path	Purpose
`POST /webextrator/render`	Headless Chromium render → raw HTML + clean text + title
`POST /webextrator/extract`	Render + structured extraction (schema.org + LLM types) + markdown
`POST /webextrator/tasks`	Look up historical render/extract task envelopes (7-day retention, free)

路径	用途
`POST /webextrator/render`	无头Chromium渲染 → 原始HTML + 纯净文本 + 标题
`POST /webextrator/extract`	渲染 + 结构化提取（schema.org + LLM类型） + Markdown
`POST /webextrator/tasks`	查询历史渲染/提取任务记录（免费保留7天）

Workflows

工作流程

1. Extract typed content

1. 提取类型化内容

json

POST /webextrator/extract
{
  "url": "https://example.com",
  "expected_type": "general"
}

Real response (trimmed):

json

{
  "success": true,
  "task_id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5",
  "trace_id": "f2a7c0b0-c17c-4bc9-b6e7-9c59746dd366",
  "elapsed": 0.003,
  "data": {
    "kind": "extract",
    "url": "https://example.com",
    "finalUrl": "https://example.com/",
    "contentType": "general",
    "title": "Example Domain",
    "description": "This domain is for use in documentation examples without needing permission. Avoid use in operations.",
    "language": "en",
    "images": [],
    "links": [],
    "markdown": "...",
    "text": "...",
    "structured": { "schemaOrg": {}, "openGraph": {}, "jsonLd": [] }
  }
}

json

POST /webextrator/extract
{
  "url": "https://example.com",
  "expected_type": "general"
}

真实响应（已精简）：

json

{
  "success": true,
  "task_id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5",
  "trace_id": "f2a7c0b0-c17c-4bc9-b6e7-9c59746dd366",
  "elapsed": 0.003,
  "data": {
    "kind": "extract",
    "url": "https://example.com",
    "finalUrl": "https://example.com/",
    "contentType": "general",
    "title": "Example Domain",
    "description": "This domain is for use in documentation examples without needing permission. Avoid use in operations.",
    "language": "en",
    "images": [],
    "links": [],
    "markdown": "...",
    "text": "...",
    "structured": { "schemaOrg": {}, "openGraph": {}, "jsonLd": [] }
  }
}

2. Render raw HTML

2. 渲染原始HTML

json

POST /webextrator/render
{
  "url": "https://example.com",
  "wait_until": "networkidle",
  "block_resources": ["image", "media", "font"]
}

Returns

data.html

data.text

data.title

data.status

data.finalUrl

json

POST /webextrator/render
{
  "url": "https://example.com",
  "wait_until": "networkidle",
  "block_resources": ["image", "media", "font"]
}

data.html

、

data.text

、

data.title

、

data.status

、

data.finalUrl

。

3. Look up a task

3. 查询任务记录

json

POST /webextrator/tasks
{
  "action": "retrieve",
  "id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5"
}

json

POST /webextrator/tasks
{
  "action": "retrieve",
  "id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5"
}

Parameters

参数说明

Render & Extract (shared)

渲染与提取（通用参数）

Parameter	Required	Description
`url`	Yes	Page URL to render ( `http(s)://` )
`wait_until`	No	`load` / `domcontentloaded` / `networkidle` / `commit` (default `networkidle` )
`timeout`	No	Navigation timeout in seconds (default 30)
`delay`	No	Extra wait in seconds after `wait_until` (for SPAs)
`wait_for_selector`	No	CSS selector to wait for before ready
`block_resources`	No	Drop `image` / `font` / `media` / `stylesheet` / `xhr` / `fetch`
`headers`	No	Extra request headers for the target site
`cookies`	No	Cookies to install before navigation
`mode`	No	`sync` (default) or `async` (returns job id)
`callback_url`	No	Posted the final envelope when `mode=async`
`bypass_cache`	No	Skip the Redis result cache for this request
`cache_ttl_seconds`	No	Override cache TTL; `0` disables caching

参数	是否必填	描述
`url`	是	要渲染的页面URL（需以 `http(s)://` 开头）
`wait_until`	否	`load` / `domcontentloaded` / `networkidle` / `commit` （默认值为 `networkidle` ）
`timeout`	否	导航超时时间（单位：秒，默认30）
`delay`	否	`wait_until` 触发后额外等待的时间（单位：秒，适用于单页应用）
`wait_for_selector`	否	等待指定CSS选择器加载完成后再返回结果
`block_resources`	否	屏蔽指定资源类型： `image` / `font` / `media` / `stylesheet` / `xhr` / `fetch`
`headers`	否	向目标站点发送请求时添加的额外请求头
`cookies`	否	导航前植入的Cookie
`mode`	否	`sync` （默认，同步）或 `async` （异步，返回任务ID）
`callback_url`	否	当 `mode=async` 时，任务完成后会向该URL推送最终结果
`bypass_cache`	否	跳过Redis结果缓存，强制重新处理请求
`cache_ttl_seconds`	否	覆盖缓存过期时间；设为 `0` 则禁用缓存

Extract-only

仅提取专用参数

Parameter	Required	Description
`expected_type`	No	`product` / `article` / `general` — skips the heuristic
`enable_llm`	No	Allow LLM extractor when schema.org found nothing (default false)

参数	是否必填	描述
`expected_type`	否	`product` / `article` / `general` — 跳过启发式检测
`enable_llm`	否	当未检测到schema.org数据时，允许启用LLM提取器（默认值为false）

Tasks

任务查询参数

Parameter	Required	Description
`action`	Yes	`retrieve` (single) or `retrieve_batch` (many)
`id` / `trace_id`	one of	For `retrieve`
`ids` / `trace_ids`	one of	For `retrieve_batch`

参数	是否必填	描述
`action`	是	`retrieve` （查询单个任务）或 `retrieve_batch` （批量查询任务）
`id` / `trace_id`	二选一	用于 `retrieve` 模式
`ids` / `trace_ids`	二选一	用于 `retrieve_batch` 模式

Gotchas

注意事项

Parameters use snake_case (
```
wait_until
```
,
```
block_resources
```
), not camelCase
Cache hits are still billed; identical URLs return in ~0.003s
```
expected_type
```
only allows
```
product
```
/
```
article
```
/
```
general
```
— typed kinds (recipe/video/job) are detected automatically from schema.org
```
enable_llm
```
has no effect when the page ships schema.org JSON-LD — the deterministic mapper wins for free
Tasks API is free and retains records for 7 days only
```
cache_ttl_seconds: 0
```
means "do not cache" — use
```
bypass_cache
```
to skip read

MCP:
pip install mcp-webextrator
| Hosted:
https://webextrator.mcp.acedata.cloud/mcp
| See all MCP servers

参数采用蛇形命名法（
```
wait_until
```
、
```
block_resources
```
），而非驼峰命名法
命中缓存的请求仍会计费；相同URL的请求返回耗时约0.003秒
```
expected_type
```
仅支持
```
product
```
/
```
article
```
/
```
general
```
——其他类型（食谱/视频/职位）会通过schema.org自动检测
当页面包含schema.org JSON-LD数据时，
```
enable_llm
```
参数无效——此时会优先使用确定性映射器，无需额外成本
任务查询API免费使用，记录仅保留7天
```
cache_ttl_seconds: 0
```
表示“不缓存”——若需跳过读取缓存，请使用
```
bypass_cache
```
参数

MCP:
pip install mcp-webextrator
| 托管地址:
https://webextrator.mcp.acedata.cloud/mcp
| 查看所有MCP服务器