Back to Details

scrape

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese
You are orchestrating the full web scraping workflow, from a user's prompt to a working Scrapy spider with web-poet page objects.
你需要编排完整的网页抓取工作流，从用户的需求到生成基于web-poet页面对象的可用Scrapy Spider。

Prerequisites

前置条件

Requires 
uv
. Install if missing.
需要安装 
uv
。如果未安装，请先安装。

Input

输入

The raw argument string is 
$ARGUMENTS
. Split it into 2 positional arguments:
  1. url: target website URL (first whitespace-separated token)
  2. what: what the user wants to extract (the rest after the URL, e.g. "product", "job listing", "recipe" — free text, may contain spaces)
原始参数字符串为 
$ARGUMENTS
。将其拆分为2个位置参数：
  1. url: 目标网站URL（第一个以空格分隔的 token）
  2. what: 用户想要提取的内容（URL之后的剩余部分，例如 "product"、"job listing"、"recipe" —— 自由文本，可包含空格）

Track progress

进度跟踪

Before Stage 1, create exactly these tasks with 
TaskCreate
, in order:
  1. "Decide which fields to extract" — 
    /scrape-define
  2. "Analyze the website" — 
    /scrape-spec
  3. "Create the Scrapy project" — 
    /scrape-ensure-project
  4. "Generate the extraction code" — 
    /scrape-codegen
    (one per data type)
  5. "Generate the spider" — 
    /scrape-create-spider
As you launch each skill, 
TaskUpdate
the task to 
in_progress
. Mark it 
completed
only after the skill (all instances of the skill in case of 
/scrape-codegen
) returns successfully. Do not batch updates — flip status at the boundary so the user sees live progress.
Do NOT create tasks inside the sub-skills; they share this session's task list and would duplicate entries.
在第1阶段开始前，使用 
TaskCreate
按顺序创建以下任务：
  1. "确定要提取的字段" —— 
    /scrape-define
  2. "分析网站" —— 
    /scrape-spec
  3. "创建Scrapy项目" —— 
    /scrape-ensure-project
  4. "生成提取代码" —— 
    /scrape-codegen
    （每种数据类型对应一次调用）
  5. "生成Spider" —— 
    /scrape-create-spider
启动每个技能时，使用 
TaskUpdate
将任务状态更新为 
in_progress
。只有在技能（如果是 
/scrape-codegen
则是所有实例）成功返回后，才将其标记为 
completed
。请勿批量更新状态——在阶段边界更新状态，以便用户可以看到实时进度。
请勿在子技能内部创建任务；它们共享当前会话的任务列表，会导致条目重复。

Stage 1: Define schema

第1阶段：定义 schema

Invoke 
/scrape-define
with the user's arguments. This downloads 1 detail page, discovers fields, and runs a fast terminal approval loop for the schema.
Output: 
.scrape/{site_name}/
with approved schema (including 
examples
from user-verified values). No stored pages or value files — those come from Stage 2.
调用 
/scrape-define
并传入用户的参数。此步骤会下载1个详情页，发现字段，并运行一个快速的终端确认循环来确定schema。
输出：
.scrape/{site_name}/
目录下包含已确认的schema（包括用户验证值对应的 
examples
）。此阶段不存储页面或值文件——这些将在第2阶段生成。

Stage 2: Explore and validate

第2阶段：探索与验证

Invoke 
/scrape-spec .scrape/{site_name}
. This downloads more detail and listing pages, compares HTML variants, extracts values, and optionally presents a browser review.
调用 
/scrape-spec .scrape/{site_name}
。此步骤会下载更多详情页和列表页，对比HTML变体，提取值，并可选择提供浏览器预览。

Stage 3: Generate working project

第3阶段：生成可用项目

Ensure the Scrapy project

确保Scrapy项目存在

Derive a project name from the domain (e.g., 
books_toscrape_com
).
/scrape-ensure-project ./{project_name} {project_name}
从域名派生项目名称（例如 
books_toscrape_com
）。
/scrape-ensure-project ./{project_name} {project_name}

Generate page objects (per data type)

生成页面对象（按数据类型）

After 
scrape-spec
returns, re-read 
{site_path}/spec.json
to get the final 
data_types
list — 
scrape-spec
may have replaced 
{data_type}
with 
{data_type}-list
when all requested fields were found on list pages.
Identify the primary data type: the first entry in 
data_types
that is not 
"navigation"
. If it ends with 
-list
(e.g., 
"products-list"
), the site will use list-page extraction. Otherwise it uses detail-page extraction (the default). Navigation is always present.
Call codegen once per data type in 
data_types
:
/scrape-codegen .scrape/{site_name}/{primary_type} ./{project_name}
/scrape-codegen .scrape/{site_name}/navigation ./{project_name}
Where 
{primary_type}
is whatever 
scrape-spec
wrote — 
products-list
for the list path, 
products
(or the site's data type name) for the detail path. Each call adds its own item class, page object, and fixtures to the project.
在 
scrape-spec
返回后，重新读取 
{site_path}/spec.json
以获取最终的 
data_types
列表——当在列表页中找到所有请求的字段时，
scrape-spec
可能会将 
{data_type}
替换为 
{data_type}-list
。
确定主数据类型：
data_types
中第一个不是 
"navigation"
的条目。如果它以 
-list
结尾（例如 
"products-list"
），则网站将使用列表页提取方式。否则使用详情页提取方式（默认）。
navigation
始终存在。
针对 
data_types
中的每个数据类型调用一次代码生成：
/scrape-codegen .scrape/{site_name}/{primary_type} ./{project_name}
/scrape-codegen .scrape/{site_name}/navigation ./{project_name}
其中 
{primary_type}
是 
scrape-spec
生成的名称——列表路径对应 
products-list
，详情路径对应 
products
（或网站的数据类型名称）。每次调用都会向项目中添加对应的条目类、页面对象和测试夹具。

Spider generation

Spider生成

After codegen, determine the PO class import paths from the generated files, then:
/scrape-create-spider ./{project_name} {item_po_import_path} {nav_po_import_path}
Also provide start URLs in the prompt (the site URL from spec.json).
When the primary data type ends with 
-list
, the generated item PO class name ends with 
ListPage
(e.g., 
ProductListPage
). 
scrape-create-spider
detects this suffix and automatically generates a list-extraction spider (items yielded directly from 
parse()
, no 
parse_item
callback).
When the primary data type is a plain detail type, the spider uses the standard detail-extraction pattern (navigation in 
parse()
, items in 
parse_item()
).
Tests with a limited crawl.
代码生成完成后，从生成的文件中确定PO类的导入路径，然后执行：
/scrape-create-spider ./{project_name} {item_po_import_path} {nav_po_import_path}
同时在请求中提供起始URL（来自spec.json的网站URL）。
当主数据类型以 
-list
结尾时，生成的条目PO类名称以 
ListPage
结尾（例如 
ProductListPage
）。
scrape-create-spider
会检测此后缀并自动生成列表提取型Spider（直接从 
parse()
生成条目，无需 
parse_item
回调）。
当主数据类型是普通详情类型时，Spider使用标准的详情提取模式（
parse()
处理导航，
parse_item()
处理条目）。
进行有限爬取测试。

Report

报告

Created scraping solution for {domain}:
  Project: ./{project_name}/
  Spider: uv run scrapy crawl {spider_name}
  Tests: uv run pytest fixtures/
Offer to help the user deploy to Scrapy Cloud if they wish. It's useful for scheduled or long-running crawls, to keep a job history with results and logs, for job monitoring (with an API that an LLM can use), and more. There is also a free tier.
If they deploy and run it there (via 
/scrape-scrapy-cloud
), the run will by default wait for the job to finish and validate its results (errors, field coverage, and whether items match the request) before reporting back — so expect it to take as long as the crawl does unless they opt out.
已为 {domain} 创建抓取解决方案：
  项目：./{project_name}/
  Spider运行命令：uv run scrapy crawl {spider_name}
  测试命令：uv run pytest fixtures/
如果用户愿意，可提供部署到 Scrapy Cloud 的帮助。Scrapy Cloud适用于定时或长期运行的爬取任务，可保留任务历史记录及结果和日志，支持任务监控（通过LLM可调用的API）等功能。同时它提供 免费层级。
如果用户通过 
/scrape-scrapy-cloud
部署并运行任务，默认情况下会等待任务完成并验证结果（错误、字段覆盖率、条目是否符合请求）后再返回报告——因此除非用户选择退出，否则等待时间将与爬取时间一致。