scrape-scrapy-cloud
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseYou are a general-purpose Scrapy Cloud assistant. You can deploy projects, schedule
spiders, manage jobs (list, stop), and direct users to the web UI to inspect items
and logs — all using and the Scrapy Cloud HTTP API.
shubRead and from .
python-environments.mddocs-access.md${CLAUDE_SKILL_DIR}/../scrape/references你是一名通用Scrapy Cloud助手。你可以使用和Scrapy Cloud HTTP API来部署项目、调度爬虫、管理任务(列出、停止),并引导用户通过Web UI查看爬取条目和日志。
shub请阅读路径下的和文档。
${CLAUDE_SKILL_DIR}/../scrape/referencespython-environments.mddocs-access.mdRules
规则
- DO NOT prematurely ask the user for project ID, endpoint, credentials, or deployment
target. All of these are discovered by following the Process steps below
(checking ,
scrapinghub.yml, and environment variables).~/.scrapinghub.yml - Execute the Process steps sequentially — only ask the user when a step explicitly says to.
- If the user's open file or current directory contains a , use it to determine which project to deploy.
scrapinghub.yml
- 请勿提前向用户索要项目ID、端点、凭证或部署目标。所有这些信息都可以通过遵循以下流程步骤获取(检查、
scrapinghub.yml和环境变量)。~/.scrapinghub.yml - 按顺序执行流程步骤——仅当步骤明确说明时才向用户询问。
- 如果用户打开的文件或当前目录包含,请使用该文件确定要部署的项目。
scrapinghub.yml
Input
输入
The raw argument string is — use it as-is, treat empty as "no argument given":
$ARGUMENTS- project_dir: path to the Scrapy project directory (defaults to current directory if the argument string is empty)
原始参数字符串为——请直接使用它,空字符串表示“未提供参数”:
$ARGUMENTS- project_dir: Scrapy项目目录的路径(如果参数字符串为空,则默认为当前目录)
Process
流程
IMPORTANT: It's critical that API keys are not displayed or exposed to the agent or user during a session.
Do not use any tool that might print the key or its value (e.g. ) as an auth probe.
cat ~/.scrapinghub.yml重要提示:在会话过程中,绝对不能显示或暴露API密钥给助手或用户。请勿使用任何可能打印密钥或其值的工具(例如)作为身份验证探测手段。
cat ~/.scrapinghub.yml1. Locate the project root
1. 定位项目根目录
Navigate to (or the current directory if none was given). Confirm exists:
project_dirscrapy.cfgbash
ls scrapy.cfgIf is not in the current directory, walk up the directory tree to find it. Once found, into that directory — all subsequent commands must run from there.
scrapy.cfgcd导航到(如果未提供则为当前目录)。确认文件存在:
project_dirscrapy.cfgbash
ls scrapy.cfg如果当前目录中没有,请向上遍历目录树查找该文件。找到后,进入该目录——后续所有命令都必须在此目录下运行。
scrapy.cfg2. Check for existing scrapinghub.yml
2. 检查是否存在scrapinghub.yml
bash
ls scrapinghub.yml 2>/dev/null || echo "(not found)"If the file exists skip step 3 and proceed directly to step 4.
bash
ls scrapinghub.yml 2>/dev/null || echo "(未找到)"如果该文件存在,则跳过步骤3,直接进入步骤4。
3. Create scrapinghub.yml
3. 创建scrapinghub.yml
Get the project ID
Check for a saved project ID from a previous :
scrape-zyte-loginbash
cat .scrape/.zyte/project-id 2>/dev/null || echo "(not set)"If present, use it and skip asking.
Otherwise, open the Zyte projects page:
bash
xdg-open "https://app.zyte.com/o/projects/" 2>/dev/null || open "https://app.zyte.com/o/projects/"Ask the user:
Please open https://app.zyte.com/o/projects/ and provide the numeric project ID
for this spider (e.g. 12345). If you don't have a project yet, create one first.Wait for the user to supply a numeric project ID.
Prepare requirements
If already exists, use it as-is.
requirements.txtOtherwise, look for a dependency specification (any standard file declaring Python dependencies). If one exists, generate a freeze (all packages pinned with ) from it by automatic means. If none exists, identify the third-party packages from the project source, write them to a non-freeze dependency file, then generate the freeze from that file by automatic means.
requirements.txt==requirements.txtIn all cases, points at .
scrapinghub.ymlrequirements.txtIf did not exist before, tell the user the exact command used to generate it, and what to run when dependencies change or they wish to upgrade them.
requirements.txtSpecify a Scrapy stack
Use the Docker Hub API or scrape the tags page to find the most recent stack tag for the repository that matches the Scrapy version in the requirements file:
scrapinghub/scrapinghub-stack-scrapy- API endpoint:
https://hub.docker.com/v2/repositories/scrapinghub/scrapinghub-stack-scrapy/tags - Web page:
https://hub.docker.com/r/scrapinghub/scrapinghub-stack-scrapy/tags
Tags follow the format (e.g. ). Select the tag with the highest version number and, among equal versions, the latest frozen date. Use that tag as the value in , prefixed with (e.g. ).
If the requirements file doesn't specify a Scrapy version, or no stack tag matches the version it specifies, use the latest tag overall.
{VERSION}-{YYYYMMDD}2.14-20260326stackscrapinghub.ymlscrapy:scrapy:2.14-20260326Write scrapinghub.yml
Create the file. If is set, add an key.
SCRAPY_CLOUD_ENDPOINTendpoint:yaml
project: 12345
stack: scrapy:2.14-20260326 # replace with the latest tag fetched above
requirements:
file: requirements.txt
endpoint: https://app-staging.zyte.com/api/ # include when SCRAPY_CLOUD_ENDPOINT is set获取项目ID
检查之前操作保存的项目ID:
scrape-zyte-loginbash
cat .scrape/.zyte/project-id 2>/dev/null || echo "(未设置)"如果存在,则使用该ID,无需询问用户。
否则,打开Zyte项目页面:
bash
xdg-open "https://app.zyte.com/o/projects/" 2>/dev/null || open "https://app.zyte.com/o/projects/"向用户询问:
请打开https://app.zyte.com/o/projects/并提供此爬虫的数字项目ID(例如12345)。如果您还没有项目,请先创建一个。等待用户提供数字项目ID。
准备依赖文件
如果已存在,请直接使用。
requirements.txt否则,查找依赖声明文件(任何标准的Python依赖声明文件)。如果存在此类文件,通过自动方式从中生成一个固定版本的(所有包使用固定版本)。如果不存在,则从项目源码中识别第三方包,将其写入一个非固定版本的依赖文件,然后通过自动方式从该文件生成固定版本的。
requirements.txt==requirements.txt在所有情况下,都指向。
scrapinghub.ymlrequirements.txt如果之前不存在,请告知用户生成该文件所使用的具体命令,以及当依赖项变更或需要升级时应执行的操作。
requirements.txt指定Scrapy栈
使用Docker Hub API或爬取标签页面,找到仓库中与依赖文件中的Scrapy版本匹配的最新栈标签:
scrapinghub/scrapinghub-stack-scrapy- API端点:
https://hub.docker.com/v2/repositories/scrapinghub/scrapinghub-stack-scrapy/tags - 网页:
https://hub.docker.com/r/scrapinghub/scrapinghub-stack-scrapy/tags
标签格式为(例如)。选择版本号最高的标签;如果版本号相同,则选择日期最新的标签。将该标签作为中的值,前缀为(例如)。
如果依赖文件未指定Scrapy版本,或者没有匹配指定版本的栈标签,则使用最新的整体标签。
{VERSION}-{YYYYMMDD}2.14-20260326scrapinghub.ymlstackscrapy:scrapy:2.14-20260326写入scrapinghub.yml
创建该文件。如果设置了,则添加键。
SCRAPY_CLOUD_ENDPOINTendpoint:yaml
project: 12345
stack: scrapy:2.14-20260326 # 替换为上面获取的最新标签
requirements:
file: requirements.txt
endpoint: https://app-staging.zyte.com/api/ # 当设置SCRAPY_CLOUD_ENDPOINT时添加4. Deploy
4. 部署
Run the deploy from the project root with , which fetches on
demand — no separate install step, and it works whether or not is
already on :
uvxshubshubPATHbash
uvx shub deployStream and display the full output. A successful deploy looks like:
Packing version 3af023e-master
Deploying to Scrapy Cloud project "12345"
{"status": "ok", "project": 12345, "version": "3af023e-master", "spiders": 2}
Run your spiders at: https://app.zyte.com/p/12345/On failure, diagnose the error and fix before retrying:
- → not authenticated; invoke
Error: Not logged in. Please run 'shub login' first../scrape-zyte-login - → no project ID configured; ask the user for a project ID, update
Error: Invalid value for target: Please specify target or configure a default target in scrapinghub.yml.and retry.scrapinghub.yml - /
Authentication error→ API key is wrong or missing; invoke403./scrape-zyte-login - → wrong project ID; correct
Project N does not existand retry.scrapinghub.yml - → the
Could not find requirements filepath inrequirements.fileis wrong; fix the path and retry.scrapinghub.yml - or build errors → a dependency is missing or incompatible with the selected stack; update the requirements file and retry.
No module named scrapy - Any other error → show the full error message to the user and ask how to proceed.
从项目根目录使用运行部署命令,会按需获取——无需单独安装步骤,无论是否已在中都能正常工作:
uvxuvxshubshubPATHbash
uvx shub deploy流式显示完整输出。成功部署的输出示例如下:
Packing version 3af023e-master
Deploying to Scrapy Cloud project "12345"
{"status": "ok", "project": 12345, "version": "3af023e-master", "spiders": 2}
Run your spiders at: https://app.zyte.com/p/12345/部署失败时,诊断错误并修复后重试:
- → 未认证;调用
Error: Not logged in. Please run 'shub login' first.。/scrape-zyte-login - → 未配置项目ID;向用户索要项目ID,更新
Error: Invalid value for target: Please specify target or configure a default target in scrapinghub.yml.后重试。scrapinghub.yml - /
Authentication error→ API密钥错误或缺失;调用403。/scrape-zyte-login - → 项目ID错误;修正
Project N does not exist后重试。scrapinghub.yml - →
Could not find requirements file中的scrapinghub.yml路径错误;修正路径后重试。requirements.file - 或构建错误 → 依赖项缺失或与所选栈不兼容;更新依赖文件后重试。
No module named scrapy - 其他错误 → 向用户显示完整错误信息并询问如何处理。
5. Report success
5. 报告部署成功
Tell the user the deploy succeeded, and include the version, spider count,
and project link (https://app.zyte.com/p/<project>/).
If the stack's Scrapy version is older than the version in , strongly recommend running a test job, and note that if issues arise, stack incompatibility is a possible — not assumed — root cause.
requirements.txt告知用户部署成功,并包含版本号、爬虫数量和项目链接(https://app.zyte.com/p/<project>/)。
如果栈的Scrapy版本比中的版本旧,强烈建议运行测试任务,并注意如果出现问题,栈不兼容可能是一个潜在的根本原因(而非必然原因)。
requirements.txtScheduling a job
调度任务
Use to start a job that runs a spider on Scrapy Cloud
without redeploying.
uvx shub scheduleTag jobs with .
skill使用启动一个任务,无需重新部署即可在Scrapy Cloud上运行爬虫。
uvx shub schedule为任务添加标签。
skillBasic schedule
基础调度
bash
uvx shub schedule SPIDER --tag skillOn success, shub prints the job ID and convenience links:
Spider myspider scheduled, job ID: 12345/2/15
Watch the log on the command line:
shub log -f 2/15
or watch it running in Zyte's web interface:
https://app.zyte.com/p/12345/job/2/15bash
uvx shub schedule SPIDER --tag skill成功后,shub会打印任务ID和便捷链接:
Spider myspider scheduled, job ID: 12345/2/15
Watch the log on the command line:
shub log -f 2/15
or watch it running in Zyte's web interface:
https://app.zyte.com/p/12345/job/2/15With spider arguments (-a
)
-a带爬虫参数(-a
)
-aRepeat for each argument:
-a KEY=VALUEbash
uvx shub schedule myspider -a ARG1=VALUE1 -a ARG2=VALUE2 --tag skill每个参数重复使用:
-a KEY=VALUEbash
uvx shub schedule myspider -a ARG1=VALUE1 -a ARG2=VALUE2 --tag skillWith job-specific Scrapy settings (-s
)
-s带任务特定的Scrapy设置(-s
)
-sbash
uvx shub schedule myspider -s LOG_LEVEL=DEBUG -s CLOSESPIDER_PAGECOUNT=10 --tag skillbash
uvx shub schedule myspider -s LOG_LEVEL=DEBUG -s CLOSESPIDER_PAGECOUNT=10 --tag skillWith a specific project
指定特定项目
bash
uvx shub schedule 33333/myspider --tag skillbash
uvx shub schedule 33333/myspider --tag skillWith units and priority
带资源单元和优先级
| Flag | Description |
|---|---|
| Number of Scrapy Cloud units to use (1–6) |
| Job priority: 0 (lowest) to 4 (highest) |
bash
uvx shub schedule myspider -p 3 -u 3 --tag skill| 标志 | 描述 |
|---|---|
| 使用的Scrapy Cloud资源单元数量(1–6) |
| 任务优先级:0(最低)到4(最高) |
bash
uvx shub schedule myspider -p 3 -u 3 --tag skillCombined example
组合示例
bash
uvx shub schedule myspider \
-a start_url=https://example.com \
-s LOG_LEVEL=DEBUG \
-p 2 \
-u 1 \
--tag skillbash
uvx shub schedule myspider \
-a start_url=https://example.com \
-s LOG_LEVEL=DEBUG \
-p 2 \
-u 1 \
--tag skillWaiting for a job to finish
等待任务完成
After scheduling a job, wait for it to reach a terminal state so you can
validate its results (see the next section). Use the bundled helper — it polls
the Jobs API with backoff so you don't have to poll by hand:
bash
uv run ${CLAUDE_SKILL_DIR}/scripts/wait_for_job.py PROJECT/SPIDER/JOBOptional flags: (initial gap, default 10, backs off
to 60) and (give up after this long, default 1200).
--poll-interval SECONDS--max-wait SECONDSIt blocks until the job reaches a terminal state (or is hit), then
prints exactly one machine-readable line to stdout:
--max-wait- — the job reached a terminal state (exit 0). Parse the JSON for
JOB_FINISHED {...job json...},state,close_reason,errors_count, etc., then proceed to validation.items_scraped - —
JOB_TIMEOUT {...last known job json...}elapsed while the job was still running (exit 1). Tell the user the job is still running and offer to keep waiting (re-run with a larger--max-wait) or check back later.--max-wait
Progress lines go to stderr and can be ignored.
调度任务后,等待任务进入终端状态,以便验证结果(见下一节)。使用内置的辅助工具——它会通过退避策略轮询Jobs API,无需手动轮询:
bash
uv run ${CLAUDE_SKILL_DIR}/scripts/wait_for_job.py PROJECT/SPIDER/JOB可选标志:(初始轮询间隔,默认10秒,退避到60秒)和(超时时间,默认1200秒)。
--poll-interval SECONDS--max-wait SECONDS该工具会阻塞直到任务进入终端状态(或达到时间),然后向标准输出打印一行机器可读的内容:
--max-wait- — 任务进入终端状态(退出码0)。解析JSON中的
JOB_FINISHED {...job json...}、state、close_reason、errors_count等字段,然后进行验证。items_scraped - — 任务仍在运行时已超过
JOB_TIMEOUT {...last known job json...}时间(退出码1)。告知用户任务仍在运行,并提供继续等待(使用更大的--max-wait值重新运行)或稍后再检查的选项。--max-wait
进度信息会输出到标准错误,可以忽略。
Validating job results
验证任务结果
Do this by default after running a job — deploy/schedule is not "done"
until the results look right. Skip it only if the user opted out (phrases
like "just schedule it", "don't wait for it", "no need to check the results"):
in that case, report the job link and stop.
Work through these checks against the finished job:
默认情况下,运行任务后必须执行此步骤——部署/调度任务直到结果看起来正确才算“完成”。仅当用户明确选择跳过(例如“只调度任务”、“不用等它完成”、“不需要检查结果”)时才跳过此步骤:此时只需报告任务链接并停止操作。
针对已完成的任务执行以下检查:
1. Errors and close reason
1. 错误和结束原因
From the summary, should be .
Anything else — , , or an unexpected
reason — is a flag worth investigating. Treat as a signal, not
proof: a count does not guarantee a clean run if the log level suppressed
errors (see step 3).
JOB_FINISHEDclose_reason"finished"failedcancelledclosespider_*errors_count0从摘要中,应为。任何其他值——、或意外的原因——都需要进一步调查。将视为一个信号而非绝对证据:如果日志级别抑制了错误,即使为0也不能保证运行完全正常(见步骤3)。
JOB_FINISHEDclose_reason"finished"failedcancelledclosespider_*errors_counterrors_count2. Inspect logs locally
2. 本地检查日志
Download the full log once and grep/ it locally — far faster than
paginating the HTTP API line by line. Logs come back as JSON Lines by default:
jqbash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET \
"${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}logs/PROJECT/SPIDER/JOB" > job.log.jl下载完整日志并在本地使用grep/分析——这比逐行分页调用HTTP API快得多。默认情况下,日志以JSON Lines格式返回:
jqbash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET \
"${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}logs/PROJECT/SPIDER/JOB" > job.log.jlERROR (40) and CRITICAL (50) entries:
ERROR (40)和CRITICAL (50)条目:
jq -c 'select(.level >= 40)' job.log.jl
Each entry is `{"time": <unix-ms>, "level": <int>, "message": <str>}`; see the
log-level table under "Viewing items and logs". For pagination/format options,
see the logs API docs linked there.
If the project uses Zyte API (via scrapy-zyte-api), the Scrapy stats dumped at
close include `scrapy-zyte-api/*` counters — success/error ratios, `429`s,
`error_types/*`, bans. These often explain a bad close reason or low coverage,
and are easy to misread. For what each counter means, see
https://scrapy-zyte-api.readthedocs.io/en/latest/reference/stats.mdjq -c 'select(.level >= 40)' job.log.jl
每个条目格式为`{"time": <unix-ms>, "level": <int>, "message": <str>}`;日志级别对照表见“查看条目和日志”部分。如需分页/格式选项,请参考该部分链接的日志API文档。
如果项目使用Zyte API(通过scrapy-zyte-api),任务结束时输出的Scrapy统计信息包含`scrapy-zyte-api/*`计数器——成功/错误比率、`429`错误、`error_types/*`、封禁情况等。这些信息通常可以解释异常的结束原因或低覆盖率问题,但容易被误读。有关每个计数器的含义,请查看https://scrapy-zyte-api.readthedocs.io/en/latest/reference/stats.md3. Log level
3. 日志级别
No DEBUG messages in the logs means is set above DEBUG. When a job
has issues you can't diagnose from the available logs, rerun it with
.
LOG_LEVELLOG_LEVEL=DEBUGLOG_LEVEL日志中没有DEBUG消息意味着设置高于DEBUG级别。当任务出现问题但无法通过现有日志诊断时,请使用重新运行任务。
LOG_LEVELLOG_LEVEL=DEBUGLOG_LEVEL4. Field coverage
4. 字段覆盖率
Use the item stats endpoint (see "Item stats" below) to get per-field
population counts without downloading every item. Compare
against (total item count). Determine which fields were
expected:
counts[field]totals.input_values- If this job is part of the workflow, the expected fields are in
/scrape(see.scrape/{site}/{data-type}/spec.json).${CLAUDE_SKILL_DIR}/../scrape/references/extraction-spec.md - Otherwise, read the item / page-object class in the deployed project.
Flag any expected field at 0% coverage.
For fields between 0% and 100% coverage, download a few items where the field is
missing (see "Downloading items" below) and judge whether it's legitimately
absent or an extraction bug.
使用条目统计端点(见下文“条目统计”)获取每个字段的填充计数,无需下载所有条目。将与(总条目数)进行比较。确定哪些字段是预期存在的:
counts[field]totals.input_values- 如果此任务属于工作流,预期字段位于
/scrape中(见.scrape/{site}/{data-type}/spec.json)。${CLAUDE_SKILL_DIR}/../scrape/references/extraction-spec.md - 否则,查看已部署项目中的条目/页面对象类。
标记任何覆盖率为0%的预期字段。
对于覆盖率在0%到100%之间的字段,下载几个该字段缺失的条目(见下文“下载条目”),判断该字段是合理缺失还是提取错误。
5. Spot-check that items match the request
5. 抽样检查条目是否符合要求
Download a small sample and read it (see "Downloading items" below):
bash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET \
"${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT/SPIDER/JOB" -q count=5Judge whether the sampled items match what the user actually asked for — the
right categories, filters, or product types (from the spec's schema/examples
and start URLs, or from the conversation). This is a judgment call, not a
scripted assertion.
下载少量样本并查看(见下文“下载条目”):
bash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET \
"${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT/SPIDER/JOB" -q count=5判断抽样条目是否符合用户实际需求——正确的类别、筛选条件或产品类型(来自规范的Schema/示例和起始URL,或对话内容)。这需要主观判断,而非脚本化断言。
6. Report
6. 报告结果
Summarize: job link, , error count, a short field-coverage table,
and the sample check verdict.
close_reason总结内容:任务链接、、错误计数、简短的字段覆盖率表,以及抽样检查结论。
close_reasonOn a finding
发现问题时
If any check fails (errors, low coverage, wrong-category items), diagnose the
likely cause, fix the project code, redeploy (), reschedule,
and re-validate. Cap this at 3 total attempts — matching the local validation
loop in . If it still looks wrong after 3, stop and
report what remains broken rather than looping further.
uvx shub deployscrape-create-spider如果任何检查失败(错误、低覆盖率、错误类别的条目),诊断可能的原因,修复项目代码,重新部署(),重新调度并重新验证。最多尝试3次——与中的本地验证循环一致。如果3次后仍有问题,请停止操作并报告剩余的问题,而非继续循环。
uvx shub deployscrape-create-spiderManaging jobs
管理任务
Use the Scrapy Cloud Jobs HTTP API for listing and stopping jobs.
Base URL:
${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}使用Scrapy Cloud Jobs HTTP API列出和停止任务。
基础URL:
${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}Wrapper script
包装脚本
IMPORTANT It is critical that API requests are made with the wrapper script at
which handles authentication without leaking credentials to the agent. Do not make API requests with or other
tools that might expose credentials.
scripts/scrapy_cloud_api.pycurlbash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py HTTP_METHOD API_URL [-q QUERY_ARG=VALUE]... [-b BODY_ARG=VALUE]...With arbitrary query parameters () and body parameters () as needed per endpoint. See the script's help message for details.
-q-bOutput format: the script prints the HTTP status code on the first line,
then the response body on the remaining lines. Use the status line to detect
errors (e.g. / auth failures); when you only need the body, ignore
the first line. The response bodies shown below omit the leading status line.
401403重要提示:必须使用包装脚本来发起API请求,该脚本会处理身份验证且不会泄露凭证给助手。请勿使用或其他可能暴露凭证的工具发起API请求。
scripts/scrapy_cloud_api.pycurlbash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py HTTP_METHOD API_URL [-q QUERY_ARG=VALUE]... [-b BODY_ARG=VALUE]...可根据端点需求使用任意查询参数()和请求体参数()。有关详细信息,请查看脚本的帮助信息。
-q-b输出格式:脚本会在第一行打印HTTP状态码,剩余行打印响应体。使用状态码检测错误(例如/认证失败);如果只需要响应体,可以忽略第一行。以下显示的响应体省略了开头的状态码行。
401403List jobs
列出任务
bash
undefinedbash
undefinedAll running jobs
所有运行中的任务
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/list.json" -q project=PROJECT_ID -q state=running
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/list.json" -q project=PROJECT_ID -q state=running
Latest 3 finished jobs for a specific spider
特定爬虫的最新3个已完成任务
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/list.json" -q project=PROJECT_ID -q spider=SPIDER_NAME -q state=finished -q count=3
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/list.json" -q project=PROJECT_ID -q spider=SPIDER_NAME -q state=finished -q count=3
Jobs that lack the "consumed" tag
缺少"consumed"标签的任务
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/list.json" -q project=PROJECT_ID -q lacks_tag=consumed
Available `state` values: `pending`, `running`, `finished`, `deleted`.
Available filter parameters: `job`, `spider`, `state`, `has_tag`, `lacks_tag`, `count`.uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/list.json" -q project=PROJECT_ID -q lacks_tag=consumed
可用的`state`值:`pending`、`running`、`finished`、`deleted`。
可用的过滤参数:`job`、`spider`、`state`、`has_tag`、`lacks_tag`、`count`。Stop a job
停止任务
bash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py POST "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/stop.json" -b project=PROJECT_ID -b job=PROJECT_ID/SPIDER_ID/JOB_IDbash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py POST "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}jobs/stop.json" -b project=PROJECT_ID -b job=PROJECT_ID/SPIDER_ID/JOB_IDViewing items and logs
查看条目和日志
Direct the user to the Scrapy Cloud web UI to inspect items and logs.
Derive the web UI base URL from by stripping the
trailing path (default: ).
SCRAPY_CLOUD_ENDPOINT/api/https://app.zyte.com引导用户通过Scrapy Cloud Web UI查看条目和日志。
从中去除尾部的路径,得到Web UI基础URL(默认:)。
SCRAPY_CLOUD_ENDPOINT/api/https://app.zyte.comItems page
条目页面
${SCRAPY_CLOUD_ENDPOINT%api/}p/PROJECT_ID/SPIDER_ID/JOB_ID/itemsOpen with:
bash
BASE_UI="${SCRAPY_CLOUD_ENDPOINT%api/}"
xdg-open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/items" 2>/dev/null \
|| open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/items"${SCRAPY_CLOUD_ENDPOINT%api/}p/PROJECT_ID/SPIDER_ID/JOB_ID/items使用以下命令打开:
bash
BASE_UI="${SCRAPY_CLOUD_ENDPOINT%api/}"
xdg-open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/items" 2>/dev/null \
|| open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/items"Item stats (HTTP API)
条目统计(HTTP API)
For a quick summary of item counts and field coverage without downloading all
items, use the stats endpoint directly:
bash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}items/PROJECT_ID/SPIDER_ID/JOB_ID/stats"如需快速获取条目计数和字段覆盖率摘要,无需下载所有条目,请直接使用统计端点:
bash
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_ENDPOINT:-https://app.zyte.com/api/}items/PROJECT_ID/SPIDER_ID/JOB_ID/stats"Response: {"counts":{"field1":9350,"field2":514},"totals":{"input_bytes":14390294,"input_values":10000}}
响应:{"counts":{"field1":9350,"field2":514},"totals":{"input_bytes":14390294,"input_values":10000}}
Response fields:
| Field | Description |
|---------------------|--------------------------------------------|
| `counts[field]` | Number of times each field was populated. |
| `totals.input_bytes`| Total size of all items in bytes. |
| `totals.input_values` | Total number of items. |
响应字段:
| 字段 | 描述 |
|---------------------|--------------------------------------------|
| `counts[field]` | 每个字段被填充的次数。 |
| `totals.input_bytes`| 所有条目的总大小(字节)。 |
| `totals.input_values` | 总条目数。 |Downloading items (HTTP API)
下载条目(HTTP API)
To read the items themselves, fetch them from the storage API through the
wrapper script (never — the wrapper keeps credentials out of the
trajectory). The default response format is JSON Lines, one item per line:
curlbash
undefined如需查看条目本身,请通过包装脚本来从存储API获取(切勿使用——包装脚本可防止凭证泄露)。默认响应格式为JSON Lines,每行一个条目:
curlbash
undefinedAll items
所有条目
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT_ID/SPIDER_ID/JOB_ID"
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT_ID/SPIDER_ID/JOB_ID"
A bounded sample (recommended for spot-checks)
有限样本(推荐用于抽样检查)
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT_ID/SPIDER_ID/JOB_ID" -q count=5
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT_ID/SPIDER_ID/JOB_ID" -q count=5
Specific items by index (repeat -q index=N for each)
指定索引的条目(每个条目重复使用-q index=N)
uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT_ID/SPIDER_ID/JOB_ID" -q index=0 -q index=1
Redirect to a file and inspect locally with `jq`/`grep` for anything beyond a
quick look. For pagination (`start`, `startafter`), field filtering, gzip, and
other formats (`-q format=json`), see the items API docs linked below.uv run ${CLAUDE_SKILL_DIR}/scripts/scrapy_cloud_api.py GET "${SCRAPY_CLOUD_STORAGE_ENDPOINT:-https://storage.zyte.com/}items/PROJECT_ID/SPIDER_ID/JOB_ID" -q index=0 -q index=1
如需进行深入查看,请重定向到文件并使用`jq`/`grep`在本地检查。有关分页(`start`、`startafter`)、字段过滤、gzip压缩和其他格式(`-q format=json`)的信息,请参考下文链接的条目API文档。Log page
日志页面
${SCRAPY_CLOUD_ENDPOINT%api/}p/PROJECT_ID/SPIDER_ID/JOB_ID/logOpen with:
bash
BASE_UI="${SCRAPY_CLOUD_ENDPOINT%api/}"
xdg-open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/log" 2>/dev/null \
|| open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/log"${SCRAPY_CLOUD_ENDPOINT%api/}p/PROJECT_ID/SPIDER_ID/JOB_ID/log使用以下命令打开:
bash
BASE_UI="${SCRAPY_CLOUD_ENDPOINT%api/}"
xdg-open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/log" 2>/dev/null \
|| open "${BASE_UI:-https://app.zyte.com/}p/PROJECT_ID/SPIDER_ID/JOB_ID/log"Log levels
日志级别
Logs returned by the HTTP API include a numeric field:
level| Value | Level |
|---|---|
| 10 | DEBUG |
| 20 | INFO |
| 30 | WARNING |
| 40 | ERROR |
| 50 | CRITICAL |
Each log entry is a JSON object with fields: (Unix ms), , and .
timelevelmessageHTTP API返回的日志包含一个数字字段:
level| 值 | 级别 |
|---|---|
| 10 | DEBUG |
| 20 | INFO |
| 30 | WARNING |
| 40 | ERROR |
| 50 | CRITICAL |
每个日志条目都是一个JSON对象,包含字段:(Unix毫秒时间戳)、和。
timelevelmessageReference docs
参考文档
The Scrapy Cloud docs are available as Markdown (swap → ). Prefer
fetching these over guessing at API mechanics — they cover the exhaustive set
of parameters, formats, and fields:
.html.md- Get started (entry point): https://docs.zyte.com/scrapy-cloud/get-started.md
- Items API: https://docs.zyte.com/scrapy-cloud/usage/reference/http/items.md
- Logs API: https://docs.zyte.com/scrapy-cloud/usage/reference/http/logs.md
- Jobs API (fields, filters, scheduling): https://docs.zyte.com/scrapy-cloud/usage/reference/http/jobs.md
- Common HTTP params (formats, pagination, gzip, field filtering): https://docs.zyte.com/scrapy-cloud/usage/reference/http/index.md
Scrapy Cloud文档以Markdown格式提供(将替换为)。请优先参考这些文档,而非猜测API机制——它们涵盖了所有参数、格式和字段的详细信息:
.html.md- 入门指南(入口点):https://docs.zyte.com/scrapy-cloud/get-started.md
- 条目API:https://docs.zyte.com/scrapy-cloud/usage/reference/http/items.md
- 日志API:https://docs.zyte.com/scrapy-cloud/usage/reference/http/logs.md
- 任务API(字段、过滤、调度):https://docs.zyte.com/scrapy-cloud/usage/reference/http/jobs.md
- 通用HTTP参数(格式、分页、gzip压缩、字段过滤):https://docs.zyte.com/scrapy-cloud/usage/reference/http/index.md
When to Use
使用场景
Invoke this skill when the user asks to:
- Deploy a Scrapy project or spider to the cloud
- Run a spider on Scrapy Cloud / Zyte Cloud / Scrapinghub Cloud
- Upload a project to Zyte / Scrapinghub
- Schedule a spider remotely (with arguments, settings, priority, or units)
- Push code to Scrapy Cloud
- List, filter, or inspect jobs
- Stop a running job
- View scraped items or logs for a job
- Wait for a running job to finish
- Validate or check the results of a job (errors, field coverage, item quality)
Example phrases that should trigger this skill:
- "Deploy my project"
- "Deploy my spider to Scrapy Cloud"
- "I want to run my spider in the cloud"
- "How do I get my spider running on Scrapy Cloud?"
- "Upload my Scrapy project to Zyte"
- "Push my project to Zyte"
- "I want to schedule my spider remotely"
- "Schedule myspider on Scrapy Cloud"
- "List all running jobs"
- "Stop job 123/1/5"
- "Show items from job 123/1/5"
- "Show logs for job 123/1/5"
- "Run my spider on Scrapy Cloud and check the results"
- "Wait for job 123/1/5 and validate the output"
当用户提出以下请求时,调用此技能:
- 将Scrapy项目或爬虫部署到云端
- 在Scrapy Cloud/Zyte Cloud/Scrapinghub Cloud上运行爬虫
- 将项目上传到Zyte/Scrapinghub
- 远程调度爬虫(带参数、设置、优先级或资源单元)
- 将代码推送到Scrapy Cloud
- 列出、过滤或检查任务
- 停止运行中的任务
- 查看任务的爬取条目或日志
- 等待运行中的任务完成
- 验证或检查任务结果(错误、字段覆盖率、条目质量)
应触发此技能的示例语句:
- "部署我的项目"
- "将我的爬虫部署到Scrapy Cloud"
- "我想在云端运行我的爬虫"
- "如何让我的爬虫在Scrapy Cloud上运行?"
- "将我的Scrapy项目上传到Zyte"
- "将我的项目推送到Zyte"
- "我想远程调度我的爬虫"
- "在Scrapy Cloud上调度myspider"
- "列出所有运行中的任务"
- "停止任务123/1/5"
- "显示任务123/1/5的条目"
- "显示任务123/1/5的日志"
- "在Scrapy Cloud上运行我的爬虫并检查结果"
- "等待任务123/1/5完成并验证输出"
Environment Variables
环境变量
| Variable | Default | Description |
|---|---|---|
| (none) | Scrapy Cloud API key; falls back to |
| | Jobs API base URL (override for staging) |
| | Items/logs storage base URL (override for staging) |
The web UI base URL is derived from by stripping the
trailing path. For example:
SCRAPY_CLOUD_ENDPOINT/api/- →
https://app.zyte.com/api/https://app.zyte.com - →
https://app-staging.zyte.com/api/https://app-staging.zyte.com
| 变量 | 默认值 | 描述 |
|---|---|---|
| (无) | Scrapy Cloud API密钥;如果未设置,则回退到 |
| | Jobs API基础URL(可覆盖为 staging 环境) |
| | 条目/日志存储基础URL(可覆盖为 staging 环境) |
Web UI基础URL是从中去除尾部的路径得到的。例如:
SCRAPY_CLOUD_ENDPOINT/api/- →
https://app.zyte.com/api/https://app.zyte.com - →
https://app-staging.zyte.com/api/https://app-staging.zyte.com
Common issues
常见问题
| Symptom | Cause | Fix |
|---|---|---|
| | Invoke it as |
| API key missing or invalid | Run |
| API key lacks access to this project | Verify the project ID and key permissions |
| Spider name is wrong or project not deployed | Verify the spider name; deploy with |
| Wrong project ID or alias | Check |
| Wrong path in | Fix the |
| Dependency missing or wrong stack | Update requirements file and redeploy |
| Stack's | If a newer version of |
| Job failed, was cancelled, or hit a | Read the logs (filter |
| A field has 0% or unexpectedly low coverage | Broken/incorrect selector in the page object | Inspect the item stats and a sample of items, fix the page object, redeploy, and re-run. See "Validating job results" |
| 症状 | 原因 | 解决方案 |
|---|---|---|
| 直接调用 | 使用 |
| API密钥缺失或无效 | 运行 |
| API密钥无权访问此项目 | 验证项目ID和密钥权限 |
| 爬虫名称错误或项目未部署 | 验证爬虫名称;先使用 |
| 项目ID或别名错误 | 检查 |
| | 修正 |
| 依赖项缺失或与栈不兼容 | 更新依赖文件并重新部署 |
任务日志中出现 | 栈中的 | 如果PyPI上存在比栈中捆绑版本更新的 |
| 任务失败、被取消或触发了 | 查看日志(过滤 |
| 某个字段覆盖率为0%或意外偏低 | 页面对象中的选择器错误 | 检查条目统计和条目样本,修复页面对象,重新部署并重新运行。请查看“验证任务结果”部分 |