Loading...
Loading...
Found 8 Skills
Expert guidance for building web scrapers and crawlers using the Scrapy Python framework with best practices for spider development, data extraction, and pipeline management.
General-purpose Scrapy Cloud skill — deploy projects, schedule spiders, list/stop jobs, and view items or logs. Use when asked to deploy a project or spider to Scrapy Cloud / Zyte Cloud / Scrapinghub, schedule or run a spider remotely, manage jobs, or inspect scraped items and logs.
Points to the BlockchainSpider open-source Python/Scrapy toolkit for collecting on-chain data—transfer subgraphs around an address or tx, EVM and Solana block/transaction ingestion, receipts/logs, and optional label plugins. Use when the user wants to build datasets, offline traces, or research pipelines alongside blockchain-analytics-operations and solana-tracing-specialist—not as a substitute for RPC provider ToS, rate limits, or legal review of sensitive crawls.
Add an empty web-poet page object to a Scrapy project
Apply Web Scraping with Python practices (Ryan Mitchell). Covers First Scrapers (Ch 1: urllib, BeautifulSoup), HTML Parsing (Ch 2: find, findAll, CSS selectors, regex, lambda), Crawling (Ch 3-4: single-domain, cross-site, crawl models), Scrapy (Ch 5: spiders, items, pipelines, rules), Storing Data (Ch 6: CSV, MySQL, files, email), Reading Documents (Ch 7: PDF, Word, encoding), Cleaning Data (Ch 8: normalization, OpenRefine), NLP (Ch 9: n-grams, Markov, NLTK), Forms & Logins (Ch 10: POST, sessions, cookies), JavaScript (Ch 11: Selenium, headless, Ajax), APIs (Ch 12: REST, undocumented), Image/OCR (Ch 13: Pillow, Tesseract), Avoiding Traps (Ch 14: headers, honeypots), Testing (Ch 15: unittest, Selenium), Parallel (Ch 16: threads, processes), Remote (Ch 17: Tor, proxies), Legalities (Ch 18: robots.txt, CFAA, ethics). Trigger on "web scraping", "BeautifulSoup", "Scrapy", "crawler", "spider", "scraper", "parse HTML", "Selenium scraping", "data extraction".
Set up the user's Zyte account and credentials. Use when the user asks to set up, log in, sign up, or get an API key for Zyte; when ZYTE_API_KEY is missing; when a site is blocked; or before Scrapy Cloud deployment.
Undetectable, adaptive, high-performance Python web data extraction. Automatically survives website structure changes, bypasses anti-bot systems (Cloudflare, WAFs), and outperforms BeautifulSoup/Scrapy. Includes stealth browser fetching, CSS/XPath selectors, CLI, interactive shell, and MCP AI server integration.
End-to-end web scraping workflow — from URL to working spider with web-poet page objects. Use this for full-site or multiple-page crawls that create a new scraping workflow. Do not use for fixing, debugging, or modifying an existing spider.