Web Scraper

Core Capabilities

• Navigate to any URL and render full JavaScript pages
• Snapshot pages to understand structure and find interactive elements
• Extract text, HTML, or attributes from DOM selectors
• Paginate — click through pages, infinite scroll, load more buttons
• Handle auth — log in, manage sessions, restore cookies
• Anti-detection — rotate proxies, manage fingerprints

Scraping Workflow

1. Navigate to the target URL
2. Snapshot the page to understand its structure
3. Identify patterns — find the data elements (product cards, article listings, etc.)
4. Extract data — pull text/attributes from identified selectors
5. Paginate — navigate to next page and repeat
6. Handle errors — retry on failures, screenshot for debugging

Best Practices

• Respect robots.txt — check before scraping
• Rate limit requests — don't overwhelm servers (minimum 1-2 second delays)
• Use sessions — save and restore login state to avoid re-authentication
• Handle dynamic content — wait for elements to load before extracting
• Validate data — check extracted data for completeness
• Take screenshots on errors for debugging

Anti-Detection

• Rotate user agents and viewport sizes
• Use proxy rotation when available
• Add random delays between actions
• Avoid scraping too fast from a single IP
• Handle CAPTCHAs when they appear

Data Output

• Return arrays of objects with consistent field names
• Include metadata (source URL, timestamp, page number)
• Handle missing fields gracefully (null, not undefined)