Web Fetch

Prerequisites

• Google Chrome or Chromium: Must be installed on the system.
  • macOS:

    brew install --cask google-chrome

  • Ubuntu/Debian:

    sudo apt install google-chrome-stable

  • Windows:

    winget install Google.Chrome

• Python 3: Standard library only. Zero pip dependencies. Optionally

  trafilatura

  for better content extraction.
• No API keys or tokens required.

Usage

Single URL

python <skill-path>/scripts/web_fetch.py --url "https://example.com/article"

Multiple URLs (single Chrome session, efficient)

python <skill-path>/scripts/web_fetch.py --url "https://example.com/a" --url "https://example.com/b"

Piped input (one URL per line)

echo "https://example.com/article" | python <skill-path>/scripts/web_fetch.py

JSON output to file (recommended for multiple URLs)

# Cross-platform temp dir
TMPDIR=$(python -c "import tempfile; print(tempfile.gettempdir())")
python <skill-path>/scripts/web_fetch.py --url "https://example.com" --format json -o "$TMPDIR/results.json"

Options

--url

--format

text

text

json

-o, --output

--wait-ms

1500

--timeout

15

--max-chars

15000

--batch-size

3

--port

Output

Text mode (default)

---

JSON mode

[
  {
    "url": "https://news.google.com/rss/articles/...",
    "final_url": "https://www.theguardian.com/actual-article",
    "success": true,
    "content": "Article text content..."
  }
]

[
  {
    "url": "https://example.com/broken",
    "success": false,
    "error": "Page load timeout"
  }
]

How It Works

1. Finds Chrome — Checks platform-specific paths, then searches PATH
2. Launches isolated Chrome —

   --headless=new --remote-debugging-port=PORT --user-data-dir=TMPDIR

   (never conflicts with your running Chrome)
3. Controls via CDP — Connects over WebSocket using a pure-Python CDP client (no dependencies)
4. Navigates & waits — Uses

   Page.navigate

   , listens for

   loadEventFired

   , then waits for JS rendering
5. Captures content — Gets final URL via

   Runtime.evaluate

   , extracts DOM via

   DOM.getOuterHTML

6. Extracts text — Uses

   trafilatura

   (if installed) or falls back to HTML tag stripping
7. Cleans up — Terminates Chrome and removes temp profile

Key Advantages

• No profile conflicts — Uses isolated

  --user-data-dir

  , works alongside your running Chrome
• Handles JS redirects — Google News, URL shorteners, SPA routing all work
• Zero dependencies — Pure Python 3 stdlib (WebSocket client included)
• Single Chrome session — Multiple URLs reuse one instance (fast)
• Cross-platform — Mac, Windows, Linux

Cross-Platform Support

/Applications/Google Chrome.app/...

/usr/bin/google-chrome

%ProgramFiles%\Google\Chrome\...

%LocalAppData%\...

PATH

Notes

• No configuration needed — Just have Chrome installed. No need to enable "remote debugging" or change any Chrome settings. The script handles everything automatically.
• macOS first run — The system may show a popup asking "Do you want the application Google Chrome to accept incoming network connections?" Click Allow. This is because CDP uses a local port (

  127.0.0.1

  ) — no external network traffic is involved.