crw-map — URL discovery without content

When to use

• You need to know which URLs exist on a site before committing to a crawl — map first, then crawl only the section you need.
• Step 3 in the crw ladder: if you want content, combine with crw-scrape (single pages) or crw-crawl (bulk). Map is URL-only — no page content is returned.
• Estimating crawl cost:

  crw map docs.example.com | wc -l

  tells you how many pages a subsequent crawl would touch before you commit.
• Finding a specific path: filter the URL list with

  grep

  instead of crawling the whole site.

Quick start

crw map "https://docs.example.com"                       # URLs to stdout
crw map "https://docs.example.com" -d 3 --format json   # JSON object, depth 3
crw map "https://example.com" --sitemap-only             # sitemap.xml only
crw map "https://example.com" --no-sitemap               # link crawl only
crw map "https://example.com" --format json > .crw/urls.json

crw_map(url="https://docs.example.com")
crw_map(url="https://docs.example.com", maxDepth=3, limit=200)
crw_map(url="https://example.com", useSitemap=false)       # link crawl only
crw_map(url="https://example.com", crawlFallback=false)    # sitemap only

curl -X POST "$CRW_API_URL/v1/map" -H "Authorization: Bearer $CRW_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://docs.example.com","maxDepth":2,"limit":200}'
# Response: {"success":true,"data":{"links":[...]}}  — links are under data.links
# jq tip for REST: jq '.data.links[]'

Options

-d/--depth N

maxDepth

--format text|json

--sitemap-only

crawlFallback: false

--no-sitemap

useSitemap: false

limit

0

--js

--proxy URL

--stealth

--rate-limit N

--concurrency N

--timeout MS

truncated: true

totalDiscovered

limit: 0

The map → scrape / crawl pattern

# 1. Map to see what's there
crw map "https://docs.example.com" --format json > .crw/urls.json

# 2a. Grep for the section you need
grep '"authentication"' .crw/urls.json

# 2b. Scrape a single page
crw scrape "https://docs.example.com/api/authentication"

# 2c. Or crawl the whole /api section
crw crawl "https://docs.example.com/api" -d 2 -l 50

crw_map(url="https://docs.example.com", limit=0)
# inspect links[], pick the /changelog/* subset
crw_crawl(url="https://docs.example.com/changelog", maxDepth=1, maxPages=20)

Tips

• Map before crawl, always. A 3-second map call can save a 10-minute crawl abort. If the map returns 5 000 URLs and you only need

  /blog/*

  , scope the crawl to that sub-path.
• Sitemap + crawl fallback (default) is the most complete. sitemap.xml gives canonical URLs; the BFS link scan catches pages not in the sitemap. Use

  --sitemap-only

  only when you trust the sitemap is complete.
• Depth 2 covers most docs sites. Increase to 3-4 for deeply nested wikis; 1 is enough to enumerate top-level sections.
• Filter in shell, not in context. Pipe to

  grep

  ,

  jq '.links[]'

  (CLI JSON output), or

  wc -l

  rather than loading the full list into model context. For REST responses use

  jq '.data.links[]'

  (links are nested under

  data

  ).
• No content returned. Map is intentionally URL-only. If you want page content, follow up with

  crw scrape

  or

  crw crawl

  .

See also

• crw-scrape — scrape individual URLs from the map
• crw-crawl — bulk content extraction after mapping
• crw — hub skill with the full workflow ladder