Map every URL on a website

Get a clean list of a site’s URLs, such as every product, article or listing page, without fetching their content. Crawl follows the links from one start URL, keeps the ones that match your patterns, normalizes and deduplicates them, and hands you the full list as one file. Use it to plan a scrape, audit a site or feed a Batch job. Uses: Crawl with include_patterns, exclude_patterns, depth and max_items, and the download endpoint

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to map https://www.scrapingcourse.com/ecommerce/. Keep product pages and
listing pages, skip add-to-cart links, and spend up to 15 page loads. When it ends,
tell me how many product URLs and listing pages you found and list the first 10 product URLs.

How patterns steer the crawl

include_patterns decides two things: which URLs Crawl returns, and which links it opens to find more. Plan your patterns around the path that leads to your items, not just the items themselves.

Patterns	What Crawl does
`["/product/"]`	Opens the product links on the start page and returns product URLs. Add `/page/` to walk the numbered catalog pages too.
`["/product/", "/page/"]`	Opens product pages and listing pages, so it walks the whole catalog. Returns both kinds of URL; split them by pattern afterwards.
add `exclude_patterns: ["add-to-cart"]`	Drops any URL containing that text, even when it matches an include pattern.

depth sets how many link steps Crawl follows from the start URL: 1 returns the links on the start page. max_items sets how many URLs Crawl keeps and max_pages how many pages it opens, both 100 by default. Each opened page is billed as one Fetch request.

Map the site and export a CSV

Start a crawl without output_format, wait until it ends, download every URL as NDJSON, then write a CSV.

import csv
import json
import time
import requests

API = "https://api.zenrows.com/v1/crawls"
headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}

# 1. Start a URL crawl: listing pages and product pages
crawl = requests.post(
    API,
    headers=headers,
    json={
        "url": "https://www.scrapingcourse.com/ecommerce/",
        "include_patterns": ["/page/", "/product/"],
        "exclude_patterns": ["add-to-cart"],
        "max_items": 500,
        "max_pages": 15,
    },
    timeout=30,
).json()
crawl_id = crawl["crawl_id"]

# 2. Wait until it ends
while True:
    body = requests.get(f"{API}/{crawl_id}", headers=headers, params={"limit": 1}, timeout=30).json()
    if body["status"] != "running":
        break
    time.sleep(5)
print(body["status"], body.get("stop_reason"), body["coverage"])

# 3. Download every URL as NDJSON, one JSON object per line
download = requests.get(f"{API}/{crawl_id}/download", headers=headers, timeout=120)
download.raise_for_status()
with open("urls.jsonl", "wb") as f:
    f.write(download.content)

# 4. Export a CSV with one URL per row and its type
with open("urls.jsonl") as src, open("urls.csv", "w", newline="") as dst:
    writer = csv.writer(dst)
    writer.writerow(["url", "type"])
    for line in src:
        url = json.loads(line)["url"]
        writer.writerow([url, "product" if "/product/" in url else "listing"])

# 1. Start a URL crawl
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "include_patterns": ["/page/", "/product/"],
    "exclude_patterns": ["add-to-cart"],
    "max_items": 500,
    "max_pages": 15
  }'

# 2. Download every URL; the X-Crawl-Status header reads completed once the list is final
curl -D - "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID/download" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -o urls.jsonl

zenrows crawl create https://www.scrapingcourse.com/ecommerce/ \
  --include-patterns /page/ --include-patterns /product/ \
  --exclude-patterns add-to-cart --max-items 500 --max-pages 15 --follow
zenrows crawl download YOUR_CRAWL_ID --out urls.jsonl

The CLI takes --include-patterns and --exclude-patterns once per pattern. See the CLI guide.

Example output

The Python run printed:

completed max_pages {'pages_fetched': 15, 'pages_failed': 0, 'items_found': 131}

urls.jsonl holds one URL per line:

{"url":"https://www.scrapingcourse.com/ecommerce/page/2/"}
{"url":"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"}
{"url":"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/"}

And urls.csv is ready for a spreadsheet:

url,type
https://www.scrapingcourse.com/ecommerce/page/2/,listing
https://www.scrapingcourse.com/ecommerce/page/3/,listing
https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/,product

Fifteen opened pages found 131 URLs: 12 listing pages and 119 products. stop_reason: max_pages means the site holds more, so raise max_pages to map the rest. URLs come back normalized: http and https, www. and trailing-slash spellings of one URL count as one, and tracking parameters such as utm_* are removed.

Map one level

For a quick inventory of what a single page links to, set depth: 1. Crawl opens the start page once and returns its matching links:

cURL

curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "include_patterns": ["/page/", "/product/"],
    "exclude_patterns": ["add-to-cart"],
    "depth": 1,
    "max_items": 500
  }'

On the catalog’s first page that returns 22 URLs, 16 products and 6 pagination links, for the cost of one page.

Customize it

Change	How
Map the whole site	Leave out `include_patterns` and raise `max_items` and `max_pages`
One section of a site	Start from that section’s URL and include its path, such as `["/blog/"]`
Skip filters and sorting URLs	`exclude_patterns: ["?orderby=", "?filter"]`
Each page’s HTML too	Add `"output_format": "html"`, see Crawl a website from a start URL
Scrape every mapped URL	Send `urls.jsonl` to Batch; the CLI’s `zenrows batch create` reads the same one-URL-per-line format
Read results while the crawl runs	Page through `GET /v1/crawls/{crawl_id}` with `cursor`, see Read the status and the results

Troubleshooting

Few URLs, all from the first page: your include patterns skip the pages that link onward. Add the pagination or category path, such as /page/.
status: failed with no_items_found: no link matched your patterns. Check the patterns against real URLs on the start page; patterns match plain text anywhere in the URL.
The list stops short: stop_reason is max_items or max_pages. Start a new crawl with higher values.
A download is missing URLs: the X-Crawl-Status header read running. Download again once the crawl has ended.
429 with too_many_crawls: your account’s Crawl and Batch jobs are all running. Wait the Retry-After seconds, then start the crawl.

Crawl a website from a start URL

Crawl and store each page’s HTML in the same run.

Scrape a list of URLs

Turn your URL inventory into pages or data with Batch.

Get structured JSON from any page

Extract fields from each mapped page.

Build a RAG knowledge base from a website

From a mapped site to markdown chunks for retrieval.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Map every URL on a website

Ask your AI agent

How patterns steer the crawl

Map the site and export a CSV

Example output

Map one level

Customize it

Troubleshooting

Crawl a website from a start URL

Scrape a list of URLs

Get structured JSON from any page

Build a RAG knowledge base from a website

​Ask your AI agent

​How patterns steer the crawl

​Map the site and export a CSV

​Example output

​Map one level

​Customize it

​Troubleshooting

​Related recipes

Crawl a website from a start URL

Scrape a list of URLs

Get structured JSON from any page

Build a RAG knowledge base from a website

Ask your AI agent

How patterns steer the crawl

Map the site and export a CSV

Example output

Map one level

Customize it

Troubleshooting

Related recipes