> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Map every URL on a website

> Build a complete list of a website's URLs with Zenrows Crawl, focused on the sections you care about, and export it to CSV.

Get a clean list of a site's URLs, such as every product, article or listing page, without fetching their content. Crawl follows the links from one start URL, keeps the ones that match your patterns, normalizes and deduplicates them, and hands you the full list as one file. Use it to plan a scrape, audit a site or feed a [Batch](/academy/scrape-from-a-list-of-urls) job.

**Uses:** [Crawl](/crawl/introduction) with [`include_patterns`, `exclude_patterns`, `depth` and `max_items`](/crawl/endpoints#start-a-crawl), and the [download endpoint](/crawl/endpoints#download-the-results)

## Ask your AI agent

With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"dark"}
Use Zenrows to map https://www.scrapingcourse.com/ecommerce/. Keep product pages and
listing pages, skip add-to-cart links, and spend up to 15 page loads. When it ends,
tell me how many product URLs and listing pages you found and list the first 10 product URLs.
```

## How patterns steer the crawl

`include_patterns` decides two things: which URLs Crawl returns, and which links it opens to find more. Plan your patterns around the path that leads to your items, not just the items themselves.

| Patterns | What Crawl does |
| - | - |
| `["/product/"]` | Opens the product links on the start page and returns product URLs. Add `/page/` to walk the numbered catalog pages too. |
| `["/product/", "/page/"]` | Opens product pages and listing pages, so it walks the whole catalog. Returns both kinds of URL; split them by pattern afterwards. |
| add `exclude_patterns: ["add-to-cart"]` | Drops any URL containing that text, even when it matches an include pattern. |

`depth` sets how many link steps Crawl follows from the start URL: `1` returns the links on the start page. `max_items` sets how many URLs Crawl keeps and `max_pages` how many pages it opens, both 100 by default. Each opened page is billed as one Fetch request.

## Map the site and export a CSV

Start a crawl without `output_format`, wait until it ends, download every URL as NDJSON, then write a CSV.

<CodeGroup>
  ```python Python (Requests) theme={"dark"}
  import csv
  import json
  import time
  import requests

  API = "https://api.zenrows.com/v1/crawls"
  headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}

  # 1. Start a URL crawl: listing pages and product pages
  crawl = requests.post(
      API,
      headers=headers,
      json={
          "url": "https://www.scrapingcourse.com/ecommerce/",
          "include_patterns": ["/page/", "/product/"],
          "exclude_patterns": ["add-to-cart"],
          "max_items": 500,
          "max_pages": 15,
      },
      timeout=30,
  ).json()
  crawl_id = crawl["crawl_id"]

  # 2. Wait until it ends
  while True:
      body = requests.get(f"{API}/{crawl_id}", headers=headers, params={"limit": 1}, timeout=30).json()
      if body["status"] != "running":
          break
      time.sleep(5)
  print(body["status"], body.get("stop_reason"), body["coverage"])

  # 3. Download every URL as NDJSON, one JSON object per line
  download = requests.get(f"{API}/{crawl_id}/download", headers=headers, timeout=120)
  download.raise_for_status()
  with open("urls.jsonl", "wb") as f:
      f.write(download.content)

  # 4. Export a CSV with one URL per row and its type
  with open("urls.jsonl") as src, open("urls.csv", "w", newline="") as dst:
      writer = csv.writer(dst)
      writer.writerow(["url", "type"])
      for line in src:
          url = json.loads(line)["url"]
          writer.writerow([url, "product" if "/product/" in url else "listing"])
  ```

  ```bash cURL theme={"dark"}
  # 1. Start a URL crawl
  curl -X POST "https://api.zenrows.com/v1/crawls" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://www.scrapingcourse.com/ecommerce/",
      "include_patterns": ["/page/", "/product/"],
      "exclude_patterns": ["add-to-cart"],
      "max_items": 500,
      "max_pages": 15
    }'

  # 2. Download every URL; the X-Crawl-Status header reads completed once the list is final
  curl -D - "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID/download" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
    -o urls.jsonl
  ```

  ```bash CLI theme={"dark"}
  zenrows crawl create https://www.scrapingcourse.com/ecommerce/ \
    --include-patterns /page/ --include-patterns /product/ \
    --exclude-patterns add-to-cart --max-items 500 --max-pages 15 --follow
  zenrows crawl download YOUR_CRAWL_ID --out urls.jsonl
  ```
</CodeGroup>

The CLI takes `--include-patterns` and `--exclude-patterns` once per pattern. See the [CLI guide](/cli/crawl/introduction).

## Example output

The Python run printed:

```text theme={"dark"}
completed max_pages {'pages_fetched': 15, 'pages_failed': 0, 'items_found': 131}
```

`urls.jsonl` holds one URL per line:

```json theme={"dark"}
{"url":"https://www.scrapingcourse.com/ecommerce/page/2/"}
{"url":"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"}
{"url":"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/"}
```

And `urls.csv` is ready for a spreadsheet:

```text theme={"dark"}
url,type
https://www.scrapingcourse.com/ecommerce/page/2/,listing
https://www.scrapingcourse.com/ecommerce/page/3/,listing
https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/,product
```

Fifteen opened pages found 131 URLs: 12 listing pages and 119 products. `stop_reason: max_pages` means the site holds more, so raise `max_pages` to map the rest. URLs come back normalized: `http` and `https`, `www.` and trailing-slash spellings of one URL count as one, and tracking parameters such as `utm_*` are removed.

## Map one level

For a quick inventory of what a single page links to, set `depth: 1`. Crawl opens the start page once and returns its matching links:

```bash cURL theme={"dark"}
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "include_patterns": ["/page/", "/product/"],
    "exclude_patterns": ["add-to-cart"],
    "depth": 1,
    "max_items": 500
  }'
```

On the catalog's first page that returns 22 URLs, 16 products and 6 pagination links, for the cost of one page.

## Customize it

| Change | How |
| - | - |
| Map the whole site | Leave out `include_patterns` and raise `max_items` and `max_pages` |
| One section of a site | Start from that section's URL and include its path, such as `["/blog/"]` |
| Skip filters and sorting URLs | `exclude_patterns: ["?orderby=", "?filter"]` |
| Each page's HTML too | Add `"output_format": "html"`, see [Crawl a website from a start URL](/academy/scrape-and-crawl-from-a-seed-url) |
| Scrape every mapped URL | Send `urls.jsonl` to [Batch](/academy/scrape-from-a-list-of-urls); the CLI's `zenrows batch create` reads the same one-URL-per-line format |
| Read results while the crawl runs | Page through `GET /v1/crawls/{crawl_id}` with `cursor`, see [Read the status and the results](/crawl/setup#read-the-status-and-the-results) |

## Troubleshooting

* **Few URLs, all from the first page**: your include patterns skip the pages that link onward. Add the pagination or category path, such as `/page/`.
* **`status: failed` with `no_items_found`**: no link matched your patterns. Check the patterns against real URLs on the start page; patterns match plain text anywhere in the URL.
* **The list stops short**: `stop_reason` is `max_items` or `max_pages`. Start a new crawl with higher values.
* **A download is missing URLs**: the `X-Crawl-Status` header read `running`. Download again once the crawl has ended.
* **`429` with `too_many_crawls`**: your account's Crawl and Batch jobs are all running. Wait the `Retry-After` seconds, then start the crawl.

## Related recipes

<CardGroup cols={2}>
  <Card title="Crawl a website from a start URL" icon="sitemap" href="/academy/scrape-and-crawl-from-a-seed-url">
    Crawl and store each page's HTML in the same run.
  </Card>

  <Card title="Scrape a list of URLs" icon="list" href="/academy/scrape-from-a-list-of-urls">
    Turn your URL inventory into pages or data with Batch.
  </Card>

  <Card title="Get structured JSON from any page" icon="wand" href="/academy/extract/auto-extraction">
    Extract fields from each mapped page.
  </Card>

  <Card title="Build a RAG knowledge base from a website" icon="database" href="/academy/pipelines/website-to-rag">
    From a mapped site to markdown chunks for retrieval.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.