> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Crawl

> Collect the URLs behind one start page from the terminal, and each page's HTML when you ask for it.

<Note>
  Crawl is **New**, and `zenrows crawl` ships in the Zenrows CLI from version 1.5.0. Any active Zenrows API key can use it. Install or update with `npx -y @zenrows/cli`.
</Note>

`zenrows crawl` starts from one URL and returns the URLs it finds from there. A crawl runs in the background: `create` returns at once, then `wait`, `results`, `content`, and `download` read it.

```bash theme={"dark"}
zenrows crawl <create|get|list|results|content|download|stop|wait> [flags]
```

`depth` is optional. Leave it out and the crawl follows links until `max_items` or `max_pages` stops it, or nothing is left to open. Both limits default to 100 and cap at 100,000. The crawl stays on the start URL's domain, subdomains included.

## Examples

```bash theme={"dark"}
# Start a crawl and wait for it to finish
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --depth 1 --include-patterns /product/ --follow

# No depth limit. Stop after 20 fetched pages
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --max-pages 20

# Collect the URLs
zenrows crawl results <crawl-id> --out urls.jsonl

# Also fetch each kept page's HTML
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --output-format html --max-pages 20 --follow
zenrows crawl content <crawl-id> <content-id>
zenrows crawl download <crawl-id>

# Stop an in-flight crawl
zenrows crawl stop <crawl-id>
```

<Note>
  `--include-patterns` chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only `/product/` skips listing paths that lack that text, such as `/page/`. See [Crawl setup](/crawl/setup#start-a-crawl).
</Note>

<Tip>
  `--follow` and `wait` stop waiting after `--timeout` seconds (default 600). The crawl keeps running. Ctrl-C stops the wait, not the crawl.
</Tip>

## Flags

`--json` prints structured output on every subcommand. The rest are subcommand-specific:

| Subcommand | Flags |
| - | - |
| `create <url>` | `--depth <n>` (1 to 100,000; omit for no depth limit), `--max-items <n>`, `--max-pages <n>` (API default 100 each, cap 100,000; `--max-pages` is also capped by the local policy `max_pages_per_run`, default 1,000), `--include-patterns <p>` (repeatable), `--exclude-patterns <p>` (repeatable), `--output-format html`, `--follow`, `--timeout <s>`, `--idempotency-key <key>`, `--no-signup` |
| `get <id>` | `--cursor <c>`, `--limit <n>` (up to 10,000) |
| `list` | `--cursor <c>`, `--limit <n>` (up to 100) |
| `results <id>` | `--limit <n>` (page size), `--out <file>` |
| `content <id> <content_id>` | `--out <file>`. `content_id` is a content id or a result's `content_url` |
| `download <id>` | `--out <file>` (default `<id>.jsonl`) |
| `wait <id>` | `--timeout <s>` |
| `stop <id>` | |

`results` on a running crawl prints the URLs kept so far. `download` while the crawl runs is that same partial set. Call `wait` first to read every result.

Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See [Crawl pricing](/crawl/endpoints#pricing).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.