Crawl

Crawl is New, and zenrows crawl ships in the Zenrows CLI from version 1.5.0. Any active Zenrows API key can use it. Install or update with npx -y @zenrows/cli.

zenrows crawl starts from one URL and returns the URLs it finds from there. A crawl runs in the background: create returns at once, then wait, results, content, and download read it.

zenrows crawl <create|get|list|results|content|download|stop|wait> [flags]

depth is optional. Leave it out and the crawl follows links until max_items or max_pages stops it, or nothing is left to open. Both limits default to 100 and cap at 100,000. The crawl stays on the start URL’s domain, subdomains included.

Examples

# Start a crawl and wait for it to finish
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --depth 1 --include-patterns /product/ --follow

# No depth limit. Stop after 20 fetched pages
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --max-pages 20

# Collect the URLs
zenrows crawl results <crawl-id> --out urls.jsonl

# Also fetch each kept page's HTML
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --output-format html --max-pages 20 --follow
zenrows crawl content <crawl-id> <content-id>
zenrows crawl download <crawl-id>

# Stop an in-flight crawl
zenrows crawl stop <crawl-id>

--include-patterns chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only /product/ skips listing paths that lack that text, such as /page/. See Crawl setup.

--follow and wait stop waiting after --timeout seconds (default 600). The crawl keeps running. Ctrl-C stops the wait, not the crawl.

Flags

--json prints structured output on every subcommand. The rest are subcommand-specific:

Subcommand	Flags
`create <url>`	`--depth <n>` (1 to 100,000; omit for no depth limit), `--max-items <n>`, `--max-pages <n>` (API default 100 each, cap 100,000; `--max-pages` is also capped by the local policy `max_pages_per_run`, default 1,000), `--include-patterns <p>` (repeatable), `--exclude-patterns <p>` (repeatable), `--output-format html`, `--follow`, `--timeout <s>`, `--idempotency-key <key>`, `--no-signup`
`get <id>`	`--cursor <c>`, `--limit <n>` (up to 10,000)
`list`	`--cursor <c>`, `--limit <n>` (up to 100)
`results <id>`	`--limit <n>` (page size), `--out <file>`
`content <id> <content_id>`	`--out <file>`. `content_id` is a content id or a result’s `content_url`
`download <id>`	`--out <file>` (default `<id>.jsonl`)
`wait <id>`	`--timeout <s>`
`stop <id>`

results on a running crawl prints the URLs kept so far. download while the crawl runs is that same partial set. Call wait first to read every result. Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See Crawl pricing.

Get Started

Fetch

Extract

Batch

Browser Sessions

Agent Toolkit

Reference

Crawl

Examples

Flags

​Examples

​Flags

Examples

Flags