Crawl Endpoints

Crawl’s endpoints live under https://api.zenrows.com/v1/crawls. Send your API key in the X-API-Key header or the apikey query parameter.

Method	Path	Does
`POST`	`/v1/crawls`	Start a crawl
`GET`	`/v1/crawls`	List your crawls
`GET`	`/v1/crawls/{crawl_id}`	Read a crawl and its results
`GET`	`/v1/crawls/{crawl_id}/download`	Download all results
`GET`	`/v1/crawls/{crawl_id}/contents/{content_id}`	Read one page’s HTML
`POST`	`/v1/crawls/{crawl_id}/stop`	Stop a crawl

Start a crawl

`POST /v1/crawls`

Starts a crawl and returns 202 Accepted at once, with the crawl in the body and its path in the Location header. The body is JSON.

PARAMETER	TYPE	DEFAULT	DESCRIPTION
url `required`	`string`		The start URL: a public `http` or `https` URL, up to 2,048 characters. The crawl starts on this exact URL, so one ending in `?page=3` starts on page 3. It stays on the URL’s domain, subdomains included. A link longer than 2,048 characters on a crawled page is skipped.
depth `required`	`integer`		How many link steps to follow from the start URL, from 1 to 100,000. At `1`, Crawl returns the links on the start page. At `2`, it also opens those pages and returns their links.
max_items	`integer`	`10`	Stops the crawl after it keeps this many URLs, from 1 to 100,000. With `output_format`, `max_pages` can stop it first, because each kept page takes a fetch.
max_pages	`integer`	`10`	Stops the crawl after it fetches this many pages, from 1 to 100,000. With `output_format`, each kept page’s fetch counts too. If both limits are reached together, `stop_reason` is `max_pages`.
include_patterns	`array` of `string`		Keeps a URL only if it contains at least one of these texts, such as `/product/`, and opens only those URLs. A homepage crawl with only `/product/` skips listing URLs that lack that text, such as `/page/`. The start URL is never returned.
exclude_patterns	`array` of `string`		Drops a URL that contains any of these texts, even if it matches an include pattern.
output_format	`string`		`html` also returns the HTML of each kept page. Leave it out to get URLs only.

Send an Idempotency-Key header (up to 255 characters) to make a retry safe: a retry with the same key and body returns the same crawl and starts no second one.

List your crawls

`GET /v1/crawls`

Returns your crawls, newest first, without their results.

PARAMETER	TYPE	DEFAULT	DESCRIPTION
cursor	`string`		The `next_cursor` from the previous page.
limit	`integer`	`20`	How many crawls to return, from 1 to 100.

The response holds a crawls array and a next_cursor. next_cursor is absent on the last page.

Read a crawl

`GET /v1/crawls/{crawl_id}`

Returns the crawl, its coverage so far, and one page of results.

PARAMETER	TYPE	DEFAULT	DESCRIPTION
cursor	`string`		The `next_cursor` from the previous read. Leave it out to start from the first result.
limit	`integer`	`1000`	How many results to return, from 1 to 10,000.

While the crawl runs, and while a stopped crawl still reads the pages fetched before the stop, next_cursor is never null: read again with the same cursor to get only the new URLs. next_cursor is null once the crawl has ended and you have read its last results. Each URL is returned once.

Response fields

Field	Description
`crawl_id`	The crawl’s ID.
`status`	`running`, `completed`, `stopped` or `failed`. See Statuses.
`stop_reason`	Why the crawl ended early: `max_items`, `max_pages` or `user`. Absent when nothing was left to open, and when the crawl failed.
`error`	Why the crawl failed, with a `code` and a `detail`. Present only when `status` is `failed`.
`url`, `depth`, `max_items`, `max_pages`, `include_patterns`, `exclude_patterns`, `output_format`	The parameters the crawl runs with.
`discovery`, `extract_calls`	`["links"]` and `0`: the crawl follows the links in each page’s HTML.
`coverage.pages_fetched`	Pages fetched so far, successful or not, at most `max_pages`. With `output_format`, this includes each kept page fetched for its HTML.
`coverage.pages_failed`	Pages that failed to load.
`coverage.items_found`	URLs the crawl returns so far, after duplicates and your patterns are applied. At most `max_items`.
`duplicates_removed`	Links the crawl met again and dropped. The results never hold a duplicate.
`created_at`, `finished_at`	When the crawl started and ended, in UTC. `finished_at` is absent while it runs.
`results[].url`	A URL the crawl found, normalized. Spellings of one URL count as one: `http` or `https`, with or without `www.` and a trailing slash. Tracking parameters, such as `utm_*`, `gclid` and `fbclid`, are removed.
`results[].content_status`	With `output_format` only: `pending`, `fetched` or `failed`.
`results[].content_url`	With `output_format` only, once `content_status` is `fetched`: the path to read the page.
`next_cursor`	Send it as `cursor` on the next read.

Statuses

`status`	Meaning
`running`	The crawl is opening pages. Results grow as it goes.
`completed`	The crawl finished. Nothing was left to open, or it reached `max_items` or `max_pages` (see `stop_reason`).
`stopped`	You stopped it with `POST /v1/crawls/{crawl_id}/stop`. `stop_reason` is `user`. The results hold the URLs kept before the stop and the links on the pages already fetched by then.
`failed`	The crawl could not go on. `error.code` says why. The URLs found so far stay readable.

`error.code`	Meaning	What to do
`seed_unreachable`	The start URL could not be fetched.	Check the URL. The site may be blocking it.
`no_items_found`	The crawl found no URL to return.	Check your `include_patterns` and `exclude_patterns`. Without patterns, the site may have served a block or consent page.
`insufficient_credits`	Your account ran out of credits.	Add credits or upgrade your plan, then start a new crawl.
`internal_error`	Something failed on our side.	Start a new crawl. If it fails again, contact support with the `crawl_id`.

Download the results

`GET /v1/crawls/{crawl_id}/download`

Returns every result as one NDJSON file (one JSON object per line), named {crawl_id}.jsonl. Each line has the url. With output_format, each line also has content_status and, once fetched, the page HTML in content. The X-Crawl-Status header holds the crawl’s status. While it is running, a later download may hold more lines. After a stop, it reads stopped, yet a later download may still hold more lines for up to 10 minutes: read the crawl until next_cursor is null to know the results are final.

Read a page

`GET /v1/crawls/{crawl_id}/contents/{content_id}`

Returns the HTML of one kept page, for a crawl started with output_format. Take the path from the result’s content_url.

PARAMETER	TYPE	DEFAULT	DESCRIPTION
download	`boolean`	`false`	`true` returns the page as a file to save.

Stop a crawl

`POST /v1/crawls/{crawl_id}/stop`

Stops a running crawl and returns 200 with its crawl_id, status, stop_reason and finished_at. Send no request body.

Response

{
  "crawl_id": "c_01M3XP7RZD4JT88F7PNPGBFRAF",
  "status": "stopped",
  "stop_reason": "user",
  "finished_at": "2026-10-09T09:01:30Z"
}

No new page is fetched after the stop. Pages already being fetched finish and are billed, and the crawl keeps them: for up to 10 minutes after the stop, it reads each one, counts it in coverage, and keeps the links on it within your patterns and max_items. With output_format, a URL kept after the stop reads content_status: pending, since no new page is fetched. status, stop_reason and finished_at stay as the stop set them. Read the crawl until next_cursor is null for its final results. A crawl that already ended returns its final state. If the stop returns 500 internal_error, send it again.

Limits

Limit	Value
Active crawls per account	3 by default, shared with your Batch jobs
`max_items`	10 by default, 100,000 at most
`max_pages`	10 by default, 100,000 at most
`depth`	Required, from 1 to 100,000
Results per read	1,000 by default, 10,000 at most

An account runs 3 active jobs by default, shared with Batch. One more returns 429 too_many_crawls, with a Retry-After header of 30 seconds. To run more than 3 at once, use the chat in your dashboard. Email success@zenrows.com if you can’t use the chat.

Pricing

Crawl has no fee of its own. Each page a crawl opens is one Fetch (formerly, Universal Scraper API) request with Adaptive Stealth Mode, billed at that mode’s price: from 1 to 25 credits per page, depending on the site. A page that fails to load (blocked, timed out or server error) is not billed. A page that answers 404 or 410 counts as failed but is billed, as in Fetch. A refused request starts no crawl and costs nothing. max_pages caps how many pages a crawl fetches, so it caps what the crawl costs. With output_format, each kept page counts as one fetch. A kept page that the crawl also opens is fetched once, unless the crawl fetches it for its HTML first and reaches it later from a page nearer the start: then it is fetched, and billed, twice.

Errors

Errors return as application/problem+json, with a code, a title, a detail that says what to do, the status and an instance to quote to support.

Code	Status	Meaning
`invalid_request`	`400`	The body is not valid JSON, the `Idempotency-Key` is over 255 characters, or the list’s `limit` or `cursor` is not valid.
`unknown_parameter`	`400`	The request has a body field or query parameter this endpoint does not accept. `detail` names it.
`invalid_cursor`	`400`	The `cursor` does not belong to this crawl.
`invalid_parameter`	`400`, `422`	A value is missing or out of range. `detail` names the field.
`crawl_not_found`	`404`	No crawl with this ID exists on your account.
`content_not_found`	`404`	The page is not fetched yet, its fetch failed, or the crawl has no `output_format`.
`idempotency_request_in_flight`	`409`	A request with this `Idempotency-Key` is still in progress. Retry when it ends.
`invalid_start_url`	`422`	The start URL is not a public `http` or `https` URL: its host is an IP address or a private address, or it is over 2,048 characters.
`idempotency_key_reused`	`422`	This `Idempotency-Key` was used with a different body.
`too_many_crawls`	`429`	Your account already runs 3 active jobs by default, shared with Batch. Wait for one to end.
`internal_error`	`500`	Something failed on our side. Retry later.

For authentication and plan errors, see the API error codes.

Get Started

Fetch

Extract

Crawl

Batch

Browser Sessions

Residential Proxies (Legacy)

Guides

Start a crawl

`POST /v1/crawls`

List your crawls

`GET /v1/crawls`

Read a crawl

`GET /v1/crawls/{crawl_id}`

Response fields

Statuses

Download the results

`GET /v1/crawls/{crawl_id}/download`

Read a page

`GET /v1/crawls/{crawl_id}/contents/{content_id}`

Stop a crawl

`POST /v1/crawls/{crawl_id}/stop`

Limits

Pricing

Errors

​Start a crawl

​POST /v1/crawls

​List your crawls

​GET /v1/crawls

​Read a crawl

​GET /v1/crawls/{crawl_id}

​Response fields

​Statuses

​Download the results

​GET /v1/crawls/{crawl_id}/download

​Read a page

​GET /v1/crawls/{crawl_id}/contents/{content_id}

​Stop a crawl

​POST /v1/crawls/{crawl_id}/stop

​Limits

​Pricing

​Errors

Start a crawl

`POST /v1/crawls`

List your crawls

`GET /v1/crawls`

Read a crawl

`GET /v1/crawls/{crawl_id}`

Response fields

Statuses

Download the results

`GET /v1/crawls/{crawl_id}/download`

Read a page

`GET /v1/crawls/{crawl_id}/contents/{content_id}`

Stop a crawl

`POST /v1/crawls/{crawl_id}/stop`

Limits

Pricing

Errors