Skip to main content
POST
Start a crawl

Authorizations

X-API-Key
string
header
required

Batch, Crawl, the Extract endpoints and the account endpoint read the same key from the X-API-Key header.

Headers

Idempotency-Key
string

Makes a retry safe: a retry with the same key and body returns the same crawl and starts no second one. The same key with a different body is 422 idempotency_key_reused.

Maximum string length: 255

Body

application/json
url
string<uri>
required

The start URL: a public http or https URL. The crawl starts on this exact URL and stays on its domain, subdomains included.

Maximum string length: 2048
depth
integer

How many link steps to follow from the start URL. At 1, the links on the start page. Omit it to follow links until max_items or max_pages stops the crawl.

Required range: 1 <= x <= 100000
max_items
integer
default:100

Stop after keeping this many URLs.

Required range: 1 <= x <= 100000
max_pages
integer
default:100

Stop after fetching this many pages. Caps what the crawl costs. If both limits are reached together, stop_reason is max_pages.

Required range: 1 <= x <= 100000
include_patterns
string[]

Keep, and open, only URLs that contain at least one of these texts, such as /product/. The start URL is never returned.

exclude_patterns
string[]

Drop URLs that contain any of these texts, even when they match an include pattern.

output_format
enum<string>

html also returns the HTML of each kept page. Omit it to get URLs only.

Available options:
html

Response

The crawl started.

A crawl, its coverage so far and one page of results.

crawl_id
string
status
enum<string>

completed: nothing was left to open, or a limit was reached (see stop_reason). stopped: you stopped it. failed: see error.

Available options:
running,
completed,
stopped,
failed
stop_reason
enum<string>

Why the crawl ended early. Absent when nothing was left to open, and when the crawl failed.

Available options:
max_items,
max_pages,
user
error
object

Present only when status is failed. Older crawls can carry domain_not_allowed (the domain was not prepared for Extract).

url
string<uri>

The start URL: a public http or https URL. The crawl starts on this exact URL and stays on its domain, subdomains included.

Maximum string length: 2048
depth
integer

How many link steps to follow from the start URL. At 1, the links on the start page. Omit it to follow links until max_items or max_pages stops the crawl.

Required range: 1 <= x <= 100000
max_items
integer
default:100

Stop after keeping this many URLs.

Required range: 1 <= x <= 100000
max_pages
integer
default:100

Stop after fetching this many pages. Caps what the crawl costs. If both limits are reached together, stop_reason is max_pages.

Required range: 1 <= x <= 100000
include_patterns
string[]

Keep, and open, only URLs that contain at least one of these texts, such as /product/. The start URL is never returned.

exclude_patterns
string[]

Drop URLs that contain any of these texts, even when they match an include pattern.

output_format
enum<string>

html also returns the HTML of each kept page. Omit it to get URLs only.

Available options:
html
discovery
enum<string>[]

["links"]: the crawl follows the links in each page's HTML. Older crawls can also list pagination.

Available options:
links,
pagination
coverage
object
duplicates_removed
integer

Links met again and dropped. Results never hold a duplicate.

created_at
string<date-time>
finished_at
string<date-time>

Absent while the crawl runs.

results
object[]
next_cursor
string | null

Send it as cursor on the next read. Never null while the crawl runs; null once it has ended and its last results were read.