> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Introduction to Crawl

> Give Zenrows Crawl one start URL, such as a homepage, a category or a search page, and get back the pages it finds: their URLs, and their HTML if you want it.

<Note>
  Crawl is in **Beta**. Any active Zenrows API key can use it, with no extra setup.
</Note>

Zenrows Crawl takes one start URL and returns the pages it finds from there. Point it at a homepage, a category page or a search results page, and Crawl follows the links for you and removes duplicates. You choose what you get back for each page: only its URL, or its HTML. You don't write a link loop, a filter or a deduplication step.

A crawl is a background job. You start it with one request, then read its status and its results while it runs.

## How it works

1. Crawl opens the start URL with Fetch (formerly, Universal Scraper API) and [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode) (`mode=auto`).
2. It reads every link in the page's HTML. It keeps the links on the same domain (subdomains included) that match your filters, and drops duplicates: `http` and `https` links to one page count as one. It skips a link longer than 2,048 characters.
3. While `depth` allows, it opens the kept links and reads their links too. One step from a page to its links is one level of `depth`.
4. It stops when nothing is left to open, or when it reaches `max_items` or `max_pages`.

`include_patterns` chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only `/product/` opens the product links on that page and skips listing and pagination URLs that don't contain that text, such as `/page/`, so it returns almost nothing from the rest of the site. To walk those pages, include their path as well and raise `depth`. Crawl returns the URLs that match.

## Basic usage

Start a crawl with a `POST` to `/v1/crawls`:

```bash cURL theme={"dark"}
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "depth": 1,
    "include_patterns": ["/product/"]
  }'
```

With only `/product/`, the request above returns product URLs linked from the homepage. It doesn't open `/page/` or other listing URLs.

The response is `202 Accepted` with a `crawl_id`. Read the crawl with `GET /v1/crawls/{crawl_id}`, sending each response's `next_cursor` as `cursor` on the next read, until `next_cursor` is `null`: then it has ended and you have every result.

You can also try Crawl without code in the [Crawl playground](https://app.zenrows.com/crawl).

## Next steps

<CardGroup cols={2}>
  <Card title="Crawl Setup" icon="rocket" href="/crawl/setup">
    Start a crawl, read its results, download them, and get each page's HTML.
  </Card>

  <Card title="Crawl Endpoints" icon="book" href="/crawl/endpoints">
    Every endpoint, parameter, limit and error code, plus pricing.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.