> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Python SDK

> Start a crawl, wait for it, and read its URLs or HTML with the official Zenrows Python SDK. Crawl is included from version 1.6.0.

<Note>
  Crawl is **New**. Any active Zenrows API key can use it. The `ZenRowsCrawlClient` shown here is separate from the SDK's synchronous scraping client, `ZenRowsClient`, and it ships in the Python SDK from version 1.6.0. Python 3.10 or later is required.
</Note>

Use this guide to start a crawl, wait for it, and read the URLs it keeps, or each page's HTML, with the official Zenrows Python SDK.

<Tip>
  Prefer raw HTTP? [Crawl setup](/crawl/setup) covers the same calls against the REST API. Parameters and errors are in the [endpoint reference](/crawl/endpoints). For the complete method reference, see the [SDK repository on GitHub](https://github.com/ZenRows/zenrows-python-sdk).
</Tip>

## Install and authenticate

```bash theme={"dark"}
pip install "zenrows>=1.6.0"
```

```python Python theme={"dark"}
import os
from zenrows import ZenRowsCrawlClient

client = ZenRowsCrawlClient(api_key=os.environ["ZENROWS_API_KEY"])
```

The client also works as a context manager (`with ZenRowsCrawlClient(...) as client:`) when you want the connection to close automatically. It targets `https://api.zenrows.com/v1`. Pass `base_url=...` to point at a different host. `retries` defaults to 3 (set `0` to disable) and `timeout` defaults to 30 seconds per request.

## Quickstart

`create` returns as soon as the crawl is accepted, with `status` `running`. `wait` polls until the crawl ends, or until `timeout` seconds pass (default 600), and returns the crawl. It does not raise on timeout: the crawl is still `running`, and you can call `wait` again. A crawl that ends `failed` is returned too, with `error` set. `results` follows `next_cursor` and does not poll, so call `wait` first to read every URL.

`depth` is optional. Leave it out and the crawl follows links until `max_items` or `max_pages` stops it. Both default to 100 and cap at 100,000.

```python Python theme={"dark"}
crawl = client.create(
    "https://www.scrapingcourse.com/ecommerce/",
    depth=1,                         # omit for no depth limit
    max_pages=20,                    # stop after fetching 20 pages (default 100)
    include_patterns=["/product/"],  # keep only URLs containing this
)

crawl = client.wait(crawl.crawl_id)
print(crawl.status.value, crawl.coverage.items_found)

for result in client.results(crawl.crawl_id):
    print(result.url)
```

<Note>
  `include_patterns` chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only `/product/` skips listing paths that lack that text, such as `/page/`. See [Crawl setup](/crawl/setup#start-a-crawl).
</Note>

Pass `idempotency_key` to `create` so a retry with the same key and body returns the crawl the first request created.

`get(crawl_id, cursor=..., limit=...)` returns one page of results. `limit` is 1 to 10,000 (default 1,000). `next_cursor` is `None` only once the crawl has ended and you have read its last results.

## Read the pages

Pass `output_format="html"` to also fetch the page of every kept URL. `content` takes a result, its `content_url`, or its content id. `download` streams every result as NDJSON (one JSON object per line) and reports the crawl's status from the `X-Crawl-Status` header.

```python Python theme={"dark"}
crawl = client.create(
    "https://www.scrapingcourse.com/ecommerce/",
    max_pages=20,
    include_patterns=["/product/"],
    output_format="html",
)
client.wait(crawl.crawl_id)

for result in client.results(crawl.crawl_id):
    if result.content_url:  # content_status is fetched
        html = client.content(crawl.crawl_id, result)
        print(result.url, len(html))

with client.download(crawl.crawl_id) as download:
    print(download.status.value if download.status else None)
    for line in download.lines:
        print(line.url)
```

On a running crawl, `download` holds only the results kept so far. Call `wait` first for the full file.

With `output_format`, each kept page is one more fetch, and `max_pages` counts it. Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See [Crawl pricing](/crawl/endpoints#pricing).

## List and stop

`list` returns one page of the account's crawls, newest first, without their results. `stop` is safe to repeat: a crawl that already ended answers as it ended. Read the final counts with `get`.

```python Python theme={"dark"}
page = client.list(limit=20)
for item in page.crawls:
    print(item.crawl_id, item.status.value)

client.stop(crawl.crawl_id)  # the id from create
```

Pass `page.next_cursor` as `cursor` for the next page. It is `None` on the last page.

## Errors

Every non-2xx response raises `CrawlAPIError`. Branch on `status_code` and `code`. A crawl that ended `failed` is not an exception: read `crawl.error.code` and `crawl.error.detail`.

```python Python theme={"dark"}
from zenrows import CrawlAPIError

try:
    client.create("https://www.scrapingcourse.com/ecommerce/", max_pages=20)
except CrawlAPIError as exc:
    print(exc.status_code, exc.code, exc.detail)
    if exc.code == "too_many_crawls":
        print("retry in", exc.retry_after, "s")
```

An account already running 3 active jobs, shared with [Batch](/batch/introduction), gets `429` `too_many_crawls`. `create` does not retry that response. The other codes are in the [endpoint reference](/crawl/endpoints#errors).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.