Python SDK

Crawl is New. Any active Zenrows API key can use it. The ZenRowsCrawlClient shown here is separate from the SDK’s synchronous scraping client, ZenRowsClient, and it ships in the Python SDK from version 1.6.0. Python 3.10 or later is required.

Use this guide to start a crawl, wait for it, and read the URLs it keeps, or each page’s HTML, with the official Zenrows Python SDK.

Prefer raw HTTP? Crawl setup covers the same calls against the REST API. Parameters and errors are in the endpoint reference. For the complete method reference, see the SDK repository on GitHub.

Install and authenticate

pip install "zenrows>=1.6.0"

Python

import os
from zenrows import ZenRowsCrawlClient

client = ZenRowsCrawlClient(api_key=os.environ["ZENROWS_API_KEY"])

The client also works as a context manager (with ZenRowsCrawlClient(...) as client:) when you want the connection to close automatically. It targets https://api.zenrows.com/v1. Pass base_url=... to point at a different host. retries defaults to 3 (set 0 to disable) and timeout defaults to 30 seconds per request.

Quickstart

create returns as soon as the crawl is accepted, with status running. wait polls until the crawl ends, or until timeout seconds pass (default 600), and returns the crawl. It does not raise on timeout: the crawl is still running, and you can call wait again. A crawl that ends failed is returned too, with error set. results follows next_cursor and does not poll, so call wait first to read every URL. depth is optional. Leave it out and the crawl follows links until max_items or max_pages stops it. Both default to 100 and cap at 100,000.

Python

crawl = client.create(
    "https://www.scrapingcourse.com/ecommerce/",
    depth=1,                         # omit for no depth limit
    max_pages=20,                    # stop after fetching 20 pages (default 100)
    include_patterns=["/product/"],  # keep only URLs containing this
)

crawl = client.wait(crawl.crawl_id)
print(crawl.status.value, crawl.coverage.items_found)

for result in client.results(crawl.crawl_id):
    print(result.url)

include_patterns chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only /product/ skips listing paths that lack that text, such as /page/. See Crawl setup.

Pass idempotency_key to create so a retry with the same key and body returns the crawl the first request created. get(crawl_id, cursor=..., limit=...) returns one page of results. limit is 1 to 10,000 (default 1,000). next_cursor is None only once the crawl has ended and you have read its last results.

Read the pages

Pass output_format="html" to also fetch the page of every kept URL. content takes a result, its content_url, or its content id. download streams every result as NDJSON (one JSON object per line) and reports the crawl’s status from the X-Crawl-Status header.

Python

crawl = client.create(
    "https://www.scrapingcourse.com/ecommerce/",
    max_pages=20,
    include_patterns=["/product/"],
    output_format="html",
)
client.wait(crawl.crawl_id)

for result in client.results(crawl.crawl_id):
    if result.content_url:  # content_status is fetched
        html = client.content(crawl.crawl_id, result)
        print(result.url, len(html))

with client.download(crawl.crawl_id) as download:
    print(download.status.value if download.status else None)
    for line in download.lines:
        print(line.url)

On a running crawl, download holds only the results kept so far. Call wait first for the full file. With output_format, each kept page is one more fetch, and max_pages counts it. Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See Crawl pricing.

List and stop

list returns one page of the account’s crawls, newest first, without their results. stop is safe to repeat: a crawl that already ended answers as it ended. Read the final counts with get.

Python

page = client.list(limit=20)
for item in page.crawls:
    print(item.crawl_id, item.status.value)

client.stop(crawl.crawl_id)  # the id from create

Pass page.next_cursor as cursor for the next page. It is None on the last page.

Errors

Every non-2xx response raises CrawlAPIError. Branch on status_code and code. A crawl that ended failed is not an exception: read crawl.error.code and crawl.error.detail.

Python

from zenrows import CrawlAPIError

try:
    client.create("https://www.scrapingcourse.com/ecommerce/", max_pages=20)
except CrawlAPIError as exc:
    print(exc.status_code, exc.code, exc.detail)
    if exc.code == "too_many_crawls":
        print("retry in", exc.retry_after, "s")

An account already running 3 active jobs, shared with Batch, gets 429 too_many_crawls. create does not retry that response. The other codes are in the endpoint reference.

Get Started

Fetch

Extract

Crawl

Batch

Browser Sessions

Install and authenticate

Quickstart

Read the pages

List and stop

Errors

​Install and authenticate

​Quickstart

​Read the pages

​List and stop

​Errors

Install and authenticate

Quickstart

Read the pages

List and stop

Errors