> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Python SDK

> Submit batch scraping jobs, track them, and collect results with the official Zenrows Python SDK. Covers every Batch feature with runnable examples.

> [!NOTE]
>
> Batch is available to every Zenrows account. You need an active Zenrows API key. The `ZenRowsBatchClient` shown here is separate from the SDK's synchronous scraping client, `ZenRowsClient`.

Use this guide to submit batch scraping jobs, track them, and collect results with the official Zenrows Python SDK. It walks through every Batch feature, and each section runs on its own.

> [!TIP]
>
> Prefer raw HTTP, or work in another language? The [REST API guide](https://docs.zenrows.com/batch/developer-guide-restapi.md) covers the same use cases against the plain REST API. Limits are in the [introduction](https://docs.zenrows.com/batch/introduction.md#limits) and error codes in [API error codes](https://docs.zenrows.com/api-error-codes.md#batch). For the complete method reference, see the [SDK repository on GitHub](https://github.com/ZenRows/zenrows-python-sdk).

## Install and authenticate

Install the SDK from PyPI. The batch client ships in the same `zenrows` package as the synchronous client:

```bash theme={"theme":"css-variables"}
pip install zenrows
```

This page describes `zenrows` 1.10.0 or later. Webhook verification, `BatchErrorCode`, `on_skip` and the paused-run waiter behavior arrived in 1.10.0.

```python Python theme={"theme":"css-variables"}
import os
from zenrows import ZenRowsBatchClient

client = ZenRowsBatchClient(api_key=os.environ["ZENROWS_API_KEY"])
```

The client also works as a context manager (`with ZenRowsBatchClient(...) as client: ...`) when you want connections to close automatically. It also takes `timeout` (seconds per API request, default 30) and `retries` (default 3). By default it targets the Batch API at `https://async.api.zenrows.com/v1`; pass `base_url=...` (or set `ZENROWS_BATCH_BASE_URL`) to point at a different environment.

## Quickstart: submit, wait, get results

This is the standard end-to-end flow. `submit_regular` returns a `JobRef` (no GET yet), `job.run.wait()` blocks until the current run reaches a **terminal** state and returns a `RunHandle`, and `run.results(...)` iterates the per-task results.

```python Python theme={"theme":"css-variables"}
job = client.submit_regular(
    [
        "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie",
        "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket",
        "https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee",
    ],
    zenrows_params={"js_render": "true", "premium_proxy": "true"},
)

run = job.run.wait(timeout=600.0)   # blocks until the current run is terminal
s = run.stats
print(f"{s.successful}/{s.total} successful, {s.failed} failed")

for task in run.results(status="successful"):
    print(task.external_id or task.task_id, "->", task.result_url)
```

`job.run.wait()` returns a `RunHandle` whose `.data` snapshot is ready, so `run.results(...)` and `run.download_to_dir(...)` work without re-passing IDs. It returns on every terminal status (`completed`, `stopped`, `failed`), so check `run.status`. A run that ended `failed` stopped for an account reason: `run.data.failure_reason` is `insufficient_credits`, `subscription_inactive` or `api_key_cap_reached`, and `run.data.failure_detail` explains it.

A paused run makes no progress until someone resumes it, so the waiter doesn't poll it until its timeout: it raises `RunPaused` (a `WaiterError`) with the paused run on `.run`. Pass `wait_while_paused=True` to keep polling, for example when another process resumes it.

> [!TIP]
>
> Pass `progress=True` to the waiter for a live progress bar on long batches. A submit of 10,000 tasks or more returns `202 Accepted` while the API is still writing the run's tasks. Pass `wait_for_ingest=True` to `submit_regular` so results pages are complete before you read them.

## Configuring the scrape

`zenrows_params` sets scraping options for every URL in the job. Values can be strings, booleans, or integers.

```python Python theme={"theme":"css-variables"}
job = client.submit_regular(
    ["https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie"],
    zenrows_params={
        "js_render": "true",          # render JavaScript in a headless browser
        "premium_proxy": "true",      # route through premium residential proxies
        "proxy_country": "us",        # geo-target the request
        "wait_for": ".price",         # wait for this CSS selector before capturing
        "response_type": "markdown",  # markdown | plaintext | pdf; omit it for HTML
    },
)
```

Zenrows derives the job's **output format** from these options: `response_type: markdown` produces Markdown; `extract`, `json_response`, or a CSS extractor produce JSON; otherwise the result is HTML. Zenrows records this format on each result. Other common keys include `mode`, `js_instructions`, `wait`, `block_resources`, `json_response`, `css_extractor`, `extract`, `extract_fields`, `outputs`, `original_status`, `allowed_status_codes`, and `session_id`. The `autoparse` parameter is still accepted but [deprecated](https://docs.zenrows.com/fetch/features/autoparse.md). Batch doesn't take the screenshot parameters, and the API rejects an unknown key with `400`.

> [!NOTE]
>
> `custom_headers` accepts a dict of header name to value, for example `{"custom_headers": {"Referer": "https://www.google.com/"}}`, and is also how you set the `Content-Type` for a [POST task](https://docs.zenrows.com/batch/developer-guide-restapi.md#post-target-fetches). Up to 32 headers, 8 KB serialized.

## Per-URL options and correlating results

To set a per-URL `external_id`, `metadata`, or `zenrows_params`, pass dictionaries instead of plain strings. Per-URL `zenrows_params` override the job-level ones when a key appears in both. A task can also fetch its target with `POST`: set `"method": "POST"` and a `"body"` (a JSON value or a string, up to 16 KiB). A POST task can't use `js_render`, `js_instructions` or `json_response`.

```python Python theme={"theme":"css-variables"}
job = client.submit_regular(
    [
        {"url": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short",
         "external_id": "order-42",
         "zenrows_params": {"proxy_country": "de"}},
        {"url": "https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee",
         "external_id": "order-43",
         "metadata": {"sku": "ABC-1"}},
        {"url": "https://httpbin.io/post", "method": "POST", "body": {"q": "running shoes"}},
    ],
    zenrows_params={"premium_proxy": "true"},   # applies to every task unless overridden
)
```

> [!TIP]
>
> Zenrows returns your `external_id` unchanged on every result, so you can match results back to your own records without tracking the API's task IDs. It must match `^[A-Za-z0-9._-]+$` and be 128 characters or fewer, and it doesn't have to be unique. `metadata` takes up to 20 string values.

## Downloading content

`results()` gives you metadata plus a presigned `result_url` per task; the SDK downloads bodies straight from that URL with no extra API round-trip. Four options, on the current run (`job.run.*`) or any specific run (`client.get_run(job_id, run_id=...)`):

| You want | Use | Where it lands |
| - | - | - |
| Every body, one file per task | `run.download_to_dir(dir)` | files on disk |
| Every body, kept in memory | `run.download_to_memory()` | `list[DownloadedResult]` |
| The whole run as one artifact | `run.download_all_results("out.zip")` | one `.zip` |
| One task's body (inside a loop) | `run.download_task_to_file(task, path)` / `run.download_task_to_memory(task)` | file / `bytes` |

```python Python theme={"theme":"css-variables"}
# Bulk to disk, named by external_id, 8 downloads in parallel:
n = run.download_to_dir(
    "./out",
    status="successful",       # default; pass status=None for everything
    use_external_id=True,      # name files <external_id>.<ext> instead of <task_id>.<ext>
    concurrency=8,
    progress=True,
    max_files=20_000,          # safety caps that raise DownloadLimitExceeded
    max_bytes_per_file=10 * 1024 * 1024,
    on_skip=lambda task: print("skipped", task.task_id, task.status),
)
print(f"downloaded {n} files")

# One task at a time, when you want a Python-side filter:
for task in run.results(status="successful"):
    if not task.result_url:
        continue                               # expired: past the retention window
    body = run.download_task_to_memory(task)   # bytes; or download_task_to_file(task, path)
```

Each `result_url` is a plain presigned URL (no auth header) valid for 2 hours, so fetch it promptly or re-list the results to get a fresh one.

Results are kept for 14 days by default. After that, a successful task reads `expired` (`TaskStatus.EXPIRED`) and has no `result_url`, and a `status="successful"` filter still returns it. `download_to_dir` and `download_to_memory` skip `expired` and `failed` rows, which have no body to fetch, pass each one to `on_skip`, and log one warning with the counts. Reading an expired task's content through the API raises `BatchAPIError` with status 410 and code `result_expired`.

## Open jobs: streaming URLs in batches

When you don't have every URL up front, open a job, add tasks in batches, and close it. The run starts with the first tasks and completes once it finishes the final batch.

```python Python theme={"theme":"css-variables"}
job = client.submit_open(zenrows_params={"js_render": "true"})

job.add_tasks({"tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/page/1"}]},
              idempotency_key="pages-batch-1")
job.add_tasks({"tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/page/2"}],
               "last_batch": True},  # closes the job
              idempotency_key="pages-batch-2")

run = job.run.wait(timeout=900.0)
```

Each call takes up to 10,000 URLs, and an open job holds up to 100,000 in total. With an `idempotency_key`, a repeat of the same call adds its tasks once, so the SDK retries it on a transient failure. Without one, every call appends and a failed call isn't retried. Setting `"last_batch": True` (or calling `job.close()`) locks the job, and the API rejects further adds. An open job counts as one of your active jobs until you delete it, or close it once its run has ended. File-input (CSV) upload isn't supported for open jobs; use a closed job for CSV.

## Large inputs: CSV upload

For lists too large to inline, upload a CSV once and reference it. `upload_csv` allocates the slot, PUTs your file to the presigned URL, and returns a `file_input_id` that `submit_regular(file_input_id=...)` consumes.

```python Python theme={"theme":"css-variables"}
file_input_id = client.upload_csv(
    "urls.csv",
    fields={"url": "URL", "external_id": "Customer Ref"},  # column name (header=True) or 0-based index
    header=True,
)
job = client.submit_regular(file_input_id=file_input_id, zenrows_params={"js_render": "true"})
```

Each CSV row becomes a task. Zenrows reads only the `url` and optional `external_id` columns, and applies the job-level `zenrows_params` to all of them. Limits: 50 MB and 100,000 rows. The upload link works for 30 minutes and the uploaded file for 24 hours. A CSV can feed a closed regular job or a scheduled job.

## Listing and pagination

The SDK offers both single-page calls (`list_jobs`, `list_runs`, `list_results`) and lazy iterators that page for you. The iterators yield loaded handles.

```python Python theme={"theme":"css-variables"}
for job in client.iter_jobs(job_type="regular", status="closed"):
    print(job.job_id, job.data.status)

for run in client.iter_runs("01J...", page_size=50):
    print(run.run_id, run.data.status)

for task in client.iter_results("01J...", status="failed"):
    print(task.task_id, task.error.code if task.error else None)
```

## Reruns and retrying failures

Each rerun creates a brand-new run on the same job. A job runs one run at a time: start a rerun only after the previous run reaches a terminal state, or the API answers `409 run_not_terminal`. Pick one of the two calls below.

```python Python theme={"theme":"css-variables"}
# Option 1: re-run every task.
new_run = job.rerun(wait_for_ingest=True)
new_run.wait(timeout=600.0)
```

```python Python theme={"theme":"css-variables"}
# Option 2: re-run only the failures (include_pending=True also picks up tasks that
# never started). Successful results carry over, so you don't pay to scrape them again.
new_run = job.retry_failed(include_pending=True, wait_for_ingest=True)
new_run.wait(timeout=600.0)
```

The API accepts every rerun with `202` and writes the new run's tasks in the background, so results read straight after a rerun can be partial. `wait_for_ingest=True` returns only once they're written. A partial rerun (`retry_failed`, or `rerun(status=...)`) must start within 10 days of when the run you retry started, or the API answers `409 partial_rerun_window_expired`; a full rerun still works. When no task has the statuses you asked for, the API answers `409 no_matching_tasks`.

> [!TIP]
>
> `retry_failed()` is the usual post-run cleanup, a thin shortcut for `rerun(status="failed")`. You can act on an id you kept without a preceding GET: `client.job(job_id).retry_failed()`.

## Scheduled jobs

Build a schedule with the typed builders and submit a scheduled job. Zenrows stores its URLs as a template, and each fire produces a fresh run. A `Rate` fires at least 60 minutes apart, `Calendar` times must be full hours, and an account holds up to 5 scheduled jobs.

```python Python theme={"theme":"css-variables"}
from datetime import datetime
from zenrows.batch import At, Calendar, Rate, Weekly

# Every 6 hours:
client.submit_scheduled(Rate(every=6, unit="hour"),
                        ["https://www.scrapingcourse.com/ecommerce/"])

# 09:00 and 18:00 on Mon/Wed/Fri, Berlin time:
client.submit_scheduled(
    Calendar(times_of_day=["09:00", "18:00"],
             cadence=Weekly(days=["mon", "wed", "fri"]),
             timezone="Europe/Berlin"),
    ["https://www.scrapingcourse.com/ecommerce/"],
)

# One-shot at a wall-clock time. A naive datetime plus an IANA zone
# keeps daylight-saving transitions correct:
client.submit_scheduled(At(datetime(2027, 1, 15, 9, 0), timezone="Europe/Berlin"),
                        ["https://www.scrapingcourse.com/ecommerce/"])
```

A job carries two facets, matching the API's two scopes. `job.schedule.*` controls the recurring schedule, and `job.run.*` controls the **current run**. This disambiguates the API's two distinct "pause" actions:

```python Python theme={"theme":"css-variables"}
job.schedule.pause()                            # stop firing new runs
job.schedule.resume()
job.schedule.update(Rate(every=1, unit="day"))  # replace the schedule

job.run.pause()   # reversibly suspend the current run (a different endpoint)
job.run.resume()
job.run.stop()    # end the current run
```

## Completion notifications: webhooks

Attach a webhook at submit time, and Zenrows sends one event to your HTTPS URL when a run finishes on its own: `run.completed` when it completes, or `run.failed` when an account-level error ends it. Stopping a run yourself sends no event. A delivery that fails or takes longer than 10 seconds is retried with backoff for up to 3 days and then dropped, so the same event can arrive more than once: deduplicate on the `X-ZenRows-Event-Id` header. The body is a `WebhookEvent` (schema `zenrows.webhook/v1`) with the job and run ids, your `external_id` and `metadata`, the run's `stats`, and, on `run.failed`, `failure_reason` and `failure_detail`.

Set `signature: True` to receive HMAC-signed deliveries (see [Signed webhooks](#signed-webhooks-hmac-keys)); it defaults to unsigned. Manage the webhook later with `job.set_webhook(url, signature=...)` / `job.delete_webhook()`, and probe a receiver with `client.test_webhook(...)`.

```python Python theme={"theme":"css-variables"}
job = client.submit_regular(
    ["https://www.scrapingcourse.com/ecommerce/"],
    webhook={"url": "https://your-app.com/webhooks/zenrows", "signature": True},
)
```

> [!TIP]
>
> No webhook receiver? Just poll for completion with `job.run.wait()` or `run.wait()`.

## Bulk export (ZIP)

Bundle an entire run's result bodies into a single server-side zip: `download_all_results` starts the export, waits for it, and streams the file to disk.

```python Python theme={"theme":"css-variables"}
run.download_all_results("results.zip")
```

> [!NOTE]
>
> The server-side zip is capped at **100 GiB** per run. Its download link is valid for 2 hours, and the export for 12 hours. For larger runs, or one file per task, use `download_to_dir`, which fetches bodies client-side with no size limit (slower; tune with `concurrency`). For manual control, `client.start_results_export(...)` and `client.wait_for_export(...)` expose the two steps separately.

## Estimating cost

Estimate credits **before** submitting with `client.estimate_cost`. It takes the same body you would submit and runs client-side with no API call. The tiers are 1 (base), 5 (`js_render`), 10 (`premium_proxy`), 25 (both), and a dynamic 1 to 25 for `mode=auto`.

```python Python theme={"theme":"css-variables"}
est = client.estimate_cost({
    "type": "regular",
    "tasks": [{"url": f"https://www.scrapingcourse.com/ecommerce/page/{i}"} for i in range(1000)],
    "zenrows_params": {"js_render": "true"},
})
print(est)                       # "5000 credits (1000 tasks)"
print(est.min, est.max, est.exact)
print(est.format())              # per-tier breakdown table
```

An estimate assumes each URL succeeds once. `min == max` (`est.exact`) unless the job uses `mode=auto`, which yields a per-task range.

## Checking actual cost

Once a run finishes, read the **credits it consumed** from `run.stats.spend` (`{credits, cost}`).

```python Python theme={"theme":"css-variables"}
run = job.run.wait(timeout=600.0)
spend = run.stats.spend
if spend:
    print(f"this run used {spend.credits} credits (cost {spend.cost})")
```

Batch bills like Fetch. Successful tasks are billed, and so are tasks that fail because the target answered `404` or `410`. Other failed tasks aren't billed. Result rows don't carry a spend value, so use the run-level rollup, or a task's attempt history (`run.task_history(task_id)`) for its per-attempt spend. `stats.spend` is indicative; your Zenrows account statement is the authoritative record for billing.

## Signed webhooks: HMAC keys

Signed deliveries (see [Webhooks](#completion-notifications-webhooks)) need an active HMAC key for your organization. Rotation uses two slots, active plus candidate, so you can roll keys without downtime.

```python Python theme={"theme":"css-variables"}
created = client.rotate_hmac_key()   # the first call creates the active key
print(created.kid, created.secret)   # Zenrows returns the secret ONCE, so store it now

# Phased rotation:
client.rotate_hmac_key()             # stage a candidate (start accepting both)
client.finalize_hmac_key()           # promote the candidate to active, retire the old one
# client.cancel_hmac_rotation()      # or discard the candidate
client.list_hmac_keys()              # metadata only; never returns secrets
```

Each signed delivery carries `X-Signature: t=<unix>,v1=<hex>,kid=<key id>`. `verify_webhook_signature(raw_body, header, secrets)` checks it: it picks the secret by `kid`, decodes it from base64 (the `secret` is base64-encoded), recomputes `HMAC-SHA256(key, f"{t}.{raw_body}")`, compares in constant time, and rejects a `t` more than 300 seconds from now (`tolerance_seconds` changes that). It returns `False` instead of raising. Pass the raw body, before any JSON parsing, and keep both keys in `secrets` during a rotation.

```python Python theme={"theme":"css-variables"}
from flask import Flask, request
from zenrows.batch import (
    WEBHOOK_EVENT_ID_HEADER,
    WEBHOOK_SIGNATURE_HEADER,
    verify_webhook_signature,
)

app = Flask(__name__)
# The kid and base64 secret that rotate_hmac_key() returned, from your secret store.
SECRETS = {"YOUR_KEY_ID": "YOUR_BASE64_SECRET"}

@app.post("/webhooks/zenrows")
def zenrows_webhook():
    raw = request.get_data()
    if not verify_webhook_signature(raw, request.headers.get(WEBHOOK_SIGNATURE_HEADER, ""), SECRETS):
        return "", 401
    event = request.get_json()
    print(request.headers.get(WEBHOOK_EVENT_ID_HEADER), event["event_type"], event["stats"])
    return "", 204  # answer fast, then process the event
```

## Error handling

Every non-2xx raises `BatchAPIError`. Its `code` attribute carries the stable code from the problem body, `status_code` the HTTP status, and `detail` the explanation. `BatchErrorCode` names every code the API sends; its members are strings, so they compare equal to `code`. The [error code reference](https://docs.zenrows.com/api-error-codes.md#batch) explains each one.

```python Python theme={"theme":"css-variables"}
from zenrows.batch import BatchAPIError, BatchErrorCode

try:
    client.submit_regular(["https://www.scrapingcourse.com/ecommerce/"])
except BatchAPIError as e:
    if e.code in (BatchErrorCode.PAYMENT_REQUIRED, BatchErrorCode.SUBSCRIPTION_INACTIVE):
        print("Check Billing in your dashboard:", e.detail)
    elif e.code == BatchErrorCode.API_KEY_CAP_REACHED:
        print(e.detail)   # names the cap and when it resets
    elif e.code == BatchErrorCode.QUOTA_EXCEEDED:
        print("Wait for a run to end, or delete a job:", e.detail)
    else:
        raise
```

`quota_exceeded` is one of three account limits: 3 active jobs (shared with Crawl), 5 scheduled jobs, or 100,000 tasks in one open job. It clears only when a job's run ends (for an open job, once it's also closed) or a job is deleted, so the client doesn't retry it. Other transient failures (network errors, `502`, `503`, `504`, and any other `429`) are retried with backoff on reads and on the calls that carry an `idempotency_key`.

Task-level failures don't raise exceptions. They appear as `failed` results with an `error` object you can read through `task.error.code` and `task.error.detail`.

## Idempotency

Pass `idempotency_key` to `submit_*`, to `rerun` / `retry_failed`, and to `add_tasks` to make a retry safe. The API keeps each key for 24 hours:

* The same key with the same request gets the original response replayed, so a dropped connection never creates a second job, run or set of tasks.
* The same key with a different request, or a repeat while the first call is still being processed, raises a `409` with code `idempotency_key_conflict`.
* A submit or rerun the API refused, for example with `429 quota_exceeded`, doesn't use up its key: fix the cause and retry with the same key. An add-tasks call keeps its key, refused or not: a retry with the same body is checked again, and a corrected body needs a new key.
* A key can be up to 128 characters.

```python Python theme={"theme":"css-variables"}
job = client.submit_regular(
    ["https://www.scrapingcourse.com/ecommerce/"],
    idempotency_key="nightly-2027-01-15",
)
```

The full Batch surface (jobs, runs, tasks, results, content, history, file inputs, and HMAC keys) is reachable through methods on `ZenRowsBatchClient`. See the [SDK repository](https://github.com/ZenRows/zenrows-python-sdk) or run `help(ZenRowsBatchClient)`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.