> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Python SDK
> Submit batch scraping jobs, track them, and collect results with the official ZenRows Python SDK. Covers every Batch Scraper API feature with runnable examples.
The Batch Scraper API is in **public beta**, available to every ZenRows account. You need an active ZenRows API key. The `ZenRowsBatchClient` shown here is separate from the SDK's synchronous scraping client, `ZenRowsClient`.
Use this guide to submit batch scraping jobs, track them, and collect results with the official ZenRows Python SDK. It walks through every Batch Scraper API feature, and each section runs on its own.
Prefer raw HTTP, or work in another language? The [REST API guide](/batch-scraper-api/developer-guide-restapi) covers the same use cases against the plain REST API. For the complete method reference, see the [SDK repository on GitHub](https://github.com/ZenRows/zenrows-python-sdk).
## Install and Authenticate
While the batch client is being published to PyPI, install the SDK from the beta branch:
```bash theme={null}
pip install "git+https://github.com/ZenRows/zenrows-python-sdk.git@feature/batch-api/rc"
```
This git install is temporary. The batch client is not on PyPI yet; once it is published, `pip install zenrows` will include it.
```python Python theme={null}
import os
from zenrows import ZenRowsBatchClient
client = ZenRowsBatchClient(api_key=os.environ["ZENROWS_API_KEY"])
```
The client also works as a context manager (`with ZenRowsBatchClient(...) as client: ...`) when you want connections to close automatically. By default it targets the production Batch Scraper API; pass `base_url=...` (or set `ZENROWS_BATCH_BASE_URL`) to point at a different environment.
## Quickstart: Submit, Wait, Get Results
This is the standard end-to-end flow. `submit_regular` returns a `JobRef` (no GET yet), `job.run.wait()` blocks until the current run reaches a **terminal** state and returns a `RunHandle`, and `run.results(...)` iterates the per-task results.
```python Python theme={null}
job = client.submit_regular(
[
"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie",
"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket",
"https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee",
],
zenrows_params={"js_render": "true", "premium_proxy": "true"},
)
run = job.run.wait(timeout=600.0) # blocks until the current run is terminal
s = run.stats
print(f"{s.successful}/{s.total} successful, {s.failed} failed")
for task in run.results(status="successful"):
print(task.external_id or task.task_id, "->", task.result_url)
```
`job.run.wait()` returns a `RunHandle` whose `.data` snapshot is ready, so `run.results(...)` and `run.download_to_dir(...)` work without re-passing IDs.
Pass `progress=True` to the waiter for a live progress bar on long batches. For very large submissions that return `202 Accepted`, pass `wait_for_ingest=True` to `submit_regular` so results pages are complete before you read them.
## Configuring the Scrape
`zenrows_params` sets scraping options for every URL in the job. Values can be strings, booleans, or integers; the SDK converts them to strings before sending.
```python Python theme={null}
job = client.submit_regular(
["https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie"],
zenrows_params={
"js_render": "true", # render JavaScript in a headless browser
"premium_proxy": "true", # route through premium residential proxies
"proxy_country": "us", # geo-target the request
"wait_for": ".price", # wait for this CSS selector before capturing
"response_type": "markdown", # html | markdown | plaintext | pdf
},
)
```
ZenRows derives the job's **output format** from these options: `response_type: markdown` produces Markdown; `autoparse`, `json_response`, or a CSS extractor produce JSON; otherwise the result is HTML. ZenRows records this format on each result. Other common keys include `js_instructions`, `wait`, `block_resources`, `json_response`, `css_extractor`, `autoparse`, `outputs`, `original_status`, `allowed_status_codes`, and `session_id`. Custom request headers (`custom_headers`) are not supported yet.
## Per-URL Options and Correlating Results
To set a per-URL `external_id`, `metadata`, or `zenrows_params`, pass dictionaries instead of plain strings. Per-URL `zenrows_params` override the job-level ones when a key appears in both.
```python Python theme={null}
job = client.submit_regular(
[
{"url": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short",
"external_id": "order-42",
"zenrows_params": {"proxy_country": "de"}},
{"url": "https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee",
"external_id": "order-43",
"metadata": {"sku": "ABC-1"}},
],
zenrows_params={"js_render": "true"}, # applies to both unless overridden
)
```
ZenRows returns your `external_id` unchanged on every result, so you can match results back to your own records without tracking the API's task IDs. It must match `^[A-Za-z0-9._-]+$` and be 128 characters or fewer.
## Downloading Content
`results()` gives you metadata plus a presigned `result_url` per task; the SDK downloads bodies straight from that URL with no extra API round-trip. Four options, on the current run (`job.run.*`) or any specific run (`client.get_run(job_id, run_id=...)`):
| You want | Use | Where it lands |
| ------------------------------- | ----------------------------------------------------------------------------- | ------------------------ |
| Every body, one file per task | `run.download_to_dir(dir)` | files on disk |
| Every body, kept in memory | `run.download_to_memory()` | `list[DownloadedResult]` |
| The whole run as one artifact | `run.download_all_results("out.zip")` | one `.zip` |
| One task's body (inside a loop) | `run.download_task_to_file(task, path)` / `run.download_task_to_memory(task)` | file / `bytes` |
```python Python theme={null}
# Bulk to disk, named by external_id, 8 downloads in parallel:
n = run.download_to_dir(
"./out",
status="successful", # default; pass status=None for everything
use_external_id=True, # name files . instead of .
concurrency=8,
progress=True,
max_files=20_000, # safety caps that raise DownloadLimitExceeded
max_bytes_per_file=10 * 1024 * 1024,
)
print(f"downloaded {n} files")
# One task at a time, when you want a Python-side filter:
for task in run.results(status="successful"):
body = run.download_task_to_memory(task) # bytes; or download_task_to_file(task, path)
```
Each `result_url` is a plain presigned URL (no auth header) valid for a short window, so fetch it promptly or re-list the results to get a fresh one.
## Open Jobs: Streaming URLs in Batches
When you don't have every URL up front, open a job, add tasks in batches, and close it. The run stays active and completes once it finishes the final batch.
```python Python theme={null}
job = client.submit_open(zenrows_params={"js_render": "true"})
job.add_tasks({"tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/page/1"}]})
job.add_tasks({"tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/page/2"}],
"last_batch": True}) # closes the job
run = job.run.wait(timeout=900.0)
```
Setting `last_batch=True` (or calling `job.close()`) locks the job, and the API rejects further adds. File-input (CSV) upload is not supported for open jobs; use a closed job for CSV.
## Large Inputs: CSV Upload
For lists too large to inline, upload a CSV once and reference it. `upload_csv` allocates the slot, PUTs your file to the presigned URL, and returns a `file_input_id` that `submit_regular(file_input_id=...)` consumes.
```python Python theme={null}
file_input_id = client.upload_csv(
"urls.csv",
fields={"url": "URL", "external_id": "Customer Ref"}, # column name (header=True) or 0-based index
header=True,
)
job = client.submit_regular(file_input_id=file_input_id, zenrows_params={"js_render": "true"})
```
Each CSV row becomes a task. ZenRows reads only the `url` and optional `external_id` columns, and applies the job-level `zenrows_params` to all of them. Limits: 50 MB / 100,000 rows.
## Listing and Pagination
The SDK offers both single-page calls (`list_jobs`, `list_runs`, `list_results`) and lazy iterators that page for you. The iterators yield loaded handles.
```python Python theme={null}
for job in client.iter_jobs(job_type="regular", status="closed"):
print(job.job_id, job.data.status)
for run in client.iter_runs("01J...", page_size=50):
print(run.run_id, run.data.status)
for task in client.iter_results("01J...", status="failed"):
print(task.task_id, task.error.code if task.error else None)
```
## Reruns and Retrying Failures
Each rerun creates a brand-new run on the same job.
```python Python theme={null}
# Re-run every task:
new_run = job.rerun()
# Re-run only the failures (optionally also tasks that never started).
# Successful results carry over, so you don't pay to scrape them again:
new_run = job.retry_failed(include_pending=False)
new_run.wait(timeout=600.0)
```
`retry_failed()` is the usual post-run cleanup, a thin shortcut for `rerun(status="failed")`. You can act on an id you kept without a preceding GET: `client.job(job_id).retry_failed()`.
## Scheduled Jobs
Build a schedule with the typed builders and submit a scheduled job. ZenRows stores its URLs as a template, and each fire produces a fresh run.
```python Python theme={null}
from datetime import datetime
from zenrows.batch import At, Calendar, Rate, Weekly
# Every 6 hours:
client.submit_scheduled(Rate(every=6, unit="hour"),
["https://www.scrapingcourse.com/ecommerce/"])
# 09:00 and 18:00 on Mon/Wed/Fri, Berlin time:
client.submit_scheduled(
Calendar(times_of_day=["09:00", "18:00"],
cadence=Weekly(days=["mon", "wed", "fri"]),
timezone="Europe/Berlin"),
["https://www.scrapingcourse.com/ecommerce/"],
)
# One-shot at a wall-clock time. A naive datetime plus an IANA zone
# keeps daylight-saving transitions correct:
client.submit_scheduled(At(datetime(2026, 9, 1, 9, 0), timezone="Europe/Berlin"),
["https://www.scrapingcourse.com/ecommerce/"])
```
A job carries two facets, matching the API's two scopes. `job.schedule.*` controls the recurring schedule, and `job.run.*` controls the **current run**. This disambiguates the API's two distinct "pause" actions:
```python Python theme={null}
job.schedule.pause() # stop firing new runs
job.schedule.resume()
job.schedule.update(Rate(every=1, unit="day")) # replace the schedule
job.run.pause() # reversibly suspend the current run (a different endpoint)
job.run.resume()
job.run.stop() # end the current run
```
## Completion Notifications: Webhooks
Attach a webhook at submit time, and ZenRows sends a `run.completed` event to your HTTPS URL when a run finishes. Set `signature: True` to receive HMAC-signed deliveries (see [Signed Webhooks](#signed-webhooks-hmac-keys)); it defaults to unsigned. You can also manage the webhook later with `client.put_job_webhook(...)` / `client.delete_job_webhook(...)`, and probe a receiver with `client.test_webhook(...)`.
```python Python theme={null}
job = client.submit_regular(
["https://www.scrapingcourse.com/ecommerce/"],
webhook={"url": "https://your-app.com/webhooks/zenrows", "signature": True},
)
```
ZenRows delivers each event **at least once**, so your receiver must deduplicate on the `X-ZenRows-Event-Id` header.
No webhook receiver? Just poll for completion with `job.run.wait()` or `run.wait()`.
## Bulk Export (ZIP)
Bundle an entire run's result bodies into a single server-side zip: `download_all_results` starts the export, waits for it, and streams the file to disk.
```python Python theme={null}
run.download_all_results("results.zip")
```
The server-side zip is capped at **1 GiB** per run. For larger runs, or one file per task, use `download_to_dir`, which fetches bodies client-side with no size limit (slower; tune with `concurrency`). For manual control, `client.start_results_export(...)` and `client.wait_for_export(...)` expose the two steps separately.
## Estimating Cost
Estimate usage **before** submitting with `client.estimate_cost`. It takes the same body you would submit and runs client-side with no API call. The tiers are 1 (base), 5 (`js_render`), 10 (`premium_proxy`), 25 (both), and a dynamic 1 to 25 for `mode=auto`.
```python Python theme={null}
est = client.estimate_cost({
"type": "regular",
"tasks": [{"url": f"https://www.scrapingcourse.com/ecommerce/page/{i}"} for i in range(1000)],
"zenrows_params": {"js_render": "true"},
})
print(est) # "5000 credits (1000 tasks)"
print(est.min, est.max, est.exact)
print(est.format()) # per-tier breakdown table
```
An estimate assumes each URL succeeds once. `min == max` (`est.exact`) unless the job uses `mode=auto`, which yields a per-task range.
## Checking Actual Cost
Once a run finishes, read the **actual usage it consumed** from `run.stats.spend` (`{credits, cost}`).
```python Python theme={null}
run = job.run.wait(timeout=600.0)
spend = run.stats.spend
if spend:
print(f"this run used {spend.credits} usage (cost {spend.cost})")
```
Only successful requests are charged, so realized spend is at most the estimate. Per-task `task.spend` is indicative and often absent on successful rows, so rely on the run-level rollup for actuals. Your ZenRows account statement remains the authoritative record for billing.
## Signed Webhooks: HMAC Keys
Signed deliveries (see [Webhooks](#completion-notifications-webhooks)) need an active HMAC key for your organization. Rotation uses two slots, active plus candidate, so you can roll keys without downtime.
```python Python theme={null}
created = client.rotate_hmac_key() # the first call creates the active key
print(created.kid, created.secret) # ZenRows returns the secret ONCE, so store it now
# Phased rotation:
client.rotate_hmac_key() # stage a candidate (start accepting both)
client.finalize_hmac_key() # promote the candidate to active, retire the old one
# client.cancel_hmac_rotation() # or discard the candidate
client.list_hmac_keys() # metadata only; never returns secrets
```
Your receiver verifies the `X-Signature: t=,v1=,kid=` header by recomputing `HMAC-SHA256(secret, f"{t}.{raw_body}")` and comparing the result to `v1`, selecting the secret by `kid`.
## Error Handling
Every non-2xx raises `BatchAPIError`. Its `code` attribute carries the stable code from the RFC 7807 body, and `status_code` the HTTP status.
```python Python theme={null}
from zenrows.batch import BatchAPIError
try:
client.get_job("does-not-exist")
except BatchAPIError as e:
print(e.status_code, e.code) # e.g. 404 not_found
raise
```
Task-level failures don't raise exceptions. They appear as `failed` results with an `error` object you can read through `task.error.code` and `task.error.detail`.
## Idempotency
Pass `idempotency_key` to `submit_*` and to `rerun` / `retry_failed` to guard against accidentally creating duplicate jobs. Each key can be used once: a later call that reuses the same key raises a `409` conflict, whether or not the body is identical, and the API does not store or replay the original result. An idempotency key is therefore not a safe-retry token, so if a call fails midway, check whether the job was created before retrying with a new key.
```python Python theme={null}
job = client.submit_regular(
["https://www.scrapingcourse.com/ecommerce/"],
idempotency_key="nightly-2026-06-08",
)
```
The full Batch surface (jobs, runs, tasks, results, content, history, file inputs, and HMAC keys) is reachable through methods on `ZenRowsBatchClient`. See the [SDK repository](https://github.com/ZenRows/zenrows-python-sdk) or run `help(ZenRowsBatchClient)`.