Introduction to the Batch Scraper API

The ZenRows® Batch Scraper API is a managed service that scrapes large lists of URLs for you. You hand it the URLs and the scraping configuration you want, and the API runs the work reliably: it queues each request, retries transient failures, stores every result, and notifies you when the batch finishes. Whether you scrape a few hundred pages or a few hundred thousand, the Batch Scraper API gives you one consistent way to submit work, track progress, and collect results, without operating the orchestration yourself.

​

Use It from Any Language

Every Batch Scraper API feature is available over a plain HTTPS REST API, so you can integrate from any language with nothing more than an HTTP client. The developer guide is a complete, dependency-free fetch walkthrough whose patterns apply equally to Python, Go, PHP, Java, or anything else.

​

Why Use It

Scraping a single page is easy. Scraping tens of thousands of pages reliably is not: you have to spread the load so you don’t overwhelm target sites, retry the requests that fail for transient reasons, store and serve results durably, and know when everything finished and what didn’t work. The Batch Scraper API is the managed layer for exactly this. You integrate once, submit jobs, and it handles the operational concerns for you:

• Scale without orchestration. Submit up to 100,000 URLs in a single job and let the API pace the work.
• Reliable by default. The API retries transient failures automatically, so a page that’s briefly unavailable still gets fetched.
• Results you can trust. The API stores every successful page and serves it back through time-limited download links.
• Know when it’s done. Receive a webhook on completion, or poll for status, whichever fits your pipeline.

​

How It Works

1. Submit a job: a list of URLs plus the scraping options to apply to them.
2. The API queues and processes each URL, calling the ZenRows scraping engine and retrying transient failures automatically.
3. The API stores results as they complete, so you can watch progress at any time.
4. Collect the results when the batch finishes. Fetch them directly, or receive a webhook.

​

Core Concepts

Three terms map cleanly onto how teams work: A job carries a snapshot of its latest run, so a single request tells you where things stand without extra lookups.

​

Key Features

​

Flexible submission

Submit up to 100,000 URLs in a single job. An open job accepts URLs in batches over time (up to 10,000 per add-tasks call) before you close it. For very large lists, upload a CSV (up to 50 MB / 100,000 rows) and reference it instead of inlining URLs. Use an optional idempotency key to guard against accidental duplicate submissions: each key can be used once, and reusing it returns a conflict rather than creating a second job.

​

Full control over scraping

Configure each job, and override per URL, with the scraping options you need:

• JavaScript rendering for dynamic pages.
• Premium proxies and geo-targeting by country.
• Output formats: raw HTML, parsed JSON, Markdown, plain text, or PDF.
• Structured extraction with CSS selectors or automatic parsing, plus custom headers, waits, and more.

​

Results, your way

• Each successful task yields a presigned download URL (a temporary, signed link) valid for 2 hours, or you can stream the content directly.
• Page through results efficiently and filter by status: successful, failed, or in-progress.
• Correlate every result back to your own records with a caller-supplied external_id.
• Archive export: download an entire run’s results as a single ZIP archive in one request.
• Each task ends as successful or failed, and a failed task carries a structured error with a code and detail. During the beta, these codes don’t yet reliably distinguish blocked, not-found, and timeout, so treat them as a hint rather than a definitive reason.

​

Completion notifications

• Webhooks: the API sends an HTTPS callback when a run completes, optionally HMAC-signed so your receiver can verify it genuinely came from ZenRows.
• Polling: status and progress counters are always available on demand.

​

Reruns and recovery

• Full rerun: replay an entire job as a fresh run.
• Retry failed only: re-execute just the tasks that failed (and optionally those that never started). Successful results carry over, so you only pay to re-scrape what failed.

​

Scheduled jobs

Run recurring work on a one-shot, interval (every N minutes, hours, or days), or calendar (specific times on specific days) schedule. Schedules are timezone-aware using IANA names, so daylight-saving transitions stay correct. Pause and resume any time without losing the schedule.

​

Cost transparency

Pricing is per successful request, and the rate card is small and stable, so you can estimate the cost before you submit with no extra round-trip and show your users an accurate usage range up front. Once a job finishes, you can also see the total usage it actually consumed, which makes it easy to reconcile the estimate against realized spend.

​

Reliability

• Pages still come back. The API retries transient failures for you, so a page that’s temporarily unavailable still has a strong chance of being fetched.
• Easy on your targets. The API paces requests so a large job stays polite to the sites you scrape.
• Durable results. The API stores every successful page and serves it back through secure, time-limited download links.
• Built for big batches. Submit tens of thousands of URLs and collect the results without managing any of the moving parts.

​

Security

• API-key authentication on every request. Keys travel in a header, never in the URL.
• HTTPS everywhere, including a requirement that webhook receivers use HTTPS.
• Optional HMAC-signed webhooks with per-organization keys and phased key rotation, so you can verify that a callback genuinely came from ZenRows.

​

Limits

Submitting a job while 3 are already active returns 429 quota_exceeded. Wait for a job to finish, or contact ZenRows if you need a higher concurrency limit.

​

Pricing

Pricing is identical to the ZenRows Universal Scraper API. You’re billed in usage, per successful request, and the API never charges for failed requests. The usage cost depends on the scraping tier each URL uses: Optimize cost with Adaptive Stealth Mode. The Batch Scraper API supports Adaptive Stealth Mode (mode=auto): it starts each request with the cheapest viable setup and escalates to JavaScript rendering or premium proxies only when a site actually needs them. You pay for the lightest configuration that succeeds rather than over-provisioning every URL. Because the rate card is small and stable, you can present an accurate cost range to your own users before a job is submitted. See the Estimating Cost section in the developer guide.

​

Next Steps