> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Scrape a list of URLs > Scrape thousands of URLs as one Zenrows Batch job, from code, the CLI or your AI agent. Have a list of URLs? Submit them to Batch as one job. Zenrows runs them in parallel with Adaptive Stealth Mode, tracks every task, and stores each page for you to download. You get no concurrency to tune, no retry loop to write, and nothing to keep running while it works. For a handful of URLs, a concurrent loop over Fetch is just as good, and the last section shows it. **Uses:** [Batch](/batch/introduction) (`POST /jobs`, `GET /jobs/{id}/results`) with [`zenrows_params`](/batch/developer-guide-restapi#configure-the-scrape-and-per-url-options), and [Fetch](/fetch/api-reference) for small lists ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use Zenrows to scrape these URLs in one batch, as markdown, and wait for the job to finish: https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/ https://www.scrapingcourse.com/ecommerce/product/aeon-capri/ Then give me each product's name and price. ``` For a long list, point the agent at a file instead: "Use Zenrows to scrape every URL in urls.txt in one batch." It runs the same way for 10 URLs or 100,000. ## Submit the list as one Batch job Send every URL in `tasks`, with the scrape settings for all of them in `zenrows_params`. Give each task an `external_id`: Zenrows returns it on every result, so you can match results to your own records. Then wait for the run to finish and download each result from its `result_url`. ```python Python (Requests) theme={"dark"} import os import time import requests BASE = "https://async.api.zenrows.com/v1" headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"} urls = [ "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", ] # 1. Submit every URL as one job job = requests.post( f"{BASE}/jobs", headers=headers, json={ "type": "regular", "status": "closed", "zenrows_params": {"mode": "auto", "response_type": "markdown"}, "tasks": [{"url": url, "external_id": url.rstrip("/").split("/")[-1]} for url in urls], }, timeout=60, ).json() job_id = job["job_id"] print("Submitted", job_id) # 2. Wait for the run to finish while True: run = requests.get(f"{BASE}/jobs/{job_id}", headers=headers, timeout=30).json()["latest_run"] if run["status"] in {"completed", "failed", "stopped"}: break time.sleep(5) stats = run["stats"] print(run["status"], f"{stats['successful']}/{stats['total']} successful") # 3. Download each result body os.makedirs("pages", exist_ok=True) cursor = None while True: page = requests.get( f"{BASE}/jobs/{job_id}/results", headers=headers, params={"status": "successful", "cursor": cursor}, timeout=30, ).json() for row in page["results"]: body = requests.get(row["result_url"], timeout=60).content # presigned link, no key needed with open(f"pages/{row['external_id']}.md", "wb") as f: f.write(body) print("Saved", row["external_id"], len(body), "bytes") cursor = page.get("next_cursor") if not cursor: break ``` ```javascript Node.js (SDK) theme={"dark"} // npm install zenrows import { ZenRowsBatchClient } from "zenrows"; const client = new ZenRowsBatchClient("YOUR_ZENROWS_API_KEY"); // submitRegular(urls, fileInputId, options) const job = await client.submitRegular( [ { url: "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", external_id: "abominable-hoodie" }, { url: "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", external_id: "adrienne-trek-jacket" }, { url: "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", external_id: "aeon-capri" }, ], undefined, { zenrowsParams: { mode: "auto", response_type: "markdown" } }, ); const run = await job.run.wait(); const { successful, total } = run.stats; console.log(`${successful}/${total} pages scraped`); for await (const row of run.results({ status: "successful" })) { const body = await run.downloadTaskToMemory(row); console.log(row.external_id, body.length, "bytes"); } ``` ```bash cURL theme={"dark"} # 1. Submit the job curl -X POST "https://async.api.zenrows.com/v1/jobs" \ -H "X-API-Key: YOUR_ZENROWS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "type": "regular", "status": "closed", "zenrows_params": { "mode": "auto", "response_type": "markdown" }, "tasks": [ { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "external_id": "abominable-hoodie" }, { "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "external_id": "adrienne-trek-jacket" } ] }' # 2. Check progress: latest_run.status and latest_run.stats curl "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID" -H "X-API-Key: YOUR_ZENROWS_API_KEY" # 3. List the results; download each result_url curl "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID/results?status=successful" -H "X-API-Key: YOUR_ZENROWS_API_KEY" ``` ```bash CLI theme={"dark"} # jobs.jsonl holds one JSON object per line: # {"url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "external_id": "abominable-hoodie"} zenrows batch estimate jobs.jsonl zenrows batch create jobs.jsonl --output markdown --follow zenrows batch results YOUR_JOB_ID --download ./pages ``` The Node.js tab uses the Zenrows SDK, which submits, waits and pages through results for you; the Python tab shows the same flow over the raw REST API. Every task runs with Adaptive Stealth Mode (`mode=auto`), which picks the cheapest configuration that works for each page. The CLI and MCP send `mode=auto` by default. Any Fetch parameter fits in `zenrows_params`, and a task's own `zenrows_params` override the job's. ## Example output The Python run: ```text theme={"dark"} Submitted 01M4Y9YHTERAYFSFPWPK31HHHX completed 3/3 successful Saved adrienne-trek-jacket 4712 bytes Saved aeon-capri 5437 bytes Saved abominable-hoodie 4349 bytes ``` One row of `GET /jobs/{job_id}/results`, with the presigned link shortened: ```json theme={"dark"} { "task_id": "01M4Y9YHTEEAVY4WFNMMHHPP6Q", "external_id": "adrienne-trek-jacket", "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "status": "successful", "type": "markdown", "result_url": "https://conveyor-prod-results-....s3.us-east-1.amazonaws.com/..." } ``` The CLI run: ```text theme={"dark"} ✓ Spec is valid. ✓ Submitted job 01M4Y9ZQPXBAWMCYYQ06MY1JGB · status: completed 3/3 completed · 3 successful · 0 failed ✓ Downloaded 3/3 result(s) → ./pages ``` ## Use the Python SDK `ZenRowsBatchClient` submits the job, waits for it and downloads every result in a few lines: ```python Python (SDK) theme={"dark"} # pip install zenrows from zenrows import ZenRowsBatchClient client = ZenRowsBatchClient(api_key="YOUR_ZENROWS_API_KEY") job = client.submit_regular( [ "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", ], zenrows_params={"mode": "auto", "response_type": "markdown"}, ) run = job.run.wait(timeout=900) print(f"{run.stats.successful}/{run.stats.total} pages scraped") saved = run.download_to_dir("./pages") print(f"Saved {saved} files to ./pages") ``` ```text Output theme={"dark"} 3/3 pages scraped Saved 3 files to ./pages ``` Pass dictionaries such as `{"url": ..., "external_id": "abominable-hoodie"}` to set per-task IDs, and `use_external_id=True` on `download_to_dir` to name files by them. More in the [Python](/batch/sdk/python) and [Node.js](/batch/sdk/nodejs) SDK guides. ## Small lists: send requests concurrently For a few dozen URLs that you want back in seconds, call Fetch directly and let the SDK run requests in parallel. Set `concurrency` to your plan's concurrent request count, which every response reports in the `Concurrency-Limit` header. ```python Python (SDK) theme={"dark"} # pip install zenrows import asyncio from zenrows import ZenRowsClient client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1) urls = [ "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", ] async def main(): responses = await asyncio.gather( *[client.fetch_async(url, params={"mode": "auto"}, timeout=180) for url in urls], return_exceptions=True, ) for url, response in zip(urls, responses): if isinstance(response, Exception): print("ERROR", url, response) else: print(response.status_code, url, len(response.text), "bytes") asyncio.run(main()) ``` ```javascript Node.js (SDK) theme={"dark"} // npm install zenrows import { ZenRows } from "zenrows"; const client = new ZenRows("YOUR_ZENROWS_API_KEY", { concurrency: 5, retries: 1 }); const urls = [ "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", ]; const results = await Promise.allSettled(urls.map((url) => client.fetch(url, { mode: "auto" }))); for (const [i, result] of results.entries()) { if (result.status === "rejected") { console.log("ERROR", urls[i], result.reason); } else { const html = await result.value.text(); console.log(result.value.status, urls[i], html.length, "bytes"); } } ``` ```text Output theme={"dark"} 200 https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ 120588 bytes 200 https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/ 119577 bytes 200 https://www.scrapingcourse.com/ecommerce/product/aeon-capri/ 91543 bytes ``` Both SDKs queue the requests and keep `concurrency` of them in flight. Move to Batch once the list grows, or when you'd rather not keep a process running while it works. More in [Run many requests concurrently](/academy/fetch/concurrent-requests). ## Customize it | Change | How | | - | - | | Structured JSON instead of pages | `"extract": "auto"` in `zenrows_params`, see [Get structured JSON from any page](/academy/extract/auto-extraction) | | Your own fields from every page | `"css_extractor"` in `zenrows_params`, see [Extract data with CSS selectors](/academy/how-to-extract-data) | | URLs from a CSV file | Upload it with `POST /job_inputs`, see [Start a Batch job from a CSV file](/academy/batch/csv-input) | | URLs that arrive over time | Open the job with `"status": "open"` and add tasks, see [Add URLs to a job while it runs](/academy/batch/open-jobs) | | A notification when it finishes | Add a `webhook`, see [Get notified when a Batch job finishes](/academy/batch/webhooks) | | The same list every day | Make it a scheduled job, see [Run a Batch job on a schedule](/academy/batch/scheduled-jobs) | | Know the cost first | `zenrows batch estimate jobs.jsonl`, or `estimate_cost` in the SDKs | ## Troubleshooting * **Some tasks failed**: read them with `status=failed` on the results endpoint; each row has an `error` with a `code`. Rerun just those with `POST /jobs/{job_id}/rerun?status=failed`, `retry_failed()` in Python, or `zenrows batch retry-failed `. Successful results carry over. See [Retry the tasks that failed](/academy/batch/retry-failed-tasks). * **A `result_url` returns `403`**: the presigned link is stale. List the results again for a fresh link. * **The run ends `failed` with a `failure_reason`**: the account or key needs credits or a raised [credit cap](/first-steps/api-key-credit-caps). Fix it, then rerun the tasks that didn't complete. * **`429` when submitting**: your account's Crawl and Batch jobs are all running. Wait for one to finish, then submit. * **Concurrent requests return `429`**: lower `concurrency` to the `Concurrency-Limit` header value, or move the list to Batch. ## Related recipes Receive a signed webhook instead of polling. Scrape the same list hourly, daily or weekly. Build the URL list with Crawl first. Handle errors when you call Fetch directly. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.