Scrape a list of URLs

Have a list of URLs? Submit them to Batch as one job. Zenrows runs them in parallel with Adaptive Stealth Mode, tracks every task, and stores each page for you to download. You get no concurrency to tune, no retry loop to write, and nothing to keep running while it works. For a handful of URLs, a concurrent loop over Fetch is just as good, and the last section shows it. Uses: Batch (POST /jobs, GET /jobs/{id}/results) with zenrows_params, and Fetch for small lists

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to scrape these URLs in one batch, as markdown, and wait for the job to finish:
https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/
https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/
https://www.scrapingcourse.com/ecommerce/product/aeon-capri/
Then give me each product's name and price.

For a long list, point the agent at a file instead: “Use Zenrows to scrape every URL in urls.txt in one batch.” It runs the same way for 10 URLs or 100,000.

Submit the list as one Batch job

Send every URL in tasks, with the scrape settings for all of them in zenrows_params. Give each task an external_id: Zenrows returns it on every result, so you can match results to your own records. Then wait for the run to finish and download each result from its result_url.

import os
import time
import requests

BASE = "https://async.api.zenrows.com/v1"
headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}

urls = [
    "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
    "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
    "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/",
]

# 1. Submit every URL as one job
job = requests.post(
    f"{BASE}/jobs",
    headers=headers,
    json={
        "type": "regular",
        "status": "closed",
        "zenrows_params": {"mode": "auto", "response_type": "markdown"},
        "tasks": [{"url": url, "external_id": url.rstrip("/").split("/")[-1]} for url in urls],
    },
    timeout=60,
).json()
job_id = job["job_id"]
print("Submitted", job_id)

# 2. Wait for the run to finish
while True:
    run = requests.get(f"{BASE}/jobs/{job_id}", headers=headers, timeout=30).json()["latest_run"]
    if run["status"] in {"completed", "failed", "stopped"}:
        break
    time.sleep(5)
stats = run["stats"]
print(run["status"], f"{stats['successful']}/{stats['total']} successful")

# 3. Download each result body
os.makedirs("pages", exist_ok=True)
cursor = None
while True:
    page = requests.get(
        f"{BASE}/jobs/{job_id}/results",
        headers=headers,
        params={"status": "successful", "cursor": cursor},
        timeout=30,
    ).json()
    for row in page["results"]:
        body = requests.get(row["result_url"], timeout=60).content  # presigned link, no key needed
        with open(f"pages/{row['external_id']}.md", "wb") as f:
            f.write(body)
        print("Saved", row["external_id"], len(body), "bytes")
    cursor = page.get("next_cursor")
    if not cursor:
        break

// npm install zenrows
import { ZenRowsBatchClient } from "zenrows";

const client = new ZenRowsBatchClient("YOUR_ZENROWS_API_KEY");

// submitRegular(urls, fileInputId, options)
const job = await client.submitRegular(
  [
    { url: "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", external_id: "abominable-hoodie" },
    { url: "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", external_id: "adrienne-trek-jacket" },
    { url: "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", external_id: "aeon-capri" },
  ],
  undefined,
  { zenrowsParams: { mode: "auto", response_type: "markdown" } },
);

const run = await job.run.wait();
const { successful, total } = run.stats;
console.log(`${successful}/${total} pages scraped`);

for await (const row of run.results({ status: "successful" })) {
  const body = await run.downloadTaskToMemory(row);
  console.log(row.external_id, body.length, "bytes");
}

# 1. Submit the job
curl -X POST "https://async.api.zenrows.com/v1/jobs" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "regular",
    "status": "closed",
    "zenrows_params": { "mode": "auto", "response_type": "markdown" },
    "tasks": [
      { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "external_id": "abominable-hoodie" },
      { "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "external_id": "adrienne-trek-jacket" }
    ]
  }'

# 2. Check progress: latest_run.status and latest_run.stats
curl "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID" -H "X-API-Key: YOUR_ZENROWS_API_KEY"

# 3. List the results; download each result_url
curl "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID/results?status=successful" -H "X-API-Key: YOUR_ZENROWS_API_KEY"

# jobs.jsonl holds one JSON object per line:
# {"url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "external_id": "abominable-hoodie"}
zenrows batch estimate jobs.jsonl
zenrows batch create jobs.jsonl --output markdown --follow
zenrows batch results YOUR_JOB_ID --download ./pages

The Node.js tab uses the Zenrows SDK, which submits, waits and pages through results for you; the Python tab shows the same flow over the raw REST API. Every task runs with Adaptive Stealth Mode (mode=auto), which picks the cheapest configuration that works for each page. The CLI and MCP send mode=auto by default. Any Fetch parameter fits in zenrows_params, and a task’s own zenrows_params override the job’s.

Example output

The Python run:

Submitted 01M4Y9YHTERAYFSFPWPK31HHHX
completed 3/3 successful
Saved adrienne-trek-jacket 4712 bytes
Saved aeon-capri 5437 bytes
Saved abominable-hoodie 4349 bytes

One row of GET /jobs/{job_id}/results, with the presigned link shortened:

{
  "task_id": "01M4Y9YHTEEAVY4WFNMMHHPP6Q",
  "external_id": "adrienne-trek-jacket",
  "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
  "status": "successful",
  "type": "markdown",
  "result_url": "https://conveyor-prod-results-....s3.us-east-1.amazonaws.com/..."
}

The CLI run:

✓ Spec is valid.
✓ Submitted job 01M4Y9ZQPXBAWMCYYQ06MY1JGB · status: completed
  3/3 completed · 3 successful · 0 failed
✓ Downloaded 3/3 result(s) → ./pages

Use the Python SDK

ZenRowsBatchClient submits the job, waits for it and downloads every result in a few lines:

Python (SDK)

# pip install zenrows
from zenrows import ZenRowsBatchClient

client = ZenRowsBatchClient(api_key="YOUR_ZENROWS_API_KEY")

job = client.submit_regular(
    [
        "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
        "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
        "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/",
    ],
    zenrows_params={"mode": "auto", "response_type": "markdown"},
)

run = job.run.wait(timeout=900)
print(f"{run.stats.successful}/{run.stats.total} pages scraped")

saved = run.download_to_dir("./pages")
print(f"Saved {saved} files to ./pages")

Output

3/3 pages scraped
Saved 3 files to ./pages

Pass dictionaries such as {"url": ..., "external_id": "abominable-hoodie"} to set per-task IDs, and use_external_id=True on download_to_dir to name files by them. More in the Python and Node.js SDK guides.

Small lists: send requests concurrently

For a few dozen URLs that you want back in seconds, call Fetch directly and let the SDK run requests in parallel. Set concurrency to your plan’s concurrent request count, which every response reports in the Concurrency-Limit header.

# pip install zenrows
import asyncio
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1)

urls = [
    "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
    "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
    "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/",
]

async def main():
    responses = await asyncio.gather(
        *[client.fetch_async(url, params={"mode": "auto"}, timeout=180) for url in urls],
        return_exceptions=True,
    )
    for url, response in zip(urls, responses):
        if isinstance(response, Exception):
            print("ERROR", url, response)
        else:
            print(response.status_code, url, len(response.text), "bytes")

asyncio.run(main())

// npm install zenrows
import { ZenRows } from "zenrows";

const client = new ZenRows("YOUR_ZENROWS_API_KEY", { concurrency: 5, retries: 1 });

const urls = [
  "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
  "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
  "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/",
];

const results = await Promise.allSettled(urls.map((url) => client.fetch(url, { mode: "auto" })));

for (const [i, result] of results.entries()) {
  if (result.status === "rejected") {
    console.log("ERROR", urls[i], result.reason);
  } else {
    const html = await result.value.text();
    console.log(result.value.status, urls[i], html.length, "bytes");
  }
}

Output

https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ 120588 bytes
https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/ 119577 bytes
https://www.scrapingcourse.com/ecommerce/product/aeon-capri/ 91543 bytes

Both SDKs queue the requests and keep concurrency of them in flight. Move to Batch once the list grows, or when you’d rather not keep a process running while it works. More in Run many requests concurrently.

Customize it

Change	How
Structured JSON instead of pages	`"extract": "auto"` in `zenrows_params`, see Get structured JSON from any page
Your own fields from every page	`"css_extractor"` in `zenrows_params`, see Extract data with CSS selectors
URLs from a CSV file	Upload it with `POST /job_inputs`, see Start a Batch job from a CSV file
URLs that arrive over time	Open the job with `"status": "open"` and add tasks, see Add URLs to a job while it runs
A notification when it finishes	Add a `webhook`, see Get notified when a Batch job finishes
The same list every day	Make it a scheduled job, see Run a Batch job on a schedule
Know the cost first	`zenrows batch estimate jobs.jsonl`, or `estimate_cost` in the SDKs

Troubleshooting

Some tasks failed: read them with status=failed on the results endpoint; each row has an error with a code. Rerun just those with POST /jobs/{job_id}/rerun?status=failed, retry_failed() in Python, or zenrows batch retry-failed <id>. Successful results carry over. See Retry the tasks that failed.
A result_url returns 403: the presigned link is stale. List the results again for a fresh link.
The run ends failed with a failure_reason: the account or key needs credits or a raised credit cap. Fix it, then rerun the tasks that didn’t complete.
429 when submitting: your account’s Crawl and Batch jobs are all running. Wait for one to finish, then submit.
Concurrent requests return 429: lower concurrency to the Concurrency-Limit header value, or move the list to Batch.

Get notified when a Batch job finishes

Receive a signed webhook instead of polling.

Run a Batch job on a schedule

Scrape the same list hourly, daily or weekly.

Map every URL on a website

Build the URL list with Crawl first.

Retry failed requests

Handle errors when you call Fetch directly.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Scrape a list of URLs

Ask your AI agent

Submit the list as one Batch job

Example output

Use the Python SDK

Small lists: send requests concurrently

Customize it

Troubleshooting

Get notified when a Batch job finishes

Run a Batch job on a schedule

Map every URL on a website

Retry failed requests

​Ask your AI agent

​Submit the list as one Batch job

​Example output

​Use the Python SDK

​Small lists: send requests concurrently

​Customize it

​Troubleshooting

​Related recipes

Get notified when a Batch job finishes

Run a Batch job on a schedule

Map every URL on a website

Retry failed requests

Ask your AI agent

Submit the list as one Batch job

Example output

Use the Python SDK

Small lists: send requests concurrently

Customize it

Troubleshooting

Related recipes