Add URLs to a job while it runs

When URLs arrive over time, from a crawler, a message queue or a nightly export that lands in pieces, open a Batch job first and feed it as you go. Zenrows starts scraping the first URLs right away, keeps accepting new ones, and finishes the run once you close the job. One job, one set of results, no matter how many groups you send. Uses: Batch open jobs ("status": "open", POST /jobs/{id}/tasks, last_batch, POST /jobs/{id}/close)

Ask your AI agent

Paste this prompt into a coding agent such as Claude Code or Cursor, with the Zenrows docs connected. The agent looks up the details and writes the calls.

Using the Zenrows docs, change this worker so that every product URL it pulls from the
queue goes into one Zenrows Batch job that stays open while the worker runs. Send the URLs
in groups of 500, scrape them as markdown, and close the job when the queue is empty.
It should handle millions of URLs over a day.

Open a job and stream URLs into it

Create the job with "status": "open". Tasks are optional at this point. The zenrows_params you set here apply to every URL you add later.
Send each group of URLs to POST /jobs/{job_id}/tasks as {"tasks": [...]}. Scraping starts with the first group.
Close the job when your producer is done: either send the last group with "last_batch": true, or call POST /jobs/{job_id}/close. The run then finishes the queued tasks and ends completed.

import time
import requests

BASE = "https://async.api.zenrows.com/v1"
headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}

# 1. Open a job with no tasks yet: the settings apply to every URL you add later
job = requests.post(
    f"{BASE}/jobs",
    headers=headers,
    json={"type": "regular", "status": "open", "zenrows_params": {"mode": "auto", "response_type": "markdown"}},
    timeout=60,
).json()
job_id = job["job_id"]
print("Opened", job_id, job["status"])

# 2. Your producer (a crawler, a queue consumer) sends URLs as it finds them
def producer():
    yield ["https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
           "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/"]
    yield ["https://www.scrapingcourse.com/ecommerce/product/aeon-capri/"]

for urls in producer():
    added = requests.post(
        f"{BASE}/jobs/{job_id}/tasks",
        headers=headers,
        json={"tasks": [{"url": u, "external_id": u.rstrip("/").split("/")[-1]} for u in urls]},
        timeout=60,
    ).json()
    print("Added", added["accepted_tasks"], "tasks, run total:", added["latest_run"]["stats"]["total"])

# 3. Close the job when the producer is done; the run finishes the queued tasks
closed = requests.post(f"{BASE}/jobs/{job_id}/close", headers=headers, timeout=60).json()
print("Job", closed["status"])

while True:
    run = requests.get(f"{BASE}/jobs/{job_id}", headers=headers, timeout=60).json()["latest_run"]
    if run["status"] in {"completed", "failed", "stopped"}:
        break
    time.sleep(5)
print(run["status"], f"{run['stats']['successful']}/{run['stats']['total']} successful")

// npm install zenrows
import { ZenRowsBatchClient } from "zenrows";

const client = new ZenRowsBatchClient("YOUR_ZENROWS_API_KEY");

// 1. Open a job with no tasks yet
const job = await client.submitOpen([], {
  zenrowsParams: { mode: "auto", response_type: "markdown" },
});
console.log("Opened", job.jobId);

// 2. Add URLs as they arrive
await job.addTasks([
  { url: "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", external_id: "abominable-hoodie" },
  { url: "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", external_id: "adrienne-trek-jacket" },
]);

// 3. The final batch closes the job
const last = await job.addTasks(
  [{ url: "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", external_id: "aeon-capri" }],
  { lastBatch: true },
);
console.log("Job", last.job_status);

const run = await job.run.wait();
console.log(run.status, `${run.stats.successful}/${run.stats.total} successful`);

# 1. Open the job and note its job_id
curl -X POST "https://async.api.zenrows.com/v1/jobs" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "type": "regular", "status": "open", "zenrows_params": { "mode": "auto" } }'

# 2. Add a group of URLs (repeat as they arrive)
curl -X POST "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID/tasks" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/" }] }'

# 3. Send the last group and close the job in one call
curl -X POST "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID/tasks" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/" }], "last_batch": true }'

Read the results like any other job, see Scrape a list of URLs or Download all results of a job at once.

Example output

The Python run:

Opened 01M4YHTTN8HZC4HD98YW0EVEV9 open
Added 2 tasks, run total: 2
Added 1 tasks, run total: 3
Job closed
completed 3/3 successful

The response to the last cURL call, with "last_batch": true:

{
  "accepted_tasks": 1,
  "job_status": "closed",
  "latest_run": {
    "run_id": "01M4YK57WFDA6HYSJVH44VHNK0",
    "status": "running",
    "last_batch_received": true,
    "stats": { "total": 2, "completed": 0, "successful": 0, "failed": 0 }
  }
}

Use the Python SDK

submit_open opens the job, add_tasks appends a group, and "last_batch": True on the final group closes it. job.close() does the same without adding tasks.

Python (SDK)

# pip install zenrows
from zenrows import ZenRowsBatchClient

client = ZenRowsBatchClient(api_key="YOUR_ZENROWS_API_KEY")

# 1. Open a job with no tasks yet
job = client.submit_open(zenrows_params={"mode": "auto", "response_type": "markdown"})
print("Opened", job.job_id)

# 2. Add URLs as they arrive
job.add_tasks({"tasks": [
    {"url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "external_id": "abominable-hoodie"},
    {"url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "external_id": "adrienne-trek-jacket"},
]})

# 3. The final batch closes the job
last = job.add_tasks({
    "tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", "external_id": "aeon-capri"}],
    "last_batch": True,
})
print("Job", last.job_status.value)

run = job.run.wait(timeout=900)
print(run.status.value, f"{run.stats.successful}/{run.stats.total} successful")

Output

Opened 01M4YHVM70PF357C5MX3CNAW1B
Job closed
completed 3/3 successful

Customize it

Change	How
Settings for one URL only	Add `zenrows_params` to that task, see Use different settings for each URL in one job
Large groups	Send thousands of tasks per `POST /jobs/{job_id}/tasks` call and loop over your queue
A notification when it finishes	Add a `webhook` when you open the job, see Get notified when a Batch job finishes
URLs from a crawl	Feed each page of Crawl results into the open job, see Map every URL on a website
Stop early	`POST /jobs/{job_id}/stop` ends the current run, see Control the current run

Troubleshooting

Adding tasks returns 409 with job_not_open: the job is already closed by last_batch or /close. Open a new job for the next set of URLs.
Adding tasks right after a large submission returns 409: the job is still ingesting the previous group. Wait a few seconds and send the group again.
The run stays running after your producer finishes: the job is still open. Call POST /jobs/{job_id}/close so the run can complete.
429 with too_many_crawls: other Crawl and Batch jobs on your account are still running. Wait the number of seconds in the Retry-After header, then open the job.

Scrape a list of URLs

Send every URL up front as one job.

Start a Batch job from a CSV file

Upload a whole CSV of URLs at once.

Use different settings for each URL in one job

Give one task markdown, another JSON fields.

Get notified when a Batch job finishes

Receive a webhook instead of polling.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Add URLs to a job while it runs

Ask your AI agent

Open a job and stream URLs into it

Example output

Use the Python SDK

Customize it

Troubleshooting

Scrape a list of URLs

Start a Batch job from a CSV file

Use different settings for each URL in one job

Get notified when a Batch job finishes

​Ask your AI agent

​Open a job and stream URLs into it

​Example output

​Use the Python SDK

​Customize it

​Troubleshooting

​Related recipes

Scrape a list of URLs

Start a Batch job from a CSV file

Use different settings for each URL in one job

Get notified when a Batch job finishes

Ask your AI agent

Open a job and stream URLs into it

Example output

Use the Python SDK

Customize it

Troubleshooting

Related recipes