> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.
# Add URLs to a job while it runs
> Keep a Zenrows Batch job open and add URLs as your crawler or queue finds them, then close it when you're done.
When URLs arrive over time, from a crawler, a message queue or a nightly export that lands in pieces, open a Batch job first and feed it as you go. Zenrows starts scraping the first URLs right away, keeps accepting new ones, and finishes the run once you close the job. One job, one set of results, no matter how many groups you send.
**Uses:** [Batch open jobs](/batch/developer-guide-restapi#open-jobs-adding-tasks-in-batches) (`"status": "open"`, `POST /jobs/{id}/tasks`, `last_batch`, `POST /jobs/{id}/close`)
## Ask your AI agent
Paste this prompt into a coding agent such as Claude Code or Cursor, with the [Zenrows docs connected](/academy/connect-your-ai-agent#give-your-agent-the-zenrows-docs). The agent looks up the details and writes the calls.
```text theme={"dark"}
Using the Zenrows docs, change this worker so that every product URL it pulls from the
queue goes into one Zenrows Batch job that stays open while the worker runs. Send the URLs
in groups of 500, scrape them as markdown, and close the job when the queue is empty.
It should handle millions of URLs over a day.
```
## Open a job and stream URLs into it
1. Create the job with `"status": "open"`. Tasks are optional at this point. The `zenrows_params` you set here apply to every URL you add later.
2. Send each group of URLs to `POST /jobs/{job_id}/tasks` as `{"tasks": [...]}`. Scraping starts with the first group.
3. Close the job when your producer is done: either send the last group with `"last_batch": true`, or call `POST /jobs/{job_id}/close`. The run then finishes the queued tasks and ends `completed`.
```python Python (Requests) theme={"dark"}
import time
import requests
BASE = "https://async.api.zenrows.com/v1"
headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}
# 1. Open a job with no tasks yet: the settings apply to every URL you add later
job = requests.post(
f"{BASE}/jobs",
headers=headers,
json={"type": "regular", "status": "open", "zenrows_params": {"mode": "auto", "response_type": "markdown"}},
timeout=60,
).json()
job_id = job["job_id"]
print("Opened", job_id, job["status"])
# 2. Your producer (a crawler, a queue consumer) sends URLs as it finds them
def producer():
yield ["https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/"]
yield ["https://www.scrapingcourse.com/ecommerce/product/aeon-capri/"]
for urls in producer():
added = requests.post(
f"{BASE}/jobs/{job_id}/tasks",
headers=headers,
json={"tasks": [{"url": u, "external_id": u.rstrip("/").split("/")[-1]} for u in urls]},
timeout=60,
).json()
print("Added", added["accepted_tasks"], "tasks, run total:", added["latest_run"]["stats"]["total"])
# 3. Close the job when the producer is done; the run finishes the queued tasks
closed = requests.post(f"{BASE}/jobs/{job_id}/close", headers=headers, timeout=60).json()
print("Job", closed["status"])
while True:
run = requests.get(f"{BASE}/jobs/{job_id}", headers=headers, timeout=60).json()["latest_run"]
if run["status"] in {"completed", "failed", "stopped"}:
break
time.sleep(5)
print(run["status"], f"{run['stats']['successful']}/{run['stats']['total']} successful")
```
```javascript Node.js (SDK) theme={"dark"}
// npm install zenrows
import { ZenRowsBatchClient } from "zenrows";
const client = new ZenRowsBatchClient("YOUR_ZENROWS_API_KEY");
// 1. Open a job with no tasks yet
const job = await client.submitOpen([], {
zenrowsParams: { mode: "auto", response_type: "markdown" },
});
console.log("Opened", job.jobId);
// 2. Add URLs as they arrive
await job.addTasks([
{ url: "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", external_id: "abominable-hoodie" },
{ url: "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", external_id: "adrienne-trek-jacket" },
]);
// 3. The final batch closes the job
const last = await job.addTasks(
[{ url: "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", external_id: "aeon-capri" }],
{ lastBatch: true },
);
console.log("Job", last.job_status);
const run = await job.run.wait();
console.log(run.status, `${run.stats.successful}/${run.stats.total} successful`);
```
```bash cURL theme={"dark"}
# 1. Open the job and note its job_id
curl -X POST "https://async.api.zenrows.com/v1/jobs" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "type": "regular", "status": "open", "zenrows_params": { "mode": "auto" } }'
# 2. Add a group of URLs (repeat as they arrive)
curl -X POST "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID/tasks" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/" }] }'
# 3. Send the last group and close the job in one call
curl -X POST "https://async.api.zenrows.com/v1/jobs/YOUR_JOB_ID/tasks" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/" }], "last_batch": true }'
```
Read the results like any other job, see [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls) or [Download all results of a job at once](/academy/batch/export-results).
## Example output
The Python run:
```text theme={"dark"}
Opened 01M4YHTTN8HZC4HD98YW0EVEV9 open
Added 2 tasks, run total: 2
Added 1 tasks, run total: 3
Job closed
completed 3/3 successful
```
The response to the last cURL call, with `"last_batch": true`:
```json theme={"dark"}
{
"accepted_tasks": 1,
"job_status": "closed",
"latest_run": {
"run_id": "01M4YK57WFDA6HYSJVH44VHNK0",
"status": "running",
"last_batch_received": true,
"stats": { "total": 2, "completed": 0, "successful": 0, "failed": 0 }
}
}
```
## Use the Python SDK
`submit_open` opens the job, `add_tasks` appends a group, and `"last_batch": True` on the final group closes it. `job.close()` does the same without adding tasks.
```python Python (SDK) theme={"dark"}
# pip install zenrows
from zenrows import ZenRowsBatchClient
client = ZenRowsBatchClient(api_key="YOUR_ZENROWS_API_KEY")
# 1. Open a job with no tasks yet
job = client.submit_open(zenrows_params={"mode": "auto", "response_type": "markdown"})
print("Opened", job.job_id)
# 2. Add URLs as they arrive
job.add_tasks({"tasks": [
{"url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "external_id": "abominable-hoodie"},
{"url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/", "external_id": "adrienne-trek-jacket"},
]})
# 3. The final batch closes the job
last = job.add_tasks({
"tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", "external_id": "aeon-capri"}],
"last_batch": True,
})
print("Job", last.job_status.value)
run = job.run.wait(timeout=900)
print(run.status.value, f"{run.stats.successful}/{run.stats.total} successful")
```
```text Output theme={"dark"}
Opened 01M4YHVM70PF357C5MX3CNAW1B
Job closed
completed 3/3 successful
```
## Customize it
| Change | How |
| - | - |
| Settings for one URL only | Add `zenrows_params` to that task, see [Use different settings for each URL in one job](/academy/batch/per-task-settings) |
| Large groups | Send thousands of tasks per `POST /jobs/{job_id}/tasks` call and loop over your queue |
| A notification when it finishes | Add a `webhook` when you open the job, see [Get notified when a Batch job finishes](/academy/batch/webhooks) |
| URLs from a crawl | Feed each page of Crawl results into the open job, see [Map every URL on a website](/academy/crawl/map-a-website) |
| Stop early | `POST /jobs/{job_id}/stop` ends the current run, see [Control the current run](/batch/developer-guide-restapi#control-the-current-run) |
## Troubleshooting
* **Adding tasks returns `409` with `job_not_open`**: the job is already closed by `last_batch` or `/close`. Open a new job for the next set of URLs.
* **Adding tasks right after a large submission returns `409`**: the job is still ingesting the previous group. Wait a few seconds and send the group again.
* **The run stays `running` after your producer finishes**: the job is still open. Call `POST /jobs/{job_id}/close` so the run can complete.
* **`429` with `too_many_crawls`**: other Crawl and Batch jobs on your account are still running. Wait the number of seconds in the `Retry-After` header, then open the job.
## Related recipes
Send every URL up front as one job.
Upload a whole CSV of URLs at once.
Give one task markdown, another JSON fields.
Receive a webhook instead of polling.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.