> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Lesson 4: Scrape every product with Batch

> Foundations lesson 4: scrape every product page in one Zenrows Batch job and save the results as JSON and CSV.

In [Lesson 3](/academy/foundations/discover-every-product) you collected product URLs into `product_urls.txt`. Now you scrape all of them. Rather than looping over the URLs and managing retries and concurrency yourself, you submit them as one Batch job and collect the results when it finishes.

**What you'll build:** `scrape_all.py`, a script that scrapes every URL in `product_urls.txt` in one Batch job and writes the tracker's first dataset to `products.json` and `products.csv`.

**What you'll learn:**

* How a Batch job runs many Fetch requests for you, with retries and concurrency handled on Zenrows' side.
* How `zenrows_params` applies the same Fetch parameters to every URL in the job.
* How to wait for a run, check its stats and download its results with `ZenRowsBatchClient`.

**Time:** about 10 minutes.

## How a Batch job works

[Batch](/batch/introduction) takes a list of tasks (one URL each) and runs them on Zenrows' infrastructure. You submit the job and get back an ID at once. Zenrows works through the URLs at your plan's concurrency, retries the ones that fail, and stores each result. Each task costs the same as the Fetch request it runs, and failed tasks are not charged.

Every job has a **run**: the execution of its tasks. You wait for the run to finish, then read its stats and download its results.

## Write the scraping script

The script reuses `PRODUCT_SELECTORS` from Lesson 2, so every product page comes back as the same JSON fields. Create `scrape_all.py`:

```python Python (SDK) theme={"dark"}
import csv
import json
from zenrows import ZenRowsBatchClient

ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"

PRODUCT_SELECTORS = {
    "name": "h1.product_title",
    "price": "p.price",
    "sku": "span.sku",
    "category": "span.posted_in a",
    "sizes": "select#size option:not(:first-child) @value",
    "colors": "select#color option:not(:first-child) @value",
}

def as_list(value):
    if not value:
        return []
    return value if isinstance(value, list) else [value]

with open("product_urls.txt") as f:
    product_urls = [line.strip() for line in f if line.strip()]

client = ZenRowsBatchClient(api_key=ZENROWS_API_KEY)

job = client.submit_regular(
    [{"url": url, "external_id": url.rstrip("/").split("/")[-1]} for url in product_urls],
    zenrows_params={"mode": "auto", "css_extractor": json.dumps(PRODUCT_SELECTORS)},
    name="product-catalog-tracker",
)
print("Submitted job", job.job_id, "with", len(product_urls), "URLs")

run = job.run.wait(timeout=900)
print(f"Run {run.data.status.value}: {run.stats.successful}/{run.stats.total} succeeded")

products = []
for result in run.download_to_memory(status="successful"):
    fields = json.loads(result.body)
    products.append({
        "slug": result.external_id,
        "name": fields.get("name"),
        "price": fields.get("price"),
        "sku": fields.get("sku"),
        "category": fields.get("category"),
        "sizes": ", ".join(as_list(fields.get("sizes"))),
        "colors": ", ".join(as_list(fields.get("colors"))),
    })

products.sort(key=lambda p: p["slug"])

with open("products.json", "w") as f:
    json.dump(products, f, indent=2)

with open("products.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=list(products[0]))
    writer.writeheader()
    writer.writerows(products)

print(f"Wrote {len(products)} products to products.json and products.csv")
```

Here is what happens in it:

<Steps>
  <Step title="Submit the job">
    `client.submit_regular()` creates a one-time job from the URL list. Each task gets an `external_id`, here the product slug, so you can match every result to its product. `zenrows_params` applies `mode=auto` and your CSS selectors to every task.
  </Step>

  <Step title="Wait for the run">
    `job.run.wait()` polls the run until it finishes and returns it. `run.stats` counts the tasks: total, successful and failed.
  </Step>

  <Step title="Download the results">
    `run.download_to_memory()` fetches each successful task's body. With `css_extractor`, each body is the JSON object you saw in Lesson 2.
  </Step>

  <Step title="Write the dataset">
    `as_list()` turns a single match into a one-item list, so sizes and colors read the same for every product. The rows go to `products.json` and `products.csv`.
  </Step>
</Steps>

## Run it

```bash theme={"dark"}
python scrape_all.py
```

The 42 URLs from Lesson 3 finish in about a minute:

```text theme={"dark"}
Submitted job 01M4Y9JHJVNDZ356A7FSMH0PDQ with 42 URLs
Run completed: 42/42 succeeded
Wrote 42 products to products.json and products.csv
```

The first rows of `products.csv`:

```text theme={"dark"}
slug,name,price,sku,category,sizes,colors
abominable-hoodie,Abominable Hoodie,$69.00,MH09,Hoodies & Sweatshirts,"XS, S, M, L, XL","Blue, Green, Red"
adrienne-trek-jacket,Adrienne Trek Jacket,$57.00,WJ08,Erin Recommends|Clothing,"XS, S, M, L, XL","Gray, Orange, Purple"
aeon-capri,Aeon Capri,$48.00,WP07,Performance Fabrics|Clothing,"28, 29","Black, Blue, Orange"
```

Products without options, such as accessories, come back with empty `sizes` and `colors`:

```json theme={"dark"}
{
  "slug": "affirm-water-bottle",
  "name": "Affirm Water Bottle",
  "price": "$7.00",
  "sku": "24-UG06",
  "category": "Fitness Equipment",
  "sizes": "",
  "colors": ""
}
```

That's the tracker's first full dataset. Run `discover.py` with a larger `max_pages` and then `scrape_all.py`, and the same two scripts cover the whole store.

## Make it your own

| Change | How |
| - | - |
| Retry the tasks that failed | `job.retry_failed()` starts a run of the failed tasks, see [Reruns and retrying failures](/batch/sdk/python#reruns-and-retrying-failures) |
| Save each raw result as a file | `run.download_to_dir("./out", use_external_id=True)` |
| Override a parameter for one URL | Add `zenrows_params` to that task's dict, for example `{"proxy_country": "us"}` |
| Get markdown instead of fields | `zenrows_params={"mode": "auto", "response_type": "markdown"}` |
| Estimate the cost before you submit | `client.estimate_cost({...})`, see [Estimating cost](/batch/sdk/python#estimating-cost) |

## Ask your AI agent

With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"dark"}
Use Zenrows to scrape these product pages in one batch:
https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/
https://www.scrapingcourse.com/ecommerce/product/hero-hoodie/
https://www.scrapingcourse.com/ecommerce/product/aeon-capri/
Wait for it to finish, then give me a table with each product's name, price and SKU.
```

Paste three URLs or three thousand: the agent sends them as one job either way.

## What you built

`scrape_all.py` turns the URL list from Lesson 3 into `products.json` and `products.csv`. You can:

* Submit many URLs as one Batch job with `ZenRowsBatchClient.submit_regular()`.
* Apply Fetch parameters to every task with `zenrows_params`, and tag tasks with `external_id`.
* Wait for a run, read its stats and download its results.

## Next lesson

<Card title="Lesson 5: Add a product to the cart with Browser" icon="arrow-right" href="/academy/foundations/automate-a-browser">
  Drive a real browser with Playwright: pick a size and color, add to cart, and read the total.
</Card>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.