Lesson 4: Scrape every product with Batch

In Lesson 3 you collected product URLs into product_urls.txt. Now you scrape all of them. Rather than looping over the URLs and managing retries and concurrency yourself, you submit them as one Batch job and collect the results when it finishes. What you’ll build: scrape_all.py, a script that scrapes every URL in product_urls.txt in one Batch job and writes the tracker’s first dataset to products.json and products.csv. What you’ll learn:

How a Batch job runs many Fetch requests for you, with retries and concurrency handled on Zenrows’ side.
How zenrows_params applies the same Fetch parameters to every URL in the job.
How to wait for a run, check its stats and download its results with ZenRowsBatchClient.

Time: about 10 minutes.

How a Batch job works

Batch takes a list of tasks (one URL each) and runs them on Zenrows’ infrastructure. You submit the job and get back an ID at once. Zenrows works through the URLs at your plan’s concurrency, retries the ones that fail, and stores each result. Each task costs the same as the Fetch request it runs, and failed tasks are not charged. Every job has a run: the execution of its tasks. You wait for the run to finish, then read its stats and download its results.

Write the scraping script

The script reuses PRODUCT_SELECTORS from Lesson 2, so every product page comes back as the same JSON fields. Create scrape_all.py:

Python (SDK)

import csv
import json
from zenrows import ZenRowsBatchClient

ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"

PRODUCT_SELECTORS = {
    "name": "h1.product_title",
    "price": "p.price",
    "sku": "span.sku",
    "category": "span.posted_in a",
    "sizes": "select#size option:not(:first-child) @value",
    "colors": "select#color option:not(:first-child) @value",
}

def as_list(value):
    if not value:
        return []
    return value if isinstance(value, list) else [value]

with open("product_urls.txt") as f:
    product_urls = [line.strip() for line in f if line.strip()]

client = ZenRowsBatchClient(api_key=ZENROWS_API_KEY)

job = client.submit_regular(
    [{"url": url, "external_id": url.rstrip("/").split("/")[-1]} for url in product_urls],
    zenrows_params={"mode": "auto", "css_extractor": json.dumps(PRODUCT_SELECTORS)},
    name="product-catalog-tracker",
)
print("Submitted job", job.job_id, "with", len(product_urls), "URLs")

run = job.run.wait(timeout=900)
print(f"Run {run.data.status.value}: {run.stats.successful}/{run.stats.total} succeeded")

products = []
for result in run.download_to_memory(status="successful"):
    fields = json.loads(result.body)
    products.append({
        "slug": result.external_id,
        "name": fields.get("name"),
        "price": fields.get("price"),
        "sku": fields.get("sku"),
        "category": fields.get("category"),
        "sizes": ", ".join(as_list(fields.get("sizes"))),
        "colors": ", ".join(as_list(fields.get("colors"))),
    })

products.sort(key=lambda p: p["slug"])

with open("products.json", "w") as f:
    json.dump(products, f, indent=2)

with open("products.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=list(products[0]))
    writer.writeheader()
    writer.writerows(products)

print(f"Wrote {len(products)} products to products.json and products.csv")

Here is what happens in it:

Submit the job

client.submit_regular() creates a one-time job from the URL list. Each task gets an external_id, here the product slug, so you can match every result to its product. zenrows_params applies mode=auto and your CSS selectors to every task.

Wait for the run

job.run.wait() polls the run until it finishes and returns it. run.stats counts the tasks: total, successful and failed.

Download the results

run.download_to_memory() fetches each successful task’s body. With css_extractor, each body is the JSON object you saw in Lesson 2.

Write the dataset

as_list() turns a single match into a one-item list, so sizes and colors read the same for every product. The rows go to products.json and products.csv.

Run it

python scrape_all.py

The 42 URLs from Lesson 3 finish in about a minute:

Submitted job 01M4Y9JHJVNDZ356A7FSMH0PDQ with 42 URLs
Run completed: 42/42 succeeded
Wrote 42 products to products.json and products.csv

The first rows of products.csv:

slug,name,price,sku,category,sizes,colors
abominable-hoodie,Abominable Hoodie,$69.00,MH09,Hoodies & Sweatshirts,"XS, S, M, L, XL","Blue, Green, Red"
adrienne-trek-jacket,Adrienne Trek Jacket,$57.00,WJ08,Erin Recommends|Clothing,"XS, S, M, L, XL","Gray, Orange, Purple"
aeon-capri,Aeon Capri,$48.00,WP07,Performance Fabrics|Clothing,"28, 29","Black, Blue, Orange"

Products without options, such as accessories, come back with empty sizes and colors:

{
  "slug": "affirm-water-bottle",
  "name": "Affirm Water Bottle",
  "price": "$7.00",
  "sku": "24-UG06",
  "category": "Fitness Equipment",
  "sizes": "",
  "colors": ""
}

That’s the tracker’s first full dataset. Run discover.py with a larger max_pages and then scrape_all.py, and the same two scripts cover the whole store.

Make it your own

Change	How
Retry the tasks that failed	`job.retry_failed()` starts a run of the failed tasks, see Reruns and retrying failures
Save each raw result as a file	`run.download_to_dir("./out", use_external_id=True)`
Override a parameter for one URL	Add `zenrows_params` to that task’s dict, for example `{"proxy_country": "us"}`
Get markdown instead of fields	`zenrows_params={"mode": "auto", "response_type": "markdown"}`
Estimate the cost before you submit	`client.estimate_cost({...})`, see Estimating cost

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to scrape these product pages in one batch:
https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/
https://www.scrapingcourse.com/ecommerce/product/hero-hoodie/
https://www.scrapingcourse.com/ecommerce/product/aeon-capri/
Wait for it to finish, then give me a table with each product's name, price and SKU.

Paste three URLs or three thousand: the agent sends them as one job either way.

What you built

scrape_all.py turns the URL list from Lesson 3 into products.json and products.csv. You can:

Submit many URLs as one Batch job with ZenRowsBatchClient.submit_regular().
Apply Fetch parameters to every task with zenrows_params, and tag tasks with external_id.
Wait for a run, read its stats and download its results.

Next lesson

Lesson 5: Add a product to the cart with Browser

Drive a real browser with Playwright: pick a size and color, add to cart, and read the total.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Lesson 4: Scrape every product with Batch

How a Batch job works

Write the scraping script

Run it

Make it your own

Ask your AI agent

What you built

Next lesson

Lesson 5: Add a product to the cart with Browser

​How a Batch job works

​Write the scraping script

​Run it

​Make it your own

​Ask your AI agent

​What you built

​Next lesson

Lesson 5: Add a product to the cart with Browser

How a Batch job works

Write the scraping script

Run it

Make it your own

Ask your AI agent

What you built

Next lesson