> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Turn a whole catalog into a structured dataset

> Turn an entire online store into a structured dataset: find every product with Zenrows Crawl, extract them with Batch, and save CSV and JSON.

Point Crawl at a store, let it find every listing page, then send those pages to Batch with `extract=auto`. You get the full catalog (names, prices, SKUs, URLs, images) as one CSV or JSON file, with no selectors to write or maintain.

**Uses:** [Crawl](/crawl/introduction) with `include_patterns`, [Batch](/batch/introduction) with [Extract](/extract/introduction) (`extract=auto`)

## How it works

1. **Crawl discovers the pages.** Starting from the store's home page, Crawl follows links and keeps the URLs that match `include_patterns`, here the paginated listing pages (`/ecommerce/page/`).
2. **Batch extracts every page.** One Batch job runs `extract=auto` on every listing page in parallel. Each result is JSON with a `products` array.
3. **Your code merges the rows.** The script flattens every page's `products` into one list, removes duplicates by product URL, and writes `catalog.csv` and `catalog.json`.

Listing pages carry a full row per product, so 12 requests cover a 188-product store. To collect detail-page fields as well, point `include_patterns` at product URLs (`/product/`) and run a Batch step with a `css_extractor` for the fields you need.

## Ask your AI agent

With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"dark"}
Use Zenrows to find every listing page of https://www.scrapingcourse.com/ecommerce/,
scrape them all in one batch, and combine every product into one table with
name, price, SKU and product URL. Save it as catalog.csv.
```

The same prompt works on a catalog with thousands of pages: the crawl and the batch run in the cloud, not in the conversation.

## Build the dataset

Install the [Zenrows SDK](/sdk/overview) (`pip install zenrows` or `npm install zenrows`). The script runs the three steps end to end. `max_items=3` keeps the first run small: raise it, or remove it, to cover the whole catalog.

<CodeGroup>
  ```python Python (SDK) theme={"dark"}
  import csv
  import json

  from zenrows import ZenRowsBatchClient, ZenRowsCrawlClient

  API_KEY = "YOUR_ZENROWS_API_KEY"
  START_URL = "https://www.scrapingcourse.com/ecommerce/"

  # 1. Crawl: discover every listing page of the catalog
  crawler = ZenRowsCrawlClient(api_key=API_KEY)
  crawl = crawler.create(
      START_URL,
      include_patterns=["/ecommerce/page/"],
      max_items=3,  # raise to cover the whole catalog
  )
  crawl = crawler.wait(crawl.crawl_id)
  listing_urls = [START_URL] + [result.url for result in crawler.results(crawl.crawl_id)]
  print(f"Crawl {crawl.status.value}: {len(listing_urls)} listing pages")

  # 2. Batch: extract every listing page as structured JSON
  batch = ZenRowsBatchClient(api_key=API_KEY)
  job = batch.submit_regular(listing_urls, zenrows_params={"extract": "auto"})
  run = job.run.wait(timeout=900)
  print(f"Batch {run.data.status.value}: {run.stats.successful}/{run.stats.total} pages extracted")

  # 3. Merge the products into one dataset, one row per product
  products = {}
  for task in run.results(status="successful"):
      parsed = json.loads(run.download_task_to_memory(task))["parsed"]
      for product in parsed.get("products", []):
          products[product["product_url"]] = product

  rows = list(products.values())
  fields = ["name", "price", "currency", "sku", "product_id", "product_url", "image_url"]

  with open("catalog.csv", "w", newline="", encoding="utf-8") as f:
      writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
      writer.writeheader()
      writer.writerows(rows)

  with open("catalog.json", "w", encoding="utf-8") as f:
      json.dump(rows, f, indent=2, ensure_ascii=False)

  print(f"Saved {len(rows)} products to catalog.csv and catalog.json")
  ```

  ```javascript Node.js theme={"dark"}
  import { writeFile } from "node:fs/promises";
  import { ZenRowsBatchClient, ZenRowsCrawlClient } from "zenrows";

  const API_KEY = "YOUR_ZENROWS_API_KEY";
  const START_URL = "https://www.scrapingcourse.com/ecommerce/";

  // 1. Crawl: discover every listing page of the catalog
  const crawler = new ZenRowsCrawlClient(API_KEY);
  const crawl = await crawler.create({
    url: START_URL,
    includePatterns: ["/ecommerce/page/"],
    maxItems: 3, // raise to cover the whole catalog
  });
  const ended = await crawler.wait(crawl.crawl_id);
  const listingUrls = [START_URL];
  for await (const result of crawler.results(crawl.crawl_id)) listingUrls.push(result.url);
  console.log(`Crawl ${ended.status}: ${listingUrls.length} listing pages`);

  // 2. Batch: extract every listing page as structured JSON
  const batch = new ZenRowsBatchClient(API_KEY);
  const job = await batch.submitRegular(listingUrls, undefined, { zenrowsParams: { extract: "auto" } });
  const run = await job.run.wait();
  console.log(`Batch ${run.data.status}: ${run.data.stats.successful}/${run.data.stats.total} pages extracted`);

  // 3. Merge the products into one dataset, one row per product
  const products = new Map();
  for await (const task of run.results({ status: "successful" })) {
    const { parsed } = JSON.parse(await run.downloadTaskToMemory(task));
    for (const product of parsed.products ?? []) products.set(product.product_url, product);
  }

  const rows = [...products.values()];
  const fields = ["name", "price", "currency", "sku", "product_id", "product_url", "image_url"];
  const csvCell = (value) => `"${String(value ?? "").replaceAll('"', '""')}"`;
  const csv = [fields.join(","), ...rows.map((row) => fields.map((f) => csvCell(row[f])).join(","))].join("\n");

  await writeFile("catalog.csv", csv);
  await writeFile("catalog.json", JSON.stringify(rows, null, 2));
  console.log(`Saved ${rows.length} products to catalog.csv and catalog.json`);
  ```
</CodeGroup>

`crawler.wait` and `job.run.wait` poll until each step finishes, so the script runs top to bottom with no extra polling code.

## Build it from the terminal

The [Zenrows CLI](/cli/introduction) runs the same pipeline with four commands and <a href="https://jqlang.org/" rel="nofollow">jq</a>. Install it with `npm install -g @zenrows/cli` and log in with `zenrows login`.

```bash CLI theme={"dark"}
# 1. Crawl the listing pages and save their URLs as JSONL
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ \
  --include-patterns /ecommerce/page/ --max-items 3 --follow --json > crawl.json
zenrows crawl results "$(jq -r .crawl_id crawl.json)" --out pages.jsonl
echo '{"url": "https://www.scrapingcourse.com/ecommerce/"}' >> pages.jsonl  # the start page has products too

# 2. Turn each URL into a Batch task with extract=auto, then run the job
jq -c '{url, extract: "auto"}' pages.jsonl > jobs.jsonl
zenrows batch create jobs.jsonl --wait --json > job.json

# 3. Download every result and merge the products into CSV
zenrows batch results "$(jq -r .jobId job.json)" --download ./pages
jq -s '[.[].parsed.products[]] | unique_by(.product_url)' pages/*.json > catalog.json
jq -r '(["name","price","sku","product_url"]), (.[] | [.name, .price, .sku, .product_url]) | @csv' \
  catalog.json > catalog.csv
```

## Example output

The Python script's console output, from a run with `max_items=3`:

```text theme={"dark"}
Crawl completed: 4 listing pages
Batch completed: 4/4 pages extracted
Saved 64 products to catalog.csv and catalog.json
```

The first rows of `catalog.csv`:

```text theme={"dark"}
name,price,currency,sku,product_id,product_url,image_url
Atlas Fitness Tank,18,USD,MT11,1364,https://www.scrapingcourse.com/ecommerce/product/atlas-fitness-tank/,https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mt11-blue_main.jpg
Atomic Endurance Running Tee (Crew-Neck),29,USD,MS12,1025,https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-running-tee-crew-neck/,https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/ms12-red_main.jpg
Augusta Pullover Jacket,57,USD,WJ03,1761,https://www.scrapingcourse.com/ecommerce/product/augusta-pullover-jacket/,https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/wj03-red_main.jpg
```

Each entry in `catalog.json` keeps every field Extract returned:

```json theme={"dark"}
{
  "availability": null,
  "currency": "USD",
  "image_url": "https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mt11-blue_main.jpg",
  "name": "Atlas Fitness Tank",
  "price": 18,
  "product_id": "1364",
  "product_type": "variable",
  "product_url": "https://www.scrapingcourse.com/ecommerce/product/atlas-fitness-tank/",
  "sku": "MT11"
}
```

## Customize it

| Change | How |
| - | - |
| Cover the whole catalog | Remove `max_items` or raise it. `max_pages` sets how many pages the crawl fetches, see [Crawl endpoints](/crawl/endpoints#start-a-crawl) |
| Crawl one category | Start from the category URL and set `include_patterns` to its path, for example `/product-category/clothing/men/` |
| Keep only some fields | Add `extract_fields` to `zenrows_params`, for example `"extract_fields": "products.name,products.price"`, see [Extract](/extract/endpoints) |
| Use your own selectors | Replace `extract` with `css_extractor` in `zenrows_params`, see [CSS extractor](/fetch/features/css-extractor) |
| Target a country's prices | Add `"mode": "auto", "proxy_country": "us"` to `zenrows_params`, see [Proxy country](/fetch/features/proxy-country) |
| Get notified instead of waiting | Attach a webhook to the Batch job, see [Batch webhooks](/batch/developer-guide-restapi#set-up-webhooks) |
| Refresh the dataset every day | Submit the Batch step as a scheduled job, see [Scheduled jobs](/batch/developer-guide-restapi#scheduled-jobs) |

## Troubleshooting

* **The crawl returns no listing pages**: the pattern doesn't match the store's pagination URLs. Open page 2 in your browser, copy the part of the URL that repeats (`/page/`, `?page=`, `/p/`) and use it as `include_patterns`.
* **The dataset is smaller than the store**: `max_items` stopped the crawl early. Raise it, and check `crawl.coverage.items_found` against the store's page count.
* **A page returns `REQS004`**: the `extract_fields` you asked for aren't on that page. The error's `detail` lists the available fields: copy the ones you need, or remove `extract_fields`.
* **Some Batch tasks failed**: rerun only the failures with `job.retry_failed()` (Python), `job.retryFailed()` (Node.js) or `zenrows batch retry-failed <job-id>`. Successful results carry over.

## Related recipes

<CardGroup cols={2}>
  <Card title="Map every URL on a website" icon="sitemap" href="/academy/crawl/map-a-website">
    Run Crawl on its own to list a site's pages.
  </Card>

  <Card title="Get structured JSON from any page" icon="wand" href="/academy/extract/auto-extraction">
    How `extract=auto` works on one page.
  </Card>

  <Card title="Scrape a list of URLs" icon="list-check" href="/academy/scrape-from-a-list-of-urls">
    Send thousands of URLs as one Batch job.
  </Card>

  <Card title="Build a price monitor" icon="chart-line" href="/academy/build-price-monitoring">
    Track the prices in your dataset over time.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.