> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Turn search and listing pages into JSON > Turn search results and category pages into a JSON list of products with Zenrows Extract, across every page of results. Search results and category pages hold a full row per item: name, price, URL, image, SKU. Send one to Extract with `extract=auto` and you get every row as JSON, plus the page's pagination details, so you can walk every page and merge the results into one dataset. **Uses:** [Extract](/extract/setup) (`extract=auto`, [`extract_fields`](/extract/setup#selecting-the-fields-you-need)), [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode) (`mode=auto`) and [Batch](/batch/introduction) for many pages ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use Zenrows to search https://www.scrapingcourse.com/ecommerce/ for "jacket", go through every page of results, and give me one table of every product with its name, price and URL. ``` The same prompt works on whole categories with hundreds of pages: ask for "every page of this category in one batch". ## Extract one search results page Point Extract at the search URL. `parsed.products` holds one object per result and `parsed.page_context` tells you which page you're on and how many there are. Add `extract_fields` to keep the list fields you need: a dot path such as `products.price` applies to every item in the list. ```python Python (Requests) theme={"dark"} import json import requests params = { "apikey": "YOUR_ZENROWS_API_KEY", "url": "https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product", "mode": "auto", "extract": "auto", "extract_fields": "products.name,products.price,products.product_url,page_context", } response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180) response.raise_for_status() print(json.dumps(response.json()["parsed"], indent=2)) ``` ```javascript Node.js theme={"dark"} const params = new URLSearchParams({ apikey: "YOUR_ZENROWS_API_KEY", url: "https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product", mode: "auto", extract: "auto", extract_fields: "products.name,products.price,products.product_url,page_context", }); const response = await fetch(`https://api.zenrows.com/v1/?${params}`, { signal: AbortSignal.timeout(180_000), }); if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`); const { parsed } = await response.json(); console.log(JSON.stringify(parsed, null, 2)); ``` ```bash cURL theme={"dark"} curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F%3Fs%3Djacket%26post_type%3Dproduct&mode=auto&extract=auto&extract_fields=products.name%2Cproducts.price%2Cproducts.product_url%2Cpage_context" ``` ```bash CLI theme={"dark"} zenrows extract "https://www.scrapingcourse.com/ecommerce/?s=jacket&post_type=product" ``` The CLI prints every field in `parsed`. Its `zenrows extract` command sends `extract=auto` with `mode=auto` by default, and when a domain isn't prepared yet it retries the page with `autoparse`, so you always get JSON back. See the [CLI guide](/cli/extract/introduction). ```json Output (trimmed) theme={"dark"} { "page_context": { "current_page": 1, "total_pages": 2, "total_results": 21 }, "products": [ { "name": "Olivia 1/4 Zip Light Jacket", "price": 77, "product_url": "https://www.scrapingcourse.com/ecommerce/product/olivia-1-4-zip-light-jacket/" }, { "name": "Neve Studio Dance Jacket", "price": 69, "product_url": "https://www.scrapingcourse.com/ecommerce/product/neve-studio-dance-jacket/" } ] } ``` ## Follow every page and merge the results Read `page_context.total_pages` from the first page, build the URL of every other page, and extract each one. Keying rows by `product_url` merges them into one dataset and drops any item that shows up on two pages. ```python Python (Requests) theme={"dark"} import json import requests SEARCH = "https://www.scrapingcourse.com/ecommerce/{page}?s=jacket&post_type=product" def page_url(number): return SEARCH.format(page="" if number == 1 else f"page/{number}/") def extract(url): params = { "apikey": "YOUR_ZENROWS_API_KEY", "url": url, "mode": "auto", "extract": "auto", "extract_fields": "products.name,products.price,products.product_url,page_context.total_pages", } response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180) response.raise_for_status() return response.json()["parsed"] first = extract(page_url(1)) total_pages = first["page_context"]["total_pages"] rows = {p["product_url"]: p for p in first["products"]} for number in range(2, total_pages + 1): parsed = extract(page_url(number)) for product in parsed["products"]: rows[product["product_url"]] = product print(f"page {number}/{total_pages}: {len(rows)} products so far") dataset = list(rows.values()) with open("jackets.json", "w") as f: json.dump(dataset, f, indent=2) print(f"Saved {len(dataset)} products from {total_pages} pages") ``` ```javascript Node.js theme={"dark"} import { writeFile } from "node:fs/promises"; const pageUrl = (n) => `https://www.scrapingcourse.com/ecommerce/${n === 1 ? "" : `page/${n}/`}?s=jacket&post_type=product`; async function extract(url) { const params = new URLSearchParams({ apikey: "YOUR_ZENROWS_API_KEY", url, mode: "auto", extract: "auto", extract_fields: "products.name,products.price,products.product_url,page_context.total_pages", }); const response = await fetch(`https://api.zenrows.com/v1/?${params}`, { signal: AbortSignal.timeout(180_000), }); if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`); return (await response.json()).parsed; } const first = await extract(pageUrl(1)); const totalPages = first.page_context.total_pages; const rows = new Map(first.products.map((p) => [p.product_url, p])); for (let n = 2; n <= totalPages; n++) { const parsed = await extract(pageUrl(n)); for (const product of parsed.products) rows.set(product.product_url, product); console.log(`page ${n}/${totalPages}: ${rows.size} products so far`); } const dataset = [...rows.values()]; await writeFile("jackets.json", JSON.stringify(dataset, null, 2)); console.log(`Saved ${dataset.length} products from ${totalPages} pages`); ``` ```text Output theme={"dark"} page 2/2: 21 products so far Saved 21 products from 2 pages ``` Every site has its own page URL pattern (`/page/2/`, `?page=2`, `&p=2`). Open page 2 in your browser, copy its URL, and put the page number where it changes. ## Extract a search page from a major retailer The same request works on search pages of major retailers such as Walmart, Amazon, eBay and Best Buy. Field names follow each website, so run one request without `extract_fields` first and read what `parsed` offers. ```bash cURL theme={"dark"} curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.walmart.com%2Fsearch%3Fq%3Dmechanical%2Bkeyboard&mode=auto&extract=auto" ``` ```json Output (parsed, trimmed) theme={"dark"} { "search_query": { "query": "mechanical keyboard" }, "products": [ { "item_id": "1133256395", "title": "SteelSeries Apex Pro Mini WL Mechanical Gaming Keyboard - Adjustable Actuation - Compact 60% Form Factor - RGB - PBT Keycaps - BT5.0 /2.4GHz - USB-C", "current_price": 239.99, "fulfillment_label": "Shipping, arrives Sun, Oct 18" }, { "item_id": "579131893", "title": "SteelSeries Apex 5 Mechanical Gaming Keyboard - Per-Key RGB Illumination , OLED Smart Display - Black", "rating": 4.7, "review_count": 177, "fulfillment_label": "Free shipping, arrives Sat, Oct 17" } ] } ``` ## Hand many listing pages to Batch Once you know the page count, put every page URL in one Batch job with `extract=auto` in `zenrows_params`. Zenrows runs the pages in parallel and stores each result as JSON with the same `parsed` field, ready to merge. This example uses the full catalog's 12 listing pages. ```python Python (Requests) theme={"dark"} import json import time import requests BASE = "https://async.api.zenrows.com/v1" headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"} total_pages = 12 # page_context.total_pages from page 1 urls = ["https://www.scrapingcourse.com/ecommerce/"] + [ f"https://www.scrapingcourse.com/ecommerce/page/{n}/" for n in range(2, total_pages + 1) ] # 1. One job, one task per listing page job = requests.post( f"{BASE}/jobs", headers=headers, json={ "type": "regular", "status": "closed", "zenrows_params": { "mode": "auto", "extract": "auto", "extract_fields": "products.name,products.price,products.product_url", }, "tasks": [{"url": url, "external_id": f"page-{n}"} for n, url in enumerate(urls, 1)], }, timeout=60, ).json() job_id = job["job_id"] # 2. Wait for the run to finish while True: run = requests.get(f"{BASE}/jobs/{job_id}", headers=headers, timeout=30).json()["latest_run"] if run["status"] in {"completed", "failed", "stopped"}: break time.sleep(5) print(run["status"], f"{run['stats']['successful']}/{run['stats']['total']} pages extracted") # 3. Merge every page's products into one dataset products = {} cursor = None while True: page = requests.get( f"{BASE}/jobs/{job_id}/results", headers=headers, params={"status": "successful", "cursor": cursor}, timeout=30, ).json() for row in page["results"]: result = requests.get(row["result_url"], timeout=60).json() # presigned link for product in result["parsed"]["products"]: products[product["product_url"]] = product cursor = page.get("next_cursor") if not cursor: break with open("catalog.json", "w") as f: json.dump(list(products.values()), f, indent=2) print("Saved", len(products), "products") ``` ```text Output theme={"dark"} completed 12/12 pages extracted Saved 188 products ``` To discover listing pages without a URL pattern, let Crawl find them first, as in [Turn a whole catalog into a structured dataset](/academy/pipelines/catalog-dataset). ## Example output The first entries of `jackets.json`, written by the pagination script above (21 products from 2 search pages): ```json theme={"dark"} [ { "name": "Olivia 1/4 Zip Light Jacket", "price": 77, "product_url": "https://www.scrapingcourse.com/ecommerce/product/olivia-1-4-zip-light-jacket/" }, { "name": "Neve Studio Dance Jacket", "price": 69, "product_url": "https://www.scrapingcourse.com/ecommerce/product/neve-studio-dance-jacket/" }, { "name": "Jade Yoga Jacket", "price": 32, "product_url": "https://www.scrapingcourse.com/ecommerce/product/jade-yoga-jacket/" } ] ``` ## Customize it | Change | How | | - | - | | Every field of each item | Use a bare group name, `extract_fields=products`, or omit `extract_fields`, see [Selecting the fields you need](/extract/setup#selecting-the-fields-you-need) | | Several searches at once | Build the page URLs of each search term and send them all in one Batch job, with the term in each task's `external_id` | | Results from a specific country | Add `proxy_country=us` (any ISO code), see [Proxy country](/fetch/features/proxy-country) | | A new domain | Prepare it with a search page in `example_urls`, see [Prepare any website for Extract](/academy/extract/add-a-new-site) | | Typed, validated rows | Run each item through a schema, see [Validate extracted data with a schema](/academy/extract/validate-with-a-schema) | | Detail fields for each item | Send every `product_url` to Batch with `extract=auto`, see [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls) | ## Troubleshooting * **`400` with `REQS004` naming `extract_fields`**: none of the names exist on this page. The error lists the fields the page offers; copy names from there. * **`403` with `REQS007`**: the domain needs preparing for Extract. [Prepare it](/academy/extract/add-a-new-site) with the search URL in `example_urls`, then retry. * **Page 2 returns the same items as page 1**: the page URL pattern is different on this site. Open page 2 in your browser and copy its exact URL format. * **The dataset has fewer rows than `total_results`**: merging by `product_url` removed items listed twice, such as sponsored results. Key by another field, such as `item_id`, to keep them. * **The request takes long to return**: set your HTTP client timeout to 180 seconds or more so `mode=auto` has time to escalate. ## Related recipes How extract=auto works on a single page. Crawl a store, then extract every listing page with Batch. Type, normalize and check every row before you store it. Add a new domain's search and product pages to Extract. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.