> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Run many requests concurrently > Scrape many URLs at once with Zenrows Fetch, using all of your plan's capacity, or let the Zenrows SDKs run them in parallel for you. Sending Fetch requests one at a time leaves most of your plan idle. Run them in parallel, with as many in flight as your plan's concurrency allows, and a list of URLs finishes in a fraction of the time. Each response tells you that number in its `Concurrency-Limit` header. **Uses:** [Fetch](/fetch/api-reference) with [concurrency headers](/fetch/features/concurrency#concurrency-headers), the [Python SDK](/fetch/sdk/python/async-concurrency) and the [Node.js SDK](/fetch/sdk/nodejs/async-concurrency) ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use Zenrows to scrape all of these pages at once and give me each product's name and price: https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/ https://www.scrapingcourse.com/ecommerce/product/aeon-capri/ https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee/ ``` Your agent sends the list as one Batch job, so the same prompt scales from four URLs to thousands. ## Size a worker pool to your plan Make one request, read `Concurrency-Limit`, then run that many workers over the rest of the list. Every worker reuses the same pattern: send, check, parse. ```python Python (Requests) theme={"dark"} import re from concurrent.futures import ThreadPoolExecutor, as_completed import requests API_KEY = "YOUR_ZENROWS_API_KEY" SLUGS = [ "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee", "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt", ] URLS = [f"https://www.scrapingcourse.com/ecommerce/product/{slug}/" for slug in SLUGS] session = requests.Session() def scrape(url): params = {"apikey": API_KEY, "url": url, "mode": "auto"} response = session.get("https://api.zenrows.com/v1/", params=params, timeout=180) response.raise_for_status() title = re.search(r"]*>(.*?)", response.text, re.S).group(1).strip() return title, response.headers # Read your plan's limit from the first response, then size the pool to it. first_title, headers = scrape(URLS[0]) limit = int(headers["Concurrency-Limit"]) print(f"Concurrency-Limit: {limit}") print(first_title) with ThreadPoolExecutor(max_workers=limit) as pool: futures = {pool.submit(scrape, url): url for url in URLS[1:]} for future in as_completed(futures): try: title, _ = future.result() print(title) except Exception as error: # HTTP errors, timeouts, pages without an

print(f"{futures[future]} failed: {error}") ``` ```javascript Node.js theme={"dark"} const API_KEY = "YOUR_ZENROWS_API_KEY"; const slugs = [ "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee", "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt", ]; const urls = slugs.map((slug) => `https://www.scrapingcourse.com/ecommerce/product/${slug}/`); async function scrape(url) { const params = new URLSearchParams({ apikey: API_KEY, url, mode: "auto" }); const response = await fetch(`https://api.zenrows.com/v1/?${params}`, { signal: AbortSignal.timeout(180_000), }); if (!response.ok) throw new Error(`${url} failed: ${response.status}`); const html = await response.text(); const title = html.match(/]*>([\s\S]*?)<\/h1>/)[1].trim(); return { title, limit: Number(response.headers.get("Concurrency-Limit")) }; } // Read your plan's limit from the first response, then run that many workers. const first = await scrape(urls[0]); console.log(`Concurrency-Limit: ${first.limit}`); console.log(first.title); const queue = urls.slice(1); async function worker() { while (queue.length > 0) { const url = queue.shift(); try { const { title } = await scrape(url); console.log(title); } catch (error) { console.error(error.message); } } } await Promise.all(Array.from({ length: first.limit }, worker)); ``` ## Example output Results print in the order they finish, not the order of the list: ```text theme={"dark"} Concurrency-Limit: 5 Abominable Hoodie Aeon Capri Affirm Water Bottle Aero Daily Fitness Tee Adrienne Trek Jacket Aether Gym Pant Aim Analog Watch Ajax Full-Zip Sweatshirt ``` ## Let the SDK queue requests for you The Zenrows SDKs keep an internal queue: fire every request at once and the client keeps `concurrency` of them in flight. `retries` re-sends transient failures. Install with `pip install zenrows` or `npm install zenrows`. ```python Python (SDK) theme={"dark"} import asyncio import re from zenrows import ZenRowsClient SLUGS = [ "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee", "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt", ] URLS = [f"https://www.scrapingcourse.com/ecommerce/product/{slug}/" for slug in SLUGS] # concurrency sets the requests in flight; retries re-sends transient failures. client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1) async def main(): responses = await asyncio.gather( *[client.fetch_async(url, params={"mode": "auto"}, timeout=180) for url in URLS] ) for url, response in zip(URLS, responses): match = re.search(r"]*>(.*?)

", response.text, re.S) print(response.status_code, match.group(1).strip() if match else url) asyncio.run(main()) ``` ```javascript Node.js (SDK) theme={"dark"} import { ZenRows } from "zenrows"; const slugs = [ "abominable-hoodie", "adrienne-trek-jacket", "aeon-capri", "aero-daily-fitness-tee", "aether-gym-pant", "affirm-water-bottle", "aim-analog-watch", "ajax-full-zip-sweatshirt", ]; const urls = slugs.map((slug) => `https://www.scrapingcourse.com/ecommerce/product/${slug}/`); // concurrency sets the requests in flight; retries re-sends transient failures. const client = new ZenRows("YOUR_ZENROWS_API_KEY", { concurrency: 5, retries: 1 }); const results = await Promise.allSettled(urls.map((url) => client.fetch(url, { mode: "auto" }))); for (const [i, result] of results.entries()) { if (result.status === "rejected") { console.error(urls[i], result.reason); continue; } const html = await result.value.text(); const title = html.match(/]*>([\s\S]*?)<\/h1>/)?.[1].trim(); console.log(result.value.status, title); } ``` Output, in list order: ```text theme={"dark"} 200 Abominable Hoodie 200 Adrienne Trek Jacket 200 Aeon Capri 200 Aero Daily Fitness Tee 200 Aether Gym Pant 200 Affirm Water Bottle 200 Aim Analog Watch 200 Ajax Full-Zip Sweatshirt ``` Set `concurrency` to your plan's `Concurrency-Limit`. Use one client for the whole run, so every request shares the same queue. ## Send very large lists to Batch For thousands of URLs, or jobs you want to run without keeping a process alive, send the list to [Batch](/batch/introduction). Zenrows queues the URLs, runs them at your plan's concurrency, retries and stores every result for you to download. See [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls). ## Customize it | Change | How | | - | - | | Back off and retry failures | See [Retry failed requests](/academy/retry-failed-requests) | | Watch free slots live | Read `Concurrency-Remaining` on each response, see [Concurrency headers](/fetch/features/concurrency#concurrency-headers) | | Different parameters per URL | Build `params` inside `scrape()` from the URL or a per-URL dict | | Thousands of URLs, results stored for you | Use [Batch](/academy/scrape-from-a-list-of-urls) | | Many parallel real browsers | See [Run browser sessions in parallel](/academy/browser/parallel-sessions) | ## Troubleshooting * **`429` with code `AUTH006`**: more requests are in flight than your plan's concurrency. Lower the pool size to `Concurrency-Limit`, and keep one pool or one SDK client for the whole run. * **`429` right after a burst of cancelled requests**: a request cancelled on your side keeps its slot until Zenrows finishes it. Keep client timeouts at 180 seconds or more so you don't cancel work in progress. * **Throughput is lower than expected**: slow pages hold their slot longer. Use the cheapest configuration that works, for example `mode=auto` instead of a fixed `js_render`, so each request returns sooner. * **One failure stops the whole run**: an exception escaped a worker. Catch errors per URL, as the samples do, and collect failures for a retry pass. ## Related recipes Send thousands of URLs as one Batch job. Retry 429s and 5xx errors with backoff. The Foundations lesson on moving from one URL to many. Keep the per-request cost low on large runs. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.