> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Crawl a website from a start URL > Give Zenrows Crawl one start URL and get back every matching page on the site, with each page's content. Point Crawl at a homepage, category or search page and it follows the links for you: it opens each page with Fetch in Adaptive Stealth Mode, keeps the URLs that match your patterns, removes duplicates, and stores each page's HTML. You write no link loop, queue or deduplication. **Uses:** [Crawl](/crawl/introduction) (`POST /v1/crawls`, [`output_format`](/crawl/setup#get-each-pages-html), [`include_patterns`](/crawl/endpoints#start-a-crawl)) ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use Zenrows to crawl https://www.scrapingcourse.com/ecommerce/ for product pages, keeping 10 products and their content. When it finishes, read every product page and give me a table of product name and price. ``` ## Start a crawl and read every page A crawl is a background job: start it, wait until it ends, then read its results and each page's HTML. 1. `POST /v1/crawls` with the start `url`, your `include_patterns` and `"output_format": "html"`. Keep the `crawl_id`. 2. Poll `GET /v1/crawls/{crawl_id}` until `status` is no longer `running`. 3. Read the results, sending each response's `next_cursor` as `cursor` until it is `null`. 4. Download each page from its `content_url`. ```python Python (Requests) theme={"dark"} import time import requests API = "https://api.zenrows.com/v1/crawls" headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"} # 1. Start the crawl crawl = requests.post( API, headers=headers, json={ "url": "https://www.scrapingcourse.com/ecommerce/", "include_patterns": ["/product/"], "max_items": 3, "max_pages": 6, "output_format": "html", }, timeout=30, ).json() crawl_id = crawl["crawl_id"] print("Started", crawl_id) # 2. Wait until the crawl ends while True: body = requests.get(f"{API}/{crawl_id}", headers=headers, params={"limit": 1}, timeout=30).json() if body["status"] != "running": break time.sleep(5) print(body["status"], body["coverage"]) # 3. Read every result, following next_cursor results, cursor = [], None while True: params = {"cursor": cursor} if cursor else {} page = requests.get(f"{API}/{crawl_id}", headers=headers, params=params, timeout=30).json() results += page["results"] cursor = page["next_cursor"] if cursor is None: break # 4. Get each page's HTML from its content_url for result in results: if result.get("content_status") == "fetched": html = requests.get(f"https://api.zenrows.com{result['content_url']}", headers=headers, timeout=60).text print(result["url"], len(html), "bytes") ``` ```bash cURL theme={"dark"} # 1. Start the crawl curl -X POST "https://api.zenrows.com/v1/crawls" \ -H "X-API-Key: YOUR_ZENROWS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://www.scrapingcourse.com/ecommerce/", "include_patterns": ["/product/"], "max_items": 3, "max_pages": 6, "output_format": "html" }' # 2 and 3. Read its status and results (repeat with ?cursor=NEXT_CURSOR until next_cursor is null) curl "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID" \ -H "X-API-Key: YOUR_ZENROWS_API_KEY" # 4. Download one page's HTML from a result's content_url curl "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID/contents/YOUR_CONTENT_ID" \ -H "X-API-Key: YOUR_ZENROWS_API_KEY" \ -o page.html ``` Read the results once the crawl has ended: that's when every kept page has its `content_url`. Each page Crawl opens is one Fetch request in [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode), and `max_pages` sets how many it opens. ## Example output ```text theme={"dark"} Started c_01M4Y9RCGK0QEZ0QB9MC9S5AK7 completed {'pages_fetched': 4, 'pages_failed': 0, 'items_found': 3} https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ 120468 bytes https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/ 119577 bytes https://www.scrapingcourse.com/ecommerce/product/aeon-capri/ 91543 bytes ``` A finished crawl reads like this. Each result carries the page's `content_url`: ```json theme={"dark"} { "crawl_id": "c_01M4Y9RCGK0QEZ0QB9MC9S5AK7", "status": "completed", "stop_reason": "max_items", "coverage": { "pages_fetched": 4, "pages_failed": 0, "items_found": 3 }, "results": [ { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "content_status": "fetched", "content_url": "/v1/crawls/c_01M4Y9RCGK0QEZ0QB9MC9S5AK7/contents/ct_4bf794e572e17fdd50710f61cc730066" } ], "next_cursor": null } ``` ## Crawl with the SDKs `ZenRowsCrawlClient` wraps the same calls. `create` starts the crawl, `wait` polls until it ends, `results` follows the cursor for you, and `content` downloads a page's HTML. ```python Python (SDK) theme={"dark"} # pip install zenrows from zenrows import ZenRowsCrawlClient client = ZenRowsCrawlClient(api_key="YOUR_ZENROWS_API_KEY") crawl = client.create( "https://www.scrapingcourse.com/ecommerce/", include_patterns=["/product/"], max_items=3, max_pages=6, output_format="html", ) crawl = client.wait(crawl.crawl_id) print(crawl.status.value, crawl.coverage.pages_fetched, "pages fetched") for result in client.results(crawl.crawl_id): if result.content_url: html = client.content(crawl.crawl_id, result) print(result.url, len(html), "bytes") ``` ```javascript Node.js (SDK) theme={"dark"} // npm install zenrows import { ZenRowsCrawlClient } from "zenrows"; const client = new ZenRowsCrawlClient("YOUR_ZENROWS_API_KEY"); const crawl = await client.create({ url: "https://www.scrapingcourse.com/ecommerce/", includePatterns: ["/product/"], maxItems: 3, maxPages: 6, outputFormat: "html", }); const ended = await client.wait(crawl.crawl_id); console.log(ended.status, ended.coverage.pages_fetched, "pages fetched"); for await (const result of client.results(crawl.crawl_id)) { if (result.content_status === "fetched") { const html = await client.content(crawl.crawl_id, result); console.log(result.url, html.length, "bytes"); } } ``` The Node.js SDK takes camelCase arguments (`maxPages`, `includePatterns`) and returns fields in the API's form (`crawl_id`, `content_status`). See the [Python](/crawl/sdk/python) and [Node.js](/crawl/sdk/nodejs) SDK guides. ## Crawl from the CLI `--output-format html` stores each kept page, and `--follow` waits until the crawl ends. Then list the URLs, or save everything, HTML included, to one NDJSON file: ```bash CLI theme={"dark"} zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --include-patterns /product/ --max-items 3 --max-pages 6 --output-format html --follow zenrows crawl results c_01M4YD5KB00MSQKQ26QKZ8VCZN zenrows crawl download c_01M4YD5KB00MSQKQ26QKZ8VCZN --out products.jsonl ``` ```text Output (content URLs shortened) theme={"dark"} ✓ Crawl c_01M4YD5KB00MSQKQ26QKZ8VCZN finished · status: completed (max_items) 3 item(s) found · 4 page(s) fetched · 0 failed 3 result(s) for crawl c_01M4YD5KB00MSQKQ26QKZ8VCZN (status: completed): {"url":"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/","content_status":"fetched","content_url":"/v1/crawls/c_01M4YD5KB00MSQKQ26QKZ8VCZN/contents/ct_4bf7..."} {"url":"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/","content_status":"fetched","content_url":"/v1/crawls/c_01M4YD5KB00MSQKQ26QKZ8VCZN/contents/ct_a3eb..."} {"url":"https://www.scrapingcourse.com/ecommerce/product/aeon-capri/","content_status":"fetched","content_url":"/v1/crawls/c_01M4YD5KB00MSQKQ26QKZ8VCZN/contents/ct_fa8a..."} ✓ Downloaded 3 result(s) → products.jsonl ``` Each line of `products.jsonl` holds `url`, `content_status` and the page's HTML in `content`. To read one page, run `zenrows crawl content `. See the [CLI guide](/cli/crawl/introduction). ## Customize it | Change | How | | - | - | | Follow a set number of link steps | Set `depth`, for example `1` for the links on the start page, see [Start a crawl](/crawl/endpoints#start-a-crawl) | | Walk listing pages to reach more items | Add their path to `include_patterns`, for example `["/product/", "/page/"]` | | Skip sections | Add `exclude_patterns`, for example `["/cart", "?orderby="]` | | URLs without HTML | Leave out `output_format`, see [Map every URL on a website](/academy/crawl/map-a-website) | | Structured fields from every page | Send the kept URLs to Batch with `extract=auto`, see [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls) | | Stop a crawl early | `POST /v1/crawls/{crawl_id}/stop`, see [Stop a crawl](/crawl/setup#stop-a-crawl) | ## Troubleshooting * **The crawl returns few URLs**: `include_patterns` also decides which links Crawl opens. Add the paths of the listing or pagination pages that lead to your items, such as `/page/` or `/category/`. * **`content_status` reads `pending`**: the crawl is still fetching that page. Wait for the crawl to end, then read the results again from the start. * **`stop_reason` is `max_items` or `max_pages`**: the site holds more. Start a new crawl with higher values. * **`429` with `too_many_crawls`**: your account's Crawl and Batch jobs are all running. Wait the `Retry-After` seconds, then start the crawl. * **`status: failed` with `seed_unreachable`**: the start URL didn't load. Open it in a browser to check it, then start again. ## Related recipes Build a URL inventory and export it as CSV. Feed the URLs you found into one Batch job. Crawl a site and turn every page into markdown chunks. Parse each crawled page with BeautifulSoup or Cheerio. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.