Crawl a website from a start URL

Point Crawl at a homepage, category or search page and it follows the links for you: it opens each page with Fetch in Adaptive Stealth Mode, keeps the URLs that match your patterns, removes duplicates, and stores each page’s HTML. You write no link loop, queue or deduplication. Uses: Crawl (POST /v1/crawls, output_format, include_patterns)

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to crawl https://www.scrapingcourse.com/ecommerce/ for product pages, keeping
10 products and their content. When it finishes, read every product page and give me a
table of product name and price.

Start a crawl and read every page

A crawl is a background job: start it, wait until it ends, then read its results and each page’s HTML.

POST /v1/crawls with the start url, your include_patterns and "output_format": "html". Keep the crawl_id.
Poll GET /v1/crawls/{crawl_id} until status is no longer running.
Read the results, sending each response’s next_cursor as cursor until it is null.
Download each page from its content_url.

import time
import requests

API = "https://api.zenrows.com/v1/crawls"
headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}

# 1. Start the crawl
crawl = requests.post(
    API,
    headers=headers,
    json={
        "url": "https://www.scrapingcourse.com/ecommerce/",
        "include_patterns": ["/product/"],
        "max_items": 3,
        "max_pages": 6,
        "output_format": "html",
    },
    timeout=30,
).json()
crawl_id = crawl["crawl_id"]
print("Started", crawl_id)

# 2. Wait until the crawl ends
while True:
    body = requests.get(f"{API}/{crawl_id}", headers=headers, params={"limit": 1}, timeout=30).json()
    if body["status"] != "running":
        break
    time.sleep(5)
print(body["status"], body["coverage"])

# 3. Read every result, following next_cursor
results, cursor = [], None
while True:
    params = {"cursor": cursor} if cursor else {}
    page = requests.get(f"{API}/{crawl_id}", headers=headers, params=params, timeout=30).json()
    results += page["results"]
    cursor = page["next_cursor"]
    if cursor is None:
        break

# 4. Get each page's HTML from its content_url
for result in results:
    if result.get("content_status") == "fetched":
        html = requests.get(f"https://api.zenrows.com{result['content_url']}", headers=headers, timeout=60).text
        print(result["url"], len(html), "bytes")

# 1. Start the crawl
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "include_patterns": ["/product/"],
    "max_items": 3,
    "max_pages": 6,
    "output_format": "html"
  }'

# 2 and 3. Read its status and results (repeat with ?cursor=NEXT_CURSOR until next_cursor is null)
curl "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY"

# 4. Download one page's HTML from a result's content_url
curl "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID/contents/YOUR_CONTENT_ID" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -o page.html

Read the results once the crawl has ended: that’s when every kept page has its content_url. Each page Crawl opens is one Fetch request in Adaptive Stealth Mode, and max_pages sets how many it opens.

Example output

Started c_01M4Y9RCGK0QEZ0QB9MC9S5AK7
completed {'pages_fetched': 4, 'pages_failed': 0, 'items_found': 3}
https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ 120468 bytes
https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/ 119577 bytes
https://www.scrapingcourse.com/ecommerce/product/aeon-capri/ 91543 bytes

A finished crawl reads like this. Each result carries the page’s content_url:

{
  "crawl_id": "c_01M4Y9RCGK0QEZ0QB9MC9S5AK7",
  "status": "completed",
  "stop_reason": "max_items",
  "coverage": { "pages_fetched": 4, "pages_failed": 0, "items_found": 3 },
  "results": [
    {
      "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
      "content_status": "fetched",
      "content_url": "/v1/crawls/c_01M4Y9RCGK0QEZ0QB9MC9S5AK7/contents/ct_4bf794e572e17fdd50710f61cc730066"
    }
  ],
  "next_cursor": null
}

Crawl with the SDKs

ZenRowsCrawlClient wraps the same calls. create starts the crawl, wait polls until it ends, results follows the cursor for you, and content downloads a page’s HTML.

# pip install zenrows
from zenrows import ZenRowsCrawlClient

client = ZenRowsCrawlClient(api_key="YOUR_ZENROWS_API_KEY")

crawl = client.create(
    "https://www.scrapingcourse.com/ecommerce/",
    include_patterns=["/product/"],
    max_items=3,
    max_pages=6,
    output_format="html",
)
crawl = client.wait(crawl.crawl_id)
print(crawl.status.value, crawl.coverage.pages_fetched, "pages fetched")

for result in client.results(crawl.crawl_id):
    if result.content_url:
        html = client.content(crawl.crawl_id, result)
        print(result.url, len(html), "bytes")

// npm install zenrows
import { ZenRowsCrawlClient } from "zenrows";

const client = new ZenRowsCrawlClient("YOUR_ZENROWS_API_KEY");

const crawl = await client.create({
  url: "https://www.scrapingcourse.com/ecommerce/",
  includePatterns: ["/product/"],
  maxItems: 3,
  maxPages: 6,
  outputFormat: "html",
});
const ended = await client.wait(crawl.crawl_id);
console.log(ended.status, ended.coverage.pages_fetched, "pages fetched");

for await (const result of client.results(crawl.crawl_id)) {
  if (result.content_status === "fetched") {
    const html = await client.content(crawl.crawl_id, result);
    console.log(result.url, html.length, "bytes");
  }
}

The Node.js SDK takes camelCase arguments (maxPages, includePatterns) and returns fields in the API’s form (crawl_id, content_status). See the Python and Node.js SDK guides.

Crawl from the CLI

--output-format html stores each kept page, and --follow waits until the crawl ends. Then list the URLs, or save everything, HTML included, to one NDJSON file:

CLI

zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --include-patterns /product/ --max-items 3 --max-pages 6 --output-format html --follow
zenrows crawl results c_01M4YD5KB00MSQKQ26QKZ8VCZN
zenrows crawl download c_01M4YD5KB00MSQKQ26QKZ8VCZN --out products.jsonl

Output (content URLs shortened)

✓ Crawl c_01M4YD5KB00MSQKQ26QKZ8VCZN finished · status: completed (max_items)
  3 item(s) found · 4 page(s) fetched · 0 failed
3 result(s) for crawl c_01M4YD5KB00MSQKQ26QKZ8VCZN (status: completed):
{"url":"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/","content_status":"fetched","content_url":"/v1/crawls/c_01M4YD5KB00MSQKQ26QKZ8VCZN/contents/ct_4bf7..."}
{"url":"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/","content_status":"fetched","content_url":"/v1/crawls/c_01M4YD5KB00MSQKQ26QKZ8VCZN/contents/ct_a3eb..."}
{"url":"https://www.scrapingcourse.com/ecommerce/product/aeon-capri/","content_status":"fetched","content_url":"/v1/crawls/c_01M4YD5KB00MSQKQ26QKZ8VCZN/contents/ct_fa8a..."}
✓ Downloaded 3 result(s) → products.jsonl

Each line of products.jsonl holds url, content_status and the page’s HTML in content. To read one page, run zenrows crawl content <crawl_id> <content_id>. See the CLI guide.

Customize it

Change	How
Follow a set number of link steps	Set `depth`, for example `1` for the links on the start page, see Start a crawl
Walk listing pages to reach more items	Add their path to `include_patterns`, for example `["/product/", "/page/"]`
Skip sections	Add `exclude_patterns`, for example `["/cart", "?orderby="]`
URLs without HTML	Leave out `output_format`, see Map every URL on a website
Structured fields from every page	Send the kept URLs to Batch with `extract=auto`, see Scrape a list of URLs
Stop a crawl early	`POST /v1/crawls/{crawl_id}/stop`, see Stop a crawl

Troubleshooting

The crawl returns few URLs: include_patterns also decides which links Crawl opens. Add the paths of the listing or pagination pages that lead to your items, such as /page/ or /category/.
content_status reads pending: the crawl is still fetching that page. Wait for the crawl to end, then read the results again from the start.
stop_reason is max_items or max_pages: the site holds more. Start a new crawl with higher values.
429 with too_many_crawls: your account’s Crawl and Batch jobs are all running. Wait the Retry-After seconds, then start the crawl.
status: failed with seed_unreachable: the start URL didn’t load. Open it in a browser to check it, then start again.

Map every URL on a website

Build a URL inventory and export it as CSV.

Scrape a list of URLs

Feed the URLs you found into one Batch job.

Build a RAG knowledge base from a website

Crawl a site and turn every page into markdown chunks.

Extract data with CSS selectors

Parse each crawled page with BeautifulSoup or Cheerio.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Crawl a website from a start URL

Ask your AI agent

Start a crawl and read every page

Example output

Crawl with the SDKs

Crawl from the CLI

Customize it

Troubleshooting

Map every URL on a website

Scrape a list of URLs

Build a RAG knowledge base from a website

Extract data with CSS selectors

​Ask your AI agent

​Start a crawl and read every page

​Example output

​Crawl with the SDKs

​Crawl from the CLI

​Customize it

​Troubleshooting

​Related recipes

Map every URL on a website

Scrape a list of URLs

Build a RAG knowledge base from a website

Extract data with CSS selectors

Ask your AI agent

Start a crawl and read every page

Example output

Crawl with the SDKs

Crawl from the CLI

Customize it

Troubleshooting

Related recipes