Save every page of a site section

To crawl and read pages in memory, see Crawl a website from a start URL. This recipe saves them to disk: every product, article or docs page of a section as its own .html file, named from its URL path, ready to archive, parse offline or load into another tool. Uses: Crawl with output_format, the download and contents endpoints, and Batch with response_type=markdown for markdown copies

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to crawl https://www.scrapingcourse.com/ecommerce/ and keep the first five product
pages with their full HTML. Save each page to a pages folder as an .html file named after its
URL path, then tell me the file names and sizes.

Save the pages with the API

The download endpoint returns every page, HTML included, in one NDJSON file. The script writes each page to pages/, turning /ecommerce/product/aeon-capri/ into ecommerce_product_aeon-capri.html, so every file name is unique within the section.

import json
import time
from pathlib import Path
from urllib.parse import urlparse

import requests

API = "https://api.zenrows.com/v1/crawls"
headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}
out = Path("pages")
out.mkdir(exist_ok=True)

# 1. Crawl the section and keep each page's HTML
response = requests.post(
    API,
    headers=headers,
    json={
        "url": "https://www.scrapingcourse.com/ecommerce/",
        "include_patterns": ["/product/"],
        "max_items": 5,
        "max_pages": 10,
        "output_format": "html",
    },
    timeout=30,
)
response.raise_for_status()
crawl_id = response.json()["crawl_id"]

# 2. Wait until it ends
while requests.get(f"{API}/{crawl_id}", headers=headers, params={"limit": 1}, timeout=30).json()["status"] == "running":
    time.sleep(5)

# 3. Download every page in one NDJSON file and save each one
download = requests.get(f"{API}/{crawl_id}/download", headers=headers, timeout=300)
download.raise_for_status()
for line in download.text.splitlines():
    page = json.loads(line)
    if page.get("content_status") == "fetched":
        name = (urlparse(page["url"]).path.strip("/").replace("/", "_") or "index") + ".html"
        (out / name).write_text(page["content"], encoding="utf-8")
        print("Saved", out / name)

import { mkdir, writeFile } from "node:fs/promises";

const API = "https://api.zenrows.com/v1/crawls";
const headers = { "X-API-Key": "YOUR_ZENROWS_API_KEY", "Content-Type": "application/json" };
await mkdir("pages", { recursive: true });

// 1. Crawl the section and keep each page's HTML
const create = await fetch(API, {
  method: "POST",
  headers,
  body: JSON.stringify({
    url: "https://www.scrapingcourse.com/ecommerce/",
    include_patterns: ["/product/"],
    max_items: 5,
    max_pages: 10,
    output_format: "html",
  }),
});
if (!create.ok) throw new Error(`${create.status}: ${await create.text()}`);
const { crawl_id } = await create.json();

// 2. Wait until it ends
let status = "running";
while (status === "running") {
  await new Promise((resolve) => setTimeout(resolve, 5000));
  ({ status } = await (await fetch(`${API}/${crawl_id}?limit=1`, { headers })).json());
}

// 3. Download every page in one NDJSON file and save each one
const ndjson = await (await fetch(`${API}/${crawl_id}/download`, { headers })).text();
for (const line of ndjson.split("\n").filter(Boolean)) {
  const page = JSON.parse(line);
  if (page.content_status !== "fetched") continue;
  const name = (new URL(page.url).pathname.replace(/^\/+|\/+$/g, "").replaceAll("/", "_") || "index") + ".html";
  await writeFile(`pages/${name}`, page.content);
  console.log("Saved", `pages/${name}`);
}

Download once the crawl has ended, when every kept page reads content_status: fetched.

Save the pages from the CLI

--output-format html stores each kept page and --follow waits until the crawl ends. zenrows crawl content then writes each page to its own file. The script uses jq to read the results:

CLI

#!/bin/sh
set -e
mkdir -p pages

# 1. Crawl the section, keep each page's HTML, and wait until it ends
CRAWL_ID=$(zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --include-patterns /product/ \
  --max-items 5 --max-pages 10 --output-format html --follow --json | jq -r .crawl_id)
[ "$CRAWL_ID" != null ] || exit 1

# 2. Save each fetched page, named from its URL path
zenrows crawl results "$CRAWL_ID" \
  | jq -r 'select(.content_status == "fetched") | [.url, .content_url] | @tsv' \
  | while IFS="$(printf '\t')" read -r url content_url; do
      name=$(echo "$url" | sed -E 's#^https?://[^/]+/##; s#/$##; s#/#_#g')
      zenrows crawl content "$CRAWL_ID" "$content_url" --out "pages/${name:-index}.html"
    done

Output

✓ Wrote 120512 bytes (text/html; charset=utf-8) → pages/ecommerce_product_abominable-hoodie.html
✓ Wrote 119577 bytes (text/html; charset=utf-8) → pages/ecommerce_product_adrienne-trek-jacket.html
✓ Wrote 91543 bytes (text/html; charset=utf-8) → pages/ecommerce_product_aeon-capri.html
✓ Wrote 119256 bytes (text/html; charset=utf-8) → pages/ecommerce_product_aero-daily-fitness-tee.html
✓ Wrote 106662 bytes (text/html; charset=utf-8) → pages/ecommerce_product_aether-gym-pant.html

To keep the whole section in one file instead, run zenrows crawl download <crawl_id> --out section.jsonl: each line holds a page’s url, content_status and HTML in content. See the CLI guide.

Example output

The Python (Requests) run:

Saved pages/ecommerce_product_abominable-hoodie.html
Saved pages/ecommerce_product_adrienne-trek-jacket.html
Saved pages/ecommerce_product_aeon-capri.html
Saved pages/ecommerce_product_aero-daily-fitness-tee.html
Saved pages/ecommerce_product_aether-gym-pant.html

The pages folder afterwards, with each file’s size in bytes:

120556  ecommerce_product_abominable-hoodie.html
119577  ecommerce_product_adrienne-trek-jacket.html
 91543  ecommerce_product_aeon-capri.html
119188  ecommerce_product_aero-daily-fitness-tee.html
106662  ecommerce_product_aether-gym-pant.html

The Node.js version saved the same five pages. File sizes can vary slightly from run to run.

Save markdown copies too

For an LLM, a search index or plain reading, save each page as markdown as well. Send the saved URLs to one Batch job with response_type=markdown:

Python (SDK)

# pip install zenrows
from pathlib import Path
from urllib.parse import urlparse

from zenrows import ZenRowsBatchClient

urls = [
    "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
    "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/",
]
out = Path("pages")
out.mkdir(exist_ok=True)

batch = ZenRowsBatchClient(api_key="YOUR_ZENROWS_API_KEY")
job = batch.submit_regular(urls, zenrows_params={"mode": "auto", "response_type": "markdown"})
run = job.run.wait(timeout=900)

for task in run.results(status="successful"):
    name = (urlparse(str(task.url)).path.strip("/").replace("/", "_") or "index") + ".md"
    (out / name).write_bytes(run.download_task_to_memory(task))
    print("Saved", out / name)

Output

Saved pages/ecommerce_product_aeon-capri.md
Saved pages/ecommerce_product_abominable-hoodie.md

Each .md file holds the whole page as markdown, starting with the site’s navigation links. Batch saves the files in the order its tasks finish. To split the markdown into chunks for retrieval, see Build a RAG knowledge base from a website.

Customize it

Change	How
Save a different section	Start from the section URL and set `include_patterns` to its path, such as `["/blog/"]` or `["/docs/"]`
Save more pages	Raise `max_items`, and `max_pages` to at least `max_items` plus the listing pages Crawl walks, see Get each page’s HTML
Keep the folder structure	Write each page to `pages/<path>/index.html` instead of joining the path with `_`
Skip sections	Add `exclude_patterns`, for example `["/cart", "?orderby="]`
Save a list of URLs you already have	Skip the crawl and send the URLs to Batch, see Scrape a list of URLs
Refresh the copy on a schedule	Run the script from cron, and see Get alerted when a website adds new pages to save only new pages

Troubleshooting

Fewer files than pages in the section: include_patterns also decides which links Crawl opens. Add the path of the listing or pagination pages, such as /page/, and raise max_pages.
A result reads content_status: pending: the crawl is still fetching that page. Read the results after the crawl ends.
stop_reason is max_pages: each kept page’s HTML counts toward max_pages. Set max_pages to at least max_items plus the pages the crawl walks.
Two URLs save to the same file name: they differ only in their query string. Add the query to the name, for example urlparse(url).query, or exclude those URLs with exclude_patterns.
429 with too_many_crawls: other Crawl and Batch jobs on your account are still running. Wait the number of seconds in the Retry-After header, then start the crawl again.

Crawl a website from a start URL

Crawl a site and read each page in memory, with the SDKs.

Build a RAG knowledge base from a website

Turn a crawled site into markdown chunks.

Map every URL on a website

List a site’s URLs before you save them.

Extract data with CSS selectors

Parse the saved HTML with BeautifulSoup or Cheerio.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Save every page of a site section

Ask your AI agent

Save the pages with the API

Save the pages from the CLI

Example output

Save markdown copies too

Customize it

Troubleshooting

Crawl a website from a start URL

Build a RAG knowledge base from a website

Map every URL on a website

Extract data with CSS selectors

​Ask your AI agent

​Save the pages with the API

​Save the pages from the CLI

​Example output

​Save markdown copies too

​Customize it

​Troubleshooting

​Related recipes

Crawl a website from a start URL

Build a RAG knowledge base from a website

Map every URL on a website

Extract data with CSS selectors

Ask your AI agent

Save the pages with the API

Save the pages from the CLI

Example output

Save markdown copies too

Customize it

Troubleshooting

Related recipes