> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Save every page of a site section > Save every page of a website section to disk with Zenrows Crawl, as HTML files and optionally as markdown. To crawl and read pages in memory, see [Crawl a website from a start URL](/academy/scrape-and-crawl-from-a-seed-url). This recipe saves them to disk: every product, article or docs page of a section as its own `.html` file, named from its URL path, ready to archive, parse offline or load into another tool. **Uses:** [Crawl](/crawl/introduction) with [`output_format`](/crawl/setup#get-each-pages-html), the [download](/crawl/endpoints#download-the-results) and [contents](/crawl/endpoints#read-a-page) endpoints, and [Batch](/batch/introduction) with [`response_type=markdown`](/fetch/features/markdown) for markdown copies ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use Zenrows to crawl https://www.scrapingcourse.com/ecommerce/ and keep the first five product pages with their full HTML. Save each page to a pages folder as an .html file named after its URL path, then tell me the file names and sizes. ``` ## Save the pages with the API The download endpoint returns every page, HTML included, in one NDJSON file. The script writes each page to `pages/`, turning `/ecommerce/product/aeon-capri/` into `ecommerce_product_aeon-capri.html`, so every file name is unique within the section. ```python Python (Requests) theme={"dark"} import json import time from pathlib import Path from urllib.parse import urlparse import requests API = "https://api.zenrows.com/v1/crawls" headers = {"X-API-Key": "YOUR_ZENROWS_API_KEY"} out = Path("pages") out.mkdir(exist_ok=True) # 1. Crawl the section and keep each page's HTML response = requests.post( API, headers=headers, json={ "url": "https://www.scrapingcourse.com/ecommerce/", "include_patterns": ["/product/"], "max_items": 5, "max_pages": 10, "output_format": "html", }, timeout=30, ) response.raise_for_status() crawl_id = response.json()["crawl_id"] # 2. Wait until it ends while requests.get(f"{API}/{crawl_id}", headers=headers, params={"limit": 1}, timeout=30).json()["status"] == "running": time.sleep(5) # 3. Download every page in one NDJSON file and save each one download = requests.get(f"{API}/{crawl_id}/download", headers=headers, timeout=300) download.raise_for_status() for line in download.text.splitlines(): page = json.loads(line) if page.get("content_status") == "fetched": name = (urlparse(page["url"]).path.strip("/").replace("/", "_") or "index") + ".html" (out / name).write_text(page["content"], encoding="utf-8") print("Saved", out / name) ``` ```javascript Node.js theme={"dark"} import { mkdir, writeFile } from "node:fs/promises"; const API = "https://api.zenrows.com/v1/crawls"; const headers = { "X-API-Key": "YOUR_ZENROWS_API_KEY", "Content-Type": "application/json" }; await mkdir("pages", { recursive: true }); // 1. Crawl the section and keep each page's HTML const create = await fetch(API, { method: "POST", headers, body: JSON.stringify({ url: "https://www.scrapingcourse.com/ecommerce/", include_patterns: ["/product/"], max_items: 5, max_pages: 10, output_format: "html", }), }); if (!create.ok) throw new Error(`${create.status}: ${await create.text()}`); const { crawl_id } = await create.json(); // 2. Wait until it ends let status = "running"; while (status === "running") { await new Promise((resolve) => setTimeout(resolve, 5000)); ({ status } = await (await fetch(`${API}/${crawl_id}?limit=1`, { headers })).json()); } // 3. Download every page in one NDJSON file and save each one const ndjson = await (await fetch(`${API}/${crawl_id}/download`, { headers })).text(); for (const line of ndjson.split("\n").filter(Boolean)) { const page = JSON.parse(line); if (page.content_status !== "fetched") continue; const name = (new URL(page.url).pathname.replace(/^\/+|\/+$/g, "").replaceAll("/", "_") || "index") + ".html"; await writeFile(`pages/${name}`, page.content); console.log("Saved", `pages/${name}`); } ``` Download once the crawl has ended, when every kept page reads `content_status: fetched`. ## Save the pages from the CLI `--output-format html` stores each kept page and `--follow` waits until the crawl ends. `zenrows crawl content` then writes each page to its own file. The script uses `jq` to read the results: ```bash CLI theme={"dark"} #!/bin/sh set -e mkdir -p pages # 1. Crawl the section, keep each page's HTML, and wait until it ends CRAWL_ID=$(zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --include-patterns /product/ \ --max-items 5 --max-pages 10 --output-format html --follow --json | jq -r .crawl_id) [ "$CRAWL_ID" != null ] || exit 1 # 2. Save each fetched page, named from its URL path zenrows crawl results "$CRAWL_ID" \ | jq -r 'select(.content_status == "fetched") | [.url, .content_url] | @tsv' \ | while IFS="$(printf '\t')" read -r url content_url; do name=$(echo "$url" | sed -E 's#^https?://[^/]+/##; s#/$##; s#/#_#g') zenrows crawl content "$CRAWL_ID" "$content_url" --out "pages/${name:-index}.html" done ``` ```text Output theme={"dark"} ✓ Wrote 120512 bytes (text/html; charset=utf-8) → pages/ecommerce_product_abominable-hoodie.html ✓ Wrote 119577 bytes (text/html; charset=utf-8) → pages/ecommerce_product_adrienne-trek-jacket.html ✓ Wrote 91543 bytes (text/html; charset=utf-8) → pages/ecommerce_product_aeon-capri.html ✓ Wrote 119256 bytes (text/html; charset=utf-8) → pages/ecommerce_product_aero-daily-fitness-tee.html ✓ Wrote 106662 bytes (text/html; charset=utf-8) → pages/ecommerce_product_aether-gym-pant.html ``` To keep the whole section in one file instead, run `zenrows crawl download --out section.jsonl`: each line holds a page's `url`, `content_status` and HTML in `content`. See the [CLI guide](/cli/crawl/introduction). ## Example output The Python (Requests) run: ```text theme={"dark"} Saved pages/ecommerce_product_abominable-hoodie.html Saved pages/ecommerce_product_adrienne-trek-jacket.html Saved pages/ecommerce_product_aeon-capri.html Saved pages/ecommerce_product_aero-daily-fitness-tee.html Saved pages/ecommerce_product_aether-gym-pant.html ``` The `pages` folder afterwards, with each file's size in bytes: ```text theme={"dark"} 120556 ecommerce_product_abominable-hoodie.html 119577 ecommerce_product_adrienne-trek-jacket.html 91543 ecommerce_product_aeon-capri.html 119188 ecommerce_product_aero-daily-fitness-tee.html 106662 ecommerce_product_aether-gym-pant.html ``` The Node.js version saved the same five pages. File sizes can vary slightly from run to run. ## Save markdown copies too For an LLM, a search index or plain reading, save each page as markdown as well. Send the saved URLs to one Batch job with `response_type=markdown`: ```python Python (SDK) theme={"dark"} # pip install zenrows from pathlib import Path from urllib.parse import urlparse from zenrows import ZenRowsBatchClient urls = [ "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/", "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", ] out = Path("pages") out.mkdir(exist_ok=True) batch = ZenRowsBatchClient(api_key="YOUR_ZENROWS_API_KEY") job = batch.submit_regular(urls, zenrows_params={"mode": "auto", "response_type": "markdown"}) run = job.run.wait(timeout=900) for task in run.results(status="successful"): name = (urlparse(str(task.url)).path.strip("/").replace("/", "_") or "index") + ".md" (out / name).write_bytes(run.download_task_to_memory(task)) print("Saved", out / name) ``` ```text Output theme={"dark"} Saved pages/ecommerce_product_aeon-capri.md Saved pages/ecommerce_product_abominable-hoodie.md ``` Each `.md` file holds the whole page as markdown, starting with the site's navigation links. Batch saves the files in the order its tasks finish. To split the markdown into chunks for retrieval, see [Build a RAG knowledge base from a website](/academy/pipelines/website-to-rag). ## Customize it | Change | How | | - | - | | Save a different section | Start from the section URL and set `include_patterns` to its path, such as `["/blog/"]` or `["/docs/"]` | | Save more pages | Raise `max_items`, and `max_pages` to at least `max_items` plus the listing pages Crawl walks, see [Get each page's HTML](/crawl/setup#get-each-pages-html) | | Keep the folder structure | Write each page to `pages//index.html` instead of joining the path with `_` | | Skip sections | Add `exclude_patterns`, for example `["/cart", "?orderby="]` | | Save a list of URLs you already have | Skip the crawl and send the URLs to Batch, see [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls) | | Refresh the copy on a schedule | Run the script from cron, and see [Get alerted when a website adds new pages](/academy/crawl/new-pages) to save only new pages | ## Troubleshooting * **Fewer files than pages in the section**: `include_patterns` also decides which links Crawl opens. Add the path of the listing or pagination pages, such as `/page/`, and raise `max_pages`. * **A result reads `content_status: pending`**: the crawl is still fetching that page. Read the results after the crawl ends. * **`stop_reason` is `max_pages`**: each kept page's HTML counts toward `max_pages`. Set `max_pages` to at least `max_items` plus the pages the crawl walks. * **Two URLs save to the same file name**: they differ only in their query string. Add the query to the name, for example `urlparse(url).query`, or exclude those URLs with `exclude_patterns`. * **`429` with `too_many_crawls`**: other Crawl and Batch jobs on your account are still running. Wait the number of seconds in the `Retry-After` header, then start the crawl again. ## Related recipes Crawl a site and read each page in memory, with the SDKs. Turn a crawled site into markdown chunks. List a site's URLs before you save them. Parse the saved HTML with BeautifulSoup or Cheerio. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.