> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Get alerted when a website adds new pages > Get alerted when a website adds or removes pages: crawl it on a schedule with Zenrows Crawl and compare each run with the last. Know the moment a competitor lists a new product, a blog publishes a post or a job board adds a role. Each run, Crawl lists every URL in the section you watch, your script compares that list with the one saved last time, and you get an alert with the pages that appeared or went away. **Uses:** [Crawl](/crawl/introduction) with [`include_patterns` and `depth`](/crawl/endpoints#start-a-crawl), and the [download endpoint](/crawl/endpoints#download-the-results) ## How it works 1. **Crawl lists the section.** A URL crawl (no `output_format`) follows the links from your start URL and keeps the URLs that match `include_patterns`. Each page Crawl opens is one Fetch request. 2. **Your script diffs the URL sets.** It compares the URLs with the set saved in `known_urls.json`: URLs only in the new set are new pages, URLs only in the old set are removed pages. 3. **The alert goes out.** New and removed pages go to a Slack incoming webhook when `SLACK_WEBHOOK_URL` is set, and to the console otherwise. 4. **A schedule repeats it.** cron, a CI scheduler or any job runner starts the script every hour or every day. ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use Zenrows to list every product page linked from https://www.scrapingcourse.com/ecommerce/ and save the list to known_products.txt. Next time I ask, list them again, compare with the saved file, and tell me which product pages are new and which are gone. ``` ## Watch a section for new pages Set the section in `CRAWL` and save this as `new_pages.py`. The first run saves a baseline; every later run reports what changed since the previous one. ```python Python (Requests) theme={"dark"} import json import os import time from pathlib import Path import requests API = "https://api.zenrows.com/v1/crawls" HEADERS = {"X-API-Key": "YOUR_ZENROWS_API_KEY"} SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL") # optional STATE = Path("known_urls.json") # The section to watch CRAWL = { "url": "https://www.scrapingcourse.com/ecommerce/", "include_patterns": ["/product/"], "depth": 1, "max_items": 1000, } def crawl_section() -> tuple[dict, set[str]]: """Run one crawl and return its final state and every URL it kept.""" response = requests.post(API, headers=HEADERS, json=CRAWL, timeout=30) response.raise_for_status() crawl_id = response.json()["crawl_id"] while True: crawl = requests.get(f"{API}/{crawl_id}", headers=HEADERS, params={"limit": 1}, timeout=30).json() if crawl["status"] != "running": break time.sleep(5) download = requests.get(f"{API}/{crawl_id}/download", headers=HEADERS, timeout=120) download.raise_for_status() urls = {json.loads(line)["url"] for line in download.text.splitlines() if line.strip()} return crawl, urls def alert(message: str) -> None: if SLACK_WEBHOOK_URL: requests.post(SLACK_WEBHOOK_URL, json={"text": message}, timeout=30) else: print(message) def main() -> None: crawl, current = crawl_section() if crawl["status"] == "failed": raise SystemExit(f"Crawl {crawl['crawl_id']} failed: {crawl['error']}") print(f"Crawl {crawl['crawl_id']}: {len(current)} URLs, {crawl['status']}") if not STATE.exists(): STATE.write_text(json.dumps(sorted(current), indent=2)) print("Baseline saved") return previous = set(json.loads(STATE.read_text())) complete = "stop_reason" not in crawl # the crawl covered the whole section added = sorted(current - previous) removed = sorted(previous - current) if complete else [] if added: alert(f"{len(added)} new page(s) on {CRAWL['url']}\n" + "\n".join(f"+ {url}" for url in added)) if removed: alert(f"{len(removed)} page(s) gone from {CRAWL['url']}\n" + "\n".join(f"- {url}" for url in removed)) if not added and not removed: print("No new or removed pages") # Keep URLs a partial crawl did not reach, so they don't count as new next time STATE.write_text(json.dumps(sorted(current if complete else current | previous), indent=2)) if __name__ == "__main__": main() ``` A crawl without `stop_reason` reached every matching page, so a missing URL really is gone. When `max_items` or `max_pages` ends the crawl first, the script reports new pages only and keeps the URLs it did not reach. To send alerts to Slack, create an incoming webhook and export its URL as `SLACK_WEBHOOK_URL`. The same `{"text": ...}` payload works with Microsoft Teams workflows and most chat tools. ## Run it on a schedule Start the script from cron. This checks every morning at 08:00: ```bash theme={"dark"} 0 8 * * * cd /path/to/watch && /usr/bin/python3 new_pages.py >> new_pages.log 2>&1 ``` Any scheduler works the same way: a GitHub Actions `schedule` workflow, a Kubernetes CronJob or your task queue. Keep `known_urls.json` somewhere that persists between runs. ## Watch from the CLI The same job as a shell script, with the [Zenrows CLI](/cli/crawl/introduction), `jq` and `comm`. `--follow` waits until the crawl ends and `--json` prints the `crawl_id`: ```bash CLI theme={"dark"} #!/bin/sh set -e START_URL="https://www.scrapingcourse.com/ecommerce/" # 1. Crawl the section and wait until it ends CRAWL_ID=$(zenrows crawl create "$START_URL" --include-patterns /product/ --depth 1 \ --max-items 1000 --follow --json | jq -r .crawl_id) [ "$CRAWL_ID" != null ] || exit 1 # 2. Save the sorted URL list zenrows crawl download "$CRAWL_ID" --out latest.jsonl jq -r .url latest.jsonl | sort > current.txt # 3. Compare with the previous run if [ -f known.txt ]; then comm -13 known.txt current.txt | sed 's/^/+ /' # new pages comm -23 known.txt current.txt | sed 's/^/- /' # removed pages else echo "Baseline saved: $(wc -l < current.txt | tr -d ' ') URLs" fi mv current.txt known.txt ``` ## Example output The first run of `new_pages.py` saves the baseline: ```text theme={"dark"} Crawl c_01M4YGK3M8JWQY4DNV1CZ91982: 16 URLs, completed Baseline saved ``` The test site's catalog stays the same between runs, so to show a change, edit `known_urls.json` before the second run: remove the Aeon Capri URL and add a made-up Zeppelin Yoga Pant URL. The second run reports one new page and one removed page: ```text theme={"dark"} Crawl c_01M4YGKJKB2GCA1KNDYSH313PG: 16 URLs, completed 1 new page(s) on https://www.scrapingcourse.com/ecommerce/ + https://www.scrapingcourse.com/ecommerce/product/aeon-capri/ 1 page(s) gone from https://www.scrapingcourse.com/ecommerce/ - https://www.scrapingcourse.com/ecommerce/product/zeppelin-yoga-pant/ ``` A run with nothing to report: ```text theme={"dark"} Crawl c_01M4YHV4BRH4664V89JY8TFA4E: 16 URLs, completed No new or removed pages ``` The CLI script printed the same change, `+ .../aeon-capri/` and `- .../zeppelin-yoga-pant/`, after the same simulated edit of `known.txt`. With `depth: 1`, each run opens one page, the start page, and costs one Fetch request. ## Customize it | Change | How | | - | - | | Watch the whole section, not one page of it | Remove `depth` and add the listing path to `include_patterns`, for example `["/product/", "/page/"]`, then raise `max_pages`, see [Map every URL on a website](/academy/crawl/map-a-website) | | Watch a blog or a docs site | Start from the section URL and set `include_patterns` to its path, such as `["/blog/"]` | | Skip filter and sorting URLs | Add `exclude_patterns`, for example `["?orderby=", "?filter"]` | | Read each new page | Fetch only the `added` URLs, or send them to Batch, see [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls) | | Watch what changes inside a page | Use [Detect changes on any page and get alerted](/academy/pipelines/change-detection) | | Alert by email | Replace the body of `alert()` with your email provider's API call | ## Troubleshooting * **Pages show up as new that existed before**: the earlier crawl ended on `max_items` or `max_pages` before reaching them. Raise both values so the crawl finishes without a `stop_reason`. * **Every run reports the same URLs as new and removed**: the URLs carry a parameter that changes per visit, such as a session ID. Add that parameter to `exclude_patterns`, or strip it before comparing. * **The crawl keeps fewer URLs than the section holds**: `include_patterns` also decides which links Crawl opens. Add the path of the listing or pagination pages that lead to the items. * **`429` with `too_many_crawls`**: other Crawl and Batch jobs on your account are still running. Wait the number of seconds in the `Retry-After` header, then start the crawl again. ## Related recipes Build the full URL inventory of a site. Diff the content of known pages on a schedule. Crawl a site and check every link it holds. Get every matching page with its HTML. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.