> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Find broken links on a website > Find every broken link on a website with Zenrows Crawl, and see which pages link to it. Find the links on your site that lead to a 404 or 410 page, and the exact pages that hold them, so you know what to fix. Crawl reads the site and keeps each page's HTML, your script collects the links on those pages, and Fetch checks each link and returns the target's status code. **Uses:** [Crawl](/crawl/introduction) with [`output_format`](/crawl/setup#get-each-pages-html), [Fetch](/fetch/api-reference) with [`allowed_status_codes`](/fetch/features/other#return-content-on-error), [Batch](/batch/introduction) ## How it works 1. **Crawl reads the site.** A crawl with `"output_format": "html"` keeps each matching page's HTML. The download endpoint returns every page in one NDJSON file. 2. **Your script collects the links.** It parses each page's `` links, keeps the ones on the site's own domain, and records which pages link to each one. 3. **Fetch checks each link.** With `allowed_status_codes=404,410`, Fetch answers `200` and puts the target's status in the `Zr-Status-Code` header. A link that loads normally has no `Zr-Status-Code` header and answers `200`. 4. **You get the report.** Every link with a status of 400 or more is listed with the pages to fix. ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use Zenrows to crawl https://www.scrapingcourse.com/ecommerce/ and read three product pages. Collect every link on those pages that stays on scrapingcourse.com, check whether each one loads, and list any link that returns "not found", with the pages that link to it. ``` ## Check every link on a site Save this as `broken_links.py` and run it. It needs only `requests`. ```python Python (Requests) theme={"dark"} import json import time from collections import defaultdict from concurrent.futures import ThreadPoolExecutor from html.parser import HTMLParser from urllib.parse import urldefrag, urljoin, urlparse import requests API_KEY = "YOUR_ZENROWS_API_KEY" START_URL = "https://www.scrapingcourse.com/ecommerce/" CRAWLS = "https://api.zenrows.com/v1/crawls" HEADERS = {"X-API-Key": API_KEY} SKIP = ["add-to-cart"] # links you don't want checked class LinkParser(HTMLParser): def __init__(self): super().__init__() self.hrefs = [] def handle_starttag(self, tag, attrs): if tag == "a": self.hrefs += [value for name, value in attrs if name == "href" and value] # 1. Crawl the site and keep each page's HTML response = requests.post( CRAWLS, headers=HEADERS, json={ "url": START_URL, "include_patterns": ["/product/"], "max_items": 3, "max_pages": 6, "output_format": "html", }, timeout=30, ) response.raise_for_status() crawl_id = response.json()["crawl_id"] while requests.get(f"{CRAWLS}/{crawl_id}", headers=HEADERS, params={"limit": 1}, timeout=30).json()["status"] == "running": time.sleep(5) # 2. Collect every link on the site's own domain, and the pages it appears on linked_from = defaultdict(set) download = requests.get(f"{CRAWLS}/{crawl_id}/download", headers=HEADERS, timeout=120) for line in download.text.splitlines(): page = json.loads(line) if page.get("content_status") != "fetched": continue parser = LinkParser() parser.feed(page["content"]) for href in parser.hrefs: link = urldefrag(urljoin(page["url"], href)).url if urlparse(link).netloc == urlparse(START_URL).netloc and not any(s in link for s in SKIP): linked_from[link].add(page["url"]) print(f"{len(linked_from)} links found on the crawled pages") # 3. Check each link: allowed_status_codes returns 404 and 410 pages with their status code def check(link: str) -> tuple[str, int]: params = {"apikey": API_KEY, "url": link, "mode": "auto", "allowed_status_codes": "404,410"} # stream=True: the status is in the headers, so the body is never downloaded with requests.get("https://api.zenrows.com/v1/", params=params, timeout=180, stream=True) as response: return link, int(response.headers.get("Zr-Status-Code", response.status_code)) with ThreadPoolExecutor(max_workers=5) as pool: statuses = dict(pool.map(check, linked_from)) # 4. Report the broken links and where they appear broken = {link: status for link, status in statuses.items() if status >= 400} print(f"{len(statuses)} links checked, {len(broken)} broken") for link, status in sorted(broken.items()): print(f"\n{status} {link}") for source in sorted(linked_from[link]): print(f" linked from {source}") ``` `stream=True` reads the response headers and leaves the page body undownloaded, which keeps each check fast. Each check is one Fetch request, and [`mode=auto`](/fetch/features/adaptive-stealth-mode) loads links on protected sites too. Fetch bills a 404 or 410 answer like any other page. ## Check one link To check a single URL, send it with `allowed_status_codes` and read the `Zr-Status-Code` header: ```bash cURL theme={"dark"} curl -s -o /dev/null -D - "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2Fproduct%2Fdoes-not-exist%2F&mode=auto&allowed_status_codes=404,410" | grep -i "^zr-status-code" ``` ```text Output theme={"dark"} Zr-Status-Code: 404 ``` Without `allowed_status_codes`, Fetch answers a missing page with its own `404` and the error code [`RESP002`](/api-error-codes#RESP002), which also tells you the link is broken. ## Check thousands of links with Batch For a large site, send the links to one [Batch](/academy/scrape-from-a-list-of-urls) job instead of a thread pool. Leave out `allowed_status_codes`: each link that answers 404 then becomes a `failed` row whose `error` holds `"status": 404` and `"code": "RESP002"`. ```python Python (Requests) theme={"dark"} import time import requests BATCH = "https://async.api.zenrows.com/v1" HEADERS = {"X-API-Key": "YOUR_ZENROWS_API_KEY"} links = [ "https://www.scrapingcourse.com/ecommerce/product/aeon-capri/", "https://www.scrapingcourse.com/ecommerce/product/does-not-exist/", ] job = requests.post( f"{BATCH}/jobs", headers=HEADERS, json={ "type": "regular", "status": "closed", "zenrows_params": {"mode": "auto"}, "tasks": [{"url": link} for link in links], }, timeout=60, ).json() while requests.get(f"{BATCH}/jobs/{job['job_id']}", headers=HEADERS, timeout=30).json()["latest_run"]["status"] not in ("completed", "failed", "stopped"): time.sleep(5) failed = requests.get(f"{BATCH}/jobs/{job['job_id']}/results", headers=HEADERS, params={"status": "failed"}, timeout=30).json() for row in failed["results"]: print(row["error"]["status"], row["error"]["code"], row["url"]) ``` ```text Output theme={"dark"} 404 RESP002 https://www.scrapingcourse.com/ecommerce/product/does-not-exist/ ``` For more than one page of results, follow `next_cursor` as shown in [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls). The run's `stats.failure_reasons` counts failures by kind, for example `{"bad_target": 1}`. ## Example output Every link on scrapingcourse.com loads, so this run adds one made-up URL, `/product/does-not-exist/`, to the links found on the Abominable Hoodie page before the checks. Everything else is the real run: ```text theme={"dark"} 42 links found on the crawled pages 42 links checked, 1 broken 404 https://www.scrapingcourse.com/ecommerce/product/does-not-exist/ linked from https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ ``` The crawl read three product pages, which hold 41 links on the site's own domain: category pages, other products, the cart and checkout pages, and product images. All 41 returned 200. ## Customize it | Change | How | | - | - | | Check the whole site | Remove `include_patterns` and raise `max_items` and `max_pages`, see [Start a crawl](/crawl/endpoints#start-a-crawl) | | Check one section | Start from the section URL and set `include_patterns` to its path, such as `["/blog/"]` | | Check outgoing links too | Drop the domain check in step 2 to keep links to other sites | | Check images and scripts | Also collect `src` attributes from `img` and `script` tags in `LinkParser` | | Report redirects | Compare the `Zr-Final-Url` header with the link to find links that redirect, see the [Fetch API reference](/fetch/api-reference) | | Treat more status codes as broken | Add them to `allowed_status_codes`, for example `404,410,500,503`, see [Return content on error](/fetch/features/other#return-content-on-error) | ## Troubleshooting * **A link reports `422` or `429` instead of a page status**: that code is Fetch's answer, not the target's. Check the link again. For `429`, set `max_workers` to the value of the `Concurrency-Limit` response header. * **The script finds few links**: the crawl kept few pages. Raise `max_items` and `max_pages`, and add the paths of listing pages to `include_patterns`. * **Links show up twice with and without a trailing slash**: the site links to both spellings. Normalize them before checking, for example with `link.rstrip("/")`. * **`429` with `too_many_crawls`**: other Crawl and Batch jobs on your account are still running. Wait the number of seconds in the `Retry-After` header, then start the crawl again. ## Related recipes List a site's URLs without their HTML. Compare a section's URLs run after run. Send thousands of URLs as one Batch job. Get every matching page with its HTML. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.