> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.
# Build a RAG knowledge base from a website
> Turn a whole website into a searchable knowledge base for your AI assistant: crawl it with Zenrows, get every page as markdown, and index it.
Turn a website into a knowledge base your LLM can search: Crawl finds the pages, Batch returns each one as clean markdown, and your code splits the markdown by headings, embeds the chunks and stores them. Ask a question, retrieve the best chunks, and send them to your model with their source URLs.
**Uses:** [Crawl](/crawl/introduction), [Batch](/batch/introduction) with [`response_type=markdown`](/fetch/features/markdown) and [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode) (`mode=auto`)
## How it works
1. **Crawl finds the pages.** Crawl starts at one URL, follows links on the same domain, and keeps the URLs that match `include_patterns`.
2. **Batch fetches every page as markdown.** One Batch job runs every URL with `response_type=markdown` and `mode=auto`. Headings, lists, links and tables stay; scripts and styling go.
3. **Your code chunks by headings.** Each `#` to `######` heading starts a new chunk, and the chunk keeps its heading path (`Product > Description`) as context. Text before the first heading, usually site navigation, is skipped.
4. **Your embedding model indexes the chunks.** Every chunk becomes a vector in your store, with its URL and heading as metadata.
5. **Search returns the best chunks.** A question is embedded the same way, and the closest chunks go into your LLM prompt.
Steps 1 and 2 are Zenrows. Steps 3 to 5 are yours to adapt: the chunker below is complete, and the embedding function and store are pluggable.
## Ask your AI agent
With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.
```text theme={"dark"}
Use Zenrows to find the product pages linked from https://www.scrapingcourse.com/ecommerce/,
read the first three, and answer: which of these products is made of organic cotton? Cite the URL you used.
```
For a full site, ask the agent to crawl every page and scrape them in one batch: the pages arrive as markdown, ready to chunk.
## Build the index
Install the [Zenrows SDK](/sdk/overview) with `pip install zenrows`. The script crawls three product pages, fetches them as markdown, chunks them and searches them. Raise `max_items` to index a whole site.
```python Python (SDK) theme={"dark"}
import hashlib
import math
import re
from zenrows import ZenRowsBatchClient, ZenRowsCrawlClient
API_KEY = "YOUR_ZENROWS_API_KEY"
START_URL = "https://www.scrapingcourse.com/ecommerce/"
# 1. Crawl: find the pages to index
crawler = ZenRowsCrawlClient(api_key=API_KEY)
crawl = crawler.create(START_URL, depth=1, max_items=3, include_patterns=["/product/"])
crawler.wait(crawl.crawl_id)
urls = [result.url for result in crawler.results(crawl.crawl_id)]
print(f"Crawl found {len(urls)} pages")
# 2. Batch: get every page as markdown
batch = ZenRowsBatchClient(api_key=API_KEY)
job = batch.submit_regular(urls, zenrows_params={"mode": "auto", "response_type": "markdown"})
run = job.run.wait(timeout=900)
pages = {
str(task.url): run.download_task_to_memory(task).decode()
for task in run.results(status="successful")
}
print(f"Batch returned {len(pages)} markdown pages")
# 3. Chunk: one chunk per heading section, with the heading path kept as context
def chunk_markdown(markdown: str, url: str, max_chars: int = 2000) -> list[dict]:
chunks, path, lines = [], [], []
def flush():
text = "\n".join(lines).strip()
if text and path: # skip text before the first heading (site navigation)
for start in range(0, len(text), max_chars):
chunks.append({
"url": url,
"heading": " > ".join(path),
"text": text[start:start + max_chars],
})
lines.clear()
for line in markdown.splitlines():
match = re.match(r"^(#{1,6})\s+(.*)", line)
if match:
flush()
level = len(match.group(1))
path[:] = path[: level - 1] + [match.group(2).strip()]
else:
lines.append(line)
flush()
return chunks
chunks = [chunk for url, md in pages.items() for chunk in chunk_markdown(md, url)]
print(f"Split into {len(chunks)} chunks")
# 4. Embed and store: swap embed() for your embedding model
def embed(text: str, dims: int = 512) -> list[float]:
"""Local stand-in: a hashed bag of words. Replace with OpenAI, Voyage, Cohere, etc."""
vector = [0.0] * dims
for word in re.findall(r"[a-z0-9]+", text.lower()):
vector[int(hashlib.md5(word.encode()).hexdigest(), 16) % dims] += 1.0
norm = math.sqrt(sum(v * v for v in vector)) or 1.0
return [v / norm for v in vector]
store = [{**chunk, "vector": embed(f"{chunk['heading']}\n{chunk['text']}")} for chunk in chunks]
# 5. Search: the closest chunks by cosine similarity
def search(query: str, k: int = 3) -> list[dict]:
q = embed(query)
scored = [(sum(a * b for a, b in zip(q, item["vector"])), item) for item in store]
return [item for _, item in sorted(scored, key=lambda pair: pair[0], reverse=True)[:k]]
for hit in search("Which jacket is made of organic cotton?"):
print(f"- {hit['heading']} ({hit['url']})\n {hit['text'][:100]!r}")
```
The `embed()` function above is a dependency-free stand-in so the script runs as pasted. It matches words, not meaning: swap it for a real embedding model before you rely on the results.
## Example output
```text theme={"dark"}
Crawl found 3 pages
Batch returned 3 markdown pages
Split into 11 chunks
- Adrienne Trek Jacket > Description (https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/)
'You’re ready for a cross-country jog or a coffee on the patio in the Adrienne Trek Jacket. Its style'
- Adrienne Trek Jacket (https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/)
'$57.00\n\nThis is a variable product called a Adrienne Trek Jacket\n\nSize Choose an optionXSSMLXL Color'
- Abominable Hoodie (https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/)
'$69.00\n\nThis is a variable product called a Abominable Hoodie\n\nSize Choose an optionXSSMLXL Color Ch'
```
The top hit is the jacket's `Description` section, which lists "95% Organic Cotton / 5% Spandex".
## Plug in your embedding model and vector store
These two snippets are one choice among many: use the embedding model and vector database your stack already has. Keep the chunk's `url` and `heading` as metadata, so every answer can cite its source.
**An embedding model.** Replace `embed()` with a call to your provider. With OpenAI (`pip install openai`, `OPENAI_API_KEY` set):
```python Python theme={"dark"}
from openai import OpenAI
openai_client = OpenAI()
def embed(text: str) -> list[float]:
response = openai_client.embeddings.create(model="text-embedding-3-small", input=text)
return response.data[0].embedding
```
**A vector database.** Replace the in-memory `store` and `search()` with your database. With Chroma (`pip install chromadb`), passing the vectors from `embed()`:
```python Python theme={"dark"}
import chromadb
collection = chromadb.PersistentClient(path="./rag-index").get_or_create_collection("website")
collection.add(
ids=[f"{i}" for i in range(len(chunks))],
documents=[chunk["text"] for chunk in chunks],
embeddings=[embed(f"{chunk['heading']}\n{chunk['text']}") for chunk in chunks],
metadatas=[{"url": chunk["url"], "heading": chunk["heading"]} for chunk in chunks],
)
hits = collection.query(query_embeddings=[embed("Which jacket is made of organic cotton?")], n_results=3)
print(hits["metadatas"][0])
```
Then put the retrieved chunks in your prompt, each with its URL, and ask the model to answer from them only.
## Customize it
| Change | How |
| - | - |
| Index a whole site | Remove `depth` and raise `max_items` and `max_pages`, see [Crawl endpoints](/crawl/endpoints#start-a-crawl) |
| Index one section | Set `include_patterns` to its path, for example `["/docs/"]`, and `exclude_patterns` for paths to skip |
| Plain text instead of markdown | Set `response_type` to `plaintext` and chunk by paragraphs, see [Plain text](/fetch/features/plaintext) |
| Smaller or larger chunks | Change `max_chars`, or split on `##` only by changing `#{1,6}` to `#{1,2}` |
| Keep the index fresh | Submit the Batch step as a scheduled job and re-embed only pages whose content hash changed, see [Detect changes on any page](/academy/pipelines/change-detection) |
| Pages you already know | Skip the crawl and pass your URL list to the Batch step, see [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls) |
## Troubleshooting
* **The crawl keeps fewer pages than expected**: `include_patterns` also decides which links Crawl opens. Add the patterns of the listing or index pages that link to your content, or remove the patterns and use `exclude_patterns` instead.
* **Chunks contain menus and footers**: they sit under a heading inside the page template. Drop chunks whose heading matches template sections, or filter lines that are only links.
* **Answers cite the wrong page**: chunks are too large and mix topics. Lower `max_chars` so each chunk covers one idea.
* **Some Batch tasks failed**: rerun only the failures with `job.retry_failed()`. Successful results carry over.
## Related recipes
How `response_type=markdown` works on one page.
Run Crawl on its own to list a site's pages.
Let an LLM read the web on demand instead of from an index.
Send thousands of URLs as one Batch job.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.