Build a RAG knowledge base from a website

Turn a website into a knowledge base your LLM can search: Crawl finds the pages, Batch returns each one as clean markdown, and your code splits the markdown by headings, embeds the chunks and stores them. Ask a question, retrieve the best chunks, and send them to your model with their source URLs. Uses: Crawl, Batch with response_type=markdown and Adaptive Stealth Mode (mode=auto)

How it works

Crawl finds the pages. Crawl starts at one URL, follows links on the same domain, and keeps the URLs that match include_patterns.
Batch fetches every page as markdown. One Batch job runs every URL with response_type=markdown and mode=auto. Headings, lists, links and tables stay; scripts and styling go.
Your code chunks by headings. Each # to ###### heading starts a new chunk, and the chunk keeps its heading path (Product > Description) as context. Text before the first heading, usually site navigation, is skipped.
Your embedding model indexes the chunks. Every chunk becomes a vector in your store, with its URL and heading as metadata.
Search returns the best chunks. A question is embedded the same way, and the closest chunks go into your LLM prompt.

Steps 1 and 2 are Zenrows. Steps 3 to 5 are yours to adapt: the chunker below is complete, and the embedding function and store are pluggable.

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to find the product pages linked from https://www.scrapingcourse.com/ecommerce/,
read the first three, and answer: which of these products is made of organic cotton? Cite the URL you used.

For a full site, ask the agent to crawl every page and scrape them in one batch: the pages arrive as markdown, ready to chunk.

Build the index

Install the Zenrows SDK with pip install zenrows. The script crawls three product pages, fetches them as markdown, chunks them and searches them. Raise max_items to index a whole site.

Python (SDK)

import hashlib
import math
import re

from zenrows import ZenRowsBatchClient, ZenRowsCrawlClient

API_KEY = "YOUR_ZENROWS_API_KEY"
START_URL = "https://www.scrapingcourse.com/ecommerce/"

# 1. Crawl: find the pages to index
crawler = ZenRowsCrawlClient(api_key=API_KEY)
crawl = crawler.create(START_URL, depth=1, max_items=3, include_patterns=["/product/"])
crawler.wait(crawl.crawl_id)
urls = [result.url for result in crawler.results(crawl.crawl_id)]
print(f"Crawl found {len(urls)} pages")

# 2. Batch: get every page as markdown
batch = ZenRowsBatchClient(api_key=API_KEY)
job = batch.submit_regular(urls, zenrows_params={"mode": "auto", "response_type": "markdown"})
run = job.run.wait(timeout=900)
pages = {
    str(task.url): run.download_task_to_memory(task).decode()
    for task in run.results(status="successful")
}
print(f"Batch returned {len(pages)} markdown pages")

# 3. Chunk: one chunk per heading section, with the heading path kept as context
def chunk_markdown(markdown: str, url: str, max_chars: int = 2000) -> list[dict]:
    chunks, path, lines = [], [], []

    def flush():
        text = "\n".join(lines).strip()
        if text and path:  # skip text before the first heading (site navigation)
            for start in range(0, len(text), max_chars):
                chunks.append({
                    "url": url,
                    "heading": " > ".join(path),
                    "text": text[start:start + max_chars],
                })
        lines.clear()

    for line in markdown.splitlines():
        match = re.match(r"^(#{1,6})\s+(.*)", line)
        if match:
            flush()
            level = len(match.group(1))
            path[:] = path[: level - 1] + [match.group(2).strip()]
        else:
            lines.append(line)
    flush()
    return chunks

chunks = [chunk for url, md in pages.items() for chunk in chunk_markdown(md, url)]
print(f"Split into {len(chunks)} chunks")

# 4. Embed and store: swap embed() for your embedding model
def embed(text: str, dims: int = 512) -> list[float]:
    """Local stand-in: a hashed bag of words. Replace with OpenAI, Voyage, Cohere, etc."""
    vector = [0.0] * dims
    for word in re.findall(r"[a-z0-9]+", text.lower()):
        vector[int(hashlib.md5(word.encode()).hexdigest(), 16) % dims] += 1.0
    norm = math.sqrt(sum(v * v for v in vector)) or 1.0
    return [v / norm for v in vector]

store = [{**chunk, "vector": embed(f"{chunk['heading']}\n{chunk['text']}")} for chunk in chunks]

# 5. Search: the closest chunks by cosine similarity
def search(query: str, k: int = 3) -> list[dict]:
    q = embed(query)
    scored = [(sum(a * b for a, b in zip(q, item["vector"])), item) for item in store]
    return [item for _, item in sorted(scored, key=lambda pair: pair[0], reverse=True)[:k]]

for hit in search("Which jacket is made of organic cotton?"):
    print(f"- {hit['heading']} ({hit['url']})\n  {hit['text'][:100]!r}")

The embed() function above is a dependency-free stand-in so the script runs as pasted. It matches words, not meaning: swap it for a real embedding model before you rely on the results.

Example output

Crawl found 3 pages
Batch returned 3 markdown pages
Split into 11 chunks
- Adrienne Trek Jacket > Description (https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/)
  'You’re ready for a cross-country jog or a coffee on the patio in the Adrienne Trek Jacket. Its style'
- Adrienne Trek Jacket (https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/)
  '$57.00\n\nThis is a variable product called a Adrienne Trek Jacket\n\nSize Choose an optionXSSMLXL Color'
- Abominable Hoodie (https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/)
  '$69.00\n\nThis is a variable product called a Abominable Hoodie\n\nSize Choose an optionXSSMLXL Color Ch'

The top hit is the jacket’s Description section, which lists “95% Organic Cotton / 5% Spandex”.

Plug in your embedding model and vector store

These two snippets are one choice among many: use the embedding model and vector database your stack already has. Keep the chunk’s url and heading as metadata, so every answer can cite its source. An embedding model. Replace embed() with a call to your provider. With OpenAI (pip install openai, OPENAI_API_KEY set):

Python

from openai import OpenAI

openai_client = OpenAI()

def embed(text: str) -> list[float]:
    response = openai_client.embeddings.create(model="text-embedding-3-small", input=text)
    return response.data[0].embedding

A vector database. Replace the in-memory store and search() with your database. With Chroma (pip install chromadb), passing the vectors from embed():

Python

import chromadb

collection = chromadb.PersistentClient(path="./rag-index").get_or_create_collection("website")
collection.add(
    ids=[f"{i}" for i in range(len(chunks))],
    documents=[chunk["text"] for chunk in chunks],
    embeddings=[embed(f"{chunk['heading']}\n{chunk['text']}") for chunk in chunks],
    metadatas=[{"url": chunk["url"], "heading": chunk["heading"]} for chunk in chunks],
)

hits = collection.query(query_embeddings=[embed("Which jacket is made of organic cotton?")], n_results=3)
print(hits["metadatas"][0])

Then put the retrieved chunks in your prompt, each with its URL, and ask the model to answer from them only.

Customize it

Change	How
Index a whole site	Remove `depth` and raise `max_items` and `max_pages`, see Crawl endpoints
Index one section	Set `include_patterns` to its path, for example `["/docs/"]`, and `exclude_patterns` for paths to skip
Plain text instead of markdown	Set `response_type` to `plaintext` and chunk by paragraphs, see Plain text
Smaller or larger chunks	Change `max_chars`, or split on `##` only by changing `#{1,6}` to `#{1,2}`
Keep the index fresh	Submit the Batch step as a scheduled job and re-embed only pages whose content hash changed, see Detect changes on any page
Pages you already know	Skip the crawl and pass your URL list to the Batch step, see Scrape a list of URLs

Troubleshooting

The crawl keeps fewer pages than expected: include_patterns also decides which links Crawl opens. Add the patterns of the listing or index pages that link to your content, or remove the patterns and use exclude_patterns instead.
Chunks contain menus and footers: they sit under a heading inside the page template. Drop chunks whose heading matches template sections, or filter lines that are only links.
Answers cite the wrong page: chunks are too large and mix topics. Lower max_chars so each chunk covers one idea.
Some Batch tasks failed: rerun only the failures with job.retry_failed(). Successful results carry over.

Turn any web page into LLM-ready markdown

How response_type=markdown works on one page.

Map every URL on a website

Run Crawl on its own to list a site’s pages.

Build a research agent with live web data

Let an LLM read the web on demand instead of from an index.

Scrape a list of URLs

Send thousands of URLs as one Batch job.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Build a RAG knowledge base from a website

How it works

Ask your AI agent

Build the index

Example output

Plug in your embedding model and vector store

Customize it

Troubleshooting

Turn any web page into LLM-ready markdown

Map every URL on a website

Build a research agent with live web data

Scrape a list of URLs

​How it works

​Ask your AI agent

​Build the index

​Example output

​Plug in your embedding model and vector store

​Customize it

​Troubleshooting

​Related recipes

Turn any web page into LLM-ready markdown

Map every URL on a website

Build a research agent with live web data

Scrape a list of URLs

How it works

Ask your AI agent

Build the index

Example output

Plug in your embedding model and vector store

Customize it

Troubleshooting

Related recipes