> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.
# Build a research agent with live web data
> Build an AI research agent that reads live web pages with Zenrows and cites its sources, with Claude or GPT.
A research agent answers a question by reading the web: it decides which pages to open, reads them, and writes an answer with sources. Zenrows gives it the reading part, on any site, including pages protected by anti-bot systems or rendered with JavaScript.
**Uses:** [Zenrows MCP server](/mcp/overview) (`scrape` and every other tool), or [Fetch](/fetch/api-reference) with [`response_type=markdown`](/fetch/features/markdown) and `mode=auto` as a function tool
## How it works
1. **The model plans.** Given a question, the LLM decides which URL to read first.
2. **Zenrows reads the page.** The model calls a Zenrows tool: the MCP `scrape` tool, or your own function that calls Fetch with `mode=auto` and `response_type=markdown`. The page comes back as markdown.
3. **The loop repeats.** The model reads the result, follows up with more pages if it needs them, and stops when it can answer.
4. **The model answers with sources.** The final answer cites the URLs it read.
Pick the path that fits your stack:
| Path | Best for | Zenrows part |
| - | - | - |
| [No code](#no-code-claude-or-cursor-with-the-mcp-server) | Research in Claude, Cursor or another AI app | MCP server, connected once |
| [MCP connector](#connect-the-mcp-server-to-the-llm-api) | An agent in your code, with every Zenrows tool | `https://mcp.zenrows.com/mcp` with your API key as a Bearer token |
| [Function tool](#add-a-function-tool-that-calls-fetch) | Full control over what the agent can do | One function that calls Fetch |
## Ask your AI agent
With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.
```text theme={"dark"}
Use Zenrows to research the three cheapest hoodies on https://www.scrapingcourse.com/ecommerce/.
Read the catalog, open each hoodie's product page, and write a short buying guide with each
product's name, price, materials and a link to its page.
```
## No code: Claude or Cursor with the MCP server
Connect the Zenrows MCP server to your AI app once, then research in plain language.
In Claude, add a custom connector with the URL `https://mcp.zenrows.com/mcp` and sign in, see [Claude](/integrations/claude). In Cursor, Claude Code, VS Code, Windsurf and other clients, follow the client's page in [MCP overview](/mcp/overview#per-client-setup-guides).
Paste the prompt from [Ask your AI agent](#ask-your-ai-agent), or your own. Name Zenrows in the prompt ("Use Zenrows to...") so the app picks its tools.
The same server gives the agent `extract` for structured data, `crawl_*` to discover a site's pages, `batch_*` for many URLs at once, and `browser_*` for pages that need a login or clicks. See the [tool reference](/mcp/tool-reference).
## Connect the MCP server to the LLM API
The Anthropic and OpenAI APIs connect to remote MCP servers on their side: you pass the server URL and your Zenrows API key, and the model calls the tools directly. Check the connection first. This request lists the server's tools, with your key as a Bearer token:
```bash cURL theme={"dark"}
curl https://mcp.zenrows.com/mcp \
-H "Authorization: Bearer YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
```
The response lists every tool, starting with `scrape`, `extract`, `batch_create`, `batch_status` and the `crawl_*` and `browser_*` families. Then give the server to the model:
```python Anthropic (Python) theme={"dark"}
import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY
response = client.beta.messages.create(
model="claude-opus-5-5",
max_tokens=16000,
betas=["mcp-client-2025-11-20"],
mcp_servers=[
{
"type": "url",
"url": "https://mcp.zenrows.com/mcp",
"name": "zenrows",
"authorization_token": "YOUR_ZENROWS_API_KEY",
}
],
tools=[{"type": "mcp_toolset", "mcp_server_name": "zenrows"}],
messages=[
{
"role": "user",
"content": "Research the three cheapest hoodies on https://www.scrapingcourse.com/ecommerce/ and write a short buying guide with prices and links.",
}
],
)
for block in response.content:
if block.type == "text":
print(block.text)
```
```python OpenAI (Python) theme={"dark"}
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY
response = client.responses.create(
model="gpt-5",
tools=[
{
"type": "mcp",
"server_label": "zenrows",
"server_url": "https://mcp.zenrows.com/mcp",
"authorization": "YOUR_ZENROWS_API_KEY",
"require_approval": "never",
}
],
input="Research the three cheapest hoodies on https://www.scrapingcourse.com/ecommerce/ and write a short buying guide with prices and links.",
)
print(response.output_text)
```
The API runs the whole agent loop: it lists the tools, calls `scrape` as many times as the model needs, and returns the final answer. To limit which tools the model can call, configure the toolset, see the Anthropic MCP connector and OpenAI remote MCP docs. More setups: [Anthropic Claude API](/integrations/anthropic-claude-api), [OpenAI](/integrations/openai), [OpenAI Agents SDK](/integrations/openai-agents-sdk).
## Add a function tool that calls Fetch
For full control, give the model one function of your own. It calls Fetch with `mode=auto` and `response_type=markdown`, so the model reads clean markdown from any page. This example uses the Anthropic SDK's tool runner, which runs the loop for you (`pip install anthropic requests`):
```python Python theme={"dark"}
import anthropic
import requests
from anthropic import beta_tool
ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"
@beta_tool
def read_web_page(url: str) -> str:
"""Read a live web page and return its content as markdown.
Call this whenever you need current facts from a specific URL, including pages
protected by anti-bot systems or rendered with JavaScript.
Args:
url: The full URL of the page to read, starting with http:// or https://.
"""
response = requests.get(
"https://api.zenrows.com/v1/",
params={
"apikey": ZENROWS_API_KEY,
"url": url,
"mode": "auto",
"response_type": "markdown",
},
timeout=180,
)
if not response.ok:
return f"Error {response.status_code}: {response.text[:500]}"
return response.text[:50_000]
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY
runner = client.beta.messages.tool_runner(
model="claude-opus-5-5",
max_tokens=16000,
tools=[read_web_page],
messages=[
{
"role": "user",
"content": "Compare the prices and materials of https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ and https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
}
],
)
for message in runner:
for block in message.content:
if block.type == "text":
print(block.text)
```
The function returns errors as text instead of raising, so the model sees what went wrong and can try another URL. `response.text[:50_000]` keeps very long pages within a sensible context budget.
With another framework, keep `read_web_page` as it is and register it the way your framework expects. Its JSON schema, which `@beta_tool` builds from the signature and docstring, is:
```json theme={"dark"}
{
"name": "read_web_page",
"description": "Read a live web page and return its content as markdown. Call this whenever you need current facts from a specific URL, including pages protected by anti-bot systems or rendered with JavaScript.",
"input_schema": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "The full URL of the page to read, starting with http:// or https://."}
},
"required": ["url"],
"additionalProperties": false
}
}
```
## Example output
The `tools/list` check returns every tool. Trimmed to the start of the first one:
```json theme={"dark"}
{
"result": {
"tools": [
{
"name": "scrape",
"inputSchema": {
"type": "object",
"properties": {
"url": {"type": "string", "format": "uri", "description": "The webpage URL to scrape"}
}
}
}
]
},
"jsonrpc": "2.0",
"id": 1
}
```
What `read_web_page` returns to the model for the hoodie page, trimmed to the product section:
```markdown theme={"dark"}
# Abominable Hoodie
$69.00
This is a variable product called a Abominable Hoodie
## Description
It took CoolTech™ weather apparel know-how and lots of wind-resistant fabric to get the Abominable Hoodie just right.
```
## Customize it
| Change | How |
| - | - |
| Structured data instead of markdown | Use `extract=auto` in the function, or let the MCP agent call `extract`, see [Get structured JSON from any page](/academy/extract/auto-extraction) |
| Research behind a login | Add a Browser tool, see [Fetch first, open a Browser when a task needs interaction](/academy/pipelines/fetch-with-browser-fallback) |
| Research a whole site | Crawl it first and index it, see [Build a RAG knowledge base from a website](/academy/pipelines/website-to-rag) |
| Country-specific results | Add `proxy_country` (any ISO code) to the Fetch params, see [Proxy country](/fetch/features/proxy-country) |
| Run the server locally | Use `npx -y @zenrows/mcp` with `ZENROWS_API_KEY` set, see [MCP overview](/mcp/overview#local-mcp-server) |
## Troubleshooting
* **The `tools/list` check returns `401`**: the key is missing or wrong. Send it exactly as `Authorization: Bearer YOUR_ZENROWS_API_KEY`, with your key from the [dashboard](https://app.zenrows.com/settings/api-keys).
* **The agent answers without reading any page**: the model didn't see a reason to call the tool. Name it in the prompt ("Use Zenrows to read...") or make the tool description say when to call it.
* **Tool calls time out in your function**: keep the HTTP timeout at 180 seconds so `mode=auto` has time to escalate on protected pages.
* **The model's context fills up**: long pages add up over many calls. Truncate each result, or return only the fields you need with `extract=auto` or `css_extractor`.
## Related recipes
Set up the MCP server in every major client.
Give the agent a browser for logins and clicks.
Ready prompts for every Zenrows tool.
Index a site ahead of time instead of reading on demand.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.