Build a research agent with live web data

A research agent answers a question by reading the web: it decides which pages to open, reads them, and writes an answer with sources. Zenrows gives it the reading part, on any site, including pages protected by anti-bot systems or rendered with JavaScript. Uses: Zenrows MCP server (scrape and every other tool), or Fetch with response_type=markdown and mode=auto as a function tool

How it works

The model plans. Given a question, the LLM decides which URL to read first.
Zenrows reads the page. The model calls a Zenrows tool: the MCP scrape tool, or your own function that calls Fetch with mode=auto and response_type=markdown. The page comes back as markdown.
The loop repeats. The model reads the result, follows up with more pages if it needs them, and stops when it can answer.
The model answers with sources. The final answer cites the URLs it read.

Pick the path that fits your stack:

Path	Best for	Zenrows part
No code	Research in Claude, Cursor or another AI app	MCP server, connected once
MCP connector	An agent in your code, with every Zenrows tool	`https://mcp.zenrows.com/mcp` with your API key as a Bearer token
Function tool	Full control over what the agent can do	One function that calls Fetch

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to research the three cheapest hoodies on https://www.scrapingcourse.com/ecommerce/.
Read the catalog, open each hoodie's product page, and write a short buying guide with each
product's name, price, materials and a link to its page.

No code: Claude or Cursor with the MCP server

Connect the Zenrows MCP server to your AI app once, then research in plain language.

Connect the server

In Claude, add a custom connector with the URL https://mcp.zenrows.com/mcp and sign in, see Claude. In Cursor, Claude Code, VS Code, Windsurf and other clients, follow the client’s page in MCP overview.

Ask a research question

Paste the prompt from Ask your AI agent, or your own. Name Zenrows in the prompt (“Use Zenrows to…”) so the app picks its tools.

Go deeper with more tools

The same server gives the agent extract for structured data, crawl_* to discover a site’s pages, batch_* for many URLs at once, and browser_* for pages that need a login or clicks. See the tool reference.

Connect the MCP server to the LLM API

The Anthropic and OpenAI APIs connect to remote MCP servers on their side: you pass the server URL and your Zenrows API key, and the model calls the tools directly. Check the connection first. This request lists the server’s tools, with your key as a Bearer token:

cURL

curl https://mcp.zenrows.com/mcp \
  -H "Authorization: Bearer YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

The response lists every tool, starting with scrape, extract, batch_create, batch_status and the crawl_* and browser_* families. Then give the server to the model:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

response = client.beta.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    betas=["mcp-client-2025-11-20"],
    mcp_servers=[
        {
            "type": "url",
            "url": "https://mcp.zenrows.com/mcp",
            "name": "zenrows",
            "authorization_token": "YOUR_ZENROWS_API_KEY",
        }
    ],
    tools=[{"type": "mcp_toolset", "mcp_server_name": "zenrows"}],
    messages=[
        {
            "role": "user",
            "content": "Research the three cheapest hoodies on https://www.scrapingcourse.com/ecommerce/ and write a short buying guide with prices and links.",
        }
    ],
)

for block in response.content:
    if block.type == "text":
        print(block.text)

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY

response = client.responses.create(
    model="gpt-5",
    tools=[
        {
            "type": "mcp",
            "server_label": "zenrows",
            "server_url": "https://mcp.zenrows.com/mcp",
            "authorization": "YOUR_ZENROWS_API_KEY",
            "require_approval": "never",
        }
    ],
    input="Research the three cheapest hoodies on https://www.scrapingcourse.com/ecommerce/ and write a short buying guide with prices and links.",
)

print(response.output_text)

The API runs the whole agent loop: it lists the tools, calls scrape as many times as the model needs, and returns the final answer. To limit which tools the model can call, configure the toolset, see the Anthropic MCP connector and OpenAI remote MCP docs. More setups: Anthropic Claude API, OpenAI, OpenAI Agents SDK.

Add a function tool that calls Fetch

For full control, give the model one function of your own. It calls Fetch with mode=auto and response_type=markdown, so the model reads clean markdown from any page. This example uses the Anthropic SDK’s tool runner, which runs the loop for you (pip install anthropic requests):

Python

import anthropic
import requests
from anthropic import beta_tool

ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"

@beta_tool
def read_web_page(url: str) -> str:
    """Read a live web page and return its content as markdown.

    Call this whenever you need current facts from a specific URL, including pages
    protected by anti-bot systems or rendered with JavaScript.

    Args:
        url: The full URL of the page to read, starting with http:// or https://.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "apikey": ZENROWS_API_KEY,
            "url": url,
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=180,
    )
    if not response.ok:
        return f"Error {response.status_code}: {response.text[:500]}"
    return response.text[:50_000]

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
runner = client.beta.messages.tool_runner(
    model="claude-opus-5-5",
    max_tokens=16000,
    tools=[read_web_page],
    messages=[
        {
            "role": "user",
            "content": "Compare the prices and materials of https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ and https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
        }
    ],
)

for message in runner:
    for block in message.content:
        if block.type == "text":
            print(block.text)

The function returns errors as text instead of raising, so the model sees what went wrong and can try another URL. response.text[:50_000] keeps very long pages within a sensible context budget. With another framework, keep read_web_page as it is and register it the way your framework expects. Its JSON schema, which @beta_tool builds from the signature and docstring, is:

{
  "name": "read_web_page",
  "description": "Read a live web page and return its content as markdown. Call this whenever you need current facts from a specific URL, including pages protected by anti-bot systems or rendered with JavaScript.",
  "input_schema": {
    "type": "object",
    "properties": {
      "url": {"type": "string", "description": "The full URL of the page to read, starting with http:// or https://."}
    },
    "required": ["url"],
    "additionalProperties": false
  }
}

Example output

The tools/list check returns every tool. Trimmed to the start of the first one:

{
  "result": {
    "tools": [
      {
        "name": "scrape",
        "inputSchema": {
          "type": "object",
          "properties": {
            "url": {"type": "string", "format": "uri", "description": "The webpage URL to scrape"}
          }
        }
      }
    ]
  },
  "jsonrpc": "2.0",
  "id": 1
}

What read_web_page returns to the model for the hoodie page, trimmed to the product section:

# Abominable Hoodie

$69.00

This is a variable product called a Abominable Hoodie

## Description

It took CoolTech™ weather apparel know-how and lots of wind-resistant fabric to get the Abominable Hoodie just right.

Customize it

Change	How
Structured data instead of markdown	Use `extract=auto` in the function, or let the MCP agent call `extract`, see Get structured JSON from any page
Research behind a login	Add a Browser tool, see Fetch first, open a Browser when a task needs interaction
Research a whole site	Crawl it first and index it, see Build a RAG knowledge base from a website
Country-specific results	Add `proxy_country` (any ISO code) to the Fetch params, see Proxy country
Run the server locally	Use `npx -y @zenrows/mcp` with `ZENROWS_API_KEY` set, see MCP overview

Troubleshooting

The tools/list check returns 401: the key is missing or wrong. Send it exactly as Authorization: Bearer YOUR_ZENROWS_API_KEY, with your key from the dashboard.
The agent answers without reading any page: the model didn’t see a reason to call the tool. Name it in the prompt (“Use Zenrows to read…”) or make the tool description say when to call it.
Tool calls time out in your function: keep the HTTP timeout at 180 seconds so mode=auto has time to escalate on protected pages.
The model’s context fills up: long pages add up over many calls. Truncate each result, or return only the fields you need with extract=auto or css_extractor.

Connect your AI agent to Zenrows

Set up the MCP server in every major client.

Fetch first, open a Browser when a task needs interaction

Give the agent a browser for logins and clicks.

Prompts that put Zenrows MCP to work

Ready prompts for every Zenrows tool.

Build a RAG knowledge base from a website

Index a site ahead of time instead of reading on demand.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Build a research agent with live web data

How it works

Ask your AI agent

No code: Claude or Cursor with the MCP server

Connect the MCP server to the LLM API

Add a function tool that calls Fetch

Example output

Customize it

Troubleshooting

Connect your AI agent to Zenrows

Fetch first, open a Browser when a task needs interaction

Prompts that put Zenrows MCP to work

Build a RAG knowledge base from a website

​How it works

​Ask your AI agent

​No code: Claude or Cursor with the MCP server

​Connect the MCP server to the LLM API

​Add a function tool that calls Fetch

​Example output

​Customize it

​Troubleshooting

​Related recipes

Connect your AI agent to Zenrows

Fetch first, open a Browser when a task needs interaction

Prompts that put Zenrows MCP to work

Build a RAG knowledge base from a website

How it works

Ask your AI agent

No code: Claude or Cursor with the MCP server

Connect the MCP server to the LLM API

Add a function tool that calls Fetch

Example output

Customize it

Troubleshooting

Related recipes