Skip to main content
Claude has a fixed knowledge cutoff and no built-in web access at the API layer. Pairing it with the ZenRows Universal Scraper API closes that gap: your application can fetch live content from any website, including JavaScript-heavy and bot-protected pages, and feed it directly into any Claude model. This guide covers two implementation paths:
  • Direct API Call: Best for request-level control over your AI application’s scraping layer.
  • Model Context Protocol (MCP): Best for a plug-and-play setup with ZenRows’ optimal anti-bot configurations built in.
The guide uses Claude’s Messages API, tool use (function calling), and structured outputs throughout. If you are building autonomous, multi-step agents with the Claude Agent SDK, see the Claude Agent SDK integration guide.

What is Anthropic Claude API?

The Anthropic API provides programmatic access to Claude through the Messages API, with built-in support for tool use (function calling), structured outputs, extended thinking, and the MCP connector for remote MCP servers. Claude excels at reasoning over long documents, multi-step analysis, and structured generation, but it has a fixed knowledge cutoff and no built-in web access at the API layer. Pairing Claude with ZenRows for reliable scraping that bypasses anti-bot detection closes that gap.

Key benefits

  • Real-time web grounding for any Claude model
    ZenRows feeds live, up-to-date web content into any response, so responses draw from real-time data instead of training knowledge. This eliminates stale or hallucinated answers about recent events, prices, or product details.
  • Anti-bot bypass out of the box
    Adaptive Stealth Mode (mode=auto) automatically handles JavaScript rendering, premium proxies, fingerprinting, and bot detection, so Claude can read pages that would block a regular scraper.
  • Token-efficient Markdown output
    ZenRows returns clean Markdown in addition to raw HTML. This pairs well with Claude’s large context window, letting you fit more pages per request without burning through tokens on HTML noise.
  • Native tool use
    Expose ZenRows as a tool and let Claude decide when to scrape based on the user’s question, with no orchestration code required.
  • Works with every Claude API surface
    Use ZenRows with the Messages API, structured outputs (messages.parse), and extended thinking. The integration is a plain HTTP call, so it fits into any pattern.
  • MCP-ready out of the box
    Plug the ZenRows hosted MCP server directly into the Messages API through Claude’s MCP connector and give your AI application access to all of ZenRows’ scraping capabilities.

Use cases

The Claude and ZenRows combination unlocks a wide range of AI workflows:
Scrape multiple pages with ZenRows and feed them into Claude’s 200K+ context window for cross-document analysis, comparison, or synthesis.
Have Claude analyze competitor pricing pages, product launches, and changelogs as they happen.
Scrape company websites and let Claude extract industry, headcount signals, tech stack hints, and product summaries into your CRM.
Build assistants that can read product docs, status pages, or knowledge bases the user references mid-conversation.
Pull strongly-typed JSON (products, jobs, listings, reviews) from any page using Claude’s structured outputs grounded in ZenRows-scraped content.

Getting started: Basic usage

A simple example: scrape a JavaScript-heavy, anti-bot-protected demo page using ZenRows, then summarize it with claude-sonnet-4-6 via the Messages API.
1

Install the Anthropic Python library and requests

pip install anthropic requests
2

Create a .env file with your API keys

ZENROWS_API_KEY=your_zenrows_key
ANTHROPIC_API_KEY=your_anthropic_key
Find your ZenRows API key in the ZenRows dashboard. Find your Anthropic API key in the Claude Console under Settings > API keys.
3

Run the following script

Python
import os
import requests
from anthropic import Anthropic
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = Anthropic()

def scrape(url: str) -> str:
    """Scrape any URL with ZenRows and return clean Markdown."""
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "apikey": ZENROWS_API_KEY,
            "url": url,
            "mode": "auto",
            "response_type": "markdown",
        },
    )
    response.raise_for_status()
    return response.text

# Scrape a JS-rendered, anti-bot-protected page
markdown = scrape("https://www.scrapingcourse.com/antibot-challenge")

# Summarize with Claude through the Messages API
message = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system="You are a concise technical writer.",
    messages=[
        {
            "role": "user",
            "content": f"Summarize this page in 2 sentences:\n\n{markdown}",
        }
    ],
)

print(message.content[0].text)
Two things are happening here:
  1. ZenRows handles the scrape. mode=auto tells the API to start with the cheapest viable configuration and automatically escalate to JavaScript rendering or premium proxies if the target site requires it. response_type=markdown returns clean Markdown instead of raw HTML, which is ideal for Claude’s context window.
  2. Claude handles the reasoning. The Markdown is fed into client.messages.create(), with system-level guidance passed through the system parameter and the user prompt passed through messages. Claude’s response comes back as a list of content blocks; for plain-text replies, message.content[0].text is the right call.

Advanced usage: Building a web-aware AI assistant with tool use

Tool use is Anthropic’s mechanism for letting Claude decide when to call an external tool. Instead of always scraping before each prompt, you expose ZenRows as a tool and let Claude call it only when needed. This pattern is the foundation of any production-grade Claude application that needs web access.
1

Set up the environment

Install the Anthropic Python library and requests:
pip install anthropic requests
Create a .env file:
ZENROWS_API_KEY=your_zenrows_key
ANTHROPIC_API_KEY=your_anthropic_key
2

Define the scraping tool

Wrap the ZenRows REST call in a Python function and describe it to the Messages API using a JSON schema:
Python
import os
import requests
from anthropic import Anthropic
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = Anthropic()

def scrape_website(url: str) -> str:
    """Scrape any website with ZenRows and return Markdown content."""
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "apikey": ZENROWS_API_KEY,
            "url": url,
            "mode": "auto",
            "response_type": "markdown",
        },
    )
    response.raise_for_status()
    return response.text

# JSON schema to tell Claude what scrape_website does and how to call it
tools = [
    {
        "name": "scrape_website",
        "description": (
            "Scrape any public website and return its content as Markdown. "
            "Use this whenever you need up-to-date information from a specific URL."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "url": {
                    "type": "string",
                    "description": "The fully qualified URL of the website to scrape",
                }
            },
            "required": ["url"],
        },
    }
]
Each tool definition uses Anthropic’s flat shape: name, description, and input_schema at the top level.
3

Let Claude decide when to scrape

Send the user’s question along with the tool definition. Claude returns either a direct text response or a tool_use content block with stop_reason: "tool_use":
Python
def ask(question: str) -> str:
    messages = [{"role": "user", "content": question}]

    # Send the question to Claude with the scraping tool available
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        tools=tools,
        messages=messages,
    )

    # Keep looping as long as Claude wants to call a tool
    while response.stop_reason == "tool_use":
        # Extract the tool call from Claude's response
        tool_use = next(block for block in response.content if block.type == "tool_use")
        # Execute the scrape and capture the result
        result = scrape_website(**tool_use.input)

        # Append Claude's response and the tool result to the conversation history
        messages.append({"role": "assistant", "content": response.content})
        messages.append(
            {
                "role": "user",
                "content": [
                    {
                        "type": "tool_result",
                        "tool_use_id": tool_use.id,
                        "content": result,
                    }
                ],
            }
        )

        # Send the updated conversation history back to Claude to continue
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            tools=tools,
            messages=messages,
        )

    # Return Claude's final text response once no more tool calls are needed
    return next(block.text for block in response.content if block.type == "text")

# Ask Claude a question that requires live web access to answer
print(
    ask(
        "What does the Hugging Face blog cover? "
        "Visit https://huggingface.co/blog and tell me what topics they post about."
    )
)
The while response.stop_reason == "tool_use" loop is the standard Claude tool-use pattern. Claude can chain multiple tool calls in a single conversation, so the loop continues until Claude returns stop_reason: "end_turn". Each iteration appends the assistant’s tool use and your tool result to the messages list before the next API call.
4

Complete code example and output

Python
import os
import requests
from anthropic import Anthropic
from dotenv import load_dotenv

load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = Anthropic()

def scrape_website(url: str) -> str:
    """Scrape any website with ZenRows and return Markdown content."""
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "apikey": ZENROWS_API_KEY,
            "url": url,
            "mode": "auto",
            "response_type": "markdown",
        },
    )
    response.raise_for_status()
    return response.text

tools = [
    {
        "name": "scrape_website",
        "description": (
            "Scrape any public website and return its content as Markdown. "
            "Use this whenever you need up-to-date information from a specific URL."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "url": {
                    "type": "string",
                    "description": "The fully qualified URL of the website to scrape",
                }
            },
            "required": ["url"],
        },
    }
]

def ask(question: str) -> str:
    messages = [{"role": "user", "content": question}]

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        tools=tools,
        messages=messages,
    )

    while response.stop_reason == "tool_use":
        tool_use = next(block for block in response.content if block.type == "tool_use")
        result = scrape_website(**tool_use.input)

        messages.append({"role": "assistant", "content": response.content})
        messages.append(
            {
                "role": "user",
                "content": [
                    {
                        "type": "tool_result",
                        "tool_use_id": tool_use.id,
                        "content": result,
                    }
                ],
            }
        )

        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            tools=tools,
            messages=messages,
        )

    return next(block.text for block in response.content if block.type == "text")

print(
    ask(
        "What does the Hugging Face blog cover? "
        "Visit https://huggingface.co/blog and tell me what topics they post about."
    )
)
Example output:
Output
The **Hugging Face Blog** (https://huggingface.co/blog) is a community-driven publication covering a wide range of topics related to AI, machine learning, and open-source development. Here's a breakdown of what they post abo

### 🏷️ Main Topic Categories

The blog uses tags to organize content, including:
- **NLP** (Natural Language Processing)
- ...truncated for brevity

### 📝 Recent Article Highlights

Here are some of the specific topics covered in recent posts:
1. **Robotics** – e.g.,...
...truncated for brevity

### 👥 Who Writes It?

The blog is a **community blog**, meaning articles are contributed by a mix of:

- **Hugging Face team members**
- ...truncated for brevity

In summary, the Hugging Face blog is a rich resource for anyone interested in **cutting-edge AI research, practical ML tutorials, model releases, open-source tooling, and industry insights**.
Claude decides on its own that the question requires fresh information from huggingface.co/blog, calls scrape_website with the right URL, then synthesizes an answer from the scraped Markdown.

Structured outputs

Anthropic’s structured outputs let you guarantee that Claude returns JSON conforming to a specific schema. Combined with ZenRows, this is the cleanest way to extract typed data from any web page. The Messages API exposes structured outputs through client.messages.parse(), which takes a Pydantic model and returns a parsed Python object directly through response.parsed_output. The example below scrapes a company homepage and extracts a strongly-typed CompanyInfo object:
Python
import os
import requests
from anthropic import Anthropic
from pydantic import BaseModel
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = Anthropic()

# Define the Pydantic schema
class CompanyInfo(BaseModel):
    name: str
    industry: str
    description: str

# Scrape the company website
response = requests.get(
    "https://api.zenrows.com/v1/",
    params={
        "apikey": ZENROWS_API_KEY,
        "url": "https://huggingface.co/",
        "mode": "auto",
        "response_type": "markdown",
    },
)
response.raise_for_status()
markdown = response.text

# Extract structured data through the Messages API
parsed = client.messages.parse(
    model="claude-opus-4-7",
    max_tokens=1024,
    system="Extract company information from the provided website content.",
    messages=[{"role": "user", "content": markdown}],
    output_format=CompanyInfo,
)

company = parsed.parsed_output
print(company.model_dump_json(indent=2))
Example output:
JSON
{
  "name": "Hugging Face",
  "industry": "Artificial intelligence / Machine learning platform",
  "description": "Hugging Face is an open-source AI community and platform that hosts machine learning models, datasets, and applications. It provides tooling for building, sharing, and deploying models, including the Transformers library and Inference Endpoints, and is widely used as the central hub for the open-source ML ecosystem."
}
The same pattern works for any structured extraction job: product listings, job postings, real estate, reviews, articles, and so on. Define the Pydantic schema, scrape the page with mode=auto, and let Claude return validated JSON.
Skip the schema with ZenRows Autoparse. For common site types, you can skip LLM-based parsing entirely. Adding autoparse=true to your ZenRows request automatically identifies and extracts product details, article content, job listings, property data, and similar information into clean JSON, with no Pydantic schema or model call required. ZenRows Autoparse is included at no additional cost. Note that Autoparse cannot be combined with response_type=markdown in the same request.

Using ZenRows MCP with the Messages API

The Messages API natively supports remote Model Context Protocol (MCP) servers through Claude’s MCP connector. ZenRows has a hosted MCP server that exposes web scraping capabilities, so you can give Claude real-time web access without writing any tool use boilerplate.
Python
import os
from anthropic import Anthropic
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = Anthropic()

response = client.beta.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Visit https://huggingface.co/blog and summarize the three most recent posts.",
        }
    ],
    mcp_servers=[
        {
            "type": "url",
            "url": "https://mcp.zenrows.com/mcp",
            "name": "zenrows",
            "authorization_token": ZENROWS_API_KEY,
        }
    ],
    tools=[
        {
            "type": "mcp_toolset",
            "mcp_server_name": "zenrows",
        }
    ],
    betas=["mcp-client-2025-11-20"],
)

# Print Claude's final text response
for block in response.content:
    if block.type == "text":
        print(block.text)
A few notes on this configuration:
  • The MCP connector lives under client.beta.messages.create() (not the standard messages.create()), and requires the betas=["mcp-client-2025-11-20"] parameter.
  • The configuration has two parts: mcp_servers defines the connection (URL, name, authorization token), and tools references that server with a mcp_toolset entry that controls which tools are enabled.
  • authorization_token carries your ZenRows API key as the Bearer token. Anthropic does not retain the value between requests, so it must be present on every API call.
  • Responses include new mcp_tool_use and mcp_tool_result content blocks alongside regular text blocks. Filter by block.type == "text" for the final answer, or inspect all blocks for full tool call traces.
For details on the ZenRows MCP server and its capabilities, see the MCP overview.

Troubleshooting

Token limit exceeded

  1. Use response_type=markdown (shown in every example above). Markdown reduces token usage significantly compared to raw HTML.
  2. Use css_extractor, autoparse for supported sites, or outputs to scrape only the parts of the page you need (a product card, a pricing table, an article body) instead of the entire DOM.
  3. Claude has one of the largest context windows in the industry (200K+ tokens for Sonnet 4.6 and Opus 4.7). For multi-page workflows, you can often fit all scraped content in a single request without chunking.

API key errors

  1. Confirm both ZENROWS_API_KEY and ANTHROPIC_API_KEY are set in your environment.
  2. Verify your ZenRows API key in the ZenRows dashboard and your Anthropic key in the Anthropic Console.
  3. Check that your ZenRows subscription is active and has remaining quota on the Analytics page.

Empty or incomplete tool responses

  1. Confirm mode=auto is set. Without it, JavaScript rendering and premium proxies are off by default and protected sites return blocked or empty pages.
  2. For sites that load content asynchronously, add wait_for=<css-selector> (waits for a specific element) or wait=5000 (waits a fixed duration in milliseconds).
  3. If Claude is calling the tool with a malformed URL, tighten the tool description and input_schema, or add validation in your scrape_website wrapper before sending the request to ZenRows.

Infinite tool use loop

If your while response.stop_reason == "tool_use" loop runs more times than expected, Claude is asking to scrape the same URL repeatedly. Tighten the system prompt with a stop condition such as “Once you have the content of the requested URL, summarize and stop.” For complex workflows, add an explicit iteration counter and break after a reasonable maximum.

424 failed dependency or 401 unauthorized from the MCP connector

  1. Confirm the authorization_token field is set on the MCP server entry and that the value is your active ZenRows API key.
  2. Confirm you are using client.beta.messages.create() (not client.messages.create()) and that betas=["mcp-client-2025-11-20"] is set.
  3. Confirm the URL in the mcp_servers entry is exactly https://mcp.zenrows.com/mcp.

FAQ (Frequently Asked Questions)

All current Claude 4 models. Tool use and structured outputs work best with claude-opus-4-7 and claude-sonnet-4-6. For high-throughput, cost-sensitive workloads, claude-haiku-4-5-20251001 is a faster, cheaper option. The MCP connector is currently available on the Claude API, Claude Platform on AWS, and Microsoft Foundry. It is not yet supported on Amazon Bedrock or Vertex AI.
Three options, in order of preference: use response_type=markdown to keep token usage low; use css_extractor, autoparse (JSON output for supported sites), or outputs to scrape only the relevant section of the page; use Claude models with large context windows (Sonnet 4.6 and Opus 4.7 support 200K+ tokens) to fit more pages per request without chunking.
Yes. Setting mode=auto enables Adaptive Stealth Mode, which automatically activates JavaScript rendering, premium residential proxies, and stealth fingerprinting only when the target site requires them. You only pay for what succeeds.
Yes. The Claude Agent SDK is built on top of the Messages API and has native MCP support, so the same hosted MCP server URL works there. For agent-specific workflows including autonomous research, multi-step tool chains, and the SDK’s built-in MCP support, see the Claude Agent SDK integration guide.
Yes. Tool use supports multi-turn loops, so Claude can call scrape_website repeatedly with different URLs, read what it gets back, and decide what to scrape next. For more complex agent behavior, pair this with the MCP connector, which exposes a richer scraping surface without requiring you to maintain the tool use loop yourself. For autonomous, multi-step research workflows, see the Claude Agent SDK integration guide.
Anthropic offers built-in web_search and web_fetch server tools, but they’re limited: web search is a black-box query interface, and web fetch is a basic HTTP getter without anti-bot bypass, JavaScript rendering, or geo-routing. ZenRows gives you full control over which URL is fetched, how the page is rendered, what format the response comes back in, and which countries the request is routed through. It also handles pages that block standard scrapers, such as geo-restricted content, JS-gated sites, and heavily protected pages.
ZenRows has plan-based concurrency limits documented on the Concurrency page. Anthropic also has model-specific rate limits on the Messages API. When using tool use loops or the MCP connector with many parallel scrapes, monitor your usage on both dashboards and consider adding retry logic with exponential backoff in your scrape_website wrapper.