- Direct API Call: Best for request-level control over your AI application’s scraping layer.
- Model Context Protocol (MCP): Best for a plug-and-play setup with ZenRows’ optimal anti-bot configurations built in.
What is Anthropic Claude API?
The Anthropic API provides programmatic access to Claude through the Messages API, with built-in support for tool use (function calling), structured outputs, extended thinking, and the MCP connector for remote MCP servers. Claude excels at reasoning over long documents, multi-step analysis, and structured generation, but it has a fixed knowledge cutoff and no built-in web access at the API layer. Pairing Claude with ZenRows for reliable scraping that bypasses anti-bot detection closes that gap.Key benefits
-
Real-time web grounding for any Claude model
ZenRows feeds live, up-to-date web content into any response, so responses draw from real-time data instead of training knowledge. This eliminates stale or hallucinated answers about recent events, prices, or product details. -
Anti-bot bypass out of the box
Adaptive Stealth Mode (mode=auto) automatically handles JavaScript rendering, premium proxies, fingerprinting, and bot detection, so Claude can read pages that would block a regular scraper. -
Token-efficient Markdown output
ZenRows returns clean Markdown in addition to raw HTML. This pairs well with Claude’s large context window, letting you fit more pages per request without burning through tokens on HTML noise. -
Native tool use
Expose ZenRows as a tool and let Claude decide when to scrape based on the user’s question, with no orchestration code required. -
Works with every Claude API surface
Use ZenRows with the Messages API, structured outputs (messages.parse), and extended thinking. The integration is a plain HTTP call, so it fits into any pattern. -
MCP-ready out of the box
Plug the ZenRows hosted MCP server directly into the Messages API through Claude’s MCP connector and give your AI application access to all of ZenRows’ scraping capabilities.
Use cases
The Claude and ZenRows combination unlocks a wide range of AI workflows:Long-context document grounding
Long-context document grounding
Real-time competitive intelligence
Real-time competitive intelligence
Lead enrichment
Lead enrichment
Web-aware support assistants
Web-aware support assistants
Structured data extraction
Structured data extraction
Getting started: Basic usage
A simple example: scrape a JavaScript-heavy, anti-bot-protected demo page using ZenRows, then summarize it withclaude-sonnet-4-6 via the Messages API.
Install the Anthropic Python library and requests
Create a .env file with your API keys
Run the following script
- ZenRows handles the scrape.
mode=autotells the API to start with the cheapest viable configuration and automatically escalate to JavaScript rendering or premium proxies if the target site requires it.response_type=markdownreturns clean Markdown instead of raw HTML, which is ideal for Claude’s context window. - Claude handles the reasoning. The Markdown is fed into
client.messages.create(), with system-level guidance passed through thesystemparameter and the user prompt passed throughmessages. Claude’s response comes back as a list of content blocks; for plain-text replies,message.content[0].textis the right call.
Advanced usage: Building a web-aware AI assistant with tool use
Tool use is Anthropic’s mechanism for letting Claude decide when to call an external tool. Instead of always scraping before each prompt, you expose ZenRows as a tool and let Claude call it only when needed. This pattern is the foundation of any production-grade Claude application that needs web access.Set up the environment
.env file:Define the scraping tool
name, description, and input_schema at the top level.Let Claude decide when to scrape
tool_use content block with stop_reason: "tool_use":while response.stop_reason == "tool_use" loop is the standard Claude tool-use pattern. Claude can chain multiple tool calls in a single conversation, so the loop continues until Claude returns stop_reason: "end_turn". Each iteration appends the assistant’s tool use and your tool result to the messages list before the next API call.Complete code example and output
huggingface.co/blog, calls scrape_website with the right URL, then synthesizes an answer from the scraped Markdown.Structured outputs
Anthropic’s structured outputs let you guarantee that Claude returns JSON conforming to a specific schema. Combined with ZenRows, this is the cleanest way to extract typed data from any web page. The Messages API exposes structured outputs throughclient.messages.parse(), which takes a Pydantic model and returns a parsed Python object directly through response.parsed_output.
The example below scrapes a company homepage and extracts a strongly-typed CompanyInfo object:
mode=auto, and let Claude return validated JSON.
Using ZenRows MCP with the Messages API
The Messages API natively supports remote Model Context Protocol (MCP) servers through Claude’s MCP connector. ZenRows has a hosted MCP server that exposes web scraping capabilities, so you can give Claude real-time web access without writing any tool use boilerplate.- The MCP connector lives under
client.beta.messages.create()(not the standardmessages.create()), and requires thebetas=["mcp-client-2025-11-20"]parameter. - The configuration has two parts:
mcp_serversdefines the connection (URL, name, authorization token), andtoolsreferences that server with amcp_toolsetentry that controls which tools are enabled. authorization_tokencarries your ZenRows API key as the Bearer token. Anthropic does not retain the value between requests, so it must be present on every API call.- Responses include new
mcp_tool_useandmcp_tool_resultcontent blocks alongside regular text blocks. Filter byblock.type == "text"for the final answer, or inspect all blocks for full tool call traces.
Troubleshooting
Token limit exceeded
- Use
response_type=markdown(shown in every example above). Markdown reduces token usage significantly compared to raw HTML. - Use
css_extractor,autoparsefor supported sites, oroutputsto scrape only the parts of the page you need (a product card, a pricing table, an article body) instead of the entire DOM. - Claude has one of the largest context windows in the industry (200K+ tokens for Sonnet 4.6 and Opus 4.7). For multi-page workflows, you can often fit all scraped content in a single request without chunking.
API key errors
- Confirm both
ZENROWS_API_KEYandANTHROPIC_API_KEYare set in your environment. - Verify your ZenRows API key in the ZenRows dashboard and your Anthropic key in the Anthropic Console.
- Check that your ZenRows subscription is active and has remaining quota on the Analytics page.
Empty or incomplete tool responses
- Confirm
mode=autois set. Without it, JavaScript rendering and premium proxies are off by default and protected sites return blocked or empty pages. - For sites that load content asynchronously, add
wait_for=<css-selector>(waits for a specific element) orwait=5000(waits a fixed duration in milliseconds). - If Claude is calling the tool with a malformed URL, tighten the tool description and
input_schema, or add validation in yourscrape_websitewrapper before sending the request to ZenRows.
Infinite tool use loop
If yourwhile response.stop_reason == "tool_use" loop runs more times than expected, Claude is asking to scrape the same URL repeatedly. Tighten the system prompt with a stop condition such as “Once you have the content of the requested URL, summarize and stop.” For complex workflows, add an explicit iteration counter and break after a reasonable maximum.
424 failed dependency or 401 unauthorized from the MCP connector
- Confirm the
authorization_tokenfield is set on the MCP server entry and that the value is your active ZenRows API key. - Confirm you are using
client.beta.messages.create()(notclient.messages.create()) and thatbetas=["mcp-client-2025-11-20"]is set. - Confirm the URL in the
mcp_serversentry is exactlyhttps://mcp.zenrows.com/mcp.
FAQ (Frequently Asked Questions)
Which Claude models work with this integration?
Which Claude models work with this integration?
claude-opus-4-7 and claude-sonnet-4-6. For high-throughput, cost-sensitive workloads, claude-haiku-4-5-20251001 is a faster, cheaper option. The MCP connector is currently available on the Claude API, Claude Platform on AWS, and Microsoft Foundry. It is not yet supported on Amazon Bedrock or Vertex AI.How do I handle pages that exceed the model's token limit?
How do I handle pages that exceed the model's token limit?
response_type=markdown to keep token usage low; use css_extractor, autoparse (JSON output for supported sites), or outputs to scrape only the relevant section of the page; use Claude models with large context windows (Sonnet 4.6 and Opus 4.7 support 200K+ tokens) to fit more pages per request without chunking.Does ZenRows handle JavaScript-heavy and anti-bot-protected sites automatically?
Does ZenRows handle JavaScript-heavy and anti-bot-protected sites automatically?
mode=auto enables Adaptive Stealth Mode, which automatically activates JavaScript rendering, premium residential proxies, and stealth fingerprinting only when the target site requires them. You only pay for what succeeds.Can I use this with the Claude Agent SDK?
Can I use this with the Claude Agent SDK?
Can I let Claude browse multiple pages autonomously?
Can I let Claude browse multiple pages autonomously?
scrape_website repeatedly with different URLs, read what it gets back, and decide what to scrape next. For more complex agent behavior, pair this with the MCP connector, which exposes a richer scraping surface without requiring you to maintain the tool use loop yourself. For autonomous, multi-step research workflows, see the Claude Agent SDK integration guide.How does this compare to Claude's built-in web search and web fetch tools?
How does this compare to Claude's built-in web search and web fetch tools?
web_search and web_fetch server tools, but they’re limited: web search is a black-box query interface, and web fetch is a basic HTTP getter without anti-bot bypass, JavaScript rendering, or geo-routing. ZenRows gives you full control over which URL is fetched, how the page is rendered, what format the response comes back in, and which countries the request is routed through. It also handles pages that block standard scrapers, such as geo-restricted content, JS-gated sites, and heavily protected pages.Is there a rate limit I should be aware of?
Is there a rate limit I should be aware of?
scrape_website wrapper.