Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt

Use this file to discover all available pages before exploring further.

Combine the Claude Agent SDK with the ZenRows hosted MCP server to build autonomous, multi-step agents that can read any website in real time. The Agent SDK has native support for MCP servers through its mcp_servers option, so plugging in ZenRows requires a single block of code, with no tool-use boilerplate, no scraping infrastructure, and no anti-bot tuning. This guide covers the Claude Agent SDK. If you want to use ZenRows with the standard Anthropic API (Messages API, tool use, structured outputs, MCP connector), use the ZenRows MCP server with Anthropic’s MCP connector.

What is Claude Agent SDK?

The Claude Agent SDK is Anthropic’s Python and TypeScript library for building production AI agents using the same agent loop, tools, and context management that power Claude Code. It exposes two main entry points: query() for stateless one-shot agent runs, and ClaudeSDKClient for stateful, interactive sessions. Both support MCP servers, custom Python tools, subagents, hooks, and Claude Code’s built-in tools (Read, Write, Edit, Bash, WebSearch, WebFetch, and more). Agents built with the SDK are well-suited to tasks that need multiple steps, autonomous tool selection, or long-running workflows. Adding ZenRows MCP gives every agent in your workflow real-time access to any website, including JavaScript-heavy and anti-bot-protected ones.

Key benefits

  • Native MCP support, zero glue code
    The Agent SDK’s mcp_servers option takes the ZenRows server URL and your API key. The agent then sees ZenRows’ tools (scrape and 30+ browser automation tools) automatically, with no tool use loop to maintain.
  • Autonomous, multi-step web tasks
    Agents can scrape one page, decide what to scrape next based on what they read, and chain together browser interactions like form fills and clicks across multiple pages, without you writing the orchestration loop.
  • Anti-bot bypass on every tool call
    Premium residential proxies, JavaScript rendering, and stealth fingerprinting are built-in standards. Your agents reach JavaScript-heavy SPAs, Cloudflare-protected sites, and geo-restricted pages without configuration.
  • Composes with the full Agent SDK feature set
    ZenRows tools work alongside Claude Code’s built-in tools, custom Python tools, subagents, hooks, and permission gates. Pass them to a single agent or many; the integration is the same.
  • Hosted execution, no local installation
    The hosted MCP server runs on ZenRows infrastructure. Your agent code stays clean, with no proxy management, scraping retries, or headless browser orchestration in your codebase.

Use cases

The Agent SDK and ZenRows combination unlocks a wide range of agent workflows:
Agents that visit multiple URLs, follow citations, and synthesize findings from across the web into a single brief.
Agents that take a company name, scrape the company website, find pricing pages and product launches, and write enriched profiles into your CRM.
Agents that periodically check competitor pricing pages, blog posts, and changelogs, then surface what changed since the last run.
A research subagent that gathers information and hands off to a writer subagent that produces the final report, both pulling from live web data.
Agents that navigate multi-step checkout flows, login-walled forms, or job applications using ZenRows’ browser tools.

Getting started: Basic usage

A minimal working example: a single agent equipped with ZenRows MCP that can answer questions by reading any website in real time.
1

Install the Claude Agent SDK

Python 3.10 or later is required.
pip install claude-agent-sdk
2

Create a .env file with your API keys

ZENROWS_API_KEY=your_zenrows_key
ANTHROPIC_API_KEY=your_anthropic_key
Find your ZenRows API key in the ZenRows dashboard. Find your Anthropic API key in the Claude Console under Settings > API keys.
3

Run the following script

Python
import asyncio
import os
from claude_agent_sdk import query, ClaudeAgentOptions
from dotenv import load_dotenv
load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    options = ClaudeAgentOptions(
        model="claude-sonnet-4-6",
        system_prompt=(
            "You are a research assistant with live web access. "
            "Use the ZenRows tools to scrape pages when you need real-time information."
        ),
        mcp_servers={
            "zenrows": {
                "type": "http",
                "url": "https://mcp.zenrows.com/mcp",
                "headers": {
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            }
        },
        allowed_tools=["mcp__zenrows__*"],
        permission_mode="bypassPermissions",
    )

    async for message in query(
        prompt="Visit https://huggingface.co/blog and summarize the three most recent posts.",
        options=options,
    ):
        if hasattr(message, "result"):
            print(message.result)

asyncio.run(main())
Here’s what each part does:
  • ClaudeAgentOptions configures the agent. The mcp_servers dictionary registers the ZenRows hosted MCP server with type: "http", the server URL, and an Authorization: Bearer header carrying your ZenRows API key as the OAuth token.
  • allowed_tools=["mcp__zenrows__*"] pre-approves every tool from the ZenRows server. The Agent SDK names MCP tools using the pattern mcp__<server-name>__<tool-name>, so the wildcard enables mcp__zenrows__scrape, mcp__zenrows__browser_navigate, and so on.
  • permission_mode="bypassPermissions" lets the agent call tools without an interactive approval prompt. Use this for unattended scripts and production agents. For interactive use, leave it at the default and the SDK will prompt before each tool call.
  • query() runs the agent loop. The agent discovers ZenRows’ tools, picks the right one for the prompt, calls it, and synthesizes the answer. Messages stream back asynchronously; the final answer arrives as a message with a result attribute.

Advanced usage: Building an autonomous research agent

The real power of the Agent SDK is letting an agent decide for itself which tools to call across multiple steps. With ZenRows MCP, that means the agent can autonomously read several pages, follow relevant links, and synthesize findings, all from a single prompt. The example below builds a research agent that visits a blog page, identifies the most relevant linked articles, scrapes each one, and produces a synthesized report:
Python
import asyncio
import os
from claude_agent_sdk import query, ClaudeAgentOptions
from dotenv import load_dotenv
load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    options = ClaudeAgentOptions(
        model="claude-opus-4-7",
        system_prompt=(
            "You are a thorough web researcher. When given a topic and a starting URL, "
            "visit the URL, identify the 3 most relevant linked articles, scrape each one, "
            "and produce a synthesized report covering key findings, common themes, and "
            "any contradictions. Cite each source with its URL."
        ),
        mcp_servers={
            "zenrows": {
                "type": "http",
                "url": "https://mcp.zenrows.com/mcp",
                "headers": {
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            }
        },
        allowed_tools=["mcp__zenrows__*"],
        permission_mode="bypassPermissions",
        max_turns=20,
    )

    async for message in query(
        prompt=(
            "Research recent developments in open-source LLMs. "
            "Start at https://huggingface.co/blog and follow the most relevant links."
        ),
        options=options,
    ):
        if hasattr(message, "result"):
            print(message.result)

asyncio.run(main())
  • Don’t write an orchestration loop in your code. The agent decides on its own when to scrape, which URLs to follow, and when it has enough information to write the report. The SDK runs the multi-step loop automatically.
  • Use a capable model for multi-step tasks. claude-opus-4-7 is the right default for autonomous workflows because it plans its next action based on what it has learned so far. For lighter tasks, claude-sonnet-4-6 is faster and cheaper.
  • max_turns=20 caps the total number of agent turns. This is a useful safety net that prevents indefinite loops. Adjust upward for deeper research, downward for tighter workflows.
  • Same MCP config as the basic example. The only things that changed are the model, the system prompt, and the prompt itself.

Stateful sessions with ClaudeSDKClient

For agents that need to maintain conversation state across multiple user inputs (chatbots, interactive research assistants, follow-up questions), use ClaudeSDKClient instead of query(). The client supports bidirectional, interactive conversations with the same MCP configuration.
Python
import asyncio
import os
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions
from dotenv import load_dotenv
load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    options = ClaudeAgentOptions(
        model="claude-sonnet-4-6",
        system_prompt="You are a web research assistant with access to ZenRows scraping tools.",
        mcp_servers={
            "zenrows": {
                "type": "http",
                "url": "https://mcp.zenrows.com/mcp",
                "headers": {
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            }
        },
        allowed_tools=["mcp__zenrows__*"],
        permission_mode="bypassPermissions",
    )

    async with ClaudeSDKClient(options=options) as client:
        # First turn
        await client.query("Visit https://huggingface.co/blog and summarize the top post.")
        async for message in client.receive_response():
            if hasattr(message, "result"):
                print(message.result)

        # Follow-up turn, sharing conversation history
        await client.query("Now scrape the linked GitHub repo and tell me what license it uses.")
        async for message in client.receive_response():
            if hasattr(message, "result"):
                print(message.result)

asyncio.run(main())
The client preserves conversation context between turns, so the agent remembers what it scraped in the first turn when answering the follow-up. Use query() for one-shot scripts and ClaudeSDKClient whenever you need multi-turn conversation state.

Combining with custom Python tools

The Agent SDK also supports custom Python tools defined as in-process MCP servers. These can be used alongside the ZenRows hosted MCP server in the same agent.
Python
import asyncio
import os
from claude_agent_sdk import (
    query,
    tool,
    create_sdk_mcp_server,
    ClaudeAgentOptions,
)
from dotenv import load_dotenv
load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

# Custom in-process tool
@tool("save_summary", "Save a research summary to a file", {"filename": str, "content": str})
async def save_summary(args):
    with open(args["filename"], "w") as f:
        f.write(args["content"])
    return {
        "content": [
            {"type": "text", "text": f"Saved to {args['filename']}"}
        ]
    }

local_server = create_sdk_mcp_server(
    name="local",
    version="1.0.0",
    tools=[save_summary],
)

async def main() -> None:
    options = ClaudeAgentOptions(
        model="claude-sonnet-4-6",
        system_prompt=(
            "Research the user's topic with ZenRows, then save your final summary "
            "using the save_summary tool."
        ),
        mcp_servers={
            "zenrows": {
                "type": "http",
                "url": "https://mcp.zenrows.com/mcp",
                "headers": {
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            },
            "local": local_server,
        },
        allowed_tools=[
            "mcp__zenrows__*",
            "mcp__local__save_summary",
        ],
        permission_mode="bypassPermissions",
    )

    async for message in query(
        prompt=(
            "Scrape https://huggingface.co/blog, summarize the top three posts, "
            "and save the summary to hf_summary.txt."
        ),
        options=options,
    ):
        if hasattr(message, "result"):
            print(message.result)

asyncio.run(main())
This pattern composes two MCP servers in the same agent: ZenRows for live web access and an in-process Python server for local file writes. The agent autonomously chains them, scraping the blog first and then calling save_summary with the finished text.

Troubleshooting

424 failed dependency or 401 unauthorized from the MCP server

  1. Confirm the Authorization: Bearer YOUR_ZENROWS_API_KEY header is present in the headers dict on the MCP server entry.
  2. Verify your API key in the ZenRows dashboard and confirm your subscription has remaining quota.
  3. Confirm the server URL is exactly https://mcp.zenrows.com/mcp with type: "http".

Agent connects but never calls ZenRows tools

  1. Confirm allowed_tools includes the ZenRows tool names. Without an entry matching mcp__zenrows__* (or specific tool names), the SDK does not expose the tools to the agent.
  2. If permission_mode is at the default, the SDK prompts before each tool call. For unattended scripts, set permission_mode="bypassPermissions".
  3. Strengthen the system prompt to explicitly point the agent at the ZenRows tools, for example, “Use the ZenRows tools to fetch any URL the user mentions.”

Agent repeatedly scrapes the same page

Add explicit guidance in the system prompt about what to do once it has the data, for example, “After scraping each URL once, summarize and stop.” For complex flows, lower max_turns to bound runaway agents.

Page blocked or data missing in scraped content

If the agent connects and invokes tools correctly but the scraped content comes back blocked or incomplete, the issue is from the target page rather than the MCP connection. Add explicit guidance to your prompt, for example “Scrape this page using JavaScript rendering” or “Use Premium Proxies.” See the Universal Scraper API troubleshooting guide for parameter-level fixes.

ANTHROPIC_API_KEY not found

The Agent SDK reads ANTHROPIC_API_KEY from the environment. Confirm the variable is set in your shell or .env file. For Bedrock, Vertex AI, or Azure routing, see the Claude Agent SDK authentication docs.

FAQ (Frequently Asked Questions)

The standard Anthropic Claude integration uses the Messages API, tool use, structured outputs, and the MCP connector. In this pattern, you make one or two model calls and process the results, with the tool-use loop running in your code.The Claude Agent SDK is for autonomous, multi-step agents. The SDK runs the orchestration loop, decides which tools to call, and chains tool calls without you having to write the loop. It also ships with Claude Code’s built-in tools (file editing, shell, web search/fetch) and supports subagents and hooks. Use the standard integration when you need direct control. Use the Agent SDK when you want autonomy, multi-step behavior, and Claude Code’s broader toolset.
All current Claude 4 models. For autonomous multi-step workflows, claude-opus-4-7 is the strongest. For balanced cost-performance, claude-sonnet-4-6 is a good default. For high-throughput, cost-sensitive workloads, claude-haiku-4-5 is a faster, cheaper option.
Yes. Use the command and args form instead of type: "http":
Python
mcp_servers={
    "zenrows": {
        "command": "npx",
        "args": ["-y", "@zenrows/mcp"],
        "env": {"ZENROWS_API_KEY": ZENROWS_API_KEY},
    }
}
The hosted HTTP URL is recommended because it requires no local installation and runs the same way in development and production. See the ZenRows MCP overview for the local stdio configuration.
Yes. Build the mcp_servers dict and allowed_tools list once and reuse them across as many ClaudeAgentOptions instances as you need. Each agent gets independent tool calls, and ZenRows handles them all on the same hosted server.
Yes. Both query() and ClaudeSDKClient.receive_response() are async iterators that stream messages as the agent works. Each message can be a tool call, a tool result, a partial text response, or the final result. Use if hasattr(message, "result") to filter for the final answer, or inspect all messages for full traces.
Yes. Add the built-in tool names to allowed_tools alongside the ZenRows wildcards. This is the right pattern for agents that need to scrape the web, write to local files, run shell commands, or edit code. See the Combining with custom Python tools section above for a similar pattern with in-process MCP servers.
The Claude Agent SDK and OpenAI Agents SDK solve similar problems with different design philosophies. The Claude Agent SDK ships as “Claude Code as a library,” so it includes a rich set of built-in tools (file editing, shell, web search/fetch) and is tightly coupled with the Claude model family. The OpenAI Agents SDK is a thinner orchestration layer optimized for the OpenAI Responses API and GPT models, with a strong handoff and tracing story. ZenRows MCP works with both — pick the SDK that fits your model and toolset preferences. See the OpenAI Agents SDK integration guide for the OpenAI equivalent of this doc.