Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt

Use this file to discover all available pages before exploring further.

Combine the OpenAI Agents SDK with the ZenRows hosted MCP server to build autonomous, multi-step agents that can read any website in real time. The Agents SDK has native support for MCP servers, so plugging in ZenRows takes a single block of code, with no function-calling boilerplate, no scraping infrastructure, and no anti-bot tuning. This guide covers the OpenAI Agents SDK. If you want to use ZenRows with the standard OpenAI API, see How to Integrate OpenAI with ZenRows.

What is OpenAI Agents SDK?

The OpenAI Agents SDK is a lightweight Python framework for building multi-agent workflows. It provides a small set of primitives, including agents, tools, handoffs, guardrails, sessions, and tracing, that compose into autonomous AI applications. Agents are LLMs equipped with instructions and tools. The SDK handles the orchestration loop, tool calls, handoffs between agents, and conversation state. Agents are well-suited to tasks that need multiple steps, tool selection, or coordination across specialized roles. Adding ZenRows MCP gives every agent in your workflow real-time access to any website, including JavaScript-heavy and bot-protected ones.

Key benefits of integrating OpenAI Agents SDK with ZenRows

  • Native MCP support, zero glue code: The Agents SDK’s HostedMCPTool wrapper takes the ZenRows server URL and your API key. The agent then sees ZenRows’ tools (scrape and 30+ browser automation tools) automatically, with no function-calling code to maintain.
  • Autonomous, multi-step web tasks: Agents can scrape one page, decide what to scrape next based on what they read, and chain together browser interactions like form fills and clicks across multiple pages, without you writing the orchestration loop.
  • Anti-bot bypass on every tool call: Premium residential proxies, JavaScript rendering, and stealth fingerprinting come standard. Your agents reach JavaScript-heavy SPAs, Cloudflare-protected sites, and geo-restricted pages without configuration.
  • Composes with handoffs, tracing, and sessions: ZenRows tools work alongside every Agents SDK feature. Pass them to multiple agents, hand off mid-workflow, persist sessions across runs, and trace every tool call in the OpenAI dashboard.
  • Hosted execution, no local install: The hosted MCP server runs on ZenRows infrastructure. Your agent code stays clean, with no proxy management, scraping retries, or headless browser orchestration in your codebase.

Use cases

The Agents SDK and ZenRows combination unlocks a wide range of agent workflows:
  • Autonomous research agents: Agents that visit multiple URLs, follow citations, and synthesize findings from across the web into a single brief.
  • Lead enrichment pipelines: Agents that take a company name, scrape the company website, find pricing pages and product launches, and write enriched profiles into your CRM.
  • Competitive monitoring: Agents that periodically check competitor pricing pages, blog posts, and changelogs, then surface what changed since the last run.
  • Multi-agent web automation: A research agent that gathers information and hands off to a writing agent that produces the final report, both pulling from live web data.
  • Form-filling and extraction agents: Agents that navigate multi-step checkout flows using ZenRows’ browser tools.

Getting started: Basic Usage

Let’s start with a simple example: a single agent equipped with ZenRows MCP that can answer questions by reading any website in real time.
1

Install the OpenAI Agents SDK (Python 3.10+ required)

pip install openai-agents
2

Create a .env file with your API keys

ZENROWS_API_KEY=your_zenrows_key
OPENAI_API_KEY=your_openai_key
3

Run the following script

Python
import asyncio
import os
from agents import Agent, HostedMCPTool, Runner

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    agent = Agent(
        name="Web Research Assistant",
        instructions=(
            "You are a research assistant with live web access. "
            "Scrape pages when you need real-time information."
        ),
        tools=[
            HostedMCPTool(
                tool_config={
                    "type": "mcp",
                    "server_label": "zenrows",
                    "server_description": "Web scraping MCP server for accessing live web content.",
                    "server_url": "https://mcp.zenrows.com/mcp",
                    "authorization": ZENROWS_API_KEY,
                    "require_approval": "never",
                }
            )
        ],
    )

    result = await Runner.run(
        agent,
        "Visit https://news.ycombinator.com/ and summarize the three most recent posts.",
    )

    print(result.final_output)

asyncio.run(main())
HostedMCPTool registers the ZenRows hosted MCP server as a tool. The tool_config mirrors the Responses API MCP tool config: server_url points at ZenRows, authorization carries your API key as the OAuth Bearer token, and require_approval="never" skips the per-call approval step. Agent equips the agent with that tool. The agent now has access to all ZenRows tools (scrape, browser_navigate, browser_click, and so on) automatically. Runner.run() executes the agent. The agent reads the prompt, decides to call ZenRows’ scrape tool with the Hacker News URL, gets the content back as Markdown, and writes a summary.

Example output

Output
Here are the three most recent posts on the front page of Hacker News (https://news.ycombinator.com/) as of now:

### 1. [Accelerating Gemma 4: faster inference with multi-token prediction drafters](https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/)
- **Source:** blog.google
- **Summary:** Google shares an update on improving inference speed for its Gemma 4 AI model...
---
### 2. [Three Inverse Laws of AI](https://susam.net/inverse-laws-of-robotics.html)
- **Source:** susam.net
- **Summary:** A blog discusses three proposed "inverse laws" for artificial intelligence, likely as a counterpoint to Isaac Asimov's classic...
---
### 3. [Computer Use is 45x more expensive than structured APIs](https://reflex.dev/blog/computer-use-is-45x-more-expensive-than-structured-apis/)
- **Source:** reflex.dev
- **Summary:** This post analyzes the cost disparity between general computer use (running code, scripts, unstructured processing) and consuming structured APIs...
You’ve built a working web-aware agent in under 30 lines of code. Let’s look at more advanced patterns.

Advanced Usage: Building an autonomous research agent

The real power of the Agents SDK is letting an agent decide for itself which tools to call across multiple steps. With ZenRows MCP, that means the agent can autonomously read several pages, follow links it finds interesting, and synthesize findings, all from a single prompt. The example below builds a research agent that visits a topic landing page, identifies the most relevant linked articles, scrapes each one, and produces a synthesized report:
Python
import asyncio
import os
from agents import Agent, HostedMCPTool, Runner

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    research_agent = Agent(
        name="Research Agent",
        instructions=(
            "You are a thorough web researcher. When given a topic and a starting URL, "
            "visit the URL, identify the 3 most relevant linked articles, scrape each one, "
            "and produce a synthesized report covering key findings, common themes, and "
            "any contradictions. Cite each source with its URL."
        ),
        tools=[
            HostedMCPTool(
                tool_config={
                    "type": "mcp",
                    "server_label": "zenrows",
                    "server_description": "Web scraping MCP server for accessing live web content.",
                    "server_url": "https://mcp.zenrows.com/mcp",
                    "authorization": ZENROWS_API_KEY,
                    "require_approval": "never",
                }
            )
        ],
        model="gpt-4o",
    )

    result = await Runner.run(
        research_agent,
        "Research recent developments in AI. Start at https://huggingface.co/blog and follow the most relevant links.",
    )

    print(result.final_output)

asyncio.run(main())
A few things worth noting:
  • No orchestration loop in your code. The agent decides on its own when to scrape, which URLs to follow, and when it has enough information to write the report. The Agents SDK runs the multi-step loop automatically.
  • Use a capable model for multi-step tasks. Models like gpt-4o work better for autonomous workflows because they can plan their next action based on what they’ve learned so far.
  • Same MCP config as the basic example. The only things that changed are the agent’s instructions and the prompt. ZenRows handles the scraping; the SDK handles the loop.

Structured outputs

The Agents SDK supports strongly-typed outputs through the output_type parameter on the Agent constructor. Combined with ZenRows, this is the cleanest way to extract typed data from any web page in an agent workflow. The example below scrapes a company homepage and extracts a strongly-typed CompanyInfo object:
Python
import asyncio
import os
from agents import Agent, HostedMCPTool, Runner
from pydantic import BaseModel

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

class CompanyInfo(BaseModel):
    name: str
    industry: str
    description: str

async def main() -> None:
    agent = Agent(
        name="Company Researcher",
        instructions="Visit the URL the user provides and extract structured company information from the page.",
        tools=[
            HostedMCPTool(
                tool_config={
                    "type": "mcp",
                    "server_label": "zenrows",
                    "server_description": "Web scraping MCP server for accessing live web content.",
                    "server_url": "https://mcp.zenrows.com/mcp",
                    "authorization": ZENROWS_API_KEY,
                    "require_approval": "never",
                }
            )
        ],
        output_type=CompanyInfo,
        model="gpt-4o-mini",
    )

    result = await Runner.run(
        agent,
        "Extract company information from https://www.zillow.com/",
    )

    company: CompanyInfo = result.final_output
    print(company.model_dump_json(indent=2))

asyncio.run(main())

Example output

JSON
{
  "name": "Zillow (Zillow Group)",
  "industry": "Real estate / Online real estate marketplace",
  "description": "Zillow is an online real estate marketplace and services company offering home listings for sale and rent, rental-management and agent-finding tools, home value estimates (Zestimates), mortgage products through Zillow Home Loans, and affiliated brands such as Trulia, StreetEasy, and HotPads."
}
The output_type parameter makes the SDK enforce the Pydantic schema on the final output. The agent autonomously calls ZenRows’ scrape tool to fetch the page, then parses the Markdown content into the CompanyInfo model. Type-safe access to the result is available through result.final_output. The same pattern works for any structured extraction job: product listings, job postings, real estate, reviews, articles, and so on. Define the Pydantic schema, point the agent at the URL, and let it return validated data.
Skip the schema with ZenRows AutoparseFor common site types like e-commerce products, articles, jobs, and real estate listings, you can skip LLM-based parsing entirely. Tell your agent to scrape with autoparse enabled, and ZenRows automatically identifies and extracts the page’s structured data into clean JSON, with no Pydantic schema or extra model call required.ZenRows Autoparse is included at no additional cost. The agent can use the structured JSON directly, or pass it to GPT for downstream enrichment, normalization, or categorization. Pre-structured input also uses far fewer tokens than raw Markdown, which keeps your context window lean and your agent runs cheaper.Note that Autoparse cannot be combined with Markdown output in the same scrape call.

Handoffs and multi-agent workflows

Because ZenRows MCP is a standard tool on the agent, you can mix it with the SDK’s other primitives like handoffs, sessions, and guardrails. A common pattern is to use one agent for research (with ZenRows access) and another for writing (without web tools), passing control between them via handoffs:
Python
import os
from agents import Agent, HostedMCPTool

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

zenrows_tool = HostedMCPTool(
    tool_config={
        "type": "mcp",
        "server_label": "zenrows",
        "server_description": "Web scraping MCP server for accessing live web content.",
        "server_url": "https://mcp.zenrows.com/mcp",
        "authorization": ZENROWS_API_KEY,
        "require_approval": "never",
    }
)

writer_agent = Agent(
    name="Writer Agent",
    instructions="Take research findings and produce a polished article.",
    model="gpt-4o-mini",
)

researcher_agent = Agent(
    name="Researcher Agent",
    instructions=(
        "Gather research from the web on the user's topic. "
        "Once you have enough material, hand off to the Writer Agent."
    ),
    tools=[zenrows_tool],
    handoffs=[writer_agent],
    model="gpt-4o",
)
The researcher agent calls ZenRows tools to gather material, then hands off to the writer agent (which has no web access) for the final draft. See the Agents SDK handoffs documentation for the full pattern.

Tool filtering and approval gates

The ZenRows MCP server exposes 30+ tools by default. For most agents this is fine, but two SDK features are worth knowing about for production use.

Restrict which tools the agent sees

If you only want the agent to use specific ZenRows tools (for example, only scrape and not the browser automation tools), you can filter at the SDK level. See Tool filtering in the Agents SDK MCP docs for the exact API.

Require approval for sensitive tool calls

By default, the examples above set require_approval="never", which lets the agent call tools without confirmation. For agents that interact with logged-in pages, submit forms, or affect external state, set require_approval="always" and provide an on_approval_request callback to gate execution behind your own logic. See Optional approval flows for details.

Troubleshooting

424 failed dependency or 401 unauthorized from the MCP tool

  • Option 1: Confirm the authorization field is set on the tool_config and that the value is your active ZenRows API key. OpenAI does not retain header values between requests, so the field must be present on every call.
  • Option 2: Verify your key in the ZenRows dashboard and confirm your subscription has remaining quota.
  • Option 3: Confirm server_url is exactly https://mcp.zenrows.com/mcp.

Empty tool list or empty output

If the agent’s response indicates it didn’t find any tools, the connection to the MCP server failed silently. Check your API key and server_url.

Agent repeatedly scrapes the same page

Add explicit guidance in the agent’s instructions about what to do once it has the data, for example “After scraping, summarize and stop.” For complex flows, set a tool-call limit in model_settings.

Reasoning items missing after multi-step runs

The Agents SDK handles reasoning-item passing automatically. If you’re seeing empty model responses on multi-step runs, ensure you’re using a capable model like gpt-4o or gpt-4o-mini and that your prompt asks the model to take multiple steps.

Content-level issues (page blocked, missing data)

If the agent connects and invokes tools correctly but the scraped content comes back blocked or incomplete, the issue is on the target page rather than the MCP connection. Add explicit guidance to your prompt, for example “Scrape this page using JavaScript rendering” or “Use Premium Proxies.” See the Universal Scraper API troubleshooting guide for parameter-level fixes.

Helpful resources

Frequently asked questions

The standard OpenAI integration covers the Responses API, function calling, and structured outputs. These are patterns where you make one or two model calls and process the result.The Agents SDK is for autonomous, multi-step agents: the SDK runs the orchestration loop, decides which tools to call, and chains tool calls without you writing the glue. Use the standard integration when you need direct control. Use the Agents SDK when you want autonomy and multi-step behavior.
No. HostedMCPTool runs through OpenAI’s infrastructure, so it goes only in the agent’s tools list. mcp_servers is for SDK-managed connections (MCPServerStreamableHttp, MCPServerStdio, MCPServerSse).
Yes. Use MCPServerStdio from agents.mcp with the @zenrows/mcp npm package. This is less common in agent applications because the hosted remote URL is simpler to set up and runs on OpenAI’s infrastructure rather than in your process. See the ZenRows MCP overview for the local stdio configuration.
Yes. Build the HostedMCPTool once and pass it to as many agents as you need:
zenrows_tool = HostedMCPTool(tool_config={...})
agent_a = Agent(name="A", tools=[zenrows_tool])
agent_b = Agent(name="B", tools=[zenrows_tool])
Each agent gets independent tool calls. ZenRows handles them all on the same hosted server.
Yes. Use Runner.run_streamed() instead of Runner.run() to receive incremental events as the agent works. See Streaming hosted MCP results in the Agents SDK docs.
Yes. The Agents SDK traces every MCP list_tools and tool-call invocation. You can view them in the OpenAI dashboard alongside your agent’s other steps.
The Agents SDK is OpenAI’s first-party agent framework, optimized for the Responses API and reasoning models. LangChain and LangGraph are framework-agnostic and have a larger ecosystem of integrations. ZenRows MCP works with both. Pick the framework that fits your stack.