How to Integrate Zenrows with CrewAI

Connect CrewAI to the Zenrows MCP server and every agent in your crew can read live web pages, including the ones that return a block page to an ordinary HTTP request. No tool class to write and nothing to publish, just an MCP connection your crew opens at startup.

What Is CrewAI?

CrewAI is an open-source Python framework for orchestrating teams of autonomous AI agents. You define agents with a role, a goal, and a backstory, give each one tools, and assign tasks. CrewAI handles delegation, sequencing, and the hand-off of one agent’s output into the next agent’s input. Its crewai-tools package ships integrations for search, databases, file systems, and scraping, plus MCPServerAdapter, which pulls tools from any MCP server into a crew at runtime.

Why Connect Zenrows to CrewAI?

Agents that don’t get blocked: a research agent calling requests or a plain HTTP tool gets a challenge page from any site behind Cloudflare or DataDome, then reasons confidently over the wrong content. Zenrows returns the real page.
No tool code to maintain: MCPServerAdapter discovers the tools at runtime, so there’s no BaseTool subclass, no Pydantic schema, and no package of your own to version.
Markdown instead of HTML: the scrape tool returns clean Markdown by default, which costs far fewer tokens than raw HTML and gives the agent less noise to reason through.
Every agent in the crew shares one connection: open the adapter once and pass tools to as many agents as you like.
Model-agnostic: the tools arrive as ordinary CrewAI tools, so they work with whichever LLM your crew is configured to use.

What You Can Build with CrewAI and Zenrows

A research crew with a scraper and an analyst: one agent gathers pages, a second synthesizes them into a brief, and the scraping agent never returns a challenge page instead of content.
Competitive price monitoring: a crew that walks a list of competitor product pages, extracts prices, and reports the deltas on a schedule.
Lead enrichment: an agent reads a prospect’s site, careers page, and pricing page, and hands a structured profile to a writer agent that drafts the outreach.
Content pipelines: a crew that pulls source articles, extracts the substance, and produces a draft, with the sources fetched reliably rather than best-effort.

Prerequisites

Python 3.12 or 3.13, and an existing CrewAI project. crewai-tools requires >=3.10,<3.14 and mcpadapt requires >=3.12, so the MCP path needs a version in that overlap. On Python 3.14, pip silently resolves crewai-tools down to a placeholder 0.0.1 release that has no mcp extra.
An LLM provider key configured for CrewAI (OPENAI_API_KEY by default).
A Zenrows API key from your Zenrows dashboard.

Setup

Install the MCP extra

MCP support is an optional extra, not part of the base crewai-tools install:

pip install 'crewai-tools[mcp]'

The extra pulls in mcp and mcpadapt, which MCPServerAdapter needs. Without it, constructing the adapter offers to install the packages for you and raises an ImportError if you decline.

Connect to the Zenrows MCP server

Point MCPServerAdapter at https://mcp.zenrows.com/mcp and pass your key as a Bearer token.

Python

import os
from crewai_tools import MCPServerAdapter

server_params = {
    "url": "https://mcp.zenrows.com/mcp",
    "transport": "streamable-http",
    "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
}

with MCPServerAdapter(server_params) as tools:
    print([tool.name for tool in tools])

"transport": "streamable-http" is required. MCPServerAdapter passes your dictionary to mcpadapt, which defaults to the deprecated SSE transport when no transport is given. Omit the line and the connection is attempted over SSE against an endpoint that speaks Streamable HTTP, and it fails. CrewAI’s own docstring documents STDIO and SSE only, so this is not discoverable from the framework’s documentation.

Use the adapter as a context manager, as above, and the server connection closes with the block. If you construct it directly instead, call stop() in a finally block once the crew has finished.

Give the tools to an agent and run the crew

The adapter yields ordinary CrewAI tools, so they go straight onto an agent.

Python

import os
from crewai import Agent, Crew, Task
from crewai_tools import MCPServerAdapter

server_params = {
    "url": "https://mcp.zenrows.com/mcp",
    "transport": "streamable-http",
    "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
}

with MCPServerAdapter(server_params, "scrape") as tools:
    researcher = Agent(
        role="Web Researcher",
        goal="Read live web pages and report exactly what they say",
        backstory=(
            "You read pages through Zenrows, which returns the real content "
            "even when a site blocks ordinary requests."
        ),
        tools=tools,
        verbose=True,
    )

    task = Task(
        description=(
            "Read https://www.scrapingcourse.com/ecommerce/ and list three "
            "products with their prices."
        ),
        expected_output="A Markdown list of three product names with prices.",
        agent=researcher,
    )

    result = Crew(agents=[researcher], tasks=[task]).kickoff()
    print(result)

The second argument, "scrape", filters the tool set. The next section explains why that matters.

Filter to the Tools Your Crew Needs

The Zenrows MCP server exposes three families of tools: scrape, the batch_* job tools, and the browser_* session tools. See the MCP overview for the full list. Every tool you load contributes its schema to the prompt on every request. A crew that only reads pages does not need the batch job lifecycle or browser session control, so pass the names you want as positional arguments:

Python

with MCPServerAdapter(server_params, "scrape") as tools:
    ...

Leave the names out and the crew receives everything the server offers.

Raise the Timeout for Slow Pages

Keys in server_params other than transport are forwarded to the underlying Streamable HTTP client, whose request timeout defaults to 30 seconds. A page that needs JavaScript rendering or an anti-bot bypass can take longer than that, and the call fails on the client side while the request is still in flight.

Python

server_params = {
    "url": "https://mcp.zenrows.com/mcp",
    "transport": "streamable-http",
    "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
    "timeout": 120,
}

MCPServerAdapter’s own connect_timeout argument is separate and covers establishing the connection, not individual tool calls:

Python

MCPServerAdapter(server_params, connect_timeout=60)

Troubleshooting

ImportError or a Prompt to Install `mcp`

The MCP extra is not installed. Run pip install 'crewai-tools[mcp]'. Installing mcp alone is not enough, because MCPServerAdapter also needs mcpadapt.

The Connection Hangs or Fails Immediately

The transport is missing. Confirm "transport": "streamable-http" is present in server_params. Without it the client attempts SSE, which the Zenrows MCP endpoint does not serve.

Authentication Errors from the Server

Check the Authorization header is exactly Bearer YOUR_ZENROWS_API_KEY, with a single space and no trailing whitespace. Verify the key in your Zenrows dashboard.

Tool Calls Time Out on Heavy Pages

Raise timeout in server_params, as described above. The default is 30 seconds, which is short for a rendered or protected page.

The Agent Ignores the Scraping Tool

Make the task description name the URL explicitly and state that the page must be read rather than recalled. Filtering to scrape also helps, because a crew offered a dozen tools has more ways to choose wrongly.

The Prompt Grows Too Large

Every loaded tool’s schema is sent with each request, and page content is added on top. Filter the tool set to scrape, or move to a model with a larger context window.

Frequently Asked Questions

Do I need to write a custom CrewAI tool for Zenrows?

No. MCPServerAdapter discovers the Zenrows tools at runtime and adapts them into CrewAI tools, so there is no tool class to write, test, or keep in step with the API.

Why does the connection fail without the transport line?

MCPServerAdapter hands your dictionary to mcpadapt, which reads the transport key and falls back to SSE when it is absent. SSE is the deprecated MCP transport and is not what the Zenrows endpoint serves, so the connection cannot complete. Setting "transport": "streamable-http" selects the right client.

Can I use Zenrows alongside other MCP servers?

Yes. Create one MCPServerAdapter per server and combine the tool lists when constructing an agent.

Does this work with a local Zenrows MCP server instead?

Yes. MCPServerAdapter also accepts StdioServerParameters for a local subprocess. See the MCP overview for the local server configuration.

Get Started

AI Assistants

AI Code Editors

Agent SDKs & Frameworks

AI Tools

Agent Platforms

No-Code & Automation

Browser & Scraping Frameworks

Captcha Solvers

How to Integrate Zenrows with CrewAI

What Is CrewAI?

Why Connect Zenrows to CrewAI?

What You Can Build with CrewAI and Zenrows

Prerequisites

Setup

Filter to the Tools Your Crew Needs

Raise the Timeout for Slow Pages

Troubleshooting

ImportError or a Prompt to Install `mcp`

The Connection Hangs or Fails Immediately

Authentication Errors from the Server

Tool Calls Time Out on Heavy Pages

The Agent Ignores the Scraping Tool

The Prompt Grows Too Large

Further Reading

Frequently Asked Questions

​What Is CrewAI?

​Why Connect Zenrows to CrewAI?

​What You Can Build with CrewAI and Zenrows

​Prerequisites

​Setup

​Filter to the Tools Your Crew Needs

​Raise the Timeout for Slow Pages

​Troubleshooting

​ImportError or a Prompt to Install mcp

​The Connection Hangs or Fails Immediately

​Authentication Errors from the Server

​Tool Calls Time Out on Heavy Pages

​The Agent Ignores the Scraping Tool

​The Prompt Grows Too Large

​Further Reading

​Frequently Asked Questions

What Is CrewAI?

Why Connect Zenrows to CrewAI?

What You Can Build with CrewAI and Zenrows

Prerequisites

Setup

Filter to the Tools Your Crew Needs

Raise the Timeout for Slow Pages

Troubleshooting

ImportError or a Prompt to Install `mcp`

The Connection Hangs or Fails Immediately

Authentication Errors from the Server

Tool Calls Time Out on Heavy Pages

The Agent Ignores the Scraping Tool

The Prompt Grows Too Large

Further Reading

Frequently Asked Questions