> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How to Integrate Zenrows with CrewAI

> Connect the Zenrows MCP to CrewAI and give your agent crews reliable access to any website, including pages behind anti-bot protection and heavy JavaScript. No custom tool code required.

Connect <a href="https://www.crewai.com" target="_blank" rel="nofollow noopener noreferrer">CrewAI</a> to the Zenrows MCP server and every agent in your crew can read live web pages, including the ones that return a block page to an ordinary HTTP request. No tool class to write and nothing to publish, just an MCP connection your crew opens at startup.

## What Is CrewAI?

<a href="https://www.crewai.com" target="_blank" rel="nofollow noopener noreferrer">CrewAI</a> is an open-source Python framework for orchestrating teams of autonomous AI agents. You define agents with a role, a goal, and a backstory, give each one tools, and assign tasks. CrewAI handles delegation, sequencing, and the hand-off of one agent's output into the next agent's input.

Its `crewai-tools` package ships integrations for search, databases, file systems, and scraping, plus `MCPServerAdapter`, which pulls tools from any MCP server into a crew at runtime.

## Why Connect Zenrows to CrewAI?

* **Agents that don't get blocked:** a research agent calling `requests` or a plain HTTP tool gets a challenge page from any site behind Cloudflare or DataDome, then reasons confidently over the wrong content. Zenrows returns the real page.
* **No tool code to maintain:** `MCPServerAdapter` discovers the tools at runtime, so there's no `BaseTool` subclass, no Pydantic schema, and no package of your own to version.
* **Markdown instead of HTML:** the `scrape` tool returns clean Markdown by default, which costs far fewer tokens than raw HTML and gives the agent less noise to reason through.
* **Every agent in the crew shares one connection:** open the adapter once and pass `tools` to as many agents as you like.
* **Model-agnostic:** the tools arrive as ordinary CrewAI tools, so they work with whichever LLM your crew is configured to use.

## What You Can Build with CrewAI and Zenrows

* **A research crew with a scraper and an analyst:** one agent gathers pages, a second synthesizes them into a brief, and the scraping agent never returns a challenge page instead of content.
* **Competitive price monitoring:** a crew that walks a list of competitor product pages, extracts prices, and reports the deltas on a schedule.
* **Lead enrichment:** an agent reads a prospect's site, careers page, and pricing page, and hands a structured profile to a writer agent that drafts the outreach.
* **Content pipelines:** a crew that pulls source articles, extracts the substance, and produces a draft, with the sources fetched reliably rather than best-effort.

## Prerequisites

* Python 3.12 or 3.13, and an existing CrewAI project. `crewai-tools` requires
  `>=3.10,<3.14` and `mcpadapt` requires `>=3.12`, so the MCP path needs a
  version in that overlap. On Python 3.14, pip silently resolves
  `crewai-tools` down to a placeholder 0.0.1 release that has no `mcp` extra.
* An LLM provider key configured for CrewAI (`OPENAI_API_KEY` by default).
* A Zenrows API key from your <a href="https://app.zenrows.com/settings/api-keys" target="_blank" rel="nofollow noopener noreferrer">Zenrows dashboard</a>.

## Setup

<Steps>
  <Step title="Install the MCP extra">
    MCP support is an optional extra, not part of the base `crewai-tools` install:

    ```bash theme={"dark"}
    pip install 'crewai-tools[mcp]'
    ```

    The extra pulls in `mcp` and `mcpadapt`, which `MCPServerAdapter` needs. Without it, constructing the adapter offers to install the packages for you and raises an `ImportError` if you decline.
  </Step>

  <Step title="Connect to the Zenrows MCP server">
    Point `MCPServerAdapter` at `https://mcp.zenrows.com/mcp` and pass your key as a Bearer token.

    ```python Python theme={"dark"}
    import os
    from crewai_tools import MCPServerAdapter

    server_params = {
        "url": "https://mcp.zenrows.com/mcp",
        "transport": "streamable-http",
        "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
    }

    with MCPServerAdapter(server_params) as tools:
        print([tool.name for tool in tools])
    ```

    <Warning>
      **`"transport": "streamable-http"` is required.** `MCPServerAdapter` passes your dictionary to `mcpadapt`, which defaults to the deprecated SSE transport when no transport is given. Omit the line and the connection is attempted over SSE against an endpoint that speaks Streamable HTTP, and it fails. CrewAI's own docstring documents STDIO and SSE only, so this is not discoverable from the framework's documentation.
    </Warning>

    Use the adapter as a context manager, as above, and the server connection closes with the block. If you construct it directly instead, call `stop()` in a `finally` block once the crew has finished.
  </Step>

  <Step title="Give the tools to an agent and run the crew">
    The adapter yields ordinary CrewAI tools, so they go straight onto an agent.

    ```python Python theme={"dark"}
    import os
    from crewai import Agent, Crew, Task
    from crewai_tools import MCPServerAdapter

    server_params = {
        "url": "https://mcp.zenrows.com/mcp",
        "transport": "streamable-http",
        "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
    }

    with MCPServerAdapter(server_params, "scrape") as tools:
        researcher = Agent(
            role="Web Researcher",
            goal="Read live web pages and report exactly what they say",
            backstory=(
                "You read pages through Zenrows, which returns the real content "
                "even when a site blocks ordinary requests."
            ),
            tools=tools,
            verbose=True,
        )

        task = Task(
            description=(
                "Read https://www.scrapingcourse.com/ecommerce/ and list three "
                "products with their prices."
            ),
            expected_output="A Markdown list of three product names with prices.",
            agent=researcher,
        )

        result = Crew(agents=[researcher], tasks=[task]).kickoff()
        print(result)
    ```

    The second argument, `"scrape"`, filters the tool set. The next section explains why that matters.
  </Step>
</Steps>

## Filter to the Tools Your Crew Needs

The Zenrows MCP server exposes three families of tools: `scrape`, the `batch_*` job tools, and the `browser_*` session tools. See the [MCP overview](/mcp/overview) for the full list.

Every tool you load contributes its schema to the prompt on every request. A crew that only reads pages does not need the batch job lifecycle or browser session control, so pass the names you want as positional arguments:

```python Python theme={"dark"}
with MCPServerAdapter(server_params, "scrape") as tools:
    ...
```

Leave the names out and the crew receives everything the server offers.

## Raise the Timeout for Slow Pages

Keys in `server_params` other than `transport` are forwarded to the underlying Streamable HTTP client, whose request timeout defaults to 30 seconds. A page that needs JavaScript rendering or an anti-bot bypass can take longer than that, and the call fails on the client side while the request is still in flight.

```python Python theme={"dark"}
server_params = {
    "url": "https://mcp.zenrows.com/mcp",
    "transport": "streamable-http",
    "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
    "timeout": 120,
}
```

`MCPServerAdapter`'s own `connect_timeout` argument is separate and covers establishing the connection, not individual tool calls:

```python Python theme={"dark"}
MCPServerAdapter(server_params, connect_timeout=60)
```

## Troubleshooting

### ImportError or a Prompt to Install `mcp`

The MCP extra is not installed. Run `pip install 'crewai-tools[mcp]'`. Installing `mcp` alone is not enough, because `MCPServerAdapter` also needs `mcpadapt`.

### The Connection Hangs or Fails Immediately

The transport is missing. Confirm `"transport": "streamable-http"` is present in `server_params`. Without it the client attempts SSE, which the Zenrows MCP endpoint does not serve.

### Authentication Errors from the Server

Check the `Authorization` header is exactly `Bearer YOUR_ZENROWS_API_KEY`, with a single space and no trailing whitespace. Verify the key in your <a href="https://app.zenrows.com/settings/api-keys" target="_blank" rel="nofollow noopener noreferrer">Zenrows dashboard</a>.

### Tool Calls Time Out on Heavy Pages

Raise `timeout` in `server_params`, as described above. The default is 30 seconds, which is short for a rendered or protected page.

### The Agent Ignores the Scraping Tool

Make the task description name the URL explicitly and state that the page must be read rather than recalled. Filtering to `scrape` also helps, because a crew offered a dozen tools has more ways to choose wrongly.

### The Prompt Grows Too Large

Every loaded tool's schema is sent with each request, and page content is added on top. Filter the tool set to `scrape`, or move to a model with a larger context window.

## Further Reading

* <a href="https://docs.crewai.com/en/mcp/overview" target="_blank" rel="nofollow noopener noreferrer">CrewAI MCP Integration</a>
* <a href="https://docs.crewai.com/en/concepts/agents" target="_blank" rel="nofollow noopener noreferrer">CrewAI Agents</a>
* [Zenrows MCP Documentation](/mcp/overview)
* [Fetch API Reference](/fetch/api-reference)

## Frequently Asked Questions

<Accordion title="Do I need to write a custom CrewAI tool for Zenrows?">
  No. `MCPServerAdapter` discovers the Zenrows tools at runtime and adapts them into CrewAI tools, so there is no tool class to write, test, or keep in step with the API.
</Accordion>

<Accordion title="Why does the connection fail without the transport line?">
  `MCPServerAdapter` hands your dictionary to `mcpadapt`, which reads the transport key and falls back to SSE when it is absent. SSE is the deprecated MCP transport and is not what the Zenrows endpoint serves, so the connection cannot complete. Setting `"transport": "streamable-http"` selects the right client.
</Accordion>

<Accordion title="Can several agents in one crew share the same Zenrows tools?">
  Yes. Open the adapter once and pass the same `tools` value to each agent that needs web access. One connection serves the whole crew.
</Accordion>

<Accordion title="Can I use Zenrows alongside other MCP servers?">
  Yes. Create one `MCPServerAdapter` per server and combine the tool lists when constructing an agent.
</Accordion>

<Accordion title="Does this work with a local Zenrows MCP server instead?">
  Yes. `MCPServerAdapter` also accepts `StdioServerParameters` for a local subprocess. See the [MCP overview](/mcp/overview) for the local server configuration.
</Accordion>