> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Move an existing scraper to Zenrows

> Move an existing Requests, Puppeteer, Playwright or Scrapy scraper to Zenrows without rewriting your parsing or storage.

Keep your scraper and change only the layer that gets blocked. An HTTP scraper swaps its request for one call to Fetch. A Puppeteer or Playwright scraper swaps `launch()` for a connection to Browser. Your parsing, storage and scheduling stay as they are, and you can retire proxy rotation and header tricks.

**Uses:** [Fetch](/fetch/api-reference) with [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode) (`mode=auto`), [Browser](/browser-sessions/introduction), [`css_extractor`](/fetch/features/css-extractor), [Extract](/extract/introduction), [Batch](/batch/introduction)

## Ask your AI agent

With the [Zenrows MCP server](/mcp/overview) connected to Claude Code, Cursor or another coding agent, open your scraper's code and paste this prompt:

```text theme={"dark"}
Migrate this scraper to Zenrows, using the Zenrows docs. Send each request through Zenrows
with Adaptive Stealth Mode and a 3-minute timeout, and read the API key from the
ZENROWS_API_KEY environment variable. Remove custom proxies and User-Agent rotation and keep
the parsing code unchanged. Then use Zenrows to read one of the target URLs and confirm the selectors still match.
```

## Choose your path

| Your scraper uses | Move it to | What changes |
| - | - | - |
| Requests, httpx, axios, `fetch` | [Fetch](#replace-the-http-request-with-fetch) | The request URL and params. Parsing stays |
| BeautifulSoup or Cheerio selectors | [`css_extractor` or Extract](#drop-the-parser-with-a-css-extractor-or-extract) | Optional: Zenrows returns JSON, so the parser goes |
| Puppeteer or Playwright | [Browser](#replace-the-local-browser-with-zenrows-browser) | `launch()` becomes `connect()`. Page code stays |
| Scrapy | [Scrapy integration](/integrations/scrapy) | Requests go through Zenrows via a middleware |
| Selenium | [Selenium integration](/integrations/selenium) | Page loads go through Zenrows |
| A loop over thousands of URLs | [Batch](#scale-out-with-batch) | One job instead of your own queue |

## Replace the HTTP request with Fetch

Before, the scraper sends the request itself, with hand-made headers:

```python Python (before) theme={"dark"}
import requests

def get_page_content(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }
    response = requests.get(url, headers=headers)
    response.raise_for_status()
    return response.text
```

After, Fetch sends it. `mode=auto` renders JavaScript and uses premium proxies only when a page needs them, so you drop headers and proxies. The parsing function doesn't change:

```python Python (after) theme={"dark"}
import requests
from bs4 import BeautifulSoup

ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"

def get_page_content(url):
    params = {
        "apikey": ZENROWS_API_KEY,
        "url": url,
        "mode": "auto",
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
    response.raise_for_status()
    return response.text

# Your parsing logic stays the same
def scrape_price_data(url):
    soup = BeautifulSoup(get_page_content(url), "html.parser")
    name = soup.select_one("h1.product_title")
    price = soup.select_one("p.price")
    return {
        "name": name.get_text(strip=True) if name else None,
        "price": price.get_text(strip=True) if price else None,
    }

print(scrape_price_data("https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"))
```

## Drop the parser with a CSS extractor or Extract

When you're ready, let Zenrows return JSON instead of HTML. Move your selectors into a `css_extractor` and delete the parsing code:

```python Python theme={"dark"}
import json
import requests

ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"

def scrape_price_data(url):
    params = {
        "apikey": ZENROWS_API_KEY,
        "url": url,
        "mode": "auto",
        "css_extractor": json.dumps({"name": "h1.product_title", "price": "p.price"}),
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
    response.raise_for_status()
    return response.json()

print(scrape_price_data("https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"))
```

On retailers Zenrows has prepared, such as amazon.com, walmart.com, ebay.com and bestbuy.com, skip selectors entirely: replace `css_extractor` with `extract=auto` and read the product from the `parsed` field. See [Get structured JSON from any page](/academy/extract/auto-extraction).

## Replace the local browser with Zenrows Browser

Puppeteer and Playwright scrapers keep their page code. Only the line that starts the browser changes: connect to `wss://browser.zenrows.com` instead of launching a local Chromium. Install `puppeteer-core` or `playwright` (no bundled browser needed).

<CodeGroup>
  ```python Python (Playwright) theme={"dark"}
  from playwright.sync_api import sync_playwright

  ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"

  def scrape_with_zenrows_browser(url):
      with sync_playwright() as p:
          # Before: browser = p.chromium.launch()
          browser = p.chromium.connect_over_cdp(f"wss://browser.zenrows.com?apikey={ZENROWS_API_KEY}")
          try:
              page = browser.contexts[0].new_page()
              page.goto(url)
              return page.content()
          finally:
              browser.close()

  html = scrape_with_zenrows_browser("https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/")
  print(len(html), "<title>" in html)
  ```

  ```javascript Node.js (Puppeteer) theme={"dark"}
  import puppeteer from "puppeteer-core";

  const ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY";

  async function scrapeWithZenrowsBrowser(url) {
    // Before: const browser = await puppeteer.launch();
    const browser = await puppeteer.connect({
      browserWSEndpoint: `wss://browser.zenrows.com?apikey=${ZENROWS_API_KEY}`,
    });
    try {
      const page = await browser.newPage();
      await page.goto(url, { waitUntil: "domcontentloaded" });
      return await page.content();
    } finally {
      await browser.close();
    }
  }

  const html = await scrapeWithZenrowsBrowser("https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/");
  console.log(html.length, html.includes("Abominable Hoodie"));
  ```
</CodeGroup>

Close the browser in a `finally` block, as above, so every session ends when your code is done. If your scraper only loads pages and reads HTML, with no clicks or logins, Fetch is the lighter option: it returns the rendered page in one request.

## Scale out with Batch

If your scraper loops over a long URL list with its own queue, retries and concurrency control, hand the list to Batch instead: one job, parallel fetching, automatic tracking of failures, and results you download when the job ends. The same `mode`, `css_extractor` and `extract` parameters go in the job's `zenrows_params`. See [Scrape a list of URLs](/academy/scrape-from-a-list-of-urls).

## Migration checklist

* **Before**: list every place that sends a request or launches a browser, note the custom headers, cookies and proxies each one uses, and test Zenrows on a few real target URLs.
* **During**: replace the request or `launch()` call, keep `mode=auto` as the default, set client timeouts to 180 seconds, and read the API key from an environment variable.
* **After**: remove proxy rotation and User-Agent lists, drop parsing dependencies you replaced with `css_extractor` or Extract, and log the `X-Request-Cost` and `X-Request-Id` response headers.

## Example output

The Fetch migration and the `css_extractor` version return the same data for the test product:

```text theme={"dark"}
{'name': 'Abominable Hoodie', 'price': '$69.00'}
```

The Browser migrations return the full rendered page:

```text theme={"dark"}
122781 True
```

## Customize it

| Change | How |
| - | - |
| Keep your own headers | Add `custom_headers=true` and send the headers with the request, see [Headers](/fetch/features/headers) |
| Content from one country | Add `proxy_country=us` (any ISO code), see [Proxy country](/fetch/features/proxy-country) |
| Keep the same IP across requests | Add `session_id` (any integer), see [Other features](/fetch/features/other) |
| Wait for an element before returning | Use `js_render=true` with `wait_for=<selector>` instead of `mode=auto`, see [Wait for selector](/fetch/features/wait-for) |
| Run more requests at once | Match your workers to the `Concurrency-Limit` response header, see [Concurrency](/fetch/features/concurrency) |
| Get markdown for an LLM | Add `response_type=markdown`, see [Turn any web page into LLM-ready markdown](/academy/fetch/markdown-for-llms) |

## Troubleshooting

* **The selectors return nothing after the migration**: Zenrows returned the rendered page, which can differ from the raw HTML your old scraper saw. Save `response.text` once, check the element's selector in it, and update the selector.
* **Requests fail with `429`**: you're sending more requests at once than your plan's concurrency. Cap your workers at the `Concurrency-Limit` header value.
* **Requests fail with `402`**: the account is out of credits ([AUTH004](/api-error-codes#AUTH004)) or the API key reached one of its [credit caps](/first-steps/api-key-credit-caps) ([AUTH014](/api-error-codes#AUTH014)).
* **Requests end with a client timeout or `499`**: your HTTP client gave up first. Set its timeout to 180 seconds.
* **`page.goto` hangs in Puppeteer**: `networkidle0` and `networkidle2` wait for every background request. Use `waitUntil: "domcontentloaded"` and wait for the element you need.

## Related recipes

<CardGroup cols={2}>
  <Card title="Build a price monitor" icon="chart-line" href="/academy/build-price-monitoring">
    A complete monitor with price history and drop alerts.
  </Card>

  <Card title="Get structured JSON from any page" icon="wand" href="/academy/extract/auto-extraction">
    Replace selectors with Extract on prepared retailers.
  </Card>

  <Card title="Connect Playwright or Puppeteer to Zenrows Browser" icon="plug-connected" href="/academy/browser/connect-playwright-and-puppeteer">
    Every connection option for Browser.
  </Card>

  <Card title="Retry failed requests" icon="refresh" href="/academy/retry-failed-requests">
    Add retries with backoff around your Zenrows calls.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.