> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Capture the network requests behind a page

> Find the hidden API calls a page makes in the background and get their data as clean JSON with Zenrows.

Many pages load their data through background XHR and fetch calls, and those calls often return clean JSON. Capture them and you get structured data without parsing any HTML. Fetch returns every call in one response; Browser lets you listen to them live from Playwright or Puppeteer.

**Uses:** [Fetch](/fetch/api-reference) with [`json_response`](/fetch/features/json-response), [`js_render`](/fetch/features/js-rendering) and [`wait_for`](/fetch/features/wait-for); [Browser](/browser-sessions/introduction) with [Playwright](/browser-sessions/get-started/playwright) or [Puppeteer](/browser-sessions/get-started/puppeteer)

## Ask your AI agent

With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"dark"}
Use the Zenrows browser to open https://www.scrapingcourse.com/javascript-rendering, wait for the
products to load, and list the background data requests the page made. Close the session.
Then use Zenrows to read the request that returns the products and give me the name,
SKU and price of the first three.
```

## Find the request that carries the data

Before you capture anything, find the call in your own browser:

1. Open `https://www.scrapingcourse.com/javascript-rendering` in Chrome.
2. Right-click the page, choose **Inspect**, open the **Network** tab and select the **Fetch/XHR** filter.
3. Reload the page. A `GET` to `https://www.scrapingcourse.com/ajax/products/json` appears.
4. Click it and open **Response**: it is a JSON array with every product's name, SKU, price and images.

The URL fragment you spotted, here `/ajax/products`, is what you filter on in the code below. Calls triggered by an action, such as a "Load more" click, show up when you perform that action with DevTools open.

## Capture requests with Fetch

Add `json_response=true` to a rendered request. The response becomes JSON with the page `html` and an `xhr` list: one entry per background call, with its `url`, `method`, `status_code`, `headers` and `body`. `wait_for` holds the capture until the data has arrived.

<CodeGroup>
  ```python Python (Requests) theme={"dark"}
  import json
  import requests

  params = {
      "apikey": "YOUR_ZENROWS_API_KEY",
      "url": "https://www.scrapingcourse.com/javascript-rendering",
      "js_render": "true",
      "wait_for": ".product-name",
      "json_response": "true",
  }

  response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
  response.raise_for_status()

  data = response.json()
  for call in data["xhr"]:
      print(call["method"], call["status_code"], call["url"])

  # Keep the call that returns the product list and parse its body
  products_call = next(call for call in data["xhr"] if "/ajax/products" in call["url"])
  products = json.loads(products_call["body"])
  print(len(products), "products")
  for product in products[:3]:
      print(product["sku"], product["name"], product["regular_price"])
  ```

  ```javascript Node.js theme={"dark"}
  const params = new URLSearchParams({
    apikey: "YOUR_ZENROWS_API_KEY",
    url: "https://www.scrapingcourse.com/javascript-rendering",
    js_render: "true",
    wait_for: ".product-name",
    json_response: "true",
  });

  const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
    signal: AbortSignal.timeout(180_000),
  });
  if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);

  const data = await response.json();
  for (const call of data.xhr) console.log(call.method, call.status_code, call.url);

  // Keep the call that returns the product list and parse its body
  const productsCall = data.xhr.find((call) => call.url.includes("/ajax/products"));
  const products = JSON.parse(productsCall.body);
  console.log(products.length, "products");
  for (const p of products.slice(0, 3)) console.log(p.sku, p.name, p.regular_price);
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fjavascript-rendering&js_render=true&wait_for=.product-name&json_response=true"
  ```
</CodeGroup>

## Example output

```text theme={"dark"}
GET 200 https://www.scrapingcourse.com/ajax/products/json
POST 204 https://www.scrapingcourse.com/cdn-cgi/rum?
12 products
MH01 Chaz Kangeroo Hoodie 52
MH02 Teton Pullover Hoodie 70
MH03 Bruno Compete Hoodie 63
```

Each `body` is a string: parse it with `json.loads` or `JSON.parse` when the call returns JSON.

## Call the endpoint directly

Once you know the endpoint, you can often request it straight away. It skips rendering, so it is faster and cheaper than capturing it from the page.

```python Python (Requests) theme={"dark"}
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/ajax/products/json",
    "mode": "auto",
}

response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

products = response.json()
print(len(products), "products,", response.headers["X-Request-Credits"], "credit")
```

Output: `12 products, 1 credit`. When the endpoint needs the page's cookies or tokens, keep capturing it from the page instead.

## Capture requests with Browser

For multi-step flows, or when you want each response as it happens, connect Playwright or Puppeteer to Zenrows Browser and register a `response` listener before you navigate. Filter on the resource type and read the body with `.json()`.

<CodeGroup>
  ```python Python (Playwright) theme={"dark"}
  import asyncio

  from playwright.async_api import async_playwright

  CONNECTION_URL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

  async def main():
      captured = []

      async def on_response(response):
          if response.request.resource_type in ("xhr", "fetch"):
              captured.append(response)

      async with async_playwright() as p:
          browser = await p.chromium.connect_over_cdp(CONNECTION_URL)
          try:
              page = await browser.new_page()
              page.on("response", on_response)
              await page.goto("https://www.scrapingcourse.com/javascript-rendering")
              await page.wait_for_selector(".product-name")

              for response in captured:
                  print(response.request.method, response.status, response.url[:80])

              products_response = next(r for r in captured if "/ajax/products" in r.url)
              products = await products_response.json()
              print(len(products), "products")
              for product in products[:3]:
                  print(product["sku"], product["name"], product["regular_price"])
          finally:
              await browser.close()

  asyncio.run(main())
  ```

  ```javascript Node.js (Puppeteer) theme={"dark"}
  import puppeteer from "puppeteer-core";

  const connectionURL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY";

  const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
  try {
    const page = await browser.newPage();

    const captured = [];
    page.on("response", (response) => {
      const type = response.request().resourceType();
      if (type === "xhr" || type === "fetch") captured.push(response);
    });

    await page.goto("https://www.scrapingcourse.com/javascript-rendering");
    await page.waitForSelector(".product-name");

    for (const response of captured) {
      console.log(response.request().method(), response.status(), response.url().slice(0, 80));
    }

    const productsResponse = captured.find((r) => r.url().includes("/ajax/products"));
    const products = await productsResponse.json();
    console.log(products.length, "products");
    for (const p of products.slice(0, 3)) console.log(p.sku, p.name, p.regular_price);
  } finally {
    await browser.close();
  }
  ```
</CodeGroup>

Install the client first: `pip install playwright` or `npm install puppeteer-core`. Output:

```text theme={"dark"}
GET 200 https://www.scrapingcourse.com/ajax/products/json
POST 204 https://www.google-analytics.com/g/collect?v=2&tid=G-NZGD14H87G&gtm=45je6ad1v918
POST 204 https://www.scrapingcourse.com/cdn-cgi/rum?
12 products
MH01 Chaz Kangeroo Hoodie 52
MH02 Teton Pullover Hoodie 70
MH03 Bruno Compete Hoodie 63
```

The `finally` block closes the browser even when a step fails, which ends the session and frees its slot.

## Customize it

| Change | How |
| - | - |
| Capture calls triggered by a click or scroll | Add `js_instructions` to the Fetch request, see [Click, fill, and scroll before you scrape](/academy/fetch/page-interactions) |
| Protected site | Add `premium_proxy=true` to the Fetch request, see [Scrape pages protected by anti-bot systems](/academy/fetch/protected-sites) |
| Browser from a specific country | Add `proxy_country` to the connection URL, see [Country](/browser-sessions/features/country) |
| Send the page's cookies or tokens to the endpoint | Use `custom_headers=true`, see [Keep the same IP and submit forms](/academy/fetch/sessions-and-forms) |
| Wait for the network instead of an element | In Fetch, `js_instructions` with `{"wait_event": "networkidle"}` |

## Troubleshooting

* **The `xhr` list is empty or missing the call**: the capture finished before the call ran. Add `wait_for` with a selector of the content the call fills, or a `wait` of a few seconds.
* **`400` with code `REQS004` mentioning `json_response`**: the request has no rendering. Add `js_render=true`.
* **`json.loads` fails on a `body`**: that call returned HTML, text or an empty body (status `204`). Filter by URL first, and check `status_code` before parsing.
* **The Browser listener sees nothing**: it was registered after navigation. Call `page.on("response", ...)` before `page.goto`.
* **`await response.json()` fails in Browser**: the response was a redirect or had no body. Check `response.status` and read the body of `200` responses.

## Related recipes

<CardGroup cols={2}>
  <Card title="Scrape JavaScript-rendered pages" icon="brand-javascript" href="/academy/fetch/javascript-pages">
    Render the page and wait for the content you need.
  </Card>

  <Card title="Connect Playwright or Puppeteer to Zenrows Browser" icon="plug" href="/academy/browser/connect-playwright-and-puppeteer">
    Drive Zenrows Browser from your existing scripts.
  </Card>

  <Card title="Click, fill, and scroll before you scrape" icon="click" href="/academy/fetch/page-interactions">
    Trigger the calls you want to capture.
  </Card>

  <Card title="Get structured JSON from any page" icon="wand" href="/academy/extract/auto-extraction">
    Get product, listing or article fields as JSON.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.