# API Error Codes
Source: https://docs.zenrows.com/api-error-codes

Complete list of Zenrows API error codes with troubleshooting steps, status code meanings, and solutions for common request failures.

Understanding the Zenrows Fetch error codes is crucial for troubleshooting and optimizing your interactions with Zenrows. Below are common errors you may encounter, along with explanations and recommended actions.

Each error code corresponds to specific conditions encountered during Fetch usage, from authentication problems to request handling and server responses.

## 400 Bad Request<span />

### REQS001 Requests To This Domain Are Forbidden<span />

**Problem:** Requests to this URL are forbidden.

**Solution:**

1. Check our [target sites' access restrictions and user behavior guidelines](/forbidden-sites) to see if the domain is explicitly blocked
2. Try using a different domain that provides similar data
3. If you need to scrape this specific domain for a legitimate business purpose, contact our support team to discuss potential options

### REQS002 Request Requirements Unsatisfied<span />

**Problem:** The requested URL domain needs JavaScript rendering and/or Premium Proxies due to its high-level security defenses.

**Solution:**

1. Read the error message to understand the specific requirements for the domain
2. Add `js_render=true` and/or `premium_proxy=true` parameters to your request, depending on the domain's requirements

### REQS004 Invalid Params Provided<span />

**Problem:** Some parameters or even the URL are invalid or not properly encoded.

**Solution:**

1. Read the error message to understand the specific issue
2. Ensure your URL is properly URL-encoded (check our [guide on encoding URLs](/fetch/faq#how-to-encode-urls)) or the http you use is doing it for you
3. Validate that all parameter values match the expected types (boolean, string, etc.)
4. Remove any invalid or unsupported parameters from your request
5. Check that you are not sending query parameters for the target page as parameters for the API call

### RESP004 CSS Extractor Parameter Is Not Valid<span />

**Problem:** The `css_extractor` parameter sent in your request is not valid.

**Solution:**

1. Check your CSS selector syntax for errors (like missing brackets or quotes)
2. Simplify complex selectors that might be causing issues
3. Test your CSS selector on the actual page using browser developer tools first (`document.querySelector('.your-selector')`)
4. If extracting multiple elements, ensure your JSON structure follows the correct format:

```js theme={"dark"}
css_extractor={
  "title": "h1.product-title",
  "price": "span.product-price"
}
```

### REQS006 Invalid CAPTCHA Solver Key<span />

**Problem:** The configured CAPTCHA solver integration API key is invalid.

**Solution:**

1. Verify your CAPTCHA solver API key in your [account integrations](https://app.zenrows.com/settings/integrations) page
2. Check that your CAPTCHA solver account has sufficient funds
3. Confirm the CAPTCHA solver service is operational through their status page
4. Try re-generating a new API key from your CAPTCHA solver service

### RESP008 Non-Retryable Error not related to Zenrows<span />

**Problem:** The target returned an error tied to the site's own configuration or certificate, not something Zenrows can solve or bypass by changing request parameters. Zenrows does not retry RESP008 automatically, but the underlying condition can be temporary on the target's side.

**Solution:**

1. Check if the target website is accessible from your own browser or host
2. If you're using geolocation, try a different country as the site might only be available in specific regions
3. If the error reports a certificate problem but the certificate checks out fine when you access the site directly, the condition may be intermittent on the target's side. Wait a few minutes and retry once or twice. Don't retry in a loop, because repeated 400 errors can trigger a temporary IP block ([BLK0001](#BLK0001)). If it keeps failing, contact support with the `X-Request-Id` response header
4. For SSL certificate issues, try accessing the HTTP version of the site if available

## 401 Unauthorized<span />

### AUTH001 API Key Missing<span />

**Problem:** No `apikey` information was sent in your request.

**Solution:**

1. Add the `apikey` parameter to your request
2. Check your code to ensure the apikey is being included in every request
3. For API clients like Axios or Requests, set up default parameters to include your API key automatically
4. Verify the API endpoint structure - the key should be sent as a query parameter to the API endpoint

### AUTH002 Invalid API Key<span />

**Problem:** The `apikey` sent does not match the expected format.

**Solution:**

1. Copy your API key directly from the [Zenrows Playground](https://app.zenrows.com/fetch)
2. Ensure no extra spaces or characters were accidentally included
3. Check for any string formatting or concatenation issues in your code
4. If using environment variables, verify they're loading correctly

### AUTH003 API Key Not Found<span />

**Problem:** The `apikey` sent in your request is not valid.

**Solution:**

1. Verify you're using the correct API key from your [Zenrows Playground](https://app.zenrows.com/fetch)
2. Check if your API key has been revoked or regenerated recently
3. Ensure your account is active and working, you can perform a request from the Playground itself
4. For team accounts, confirm with your administrator that the key is still valid

## 402 Payment Required<span />

### AUTH004 Usage Exceeded<span />

**Problem:** This `apikey` has spent its credit allowance for the current billing period.

**This is not a permanent block.** The allowance renews at the end of the period, and requests succeed again from then on with no action on your part. Topping up or upgrading is how you carry on *before* that.

<Note>
  Not to be confused with [AUTH006](#AUTH006), the concurrency limit — too many requests *at once*, rather than too many across the period. AUTH006 clears in seconds; AUTH004 clears at the period boundary.
</Note>

**Solution:**

1. Check where you stand and when the period ends: call `/v1/subscriptions/self/details` with your API key in the `X-API-Key` header, or run `zenrows usage`. Neither counts against your concurrency limit, so both are safe to call from a script right after a failure.
2. If you can wait for the renewal date, retry after it. Retrying sooner returns AUTH004 again unless the allowance itself has changed — a top-up or a plan upgrade raises it immediately, mid-period.
3. To continue now, top up in the [Zenrows Billing page](https://app.zenrows.com/billing) or upgrade in the [Zenrows Plans page](https://app.zenrows.com/plans).
4. For temporary needs or punctual issues, contact our support team.

<Tip>
  **Writing a client or agent?** Read `usage_percent` and `period_ends_at` from `/v1/subscriptions/self/details` before a large run, and back off rather than retry when you see AUTH004. A Fetch request costs between 1 and 25 credits depending on `js_render` and `premium_proxy`, and Browser Sessions and residential bandwidth draw on the same balance at their own rates, so a run can consume far more of the allowance than its request count suggests.
</Tip>

### AUTH005 API Key Is No Longer Valid<span />

**Problem:** This `apikey` has reached its validity period.

**Solution:**

1. Check your subscription in the [Zenrows Billing page](https://app.zenrows.com/billing)
2. Check if your account is active and working, you can perform a request from the [Playground](https://app.zenrows.com/fetch) itself
3. Contact support if you believe there is an issue with your account

### AUTH010 Feature Is Not Included In Plan<span />

**Problem:** The requested feature is not included in your subscription plan.

**Solution:**

1. Review the [features included in each plan](https://www.zenrows.com/pricing)
2. Upgrade to a plan that includes the feature you need
3. Modify your code to avoid using premium features if you don't upgrade
4. Contact our support team if you believe there is an issue with your account

### AUTH011 No Subscription Found<span />

**Problem:** This account does not have an active subscription.

**Solution:**

1. Purchase a subscription plan from the [Zenrows plans page](https://app.zenrows.com/plans)
2. Check for any failed payment attempts in your account history
3. Verify your payment method details are correct
4. Contact our support team if you believe there is an issue with your account

### AUTH012 Subscription Does Not Allow Use Of The Product<span />

**Problem:** This account subscription does not allow the use of this service.

**Solution:**

1. Check which Zenrows products your subscription includes
2. Upgrade to a plan that includes the product you're trying to use
3. Ensure you're using the correct API endpoint for your subscription
4. Contact our support team if you believe there is an issue with your account

### AUTH014 API Key Credit Cap Reached<span />

**Problem:** This `apikey` has reached one of its [credit caps](/first-steps/api-key-credit-caps) (per billing period, daily, weekly or monthly). Your other API keys keep working.

<Note>
  Not to be confused with [AUTH004](#AUTH004), which means your account has no credits left. AUTH014 affects only the key that reached its cap.
</Note>

**Solution:**

1. Raise or remove the cap on the [API keys page](https://app.zenrows.com/settings/api-keys).
2. Or wait for the cap's window to reset. Daily, weekly and monthly caps reset in UTC; the billing-period cap resets with your billing period.

## 403 Forbidden<span />

### AUTH009 User Is Not Verified<span />

**Problem:** This `apikey` belongs to a user that has not verified the email account.

**Solution:**

1. Check your email inbox (including spam folder) for a verification email
2. Request a new verification email
3. Ensure your email address is entered correctly in your profile
4. Contact support if you continue to have verification issues

### BLK0001 IP Address Blocked<span />

**Problem:** Your IP address has been blocked for exceeding the maximum error rate allowed.

**Solution:**

1. Wait a few minutes before retrying requests
2. Implement error handling in your code to prevent excessive failed requests
3. Use exponential backoff when retrying failed requests
4. If using the service in a high-traffic environment, consider implementing a queue system to manage request rates

<Tip>
  Visit our [troubleshooting guide](/fetch/troubleshooting/ip-address-blocked) for step-by-step instructions, common causes, and best practices to quickly restore access.
</Tip>

### REQS007 Extract Is Not Prepared For This Domain<span />

**Problem:** You asked Extract for a domain it has not been prepared for. The message names the domain.

**Solution:**

1. Ask for the domain with [`POST /v1/extract/prepared-domains`](/extract/endpoints#getting-a-domain-prepared), then retry once it reports `ready`
2. Call [`GET /v1/extract/domains`](/extract/endpoints#which-domains-extract-serves) to see what Extract serves right now
3. Use [`css_extractor`](/fetch/features/css-extractor) meanwhile, which works on any website without preparation. The deprecated [`autoparse`](/fetch/features/autoparse) parameter also still works, but new integrations should not start there

<Note>
  Preparation is per domain, not per account: once a domain is prepared it works for every Zenrows API key. Changing or upgrading your plan does not affect this error.
</Note>

## 404 Not Found<span />

### RESP002 Page Not Found<span />

**Problem:** The requested URL page returned a 404 HTTP Status Code.

<Note>
  RESP002 and [RESP007](#RESP007) both return HTTP 404. Check the `code` field in the JSON error body to tell them apart: RESP002 means the page was not found on a domain that resolves, and RESP007 means the domain does not resolve.
</Note>

**Solution:**

1. Verify the URL exists by opening it in a browser
2. Check for typos or encoding issues in the URL
3. If the page was recently available, it might have been moved or deleted
4. For dynamic sites, try adding `js_render=true` as some 404 pages are generated via JavaScript
5. Note that these requests are billed

### RESP007 Site Not Found<span />

**Problem:** The requested target domain could not be resolved, or there is no DNS record associated with it.

<Note>
  This returns an HTTP 404, the same status as [RESP002](#RESP002). Check the `code` field in the JSON error body: `RESP007` means the domain does not resolve (DNS failure), and `RESP002` means the page was not found. A retry will not change the result.
</Note>

**Solution:**

1. Verify the domain exists by checking in your browser
2. Check for typos in the domain name
3. If using an IP address, ensure it's correctly formatted
4. If the domain is new or rarely accessed, DNS propagation might still be in progress
5. Try using premium proxies and geolocation since the domain might be available only in certain countries
6. Note that these requests are billed

## 405 Method Not Allowed<span />

### REQS005 Method Not Allowed<span />

**Problem:** The HTTP verb used to access this page is not allowed.

**Solution:**

1. Change your request method to one of the allowed methods: GET, POST, or PUT
2. For complex requests, consider breaking them down into multiple simpler requests
3. Check if the endpoint you're trying to access has specific method requirements

## 407 Proxy Authentication Required<span />

### AUTH007 Invalid Proxy-Authorization Header<span />

**Problem:** The Proxy-Authorization header sent does not match the expected format.

**Solution:**

1. Ensure the Proxy-Authorization header is a base64 encoded string of `<apikey>:<params>`
2. Check your base64 encoding function for errors or incorrect character handling
3. Try API calls instead of proxy mode to ensure that the API key and account are working properly

## 413 Content Too Large<span />

### RESP005 Response Size Exceeded The Limit<span />

**Problem:** The response data size is bigger than the maximum allowed download size.

**Solution:**

1. Request specific parts of the page using CSS selectors instead of the entire page
2. Split large pages into multiple smaller requests by targeting specific sections
3. Use pagination parameters if available on the target site
4. If using JSON Response, the full response will be considered for the size limit - try removing it and see if it works

<Tip>
  Looking for ways to handle large responses? Check out our [troubleshooting guide](/fetch/troubleshooting/response-too-large) for practical strategies, examples, and tips to work within response size limits.
</Tip>

## 422 Unprocessable Entity<span />

### RESP001 Could Not Get Content<span />

**Problem:** The service couldn't get the content.

**Solution:**

1. Add `js_render=true` as the site might require JavaScript to load content
2. Enable `premium_proxy=true` if the site has anti-bot measures
3. Try adding geolocation to the request set to a country where the site is available
4. Increase wait time with `wait=5000` or higher if content loads slowly
5. Check the error's body for more specific details about the failure
6. Try using custom headers with a referer to mimic a real browser:

```js theme={"dark"}
params={
  // ...
  "custom_headers": true,
}
headers={
  "Referer": "https://www.google.com"
}
```

7. If you already use `js_render=true` and `premium_proxy=true` and still get 422 after adding extras, remove `wait`, `wait_for`, and the custom `referer` header, then retest with only those two. Keep `proxy_country` if the site is region-restricted. Some protected sites respond better to the minimal configuration. See the [troubleshooting guide](/fetch/troubleshooting/troubleshooting-guide#still-failing-after-stacking-parameters-retest-with-the-minimal-config) for an example

## 424 Failed Dependency<span />

### RESP006 Failed To Solve CAPTCHA<span />

**Problem:** The CAPTCHA solver provider was unable to solve the CAPTCHA detected in the page.

**Solution:**

1. Check your CAPTCHA solver service account for sufficient balance
2. Try adding premium proxies to the request
3. Implement retry logic with increasing wait times between attempts

## 429 Too Many Requests<span />

<Note>
  When a burst of requests fails together, group the failures by HTTP status and `code` before troubleshooting. A burst can mix [404](#404) and 429 errors, and [AUTH006](#AUTH006) (concurrency) and [AUTH008](#AUTH008) (rate limit) both return 429. Fix each code with its own solution.
</Note>

### AUTH006 Concurrency Exceeded<span />

**Problem:** The concurrency limit was reached.

**Solution:**

1. Implement a queue system in your code to limit concurrency requests
2. Monitor the `Concurrency-Remaining` header to adjust your request rate dynamically
3. Increase wait times between batches of requests
4. For high-volume scraping needs, upgrade to a plan with higher concurrency limits
5. Learn more about [how Zenrows concurrency works](/fetch/features/concurrency) and implement the provided code examples
6. Note that canceled requests on the client side will not release concurrency until the processing is done in the server side - we recommend not setting a timeout below 3 minutes

### AUTH008 Rate Limit Exceeded<span />

**Problem:** The rate limit was reached.

**Solution:**

1. Implement exponential backoff between requests
2. Distribute requests evenly over time rather than sending them in bursts
3. Set up a queue system with configurable delay between requests
4. For time-sensitive projects, consider upgrading to a plan with higher rate limits
5. Monitor usage patterns to identify and optimize peak request periods

## 500 Internal Server Error<span />

### CTX0001 Context Cancelled<span />

**Problem:** The request was canceled from the client's side.

**Solution:**

1. Check your client's timeout settings - we recommend not setting a timeout below 3 minutes
2. If you're canceling requests manually, review your cancellation logic
3. Ensure your network connection is stable during the entire request

### ERR0001 Unknown Error<span />

**Problem:** An internal error occurred.

**Solution:**

1. Retry the request after a short delay (10-30 seconds)
2. Check the [Zenrows status page](https://status.zenrows.com/) for any ongoing issues
3. Implement error logging to capture the full error response for troubleshooting
4. If the error persists, contact support with details of your request

### ERR0000 Unknown Error<span />

**Problem:** An unexpected internal error occurred.

**Solution:**

1. Retry the request after a short delay (10-30 seconds)
2. Check the [Zenrows status page](https://status.zenrows.com/) for any ongoing issues
3. Implement error logging to capture the full error response for troubleshooting
4. If the error persists, contact support with details of your request

## 502 Bad Gateway<span />

### RESP003 Could Not Parse Content<span />

**Problem:** The request failed because the URL could not be automatically parsed.

**Solution:**

1. Remove the `autoparse` parameter and process the raw HTML response
2. Contact support with details of your request if the issue persists

## 503 Service Unavailable<span />

### RESP009 Service Temporarily At Capacity<span />

**Problem:** All scraping capacity is busy right now. This is a temporary condition on our side.

**Solution:**

1. Retry the request after a short delay; this condition usually clears within seconds as capacity frees up
2. Increase the wait between retries (exponential backoff) and add a small random delay to each one (jitter), so repeated retries do not all arrive at the same instant
3. Treat it like any transient 503: the request is safe to retry
4. If you see it sustained over a longer period, contact support with your account and request details

## 504 Gateway Timeout<span />

### CTX0002 Operation Timeout Exceeded<span />

**Problem:** The request exceeded the maximum allowed time and was aborted.

**Solution:**

1. Check if the target website is slow or unresponsive in a browser
2. Reduce the complexity of your request if using JS Instructions
3. Try adding premium proxies to the request or geolocation to a country where the site is available
4. Break complex scraping tasks into smaller, more focused requests
5. If the issue persists, contact support with details of your request and use case

## Batch and Extract error codes<span />

[Batch](/batch/introduction) and the [Extract prepared-domains endpoints](/extract/endpoints) answer errors with lowercase `code` values, and each error's `type` links to its row below. Extract requests made with `extract=auto` use the uppercase codes above, such as [REQS007](#REQS007). For Batch, see how to handle each one in the [error handling reference](/batch/developer-guide-restapi#error-handling-reference).

| HTTP | `code` | Meaning |
| - | - | - |
| 400 | `invalid_argument`<span /> | Batch: validation failed; `invalid_tasks[]` lists `{index, reason}`. Extract: the request body is missing or malformed; `detail` says what, and `reason` carries a tag such as `invalid_domain`. |
| 401 | `unauthenticated`<span /> | Missing or invalid API key. |
| 402 | `payment_required`<span /> | No credits available. |
| 402 | `api_key_cap_reached`<span /> | The API key reached one of its [credit caps](/first-steps/api-key-credit-caps). The Batch equivalent of [AUTH014](#AUTH014). See [API key credit caps and Batch](/batch/developer-guide-restapi#api-key-credit-caps-and-batch). |
| 404 | `not_found`<span /> | Batch: job, run, or task missing or not owned by you. Extract: this account has never submitted that domain. |
| 409 | `conflict`<span /> / `run_not_terminal`<span /> / `idempotency_key_conflict`<span /> | State conflict; see `detail`. |
| 409 | `partial_rerun_window_expired`<span /> | Batch: a partial rerun was requested more than 10 days after the original run. Start a full rerun instead. |
| 409 | `in_progress`<span /> | Extract: that domain already has a preparation running. Poll its status instead of resubmitting. |
| 409 | `idempotency_conflict`<span /> | Extract: the same `Idempotency-Key` was reused with a different body. |
| 422 | (on `/content`) | Task failed; the body is the stored error. |
| 429 | `quota_exceeded`<span /> | Account limit reached, such as the maximum of 3 concurrent active jobs. |
| 429 | `out_of_allowance`<span /> | Extract: no domain preparation slots left this month. Check `GET /v1/extract/prepared-domains/allowance`. |
| 503 | `internal`<span /> | Transient; safe to retry with a backoff. |


# API Versioning and Deprecation
Source: https://docs.zenrows.com/api-versioning

How Zenrows versions its API, how a version is retired, and the twelve-month notice you get before anything you depend on stops working.

Zenrows versions its API in the URL path. Every endpoint you call carries the version you are pinned to, so an integration written today keeps behaving the way it did on the day you wrote it.

| Surface | Base URL | Version |
| - | - | - |
| Fetch and Extract | `https://api.zenrows.com/v1/` | `v1` |
| Batch | `https://async.api.zenrows.com/v1` | `v1` |

`v1` is the only version that has ever existed, and no version has ever been retired.

## What Changes Inside a Version

A version is a compatibility promise, not a freeze. Inside `v1` we add, and we do not take away.

**You can expect, without notice:**

* New endpoints.
* New optional request parameters, with the previous behavior as their default.
* New fields in a response body.
* New response headers.

Write your integration so an unrecognized field or header is ignored rather than treated as an error. That is the one thing we ask of you in return, and it is what lets us ship improvements without moving anyone to a new version.

**You will never see inside a version:**

* A parameter removed, or its default changed.
* A field removed from a response, or its type changed.
* An endpoint removed, or its path changed.
* An error code reused for a different condition.

<Note>
  Error codes are part of the contract. `AUTH008` will always mean a rate limit was reached. See [API Error Codes](/api-error-codes) for the catalog.
</Note>

## How a Version Is Retired

If we ever need to retire a version, this is what happens, in this order.

<Steps>
  <Step title="Announcement">
    We publish the deprecation here and on the <a href="https://eu.intercom.news/zenrows">changelog</a>, and email every account with traffic on the affected version.
  </Step>

  <Step title="Twelve months of notice">
    The version keeps working, unchanged, for at least **twelve months** from the announcement. Not twelve months from when you read about it: twelve months from the day it is announced.
  </Step>

  <Step title="Machine-readable warning">
    Every response from the deprecated version carries the standard HTTP fields, so your code can detect the deadline without anyone reading an email:

    ```http theme={"dark"}
    Deprecation: @1789603200
    Sunset: Fri, 17 Sep 2027 00:00:00 GMT
    Link: <https://docs.zenrows.com/api-versioning>; rel="deprecation"
    ```

    `Deprecation` (<a href="https://www.rfc-editor.org/rfc/rfc9745.html">RFC 9745</a>) is when the version became deprecated, given as a Unix timestamp in seconds (the `@` prefix). `Sunset` (<a href="https://www.rfc-editor.org/rfc/rfc8594.html">RFC 8594</a>) is when it stops answering. Both are present from the announcement onward, and both point to the same instant twelve months apart.
  </Step>

  <Step title="Migration guide">
    We publish a guide covering every breaking difference with the announcement, not at the end of the window.
  </Step>
</Steps>

## What This Means if You Are an Agent

Read `Sunset` on any response. If it is present, the version you are calling has a deadline, and the `Link` header points at this page. Nothing else about the response changes: a deprecated version answers exactly as it always did, right up to its sunset date.

If you build integrations that run unattended, treat the presence of a `Sunset` header as a condition worth surfacing to a human, the same way you would surface a payment failure.

## Machine-Readable Description

The full API surface is described in OpenAPI 3.1 at [www.zenrows.com/openapi.json](https://www.zenrows.com/openapi.json), and listed in the RFC 9727 catalog at [app.zenrows.com/.well-known/api-catalog](https://app.zenrows.com/.well-known/api-catalog).


# Developer Guide: REST API
Source: https://docs.zenrows.com/batch/developer-guide-restapi

Drive the Zenrows Batch directly over HTTP. Each request is shown in Python, Node.js, Java, PHP, Go, Ruby, and cURL, with the full original logic preserved.

<Note>
  Batch is available to every Zenrows account. You need an active Zenrows API key. Replace `YOUR_ZENROWS_API_KEY` in every example with your key.
</Note>

## Basics

* **Base URL:** `https://async.api.zenrows.com/v1`
* **Authentication:** send `X-API-Key: YOUR_ZENROWS_API_KEY` on every request. The key travels in a header, never in the URL.
* **Bodies:** send JSON with `Content-Type: application/json` and receive JSON. Errors come back as `application/problem+json`, a standard JSON error format (RFC 7807) with a stable `code` you can branch on.

The Node.js examples below reuse this small wrapper, which adds authentication, sets the `Content-Type` header when there is a body, and turns error responses into thrown objects:

```javascript Node.js theme={"dark"}
const BASE = process.env.BATCH_API_URL ?? "https://async.api.zenrows.com/v1";
const API_KEY = process.env.ZENROWS_API_KEY;

async function api(method, path, { body, query, headers } = {}) {
  const url = new URL(BASE + path);
  for (const [k, v] of Object.entries(query ?? {})) {
    if (v !== undefined && v !== null) url.searchParams.set(k, v);
  }
  const res = await fetch(url, {
    method,
    headers: {
      "X-API-Key": API_KEY,
      ...(body ? { "Content-Type": "application/json" } : {}),
      ...headers,
    },
    body: body ? JSON.stringify(body) : undefined,
  });

  if (res.status === 204) return null;
  const isJson = (res.headers.get("content-type") ?? "").includes("json");
  const payload = isJson ? await res.json() : await res.arrayBuffer();
  if (!res.ok) {
    // RFC 7807 problem: { type, title, status, detail, code, invalid_tasks? }
    const err = new Error(payload?.detail || payload?.title || `HTTP ${res.status}`);
    err.status = res.status;
    err.code = payload?.code;
    err.problem = payload;
    throw err;
  }
  return payload;
}
```

## 1. Create your jobs

There are several ways to create a job. Pick the one that matches how your URLs arrive.

### Submit a job with all URLs up front

Create a **closed job** (one where you know all URLs up front) with `POST /jobs`. The response includes a `job_id` and the initial run status.

<Note>
  `POST /jobs` returns **`201 Created`** for submissions under 10,000 inline tasks and **`202 Accepted`** for 10,000 or more tasks (or a CSV input). The request shape and response body are identical, so treat both the same: store `job_id`, then poll until `stats.completed >= stats.total` (or use a webhook). On a `202`, `stats.total` is correct immediately, but `GET /jobs/{id}/results` may return partial pages for a few seconds while the task rows finish writing.
</Note>

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json'}
  payload = {
      'type': 'regular',
      'status': 'closed',
      'zenrows_params': {'js_render': 'true', 'premium_proxy': 'true'},
      'tasks': [
          {'url': 'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie', 'external_id': 'order-1'},
          {'url': 'https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket', 'external_id': 'order-2'},
      ],
  }
  response = requests.post('https://async.api.zenrows.com/v1/jobs', headers=headers, data=json.dumps(payload))
  print(response.json())
  ```

  ```javascript Node.js theme={"dark"}
  // Reuses the api() helper from the Basics section.
  const submitted = await api("POST", "/jobs", {
    body: {
      type: "regular",
      status: "closed",
      zenrows_params: { js_render: "true", premium_proxy: "true" },
      tasks: [
        { url: "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie", external_id: "order-1" },
        { url: "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket", external_id: "order-2" },
      ],
    },
  });
  console.log("submitted", submitted.job_id, submitted.latest_run.status);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;

  public class SubmitJob {
      public static void main(final String... args) throws Exception {
          String body = "{"
              + "\"type\":\"regular\",\"status\":\"closed\","
              + "\"zenrows_params\":{\"js_render\":\"true\",\"premium_proxy\":\"true\"},"
              + "\"tasks\":["
              + "{\"url\":\"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie\",\"external_id\":\"order-1\"},"
              + "{\"url\":\"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket\",\"external_id\":\"order-2\"}"
              + "]}";
          String response = Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .bodyString(body, ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $payload = json_encode([
      'type' => 'regular',
      'status' => 'closed',
      'zenrows_params' => ['js_render' => 'true', 'premium_proxy' => 'true'],
      'tasks' => [
          ['url' => 'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie', 'external_id' => 'order-1'],
          ['url' => 'https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket', 'external_id' => 'order-2'],
      ],
  ]);
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1/jobs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY', 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "io"
      "log"
      "net/http"
  )

  func main() {
      payload := []byte(`{
          "type": "regular",
          "status": "closed",
          "zenrows_params": { "js_render": "true", "premium_proxy": "true" },
          "tasks": [
              { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie", "external_id": "order-1" },
              { "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket", "external_id": "order-2" }
          ]
      }`)
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs", bytes.NewBuffer(payload))
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      req.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  payload = {
      type: 'regular',
      status: 'closed',
      zenrows_params: { js_render: 'true', premium_proxy: 'true' },
      tasks: [
          { url: 'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie', external_id: 'order-1' },
          { url: 'https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket', external_id: 'order-2' }
      ]
  }
  conn = Faraday.new
  res = conn.post('https://async.api.zenrows.com/v1/jobs') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.headers['Content-Type'] = 'application/json'
      req.body = payload.to_json
  end
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "type": "regular",
      "status": "closed",
      "zenrows_params": { "js_render": "true", "premium_proxy": "true" },
      "tasks": [
        { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie", "external_id": "order-1" },
        { "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket", "external_id": "order-2" }
      ]
    }'
  ```
</CodeGroup>

### Configure the scrape and per-URL options

`zenrows_params` applies to every URL in the job. A per-task `zenrows_params` overrides the job-level keys for that URL, so you can set the common case once and tune exceptions such as a different `proxy_country`.

Common keys:

* [`js_render`](/fetch/features/js-rendering)
* [`premium_proxy`](/fetch/features/premium-proxy)
* [`proxy_country`](/fetch/features/proxy-country)
* [`js_instructions`](/fetch/features/js-instructions)
* [`wait_for`](/fetch/features/wait-for)
* [`wait`](/fetch/features/wait)
* [`block_resources`](/fetch/features/block-resources)
* [`custom_headers`](/fetch/features/headers)
* [`json_response`](/fetch/features/json-response)
* [`css_extractor`](/fetch/features/css-extractor)
* [`extract`](/extract/setup)
* [`autoparse`](/fetch/features/autoparse) (deprecated, use `extract`)
* `response_type` ([`markdown`](/fetch/features/markdown), [`plaintext`](/fetch/features/plaintext), or [`pdf`](/fetch/features/pdf))
* [`outputs`](/fetch/features/output-filters)
* [`original_status`](/fetch/features/other#original-http-code)
* [`allowed_status_codes`](/fetch/features/other#return-content-on-error)
* [`session_id`](/fetch/features/other#session-id)

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json'}
  payload = {
      'type': 'regular',
      'status': 'closed',
      'zenrows_params': {'js_render': 'true', 'proxy_country': 'us', 'response_type': 'markdown'},
      'tasks': [
          {'url': 'https://www.scrapingcourse.com/ecommerce/product/artemis-running-short',
          'external_id': 'order-42', 'zenrows_params': {'proxy_country': 'de'}},
          {'url': 'https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee',
          'metadata': {'sku': 'ABC-1'}},
      ],
  }
  response = requests.post('https://async.api.zenrows.com/v1/jobs', headers=headers, data=json.dumps(payload))
  print(response.json())
  ```

  ```javascript Node.js theme={"dark"}
  const job = await api("POST", "/jobs", {
    body: {
      type: "regular",
      status: "closed",
      zenrows_params: { js_render: "true", proxy_country: "us", response_type: "markdown" },
      tasks: [
        { url: "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short",
          external_id: "order-42", zenrows_params: { proxy_country: "de" } },
        { url: "https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee",
          metadata: { sku: "ABC-1" } },
      ],
    },
  });
  console.log(job.job_id);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;

  public class ConfigureJob {
      public static void main(final String... args) throws Exception {
          String body = "{"
              + "\"type\":\"regular\",\"status\":\"closed\","
              + "\"zenrows_params\":{\"js_render\":\"true\",\"proxy_country\":\"us\",\"response_type\":\"markdown\"},"
              + "\"tasks\":["
              + "{\"url\":\"https://www.scrapingcourse.com/ecommerce/product/artemis-running-short\",\"external_id\":\"order-42\",\"zenrows_params\":{\"proxy_country\":\"de\"}},"
              + "{\"url\":\"https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee\",\"metadata\":{\"sku\":\"ABC-1\"}}"
              + "]}";
          String response = Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .bodyString(body, ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $payload = json_encode([
      'type' => 'regular',
      'status' => 'closed',
      'zenrows_params' => ['js_render' => 'true', 'proxy_country' => 'us', 'response_type' => 'markdown'],
      'tasks' => [
          ['url' => 'https://www.scrapingcourse.com/ecommerce/product/artemis-running-short',
          'external_id' => 'order-42', 'zenrows_params' => ['proxy_country' => 'de']],
          ['url' => 'https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee',
          'metadata' => ['sku' => 'ABC-1']],
      ],
  ]);
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1/jobs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY', 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "io"
      "log"
      "net/http"
  )

  func main() {
      payload := []byte(`{
          "type": "regular",
          "status": "closed",
          "zenrows_params": { "js_render": "true", "proxy_country": "us", "response_type": "markdown" },
          "tasks": [
              { "url": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short", "external_id": "order-42", "zenrows_params": { "proxy_country": "de" } },
              { "url": "https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee", "metadata": { "sku": "ABC-1" } }
          ]
      }`)
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs", bytes.NewBuffer(payload))
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      req.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  payload = {
      type: 'regular',
      status: 'closed',
      zenrows_params: { js_render: 'true', proxy_country: 'us', response_type: 'markdown' },
      tasks: [
          { url: 'https://www.scrapingcourse.com/ecommerce/product/artemis-running-short',
            external_id: 'order-42', zenrows_params: { proxy_country: 'de' } },
          { url: 'https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee',
            metadata: { sku: 'ABC-1' } }
      ]
  }
  conn = Faraday.new
  res = conn.post('https://async.api.zenrows.com/v1/jobs') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.headers['Content-Type'] = 'application/json'
      req.body = payload.to_json
  end
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "type": "regular",
      "status": "closed",
      "zenrows_params": { "js_render": "true", "proxy_country": "us", "response_type": "markdown" },
      "tasks": [
        { "url": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short", "external_id": "order-42", "zenrows_params": { "proxy_country": "de" } },
        { "url": "https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee", "metadata": { "sku": "ABC-1" } }
      ]
    }'
  ```
</CodeGroup>

<Tip>
  The API returns your `external_id` unchanged on every result, so you can match results back to your own records without tracking the API's task IDs. It does not need to be unique.
</Tip>

### POST target fetches

By default, Batch fetches every target URL with an HTTP GET. To fetch a target with POST instead, set `method` and `body` on the task object. The body is stored verbatim and sent as-is with each request; maximum size is **16 KiB** per task.

```json theme={"dark"}
{
  "type": "regular",
  "status": "closed",
  "tasks": [
    {
      "url": "https://api.example.com/search",
      "method": "POST",
      "body": "{\"query\": \"web scraping\", \"page\": 1}",
      "zenrows_params": { "custom_headers": { "Content-Type": "application/json" } }
    },
    {
      "url": "https://www.example.com/page",
      "external_id": "get-target"
    }
  ]
}
```

<Note>
  You can mix GET and POST targets in the same job. Tasks without a `method` field default to GET. Set `custom_headers` in `zenrows_params` to control the `Content-Type` sent with the POST body.
</Note>

### Open jobs: adding tasks in batches

<Note>
  Open (queue-mode) jobs let you add tasks in batches after creating the job, instead of supplying all URLs upfront.
</Note>

Create an open job, add tasks with `POST /jobs/{job_id}/tasks` over time, then close it by sending `last_batch: true` on the final batch. Each add-tasks call accepts up to 10,000 URLs.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json'}

  job = requests.post('https://async.api.zenrows.com/v1/jobs', headers=headers,
                      data=json.dumps({'type': 'regular', 'status': 'open',
                                      'zenrows_params': {'js_render': 'true'}})).json()
  job_id = job['job_id']

  requests.post(f'https://async.api.zenrows.com/v1/jobs/{job_id}/tasks', headers=headers,
                data=json.dumps({'tasks': [{'url': 'https://www.scrapingcourse.com/ecommerce/page/1'}]}))
  requests.post(f'https://async.api.zenrows.com/v1/jobs/{job_id}/tasks', headers=headers,
                data=json.dumps({'tasks': [{'url': 'https://www.scrapingcourse.com/ecommerce/page/2'}], 'last_batch': True}))
  ```

  ```javascript Node.js theme={"dark"}
  const { job_id } = await api("POST", "/jobs", {
    body: { type: "regular", status: "open", zenrows_params: { js_render: "true" } },
  });

  await api("POST", `/jobs/${job_id}/tasks`, {
    body: { tasks: [{ url: "https://www.scrapingcourse.com/ecommerce/page/1" }] },
  });
  await api("POST", `/jobs/${job_id}/tasks`, {
    body: { tasks: [{ url: "https://www.scrapingcourse.com/ecommerce/page/2" }], last_batch: true }, // closes the job
  });
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;

  public class OpenJob {
      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";
          String created = Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", key)
                  .bodyString("{\"type\":\"regular\",\"status\":\"open\",\"zenrows_params\":{\"js_render\":\"true\"}}", ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(created); // read job_id from this response

          String jobId = "JOB_ID";
          Request.post("https://async.api.zenrows.com/v1/jobs/" + jobId + "/tasks")
                  .addHeader("X-API-Key", key)
                  .bodyString("{\"tasks\":[{\"url\":\"https://www.scrapingcourse.com/ecommerce/page/1\"}]}", ContentType.APPLICATION_JSON)
                  .execute().discardContent();
          Request.post("https://async.api.zenrows.com/v1/jobs/" + jobId + "/tasks")
                  .addHeader("X-API-Key", key)
                  .bodyString("{\"tasks\":[{\"url\":\"https://www.scrapingcourse.com/ecommerce/page/2\"}],\"last_batch\":true}", ContentType.APPLICATION_JSON)
                  .execute().discardContent();
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  function post($path, $body) {
      $ch = curl_init();
      curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1' . $path);
      curl_setopt($ch, CURLOPT_POST, 1);
      curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($body));
      curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY', 'Content-Type: application/json']);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $res = curl_exec($ch);
      curl_close($ch);
      return json_decode($res, true);
  }

  $job = post('/jobs', ['type' => 'regular', 'status' => 'open', 'zenrows_params' => ['js_render' => 'true']]);
  $jobId = $job['job_id'];
  post("/jobs/$jobId/tasks", ['tasks' => [['url' => 'https://www.scrapingcourse.com/ecommerce/page/1']]]);
  post("/jobs/$jobId/tasks", ['tasks' => [['url' => 'https://www.scrapingcourse.com/ecommerce/page/2']], 'last_batch' => true]);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "log"
      "net/http"
  )

  func post(path string, body string) {
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1"+path, bytes.NewBufferString(body))
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      req.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
  }

  func main() {
      // Create the open job, then read job_id from the response body.
      post("/jobs", `{"type":"regular","status":"open","zenrows_params":{"js_render":"true"}}`)
      jobId := "JOB_ID"
      post("/jobs/"+jobId+"/tasks", `{"tasks":[{"url":"https://www.scrapingcourse.com/ecommerce/page/1"}]}`)
      post("/jobs/"+jobId+"/tasks", `{"tasks":[{"url":"https://www.scrapingcourse.com/ecommerce/page/2"}],"last_batch":true}`)
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  conn = Faraday.new
  conn.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
  conn.headers['Content-Type'] = 'application/json'

  job = JSON.parse(conn.post('https://async.api.zenrows.com/v1/jobs',
      { type: 'regular', status: 'open', zenrows_params: { js_render: 'true' } }.to_json).body)
  job_id = job['job_id']

  conn.post("https://async.api.zenrows.com/v1/jobs/#{job_id}/tasks",
      { tasks: [{ url: 'https://www.scrapingcourse.com/ecommerce/page/1' }] }.to_json)
  conn.post("https://async.api.zenrows.com/v1/jobs/#{job_id}/tasks",
      { tasks: [{ url: 'https://www.scrapingcourse.com/ecommerce/page/2' }], last_batch: true }.to_json)
  ```

  ```bash cURL theme={"dark"}
  # 1) Create the open job and note its job_id.
  curl -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "type": "regular", "status": "open", "zenrows_params": { "js_render": "true" } }'

  # 2) Add a batch of tasks.
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/page/1" }] }'

  # 3) Add the final batch and close the job.
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/page/2" }], "last_batch": true }'
  ```
</CodeGroup>

`last_batch: true` (or `POST /jobs/{id}/close`) locks the job, and later adds return `409`. To abandon a run that is still in flight, call `POST /jobs/{id}/stop` (see [Control the current run](#control-the-current-run)). A very large submission also keeps ingesting task rows for a few seconds after it responds; `POST /jobs/{id}/tasks` can return `409` during that window, so retry shortly.

### Large inputs: CSV upload

For lists too large to inline, upload a CSV in three steps:

1. Create an input slot with `POST /job_inputs`.
2. PUT the file to the returned presigned URL using the exact headers it specifies
3. Submit a job referencing `file_input_id`.

The `fields` map points the canonical `url` (required) and `external_id` (optional) fields at your CSV columns by header name (with `header: true`) or 0-based index.

<Note>Limits: 50 MB / 100,000 rows. Upload slots live 24 hours.</Note>

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}

  # 1) Create the slot.
  slot = requests.post('https://async.api.zenrows.com/v1/job_inputs',
                      headers={**headers, 'Content-Type': 'application/json'},
                      data=json.dumps({'type': 'csv',
                                        'csv': {'header': True, 'fields': {'url': 'URL', 'external_id': 'Customer Ref'}}})).json()

  # 2) Upload the file with the exact content-type the slot requires.
  with open('urls.csv', 'rb') as f:
      requests.put(slot['upload']['url'], headers=slot['upload']['headers'], data=f)

  # 3) Submit referencing the slot.
  job = requests.post('https://async.api.zenrows.com/v1/jobs',
                      headers={**headers, 'Content-Type': 'application/json'},
                      data=json.dumps({'type': 'regular', 'status': 'closed',
                                      'file_input_id': slot['file_input_id'], 'zenrows_params': {'js_render': 'true'}})).json()
  print(job['job_id'])
  ```

  ```javascript Node.js theme={"dark"}
  import { readFile } from "node:fs/promises";

  // 1) Create the slot.
  const slot = await api("POST", "/job_inputs", {
    body: { type: "csv", csv: { header: true, fields: { url: "URL", external_id: "Customer Ref" } } },
  });

  // 2) Upload the body with the EXACT content-type the slot requires.
  const csvBytes = await readFile("urls.csv");
  const put = await fetch(slot.upload.url, {
    method: slot.upload.method, // "PUT"
    headers: slot.upload.headers, // e.g. { "Content-Type": "text/csv" }
    body: csvBytes,
  });
  if (!put.ok) throw new Error(`upload failed: ${put.status}`);

  // 3) Submit referencing the slot instead of inline tasks.
  const job = await api("POST", "/jobs", {
    body: { type: "regular", status: "closed", file_input_id: slot.file_input_id, zenrows_params: { js_render: "true" } },
  });
  console.log(job.job_id);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;
  import java.io.File;

  public class CsvUpload {
      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";

          // 1) Create the slot, then read upload.url, upload.headers, and file_input_id from the response.
          String slot = Request.post("https://async.api.zenrows.com/v1/job_inputs")
                  .addHeader("X-API-Key", key)
                  .bodyString("{\"type\":\"csv\",\"csv\":{\"header\":true,\"fields\":{\"url\":\"URL\",\"external_id\":\"Customer Ref\"}}}", ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(slot);

          // 2) PUT the file to the presigned URL with the content-type the slot requires.
          Request.put("PRESIGNED_UPLOAD_URL")
                  .bodyFile(new File("urls.csv"), ContentType.create("text/csv"))
                  .execute().discardContent();

          // 3) Submit referencing the slot.
          String job = Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", key)
                  .bodyString("{\"type\":\"regular\",\"status\":\"closed\",\"file_input_id\":\"FILE_INPUT_ID\",\"zenrows_params\":{\"js_render\":\"true\"}}", ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(job);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $key = 'YOUR_ZENROWS_API_KEY';

  // 1) Create the slot.
  $ch = curl_init('https://async.api.zenrows.com/v1/job_inputs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
      'type' => 'csv',
      'csv' => ['header' => true, 'fields' => ['url' => 'URL', 'external_id' => 'Customer Ref']],
  ]));
  curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key", 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $slot = json_decode(curl_exec($ch), true);
  curl_close($ch);

  // 2) Upload the file with the exact content-type the slot requires.
  $ch = curl_init($slot['upload']['url']);
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
  curl_setopt($ch, CURLOPT_POSTFIELDS, file_get_contents('urls.csv'));
  curl_setopt($ch, CURLOPT_HTTPHEADER, $slot['upload']['headers']);
  curl_exec($ch);
  curl_close($ch);

  // 3) Submit referencing the slot.
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
      'type' => 'regular', 'status' => 'closed',
      'file_input_id' => $slot['file_input_id'], 'zenrows_params' => ['js_render' => 'true'],
  ]));
  curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key", 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "io"
      "log"
      "net/http"
      "os"
  )

  func main() {
      key := "YOUR_ZENROWS_API_KEY"

      // 1) Create the slot, then read upload.url, upload.headers, and file_input_id from the response.
      slotReq, _ := http.NewRequest("POST", "https://async.api.zenrows.com/v1/job_inputs",
          bytes.NewBufferString(`{"type":"csv","csv":{"header":true,"fields":{"url":"URL","external_id":"Customer Ref"}}}`))
      slotReq.Header.Set("X-API-Key", key)
      slotReq.Header.Set("Content-Type", "application/json")
      slotResp, err := http.DefaultClient.Do(slotReq)
      if err != nil {
          log.Fatalln(err)
      }
      defer slotResp.Body.Close()
      slotBody, _ := io.ReadAll(slotResp.Body)
      log.Println(string(slotBody))

      // 2) PUT the file to the presigned URL with the content-type the slot requires.
      file, _ := os.Open("urls.csv")
      defer file.Close()
      putReq, _ := http.NewRequest("PUT", "PRESIGNED_UPLOAD_URL", file)
      putReq.Header.Set("Content-Type", "text/csv")
      if _, err := http.DefaultClient.Do(putReq); err != nil {
          log.Fatalln(err)
      }

      // 3) Submit referencing the slot.
      jobReq, _ := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs",
          bytes.NewBufferString(`{"type":"regular","status":"closed","file_input_id":"FILE_INPUT_ID","zenrows_params":{"js_render":"true"}}`))
      jobReq.Header.Set("X-API-Key", key)
      jobReq.Header.Set("Content-Type", "application/json")
      jobResp, err := http.DefaultClient.Do(jobReq)
      if err != nil {
          log.Fatalln(err)
      }
      defer jobResp.Body.Close()
      jobBody, _ := io.ReadAll(jobResp.Body)
      log.Println(string(jobBody))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  key = 'YOUR_ZENROWS_API_KEY'
  conn = Faraday.new

  # 1) Create the slot.
  slot = JSON.parse(conn.post('https://async.api.zenrows.com/v1/job_inputs',
      { type: 'csv', csv: { header: true, fields: { url: 'URL', external_id: 'Customer Ref' } } }.to_json,
      { 'X-API-Key' => key, 'Content-Type' => 'application/json' }).body)

  # 2) Upload the file with the exact headers the slot requires.
  conn.put(slot['upload']['url'], File.read('urls.csv'), slot['upload']['headers'])

  # 3) Submit referencing the slot.
  job = JSON.parse(conn.post('https://async.api.zenrows.com/v1/jobs',
      { type: 'regular', status: 'closed', file_input_id: slot['file_input_id'], zenrows_params: { js_render: 'true' } }.to_json,
      { 'X-API-Key' => key, 'Content-Type' => 'application/json' }).body)
  puts job['job_id']
  ```

  ```bash cURL theme={"dark"}
  # 1) Create the slot. Read upload.url, upload.headers, and file_input_id from the response.
  curl -X POST "https://async.api.zenrows.com/v1/job_inputs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "type": "csv", "csv": { "header": true, "fields": { "url": "URL", "external_id": "Customer Ref" } } }'

  # 2) Upload the file to the presigned URL with the content-type it requires.
  curl -X PUT "PRESIGNED_UPLOAD_URL" -H "Content-Type: text/csv" --data-binary @urls.csv

  # 3) Submit referencing the slot.
  curl -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "type": "regular", "status": "closed", "file_input_id": "FILE_INPUT_ID", "zenrows_params": { "js_render": "true" } }'
  ```
</CodeGroup>

### Scheduled jobs

A scheduled job stores its URLs as a template, and each fire creates a new run. Provide exactly one of `at`, `rate`, or `calendar` (plus a `timezone` for `at` and `calendar`). Times of day must be full hours, and the minimum interval between runs is 60 minutes. The request below schedules a job to run every 6 hours.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json'}
  payload = {
      'type': 'scheduled', 'status': 'closed',
      'schedule': {'rate': {'every': 6, 'unit': 'hour'}},
      'tasks': [{'url': 'https://www.scrapingcourse.com/ecommerce/'}],
  }
  response = requests.post('https://async.api.zenrows.com/v1/jobs', headers=headers, data=json.dumps(payload))
  print(response.json())
  ```

  ```javascript Node.js theme={"dark"}
  const job = await api("POST", "/jobs", {
    body: {
      type: "scheduled", status: "closed",
      schedule: { rate: { every: 6, unit: "hour" } },
      tasks: [{ url: "https://www.scrapingcourse.com/ecommerce/" }],
    },
  });
  console.log(job.job_id);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;

  public class ScheduledJob {
      public static void main(final String... args) throws Exception {
          String body = "{\"type\":\"scheduled\",\"status\":\"closed\","
              + "\"schedule\":{\"rate\":{\"every\":6,\"unit\":\"hour\"}},"
              + "\"tasks\":[{\"url\":\"https://www.scrapingcourse.com/ecommerce/\"}]}";
          String response = Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .bodyString(body, ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $payload = json_encode([
      'type' => 'scheduled', 'status' => 'closed',
      'schedule' => ['rate' => ['every' => 6, 'unit' => 'hour']],
      'tasks' => [['url' => 'https://www.scrapingcourse.com/ecommerce/']],
  ]);
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY', 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "io"
      "log"
      "net/http"
  )

  func main() {
      payload := []byte(`{
          "type": "scheduled", "status": "closed",
          "schedule": { "rate": { "every": 6, "unit": "hour" } },
          "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/" }]
      }`)
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs", bytes.NewBuffer(payload))
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      req.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  payload = {
      type: 'scheduled', status: 'closed',
      schedule: { rate: { every: 6, unit: 'hour' } },
      tasks: [{ url: 'https://www.scrapingcourse.com/ecommerce/' }]
  }
  conn = Faraday.new
  res = conn.post('https://async.api.zenrows.com/v1/jobs') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.headers['Content-Type'] = 'application/json'
      req.body = payload.to_json
  end
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{
      "type": "scheduled", "status": "closed",
      "schedule": { "rate": { "every": 6, "unit": "hour" } },
      "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/" }]
    }'
  ```
</CodeGroup>

For a calendar schedule, replace the `schedule` object. The example below runs at 09:00 and 18:00 on Mon/Wed/Fri in Berlin time. Use an `at` object instead for a one-shot run at a single wall-clock time.

```json theme={"dark"}
{
  "schedule": {
    "calendar": {
      "times_of_day": ["09:00", "18:00"],
      "cadence": { "weekly": { "days": ["mon", "wed", "fri"] } }
    },
    "timezone": "Europe/Berlin"
  }
}
```

Manage a schedule with `PUT /jobs/{id}/schedule` to replace it, and `POST /jobs/{id}/schedule/state` with `{"schedule_state": "paused"}` or `{"schedule_state": "active"}` to pause or resume.

## 2. Track your jobs

Check progress and find your jobs once they are running.

### Check job status

Read a job with `GET /jobs/{job_id}`. Progress lives in `latest_run.stats`, where `completed` equals `successful` plus `failed`.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  response = requests.get('https://async.api.zenrows.com/v1/jobs/JOB_ID',
                          headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'})
  job = response.json()
  print(job['latest_run']['status'], job['latest_run']['stats'])
  ```

  ```javascript Node.js theme={"dark"}
  const job = await api("GET", "/jobs/JOB_ID");
  console.log(job.latest_run.status, job.latest_run.stats);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class GetJob {
      public static void main(final String... args) throws Exception {
          String response = Request.get("https://async.api.zenrows.com/v1/jobs/JOB_ID")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1/jobs/JOB_ID');
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      req, err := http.NewRequest("GET", "https://async.api.zenrows.com/v1/jobs/JOB_ID", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  conn = Faraday.new
  res = conn.get('https://async.api.zenrows.com/v1/jobs/JOB_ID') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
  end
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://async.api.zenrows.com/v1/jobs/JOB_ID" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

To wait for a run to finish, poll this endpoint until `latest_run.status` reaches a **terminal** state (`completed`, `failed`, `stopped`, or `deleted`), backing off between calls rather than running a tight loop.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import time, requests

  def wait_for_job(job_id, timeout=600):
      terminal = {'completed', 'failed', 'stopped', 'deleted'}
      headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}
      deadline = time.time() + timeout
      delay = 2.0
      while True:
          job = requests.get(f'https://async.api.zenrows.com/v1/jobs/{job_id}', headers=headers).json()
          run = job.get('latest_run')
          if run and run['status'] in terminal:
              return job
          if time.time() > deadline:
              raise TimeoutError('timeout waiting for job')
          time.sleep(delay)
          delay = min(delay * 1.5, 15.0)  # back off gradually, capped at 15s

  job = wait_for_job('JOB_ID')
  s = job['latest_run']['stats']  # { total, completed, successful, failed }
  print(job['latest_run']['status'], f"{s['successful']}/{s['total']} successful")
  ```

  ```javascript Node.js theme={"dark"}
  async function waitForJob(jobId, { timeoutMs = 600_000 } = {}) {
    const TERMINAL = new Set(["completed", "failed", "stopped", "deleted"]);
    const deadline = Date.now() + timeoutMs;
    let delay = 2000;
    for (;;) {
      const job = await api("GET", `/jobs/${jobId}`);
      const run = job.latest_run;
      if (run && TERMINAL.has(run.status)) return job;
      if (Date.now() > deadline) throw new Error("timeout waiting for job");
      await new Promise((r) => setTimeout(r, delay));
      delay = Math.min(delay * 1.5, 15_000); // back off gradually, capped at 15s
    }
  }

  const job = await waitForJob("JOB_ID");
  const s = job.latest_run.stats; // { total, completed, successful, failed }
  console.log(`${job.latest_run.status}: ${s.successful}/${s.total} successful`);
  ```

  ```java Java theme={"dark"}
  import com.fasterxml.jackson.databind.JsonNode;
  import com.fasterxml.jackson.databind.ObjectMapper;
  import org.apache.hc.client5.http.fluent.Request;
  import java.util.Set;

  public class WaitForJob {
      static final ObjectMapper M = new ObjectMapper();

      public static void main(final String... args) throws Exception {
          Set<String> terminal = Set.of("completed", "failed", "stopped", "deleted");
          long deadline = System.currentTimeMillis() + 600_000;
          long delay = 2000;
          while (true) {
              String body = Request.get("https://async.api.zenrows.com/v1/jobs/JOB_ID")
                      .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                      .execute().returnContent().asString();
              JsonNode run = M.readTree(body).path("latest_run");
              if (terminal.contains(run.path("status").asText())) {
                  System.out.println(run.path("stats"));
                  break;
              }
              if (System.currentTimeMillis() > deadline) throw new RuntimeException("timeout waiting for job");
              Thread.sleep(delay);
              delay = Math.min((long) (delay * 1.5), 15_000); // back off gradually, capped at 15s
          }
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  function get_job($id) {
      $ch = curl_init("https://async.api.zenrows.com/v1/jobs/$id");
      curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $res = curl_exec($ch);
      curl_close($ch);
      return json_decode($res, true);
  }

  $terminal = ['completed', 'failed', 'stopped', 'deleted'];
  $deadline = time() + 600;
  $delay = 2.0;
  while (true) {
      $job = get_job('JOB_ID');
      $run = $job['latest_run'] ?? null;
      if ($run && in_array($run['status'], $terminal, true)) { print_r($run['stats']); break; }
      if (time() > $deadline) throw new RuntimeException('timeout waiting for job');
      usleep((int)($delay * 1000000));
      $delay = min($delay * 1.5, 15.0); // back off gradually, capped at 15s
  }
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "encoding/json"
      "io"
      "log"
      "net/http"
      "time"
  )

  func getJob(id string) map[string]any {
      req, _ := http.NewRequest("GET", "https://async.api.zenrows.com/v1/jobs/"+id, nil)
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      var job map[string]any
      json.Unmarshal(body, &job)
      return job
  }

  func main() {
      terminal := map[string]bool{"completed": true, "failed": true, "stopped": true, "deleted": true}
      deadline := time.Now().Add(10 * time.Minute)
      delay := 2 * time.Second
      for {
          job := getJob("JOB_ID")
          run, _ := job["latest_run"].(map[string]any)
          status, _ := run["status"].(string)
          if run != nil && terminal[status] {
              log.Println(run["stats"])
              break
          }
          if time.Now().After(deadline) {
              log.Fatalln("timeout waiting for job")
          }
          time.Sleep(delay)
          if delay < 15*time.Second {
              delay = time.Duration(float64(delay) * 1.5)
          }
      }
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  def get_job(id)
      res = Faraday.get("https://async.api.zenrows.com/v1/jobs/#{id}") do |req|
          req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      end
      JSON.parse(res.body)
  end

  terminal = ['completed', 'failed', 'stopped', 'deleted']
  deadline = Time.now + 600
  delay = 2.0
  loop do
      job = get_job('JOB_ID')
      run = job['latest_run']
      if run && terminal.include?(run['status'])
          p run['stats']
          break
      end
      raise 'timeout waiting for job' if Time.now > deadline
      sleep(delay)
      delay = [delay * 1.5, 15.0].min  # back off gradually, capped at 15s
  end
  ```

  ```bash cURL theme={"dark"}
  # cURL has no loop of its own; poll with a small shell loop until the run is terminal.
  delay=2
  while :; do
    status=$(curl -s "https://async.api.zenrows.com/v1/jobs/JOB_ID" \
      -H "X-API-Key: YOUR_ZENROWS_API_KEY" | jq -r '.latest_run.status')
    echo "status: $status"
    case "$status" in completed|failed|stopped|deleted) break ;; esac
    sleep "$delay"
    delay=$(( delay < 15 ? delay + delay/2 + 1 : 15 ))  # back off, capped at 15s
  done
  ```
</CodeGroup>

### Why a run failed

When a run ends with status `failed` for a reason that affects the whole run, the run carries two extra fields:

* `failure_reason`: a stable code you can branch on.
* `failure_detail`: a human-readable explanation, such as which credit cap was reached and when it resets.

| `failure_reason` | Meaning |
| - | - |
| `insufficient_credits` | Your account has no credits available. |
| `subscription_inactive` | Your subscription isn't active. |
| `api_key_cap_reached` | The API key reached one of its [credit caps](/first-steps/api-key-credit-caps). See [API key credit caps and Batch](#api-key-credit-caps-and-batch). |

<Warning>
  `failure_reason` is an open list. New values can be added at any time, so your code must accept values not listed here, for example by falling back to showing `failure_detail`.
</Warning>

### API key credit caps and Batch

A Batch job uses the [credit caps](/first-steps/api-key-credit-caps) of the API key it runs with. A capped key affects Batch in three ways:

* **Refused requests.** Starting a job, adding tasks, resuming, or rerunning with a key that's at one of its caps returns `402` with the problem `code` `api_key_cap_reached`. The `detail` names the cap and when it resets.
* **A run in progress stops.** If the key reaches a cap during a run, the run stops with status `failed` and `failure_reason` `api_key_cap_reached`, and `failure_detail` says which cap and when it resets. Tasks already in progress finish. The remaining tasks stay `pending`: they aren't processed and aren't charged. Because a run processes many tasks in parallel, it can go further over a cap than single requests do before it stops, so leave room in caps for keys you use with Batch.
* **Scheduled runs are checked when they start.** If the key is capped, the account has no credits, or the subscription is inactive, the run is recorded as `failed` right away with the matching `failure_reason` and a `failure_detail`. Nothing is scraped.

To recover:

1. Raise or remove the cap in the dashboard (**Settings**, **API keys**, the key's menu, **Credit caps**), or wait for the cap window to reset.
2. Rerun only the tasks that didn't complete with `POST /jobs/{job_id}/rerun?status=failed,pending` (see [Reruns and retrying failures](#5-reruns-and-retrying-failures)).

   A scheduled run refused at start has no tasks, so there is nothing to filter. Rerun the whole job with `POST /jobs/{job_id}/rerun`, or let the next scheduled run pick it up.

### List and paginate jobs

List your jobs with `GET /jobs`, optionally filtered by `status` and `type`, and page with `cursor` until `next_cursor` is null.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  response = requests.get('https://async.api.zenrows.com/v1/jobs',
                          headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'},
                          params={'status': 'closed', 'type': 'regular'})
  print(response.json())
  ```

  ```javascript Node.js theme={"dark"}
  const page = await api("GET", "/jobs", { query: { status: "closed", type: "regular" } });
  console.log(page.jobs);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class ListJobs {
      public static void main(final String... args) throws Exception {
          String response = Request.get("https://async.api.zenrows.com/v1/jobs?status=closed&type=regular")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1/jobs?status=closed&type=regular');
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      req, err := http.NewRequest("GET", "https://async.api.zenrows.com/v1/jobs?status=closed&type=regular", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  conn = Faraday.new
  res = conn.get('https://async.api.zenrows.com/v1/jobs') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.params['status'] = 'closed'
      req.params['type'] = 'regular'
  end
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://async.api.zenrows.com/v1/jobs?status=closed&type=regular" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

The cursor loop is the same as for results, and `GET /jobs/{id}/runs` pages identically.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}
  cursor = None
  while True:
      page = requests.get('https://async.api.zenrows.com/v1/jobs', headers=headers,
                          params={'status': 'closed', 'type': 'regular', 'cursor': cursor}).json()
      for job in page['jobs']:
          latest = job.get('latest_run')
          print(job['job_id'], job['status'], latest['stats']['total'] if latest else 0)
      cursor = page['next_cursor']
      if not cursor:
          break
  ```

  ```javascript Node.js theme={"dark"}
  async function* iterJobs({ status, type } = {}) {
    let cursor;
    do {
      const page = await api("GET", "/jobs", { query: { status, type, cursor } });
      yield* page.jobs;
      cursor = page.next_cursor;
    } while (cursor);
  }

  for await (const job of iterJobs({ status: "closed", type: "regular" })) {
    console.log(job.job_id, job.status, job.latest_run?.stats.total ?? 0);
  }
  ```

  ```java Java theme={"dark"}
  import com.fasterxml.jackson.databind.JsonNode;
  import com.fasterxml.jackson.databind.ObjectMapper;
  import org.apache.hc.client5.http.fluent.Request;

  public class ListJobsPaged {
      static final ObjectMapper M = new ObjectMapper();

      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";
          String cursor = null;
          do {
              String url = "https://async.api.zenrows.com/v1/jobs?status=closed&type=regular"
                      + (cursor != null ? "&cursor=" + cursor : "");
              JsonNode page = M.readTree(Request.get(url).addHeader("X-API-Key", key)
                      .execute().returnContent().asString());
              for (JsonNode job : page.path("jobs")) {
                  long total = job.path("latest_run").path("stats").path("total").asLong(0);
                  System.out.println(job.path("job_id").asText() + " " + job.path("status").asText() + " " + total);
              }
              cursor = page.path("next_cursor").asText(null);
          } while (cursor != null && !cursor.isEmpty());
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $key = 'YOUR_ZENROWS_API_KEY';
  $cursor = null;
  do {
      $url = 'https://async.api.zenrows.com/v1/jobs?status=closed&type=regular'
          . ($cursor ? '&cursor=' . urlencode($cursor) : '');
      $ch = curl_init($url);
      curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key"]);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $page = json_decode(curl_exec($ch), true);
      curl_close($ch);
      foreach ($page['jobs'] as $job) {
          $total = $job['latest_run']['stats']['total'] ?? 0;
          echo "{$job['job_id']} {$job['status']} $total" . PHP_EOL;
      }
      $cursor = $page['next_cursor'] ?? null;
  } while ($cursor);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "encoding/json"
      "io"
      "log"
      "net/http"
  )

  func main() {
      key := "YOUR_ZENROWS_API_KEY"
      cursor := ""
      for {
          url := "https://async.api.zenrows.com/v1/jobs?status=closed&type=regular"
          if cursor != "" {
              url += "&cursor=" + cursor
          }
          req, _ := http.NewRequest("GET", url, nil)
          req.Header.Set("X-API-Key", key)
          resp, err := http.DefaultClient.Do(req)
          if err != nil {
              log.Fatalln(err)
          }
          body, _ := io.ReadAll(resp.Body)
          resp.Body.Close()
          var page struct {
              Jobs []struct {
                  JobID     string `json:"job_id"`
                  Status    string `json:"status"`
                  LatestRun *struct {
                      Stats struct {
                          Total int `json:"total"`
                      } `json:"stats"`
                  } `json:"latest_run"`
              } `json:"jobs"`
              NextCursor string `json:"next_cursor"`
          }
          json.Unmarshal(body, &page)
          for _, job := range page.Jobs {
              total := 0
              if job.LatestRun != nil {
                  total = job.LatestRun.Stats.Total
              }
              log.Println(job.JobID, job.Status, total)
          }
          cursor = page.NextCursor
          if cursor == "" {
              break
          }
      }
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  key = 'YOUR_ZENROWS_API_KEY'
  cursor = nil
  loop do
      res = Faraday.get('https://async.api.zenrows.com/v1/jobs') do |req|
          req.headers['X-API-Key'] = key
          req.params['status'] = 'closed'
          req.params['type'] = 'regular'
          req.params['cursor'] = cursor if cursor
      end
      page = JSON.parse(res.body)
      page['jobs'].each do |job|
          total = job.dig('latest_run', 'stats', 'total') || 0
          puts "#{job['job_id']} #{job['status']} #{total}"
      end
      cursor = page['next_cursor']
      break unless cursor
  end
  ```

  ```bash cURL theme={"dark"}
  cursor=""
  while :; do
    page=$(curl -s "https://async.api.zenrows.com/v1/jobs?status=closed&type=regular&cursor=$cursor" \
      -H "X-API-Key: YOUR_ZENROWS_API_KEY")
    echo "$page" | jq -r '.jobs[] | [.job_id, .status, (.latest_run.stats.total // 0)] | @tsv'
    cursor=$(echo "$page" | jq -r '.next_cursor // empty')
    [ -z "$cursor" ] && break
  done
  ```
</CodeGroup>

### Control the current run

Stop, pause, or resume a job's **latest run**. Each endpoint is a `POST` with no request body and returns the refreshed run object on `200`.

| Endpoint | Purpose |
| - | - |
| `POST /jobs/{id}/stop` | Stop the current run (terminal). |
| `POST /jobs/{id}/cancel` | Alias for `/stop`, identical behavior. |
| `POST /jobs/{id}/pause` | Reversibly suspend the current run. |
| `POST /jobs/{id}/resume` | Resume a paused run. |

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests
  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}

  # Stop (terminal). /cancel is an identical alias.
  requests.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/stop', headers=headers)

  # Pause the current run, then resume it later.
  requests.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/pause', headers=headers)
  requests.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/resume', headers=headers)
  ```

  ```javascript Node.js theme={"dark"}
  // Stop (terminal). /cancel is an identical alias.
  await api("POST", "/jobs/JOB_ID/stop");

  // Pause the current run, then resume it later.
  await api("POST", "/jobs/JOB_ID/pause");
  await api("POST", "/jobs/JOB_ID/resume");
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class ControlRun {
      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";
          String base = "https://async.api.zenrows.com/v1/jobs/JOB_ID";
          // Stop (terminal); /cancel is an identical alias.
          Request.post(base + "/stop").addHeader("X-API-Key", key).execute().discardContent();
          // Pause and resume the current run.
          Request.post(base + "/pause").addHeader("X-API-Key", key).execute().discardContent();
          Request.post(base + "/resume").addHeader("X-API-Key", key).execute().discardContent();
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  function control($action) {
      $ch = curl_init("https://async.api.zenrows.com/v1/jobs/JOB_ID/$action");
      curl_setopt($ch, CURLOPT_POST, 1);
      curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $res = curl_exec($ch);
      curl_close($ch);
      return $res;
  }
  echo control('stop');   // /cancel is an identical alias
  echo control('pause');
  echo control('resume');
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func control(action string) {
      req, _ := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs/JOB_ID/"+action, nil)
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }

  func main() {
      control("stop") // /cancel is an identical alias
      control("pause")
      control("resume")
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  conn = Faraday.new
  conn.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'

  # Stop (terminal); /cancel is an identical alias.
  conn.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/stop')
  # Pause and resume the current run.
  conn.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/pause')
  conn.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/resume')
  ```

  ```bash cURL theme={"dark"}
  # Stop the current run (terminal). /cancel is an identical alias.
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/stop" -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  # Pause the current run, then resume it later.
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/pause"  -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/resume" -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

**Stop and cancel.** `/stop` moves the latest run from `running` or `pending` to the terminal state `stopped`. No new tasks are picked up, tasks already in flight may still finish and record a result, and pending tasks are left as-is (not re-queued, not failed). Result bodies are kept; to remove them, follow up with `DELETE /jobs/{id}/runs/{run_id}` (see [Delete a job or a run](#delete-a-job-or-a-run)). It is idempotent on an already-stopped run, returns `409` if the run is `completed` or `deleted` (use a rerun to start fresh), and `404` on a deleted job. `/cancel` is a pure alias for `/stop`.

**Pause and resume.** `/pause` reversibly suspends the latest run: the dispatcher stops polling its queue, while the run's `status` (`running` or `pending`) is unchanged, since pause is orthogonal to status. Tasks already in flight may still settle. `/resume` restarts polling. Both are idempotent (pausing a paused run, or resuming an active run, is a no-op), and both return `409` when the latest run is terminal.

<Warning>
  In-flight gateway calls still consume credits while a pause takes effect. For a hard cost-stop, use `/stop` (or `/cancel`) instead.
</Warning>

<Note>
  This is not the same as pausing a **schedule**. For scheduled jobs, `POST /jobs/{id}/schedule/state` controls whether new runs fire. The `/pause` and `/resume` endpoints here act on the current run.
</Note>

### Delete a job or a run

You can remove data yourself when you no longer need it. Two levels are available: delete a single run's results, or delete an entire job.

| Endpoint | Purpose |
| - | - |
| `DELETE /jobs/{id}/runs/{run_id}` | Delete one run's stored results, keeping the job and its other runs. |
| `DELETE /jobs/{id}` | Delete the whole job, including every run and result. |

<Note>
  You are not required to delete anything to manage storage. Results and their download links are retained for **14 days** after a run finishes, then removed automatically. These endpoints are for removing data sooner than that.
</Note>

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests
  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}

  # Delete one run's results, keeping the job.
  requests.delete('https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID', headers=headers)

  # Delete the whole job (all runs and results).
  requests.delete('https://async.api.zenrows.com/v1/jobs/JOB_ID', headers=headers)
  ```

  ```javascript Node.js theme={"dark"}
  // Delete one run's results, keeping the job.
  await api("DELETE", "/jobs/JOB_ID/runs/RUN_ID");

  // Delete the whole job (all runs and results).
  await api("DELETE", "/jobs/JOB_ID");
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class DeleteJob {
      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";
          String base = "https://async.api.zenrows.com/v1/jobs/JOB_ID";
          // Delete one run's results, keeping the job.
          Request.delete(base + "/runs/RUN_ID").addHeader("X-API-Key", key).execute().discardContent();
          // Delete the whole job (all runs and results).
          Request.delete(base).addHeader("X-API-Key", key).execute().discardContent();
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  function del($path) {
      $ch = curl_init("https://async.api.zenrows.com/v1$path");
      curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'DELETE');
      curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $res = curl_exec($ch);
      curl_close($ch);
      return $res;
  }
  echo del('/jobs/JOB_ID/runs/RUN_ID'); // one run's results
  echo del('/jobs/JOB_ID');             // the whole job
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "log"
      "net/http"
  )

  func del(path string) {
      req, _ := http.NewRequest("DELETE", "https://async.api.zenrows.com/v1"+path, nil)
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      log.Println(resp.Status)
  }

  func main() {
      del("/jobs/JOB_ID/runs/RUN_ID") // one run's results
      del("/jobs/JOB_ID")             // the whole job
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  conn = Faraday.new
  conn.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'

  # Delete one run's results, keeping the job.
  conn.delete('https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID')
  # Delete the whole job (all runs and results).
  conn.delete('https://async.api.zenrows.com/v1/jobs/JOB_ID')
  ```

  ```bash cURL theme={"dark"}
  # Delete one run's results, keeping the job.
  curl -X DELETE "https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID" -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  # Delete the whole job (all runs and results).
  curl -X DELETE "https://async.api.zenrows.com/v1/jobs/JOB_ID" -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

**Delete a run.** `DELETE /jobs/{id}/runs/{run_id}` removes that run's stored result bodies while leaving the job and its other runs in place. Use it once you have downloaded what you need from the run. It is idempotent, so deleting an already-deleted run is safe.

**Delete a job.** `DELETE /jobs/{id}` removes the entire job, including every run and result. Deletion of the stored contents happens in the background, so the request is accepted right away and the data is cleared shortly after. Once deleted, `GET /jobs/{id}` returns `404` and the job no longer appears in the jobs list. Stopping a run first is not required; deleting a job with an active run stops it as part of the delete.

<Note>
  Delete is permanent. Download or export any results you want to keep before deleting a run or a job.
</Note>

## 3. Collect your results

Retrieve the scraped content once a run completes.

### Download content

List a run's results with `GET /jobs/{job_id}/results`, filtered by `status` (`successful`, `failed`, or `all`) and paged with the `cursor` query parameter until `next_cursor` is null.

<Note>Each row's `result_url` is a presigned link valid for 2 hours, so fetch it promptly. Results are retained for 14 days after a run finishes, then removed automatically; re-list the run to get fresh links within that window. To remove results sooner, see [Delete a job or a run](#delete-a-job-or-a-run).</Note>

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  response = requests.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/results',
                          headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'},
                          params={'status': 'successful'})
  page = response.json()
  for row in page['results']:
      print(row.get('external_id') or row['task_id'], '->', row['result_url'])
  print('next_cursor:', page['next_cursor'])
  ```

  ```javascript Node.js theme={"dark"}
  const page = await api("GET", "/jobs/JOB_ID/results", { query: { status: "successful" } });
  for (const row of page.results) {
    console.log(row.external_id ?? row.task_id, "->", row.result_url);
  }
  console.log("next_cursor:", page.next_cursor);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class GetResults {
      public static void main(final String... args) throws Exception {
          String response = Request.get("https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful');
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      req, err := http.NewRequest("GET", "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  conn = Faraday.new
  res = conn.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/results') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.params['status'] = 'successful'
  end
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

To page through every result and download each body, loop over the pages until `next_cursor` is null, fetching each presigned `result_url` promptly.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import os, requests

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}
  os.makedirs('out', exist_ok=True)
  cursor = None
  while True:
      page = requests.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/results',
                          headers=headers, params={'status': 'successful', 'cursor': cursor}).json()
      for row in page['results']:
          # result_url is a presigned link, valid 2h, so fetch it promptly.
          content = requests.get(row['result_url']).content
          with open(f"out/{row.get('external_id') or row['task_id']}", 'wb') as f:
              f.write(content)
      cursor = page['next_cursor']
      if not cursor:
          break
  ```

  ```javascript Node.js theme={"dark"}
  async function* iterResults(jobId, status = "all") {
    let cursor;
    do {
      const page = await api("GET", `/jobs/${jobId}/results`, { query: { status, cursor } });
      yield* page.results;
      cursor = page.next_cursor;
    } while (cursor);
  }

  import { writeFile } from "node:fs/promises";

  for await (const row of iterResults("JOB_ID", "successful")) {
    // row.result_url is a presigned link, valid 2h, so fetch it promptly.
    const bytes = Buffer.from(await (await fetch(row.result_url)).arrayBuffer());
    await writeFile(`./out/${row.external_id ?? row.task_id}`, bytes);
  }
  ```

  ```java Java theme={"dark"}
  import com.fasterxml.jackson.databind.JsonNode;
  import com.fasterxml.jackson.databind.ObjectMapper;
  import org.apache.hc.client5.http.fluent.Request;
  import java.nio.file.Files;
  import java.nio.file.Path;

  public class DownloadResults {
      static final ObjectMapper M = new ObjectMapper();

      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";
          String cursor = null;
          do {
              String url = "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful"
                      + (cursor != null ? "&cursor=" + cursor : "");
              JsonNode page = M.readTree(Request.get(url).addHeader("X-API-Key", key)
                      .execute().returnContent().asString());
              for (JsonNode row : page.path("results")) {
                  // result_url is a presigned link, valid 2h, so fetch it promptly.
                  byte[] bytes = Request.get(row.path("result_url").asText())
                          .execute().returnContent().asBytes();
                  String name = row.path("external_id").asText(row.path("task_id").asText());
                  Files.write(Path.of("out", name), bytes);
              }
              cursor = page.path("next_cursor").asText(null);
          } while (cursor != null && !cursor.isEmpty());
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $key = 'YOUR_ZENROWS_API_KEY';
  @mkdir('out');
  $cursor = null;
  do {
      $url = 'https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful'
          . ($cursor ? '&cursor=' . urlencode($cursor) : '');
      $ch = curl_init($url);
      curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key"]);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $page = json_decode(curl_exec($ch), true);
      curl_close($ch);
      foreach ($page['results'] as $row) {
          // result_url is a presigned link, valid 2h, so fetch it promptly.
          $name = $row['external_id'] ?? $row['task_id'];
          file_put_contents("out/$name", file_get_contents($row['result_url']));
      }
      $cursor = $page['next_cursor'] ?? null;
  } while ($cursor);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "encoding/json"
      "io"
      "log"
      "net/http"
      "os"
      "path/filepath"
  )

  func get(url, key string) []byte {
      req, _ := http.NewRequest("GET", url, nil)
      if key != "" {
          req.Header.Set("X-API-Key", key)
      }
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      return body
  }

  func main() {
      key := "YOUR_ZENROWS_API_KEY"
      os.MkdirAll("out", 0o755)
      cursor := ""
      for {
          url := "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful"
          if cursor != "" {
              url += "&cursor=" + cursor
          }
          var page struct {
              Results []struct {
                  TaskID     string `json:"task_id"`
                  ExternalID string `json:"external_id"`
                  ResultURL  string `json:"result_url"`
              } `json:"results"`
              NextCursor string `json:"next_cursor"`
          }
          json.Unmarshal(get(url, key), &page)
          for _, row := range page.Results {
              // result_url is a presigned link, valid 2h, so fetch it promptly.
              name := row.ExternalID
              if name == "" {
                  name = row.TaskID
              }
              os.WriteFile(filepath.Join("out", name), get(row.ResultURL, ""), 0o644)
          }
          cursor = page.NextCursor
          if cursor == "" {
              break
          }
      }
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  key = 'YOUR_ZENROWS_API_KEY'
  Dir.mkdir('out') unless Dir.exist?('out')
  cursor = nil
  loop do
      res = Faraday.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/results') do |req|
          req.headers['X-API-Key'] = key
          req.params['status'] = 'successful'
          req.params['cursor'] = cursor if cursor
      end
      page = JSON.parse(res.body)
      page['results'].each do |row|
          # result_url is a presigned link, valid 2h, so fetch it promptly.
          name = row['external_id'] || row['task_id']
          File.write("out/#{name}", Faraday.get(row['result_url']).body, mode: 'wb')
      end
      cursor = page['next_cursor']
      break unless cursor
  end
  ```

  ```bash cURL theme={"dark"}
  # Page with a shell loop, then download each presigned result_url.
  mkdir -p out
  cursor=""
  while :; do
    page=$(curl -s "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful&cursor=$cursor" \
      -H "X-API-Key: YOUR_ZENROWS_API_KEY")
    echo "$page" | jq -r '.results[] | [(.external_id // .task_id), .result_url] | @tsv' \
      | while IFS=$'\t' read -r name url; do curl -s "$url" -o "out/$name"; done
    cursor=$(echo "$page" | jq -r '.next_cursor // empty')
    [ -z "$cursor" ] && break
  done
  ```
</CodeGroup>

You can also fetch one task's body directly with `GET /jobs/{job_id}/tasks/{task_id}/content`.

* `200`: returns the content
* `409`: means the task is still processing
* `422`: means the task failed and the body holds the stored error.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  # 200 = content; 409 = still pending/processing; 422 = failed (body is the error)
  response = requests.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks/TASK_ID/content',
                          headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'})
  print(response.status_code)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // 200 = content; 409 = still pending/processing; 422 = failed (body is the error)
  const content = await api("GET", "/jobs/JOB_ID/tasks/TASK_ID/content");
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class TaskContent {
      public static void main(final String... args) throws Exception {
          // 200 = content; 409 = still pending/processing; 422 = failed (body is the error)
          String response = Request.get("https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks/TASK_ID/content")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  // 200 = content; 409 = still pending/processing; 422 = failed (body is the error)
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks/TASK_ID/content');
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      // 200 = content; 409 = still pending/processing; 422 = failed (body is the error)
      req, err := http.NewRequest("GET", "https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks/TASK_ID/content", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(resp.StatusCode, string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  # 200 = content; 409 = still pending/processing; 422 = failed (body is the error)
  conn = Faraday.new
  res = conn.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks/TASK_ID/content') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
  end
  print(res.status)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  # 200 = content; 409 = still pending/processing; 422 = failed (body is the error)
  curl -i "https://async.api.zenrows.com/v1/jobs/JOB_ID/tasks/TASK_ID/content" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

<Tip>
  A failed task's results row already carries `row.error` (`{ code, detail, ... }`), so you rarely need the content endpoint to inspect a failure. Treat the error `code` as a hint: it does not yet reliably distinguish blocked, not-found, and timeout.
</Tip>

### Bulk export (ZIP)

Bundle an entire run's result bodies into a single ZIP. Start the export with `POST /jobs/{job_id}/runs/{run_id}/exports`, poll the export until `status` is `completed`, then download the presigned `download_url`.

<Note>Exports expire 12 hours after creation, the API presigns `download_url` fresh on each poll, and a run whose combined results exceed 100 GiB fails with a clear error instead of producing a partial archive.</Note>

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  started = requests.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports',
                          headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'}).json()
  print(started['export_id'], started['status'])
  # Poll the export by id until status == 'completed', then download started['download_url'].
  ```

  ```javascript Node.js theme={"dark"}
  const started = await api("POST", "/jobs/JOB_ID/runs/RUN_ID/exports");
  console.log(started.export_id, started.status);
  // Poll the export by id until status === 'completed', then download download_url.
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class StartExport {
      public static void main(final String... args) throws Exception {
          String response = Request.post("https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .execute().returnContent().asString();
          System.out.println(response);
          // Poll the export by id until status is "completed", then download download_url.
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  // Poll the export by id until status is "completed", then download download_url.
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
      // Poll the export by id until status is "completed", then download download_url.
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  conn = Faraday.new
  res = conn.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
  end
  print(res.body)
  # Poll the export by id until status is "completed", then download download_url.
  ```

  ```bash cURL theme={"dark"}
  # Start the export, then poll the returned export_id until status is "completed".
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  curl "https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports/EXPORT_ID" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

Start the export, poll until it is `completed`, then download the ZIP.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import time, requests

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}
  base = 'https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports'

  exp = requests.post(base, headers=headers).json()
  while exp['status'] not in ('completed', 'failed'):
      time.sleep(3)
      exp = requests.get(f"{base}/{exp['export_id']}", headers=headers).json()
  if exp['status'] == 'failed':
      raise RuntimeError(exp.get('error'))

  with open('results.zip', 'wb') as f:
      f.write(requests.get(exp['download_url']).content)
  ```

  ```javascript Node.js theme={"dark"}
  import { writeFile } from "node:fs/promises";

  let exp = await api("POST", "/jobs/JOB_ID/runs/RUN_ID/exports");
  while (exp.status !== "completed" && exp.status !== "failed") {
    await new Promise((r) => setTimeout(r, 3000));
    exp = await api("GET", `/jobs/JOB_ID/runs/RUN_ID/exports/${exp.export_id}`);
  }
  if (exp.status === "failed") throw new Error(exp.error);

  const zip = Buffer.from(await (await fetch(exp.download_url)).arrayBuffer());
  await writeFile("results.zip", zip);
  ```

  ```java Java theme={"dark"}
  import com.fasterxml.jackson.databind.JsonNode;
  import com.fasterxml.jackson.databind.ObjectMapper;
  import org.apache.hc.client5.http.fluent.Request;
  import java.nio.file.Files;
  import java.nio.file.Path;

  public class ExportRun {
      static final ObjectMapper M = new ObjectMapper();

      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";
          String base = "https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports";

          JsonNode exp = M.readTree(Request.post(base).addHeader("X-API-Key", key)
                  .execute().returnContent().asString());
          String id = exp.path("export_id").asText();
          while (!exp.path("status").asText().equals("completed") && !exp.path("status").asText().equals("failed")) {
              Thread.sleep(3000);
              exp = M.readTree(Request.get(base + "/" + id).addHeader("X-API-Key", key)
                      .execute().returnContent().asString());
          }
          if (exp.path("status").asText().equals("failed")) throw new RuntimeException(exp.path("error").asText());

          byte[] zip = Request.get(exp.path("download_url").asText()).execute().returnContent().asBytes();
          Files.write(Path.of("results.zip"), zip);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $key = 'YOUR_ZENROWS_API_KEY';
  $base = 'https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports';

  function call($method, $url, $key) {
      $ch = curl_init($url);
      curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
      curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key"]);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $res = curl_exec($ch);
      curl_close($ch);
      return json_decode($res, true);
  }

  $exp = call('POST', $base, $key);
  while (!in_array($exp['status'], ['completed', 'failed'], true)) {
      sleep(3);
      $exp = call('GET', "$base/{$exp['export_id']}", $key);
  }
  if ($exp['status'] === 'failed') throw new RuntimeException($exp['error'] ?? 'export failed');

  file_put_contents('results.zip', file_get_contents($exp['download_url']));
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "encoding/json"
      "io"
      "log"
      "net/http"
      "os"
      "time"
  )

  func call(method, url, key string) map[string]any {
      req, _ := http.NewRequest(method, url, nil)
      if key != "" {
          req.Header.Set("X-API-Key", key)
      }
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      var out map[string]any
      json.Unmarshal(body, &out)
      return out
  }

  func main() {
      key := "YOUR_ZENROWS_API_KEY"
      base := "https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports"

      exp := call("POST", base, key)
      id, _ := exp["export_id"].(string)
      for exp["status"] != "completed" && exp["status"] != "failed" {
          time.Sleep(3 * time.Second)
          exp = call("GET", base+"/"+id, key)
      }
      if exp["status"] == "failed" {
          log.Fatalln(exp["error"])
      }

      req, _ := http.NewRequest("GET", exp["download_url"].(string), nil)
      resp, _ := http.DefaultClient.Do(req)
      defer resp.Body.Close()
      out, _ := os.Create("results.zip")
      defer out.Close()
      io.Copy(out, resp.Body)
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  key = 'YOUR_ZENROWS_API_KEY'
  base = 'https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports'

  exp = JSON.parse(Faraday.post(base) { |r| r.headers['X-API-Key'] = key }.body)
  until %w[completed failed].include?(exp['status'])
      sleep 3
      exp = JSON.parse(Faraday.get("#{base}/#{exp['export_id']}") { |r| r.headers['X-API-Key'] = key }.body)
  end
  raise(exp['error'] || 'export failed') if exp['status'] == 'failed'

  File.write('results.zip', Faraday.get(exp['download_url']).body, mode: 'wb')
  ```

  ```bash cURL theme={"dark"}
  base="https://async.api.zenrows.com/v1/jobs/JOB_ID/runs/RUN_ID/exports"
  started=$(curl -s -X POST "$base" -H "X-API-Key: YOUR_ZENROWS_API_KEY")
  id=$(echo "$started" | jq -r '.export_id')

  while :; do
    exp=$(curl -s "$base/$id" -H "X-API-Key: YOUR_ZENROWS_API_KEY")
    status=$(echo "$exp" | jq -r '.status')
    case "$status" in completed|failed) break ;; esac
    sleep 3
  done
  [ "$status" = "failed" ] && { echo "export failed"; exit 1; }

  curl -s "$(echo "$exp" | jq -r '.download_url')" -o results.zip
  ```
</CodeGroup>

## 4. Webhooks and notifications

Get notified when a run finishes, and secure those callbacks.

### Set up webhooks

Attach a webhook so the API POSTs a `run.completed` event to your HTTPS URL when a run completes, or a `run.failed` event when it fails. Set `signature: true` for HMAC-signed deliveries (see [Manage HMAC keys](#manage-hmac-keys)).

You can attach it on the submit body, replace it later with `PUT /jobs/{id}/webhook`, remove it with `DELETE /jobs/{id}/webhook`, and probe a receiver with `POST /webhook/test`.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json'}
  webhook = {'url': 'https://your-app.com/webhooks/zenrows', 'signature': True}

  # Set or replace the webhook on an existing job:
  requests.put('https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook', headers=headers, data=json.dumps(webhook))

  # Probe a receiver before wiring it up:
  probe = requests.post('https://async.api.zenrows.com/v1/webhook/test', headers=headers, data=json.dumps(webhook)).json()
  print(probe['delivered'], probe['status_code'])

  # Stop deliveries:
  requests.delete('https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook', headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'})
  ```

  ```javascript Node.js theme={"dark"}
  const webhook = { url: "https://your-app.com/webhooks/zenrows", signature: true };

  // At submit time, include it on the body: { ..., webhook }.
  // Or manage it on an existing job:
  await api("PUT", "/jobs/JOB_ID/webhook", { body: webhook });
  await api("DELETE", "/jobs/JOB_ID/webhook"); // stop deliveries

  // Verify a receiver before wiring it to a job:
  const probe = await api("POST", "/webhook/test", { body: webhook });
  console.log(probe.delivered, probe.status_code);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;

  public class Webhook {
      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";
          String webhook = "{\"url\":\"https://your-app.com/webhooks/zenrows\",\"signature\":true}";

          // Set or replace the webhook on an existing job:
          Request.put("https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook")
                  .addHeader("X-API-Key", key).bodyString(webhook, ContentType.APPLICATION_JSON)
                  .execute().discardContent();

          // Probe a receiver:
          String probe = Request.post("https://async.api.zenrows.com/v1/webhook/test")
                  .addHeader("X-API-Key", key).bodyString(webhook, ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(probe);

          // Stop deliveries:
          Request.delete("https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook")
                  .addHeader("X-API-Key", key).execute().discardContent();
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $key = 'YOUR_ZENROWS_API_KEY';
  $webhook = json_encode(['url' => 'https://your-app.com/webhooks/zenrows', 'signature' => true]);

  // Set or replace the webhook on an existing job:
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
  curl_setopt($ch, CURLOPT_POSTFIELDS, $webhook);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key", 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_exec($ch);
  curl_close($ch);

  // Probe a receiver:
  $ch = curl_init('https://async.api.zenrows.com/v1/webhook/test');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $webhook);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key", 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "io"
      "log"
      "net/http"
  )

  func main() {
      webhook := `{ "url": "https://your-app.com/webhooks/zenrows", "signature": true }`

      // Set or replace the webhook on an existing job:
      putReq, _ := http.NewRequest("PUT", "https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook", bytes.NewBufferString(webhook))
      putReq.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      putReq.Header.Set("Content-Type", "application/json")
      if _, err := http.DefaultClient.Do(putReq); err != nil {
          log.Fatalln(err)
      }

      // Probe a receiver:
      testReq, _ := http.NewRequest("POST", "https://async.api.zenrows.com/v1/webhook/test", bytes.NewBufferString(webhook))
      testReq.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      testReq.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(testReq)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  conn = Faraday.new
  conn.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
  conn.headers['Content-Type'] = 'application/json'
  webhook = { url: 'https://your-app.com/webhooks/zenrows', signature: true }.to_json

  # Set or replace the webhook on an existing job:
  conn.put('https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook', webhook)

  # Probe a receiver:
  puts conn.post('https://async.api.zenrows.com/v1/webhook/test', webhook).body

  # Stop deliveries:
  conn.delete('https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook')
  ```

  ```bash cURL theme={"dark"}
  # Set or replace the webhook on an existing job:
  curl -X PUT "https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "url": "https://your-app.com/webhooks/zenrows", "signature": true }'

  # Probe a receiver:
  curl -X POST "https://async.api.zenrows.com/v1/webhook/test" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "url": "https://your-app.com/webhooks/zenrows", "signature": false }'

  # Stop deliveries:
  curl -X DELETE "https://async.api.zenrows.com/v1/jobs/JOB_ID/webhook" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

<Tip>
  The API delivers each event at least once, so deduplicate on the `X-ZenRows-Event-Id` header. Webhooks are a convenience; the run state and result URLs remain the reliable source of truth.
</Tip>

When a run fails, the webhook receives a `run.failed` event. It carries the run's `failure_reason` and `failure_detail`, the same fields described in [Why a run failed](#why-a-run-failed).

### Manage HMAC keys

Signed webhook deliveries need an active HMAC key for your organization. Rotation uses two slots, an active key plus a candidate, so you can roll keys without downtime.

* `POST /hmac/keys/rotate` creates the first key or stages a candidate (the secret is returned once)
* `POST /hmac/keys/rotate/finalize` promotes the candidate and retires the old key
* `DELETE /hmac/keys/rotate` discards a staged candidate
* `GET /hmac/keys` lists key metadata without secrets

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests
  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}

  created = requests.post('https://async.api.zenrows.com/v1/hmac/keys/rotate', headers=headers).json()
  print(created)  # store created['secret'] now; it is returned only once

  requests.post('https://async.api.zenrows.com/v1/hmac/keys/rotate/finalize', headers=headers)
  # requests.delete('https://async.api.zenrows.com/v1/hmac/keys/rotate', headers=headers)  # discard candidate
  print(requests.get('https://async.api.zenrows.com/v1/hmac/keys', headers=headers).json())
  ```

  ```javascript Node.js theme={"dark"}
  const created = await api("POST", "/hmac/keys/rotate");
  console.log(created.kid, created.secret); // the secret is returned ONCE, so store it now

  await api("POST", "/hmac/keys/rotate/finalize"); // promote candidate -> active, retire the old
  // await api("DELETE", "/hmac/keys/rotate");     // or discard the candidate
  console.log(await api("GET", "/hmac/keys"));     // metadata only; never returns secrets
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class HmacKeys {
      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";

          String created = Request.post("https://async.api.zenrows.com/v1/hmac/keys/rotate")
                  .addHeader("X-API-Key", key).execute().returnContent().asString();
          System.out.println(created); // store the secret now; it is returned only once

          Request.post("https://async.api.zenrows.com/v1/hmac/keys/rotate/finalize")
                  .addHeader("X-API-Key", key).execute().discardContent();
          // Request.delete("https://async.api.zenrows.com/v1/hmac/keys/rotate")
          //         .addHeader("X-API-Key", key).execute().discardContent(); // discard candidate

          String list = Request.get("https://async.api.zenrows.com/v1/hmac/keys")
                  .addHeader("X-API-Key", key).execute().returnContent().asString();
          System.out.println(list);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $key = 'YOUR_ZENROWS_API_KEY';
  function call($method, $path, $key) {
      $ch = curl_init('https://async.api.zenrows.com/v1' . $path);
      curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
      curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key"]);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      $res = curl_exec($ch);
      curl_close($ch);
      return $res;
  }

  echo call('POST', '/hmac/keys/rotate', $key) . PHP_EOL;          // store the secret now
  call('POST', '/hmac/keys/rotate/finalize', $key);                // promote the candidate
  // call('DELETE', '/hmac/keys/rotate', $key);                    // or discard the candidate
  echo call('GET', '/hmac/keys', $key) . PHP_EOL;                  // metadata only
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func call(method, path string) string {
      req, _ := http.NewRequest(method, "https://async.api.zenrows.com/v1"+path, nil)
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      return string(body)
  }

  func main() {
      log.Println(call("POST", "/hmac/keys/rotate"))           // store the secret now
      call("POST", "/hmac/keys/rotate/finalize")               // promote the candidate
      // call("DELETE", "/hmac/keys/rotate")                   // or discard the candidate
      log.Println(call("GET", "/hmac/keys"))                   // metadata only
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  conn = Faraday.new
  conn.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'

  puts conn.post('https://async.api.zenrows.com/v1/hmac/keys/rotate').body   # store the secret now
  conn.post('https://async.api.zenrows.com/v1/hmac/keys/rotate/finalize')    # promote the candidate
  # conn.delete('https://async.api.zenrows.com/v1/hmac/keys/rotate')         # or discard the candidate
  puts conn.get('https://async.api.zenrows.com/v1/hmac/keys').body           # metadata only
  ```

  ```bash cURL theme={"dark"}
  # Create the active key or stage a candidate (the secret is returned once):
  curl -X POST "https://async.api.zenrows.com/v1/hmac/keys/rotate" -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  # Promote the candidate to active and retire the old key:
  curl -X POST "https://async.api.zenrows.com/v1/hmac/keys/rotate/finalize" -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  # Or discard the staged candidate:
  curl -X DELETE "https://async.api.zenrows.com/v1/hmac/keys/rotate" -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  # List key metadata (never returns secrets):
  curl "https://async.api.zenrows.com/v1/hmac/keys" -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

### Verify signed webhooks

When `signature: true`, each delivery includes an `X-Signature` header in the form `t=<unix>,v1=<hex>,kid=<active>`, where `t` is the timestamp, `v1` is the signature, and `kid` identifies the key used.

To verify a delivery:

1. Look up the base64 secret that matches the delivery's `kid`.
2. Compute `HMAC-SHA256(secret, "<t>.<raw_body>")`, signing the timestamp, a literal dot, and the raw request body.
3. Compare the result to `v1`. If they match, the delivery is authentic.
   Always sign the raw request body, never a re-serialized version, or the signature will not match. If `signature` is `false`, no `X-Signature` header is sent, so authenticate the callback another way, such as a secret path in the URL. Either way, deduplicate on the `X-ZenRows-Event-Id` header, since the same delivery can arrive more than once.

The examples below compute the expected signature so you can compare it to `v1` in your receiver.

<CodeGroup>
  ```python Python theme={"dark"}
  import hmac, hashlib, base64

  def verify(raw_body: bytes, t: str, v1: str, secret_b64: str) -> bool:
      secret = base64.b64decode(secret_b64)
      expected = hmac.new(secret, f"{t}.".encode() + raw_body, hashlib.sha256).hexdigest()
      return hmac.compare_digest(expected, v1)
  ```

  ```javascript Node.js theme={"dark"}
  import crypto from "node:crypto";

  // SECRETS maps kid -> the base64 secret you captured at /hmac/keys/rotate.
  function verifyWebhook(rawBody, signatureHeader, SECRETS) {
    const parts = Object.fromEntries(
      signatureHeader.split(",").map((kv) => kv.split("="))
    ); // { t, v1, kid }
    const secret = SECRETS[parts.kid];
    if (!secret) return false;

    const expected = crypto
      .createHmac("sha256", Buffer.from(secret, "base64"))
      .update(`${parts.t}.${rawBody}`)
      .digest("hex");

    // constant-time compare
    const a = Buffer.from(expected), b = Buffer.from(parts.v1);
    return a.length === b.length && crypto.timingSafeEqual(a, b);
  }

  // Express-style receiver (capture the raw body!):
  // app.post("/zr", express.raw({ type: "*/*" }), (req, res) => {
  //   const eventId = req.header("X-ZenRows-Event-Id");      // dedup on this
  //   const sig = req.header("X-Signature");                 // absent => unsigned
  //   if (sig && !verifyWebhook(req.body.toString(), sig, SECRETS)) return res.sendStatus(401);
  //   if (alreadyHandled(eventId)) return res.sendStatus(200);
  //   handle(JSON.parse(req.body.toString()));
  //   res.sendStatus(200);
  // });
  ```

  ```java Java theme={"dark"}
  import javax.crypto.Mac;
  import javax.crypto.spec.SecretKeySpec;
  import java.nio.charset.StandardCharsets;
  import java.security.MessageDigest;
  import java.util.Base64;

  public class VerifyWebhook {
      static boolean verify(String rawBody, String t, String v1, String secretB64) throws Exception {
          byte[] secret = Base64.getDecoder().decode(secretB64);
          Mac mac = Mac.getInstance("HmacSHA256");
          mac.init(new SecretKeySpec(secret, "HmacSHA256"));
          byte[] digest = mac.doFinal((t + "." + rawBody).getBytes(StandardCharsets.UTF_8));
          StringBuilder hex = new StringBuilder();
          for (byte x : digest) hex.append(String.format("%02x", x));
          return MessageDigest.isEqual(
              hex.toString().getBytes(StandardCharsets.UTF_8), v1.getBytes(StandardCharsets.UTF_8));
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  function verify($rawBody, $t, $v1, $secretB64) {
      $secret = base64_decode($secretB64);
      $expected = hash_hmac('sha256', $t . '.' . $rawBody, $secret);
      return hash_equals($expected, $v1);
  }
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "crypto/hmac"
      "crypto/sha256"
      "encoding/base64"
      "encoding/hex"
  )

  func verify(rawBody, t, v1, secretB64 string) bool {
      secret, _ := base64.StdEncoding.DecodeString(secretB64)
      mac := hmac.New(sha256.New, secret)
      mac.Write([]byte(t + "." + rawBody))
      expected := hex.EncodeToString(mac.Sum(nil))
      return hmac.Equal([]byte(expected), []byte(v1))
  }
  ```

  ```ruby Ruby theme={"dark"}
  require 'openssl'
  require 'base64'

  def verify(raw_body, t, v1, secret_b64)
      secret = Base64.decode64(secret_b64)
      expected = OpenSSL::HMAC.hexdigest('SHA256', secret, "#{t}.#{raw_body}")
      OpenSSL.fixed_length_secure_compare(expected, v1)
  rescue StandardError
      expected == v1
  end
  ```

  ```bash cURL theme={"dark"}
  # Verification happens in your receiver, not in a request. With openssl:
  # T and BODY come from the delivery; SECRET_B64 is your base64 key.
  key_hex=$(printf '%s' "$SECRET_B64" | base64 -d | xxd -p -c 256)
  printf '%s' "${T}.${BODY}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$key_hex"
  # Compare the printed hex digest against the v1 value from X-Signature.
  ```
</CodeGroup>

## 5. Reruns and retrying failures

`POST /jobs/{job_id}/rerun` replays a job as a new run. Add `?status=failed` to re-run only the failures (append `,pending` to also include tasks that never started).

Already-successful tasks carry over, so you only re-scrape, and only pay for, what failed. The response reports `retried_tasks` and `inherited_tasks`.

<Warning>
  Partial reruns (filtering by `?status=failed` or `?status=failed,pending`) must be triggered within **10 days** of the original run completing. After that, the API returns `409 partial_rerun_window_expired`; start a full rerun instead, which replays every task regardless of previous status.
</Warning>

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}

  # Full rerun:
  requests.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun', headers=headers)

  # Retry only the failures:
  response = requests.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun',
                          headers=headers, params={'status': 'failed'})
  data = response.json()
  print('new run', data['latest_run']['run_id'], 'retried', data['retried_tasks'], 'inherited', data['inherited_tasks'])
  ```

  ```javascript Node.js theme={"dark"}
  // Full rerun: replay every task as a new run:
  const full = await api("POST", "/jobs/JOB_ID/rerun");

  // Retry only failures (add ",pending" to also re-run tasks that never started):
  const retry = await api("POST", "/jobs/JOB_ID/rerun", { query: { status: "failed" } });
  console.log("new run", retry.latest_run.run_id, "retried", retry.retried_tasks, "inherited", retry.inherited_tasks);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class Rerun {
      public static void main(final String... args) throws Exception {
          String key = "YOUR_ZENROWS_API_KEY";

          // Full rerun:
          Request.post("https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun")
                  .addHeader("X-API-Key", key).execute().discardContent();

          // Retry only the failures:
          String response = Request.post("https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun?status=failed")
                  .addHeader("X-API-Key", key).execute().returnContent().asString();
          System.out.println(response); // includes retried_tasks and inherited_tasks
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $key = 'YOUR_ZENROWS_API_KEY';

  // Retry only the failures (drop the query string for a full rerun):
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun?status=failed');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ["X-API-Key: $key"]);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL; // includes retried_tasks and inherited_tasks
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      // Retry only the failures (drop the query string for a full rerun).
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun?status=failed", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body)) // includes retried_tasks and inherited_tasks
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  conn = Faraday.new
  conn.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'

  # Full rerun:
  conn.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun')

  # Retry only the failures:
  res = conn.post('https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun') do |req|
      req.params['status'] = 'failed'
  end
  print(res.body) # includes retried_tasks and inherited_tasks
  ```

  ```bash cURL theme={"dark"}
  # Full rerun:
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  # Retry only the failures:
  curl -X POST "https://async.api.zenrows.com/v1/jobs/JOB_ID/rerun?status=failed" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

## 6. Cost

Estimate spend before you submit, and read actual spend after a run.

### Estimate cost

Pricing is per successful request by tier, and the rate card is small and stable, so you can estimate cost on the client with no API call.

* 1 credit (basic request)
* 5 credits (JavaScript rendering)
* 10 credits (premium proxy)
* 25 credits (both)
* a 1 to 25 credit range for `mode=auto`

Sum the tier across your tasks. The estimate assumes each URL succeeds once, so it is a single number unless a task uses `mode=auto`.

<Note>The estimate assumes each URL succeeds once, so it is a single number unless a task uses `mode=auto`. For more details on credit costs, see the [Pricing Page](/first-steps/pricing#cost-multipliers-for-fetch).</Note>

<CodeGroup>
  ```python Python theme={"dark"}
  def task_cost(params):
      if str(params.get('mode')) == 'auto':
          return (1, 25)
      js = str(params.get('js_render')) == 'true'
      proxy = str(params.get('premium_proxy')) == 'true'
      c = 25 if js and proxy else 10 if proxy else 5 if js else 1
      return (c, c)

  def estimate(body):
      job = body.get('zenrows_params', {})
      lo = hi = 0
      for t in body.get('tasks', []):
          a, b = task_cost({**job, **t.get('zenrows_params', {})})
          lo += a
          hi += b
      return {'tasks': len(body.get('tasks', [])), 'min': lo, 'max': hi, 'exact': lo == hi}
  ```

  ```javascript Node.js theme={"dark"}
  const TIERS = { base: 1, js: 5, proxy: 10, both: 25 }; // credits per successful request

  function taskCost(params = {}) {
    const v = (x) => String(x) === "true";
    if (String(params.mode) === "auto") return { min: 1, max: 25 }; // dynamic
    const js = v(params.js_render), proxy = v(params.premium_proxy);
    const c = js && proxy ? TIERS.both : proxy ? TIERS.proxy : js ? TIERS.js : TIERS.base;
    return { min: c, max: c };
  }

  function estimate(submitBody) {
    const job = submitBody.zenrows_params ?? {};
    let min = 0, max = 0;
    for (const t of submitBody.tasks ?? []) {
      const { min: a, max: b } = taskCost({ ...job, ...(t.zenrows_params ?? {}) });
      min += a; max += b;
    }
    return { tasks: (submitBody.tasks ?? []).length, min, max, exact: min === max };
  }
  ```

  ```java Java theme={"dark"}
  import java.util.List;
  import java.util.Map;

  public class Estimate {
      static int[] taskCost(Map<String, String> p) {
          if ("auto".equals(p.get("mode"))) return new int[]{1, 25};
          boolean js = "true".equals(p.get("js_render"));
          boolean proxy = "true".equals(p.get("premium_proxy"));
          int c = js && proxy ? 25 : proxy ? 10 : js ? 5 : 1;
          return new int[]{c, c};
      }

      // Merge job-level and per-task params before calling taskCost for each task,
      // then sum the min and max across all tasks.
      static int[] estimate(List<Map<String, String>> mergedTaskParams) {
          int min = 0, max = 0;
          for (Map<String, String> p : mergedTaskParams) {
              int[] c = taskCost(p);
              min += c[0];
              max += c[1];
          }
          return new int[]{min, max};
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  function task_cost($params) {
      if (($params['mode'] ?? '') === 'auto') return [1, 25];
      $js = ($params['js_render'] ?? '') === 'true';
      $proxy = ($params['premium_proxy'] ?? '') === 'true';
      $c = $js && $proxy ? 25 : ($proxy ? 10 : ($js ? 5 : 1));
      return [$c, $c];
  }

  function estimate($body) {
      $job = $body['zenrows_params'] ?? [];
      $min = 0; $max = 0;
      foreach (($body['tasks'] ?? []) as $t) {
          [$a, $b] = task_cost(array_merge($job, $t['zenrows_params'] ?? []));
          $min += $a; $max += $b;
      }
      return ['tasks' => count($body['tasks'] ?? []), 'min' => $min, 'max' => $max, 'exact' => $min === $max];
  }
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  func taskCost(p map[string]string) (int, int) {
      if p["mode"] == "auto" {
          return 1, 25
      }
      js := p["js_render"] == "true"
      proxy := p["premium_proxy"] == "true"
      switch {
      case js && proxy:
          return 25, 25
      case proxy:
          return 10, 10
      case js:
          return 5, 5
      default:
          return 1, 1
      }
  }

  // Merge job-level and per-task params before calling taskCost for each task,
  // then sum the min and max across all tasks.
  func estimate(mergedTaskParams []map[string]string) (int, int) {
      min, max := 0, 0
      for _, p := range mergedTaskParams {
          lo, hi := taskCost(p)
          min += lo
          max += hi
      }
      return min, max
  }
  ```

  ```ruby Ruby theme={"dark"}
  def task_cost(params)
      return [1, 25] if params['mode'].to_s == 'auto'
      js = params['js_render'].to_s == 'true'
      proxy = params['premium_proxy'].to_s == 'true'
      c = js && proxy ? 25 : proxy ? 10 : js ? 5 : 1
      [c, c]
  end

  def estimate(body)
      job = body['zenrows_params'] || {}
      min = 0; max = 0
      (body['tasks'] || []).each do |t|
          a, b = task_cost(job.merge(t['zenrows_params'] || {}))
          min += a; max += b
      end
      { tasks: (body['tasks'] || []).length, min: min, max: max, exact: min == max }
  end
  ```

  ```bash cURL theme={"dark"}
  # Cost estimation is client-side arithmetic on the rate card, not an API call.
  # Tiers (credits per successful request): base=1, js_render=5, premium_proxy=10, both=25, mode=auto=1..25.
  # Sum the tier for each task. Example with jq over a submit body in body.json:
  jq '[.tasks[] | (.zenrows_params // {}) as $p
        | (if ($p.mode? == "auto") then 25
          elif ($p.js_render? == "true" and $p.premium_proxy? == "true") then 25
          elif ($p.premium_proxy? == "true") then 10
          elif ($p.js_render? == "true") then 5
          else 1 end)] | add' body.json
  ```
</CodeGroup>

### Check actual cost

Once a run finishes, you can read the credits it actually consumed at two levels:

* **Run total:** the `spend` object on `latest_run.stats`, read from the `GET /jobs/{id}` response shown below.
* **Per task:** the `spend` object on each result row, where `spend.total.credits` is the cost across all attempts and `spend.last_attempt.credits` is the cost of the most recent attempt. Read these while paging results, as shown in [Download content](#download-content).

Only successful (status 200, 404 and 410) requests cost credits, so a run with failures costs less than its estimate. Treat this figure as an indicative signal. Your Zenrows account statement remains the authoritative record for billing.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  job = requests.get('https://async.api.zenrows.com/v1/jobs/JOB_ID',
                    headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'}).json()
  spend = job['latest_run']['stats'].get('spend')
  if spend:
      print('credits:', spend['credits'], 'cost:', spend['cost'])
  ```

  ```javascript Node.js theme={"dark"}
  const job = await api("GET", "/jobs/JOB_ID");
  const spend = job.latest_run.stats.spend; // { credits, cost }, absent until there's spend
  if (spend) console.log("credits:", spend.credits, "cost:", spend.cost);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class ActualCost {
      public static void main(final String... args) throws Exception {
          String response = Request.get("https://async.api.zenrows.com/v1/jobs/JOB_ID")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .execute().returnContent().asString();
          System.out.println(response); // read latest_run.stats.spend from this JSON
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs/JOB_ID');
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $job = json_decode(curl_exec($ch), true);
  curl_close($ch);
  $spend = $job['latest_run']['stats']['spend'] ?? null;
  if ($spend) echo "credits: {$spend['credits']} cost: {$spend['cost']}" . PHP_EOL;
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      req, err := http.NewRequest("GET", "https://async.api.zenrows.com/v1/jobs/JOB_ID", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body)) // read latest_run.stats.spend from this JSON
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  conn = Faraday.new
  job = JSON.parse(conn.get('https://async.api.zenrows.com/v1/jobs/JOB_ID') { |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
  }.body)
  spend = job.dig('latest_run', 'stats', 'spend')
  puts "credits: #{spend['credits']} cost: #{spend['cost']}" if spend
  ```

  ```bash cURL theme={"dark"}
  # Read latest_run.stats.spend from the job response:
  curl "https://async.api.zenrows.com/v1/jobs/JOB_ID" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

## 7. Reference

Behaviors and error details that apply across the API.

### Idempotency

Send an `Idempotency-Key` header on a submit to guard against accidentally creating duplicate jobs. Each key can be used once: the first request with a given key creates the job, and any later request that reuses the same key returns `409 idempotency_key_conflict`, whether or not the body is identical. The API does not store or replay the original response.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json',
            'Idempotency-Key': 'nightly-2026-06-08'}
  payload = {'type': 'regular', 'status': 'closed',
            'tasks': [{'url': 'https://www.scrapingcourse.com/ecommerce/'}]}
  response = requests.post('https://async.api.zenrows.com/v1/jobs', headers=headers, data=json.dumps(payload))
  print(response.status_code, response.json())
  ```

  ```javascript Node.js theme={"dark"}
  const job = await api("POST", "/jobs", {
    headers: { "Idempotency-Key": "nightly-2026-06-08" },
    body: { type: "regular", status: "closed", tasks: [{ url: "https://www.scrapingcourse.com/ecommerce/" }] },
  });
  console.log(job.job_id);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;

  public class Idempotent {
      public static void main(final String... args) throws Exception {
          String body = "{\"type\":\"regular\",\"status\":\"closed\",\"tasks\":[{\"url\":\"https://www.scrapingcourse.com/ecommerce/\"}]}";
          String response = Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .addHeader("Idempotency-Key", "nightly-2026-06-08")
                  .bodyString(body, ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
      'type' => 'regular', 'status' => 'closed',
      'tasks' => [['url' => 'https://www.scrapingcourse.com/ecommerce/']],
  ]));
  curl_setopt($ch, CURLOPT_HTTPHEADER, [
      'X-API-Key: YOUR_ZENROWS_API_KEY',
      'Idempotency-Key: nightly-2026-06-08',
      'Content-Type: application/json',
  ]);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  echo curl_exec($ch) . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "io"
      "log"
      "net/http"
  )

  func main() {
      payload := []byte(`{"type":"regular","status":"closed","tasks":[{"url":"https://www.scrapingcourse.com/ecommerce/"}]}`)
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs", bytes.NewBuffer(payload))
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      req.Header.Set("Idempotency-Key", "nightly-2026-06-08")
      req.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(resp.StatusCode, string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  conn = Faraday.new
  res = conn.post('https://async.api.zenrows.com/v1/jobs') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.headers['Idempotency-Key'] = 'nightly-2026-06-08'
      req.headers['Content-Type'] = 'application/json'
      req.body = { type: 'regular', status: 'closed',
                  tasks: [{ url: 'https://www.scrapingcourse.com/ecommerce/' }] }.to_json
  end
  print(res.status)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
    -H "Idempotency-Key: nightly-2026-06-08" \
    -H "Content-Type: application/json" \
    -d '{ "type": "regular", "status": "closed", "tasks": [{ "url": "https://www.scrapingcourse.com/ecommerce/" }] }'
  ```
</CodeGroup>

<Warning>
  An idempotency key is not a safe-retry token, because reusing it always returns `409`. If a submit fails midway, such as a dropped connection or a `503`, do not resend with the same key. First check whether the job was created by listing your recent jobs, then either use its `job_id` or submit again with a new key. A late `503` can leave a visible job with a `stopped` run, which is safe to `DELETE`.
</Warning>

### Error handling reference

Errors use RFC 7807 problem JSON with a stable `code` you can branch on. Task-level failures are not HTTP errors; they appear as `failed` rows in the results, each with its own `error` object. A successful `POST /jobs` returns `201 Created` (synchronous) or `202 Accepted` (asynchronous, for 10,000+ tasks or a CSV input); both carry the same response body.

| HTTP | `code` | Meaning |
| - | - | - |
| 400 | `invalid_argument` | Validation failed; `invalid_tasks[]` lists `{index, reason}`. |
| 401 | `unauthenticated` | Missing or invalid API key. |
| 402 | `payment_required` / `api_key_cap_reached` | `payment_required`: no credits available. `api_key_cap_reached`: the API key reached one of its [credit caps](#api-key-credit-caps-and-batch); `detail` names the cap and when it resets. |
| 404 | `not_found` | Job, run, or task missing or not owned by you. |
| 409 | `conflict` / `run_not_terminal` / `idempotency_key_conflict` | State conflict; see `detail`. |
| 422 | (on `/content`) | Task failed; the body is the stored error. |
| 429 | `quota_exceeded` | Account limit reached, such as the maximum of 3 concurrent active jobs. |
| 503 | `internal` | Transient; safe to retry with a backoff. |

On a non-2xx response, read the `code` and, for a `400`, the `invalid_tasks` array from the problem body.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  r = requests.post('https://async.api.zenrows.com/v1/jobs',
                    headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json'},
                    json={'type': 'regular', 'status': 'closed', 'tasks': [{'url': 'not-a-url'}]})
  if not r.ok:
      problem = r.json()  # RFC 7807: { type, title, status, detail, code, invalid_tasks? }
      print(r.status_code, problem.get('code'), problem.get('invalid_tasks'))
  ```

  ```javascript Node.js theme={"dark"}
  // The api() helper throws an object carrying the structured problem details.
  try {
    await api("POST", "/jobs", { body: { type: "regular", status: "closed", tasks: [{ url: "not-a-url" }] } });
  } catch (e) {
    console.error(e.status, e.code, e.problem?.invalid_tasks);
  }
  ```

  ```java Java theme={"dark"}
  import com.fasterxml.jackson.databind.JsonNode;
  import com.fasterxml.jackson.databind.ObjectMapper;
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ClassicHttpResponse;
  import org.apache.hc.core5.http.ContentType;
  import org.apache.hc.core5.http.io.entity.EntityUtils;

  public class ErrorHandling {
      static final ObjectMapper M = new ObjectMapper();

      public static void main(final String... args) throws Exception {
          String body = "{\"type\":\"regular\",\"status\":\"closed\",\"tasks\":[{\"url\":\"not-a-url\"}]}";
          ClassicHttpResponse resp = (ClassicHttpResponse) Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .bodyString(body, ContentType.APPLICATION_JSON)
                  .execute().returnResponse();
          if (resp.getCode() >= 400) {
              JsonNode problem = M.readTree(EntityUtils.toString(resp.getEntity())); // RFC 7807
              System.out.println(resp.getCode() + " " + problem.path("code").asText() + " " + problem.path("invalid_tasks"));
          }
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode(['type' => 'regular', 'status' => 'closed', 'tasks' => [['url' => 'not-a-url']]]));
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY', 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $res = curl_exec($ch);
  $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
  curl_close($ch);
  if ($status >= 400) {
      $problem = json_decode($res, true); // RFC 7807
      echo $status, ' ', $problem['code'] ?? '', PHP_EOL;
      print_r($problem['invalid_tasks'] ?? []);
  }
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "encoding/json"
      "io"
      "log"
      "net/http"
  )

  func main() {
      payload := []byte(`{"type":"regular","status":"closed","tasks":[{"url":"not-a-url"}]}`)
      req, _ := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs", bytes.NewBuffer(payload))
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      req.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      if resp.StatusCode >= 400 {
          var problem struct {
              Code         string `json:"code"`
              InvalidTasks []any  `json:"invalid_tasks"`
          }
          json.Unmarshal(body, &problem) // RFC 7807
          log.Println(resp.StatusCode, problem.Code, problem.InvalidTasks)
      }
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  res = Faraday.post('https://async.api.zenrows.com/v1/jobs') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.headers['Content-Type'] = 'application/json'
      req.body = { type: 'regular', status: 'closed', tasks: [{ url: 'not-a-url' }] }.to_json
  end
  unless res.success?
      problem = JSON.parse(res.body) # RFC 7807
      puts "#{res.status} #{problem['code']}"
      p problem['invalid_tasks']
  end
  ```

  ```bash cURL theme={"dark"}
  # -i shows the status line; the body is the RFC 7807 problem with code and invalid_tasks.
  curl -i -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" -H "Content-Type: application/json" \
    -d '{ "type": "regular", "status": "closed", "tasks": [{ "url": "not-a-url" }] }'
  ```
</CodeGroup>

## Troubleshooting

Common issues you may hit while integrating. See the full [Troubleshooting](/batch/troubleshooting) page for the complete list.

* **`401 unauthenticated`:** send your key in the `X-API-Key` header (never in the URL); a `401` means the key is missing or invalid.
* **`400 invalid_argument`:** the body failed validation. Read the `invalid_tasks` array to see which URL failed and why.
* **`429 quota_exceeded`:** you have 3 jobs running already. Wait for one to finish, since 3 concurrent jobs is the limit.
* **`503` when opening a job:** if `POST /jobs` with `status: "open"` returns a `503`, retry after a moment or contact Zenrows support.
* **Download link returns an error:** each `result_url` is valid for 2 hours. Re-list the results to get a fresh link rather than reusing a stored one.
* **Webhook never arrives:** use an HTTPS receiver, test it with `POST /webhook/test`, and deduplicate on the `X-ZenRows-Event-Id` header, since deliveries are at-least-once.

## FAQ (Frequently Asked Questions)

Quick answers for developers. See the full [FAQ](/batch/faq) for more.

<Accordion title="What's the base URL, and do I need a separate key?">
  All requests go to `https://async.api.zenrows.com/v1`, authenticated with the `X-API-Key` header. Batch uses the same API key as Fetch.
</Accordion>

<Accordion title="Which languages are supported?">
  Any language with an HTTP client. The examples in this guide cover Python, Node.js, Java, PHP, Go, Ruby, and cURL, and the same REST calls work anywhere.
</Accordion>

<Accordion title="How many URLs and jobs can I run?">
  Up to 100,000 URLs per job submit, 10,000 per add-tasks call on an open job, and 3 active jobs concurrently. CSV uploads support 50 MB / 100,000 rows.
</Accordion>

<Accordion title="Is the idempotency key safe to retry with?">
  No. Each key can be used once, and reusing it always returns `409 idempotency_key_conflict`. If a submit fails midway, check whether the job was created before resending with a new key.
</Accordion>

<Accordion title="How am I billed?">
  In credits, per successful request, at the same rates as Fetch. Failed requests are never charged.
</Accordion>


# FAQ (Frequently Asked Questions)
Source: https://docs.zenrows.com/batch/faq

Frequently asked questions about the Zenrows Batch: access, pricing, limits, scraping options, results, scheduling, and how it relates to Fetch.

Quick answers to the questions teams ask most when they start with Batch. For step-by-step code, see the [developer guide](/batch/developer-guide-restapi). For error-specific help, see [Troubleshooting](/batch/troubleshooting).

## Getting Started

<AccordionGroup>
  <Accordion title="Is Batch available to everyone?">
    Yes. Batch is available to every Zenrows account. You can start submitting jobs right away with your existing API key, with no extra setup.
  </Accordion>

  <Accordion title="Do I need a separate API key?">
    No. Batch uses the **same API key as your Fetch**. There's nothing extra to generate.
  </Accordion>

  <Accordion title="What's the base URL?">
    All REST requests go to `https://async.api.zenrows.com/v1`. Send your key in the `X-API-Key` header on every request.
  </Accordion>

  <Accordion title="How do I integrate?">
    Batch is a REST API, so you can call it from any language with an HTTP client. The [developer guide](/batch/developer-guide-restapi) shows a dependency-free pattern using Node's `fetch` that translates directly to Python, Go, PHP, Java, and more.
  </Accordion>
</AccordionGroup>

## How It Works

<AccordionGroup>
  <Accordion title="What's the difference between a job, a run, and a task?">
    A **job** is the template: your URLs and configuration. A **run** is one execution of that job, where each rerun or scheduled fire creates a new run. A **task** is a single URL within a run, with its own status and result.
  </Accordion>

  <Accordion title="What's the difference between a rerun and retrying failures?">
    A **rerun** replays every task in the job as a fresh run. **Retry failed** re-executes only the tasks that failed (and optionally those that never started), carrying over the successful results so you only pay to re-scrape what failed. Both create a new run on the same job.
  </Accordion>

  <Accordion title="How do I know when a job is done?">
    Two ways: attach a **webhook** and the API sends a `run.completed` event when the run completes (or `run.failed` if it fails), or **poll** the job until `latest_run.status` is terminal (`completed`, `failed`, `stopped`, or `deleted`) and read the `stats` counters (`completed === successful + failed`). Webhooks are a convenience; the run state remains the source of truth.
  </Accordion>

  <Accordion title="Can I pause or stop a running job?">
    Yes. `POST /jobs/{id}/pause` reversibly suspends the current run and `POST /jobs/{id}/resume` continues it, while `POST /jobs/{id}/stop` (alias `POST /jobs/{id}/cancel`) ends it for good. These act on the job's current run, which is different from pausing a scheduled job's recurring schedule. See [Control the current run](/batch/developer-guide-restapi#control-the-current-run).
  </Accordion>

  <Accordion title="My submit returned 202 instead of 201. Is that an error?">
    No. `POST /jobs` returns `201 Created` for submissions under 10,000 tasks and `202 Accepted` for larger ones (10,000+ tasks or a CSV input). The response body is identical and you track completion the same way, so treat both the same. For a large `202`, `GET /jobs/{id}/results` may return partial pages for a few seconds while the task list finishes writing, but `stats.total` is correct from the start.
  </Accordion>

  <Accordion title="Can I submit URLs as I go, instead of all at once?">
    Yes. Open a job, add tasks in batches over time (up to 10,000 per add-tasks call), then close it with `last_batch: true`. See [open jobs](/batch/developer-guide-restapi#open-jobs-adding-tasks-in-batches).
  </Accordion>

  <Accordion title="How many jobs can I run at the same time?">
    An account can have **3 active jobs concurrently**. Submitting a 4th returns `429 quota_exceeded`. Wait for a job to finish before submitting the next, or contact Zenrows if you need a higher limit.
  </Accordion>
</AccordionGroup>

## Scraping Options

<AccordionGroup>
  <Accordion title="What scraping options can I use?">
    The same `zenrows_params` as Fetch, including `js_render`, `premium_proxy`, `proxy_country`, `wait_for`, `wait`, `css_extractor`, `extract`, `json_response`, `custom_headers`, and more. Set them once on the job, and override any of them per URL.
  </Accordion>

  <Accordion title="Can I use different options for different URLs in the same job?">
    Yes. Options set on the job apply to every URL; options set on an individual URL override the job-level ones for that URL. This lets you configure the common case once and tune the exceptions (for example, a different `proxy_country` for a handful of URLs).
  </Accordion>

  <Accordion title="Can one job include URLs from different websites?">
    Yes. A job is just a list of URLs plus the options to apply to them, so you can mix targets in a single job and use per-URL overrides where a specific site needs different handling.
  </Accordion>

  <Accordion title="What output formats are available?">
    Raw HTML, parsed **JSON**, **Markdown**, **plain text**, and **PDF**. The API derives the output format from your options (for example `response_type: markdown`) and records it on each result.
  </Accordion>
</AccordionGroup>

## Results and Limits

<AccordionGroup>
  <Accordion title="How long are result download links valid?">
    Each result's `result_url` is a presigned link valid for **2 hours**. If it expires, re-list the results to get a fresh one rather than storing it. You can also stream a task's content directly, or export an entire run as a ZIP.
  </Accordion>

  <Accordion title="How long are results stored, and do I need to manage storage?">
    Results and their download links are retained for **14 days** after a run finishes, then removed automatically. You don't need to delete anything to stay within storage limits. If you want to remove results sooner, you can delete a single run's results or an entire job yourself. See [Delete a job or a run](/batch/developer-guide-restapi#delete-a-job-or-a-run).
  </Accordion>

  <Accordion title="Can I delete a job or its results?">
    Yes. Use `DELETE /jobs/{id}/runs/{run_id}` to delete one run's stored results while keeping the job, or `DELETE /jobs/{id}` to delete the whole job (all runs and results). Stopping a run only ends it and keeps the results, so use delete when you want the data removed. See [Delete a job or a run](/batch/developer-guide-restapi#delete-a-job-or-a-run).
  </Accordion>

  <Accordion title="How many URLs can a job handle?">
    Up to **100,000 URLs per job submit**. On an open job, each add-tasks call accepts up to **10,000 URLs**, and CSV uploads support up to **50 MB / 100,000 rows**. Results page at \~1,000 per page.
  </Accordion>

  <Accordion title="How do I match results back to my own records?">
    Attach an `external_id` to each URL. The API returns it unchanged on every result, so you can correlate results to your records without tracking the API's task IDs. It doesn't need to be unique.
  </Accordion>

  <Accordion title="Can I download all results at once?">
    Yes. Start a bulk export to bundle a run's result bodies into a single ZIP. Exports expire 12 hours after creation, and a run whose combined results exceed 100 GiB can't be exported as one archive. Page through the results instead.
  </Accordion>
</AccordionGroup>

## Scheduling

<AccordionGroup>
  <Accordion title="Can I run a job on a recurring schedule?">
    Yes. Schedule a job as **one-shot** (a single future time), **interval** (every N minutes, hours, or days, minimum 60 minutes between runs), or **calendar** (specific times on specific days). Schedules are timezone-aware using IANA names. You can pause and resume any time without losing the schedule.
  </Accordion>

  <Accordion title="Why was my scheduled time rejected?">
    Times of day must be **full hours**: use `"09:00"`, not `"09:30"`. Also provide exactly one schedule type (`at`, `rate`, or `calendar`) and a valid IANA timezone for `at` and `calendar`.
  </Accordion>
</AccordionGroup>

## Pricing

<AccordionGroup>
  <Accordion title="How is Batch priced?">
    Pricing is **identical to Fetch**: you're billed in **credits, per successful request**. The credit cost depends on the scraping tier each URL uses.

    | Configuration | Credits per successful request |
    | - | - |
    | Base (no special options) | 1 |
    | JavaScript rendering | 5 |
    | Premium proxy | 10 |
    | JavaScript rendering + premium proxy | 25 |
    | Adaptive Stealth Mode (`mode=auto`) | Dynamic, 1–25 |
  </Accordion>

  <Accordion title="Am I charged for failed requests?">
    No. **Only successful requests cost credits.** A run with failures costs less than its up-front estimate.
  </Accordion>

  <Accordion title="Can I estimate the cost before running a job?">
    Yes. Because the rate card is small and stable, you can estimate credits client-side with no API call, then show your users an accurate range up front. After a run finishes, you can also read the actual credits it consumed. See [Estimate cost](/batch/developer-guide-restapi#estimate-cost) and [Check actual cost](/batch/developer-guide-restapi#check-actual-cost).
  </Accordion>

  <Accordion title="How can I keep costs as low as possible?">
    Use **Adaptive Stealth Mode** (`mode=auto`). It starts each request with the cheapest viable setup and escalates to JavaScript rendering or premium proxies only when a site actually needs them, so you pay for the lightest configuration that succeeds.
  </Accordion>
</AccordionGroup>

## Reliability and Security

<AccordionGroup>
  <Accordion title="What happens if a page is temporarily unavailable?">
    The API retries transient failures automatically, so a page that's briefly unavailable still has a strong chance of being fetched. Requests are also paced to stay polite to the sites you scrape.
  </Accordion>

  <Accordion title="What does the idempotency key do, and is it safe to retry a submit?">
    An **idempotency key** guards against accidentally creating duplicate jobs. Each key can be used once: the first submit with a given key creates the job, and any later submit reusing that key returns `409 idempotency_key_conflict`, whether or not the body is identical. The API does not replay the original response. Because of this, an idempotency key is **not** a safe-retry token. If a submit fails midway, don't resend with the same key. First check whether the job was created (list your recent jobs), then reuse its `job_id` or submit again with a new key.
  </Accordion>

  <Accordion title="How do I verify that a webhook really came from Zenrows?">
    Set `signature: true` when you attach the webhook. The API then HMAC-signs each delivery, and your receiver verifies the `X-Signature` header against the raw request body using your organization's active key. See [Manage HMAC keys](/batch/developer-guide-restapi#manage-hmac-keys).
  </Accordion>
</AccordionGroup>

## Didn't Find Your Answer?

Check the [Troubleshooting](/batch/troubleshooting) page for error-specific guidance, or reach out to support with your `job_id`, the request body (API key redacted), and the exact error `code` and `detail`.


# Introduction to Batch
Source: https://docs.zenrows.com/batch/introduction

Scrape large lists of URLs reliably with Zenrows Batch. Submit a job, track its progress, and collect the results without building your own orchestration.

<Note>
  Batch is available to every Zenrows account. You just need an active Zenrows API key; your existing Fetch key works without any extra setup.
</Note>

Zenrows Batch is a managed service that scrapes large lists of URLs for you. You hand it the URLs and the scraping configuration you want, and the API runs the work reliably: it queues each request, retries transient failures, stores every result, and notifies you when the batch finishes.

Whether you scrape a few hundred pages or a few hundred thousand, Batch gives you one consistent way to submit work, track progress, and collect results, without operating the orchestration yourself.

<CardGroup>
  <Card title="Setting Up Batch" icon="rocket" href="/batch/setup">
    Confirm access, find your API key, and submit your first job in minutes.
  </Card>

  <Card title="Developer Guide: REST API" icon="code" href="/batch/developer-guide-restapi">
    Every feature against the REST API, with runnable examples in any language.
  </Card>
</CardGroup>

## Use It from Any Language

Every Batch feature is available over a plain HTTPS REST API, so you can integrate from any language with nothing more than an HTTP client. The [developer guide](/batch/developer-guide-restapi) is a complete, dependency-free `fetch` walkthrough whose patterns apply equally to Python, Go, PHP, Java, or anything else.

## Why Use It

Scraping a single page is easy. Scraping tens of thousands of pages reliably is not: you have to spread the load so you don't overwhelm target sites, retry the requests that fail for transient reasons, store and serve results durably, and know when everything finished and what didn't work.

Batch is the managed layer for exactly this. You integrate once, submit jobs, and it handles the operational concerns for you:

* **Scale without orchestration.** Submit up to 100,000 URLs in a single job and let the API pace the work.
* **Reliable by default.** The API retries transient failures automatically, so a page that's briefly unavailable still gets fetched.
* **Results you can trust.** The API stores every successful page and serves it back through time-limited download links.
* **Know when it's done.** Receive a webhook on completion, or poll for status, whichever fits your pipeline.

## How It Works

1. **Submit a job:** a list of URLs plus the scraping options to apply to them.
2. **The API queues and processes each URL,** calling the Zenrows scraping engine and retrying transient failures automatically.
3. **The API stores results as they complete,** so you can watch progress at any time.
4. **Collect the results when the batch finishes.** Fetch them directly, or receive a webhook.

## Core Concepts

Three terms map cleanly onto how teams work:

| Concept | What it is |
| - | - |
| **Job** | The template: your reusable submission and its configuration (URLs, scraping options, schedule, webhook). |
| **Run** | One execution of a job. A job has one or more runs over its lifetime, since each rerun or scheduled fire creates a new run. |
| **Task** | One URL within a run, with its own status and result. |

A job carries a snapshot of its latest run, so a single request tells you where things stand without extra lookups.

## Key Features

### Flexible submission

Submit up to **100,000 URLs in a single job**. An **open job** accepts URLs in batches over time (up to **10,000 per add-tasks call**) before you close it. For very large lists, **upload a CSV** (up to 50 MB / 100,000 rows) and reference it instead of inlining URLs. Use an optional **idempotency key** to guard against accidental duplicate submissions: each key can be used once, and reusing it returns a conflict rather than creating a second job.

### Full control over scraping

Configure each job, and override per URL, with the scraping options you need:

* **JavaScript rendering** for dynamic pages.
* **Premium proxies** and **geo-targeting** by country.
* **Output formats:** raw HTML, parsed **JSON**, **Markdown**, **plain text**, or **PDF**.
* **Structured extraction** with CSS selectors or automatic parsing, plus custom headers, waits, and more.

<Note>
  Options set on the job apply to every URL; options set on an individual URL override them. Configure the common case once, and tune the exceptions per URL.
</Note>

### Results, your way

* Each successful task yields a **presigned download URL** (a temporary, signed link) valid for 2 hours, or you can stream the content directly.
* **Page through results** efficiently and filter by status: successful, failed, or in-progress.
* Correlate every result back to your own records with a caller-supplied `external_id`.
* **Archive export:** download an entire run's results as a single ZIP archive in one request.
* Each task ends as `successful` or `failed`, and a failed task carries a **structured error** with a `code` and `detail`. These codes don't yet reliably distinguish blocked, not-found, and timeout, so treat them as a hint rather than a definitive reason.

### Completion notifications

* **Webhooks:** the API sends an HTTPS callback when a run completes, optionally **HMAC-signed** so your receiver can verify it genuinely came from Zenrows.
* **Polling:** status and progress counters are always available on demand.

### Reruns and run control

* **Full rerun:** replay an entire job as a fresh run.
* **Retry failed only:** re-execute just the tasks that failed (and optionally those that never started). Successful results carry over, so you only pay to re-scrape what failed.
* **Pause, resume, or stop a run:** reversibly suspend an expensive batch and resume it later, or stop it entirely, without waiting for it to finish.

### Scheduled jobs

Run recurring work on a **one-shot**, **interval** (every N minutes, hours, or days, minimum 60 minutes between runs), or **calendar** (specific times on specific days) schedule. Schedules are **timezone-aware** using IANA names, so daylight-saving transitions stay correct. Pause and resume any time without losing the schedule.

### Cost transparency

Pricing is **per successful request**, and the rate card is small and stable, so you can **estimate the cost before you submit** with no extra round-trip and show your users an accurate credit range up front. Once a job finishes, you can also see the **total credits it actually consumed**, which makes it easy to reconcile the estimate against realized spend.

## Reliability

* **Pages still come back.** The API retries transient failures for you, so a page that's temporarily unavailable still has a strong chance of being fetched.
* **Easy on your targets.** The API paces requests so a large job stays polite to the sites you scrape.
* **Durable results.** The API stores every successful page and serves it back through secure, time-limited download links.
* **Built for big batches.** Submit tens of thousands of URLs and collect the results without managing any of the moving parts.

## Security

* **API-key authentication** on every request. Keys travel in a header, never in the URL.
* **HTTPS everywhere,** including a requirement that webhook receivers use HTTPS.
* **Optional HMAC-signed webhooks** with per-organization keys and phased key rotation, so you can verify that a callback genuinely came from Zenrows.

## Limits

| Limit | Value |
| - | - |
| URLs per job submit | 100,000 |
| URLs per add-tasks call (open jobs) | 10,000 |
| Concurrent active jobs per account | 3 |
| CSV upload | 50 MB / 100,000 rows |
| Results page size | \~1,000 per page |
| Presigned result URL lifetime | 2 hours |
| Result retention | 14 days |
| Bulk export lifetime | 12 hours |

Submitting a job while 3 are already active returns `429 quota_exceeded`. Wait for a job to finish, or contact Zenrows if you need a higher concurrency limit.

## Pricing

Pricing is **identical to the Zenrows Fetch**. You're billed in **credits, per successful request**, and the API never charges for failed requests. The credit cost depends on the scraping tier each URL uses:

| Configuration | Credits per successful request |
| - | - |
| Base (no special options) | 1 |
| JavaScript rendering | 5 |
| Premium proxy | 10 |
| JavaScript rendering + premium proxy | 25 |
| Adaptive Stealth Mode (`mode=auto`) | Dynamic, 1–25 |

**Optimize cost with Adaptive Stealth Mode.** Batch supports Adaptive Stealth Mode (`mode=auto`): it starts each request with the cheapest viable setup and escalates to JavaScript rendering or premium proxies only when a site actually needs them. You pay for the lightest configuration that succeeds rather than over-provisioning every URL.

Because the rate card is small and stable, you can present an accurate cost range to your own users before a job is submitted. See the Estimating Cost section in the developer guide.

## Next Steps

<CardGroup>
  <Card title="Setting Up Batch" icon="rocket" href="/batch/setup">
    Get access, authenticate, and submit your first job.
  </Card>

  <Card title="Developer Guide: REST API" icon="code" href="/batch/developer-guide-restapi">
    Every feature against the REST API, applicable to any language.
  </Card>

  <Card title="Python SDK" icon="brand-python" href="/batch/sdk/python">
    Submit, wait, and download in a few lines with the official Python SDK.
  </Card>

  <Card title="Node.js SDK" icon="brand-nodejs" href="/batch/sdk/nodejs">
    Submit, wait, and download in a few lines with the official Node.js SDK.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/batch/troubleshooting">
    Diagnose common issues fast.
  </Card>

  <Card title="FAQ" icon="circle-question" href="/batch/faq">
    Quick answers on access, pricing, and limits.
  </Card>
</CardGroup>


# Node.js SDK
Source: https://docs.zenrows.com/batch/sdk/nodejs

Submit batch scraping jobs, track them, and collect results with the official Zenrows Node.js SDK. Covers every Batch feature with runnable examples.

<Note>
  Batch is available to every Zenrows account. You need an active Zenrows API key. The `ZenRowsBatchClient` shown here ships under the `zenrows/batch` subpath, separate from the SDK's synchronous scraping client, `ZenRows`.
</Note>

Use this guide to submit batch scraping jobs, track them, and collect results with the official Zenrows Node.js SDK. The client is fully typed, with request and response models generated from the API's OpenAPI spec and exposed as camelCase (`job.jobId`, `run.data.stats.successful`, `row.resultUrl`).

<Tip>
  Prefer raw HTTP, or work in another language? The [REST API guide](/batch/developer-guide-restapi) covers the same use cases against the plain REST API. For the complete reference, see the [SDK repository on GitHub](https://github.com/ZenRows/zenrows-node-sdk).
</Tip>

## Install and Authenticate

While the batch client is being published to npm, install the SDK from GitHub:

```bash theme={"dark"}
npm install "github:ZenRows/zenrows-node-sdk#feature/batch-api/rc"
```

<Note>
  This GitHub install is temporary. Once the batch client is published to npm, `npm install zenrows` will include it.
</Note>

```typescript theme={"dark"}
import { ZenRowsBatchClient } from "zenrows/batch";

const client = new ZenRowsBatchClient("YOUR_ZENROWS_API_KEY");
```

## Quickstart: Submit, Wait, Get Results

`submitRegular` takes one params object: a task source (`urls`, bare strings or per-task objects, or `fileInputId`) plus optional job fields. It returns a `JobRef`. `job.run.wait()` blocks until the current run is terminal and returns its `RunHandle`, whose `.data` snapshot is ready. `results()` is an async stream you can iterate with `for await` or `.forEach`.

```typescript theme={"dark"}
const job = await client.submitRegular({
  urls: [
    "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie",
    "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket",
  ],
  zenrowsParams: { js_render: "true", premium_proxy: "true" },
});

const run = await job.run.wait();
console.log(`${run.data.stats.successful}/${run.data.stats.total} succeeded`);

for await (const row of run.results({ status: "successful" })) {
  console.log(row.externalId ?? row.taskId, "->", row.resultUrl);
}
```

<Tip>
  Need full control over the request body? `client.submitJob({...})` accepts the raw (camelCase) wire shape and returns the same `JobRef`. For very large submissions that return `202 Accepted`, pass `waitForIngest: true` so results pages are complete before you read them.
</Tip>

## Configuring the Scrape

`zenrowsParams` sets scraping options for every URL in the job. The parameter keys themselves stay in their API form (`js_render`, `premium_proxy`, and so on).

```typescript theme={"dark"}
const job = await client.submitRegular({
  urls: ["https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie"],
  zenrowsParams: {
    js_render: "true",
    premium_proxy: "true",
    proxy_country: "us",
    wait_for: ".price",
    response_type: "markdown", // html | markdown | plaintext | pdf
  },
});
```

Other common keys include `js_instructions`, `wait`, `block_resources`, `json_response`, `css_extractor`, `extract`, `outputs`, `original_status`, `allowed_status_codes`, and `session_id`. The `autoparse` parameter is still accepted but [deprecated](/fetch/features/autoparse). Zenrows derives the output format from these and records it on each result.

<Note>
  `custom_headers` takes a map of header name to value. This SDK's params are typed as scalars, so pass the map as a JSON string, for example `{ custom_headers: '{"Referer":"https://www.google.com/"}' }`. It is also how you set the `Content-Type` for a [POST target fetch](/batch/developer-guide-restapi#post-target-fetches). Up to 32 headers, 8 KB serialised.
</Note>

## Per-URL Options and Correlating Results

To set a per-task `externalId`, `metadata`, or `zenrowsParams`, pass objects instead of bare strings. Per-task `zenrowsParams` override the job-level ones on a key clash.

```typescript theme={"dark"}
const job = await client.submitRegular({
  urls: [
    { url: "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short",
      externalId: "order-42", zenrowsParams: { proxy_country: "de" } },
    { url: "https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee",
      externalId: "order-43", metadata: { sku: "ABC-1" } },
  ],
  zenrowsParams: { js_render: "true" },
});
```

<Tip>
  The API returns your `externalId` unchanged on every result, so you can match results back to your own records. It must match `^[A-Za-z0-9._-]+$` and be 128 characters or fewer; the SDK validates this before sending.
</Tip>

## Downloading Content

`results()` gives you metadata plus a presigned `resultUrl` per task, and the SDK downloads bodies straight from that URL with no extra API round-trip. Run these on the current run (`job.run.*`) or any specific run (`client.getRun(jobId, runId)`).

```typescript theme={"dark"}
// One file per task, client-side, tune parallelism:
await client.job(jobId).run.downloadToDir("./out", { concurrency: 8 });

// The whole run as one server-side zip:
await client.job(jobId).run.downloadAllResults("results.zip");

// A single task's body, to disk or memory, inside a loop:
for await (const row of run.results({ status: "successful" })) {
  await run.downloadTaskToDir(row, "./out");
  // or: const { body } = await run.downloadTaskToMemory(row);
}
```

<Note>
  `downloadAllResults` is a server-side zip capped at **100 GiB** per run. For larger runs, or one file per task, use `downloadToDir`, which fetches bodies client-side with no size limit.
</Note>

## Open Jobs: Streaming URLs in Batches

When you don't have every URL up front, open a job with `submitOpen`, add tasks in batches, and close it. The run stays active and completes once it finishes the final batch.

```typescript theme={"dark"}
const job = await client.submitOpen({ zenrowsParams: { js_render: "true" } });

await job.addTasks({ tasks: [{ url: "https://www.scrapingcourse.com/ecommerce/page/1" }] });
await job.addTasks({ tasks: [{ url: "https://www.scrapingcourse.com/ecommerce/page/2" }], lastBatch: true });

const run = await job.run.wait();
```

Setting `lastBatch: true` (or calling `job.close()`) locks the job, and the API rejects further adds. CSV file input is not supported for open jobs; use a closed job for CSV.

## Large Inputs: CSV Upload

For lists too large to inline, upload a CSV once and reference it. `uploadCsv` allocates the slot, PUTs your file to the presigned URL, and returns a `fileInputId` you pass to a submit call.

```typescript theme={"dark"}
const fileInputId = await client.uploadCsv("urls.csv", {
  fields: { url: "URL", externalId: "Customer Ref" }, // column name (header: true) or 0-based index
  header: true,
});
const job = await client.submitRegular({ fileInputId, zenrowsParams: { js_render: "true" } });
```

Each CSV row becomes a task. Only the `url` and optional `externalId` columns are read, and the job-level `zenrowsParams` apply to all of them. Limits: 50 MB / 100,000 rows.

## Listing and Pagination

The SDK offers both single-page calls (`listJobs`, `listRuns`, `listResults`) and auto-paginating async streams that yield loaded handles.

```typescript theme={"dark"}
for await (const job of client.iterJobs({ jobType: "regular", status: "closed" })) {
  console.log(job.jobId, job.data.status);
}

for await (const run of client.iterRuns("01J...", { pageSize: 50 })) {
  console.log(run.runId, run.data.status);
}

for await (const row of client.iterResults("01J...", { status: "failed" })) {
  console.log(row.taskId, row.error?.code);
}
```

Each iterator is an `AsyncStream`, so you can also use `.forEach(...)` instead of `for await`.

## Reruns and Retrying Failures

Each rerun creates a brand-new run on the same job.

```typescript theme={"dark"}
// Re-run every task:
const full = await client.job(jobId).rerun();

// Re-run only the failures (pass { includePending: true } to also pick up tasks that never started).
// Successful results carry over, so you don't pay to re-scrape them:
const run = await client.job(jobId).retryFailed();
await run.wait();
```

## Scheduled Jobs

Submit a recurring job with a typed schedule. The schedule is a discriminated union: `{ every }` interval, `{ at, tz }` one-shot, `{ times, tz }` daily, `{ times, days, tz }` weekly, or `{ times, dates, tz }` monthly. The compiler enforces exactly one shape and a required `tz` (except for `every`).

```typescript theme={"dark"}
const job = await client.submitScheduled({
  schedule: { every: "6h" },
  urls: ["https://www.scrapingcourse.com/ecommerce/"],
});
```

A job carries two facets that match the API's two scopes. `job.schedule.*` controls the recurring schedule, and `job.run.*` controls the **current run**. This disambiguates the API's two distinct "pause" actions:

```typescript theme={"dark"}
await job.schedule.pause();  // stop firing new runs
await job.schedule.resume();
await job.schedule.update({ times: ["09:00"], days: ["mon", "fri"], tz: "Europe/Berlin" });

await job.run.pause();       // reversibly suspend the current run (a different endpoint)
await job.run.resume();
await job.run.stop();        // end the current run
```

For a specific historical run, use `client.run(jobId, runId)`, a `RunRef` with read and download operations only.

## Completion Notifications: Webhooks

Attach a webhook at submit time, and Zenrows POSTs a `run.completed` event to your HTTPS URL when a run completes, or a `run.failed` event when it fails. Set `signature: true` for HMAC-signed deliveries (see [Signed Webhooks](#signed-webhooks-hmac-keys)); it defaults to unsigned. You can also manage it later with `client.putJobWebhook(...)` / `client.deleteJobWebhook(...)`, and probe a receiver with `client.testWebhook(...)`.

```typescript theme={"dark"}
const job = await client.submitRegular({
  urls: ["https://www.scrapingcourse.com/ecommerce/"],
  webhook: { url: "https://your-app.com/webhooks/zenrows", signature: true },
});
```

Deliveries are at-least-once, so deduplicate on the `X-ZenRows-Event-Id` header.

<Tip>
  No webhook receiver? Just poll for completion with `job.run.wait()` or `run.wait()`.
</Tip>

## Bulk Export (ZIP)

Bundle an entire run's result bodies into a single server-side zip: `downloadAllResults` starts the export, waits for it, and streams the file to disk.

```typescript theme={"dark"}
await client.job(jobId).run.downloadAllResults("results.zip");
```

<Note>
  The server-side zip is capped at **100 GiB** per run. For larger runs, or one file per task, use `downloadToDir`, which fetches bodies client-side with no size limit. For manual control, `client.startResultsExport(...)` and `client.waitForExport(...)` expose the two steps separately.
</Note>

## Estimating Cost

Estimate credits **before** submitting with `client.estimateCost`. It takes the same body as `submitJob` and runs client-side with no API call. The tiers are 1 (base), 5 (`js_render`), 10 (`premium_proxy`), 25 (both), and a dynamic 1 to 25 for `mode=auto`.

```typescript theme={"dark"}
const est = await client.estimateCost({
  type: "regular",
  tasks: [{ url: "https://a" }, { url: "https://b" }],
  zenrowsParams: { js_render: "true" },
});
console.log(String(est)); // "10 credits (2 tasks)"
console.log(est.format()); // per-tier breakdown table
```

An estimate assumes each URL succeeds once; it is a single number unless the job uses `mode=auto`, which yields a range.

## Checking Actual Cost

Once a run finishes, read the **actual credits it consumed** from `run.data.stats.spend` (`{ credits, cost }`).

```typescript theme={"dark"}
const run = await client.job(jobId).run.wait();
const spend = run.data.stats.spend;
if (spend) console.log(`this run used ${spend.credits} credits (cost ${spend.cost})`);
```

Only successful requests are charged, so realized spend is at most the estimate. Per-task `spend` is indicative and often absent on successful rows, so rely on the run-level rollup for actuals. Your Zenrows account statement remains the authoritative record for billing.

## Signed Webhooks: HMAC Keys

Signed deliveries (see [Webhooks](#completion-notifications-webhooks)) need an active HMAC key for your organization. Rotation uses two slots, active plus candidate, so you can roll keys without downtime.

```typescript theme={"dark"}
const created = await client.rotateHmacKey(); // the first call creates the active key
console.log(created.kid, created.secret);     // the secret is returned ONCE, so store it now

// Phased rotation:
await client.rotateHmacKey();       // stage a candidate (start accepting both)
await client.finalizeHmacKey();     // promote the candidate to active, retire the old one
// await client.cancelHmacRotation(); // or discard the candidate
await client.listHmacKeys();        // metadata only; never returns secrets
```

Your receiver verifies the `X-Signature: t=<unix>,v1=<hex>,kid=<active>` header by recomputing `HMAC-SHA256(secret, "<t>.<raw_body>")` and comparing the result to `v1`, selecting the secret by `kid`.

## Error Handling

Every non-2xx surfaces as `BatchApiError`. Its `code` property carries the stable code from the RFC 7807 body.

```typescript theme={"dark"}
import { BatchApiError } from "zenrows/batch";

try {
  await client.getJob("does-not-exist");
} catch (err) {
  if (err instanceof BatchApiError && err.code === "not_found") {
    // ...
  } else {
    throw err;
  }
}
```

Task-level failures don't throw. They appear as `failed` rows in `results()`, each with an `error` object you can read through `row.error.code` and `row.error.detail`.

## Timeouts and Retries

Transient failures (`429`, `502`, `503`, `504`, and network errors) are retried automatically with jittered exponential backoff, honoring `Retry-After`. Retries apply only to idempotent requests: `GET` / `PUT` / `DELETE`, plus `POST`s that carry an `idempotencyKey` (submit and rerun). Tune both through the constructor.

```typescript theme={"dark"}
const client = new ZenRowsBatchClient("YOUR_ZENROWS_API_KEY", {
  timeout: 30_000, // per-request, ms
  retries: 3,      // 0 disables
});
```

## Idempotency

Pass `idempotencyKey` to a submit or rerun to guard against accidentally creating duplicate jobs. Each key can be used once: a later call that reuses the same key raises a `409` conflict, whether or not the body is identical, and the API does not store or replay the original result. It is therefore not a safe-retry token, so if a call fails midway, check whether the job was created before retrying with a new key.

```typescript theme={"dark"}
const job = await client.submitRegular({
  urls: ["https://www.scrapingcourse.com/ecommerce/"],
  idempotencyKey: "nightly-2026-06-08",
});
```

The full Batch surface (jobs, runs, tasks, results, content, history, file inputs, and HMAC keys) is reachable through methods on `ZenRowsBatchClient`. See the [SDK repository on GitHub](https://github.com/ZenRows/zenrows-node-sdk).


# Python SDK
Source: https://docs.zenrows.com/batch/sdk/python

Submit batch scraping jobs, track them, and collect results with the official Zenrows Python SDK. Covers every Batch feature with runnable examples.

<Note>
  Batch is available to every Zenrows account. You need an active Zenrows API key. The `ZenRowsBatchClient` shown here is separate from the SDK's synchronous scraping client, `ZenRowsClient`.
</Note>

Use this guide to submit batch scraping jobs, track them, and collect results with the official Zenrows Python SDK. It walks through every Batch feature, and each section runs on its own.

<Tip>
  Prefer raw HTTP, or work in another language? The [REST API guide](/batch/developer-guide-restapi) covers the same use cases against the plain REST API. For the complete method reference, see the [SDK repository on GitHub](https://github.com/ZenRows/zenrows-python-sdk).
</Tip>

## Install and Authenticate

While the batch client is being published to PyPI, install the SDK from GitHub:

```bash theme={"dark"}
pip install "git+https://github.com/ZenRows/zenrows-python-sdk.git@main"
```

<Note>
  This GitHub install is temporary. Once the batch client is published to PyPI, `pip install zenrows` will include it.
</Note>

```python Python theme={"dark"}
import os
from zenrows import ZenRowsBatchClient

client = ZenRowsBatchClient(api_key=os.environ["ZENROWS_API_KEY"])
```

The client also works as a context manager (`with ZenRowsBatchClient(...) as client: ...`) when you want connections to close automatically. By default it targets the production Batch; pass `base_url=...` (or set `ZENROWS_BATCH_BASE_URL`) to point at a different environment.

## Quickstart: Submit, Wait, Get Results

This is the standard end-to-end flow. `submit_regular` returns a `JobRef` (no GET yet), `job.run.wait()` blocks until the current run reaches a **terminal** state and returns a `RunHandle`, and `run.results(...)` iterates the per-task results.

```python Python theme={"dark"}
job = client.submit_regular(
    [
        "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie",
        "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket",
        "https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee",
    ],
    zenrows_params={"js_render": "true", "premium_proxy": "true"},
)

run = job.run.wait(timeout=600.0)   # blocks until the current run is terminal
s = run.stats
print(f"{s.successful}/{s.total} successful, {s.failed} failed")

for task in run.results(status="successful"):
    print(task.external_id or task.task_id, "->", task.result_url)
```

`job.run.wait()` returns a `RunHandle` whose `.data` snapshot is ready, so `run.results(...)` and `run.download_to_dir(...)` work without re-passing IDs.

<Tip>
  Pass `progress=True` to the waiter for a live progress bar on long batches. For very large submissions that return `202 Accepted`, pass `wait_for_ingest=True` to `submit_regular` so results pages are complete before you read them.
</Tip>

## Configuring the Scrape

`zenrows_params` sets scraping options for every URL in the job. Values can be strings, booleans, or integers; the SDK converts them to strings before sending.

```python Python theme={"dark"}
job = client.submit_regular(
    ["https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie"],
    zenrows_params={
        "js_render": "true",          # render JavaScript in a headless browser
        "premium_proxy": "true",      # route through premium residential proxies
        "proxy_country": "us",        # geo-target the request
        "wait_for": ".price",         # wait for this CSS selector before capturing
        "response_type": "markdown",  # html | markdown | plaintext | pdf
    },
)
```

Zenrows derives the job's **output format** from these options: `response_type: markdown` produces Markdown; `extract`, `json_response`, or a CSS extractor produce JSON; otherwise the result is HTML. Zenrows records this format on each result. Other common keys include `js_instructions`, `wait`, `block_resources`, `json_response`, `css_extractor`, `extract`, `outputs`, `original_status`, `allowed_status_codes`, and `session_id`. The `autoparse` parameter is still accepted but [deprecated](/fetch/features/autoparse).

<Note>
  `custom_headers` accepts a dict of header name to value, for example `{"custom_headers": {"Referer": "https://www.google.com/"}}`, and is also how you set the `Content-Type` for a [POST target fetch](/batch/developer-guide-restapi#post-target-fetches). Up to 32 headers, 8 KB serialised.
</Note>

## Per-URL Options and Correlating Results

To set a per-URL `external_id`, `metadata`, or `zenrows_params`, pass dictionaries instead of plain strings. Per-URL `zenrows_params` override the job-level ones when a key appears in both.

```python Python theme={"dark"}
job = client.submit_regular(
    [
        {"url": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short",
         "external_id": "order-42",
         "zenrows_params": {"proxy_country": "de"}},
        {"url": "https://www.scrapingcourse.com/ecommerce/product/atomic-endurance-tee",
         "external_id": "order-43",
         "metadata": {"sku": "ABC-1"}},
    ],
    zenrows_params={"js_render": "true"},   # applies to both unless overridden
)
```

<Tip>
  Zenrows returns your `external_id` unchanged on every result, so you can match results back to your own records without tracking the API's task IDs. It must match `^[A-Za-z0-9._-]+$` and be 128 characters or fewer.
</Tip>

## Downloading Content

`results()` gives you metadata plus a presigned `result_url` per task; the SDK downloads bodies straight from that URL with no extra API round-trip. Four options, on the current run (`job.run.*`) or any specific run (`client.get_run(job_id, run_id=...)`):

| You want | Use | Where it lands |
| - | - | - |
| Every body, one file per task | `run.download_to_dir(dir)` | files on disk |
| Every body, kept in memory | `run.download_to_memory()` | `list[DownloadedResult]` |
| The whole run as one artifact | `run.download_all_results("out.zip")` | one `.zip` |
| One task's body (inside a loop) | `run.download_task_to_file(task, path)` / `run.download_task_to_memory(task)` | file / `bytes` |

```python Python theme={"dark"}
# Bulk to disk, named by external_id, 8 downloads in parallel:
n = run.download_to_dir(
    "./out",
    status="successful",       # default; pass status=None for everything
    use_external_id=True,      # name files <external_id>.<ext> instead of <task_id>.<ext>
    concurrency=8,
    progress=True,
    max_files=20_000,          # safety caps that raise DownloadLimitExceeded
    max_bytes_per_file=10 * 1024 * 1024,
)
print(f"downloaded {n} files")

# One task at a time, when you want a Python-side filter:
for task in run.results(status="successful"):
    body = run.download_task_to_memory(task)   # bytes; or download_task_to_file(task, path)
```

Each `result_url` is a plain presigned URL (no auth header) valid for a short window, so fetch it promptly or re-list the results to get a fresh one.

## Open Jobs: Streaming URLs in Batches

When you don't have every URL up front, open a job, add tasks in batches, and close it. The run stays active and completes once it finishes the final batch.

```python Python theme={"dark"}
job = client.submit_open(zenrows_params={"js_render": "true"})

job.add_tasks({"tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/page/1"}]})
job.add_tasks({"tasks": [{"url": "https://www.scrapingcourse.com/ecommerce/page/2"}],
               "last_batch": True})  # closes the job

run = job.run.wait(timeout=900.0)
```

Setting `last_batch=True` (or calling `job.close()`) locks the job, and the API rejects further adds. File-input (CSV) upload is not supported for open jobs; use a closed job for CSV.

## Large Inputs: CSV Upload

For lists too large to inline, upload a CSV once and reference it. `upload_csv` allocates the slot, PUTs your file to the presigned URL, and returns a `file_input_id` that `submit_regular(file_input_id=...)` consumes.

```python Python theme={"dark"}
file_input_id = client.upload_csv(
    "urls.csv",
    fields={"url": "URL", "external_id": "Customer Ref"},  # column name (header=True) or 0-based index
    header=True,
)
job = client.submit_regular(file_input_id=file_input_id, zenrows_params={"js_render": "true"})
```

Each CSV row becomes a task. Zenrows reads only the `url` and optional `external_id` columns, and applies the job-level `zenrows_params` to all of them. Limits: 50 MB / 100,000 rows.

## Listing and Pagination

The SDK offers both single-page calls (`list_jobs`, `list_runs`, `list_results`) and lazy iterators that page for you. The iterators yield loaded handles.

```python Python theme={"dark"}
for job in client.iter_jobs(job_type="regular", status="closed"):
    print(job.job_id, job.data.status)

for run in client.iter_runs("01J...", page_size=50):
    print(run.run_id, run.data.status)

for task in client.iter_results("01J...", status="failed"):
    print(task.task_id, task.error.code if task.error else None)
```

## Reruns and Retrying Failures

Each rerun creates a brand-new run on the same job.

```python Python theme={"dark"}
# Re-run every task:
new_run = job.rerun()

# Re-run only the failures (optionally also tasks that never started).
# Successful results carry over, so you don't pay to scrape them again:
new_run = job.retry_failed(include_pending=False)
new_run.wait(timeout=600.0)
```

<Tip>
  `retry_failed()` is the usual post-run cleanup, a thin shortcut for `rerun(status="failed")`. You can act on an id you kept without a preceding GET: `client.job(job_id).retry_failed()`.
</Tip>

## Scheduled Jobs

Build a schedule with the typed builders and submit a scheduled job. Zenrows stores its URLs as a template, and each fire produces a fresh run.

```python Python theme={"dark"}
from datetime import datetime
from zenrows.batch import At, Calendar, Rate, Weekly

# Every 6 hours:
client.submit_scheduled(Rate(every=6, unit="hour"),
                        ["https://www.scrapingcourse.com/ecommerce/"])

# 09:00 and 18:00 on Mon/Wed/Fri, Berlin time:
client.submit_scheduled(
    Calendar(times_of_day=["09:00", "18:00"],
             cadence=Weekly(days=["mon", "wed", "fri"]),
             timezone="Europe/Berlin"),
    ["https://www.scrapingcourse.com/ecommerce/"],
)

# One-shot at a wall-clock time. A naive datetime plus an IANA zone
# keeps daylight-saving transitions correct:
client.submit_scheduled(At(datetime(2026, 9, 1, 9, 0), timezone="Europe/Berlin"),
                        ["https://www.scrapingcourse.com/ecommerce/"])
```

A job carries two facets, matching the API's two scopes. `job.schedule.*` controls the recurring schedule, and `job.run.*` controls the **current run**. This disambiguates the API's two distinct "pause" actions:

```python Python theme={"dark"}
job.schedule.pause()                            # stop firing new runs
job.schedule.resume()
job.schedule.update(Rate(every=1, unit="day"))  # replace the schedule

job.run.pause()   # reversibly suspend the current run (a different endpoint)
job.run.resume()
job.run.stop()    # end the current run
```

## Completion Notifications: Webhooks

Attach a webhook at submit time, and Zenrows sends a `run.completed` event to your HTTPS URL when a run completes, or a `run.failed` event when it fails. Set `signature: True` to receive HMAC-signed deliveries (see [Signed Webhooks](#signed-webhooks-hmac-keys)); it defaults to unsigned. You can also manage the webhook later with `client.put_job_webhook(...)` / `client.delete_job_webhook(...)`, and probe a receiver with `client.test_webhook(...)`.

```python Python theme={"dark"}
job = client.submit_regular(
    ["https://www.scrapingcourse.com/ecommerce/"],
    webhook={"url": "https://your-app.com/webhooks/zenrows", "signature": True},
)
```

Zenrows delivers each event **at least once**, so your receiver must deduplicate on the `X-ZenRows-Event-Id` header.

<Tip>
  No webhook receiver? Just poll for completion with `job.run.wait()` or `run.wait()`.
</Tip>

## Bulk Export (ZIP)

Bundle an entire run's result bodies into a single server-side zip: `download_all_results` starts the export, waits for it, and streams the file to disk.

```python Python theme={"dark"}
run.download_all_results("results.zip")
```

<Note>
  The server-side zip is capped at **100 GiB** per run. For larger runs, or one file per task, use `download_to_dir`, which fetches bodies client-side with no size limit (slower; tune with `concurrency`). For manual control, `client.start_results_export(...)` and `client.wait_for_export(...)` expose the two steps separately.
</Note>

## Estimating Cost

Estimate credits **before** submitting with `client.estimate_cost`. It takes the same body you would submit and runs client-side with no API call. The tiers are 1 (base), 5 (`js_render`), 10 (`premium_proxy`), 25 (both), and a dynamic 1 to 25 for `mode=auto`.

```python Python theme={"dark"}
est = client.estimate_cost({
    "type": "regular",
    "tasks": [{"url": f"https://www.scrapingcourse.com/ecommerce/page/{i}"} for i in range(1000)],
    "zenrows_params": {"js_render": "true"},
})
print(est)                       # "5000 credits (1000 tasks)"
print(est.min, est.max, est.exact)
print(est.format())              # per-tier breakdown table
```

An estimate assumes each URL succeeds once. `min == max` (`est.exact`) unless the job uses `mode=auto`, which yields a per-task range.

## Checking Actual Cost

Once a run finishes, read the **actual credits it consumed** from `run.stats.spend` (`{credits, cost}`).

```python Python theme={"dark"}
run = job.run.wait(timeout=600.0)
spend = run.stats.spend
if spend:
    print(f"this run used {spend.credits} credits (cost {spend.cost})")
```

Only successful requests are charged, so realized spend is at most the estimate. Per-task `task.spend` is indicative and often absent on successful rows, so rely on the run-level rollup for actuals. Your Zenrows account statement remains the authoritative record for billing.

## Signed Webhooks: HMAC Keys

Signed deliveries (see [Webhooks](#completion-notifications-webhooks)) need an active HMAC key for your organization. Rotation uses two slots, active plus candidate, so you can roll keys without downtime.

```python Python theme={"dark"}
created = client.rotate_hmac_key()   # the first call creates the active key
print(created.kid, created.secret)   # Zenrows returns the secret ONCE, so store it now

# Phased rotation:
client.rotate_hmac_key()             # stage a candidate (start accepting both)
client.finalize_hmac_key()           # promote the candidate to active, retire the old one
# client.cancel_hmac_rotation()      # or discard the candidate
client.list_hmac_keys()              # metadata only; never returns secrets
```

Your receiver verifies the `X-Signature: t=<unix>,v1=<hex>,kid=<active>` header by recomputing `HMAC-SHA256(secret, f"{t}.{raw_body}")` and comparing the result to `v1`, selecting the secret by `kid`.

## Error Handling

Every non-2xx raises `BatchAPIError`. Its `code` attribute carries the stable code from the RFC 7807 body, and `status_code` the HTTP status.

```python Python theme={"dark"}
from zenrows.batch import BatchAPIError

try:
    client.get_job("does-not-exist")
except BatchAPIError as e:
    print(e.status_code, e.code)   # e.g. 404 not_found
    raise
```

Task-level failures don't raise exceptions. They appear as `failed` results with an `error` object you can read through `task.error.code` and `task.error.detail`.

## Idempotency

Pass `idempotency_key` to `submit_*` and to `rerun` / `retry_failed` to guard against accidentally creating duplicate jobs. Each key can be used once: a later call that reuses the same key raises a `409` conflict, whether or not the body is identical, and the API does not store or replay the original result. An idempotency key is therefore not a safe-retry token, so if a call fails midway, check whether the job was created before retrying with a new key.

```python Python theme={"dark"}
job = client.submit_regular(
    ["https://www.scrapingcourse.com/ecommerce/"],
    idempotency_key="nightly-2026-06-08",
)
```

The full Batch surface (jobs, runs, tasks, results, content, history, file inputs, and HMAC keys) is reachable through methods on `ZenRowsBatchClient`. See the [SDK repository](https://github.com/ZenRows/zenrows-python-sdk) or run `help(ZenRowsBatchClient)`.


# Setting Up Batch
Source: https://docs.zenrows.com/batch/setup

Get started with the Zenrows Batch: find your API key, authenticate, submit your first batch job, and retrieve the results.

This guide walks you through setting up the Zenrows Batch. You'll confirm your access, locate your API key, authenticate, submit your first batch job, and download the results end to end.

<Note>
  Batch is available to every Zenrows account. You just need an active Zenrows API key; your existing Fetch key works without any extra setup.
</Note>

## Initial Setup

Follow these steps to get ready:

<Steps>
  <Step title="Sign in to Zenrows">
    Visit the [Registration Page](https://app.zenrows.com/register) to create an account, or log in if you already have one. Batch uses the **same API key as Fetch**, which you'll find in your dashboard.
  </Step>

  <Step title="Note the base URL">
    All REST requests go to `https://async.api.zenrows.com/v1`.
  </Step>
</Steps>

## Authentication

Authenticate every request with your API key in the `X-API-Key` header. For security, the key travels in a header and never in the URL.

```bash theme={"dark"}
X-API-Key: YOUR_ZENROWS_API_KEY
```

Store the key in an environment variable so your code can read it instead of hard-coding it:

```bash theme={"dark"}
export ZENROWS_API_KEY="YOUR_ZENROWS_API_KEY"
```

## Submit Your First Job

This minimal example submits a **closed job** (one where you know all URLs up front) with JavaScript rendering enabled. Replace `YOUR_ZENROWS_API_KEY` with your key.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests, json

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY', 'Content-Type': 'application/json'}
  payload = {
      'type': 'regular',
      'status': 'closed',
      'zenrows_params': {'js_render': 'true'},
      'tasks': [
          {'url': 'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie'},
          {'url': 'https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket'},
      ],
  }
  response = requests.post('https://async.api.zenrows.com/v1/jobs', headers=headers, data=json.dumps(payload))
  job = response.json()
  print('submitted', job['job_id'], job['latest_run']['status'])
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  axios({
      url: 'https://async.api.zenrows.com/v1/jobs',
      method: 'POST',
      headers: { 'X-API-Key': 'YOUR_ZENROWS_API_KEY' },
      data: {
          type: 'regular',
          status: 'closed',
          zenrows_params: { js_render: 'true' },
          tasks: [
              { url: 'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie' },
              { url: 'https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket' },
          ],
      },
  })
      .then(response => console.log('submitted', response.data.job_id, response.data.latest_run.status))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.core5.http.ContentType;

  public class FirstJob {
      public static void main(final String... args) throws Exception {
          String body = "{"
              + "\"type\":\"regular\",\"status\":\"closed\","
              + "\"zenrows_params\":{\"js_render\":\"true\"},"
              + "\"tasks\":["
              + "{\"url\":\"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie\"},"
              + "{\"url\":\"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket\"}"
              + "]}";
          String response = Request.post("https://async.api.zenrows.com/v1/jobs")
                  .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .bodyString(body, ContentType.APPLICATION_JSON)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $payload = json_encode([
      'type' => 'regular',
      'status' => 'closed',
      'zenrows_params' => ['js_render' => 'true'],
      'tasks' => [
          ['url' => 'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie'],
          ['url' => 'https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket'],
      ],
  ]);
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://async.api.zenrows.com/v1/jobs');
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY', 'Content-Type: application/json']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "bytes"
      "io"
      "log"
      "net/http"
  )

  func main() {
      payload := []byte(`{
          "type": "regular",
          "status": "closed",
          "zenrows_params": { "js_render": "true" },
          "tasks": [
              { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie" },
              { "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket" }
          ]
      }`)
      req, err := http.NewRequest("POST", "https://async.api.zenrows.com/v1/jobs", bytes.NewBuffer(payload))
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      req.Header.Set("Content-Type", "application/json")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  payload = {
      type: 'regular',
      status: 'closed',
      zenrows_params: { js_render: 'true' },
      tasks: [
          { url: 'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie' },
          { url: 'https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket' }
      ]
  }
  conn = Faraday.new
  res = conn.post('https://async.api.zenrows.com/v1/jobs') do |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
      req.headers['Content-Type'] = 'application/json'
      req.body = payload.to_json
  end
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl -X POST "https://async.api.zenrows.com/v1/jobs" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "type": "regular",
      "status": "closed",
      "zenrows_params": { "js_render": "true" },
      "tasks": [
        { "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie" },
        { "url": "https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket" }
      ]
    }'
  ```
</CodeGroup>

The response includes a `job_id` and the initial run status. Poll `GET /jobs/{id}` until the run reaches a **terminal** state (finished and no longer changing), then collect the results.

### Get Your Results

Once the run is complete, list the successful tasks and download each one's scraped content. Every result row carries a `result_url`, a presigned link to the scraped page that is valid for 2 hours. Replace `JOB_ID` with the `job_id` from the submit response.

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  headers = {'X-API-Key': 'YOUR_ZENROWS_API_KEY'}
  page = requests.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/results',
                      headers=headers, params={'status': 'successful'}).json()
  for row in page['results']:
      html = requests.get(row['result_url']).text  # the scraped page
      print(row.get('external_id') or row['task_id'], len(html), 'chars')
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const page = (await axios.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/results', {
      headers: { 'X-API-Key': 'YOUR_ZENROWS_API_KEY' },
      params: { status: 'successful' },
  })).data;
  for (const row of page.results) {
      const html = (await axios.get(row.result_url)).data; // the scraped page
      console.log(row.external_id ?? row.task_id, String(html).length, 'chars');
  }
  ```

  ```java Java theme={"dark"}
  import com.fasterxml.jackson.databind.JsonNode;
  import com.fasterxml.jackson.databind.ObjectMapper;
  import org.apache.hc.client5.http.fluent.Request;

  public class GetResults {
      public static void main(final String... args) throws Exception {
          ObjectMapper mapper = new ObjectMapper();
          JsonNode page = mapper.readTree(
                  Request.get("https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful")
                          .addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                          .execute().returnContent().asString());
          for (JsonNode row : page.path("results")) {
              String html = Request.get(row.path("result_url").asText())
                      .execute().returnContent().asString(); // the scraped page
              String id = row.path("external_id").asText(row.path("task_id").asText());
              System.out.println(id + " " + html.length() + " chars");
          }
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init('https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful');
  curl_setopt($ch, CURLOPT_HTTPHEADER, ['X-API-Key: YOUR_ZENROWS_API_KEY']);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $page = json_decode(curl_exec($ch), true);
  curl_close($ch);
  foreach ($page['results'] as $row) {
      $html = file_get_contents($row['result_url']); // the scraped page
      echo ($row['external_id'] ?? $row['task_id']) . ' ' . strlen($html) . ' chars' . PHP_EOL;
  }
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "encoding/json"
      "io"
      "log"
      "net/http"
  )

  func main() {
      req, _ := http.NewRequest("GET", "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful", nil)
      req.Header.Set("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := http.DefaultClient.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()
      body, _ := io.ReadAll(resp.Body)
      var page struct {
          Results []struct {
              TaskID     string `json:"task_id"`
              ExternalID string `json:"external_id"`
              ResultURL  string `json:"result_url"`
          } `json:"results"`
      }
      json.Unmarshal(body, &page)
      for _, row := range page.Results {
          r, _ := http.Get(row.ResultURL) // the scraped page
          html, _ := io.ReadAll(r.Body)
          r.Body.Close()
          name := row.ExternalID
          if name == "" {
              name = row.TaskID
          }
          log.Println(name, len(html), "chars")
      }
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'
  require 'json'

  page = JSON.parse(Faraday.get('https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful') { |req|
      req.headers['X-API-Key'] = 'YOUR_ZENROWS_API_KEY'
  }.body)
  page['results'].each do |row|
      html = Faraday.get(row['result_url']).body # the scraped page
      puts "#{row['external_id'] || row['task_id']} #{html.length} chars"
  end
  ```

  ```bash cURL theme={"dark"}
  # 1) List the successful results; each row has a presigned result_url valid 2h.
  curl "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful" \
    -H "X-API-Key: YOUR_ZENROWS_API_KEY"

  # 2) Download one result's scraped HTML using its result_url from the response above.
  curl "RESULT_URL"
  ```
</CodeGroup>

<Tip>
  Want to see the cost before you run anything? You can estimate credits client-side with no API call. See [Estimate cost](/batch/developer-guide-restapi#6-cost).
</Tip>

## Next Steps

<CardGroup>
  <Card title="Developer Guide: REST API" icon="code" href="/batch/developer-guide-restapi">
    Every feature against the REST API: open jobs, CSV upload, scheduling, webhooks, and more.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/batch/troubleshooting">
    Diagnose authentication, submission, download, and webhook issues fast.
  </Card>

  <Card title="FAQ" icon="circle-question" href="/batch/faq">
    Quick answers on access, pricing, limits, and scraping options.
  </Card>
</CardGroup>


# Troubleshooting
Source: https://docs.zenrows.com/batch/troubleshooting

Diagnose and fix the most common Batch issues: authentication, submission errors, job state conflicts, downloads, webhooks, scheduling, and unexpected cost.

<Note>
  Batch is available to every Zenrows account. If requests fail with `401 unauthenticated`, check that your API key is present and valid before working through the steps below.
</Note>

Most Batch problems fall into a handful of categories. Find your symptom in the table below, then jump to the section that explains the cause and the fix.

| Symptom | Likely cause | Go to |
| - | - | - |
| `401` on every request | Missing or wrong API key | [Authentication and access](#authentication-and-access) |
| `400 invalid_argument` on submit | Malformed body, bad URL, or batch over the limit | [Submission errors](#submission-errors) |
| `429 quota_exceeded` on submit | Too many active jobs (max 3) | [Account limits](#account-limits) |
| `409 conflict` when adding tasks | The job is already closed | [Job and run state conflicts](#job-and-run-state-conflicts) |
| Download link returns an error | The presigned URL expired | [Results and downloads](#results-and-downloads) |
| Some URLs never returned data | Task-level failures, not request errors | [Tasks that fail](#tasks-that-fail) |
| Webhook never arrived | Receiver not HTTPS, or signature setup | [Webhooks not arriving](#webhooks-not-arriving) |
| Schedule didn't fire as expected | Unsupported time value or timezone | [Scheduled jobs](#scheduled-jobs) |
| Spent more credits than estimated | Retries or `mode=auto` range | [Unexpected cost](#unexpected-cost) |
| `503` responses | Transient server error | [Transient errors](#transient-errors) |

## How to Diagnose Any Request

<Steps>
  <Step title="Read the HTTP status and error code">
    Every error response is JSON in the `application/problem+json` format and carries a stable `code` (for example `invalid_argument` or `conflict`). The `detail` field usually states the cause directly. See the full table in the [developer guide](/batch/developer-guide-restapi#error-handling-reference).
  </Step>

  <Step title="Separate request errors from task errors">
    A non-2xx response means the API rejected your request. A `2xx` response with `failed` results means individual URLs failed while the job itself succeeded. These are handled differently. See [Tasks that fail](#tasks-that-fail).
  </Step>

  <Step title="Keep the job ID and request ID">
    Note the `job_id` (and `run_id` where relevant). If you need to contact support, include them so the team can trace your job in the logs.
  </Step>
</Steps>

## Authentication and Access

**`401 unauthenticated`** means the API key is missing or invalid.

* Send the key in the `X-API-Key` header on every request. Never put it in the URL.
* Use the same key as your Fetch. No separate Batch key exists.

**`402 payment_required`** means your account has no available credits. Top up your plan, then resubmit.

**`402 api_key_cap_reached`** means the API key reached one of its [credit caps](/first-steps/api-key-credit-caps). The `detail` names the cap and when it resets. Raise or remove the cap, or wait for it to reset, then retry. See [API key credit caps and Batch](/batch/developer-guide-restapi#api-key-credit-caps-and-batch).

## Submission Errors

**`400 invalid_argument`** means the request body failed validation. The response includes an `invalid_tasks` array listing each problem as `{index, reason}`, so you can see exactly which URL failed and why.

Common causes and fixes:

* **Malformed or unencoded URLs.** Make sure each `url` is a complete, valid address. Fix any entry flagged in `invalid_tasks`.
* **Too many URLs.** A job submit accepts up to **100,000 URLs**, and each add-tasks call on an [open job](/batch/developer-guide-restapi#open-jobs-adding-tasks-in-batches) accepts up to **10,000**. For very large lists, use a [CSV upload](/batch/developer-guide-restapi#large-inputs-csv-upload).
* **Wrong value types in `zenrows_params`.** Values can be strings, booleans, or numbers. Boolean options such as `js_render` are sent as `"true"` / `"false"`.

<Warning>
  **CSV uploads must use the exact content-type the upload slot requires.** When you PUT the file to the presigned upload URL, send the headers returned in `slot.upload.headers` (for example `Content-Type: text/csv`) unchanged. A mismatched content-type causes the upload to fail. Limits: 50 MB / 100,000 rows.
</Warning>

## Account Limits

**`429 quota_exceeded`** means you've hit an account limit. An account can run **3 active jobs concurrently**; submitting a 4th returns this error. Wait for one to finish, or contact Zenrows to raise the limit for your account. These limits can be raised per account on request.

## Job and Run State Conflicts

**`409 conflict`** means the job is in a state that doesn't allow your action. Check the `detail` and `code` to see which conflict it is.

* **Adding tasks to a closed job.** Once you send `last_batch: true` or call `close`, the job locks and rejects further adds. Open a new job for more URLs.
* **Ingestion still in progress.** A very large submission keeps writing its task rows for a few seconds after responding. `POST /jobs/{id}/tasks` on an open job can return `409` during this window; wait a moment and retry.
* **`run_not_terminal`.** You tried an action that requires the run to be finished (for example, exporting) while it's still running. Wait for the run to reach a terminal state, then retry.
* **Abandoning an in-flight run.** To stop a run that's still processing, call `POST /jobs/{id}/stop` (or its alias `POST /jobs/{id}/cancel`). To suspend it without ending it, use `POST /jobs/{id}/pause` and later `POST /jobs/{id}/resume`. See [Control the current run](/batch/developer-guide-restapi#control-the-current-run).

## Results and Downloads

**A `result_url` returns an error.** Each `result_url` is a **presigned URL** (a temporary, signed link) valid for **2 hours**. If it expired, re-list the results to get a fresh link rather than reusing a stored one. Don't persist these URLs long-term.

**Fetching a single task's content returns a non-200 status.** The `/tasks/{id}/content` endpoint uses status to signal task state:

| Status | Meaning |
| - | - |
| `200` | Content is ready; the body is the scraped result. |
| `409` | The task is still pending or processing. Poll again shortly. |
| `422` | The task failed; the body is the stored error. |

**A bulk export fails or its link stops working.** Exports expire **12 hours** after creation, and the API presigns the `download_url` fresh on each poll, so always use the latest one. If a run's combined results exceed **100 GiB**, the export fails with a clear error instead of producing a partial archive. Download results in pages, or split the work across smaller runs.

## Tasks That Fail

When the job succeeds but some URLs don't return data, those are **task-level failures**, not request errors. They don't raise an HTTP error or an exception.

* List results filtered by `status="failed"` and read each row's `error` object (`error.code`, `error.detail`) to see what went wrong.
* A failed task's results row already carries the error, so you rarely need the `/content` endpoint just to find out why it failed.
* To re-run only the failures, use [retry failed](/batch/developer-guide-restapi#5-reruns-and-retrying-failures). Successful results carry over, so you only pay to re-scrape what failed.
* If a partial rerun returns `409 partial_rerun_window_expired`, the 10-day window for retrying individual statuses has closed. Start a full rerun instead.

<Tip>
  If many tasks fail on a protected site, the cause is usually the scrape configuration, not Batch. Enable `js_render` for dynamic pages, add `premium_proxy` and `proxy_country` for blocked or geo-restricted sites, or use `wait_for` to wait for a selector before capturing, the same options you would use with Fetch.
</Tip>

## Webhooks Not Arriving

The API sends a `run.completed` event when a run completes, or a `run.failed` event when it fails. If you don't receive it:

* **Use an HTTPS receiver.** Webhook URLs must use HTTPS.
* **Test the endpoint first.** Call `POST /webhook/test` with your receiver URL to confirm the API can reach it. The response reports `delivered` and `status_code`.
* **Deduplicate, don't assume once.** Deliveries are **at-least-once**, so the same event can arrive more than once. Deduplicate on the `X-ZenRows-Event-Id` header.
* **Signature verification fails.** Signed deliveries (`signature: true`) require an active HMAC key for your organization. Verify the `X-Signature` header against the **raw request body**, not a re-serialized object. See [Verifying signed webhooks](/batch/developer-guide-restapi#verify-signed-webhooks).

<Tip>
  Webhooks are a convenience, not the source of truth. If you ever miss a delivery, poll the job. The run state and result URLs are always authoritative.
</Tip>

## Scheduled Jobs

* **Times of day must be full hours.** Use `"09:00"`, not `"09:30"`. A non-hour value is rejected.
* **Use IANA timezone names.** Names like `Europe/Berlin` keep daylight-saving transitions correct. Pair a timezone with `at` and `calendar` schedules.
* **Provide exactly one schedule type.** Send one of `at`, `rate`, or `calendar` per scheduled job.
* **Pausing and resuming.** Change state with `POST /jobs/{id}/schedule/state` using `{"schedule_state": "paused"}` or `{"schedule_state": "active"}`.

## Unexpected Cost

* **Only successful requests cost credits.** A run with failures costs less than its up-front estimate, never more for the failed tasks.
* **The estimate assumes one success per URL.** Compare it against the run's actual spend with [Checking actual cost](/batch/developer-guide-restapi#check-actual-cost).
* **Adaptive Stealth Mode is a range.** With `mode=auto`, each task's credit cost is dynamic (1 to 25) depending on what the site requires, so the estimate is a range rather than a single number.

## Transient Errors

**`503 internal`** is a transient server-side error and is safe to retry. Use an increasing delay (a **backoff**) between attempts rather than a tight retry loop, as shown in the [developer guide](/batch/developer-guide-restapi#check-job-status). On a `503` from a submit specifically, retry with a **fresh** idempotency key, since a key claimed by the failed attempt replays `409`. A late `503` can also leave a visible job with a `stopped` run, which is safe to `DELETE`.

## Still Stuck?

If a problem persists after you've worked through the relevant section above, reach out to support with:

* The `job_id` (and `run_id` if the issue is run-specific).
* The exact request body, with your API key redacted.
* The HTTP status and the full error `code` and `detail`.
* Whether the issue is reproducible or intermittent, and when it started.

This lets the team trace your job in the logs and respond without re-asking for the basics.


# Browser Sessions FAQ: Sessions, CDP, and Automation
Source: https://docs.zenrows.com/browser-sessions/faq

Find answers to frequently asked questions about Browser Sessions sessions, CDP protocol, proxy management, and browser automation limits.

<Accordion title="Why should I use the Zenrows Browser Sessions?">
  Using Zenrows Browser Sessions (formerly, Scraping Browser) saves you from the hassle of managing proxies, rotating IPs, and dealing with browser automation setup. With just one line of code, you get access to a fully managed browser with residential proxies.
</Accordion>

<Accordion title="Can Browser Sessions solve CAPTCHAs?">
  At the moment, Zenrows Browser Sessions cannot solve CAPTCHAs. You will need to handle CAPTCHA solving through third-party services if required.
  <Tip>You can also try our [Fetch](/fetch/api-reference), which comes with everything available on Browser Sessions, plus antibot capabilities, anti-CAPTCHAs and many more.</Tip>
</Accordion>

<Accordion title="What is the protocol used by Browser Sessions?">
  The Zenrows Browser Sessions uses the Chrome DevTools Protocol (CDP) as its communication standard. CDP is a powerful protocol used for browser automation and debugging, allowing you to control a browser instance programmatically.

  ## Why CDP?

  CDP is widely adopted because it provides deep control over browser behavior, offering features like:

  * **Page interaction:** Automate navigation, clicks, form submissions, etc.
  * **Network monitoring:** Access detailed network activity, including headers, responses, and requests.
  * **DOM manipulation:** Direct interaction with and modification of the Document Object Model (DOM).
  * **JavaScript execution:** Run JavaScript code within the page context.
  * **Performance tracking:** Monitor performance metrics, including rendering and page load speeds.
  * **Screenshots & PDFs:** Capture visual representations of the page.

  By leveraging CDP, the Zenrows Browser Sessions allows you to interact with web pages just like a human user would, while also benefiting from additional features like IP rotation and proxy management.

  ## How is CDP different from other protocols?

  Compared to other protocols like WebDriver (used by Selenium), CDP is generally faster and more capable because it communicates directly with the browser's core without the intermediary of a WebDriver layer. This direct interaction gives CDP greater flexibility and control, making it the protocol of choice for Zenrows Browser Sessions.
</Accordion>

<Accordion title="Can I change the viewport or window size with Browser Sessions?">
  No, Zenrows Browser Sessions does not allow programmatically changing the viewport or window size. Methods like <code>page.setViewportSize()</code> or similar browser commands are blocked and will not take effect. The browser window and viewport are managed automatically to optimize for anti-bot protection and reliability.

  If you need to take screenshots at different sizes, this is currently not supported by resizing the window or viewport. The browser will always use a real, consistent window size that matches the fingerprint.

  For reference, Zenrows also restricts other browser-modifying actions, such as changing the user agent or certain device metrics. If you have specific requirements or questions about supported features, please contact our support team.
</Accordion>


# Country Targeting for Zenrows Browser Sessions
Source: https://docs.zenrows.com/browser-sessions/features/country

Learn how to configure country-level geo-targeting in Zenrows Browser Sessions to access localized content and geo-restricted websites.

Zenrows Browser Sessions (formerly, Scraping Browser) enables precise country-level targeting for your web scraping operations, letting you access content as if you were browsing from a specific country. This is essential for geo-restricted content, localized pricing, and market research across nations.

## What is Country Targeting?

Country-targeting routes your scraping requests through residential IP addresses from a specific country, providing the most precise geolocation control available. Unlike world region targeting, which covers broad areas, country targeting gives you IPs from individual nations.

**Key benefits:**

* Access country-specific content and services
* Accurate localized pricing and product data
* Compliance testing for legal jurisdictions
* Ad verification and fraud prevention by country
* Market research with country-level precision

<Note>
  Auto-rotate and Residential IPs are pre-configured and enabled by default for all Zenrows Browser Sessions users. Country targeting works seamlessly with these features to provide optimal performance.
</Note>

## How to Configure Country Targeting

You can configure country targeting in two ways:

* **Zenrows SDK:** Recommended for most users (simplest, most robust)
* **Direct WebSocket URL:** For advanced or custom integrations

<Tabs>
  <Tab title="Puppeteer">
    For custom integrations or direct control, specify the country using the `proxy_country` parameter in the WebSocket URL.

    ```bash theme={"dark"}
    wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_country=es
    ```

    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const puppeteer = require('puppeteer-core');
      const wsUrl = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_country=es';

      (async () => {
          const browser = await puppeteer.connect({ browserWSEndpoint: wsUrl });
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from pyppeteer import connect

      async def main():
          ws_url = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_country=es'
          browser = await connect(browserWSEndpoint=ws_url)
          page = await browser.newPage()
          await page.goto('https://example.com')
          print(await page.title())
          await browser.close()

      asyncio.get_event_loop().run_until_complete(main())
      ```
    </CodeGroup>

    Zenrows supports residential IPs from over 100 countries worldwide. For the complete list and codes, see our [Premium Proxy Countries List](/first-steps/faq#what-is-geolocation-and-what-are-all-the-premium-proxy-countries).

    **Most Popular:**

    * **us** - United States
    * **gb** - United Kingdom
    * **br** - Brazil
    * **de** - Germany
    * **ca** - Canada
    * **au** - Australia
    * **fr** - France
    * **in** - India
    * **es** - Spain
    * **it** - Italy
    * **nl** - Netherlands
  </Tab>

  <Tab title="Playwright">
    For custom integrations or direct control, specify the country using the `proxy_country` parameter in the WebSocket URL.

    ```bash theme={"dark"}
    wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_country=es
    ```

    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const { ScrapingBrowser, ProxyCountry } = require('@zenrows/scraping-browser');
      const { chromium } = require('playwright');

      (async () => {
          const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
          // Target Germany
          const connectionURL = scrapingBrowser.getConnectURL({
              proxy: { location: ProxyCountry.DE },
          });
          const browser = await chromium.connectOverCDP(connectionURL);
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from playwright.async_api import async_playwright

      async def main():
          ws_url = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_country=es'
          async with async_playwright() as p:
              browser = await p.chromium.connect_over_cdp(ws_url)
              page = await browser.new_page()
              await page.goto('https://example.com')
              print(await page.title())
              await browser.close()

      asyncio.run(main())
      ```
    </CodeGroup>

    Zenrows supports residential IPs from over 100 countries worldwide. For the complete list and codes, see our [Premium Proxy Countries List](/first-steps/faq#what-is-geolocation-and-what-are-all-the-premium-proxy-countries).

    **Most Popular:**

    * **us** - United States
    * **gb** - United Kingdom
    * **br** - Brazil
    * **de** - Germany
    * **ca** - Canada
    * **au** - Australia
    * **fr** - France
    * **in** - India
    * **es** - Spain
    * **it** - Italy
    * **nl** - Netherlands
  </Tab>

  <Tab title="Zenrows SDK">
    The Zenrows SDK offers the easiest way to configure country targeting, with built-in error handling and connection management.

    <CodeGroup>
      ```javascript Node.js Puppeteer theme={"dark"}
      const { ScrapingBrowser, ProxyCountry } = require('@zenrows/scraping-browser');
      const puppeteer = require('puppeteer-core');

      (async () => {
          const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
          // Target Spain
          const connectionURL = scrapingBrowser.getConnectURL({
              proxy: { location: ProxyCountry.ES },
          });
          const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```

      ```javascript Node.js Playwright theme={"dark"}
      const { ScrapingBrowser, ProxyCountry } = require('@zenrows/scraping-browser');
      const { chromium } = require('playwright');

      (async () => {
          const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
          // Target Germany
          const connectionURL = scrapingBrowser.getConnectURL({
              proxy: { location: ProxyCountry.DE },
          });
          const browser = await chromium.connectOverCDP(connectionURL);
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```
    </CodeGroup>

    **Popular SDK Country Options:**

    * `ProxyCountry.US` - United States
    * `ProxyCountry.GB` - United Kingdom
    * `ProxyCountry.BR` - Brazil
    * `ProxyCountry.DE` - Germany
    * `ProxyCountry.FR` - France
    * `ProxyCountry.IN` - India
    * `ProxyCountry.ES` - Spain
    * `ProxyCountry.CA` - Canada
    * `ProxyCountry.AU` - Australia
    * `ProxyCountry.JP` - Japan
  </Tab>
</Tabs>

<Frame>
  <img alt="Country Targeting Configuration Interface" />
</Frame>

## Best Practices

**Choosing the Right Country:**

* Select countries based on your target market or content
* Consider local business hours and time zones for time-sensitive scraping
* Test with multiple countries to understand regional content differences

**SDK vs Direct Connection:**

* **Use the SDK** for most projects. It offers better error handling and automatic retries
* **Direct WebSocket** is for custom frameworks or specific connection control

**Performance Optimization:**

* Some countries have larger IP pools and better performance
* Monitor success rates and adjust the country selection accordingly
* Use fallback countries if your primary choice has issues

## Troubleshooting

**Common Issues and Solutions:**

**Country Code Errors:**

* Use the correct ISO country code (e.g., 'es' for Spain)
* Check the country list to ensure support
* Use lowercase codes in direct WebSocket connections

**No Content Differences:**

* Some sites may not vary by country. Test manually by accessing the website
* Try different countries to verify geo-targeting
* Some sites use detection beyond IP geolocation

**Connection Issues:**

* Some countries may have smaller IP pools. Try alternative countries
* Monitor the response error for indications that may help
* Use world region targeting if country-level access is problematic

<Tip>
  You cannot use both world region and country targeting at the same time. Select the level of precision you require, whether broad regional coverage or targeted to a specific country.
</Tip>

## Next Steps

* Explore [Session TTL Configuration](/browser-sessions/features/session-ttl) to control browser session duration
* Learn about [World Region Targeting](/browser-sessions/features/world-region) for broader coverage
* See the <a href="https://github.com/ZenRows/browser-js-sdk">Zenrows SDK Repository</a> for advanced features and examples
* Review [Premium Proxy Countries List](/first-steps/faq#what-is-geolocation-and-what-are-all-the-premium-proxy-countries) for all available countries
* Check the [Practical Use Cases](/browser-sessions/help/practical-use-cases) for real world examples


# Session TTL Configuration for Zenrows Browser Sessions
Source: https://docs.zenrows.com/browser-sessions/features/session-ttl

Set and customize Session TTL to manage how long your Zenrows Browser Sessions sessions stay active, from 1 to 15 minutes maximum.

Zenrows Browser Sessions (formerly, Scraping Browser) lets you control the duration of your browser sessions with Session TTL (Time To Live) configuration. This is crucial for managing resource usage, maintaining session consistency, and optimizing scraping workflows.

## What is Session TTL?

Session TTL (Time To Live) controls how long a browser session stays active before terminating. Think of it as a timer for your scraping session after the time expires, the session closes automatically, freeing up resources and ensuring clean session management.

**Why Session TTL matters:**

* **Resource Management:** Prevents sessions from running indefinitely
* **Cost Control:** Helps manage usage costs by auto-terminating idle sessions
* **Session Consistency:** Maintains stable connections for multi-step workflows
* **Workflow Optimization:** Lets you match session duration to your scraping needs

<Note>
  The default Session TTL is 3 minutes (180 seconds), which works well for most tasks. You can customize this between 1 minute (60 seconds) and 15 minutes (900 seconds).
</Note>

## How to Configure Session TTL

You can configure Session TTL in two ways:

* **Zenrows SDK:** Recommended for most users (simple, robust)
* **Direct WebSocket URL:** For advanced or custom integrations

<Tabs>
  <Tab title="Puppeteer">
    For advanced users or custom integrations, specify Session TTL in the WebSocket URL using the `session_ttl` parameter with duration strings.

    ```bash theme={"dark"}
    wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&session_ttl=2m
    ```

    **Duration String Examples:**

    * `1m` - 1 minute (60 seconds)
    * `2m` - 2 minutes (120 seconds)
    * `5m` - 5 minutes (300 seconds)
    * `10m` - 10 minutes (600 seconds)
    * `15m` - 15 minutes (900 seconds - max)

    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const puppeteer = require('puppeteer-core');
      const wsUrl = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&session_ttl=5m';
      (async () => {
          const browser = await puppeteer.connect({ browserWSEndpoint: wsUrl });
          const page = await browser.newPage();
          await page.goto('https://example.com');
          await browser.close();
      })();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from pyppeteer import connect

      async def main():
          ws_url = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&session_ttl=5m'
          browser = await connect(browserWSEndpoint=ws_url)
          page = await browser.newPage()
          await page.goto('https://example.com')
          await browser.close()

      asyncio.get_event_loop().run_until_complete(main())

      ```
    </CodeGroup>
  </Tab>

  <Tab title="Playwright">
    For advanced users or custom integrations, specify Session TTL in the WebSocket URL using the `session_ttl` parameter with duration strings.

    ```bash theme={"dark"}
    wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&session_ttl=2m
    ```

    **Duration String Examples:**

    * `1m` - 1 minute (60 seconds)
    * `2m` - 2 minutes (120 seconds)
    * `5m` - 5 minutes (300 seconds)
    * `10m` - 10 minutes (600 seconds)
    * `15m` - 15 minutes (900 seconds - max)

    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const { chromium } = require('playwright');
      const wsUrl = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&session_ttl=5m';
      (async () => {
          const browser = await chromium.connectOverCDP(wsUrl);
          const page = await browser.newPage();
          await page.goto('https://example.com');
          await browser.close();
      })();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from playwright.async_api import async_playwright

      async def main():
          ws_url = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&session_ttl=5m'
          async with async_playwright() as p:
              browser = await p.chromium.connect_over_cdp(ws_url)
              page = await browser.new_page()
              await page.goto('https://example.com')
              await browser.close()

      asyncio.run(main())
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Zenrows SDK">
    The Zenrows SDK offers the most user-friendly way to configure Session TTL, featuring automatic session management and error handling.

    <CodeGroup>
      ```javascript Node.js Puppeteer theme={"dark"}
      const { ScrapingBrowser } = require('@zenrows/scraping-browser');
      const puppeteer = require('puppeteer-core');

      (async () => {
          const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
          // 2-minute session
          const connectionURL = scrapingBrowser.getConnectURL({ sessionTTL: 120 });
          const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
          const page = await browser.newPage();
          await page.goto('https://example.com');
          await browser.close();
      })();
      ```

      ```javascript Node.js Playwright theme={"dark"}
      const { ScrapingBrowser } = require('@zenrows/scraping-browser');
      const { chromium } = require('playwright');

      (async () => {
          const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
          // 5-minute session
          const connectionURL = scrapingBrowser.getConnectURL({ sessionTTL: 300 });
          const browser = await chromium.connectOverCDP(connectionURL);
          const page = await browser.newPage();
          await page.goto('https://example.com');
          await browser.close();
      })();
      ```
    </CodeGroup>

    **SDK Configuration Examples:**

    ```javascript theme={"dark"}
    // 1-minute session
    const quickSession = scrapingBrowser.getConnectURL({ sessionTTL: 60 });
    // 5-minute session
    const mediumSession = scrapingBrowser.getConnectURL({ sessionTTL: 300 });
    // 15-minute session (maximum)
    const longSession = scrapingBrowser.getConnectURL({ sessionTTL: 900 });
    ```
  </Tab>
</Tabs>

## Choosing the Right Session TTL

**Session Duration Guidelines:**

**1-2 Minutes (60-120 seconds):**

* Quick data extraction from single pages
* Simple form submissions
* Basic content scraping
* Testing and development

**3-5 Minutes (180-300 seconds):**

* Multi-page navigation
* Workflows with several steps
* Moderate data collection
* Most general scraping

**10-15 Minutes (600-900 seconds):**

* Complex multi-step workflows
* Large-scale extraction
* User interaction simulation
* Long-running automation

## Combining Session TTL with Other Features

Session TTL works seamlessly with other Zenrows features for advanced scraping setups.

### Session TTL + Country Targeting

```javascript Node.js theme={"dark"}
const connectionURL = scrapingBrowser.getConnectURL({
    sessionTTL: 300, // 5 minutes
    proxy: { location: ProxyCountry.ES },
});
```

### Session TTL + World Region Targeting

```javascript Node.js theme={"dark"}
const connectionURL = scrapingBrowser.getConnectURL({
    sessionTTL: 600, // 10 minutes
    proxy: { location: ProxyRegion.Europe },
});
```

### Direct WebSocket with Multiple Parameters

```bash theme={"dark"}
wss://browser.zenrows.com?apikey=YOUR_API_KEY&session_ttl=8m&proxy_country=us
```

## Best Practices

**Optimize for Your Workflow:**

* Match session duration to your scraping needs
* Add a small buffer (30-60 seconds) for delays
* Use shorter sessions for simple tasks

**SDK vs Direct Connection:**

* **SDK:** Use for most projects and handles sessions automatically
* **Direct WebSocket:** Use for custom frameworks or specific control

**Monitoring and Debugging:**

* Monitor session usage in the Zenrows dashboard
* Log session start/end times
* Use proper durations to avoid premature expiry

**Error Handling:**

```javascript Node.js theme={"dark"}
try {
    const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
    // Your scraping code
} catch (error) {
    if (error.message.includes('session expired')) {
        console.log('Session TTL expired, consider increasing duration');
    }
    // Other errors
}
```

## Troubleshooting

**Common Issues and Solutions:**

**Session Expires Too Quickly:**

* Increase sessionTTL for more time
* Monitor execution time and add buffer
* Break long workflows into smaller sessions

**Invalid TTL Values:**

* TTL must be 60-900 seconds in SDK
* Use valid duration strings (1m, 2m, etc.) for WebSocket
* Match TTL to workflow

**Session Management Issues:**

* Always close browser connections properly
* Handle session expiration errors
* Monitor usage in the [Analytics dashboard](https://app.zenrows.com/overview?product=browser#performance)

<Tip>
  The sessionTTL parameter uses seconds with the SDK (e.g., 120 for 2 min) and duration strings for direct WebSocket (e.g., "2m").
</Tip>

## Next Steps

* Learn about [Country Targeting](/browser-sessions/features/country)
* Explore [World Region Targeting](/browser-sessions/features/world-region)
* See the <a href="https://github.com/ZenRows/browser-js-sdk">Zenrows SDK Repository</a> for advanced features and examples
* Review the [Practical Use Cases](/browser-sessions/help/practical-use-cases) for real world examples


# World Region Configuration for Zenrows Browser Sessions
Source: https://docs.zenrows.com/browser-sessions/features/world-region

Set up world region targeting to route your Zenrows Browser Sessions requests via residential IPs for location-specific web scraping.

Zenrows Browser Sessions (formerly, Scraping Browser) empowers you to target specific regions of the world for your web scraping operations, unlocking geo-restricted content and enabling region-specific data collection. This guide shows you how to configure world region targeting using both the Zenrows SDK (recommended) and direct WebSocket connections.

## What is World Region Targeting?

World region targeting routes your scraping requests through residential IPs from a chosen geographic area. This is crucial when websites display different content, pricing, or availability based on visitor location.

**Typical use cases:**

* Accessing geo-locked content or services
* Price comparison across markets
* Testing website localization and UX
* Circumventing geo-blocks for data gathering
* Ad verification in multiple regions

<Note>
  Auto-rotate and Residential IPs are pre-configured and enabled by default for all Zenrows Browser Sessions users, ensuring optimal performance and reliability.
</Note>

## How to Configure World Region

You can configure world region targeting in two ways:

* **Zenrows SDK:** Easiest and most robust option for most users
* **Direct WebSocket URL:** For advanced or custom integrations

<Tabs>
  <Tab title="Puppeteer">
    Specify the region in the WebSocket URL with the `proxy_region` parameter:

    ```bash theme={"dark"}
    wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_region=eu
    ```

    **Examples:**

    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const puppeteer = require('puppeteer-core');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_region=eu';

      (async () => {
          const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from pyppeteer import connect

      async def main():
          ws_url = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_region=eu'
          browser = await connect(browserWSEndpoint=ws_url)
          page = await browser.newPage()
          await page.goto('https://example.com')
          print(await page.title())
          await browser.close()

      asyncio.get_event_loop().run_until_complete(main())
      ```
    </CodeGroup>

    **Available Region Codes:**

    * `eu` - Europe
    * `na` - North America
    * `ap` - Asia Pacific
    * `sa` - South America
    * `af` - Africa
    * `me` - Middle East
  </Tab>

  <Tab title="Playwright">
    Specify the region in the WebSocket URL with the `proxy_region` parameter:

    ```bash theme={"dark"}
    wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_region=eu
    ```

    **Examples:**

    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const { chromium } = require('playwright');

      (async () => {
          const browser = await chromium.connectOverCDP('wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_region=eu');
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from playwright.async_api import async_playwright

      async def main():
          async with async_playwright() as p:
              browser = await p.chromium.connect_over_cdp('wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_region=eu')
              page = await browser.new_page()
              await page.goto('https://example.com')
              print(await page.title())
              await browser.close()

      asyncio.run(main())
      ```
    </CodeGroup>

    **Available Region Codes:**

    * `eu` - Europe
    * `na` - North America
    * `ap` - Asia Pacific
    * `sa` - South America
    * `af` - Africa
    * `me` - Middle East
  </Tab>

  <Tab title="Zenrows SDK">
    The Zenrows SDK makes region configuration simple and manages connections automatically. Recommended for most users.

    <CodeGroup>
      ```javascript Node.js Puppeteer theme={"dark"}
      const { ZenRowsBrowser, ProxyRegion } = require('zenrows');

      (async () => {
          const browser = await ZenRowsBrowser.puppeteer({
              apiKey: 'YOUR_ZENROWS_API_KEY',
              proxyRegion: ProxyRegion.Europe // or .NorthAmerica, .AsiaPacific, etc.
          });
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```

      ```javascript Node.js Playwright theme={"dark"}
      const { ZenRowsBrowser, ProxyRegion } = require('zenrows');

      (async () => {
          const browser = await ZenRowsBrowser.playwright({
              apiKey: 'YOUR_ZENROWS_API_KEY',
              proxyRegion: ProxyRegion.Europe // or .NorthAmerica, .AsiaPacific, etc.
          });
          const page = await browser.newPage();
          await page.goto('https://example.com');
          console.log(await page.title());
          await browser.close();
      })();
      ```
    </CodeGroup>

    **Available SDK Region Options:**

    * `ProxyRegion.Europe` - European residential IPs
    * `ProxyRegion.NorthAmerica` - North American residential IPs
    * `ProxyRegion.AsiaPacific` - Asia-Pacific residential IPs
    * `ProxyRegion.SouthAmerica` - South American residential IPs
    * `ProxyRegion.Africa` - African residential IPs
    * `ProxyRegion.MiddleEast` - Middle Eastern residential IPs
  </Tab>
</Tabs>

<Frame>
  <img alt="World Region Configuration Interface" />
</Frame>

## Best Practices

**Choosing the Right Region:**

* Select the region closest to your target audience or where the content is most relevant
* Consider local time zones and business hours for time-sensitive scraping

**SDK vs Direct Connection:**

* Use the SDK for most projects (better error handling and connection management)
* Use direct WebSocket only for fine-grained control or custom framework integration

**Testing and Validation:**

* Test your region config with a simple request to verify IP location
* Monitor scraping results to ensure you're seeing expected regional content

## Troubleshooting

**Common Issues:**

* **Invalid region code:** Double-check you're using the correct region codes
* **No regional content difference:** Some sites may not vary by region. Verify with manual testing
* **Connection timeouts:** Proxy pools may vary by region; try another if needed

**Debugging Tips:**

* Use browser dev tools to inspect the IP address
* Test the same URL from different regions to confirm content variation
* Check Zenrows dashboard for logs and error details

<Tip>
  For more granular targeting, see [Country Targeting](/browser-sessions/features/country).
</Tip>

## Next Steps

* Learn about [Country Targeting](/browser-sessions/features/country) for precision
* Explore [Session TTL Configuration](/browser-sessions/features/session-ttl) to manage session duration
* See the <a href="https://github.com/ZenRows/browser-js-sdk">Zenrows SDK Repository</a> for advanced features and examples
* Review the [Practical Use Cases](/browser-sessions/help/practical-use-cases) for real world examples


# Integrating Zenrows Browser Sessions with Playwright
Source: https://docs.zenrows.com/browser-sessions/get-started/playwright

Step-by-step guide to connect Zenrows Browser Sessions with Playwright for Python and Node.js web scraping and browser automation.

Learn to extract data from any website using Zenrows' Browser Sessions (formerly, Scraping Browser) with Playwright. This guide walks you through creating your first browser-based scraping request that can handle complex JavaScript-heavy sites with full browser automation.

Zenrows' Browser Sessions provides cloud-based Chrome instances you can control using Playwright. Whether dealing with dynamic content, complex user interactions, or sophisticated anti-bot protection, you can get started in minutes with Playwright's powerful automation capabilities.

## 1. Set Up Your Project

### Set Up Your Development Environment

Before diving in, ensure you have the proper development environment and Playwright installed. Browser Sessions works seamlessly with both Python and Node.js versions of Playwright.

<Note>While previous versions may work, we recommend using the latest stable versions for optimal performance and security.</Note>

<Tabs>
  <Tab title="Python">
    Python 3+ installed (latest stable version recommended). Using an IDE like <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> or <a href="https://code.visualstudio.com/docs/languages/python">Visual Studio Code</a> with the Python extension is recommended.

    ```bash theme={"dark"}
    # Install Python (if not already installed)
    # Visit https://www.python.org/downloads/ or use package managers:

    # macOS (using Homebrew)
    brew install python

    # Ubuntu/Debian
    sudo apt update && sudo apt install python3 python3-pip

    # Windows (using Chocolatey)
    choco install python

    # Install Playwright
    pip install playwright
    playwright install
    ```

    <Tip>
      If you need help setting up your environment, check out our detailed [Playwright web scraping guide](https://www.zenrows.com/blog/playwright-scraping)
    </Tip>
  </Tab>

  <Tab title="Node.js">
    Node.js 18+ installed (latest LTS version recommended). Using an IDE like <a href="https://code.visualstudio.com/">Visual Studio Code</a> or <a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a> will enhance your coding experience.

    ```bash theme={"dark"}
    # Install Node.js (if not already installed)
    # Visit https://nodejs.org/ or use package managers:

    # macOS (using Homebrew)
    brew install node

    # Ubuntu/Debian (using NodeSource)
    curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
    sudo apt-get install -y nodejs

    # Windows (using Chocolatey)
    choco install nodejs

    # Install Playwright
    npm install playwright
    npx playwright install
    ```

    <Tip>
      If you need help setting up your environment, check out our detailed [Playwright web scraping guide](https://www.zenrows.com/blog/playwright-scraping)
    </Tip>
  </Tab>
</Tabs>

### Get Your API Key and Connection URL

[Sign Up](https://app.zenrows.com/register?prod=scraping_browser) for a free Zenrows account and get your API key from the [Browser Sessions dashboard](https://app.zenrows.com/browser). You'll need this key to authenticate your WebSocket connection.

## 2. Make Your First Request

Start with a simple request to understand how Browser Sessions works with Playwright. We'll use the <a href="https://www.scrapingcourse.com/ecommerce/">E-commerce Challenge</a> page to demonstrate how to connect to the browser and extract the page title.

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install playwright
  import asyncio
  from playwright.async_api import async_playwright

  # scraping browser connection URL
  connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

  async def scraper():
      async with async_playwright() as p:
          # connect to the scraping browser
          browser = await p.chromium.connect_over_cdp(connection_url)
          context = browser.contexts[0] if browser.contexts else await browser.new_context()
          page = await context.new_page()
          
          await page.goto('https://www.scrapingcourse.com/ecommerce/')
          print(await page.title())
          
          await browser.close()

  if __name__ == "__main__":
      asyncio.run(scraper())
  ```

  ```javascript Node.js theme={"dark"}
  // npm install playwright
  const { chromium } = require('playwright');

  // scraping browser connection URL
  const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

  const scraper = async () => {
      // connect to the scraping browser
      const browser = await chromium.connectOverCDP(connectionURL);
      const page = await browser.newPage();
      
      await page.goto('https://www.scrapingcourse.com/ecommerce/');
      console.log(await page.title());
      
      await browser.close();
  };

  scraper();
  ```
</CodeGroup>

Replace `YOUR_ZENROWS_API_KEY` with your actual API key and run the script:

<CodeGroup>
  ```bash Python theme={"dark"}
  python scraper.py
  ```

  ```bash Node.js theme={"dark"}
  node scraper.js
  ```
</CodeGroup>

### Expected Output

The script will print the page title:

```plaintext theme={"dark"}
ScrapingCourse.com E-commerce Challenge
```

Perfect! You've just made your first web scraping request with the Zenrows Browser Sessions using Playwright.

## 3. Build a Real-World Scraping Scenario

Let's scale up to a practical scraping scenario by extracting product information from the e-commerce site. Using Playwright's powerful selectors and data extraction methods, we'll modify our code to extract product names, prices, and URLs from the page.

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install playwright
  import asyncio
  from playwright.async_api import async_playwright

  # scraping browser connection URL
  connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

  async def scraper(url):
      async with async_playwright() as p:
          # connect to the scraping browser
          browser = await p.chromium.connect_over_cdp(connection_url)
          page = await browser.new_page()
          
          try:
              await page.goto(url)
              
              # extract the desired data
              await page.wait_for_selector(".product")
              products = await page.query_selector_all(".product")
              data = []
              
              for product in products:
                  name = await product.query_selector(".product-name")
                  price = await product.query_selector(".price")
                  product_url = await product.query_selector(".woocommerce-LoopProduct-link")
                  
                  data.append({
                      "name": await name.text_content() or "",
                      "price": await price.text_content() or "",
                      "productURL": await product_url.get_attribute("href") or "",
                  })
              
              return data
          except Exception as error:
              return error
          finally:
              await page.close()
              await browser.close()

  if __name__ == "__main__":
      url = "https://www.scrapingcourse.com/ecommerce/"
      products = asyncio.run(scraper(url))
      print(products)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install playwright
  const { chromium } = require('playwright');

  // connection URL
  const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

  const scraper = async (url) => {
      // connect to the scraping browser
      const browser = await chromium.connectOverCDP(connectionURL);
      const page = await browser.newPage();
      
      try {
          await page.goto(url);
          await page.waitForSelector('.product');
          
          // extract the desired data
          const data = await page.$$eval('.product', (products) =>
              products.map((product) => ({
                  name: product.querySelector('.product-name')?.textContent.trim() || '',
                  price: product.querySelector('.price')?.textContent.trim() || '',
                  productURL: product.querySelector('.woocommerce-LoopProduct-link')?.href || '',
              }))
          );
          
          return data;
      } finally {
          await page.close();
          await browser.close();
      }
  };

  // execute the scraper function
  (async () => {
      const url = 'https://www.scrapingcourse.com/ecommerce/';
      const products = await scraper(url);
      console.log(products);
  })();
  ```
</CodeGroup>

### Run Your Application

Execute your script to test the scraping functionality:

<CodeGroup>
  ```bash Python theme={"dark"}
  python scraper.py
  ```

  ```bash Node.js theme={"dark"}
  node scraper.js
  ```
</CodeGroup>

**Example Output**

The script will extract and display product information:

```json theme={"dark"}
[
    {
        "name": "Abominable Hoodie",
        "price": "$69.00",
        "productURL": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"
    },
    {
        "name": "Artemis Running Short",
        "price": "$45.00",
        "productURL": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short/"
    }
    // ... more products
]
```

Congratulations! 🎉 You've successfully built a real-world scraping scenario with Playwright and the Zenrows Browser Sessions.

## 4. Alternative: Using the Zenrows Browser SDK

For a more streamlined development experience, you can use the Zenrows Browser SDK instead of managing WebSocket URLs manually. The SDK simplifies connection management and provides additional utilities.

<Note>The Zenrows Browser SDK is currently only available for JavaScript. For more details, see the <a href="https://github.com/ZenRows/browser-js-sdk">GitHub Repository</a>.</Note>

### Install the SDK

```bash Node.js theme={"dark"}
npm install @zenrows/browser-sdk
```

### Quick Migration from WebSocket URL

If you have existing Playwright code using the WebSocket connection, migrating to the SDK requires minimal changes:

**Before (WebSocket URL):**

```javascript Node.js theme={"dark"}
const { chromium } = require('playwright');
const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

const browser = await chromium.connectOverCDP(connectionURL);
```

**After (SDK):**

```javascript Node.js theme={"dark"}
const { chromium } = require('playwright');
const { ScrapingBrowser } = require('@zenrows/browser-sdk');

const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
const connectionURL = scrapingBrowser.getConnectURL();
const browser = await chromium.connectOverCDP(connectionURL);
```

### Complete Example with SDK

```javascript Node.js theme={"dark"}
// npm install @zenrows/browser-sdk playwright
const { chromium } = require('playwright');
const { ScrapingBrowser } = require('@zenrows/browser-sdk');

const scraper = async () => {
    // Initialize SDK
    const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
    const connectionURL = scrapingBrowser.getConnectURL();
    
    const browser = await chromium.connectOverCDP(connectionURL);
    const page = await browser.newPage();
    
    await page.goto('https://www.scrapingcourse.com/ecommerce/');
    console.log(await page.title());
    
    await browser.close();
};

scraper();
```

### SDK Benefits

* **Simplified configuration:** No need to manually construct WebSocket URLs
* **Better error handling:** Built-in error messages and debugging information
* **Future-proof:** Automatic updates to connection protocols and endpoints
* **Additional utilities:** Access to helper methods and advanced configuration options

<Tip>The SDK is particularly useful for production environments where you want cleaner code organization and better error handling.</Tip>

## How Playwright with Browser Sessions Helps

Combining Playwright with Zenrows' Browser Sessions provides powerful advantages for web scraping:

### Key Benefits

* **Cloud-based browser instances:** Run Playwright scripts on remote Chrome instances, freeing up local resources for other tasks.

* **Seamless integration:** Connect your existing Playwright code to Zenrows with just a WebSocket URL change - no complex setup required.

* **Advanced automation:** Use Playwright's full feature set, which includes page interactions, form submissions, file uploads, and complex user workflows.

* **Built-in anti-detection:** Benefit from residential proxy rotation and genuine browser fingerprints automatically.

* **Cross-browser support:** While we use Chromium for optimal compatibility, Playwright's API remains consistent across different browser engines.

* **High concurrency:** Scale your Playwright scripts with up to 400 concurrent browser instances on Scale, more on Enterprise.

* **Reliable execution:** Cloud infrastructure ensures consistent performance without local browser management overhead.

## Troubleshooting

Below are common issues you might encounter when using Playwright with Browser Sessions:

<Steps>
  <Step title="Connection Refused">
    If you receive a Connection Refused error, it might be due to:

    * **API Key Issues:** Verify that you're using the correct API key.
    * **Network Issues:** Check your internet connection and firewall settings.
    * **WebSocket Endpoint:** Ensure that the WebSocket URL (`wss://browser.zenrows.com`) is correct.
  </Step>

  <Step title="Empty Data or Timeout Errors">
    * Use `page.waitForSelector()` to ensure elements load before extraction

    * Increase timeout values for slow-loading pages

      ```javascript scraper.js theme={"dark"}
      await page.goto('https://example.com', { timeout: 60000 });  // 60 seconds
      ```

    * Verify CSS selectors are correct using browser developer tools

    * Add `page.waitForLoadState('networkidle')` for dynamic content
  </Step>

  <Step title="Browser Context Issues">
    * Use existing context when available: `browser.contexts[0] if browser.contexts else await browser.new_context()`
    * Properly close pages and browsers to prevent resource leaks
    * Handle exceptions properly to ensure cleanup occurs
  </Step>

  <Step title="Geolocation Blocks">
    Although Zenrows rotates IPs, some websites may block them based on location. Try adjusting the region or country settings.

    <Info>
      For more information, check our Browser Sessions [Region Documentation](https://docs.zenrows.com/browser-sessions/features/world-region) and [Country Documentation](https://docs.zenrows.com/browser-sessions/features/country).
    </Info>
  </Step>

  <Step title="Get Help From Zenrows Experts">
    Our support team is available to assist you if issues persist despite following these solutions. Use the [Browser Sessions dashboard](https://app.zenrows.com/browser) or <a href="mailto:success@zenrows.com">email us</a> for personalized help from Zenrows experts.
  </Step>
</Steps>

## Next Steps

You now have a solid foundation for Playwright-based web scraping with Zenrows. Here are some recommended next steps:

* **[Practical Use Cases](/browser-sessions/help/practical-use-cases)**: <br />Learn common scraping patterns, including screenshots, custom JavaScript execution, and form handling.
* **[Complete Browser Sessions Documentation](/browser-sessions/introduction)**: <br />Explore all available features and advanced configuration options for Browser Sessions.
* **[Playwright Web Scraping Guide](https://www.zenrows.com/blog?q=playwright)**: <br />Dive deeper into Playwright techniques for sophisticated scraping scenarios.
* **[Pricing and Plans](/first-steps/pricing)**: <br />Understand how browser usage is calculated and choose the plan that fits your scraping volume.

## Frequently Asked Questions (FAQ)

<Accordion title="Can I use Zenrows Browser Sessions with Puppeteer?">
  Yes, Zenrows Browser Sessions is compatible with Puppeteer as well. The integration is similar, requiring only a change in the connection method to utilize our WebSocket endpoint.
  <Tip>For detailed instructions, refer to our [Puppeteer Integration article](/browser-sessions/get-started/puppeteer).</Tip>
</Accordion>

<Accordion title="Do I need to configure proxies manually with Zenrows Browser Sessions?">
  No, Zenrows Browser Sessions handles proxy configuration and IP rotation automatically. You don't need to set up proxies manually.
</Accordion>

<Accordion title="Can Browser Sessions solve CAPTCHAs?">
  Currently, Zenrows Browser Sessions does not support CAPTCHA solving. For handling CAPTCHAs, you may need to use third-party services.
  <Tip>Consider using our [Fetch](/fetch/api-reference) for additional features like CAPTCHA solving, advanced anti-bot bypass, and more.</Tip>
</Accordion>

<Accordion title="Can I use all Playwright features with Browser Sessions?">
  Yes! Browser Sessions supports the full Playwright API. You can use page interactions, screenshots, PDF generation, network interception, and all other Playwright features seamlessly.
</Accordion>

<Accordion title="How do I handle multiple pages or tabs?">
  You can create multiple pages within the same browser context using `await browser.newPage()` or `await context.newPage()`. Each page operates independently while sharing the same browser session.
</Accordion>

<Accordion title="Can I use Playwright's built-in retry mechanisms?">
  Absolutely! Playwright's `expect()` assertions, `waitForSelector()`, and other built-in retry mechanisms work perfectly with Browser Sessions. These features help handle dynamic content and improve scraping reliability.
</Accordion>

<Accordion title="How do I take screenshots with Playwright and Browser Sessions?">
  Use Playwright's standard screenshot methods:

  ```python theme={"dark"}
      await page.screenshot(path='screenshot.png')
  ```

  The screenshot will be saved locally while the browser runs in the cloud.
</Accordion>

<Accordion title="Can I use Playwright's network interception features?">
  Yes! You can intercept requests, modify responses, and monitor network traffic using Playwright's `page.route()` and `page.on('request')` methods with Browser Sessions.
</Accordion>

<Accordion title="What's the difference between using Playwright locally vs. with Browser Sessions?">
  The main difference is that the browser runs in Zenrows' cloud infrastructure instead of locally. This provides better IP rotation, fingerprint management, and resource efficiency while maintaining the exact same Playwright API.
</Accordion>

<Accordion title="How do I handle file downloads with Playwright and Browser Sessions?">
  File downloads work with Playwright's standard download handling. Use `page.waitForDownload()` and the download will be transferred from the cloud browser to your local environment automatically.
</Accordion>


# Integrating Zenrows Browser Sessions with Puppeteer
Source: https://docs.zenrows.com/browser-sessions/get-started/puppeteer

Step-by-step tutorial for integrating Zenrows Browser Sessions with Puppeteer in Node.js and Pyppeteer in Python for web scraping.

Discover how to scrape data from any website using Zenrows' Browser Sessions (formerly, Scraping Browser) with Puppeteer. This comprehensive guide demonstrates how to create your first browser automation request capable of handling JavaScript-heavy sites and bypassing sophisticated anti-bot measures.

Zenrows' Browser Sessions offers cloud-hosted Chrome instances that integrate seamlessly with Puppeteer's automation framework. From scraping dynamic content to performing complex browser interactions, you can build robust scraping solutions in minutes using Puppeteer's intuitive API.

## 1. Set Up Your Project

### Set Up Your Development Environment

Ensure you have the necessary development tools and Puppeteer installed before starting. Browser Sessions supports both Node.js Puppeteer and Python Pyppeteer implementations.

<Note>We recommend using the latest stable versions to ensure optimal compatibility and access to the newest features.</Note>

<Tabs>
  <Tab title="Node.js">
    Node.js 18+ installed (latest LTS version recommended). Consider using an IDE like <a href="https://code.visualstudio.com/">Visual Studio Code</a> or <a href="https://www.jetbrains.com/webstorm/">WebStorm</a> for enhanced development experience.

    ```bash theme={"dark"}
        # Install Node.js (if not already installed)
        # Visit https://nodejs.org/ or use package managers:
        
        # macOS (using Homebrew)
        brew install node
        
        # Ubuntu/Debian (using NodeSource)
        curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
        sudo apt-get install -y nodejs
        
        # Windows (using Chocolatey)
        choco install nodejs

        # Install Puppeteer Core
        npm install puppeteer-core
    ```

    <Tip>
      Need help with your setup? Check out our comprehensive [Puppeteer web scraping guide](https://www.zenrows.com/blog/puppeteer-web-scraping)
    </Tip>
  </Tab>

  <Tab title="Python">
    Python 3+ installed (latest stable version recommended). IDEs like <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> or <a href="https://code.visualstudio.com/docs/languages/python">Visual Studio Code</a> with Python extensions provide excellent development support.

    ```bash theme={"dark"}
    # Install Python (if not already installed)
    # Visit https://www.python.org/downloads/ or use package managers:

    # macOS (using Homebrew)
    brew install python

    # Ubuntu/Debian
    sudo apt update && sudo apt install python3 python3-pip

    # Windows (using Chocolatey)
    choco install python

    # Install Pyppeteer
    pip install pyppeteer
     
    ```

    <Tip>
      For detailed setup instructions, refer to our [Pyppeteer scraping tutorial](https://www.zenrows.com/blog/pyppeteer)
    </Tip>
  </Tab>
</Tabs>

### Get Your API Key and Connection URL

[Create a Free Account](https://app.zenrows.com/register?prod=scraping_browser) with Zenrows and retrieve your API key from the [Browser Sessions Dashboard](https://app.zenrows.com/browser). This key authenticates your WebSocket connection to our cloud browsers.

## 2. Make Your First Request

Begin with a basic request to familiarize yourself with how Puppeteer connects to Browser Sessions. We'll target the <a href="https://www.scrapingcourse.com/ecommerce/">E-commerce Challenge</a> page to demonstrate browser connection and title extraction.

<CodeGroup>
  ```javascript Node.js theme={"dark"}
  // npm install puppeteer-core
  const puppeteer = require('puppeteer-core');
  // scraping browser connection URL
  const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

  const scraper = async () => {
      // connect to the scraping browser
      const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
      const page = await browser.newPage();
      
      await page.goto('https://www.scrapingcourse.com/ecommerce/');
      console.log(await page.title());
      
      await browser.close();
  };

  scraper();
  ```

  ```python Python theme={"dark"}
  # pip install pyppeteer
  import asyncio
  from pyppeteer import connect

  # scraping browser connection URL
  connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

  async def scraper():
      # connect to the scraping browser
      browser = await connect(browserWSEndpoint=connection_url)
      page = await browser.newPage()
      
      await page.goto('https://www.scrapingcourse.com/ecommerce/')
      print(await page.title())
      
      await page.close()
      await browser.disconnect()

  if __name__ == "__main__":
      asyncio.run(scraper())
  ```
</CodeGroup>

Replace `YOUR_ZENROWS_API_KEY` with your actual API key and execute the script:

<CodeGroup>
  ```bash Node.js theme={"dark"}
  node scraper.js
  ```

  ```bash Python theme={"dark"}
  python scraper.py
  ```
</CodeGroup>

**Expected Output:**

Your script will display the page title:

```plaintext theme={"dark"}
ScrapingCourse.com E-commerce Challenge
```

Excellent! You've successfully completed your first web scraping request using Zenrows Browser Sessions with Puppeteer.

## 3. Build a Real-World Scraping Scenario

Now let's advance to a comprehensive scraping example by extracting product data from the e-commerce site. We'll enhance our code to collect product names, prices, and URLs using Puppeteer's robust element selection and data extraction capabilities.

<CodeGroup>
  ```javascript Node.js theme={"dark"}
  // npm install puppeteer-core
  const puppeteer = require('puppeteer-core');

  // scraping browser connection URL
  const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

  const scraper = async (url) => {
      // connect to the scraping browser
      const browser = await puppeteer.connect({
          browserWSEndpoint: connectionURL,
      });
      const page = await browser.newPage();
      
      try {
          await page.goto(url, { waitUntil: 'networkidle2' });
          await page.waitForSelector('.product');
          
          // extract the desired data
          const data = await page.$$eval('.product', (products) =>
              products.map((product) => ({
                  name: product.querySelector('.product-name')?.textContent.trim() || '',
                  price: product.querySelector('.price')?.textContent.trim() || '',
                  productURL: product.querySelector('.woocommerce-LoopProduct-link')?.href || '',
              }))
          );
          
          return data;
      } finally {
          await page.close();
          await browser.close();
      }
  };

  // execute the scraper function
  (async () => {
      const url = 'https://www.scrapingcourse.com/ecommerce/';
      const products = await scraper(url);
      console.log(products);
  })();
  ```

  ```python Python theme={"dark"}
  # pip install pyppeteer
  import asyncio
  from pyppeteer import connect

  # scraping browser connection URL
  connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

  async def scraper(url):
      # connect to the scraping browser
      browser = await connect(browserWSEndpoint=connection_url)
      page = await browser.newPage()
      
      try:
          await page.goto(url, {"waitUntil": "networkidle2"})
          await page.waitForSelector(".product")
          
          # extract the desired data
          product_elements = await page.querySelectorAll(".product")
          products = []
          
          for product in product_elements:
              name_el = await product.querySelector(".product-name")
              name = await page.evaluate("(el) => el.textContent.trim()", name_el) if name_el else ""
              
              price_el = await product.querySelector(".price")
              price = await page.evaluate("(el) => el.textContent.trim()", price_el) if price_el else ""
              
              url_el = await product.querySelector(".woocommerce-LoopProduct-link")
              product_url = await page.evaluate("(el) => el.href", url_el) if url_el else ""
              
              products.append({
                  "name": name,
                  "price": price,
                  "productURL": product_url,
              })
          
          return products
      finally:
          await page.close()
          await browser.disconnect()

  # execute the scraper function
  if __name__ == "__main__":
      url = "https://www.scrapingcourse.com/ecommerce/"
      products = asyncio.run(scraper(url))
      print(products)
  ```
</CodeGroup>

### Run Your Application

Launch your script to verify the scraping functionality:

<CodeGroup>
  ```bash Node.js theme={"dark"}
  node scraper.js
  ```

  ```bash Python theme={"dark"}
  python scraper.py
  ```
</CodeGroup>

**Example Output:**

Your script will collect and display the product information:

```json theme={"dark"}
[
    {
        "name": "Abominable Hoodie",
        "price": "$69.00",
        "productURL": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"
    },
    {
        "name": "Artemis Running Short",
        "price": "$45.00",
        "productURL": "https://www.scrapingcourse.com/ecommerce/product/artemis-running-short/"
    }
    // ... additional products
]
```

Outstanding! 🎉 You've successfully implemented a production-ready scraping solution using Puppeteer and the Zenrows Browser Sessions.

## 4. Alternative: Using the Zenrows Browser SDK

For enhanced developer experience, consider using the Zenrows Browser SDK rather than manually managing WebSocket URLs. The SDK streamlines connection handling and offers additional development utilities.

<Note>The Zenrows Browser SDK is currently only available for JavaScript. For more details, see the <a href="https://github.com/ZenRows/browser-js-sdk">GitHub Repository</a>.</Note>

### Install the SDK

```bash Node.js theme={"dark"}
npm install @zenrows/browser-sdk
```

### Quick Migration from WebSocket URL

Transitioning from direct WebSocket connections to the SDK requires minimal code changes:

**Before (WebSocket URL):**

```javascript Node.js theme={"dark"}
const puppeteer = require('puppeteer-core');
const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
```

**After (SDK):**

```javascript Node.js theme={"dark"}
const puppeteer = require('puppeteer-core');
const { ScrapingBrowser } = require('@zenrows/browser-sdk');

const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
const connectionURL = scrapingBrowser.getConnectURL();
const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
```

### Complete Example with SDK

```javascript Node.js theme={"dark"}
// npm install @zenrows/browser-sdk puppeteer-core
const puppeteer = require('puppeteer-core');
const { ScrapingBrowser } = require('@zenrows/browser-sdk');

const scraper = async () => {
    // Initialize SDK
    const scrapingBrowser = new ScrapingBrowser({ apiKey: 'YOUR_ZENROWS_API_KEY' });
    const connectionURL = scrapingBrowser.getConnectURL();
    
    const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
    const page = await browser.newPage();
    
    await page.goto('https://www.scrapingcourse.com/ecommerce/');
    console.log(await page.title());
    
    await browser.close();
};

scraper();
```

### SDK Benefits

* **Streamlined configuration:** Eliminates manual WebSocket URL construction
* **Enhanced error handling:** Provides detailed error messages and debugging capabilities
* **Future-proof architecture:** Automatic updates to connection protocols and endpoints
* **Extended utilities:** Access to helper functions and advanced configuration options

<Tip>The SDK proves especially valuable in production environments where code maintainability and robust error handling are priorities.</Tip>

## How Puppeteer with Browser Sessions Helps

Integrating Puppeteer with Zenrows' Browser Sessions delivers significant advantages for web automation:

### Key Benefits

* **Cloud-hosted browser instances:** Execute Puppeteer scripts on remote Chrome browsers, preserving local system resources for other applications.
* **Drop-in replacement:** Transform existing Puppeteer code to use Zenrows by simply changing the connection method - no architectural changes required.
* **Full automation capabilities:** Leverage Puppeteer's complete feature set including form interactions, file handling, network monitoring, and custom JavaScript execution.
* **Automatic anti-detection:** Benefit from built-in residential proxy rotation and authentic browser fingerprints without additional configuration.
* **Proven reliability:** Cloud infrastructure delivers consistent performance without the complexity of local browser management.
* **Massive scalability:** Execute up to 400 concurrent browser instances on Scale, more on Enterprise.
* **Network optimization:** Reduced latency and improved success rates through globally distributed infrastructure.

## Troubleshooting

Common challenges when integrating Puppeteer with Browser Sessions and their solutions:

<Steps>
  <Step title="Connection Refused">
    If you encounter Connection Refused errors, verify these potential causes:

    * **API Key Validation:** Confirm you're using the correct API key from your dashboard.
    * **Network Connectivity:** Check your internet connection and firewall configurations.
    * **WebSocket Endpoint:** Ensure the WebSocket URL (`wss://browser.zenrows.com`) is properly formatted.
  </Step>

  <Step title="Timeout and Loading Issues">
    * Use `page.waitForSelector()` to ensure elements are available before interaction

    * Extend timeout values for slow-loading websites

      ```javascript Node.js theme={"dark"}
      await page.goto('https://example.com', { timeout: 60000 });  // 60 seconds
      ```

    * Validate CSS selectors using browser developer tools

    * Implement `waitUntil: 'networkidle2'` for dynamic content loading
  </Step>

  <Step title="Page Navigation Errors">
    * Handle navigation exceptions with proper try-catch blocks
    * Ensure proper browser and page cleanup to prevent memory leaks
    * Use `page.waitForNavigation()` for multi-step workflows
  </Step>

  <Step title="Geographic Restrictions">
    While Zenrows automatically rotates IP addresses, some websites implement location-based blocking. Consider adjusting regional settings for better access.

    <Info>
      Learn more about geographic targeting in our [Region Documentation](https://docs.zenrows.com/browser-sessions/features/world-region) and [Country Configuration](https://docs.zenrows.com/browser-sessions/features/country).
    </Info>
  </Step>

  <Step title="Get Help From Zenrows Experts">
    If challenges persist after implementing these solutions, our technical support team is ready to assist. Access help through the [Browser Sessions dashboard](https://app.zenrows.com/browser) or <a href="mailto:success@zenrows.com">contact our support team</a> for expert guidance.
  </Step>
</Steps>

## Next Steps

You've established a strong foundation for Puppeteer-based web scraping with Zenrows. Continue your journey with these resources:

* **[Practical Use Cases](/browser-sessions/help/practical-use-cases)**: <br />Explore common automation patterns including screenshot capture, custom JavaScript execution, and form interactions.
* **[Complete Browser Sessions Documentation](/browser-sessions/introduction)**: <br />Discover all available features and configuration options for Browser Sessions platform.
* **[Puppeteer Web Scraping Guide](https://www.zenrows.com/blog?q=puppeteer)**: <br />Explore advanced Puppeteer techniques for complex scraping challenges.
* **[Pricing and Plans](/first-steps/pricing)**: <br />Learn about browser usage calculations and select the optimal plan for your requirements.

## Frequently Asked Questions (FAQ)

<Accordion title="Can I use Zenrows Browser Sessions with Playwright?">
  Absolutely! Zenrows Browser Sessions supports both Puppeteer and Playwright automation frameworks. The integration process is similar, requiring only connection method adjustments.

  <Tip>For comprehensive instructions, see our [Playwright Integration guide](/browser-sessions/get-started/playwright).</Tip>
</Accordion>

<Accordion title="Do I need to manage proxies manually with Zenrows Browser Sessions?">
  No manual proxy configuration is required. Zenrows Browser Sessions automatically handles proxy management and IP rotation behind the scenes.
</Accordion>

<Accordion title="Does Browser Sessions handle CAPTCHA challenges?">
  Currently, Zenrows Browser Sessions doesn't include built-in CAPTCHA solving capabilities. For CAPTCHA handling, consider integrating third-party CAPTCHA solving services.

  <Tip>Explore our [Fetch](/fetch/api-reference) for additional features including CAPTCHA solving and advanced anti-bot bypass mechanisms.</Tip>
</Accordion>

<Accordion title="Can I access all Puppeteer features through Browser Sessions?">
  Yes! Browser Sessions provides full access to Puppeteer's API, including page manipulation, screenshot generation, PDF creation, network interception, and all other native features.
</Accordion>

<Accordion title="How do I manage multiple browser tabs or pages?">
  Create additional pages using `await browser.newPage()` within the same browser instance. Each page operates independently while sharing the browser session and resources.
</Accordion>

<Accordion title="Can I use Puppeteer's built-in waiting mechanisms?">
  Certainly! Puppeteer's `waitForSelector()`, `waitForNavigation()`, and other waiting functions work seamlessly with Browser Sessions, helping ensure reliable data extraction from dynamic content.
</Accordion>

<Accordion title="How do I capture screenshots with Puppeteer and Browser Sessions?">
  Use Puppeteer's standard screenshot functionality:

  ```javascript theme={"dark"}
      await page.screenshot({ path: 'screenshot.png' });
  ```

  Screenshots are captured from the cloud browser and saved to your local environment automatically.
</Accordion>

<Accordion title="Can I monitor network requests with Puppeteer and Browser Sessions?">
  Yes! Puppeteer's network monitoring capabilities, including `page.on('request')` and `page.on('response')` event handlers, function normally with Browser Sessions.
</Accordion>

<Accordion title="What's the main difference between local Puppeteer and Browser Sessions?">
  The primary distinction is execution location: browsers run in Zenrows' cloud infrastructure rather than locally. This provides superior IP management, fingerprint diversity, and resource efficiency while maintaining identical Puppeteer API functionality.
</Accordion>

<Accordion title="How do I handle file downloads with Puppeteer and Browser Sessions?">
  File downloads work through Puppeteer's standard download handling mechanisms. Files are downloaded to the cloud browser and then transferred to your local environment automatically.
</Accordion>


# Browser Sessions Practical Use Cases
Source: https://docs.zenrows.com/browser-sessions/help/practical-use-cases

Practical Browser Sessions examples including screenshots, form filling, infinite scroll, network interception, and JavaScript execution.

Discover common automation patterns and real-world scenarios when using Zenrows' Browser Sessions (formerly, Scraping Browser) with Puppeteer and Playwright. These practical examples demonstrate how to leverage browser automation for various data extraction and interaction tasks.

Browser Sessions excels at handling complex scenarios that traditional HTTP-based scraping cannot address. From capturing visual content to executing custom JavaScript, these use cases showcase the full potential of browser-based automation for your scraping projects.

<Note>**Websites often change their structure or update CSS class names and HTML tags.** This means the selectors you use for scraping (like `.product`, `.products`, or specific element tags) might stop working if the site layout changes. To keep your scraper reliable, regularly check and update your selectors as needed.</Note>

## Navigation and Page Content Extraction

Extract complete page content and metadata by navigating to target websites. This fundamental pattern forms the foundation for most scraping workflows and demonstrates how to retrieve both visible content and underlying HTML structure.

<Tabs>
  <Tab title="Puppeteer">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const puppeteer = require('puppeteer-core');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const scraper = async () => {
          const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/', { 
                  waitUntil: 'domcontentloaded' 
              });
              
              // Extract page metadata
              const title = await page.title();
              console.log('Page title:', title);
              
              // Get complete HTML content
              console.log('Extracting page content...');
              const html = await page.content();
              
              // Extract specific elements
              const productCount = await page.$$eval('.product', products => products.length);
              console.log(`Found ${productCount} products on the page`);
              
              // Extract text content from specific elements
              const headings = await page.$$eval('h1, h2, h3', elements => 
                  elements.map(el => el.textContent.trim())
              );
              console.log('Page headings:', headings);
              
              return {
                  title,
                  productCount,
                  headings,
                  htmlLength: html.length
              };
          } finally {
              await browser.close();
          }
      };

      scraper().then(result => console.log('Extraction complete:', result));
      ```

      ```python Python theme={"dark"}
      # pip install pyppeteer
      import asyncio
      from pyppeteer import connect

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def scraper():
          browser = await connect(browserWSEndpoint=connection_url)
          page = await browser.newPage()
          
          try:
              print('Navigating to target page...')
              await page.goto('https://www.scrapingcourse.com/ecommerce/', {
                  'waitUntil': 'domcontentloaded'
              })
              
              # Extract page metadata
              title = await page.title()
              print(f'Page title: {title}')
              
              # Get complete HTML content
              print('Extracting page content...')
              html = await page.content()
              
              # Extract specific elements
              product_count = await page.querySelectorAllEval('.product', 'products => products.length')
              print(f'Found {product_count} products on the page')
              
              # Extract text content from specific elements
              headings = await page.querySelectorAllEval('h1, h2, h3', '''elements => 
                  elements.map(el => el.textContent.trim())
              ''')
              print(f'Page headings: {headings}')
              
              return {
                  'title': title,
                  'product_count': product_count,
                  'headings': headings,
                  'html_length': len(html)
              }
          finally:
              await page.close()
              await browser.disconnect()

      if __name__ == "__main__":
          result = asyncio.run(scraper())
          print('Extraction complete:', result)
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Playwright">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const { chromium } = require('playwright');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const scraper = async () => {
          const browser = await chromium.connectOverCDP(connectionURL);
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/');
              
              // Extract page metadata
              const title = await page.title();
              console.log('Page title:', title);
              
              // Get complete HTML content
              console.log('Extracting page content...');
              const html = await page.content();
              
              // Extract specific elements
              const products = await page.locator('.product').count();
              console.log(`Found ${products} products on the page`);
              
              // Extract text content from headings
              const headings = await page.locator('h1, h2, h3').allTextContents();
              console.log('Page headings:', headings);
              
              return {
                  title,
                  productCount: products,
                  headings,
                  htmlLength: html.length
              };
          } finally {
              await browser.close();
          }
      };

      scraper().then(result => console.log('Extraction complete:', result));
      ```

      ```python Python theme={"dark"}
      import asyncio
      from playwright.async_api import async_playwright

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def scraper():
          async with async_playwright() as p:
              browser = await p.chromium.connect_over_cdp(connection_url)
              page = await browser.new_page()
              
              try:
                  print('Navigating to target page...')
                  await page.goto('https://www.scrapingcourse.com/ecommerce/')
                  
                  # Extract page metadata
                  title = await page.title()
                  print(f'Page title: {title}')
                  
                  # Get complete HTML content
                  print('Extracting page content...')
                  html = await page.content()
                  
                  # Extract specific elements
                  products = await page.locator('.product').count()
                  print(f'Found {products} products on the page')
                  
                  # Extract text content from headings
                  headings = await page.locator('h1, h2, h3').all_text_contents()
                  print(f'Page headings: {headings}')
                  
                  return {
                      'title': title,
                      'product_count': products,
                      'headings': headings,
                      'html_length': len(html)
                  }
              finally:
                  await browser.close()

      if __name__ == "__main__":
          result = asyncio.run(scraper())
          print('Extraction complete:', result)
      ```
    </CodeGroup>
  </Tab>
</Tabs>

### Key Benefits

* **Complete content access:** Retrieve both rendered content and raw HTML source
* **Metadata extraction:** Access page titles, descriptions, and other document properties
* **Element counting:** Quickly assess page structure and content volume
* **Structured data collection:** Extract specific elements using CSS selectors

## Taking Screenshots

Capture visual representations of web pages for monitoring, documentation, or visual verification purposes. Screenshots prove invaluable for debugging scraping workflows and creating visual records of dynamic content.

<Tabs>
  <Tab title="Puppeteer">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const puppeteer = require('puppeteer-core');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const screenshotScraper = async () => {
          const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/', { 
                  waitUntil: 'domcontentloaded' 
              });
              
              console.log('Page loaded:', await page.title());
              
              // Take full page screenshot
              console.log('Capturing full page screenshot...');
              await page.screenshot({ 
                  path: 'full-page-screenshot.png',
                  fullPage: true 
              });
              
              // Take viewport screenshot (uses default 1920x1080 viewport)
              console.log('Capturing viewport screenshot...');
              await page.screenshot({ 
                  path: 'viewport-screenshot.png' 
              });
              
              // Take screenshot of specific element
              console.log('Capturing product grid screenshot...');
              const productGrid = await page.$('.products');
              if (productGrid) {
                  await productGrid.screenshot({ 
                      path: 'product-grid-screenshot.png' 
                  });
              }
              
              // Take screenshot with custom clipping (alternative to viewport resizing)
              console.log('Capturing custom-sized screenshot...');
              await page.screenshot({
                  path: 'custom-size-screenshot.png',
                  type: 'jpeg',
                  quality: 100,
                  clip: { x: 0, y: 0, width: 1200, height: 800 }
              });
              
              console.log('All screenshots saved successfully');
              
          } finally {
              await browser.close();
          }
      };

      screenshotScraper();
      ```

      ```python Python theme={"dark"}
      # pip install pyppeteer
      import asyncio
      from pyppeteer import connect

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def screenshot_scraper():
          browser = await connect(browserWSEndpoint=connection_url)
          page = await browser.newPage()
          
          try:
              print('Navigating to target page...')
              await page.goto('https://www.scrapingcourse.com/ecommerce/', {
                  'waitUntil': 'domcontentloaded'
              })
              
              print(f'Page loaded: {await page.title()}')
              
              # Take full page screenshot
              print('Capturing full page screenshot...')
              await page.screenshot({
                  'path': 'full-page-screenshot.png',
                  'fullPage': True
              })
              
              # Take viewport screenshot (uses default 1920x1080 viewport)
              print('Capturing viewport screenshot...')
              await page.screenshot({
                  'path': 'viewport-screenshot.png'
              })
              
              # Take screenshot of specific element
              print('Capturing product grid screenshot...')
              product_grid = await page.querySelector('.products')
              if product_grid:
                  await product_grid.screenshot({
                      'path': 'product-grid-screenshot.png'
                  })
              
              # Take high-quality JPEG screenshot
              print('Capturing high-quality JPEG screenshot...')
              await page.screenshot({
                  'path': 'high-quality-screenshot.jpg',
                  'type': 'jpeg',
                  'quality': 90,
                  'fullPage': True
              })
              
              # Take screenshot with custom clipping (alternative to viewport resizing)
              print('Capturing custom-sized screenshot...')
              await page.screenshot({
                  'path': 'custom-size-screenshot.png',
                  'type': 'png',
                  'fullPage': True,
                  'clip': {'x': 0, 'y': 0, 'width': 1200, 'height': 800}
              })
              
              print('All screenshots saved successfully')
              
          finally:
              await page.close()
              await browser.disconnect()

      if __name__ == "__main__":
          asyncio.run(screenshot_scraper())
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Playwright">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const { chromium } = require('playwright');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const screenshotScraper = async () => {
          const browser = await chromium.connectOverCDP(connectionURL);
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/', {
                  waitUntil: 'domcontentloaded'
              });
              
              // Wait for page to be fully loaded
              await page.waitForSelector('body', { timeout: 10000 });
              
              console.log('Page loaded:', await page.title());
              
              // Take full page screenshot
              console.log('Capturing full page screenshot...');
              await page.screenshot({ 
                  path: 'full-page-screenshot.png',
                  fullPage: true 
              });
              
              // Take viewport screenshot (uses default 1920x1080 viewport)
              console.log('Capturing viewport screenshot...');
              await page.screenshot({ 
                  path: 'viewport-screenshot.png' 
              });
              
              // Take screenshot of specific element (with error handling)
              console.log('Capturing product grid screenshot...');
              try {
                  await page.waitForSelector('.products', { timeout: 5000 });
                  await page.locator('.products').screenshot({ 
                      path: 'product-grid-screenshot.png' 
                  });
              } catch (error) {
                  console.log('Product grid not found, skipping element screenshot');
              }
              
              // Take high-quality JPEG screenshot
              console.log('Capturing high-quality JPEG screenshot...');
              await page.screenshot({
                  path: 'high-quality-screenshot.jpg',
                  type: 'jpeg',
                  quality: 90,
                  fullPage: true
              });
              
              // Take screenshot with custom clipping (alternative to viewport resizing)
              console.log('Capturing custom-sized screenshot...');
              await page.screenshot({
                  path: 'custom-size-screenshot.png',
                  type: 'png',
                  clip: { x: 0, y: 0, width: 1200, height: 800 }
              });
              
              console.log('All screenshots saved successfully');
              
          } catch (error) {
              console.error('Screenshot scraper error:', error.message);
              
              // Take emergency screenshot if navigation fails
              try {
                  await page.screenshot({ path: 'error-screenshot.png' });
                  console.log('Emergency screenshot saved');
              } catch (screenshotError) {
                  console.error('Could not take emergency screenshot:', screenshotError.message);
              }
          } finally {
              await browser.close();
          }
      };

      screenshotScraper();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from playwright.async_api import async_playwright

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def screenshot_scraper():
          async with async_playwright() as p:
              browser = await p.chromium.connect_over_cdp(connection_url)
              page = await browser.new_page()
              
              try:
                  print('Navigating to target page...')
                  await page.goto('https://www.scrapingcourse.com/ecommerce/', 
                      wait_until='domcontentloaded'
                  )
                  
                  # Wait for page to be fully loaded
                  await page.wait_for_selector('body', timeout=10000)
                  
                  print(f'Page loaded: {await page.title()}')
                  
                  # Take full page screenshot
                  print('Capturing full page screenshot...')
                  await page.screenshot(path='full-page-screenshot.png', full_page=True)
                  
                  # Take viewport screenshot (uses default 1920x1080 viewport)
                  print('Capturing viewport screenshot...')
                  await page.screenshot(path='viewport-screenshot.png')
                  
                  # Take screenshot of specific element (with error handling)
                  print('Capturing product grid screenshot...')
                  try:
                      await page.wait_for_selector('.products', timeout=5000)
                      await page.locator('.products').screenshot(path='product-grid-screenshot.png')
                  except Exception:
                      print('Product grid not found, skipping element screenshot')
                  
                  # Take high-quality JPEG screenshot
                  print('Capturing high-quality JPEG screenshot...')
                  await page.screenshot(
                      path='high-quality-screenshot.jpg',
                      type='jpeg',
                      quality=90,
                      full_page=True
                  )
                  
                  # Take screenshot with custom clipping (alternative to viewport resizing)
                  print('Capturing custom-sized screenshot...')
                  await page.screenshot(
                      path='custom-size-screenshot.png',
                      type='png',
                      clip={'x': 0, 'y': 0, 'width': 1200, 'height': 800}
                  )
                  
                  print('All screenshots saved successfully')
                  
              except Exception as error:
                  print(f'Screenshot scraper error: {error}')
                  
                  # Take emergency screenshot if navigation fails
                  try:
                      await page.screenshot(path='error-screenshot.png')
                      print('Emergency screenshot saved')
                  except Exception as screenshot_error:
                      print(f'Could not take emergency screenshot: {screenshot_error}')
              finally:
                  await browser.close()

      if __name__ == "__main__":
          asyncio.run(screenshot_scraper())
      ```
    </CodeGroup>
  </Tab>
</Tabs>

### Screenshot Options

* **Full page capture:** Include content below the fold with `fullPage: true`
* **Element-specific screenshots:** Target individual components or sections
* **Custom clipping:** Focus on specific page areas using coordinate-based clipping
* **Format options:** PNG (lossless) or JPEG (with quality control from 0-100)
* **Default viewport:** Screenshots use the standard 1920x1080 viewport size

<Note>Screenshots are captured from the cloud browser and automatically transferred to your local environment. Large full-page screenshots may take additional time to process and download.</Note>

## Running Custom JavaScript Code

Execute custom JavaScript within the browser context to manipulate pages, extract computed values, or perform complex data transformations. This powerful capability enables sophisticated automation scenarios beyond standard element selection.

<Tabs>
  <Tab title="Puppeteer">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const puppeteer = require('puppeteer-core');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const customJavaScriptScraper = async () => {
          const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/', { 
                  waitUntil: 'domcontentloaded' 
              });
              
              // Extract page title using custom JavaScript
              const pageTitle = await page.evaluate(() => {
                  return document.title;
              });
              console.log('Page title via JavaScript:', pageTitle);
              
              // Get page statistics
              const pageStats = await page.evaluate(() => {
                  return {
                      totalLinks: document.querySelectorAll('a').length,
                      totalImages: document.querySelectorAll('img').length,
                      totalForms: document.querySelectorAll('form').length,
                      pageHeight: document.body.scrollHeight,
                      viewportHeight: window.innerHeight,
                      currentURL: window.location.href,
                      userAgent: navigator.userAgent
                  };
              });
              console.log('Page statistics:', pageStats);
              
              // Extract product data with custom processing
              const productData = await page.evaluate(() => {
                  const products = Array.from(document.querySelectorAll('.product'));
                  return products.map((product, index) => {
                      const name = product.querySelector('.product-name')?.textContent?.trim();
                      const priceText = product.querySelector('.price')?.textContent?.trim();
                      
                      // Custom price processing
                      const priceMatch = priceText?.match(/\$(\d+(?:\.\d{2})?)/);
                      const priceNumber = priceMatch ? parseFloat(priceMatch[1]) : null;
                      
                      return {
                          id: index + 1,
                          name: name || 'Unknown Product',
                          originalPrice: priceText || 'Price not available',
                          numericPrice: priceNumber,
                          isOnSale: product.querySelector('.sale-badge') !== null,
                          position: index + 1
                      };
                  });
              });
              console.log('Processed product data:', productData);
              
              // Scroll and capture dynamic content
              const scrollResults = await page.evaluate(async () => {
                  // Scroll to bottom of page
                  window.scrollTo(0, document.body.scrollHeight);
                  
                  // Wait for any lazy-loaded content
                  await new Promise(resolve => setTimeout(resolve, 2000));
                  
                  return {
                      finalScrollPosition: window.pageYOffset,
                      totalHeight: document.body.scrollHeight,
                      lazyImagesLoaded: document.querySelectorAll('img[data-src]').length
                  };
              });
              console.log('Scroll results:', scrollResults);
              
              // Inject custom CSS and modify page appearance
              await page.evaluate(() => {
                  const style = document.createElement('style');
                  style.textContent = `
                      .product { border: 2px solid red !important; }
                      .price { background-color: yellow !important; }
                  `;
                  document.head.appendChild(style);
              });
              
              console.log('Custom styles applied');
              
          } finally {
              await browser.close();
          }
      };

      customJavaScriptScraper();
      ```

      ```python Python theme={"dark"}
      # pip install pyppeteer
      import asyncio
      from pyppeteer import connect

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def custom_javascript_scraper():
          browser = await connect(browserWSEndpoint=connection_url)
          page = await browser.newPage()
          
          try:
              print('Navigating to target page...')
              await page.goto('https://www.scrapingcourse.com/ecommerce/', {
                  'waitUntil': 'domcontentloaded',
                  'timeout': 60000
              })
              
              # Extract page title using custom JavaScript
              page_title = await page.evaluate('() => document.title')
              print(f'Page title via JavaScript: {page_title}')
              
              # Get page statistics
              page_stats = await page.evaluate('''() => ({
                  totalLinks: document.querySelectorAll('a').length,
                  totalImages: document.querySelectorAll('img').length,
                  totalForms: document.querySelectorAll('form').length,
                  pageHeight: document.body.scrollHeight,
                  viewportHeight: window.innerHeight,
                  currentURL: window.location.href,
                  userAgent: navigator.userAgent
              })''')
              print(f'Page statistics: {page_stats}')
              
              # Extract and process product data
              product_data = await page.evaluate('''() => {
                  const products = Array.from(document.querySelectorAll('.product'));
                  return products.map((product, index) => {
                      const name = product.querySelector('.product-name')?.textContent?.trim();
                      const priceText = product.querySelector('.price')?.textContent?.trim();
                      
                      // Custom price processing
                      const priceMatch = priceText?.match(/\\$([0-9]+(?:\\.[0-9]{2})?)/);
                      const priceNumber = priceMatch ? parseFloat(priceMatch[1]) : null;
                      
                      return {
                          id: index + 1,
                          name: name || 'Unknown Product',
                          originalPrice: priceText || 'Price not available',
                          numericPrice: priceNumber,
                          isOnSale: product.querySelector('.sale-badge') !== null,
                          position: index + 1
                      };
                  });
              }''')
              print(f'Processed product data: {product_data}')
              
              # Scroll and capture dynamic content
              scroll_results = await page.evaluate('''async () => {
                  // Scroll to bottom of page
                  window.scrollTo(0, document.body.scrollHeight);
                  
                  // Wait for any lazy-loaded content
                  await new Promise(resolve => setTimeout(resolve, 2000));
                  
                  return {
                      finalScrollPosition: window.pageYOffset,
                      totalHeight: document.body.scrollHeight,
                      lazyImagesLoaded: document.querySelectorAll('img[data-src]').length
                  };
              }''')
              print(f'Scroll results: {scroll_results}')
              
              # Inject custom CSS and modify page appearance
              await page.evaluate('''() => {
                  const style = document.createElement('style');
                  style.textContent = `
                      .product { border: 2px solid red !important; }
                      .price { background-color: yellow !important; }
                  `;
                  document.head.appendChild(style);
              }''')
              
              print('Custom styles applied')
              
          except Exception as error:
              print(f'Custom JavaScript scraper error: {error}')
          finally:
              await page.close()
              await browser.disconnect()

      if __name__ == "__main__":
          asyncio.run(custom_javascript_scraper())
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Playwright">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const { chromium } = require('playwright');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const customJavaScriptScraper = async () => {
          const browser = await chromium.connectOverCDP(connectionURL);
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/');
              
              // Extract page title using custom JavaScript
              const pageTitle = await page.evaluate(() => document.title);
              console.log('Page title via JavaScript:', pageTitle);
              
              // Get page statistics
              const pageStats = await page.evaluate(() => ({
                  totalLinks: document.querySelectorAll('a').length,
                  totalImages: document.querySelectorAll('img').length,
                  totalForms: document.querySelectorAll('form').length,
                  pageHeight: document.body.scrollHeight,
                  viewportHeight: window.innerHeight,
                  currentURL: window.location.href,
                  userAgent: navigator.userAgent
              }));
              console.log('Page statistics:', pageStats);
              
              // Extract and process product data
              const productData = await page.evaluate(() => {
                  const products = Array.from(document.querySelectorAll('.product'));
                  return products.map((product, index) => {
                      const name = product.querySelector('.product-name')?.textContent?.trim();
                      const priceText = product.querySelector('.price')?.textContent?.trim();
                      
                      // Custom price processing
                      const priceMatch = priceText?.match(/\$(\d+(?:\.\d{2})?)/);
                      const priceNumber = priceMatch ? parseFloat(priceMatch[1]) : null;
                      
                      return {
                          id: index + 1,
                          name: name || 'Unknown Product',
                          originalPrice: priceText || 'Price not available',
                          numericPrice: priceNumber,
                          isOnSale: product.querySelector('.sale-badge') !== null,
                          position: index + 1
                      };
                  });
              });
              console.log('Processed product data:', productData);
              
              // Scroll and capture dynamic content
              await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
              await page.waitForTimeout(2000);
              
              const scrollResults = await page.evaluate(() => ({
                  finalScrollPosition: window.pageYOffset,
                  totalHeight: document.body.scrollHeight,
                  lazyImagesLoaded: document.querySelectorAll('img[data-src]').length
              }));
              console.log('Scroll results:', scrollResults);
              
              // Inject custom CSS
              await page.addStyleTag({
                  content: `
                      .product { border: 2px solid red !important; }
                      .price { background-color: yellow !important; }
                  `
              });
              
              console.log('Custom styles applied');
              
          } finally {
              await browser.close();
          }
      };

      customJavaScriptScraper();
      ```

      ```python Python theme={"dark"}
      import asyncio
      from playwright.async_api import async_playwright

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def custom_javascript_scraper():
          async with async_playwright() as p:
              browser = await p.chromium.connect_over_cdp(connection_url)
              page = await browser.new_page()
              
              try:
                  print('Navigating to target page...')
                  await page.goto('https://www.scrapingcourse.com/ecommerce/')
                  
                  # Extract page title using custom JavaScript
                  page_title = await page.evaluate('() => document.title')
                  print(f'Page title via JavaScript: {page_title}')
                  
                  # Get page statistics
                  page_stats = await page.evaluate('''() => ({
                      totalLinks: document.querySelectorAll('a').length,
                      totalImages: document.querySelectorAll('img').length,
                      totalForms: document.querySelectorAll('form').length,
                      pageHeight: document.body.scrollHeight,
                      viewportHeight: window.innerHeight,
                      currentURL: window.location.href,
                      userAgent: navigator.userAgent
                  })''')
                  print(f'Page statistics: {page_stats}')
                  
                  # Extract and process product data
                  product_data = await page.evaluate('''() => {
                      const products = Array.from(document.querySelectorAll('.product'));
                      return products.map((product, index) => {
                          const name = product.querySelector('.product-name')?.textContent?.trim();
                          const priceText = product.querySelector('.price')?.textContent?.trim();
                          
                          // Custom price processing
                          const priceMatch = priceText?.match(/\\$([0-9]+(?:\\.[0-9]{2})?)/);
                          const priceNumber = priceMatch ? parseFloat(priceMatch[1]) : null;
                          
                          return {
                              id: index + 1,
                              name: name || 'Unknown Product',
                              originalPrice: priceText || 'Price not available',
                              numericPrice: priceNumber,
                              isOnSale: product.querySelector('.sale-badge') !== null,
                              position: index + 1
                          };
                      });
                  }''')
                  print(f'Processed product data: {product_data}')
                  
                  # Scroll and capture dynamic content
                  await page.evaluate('() => window.scrollTo(0, document.body.scrollHeight)')
                  await page.wait_for_timeout(2000)
                  
                  scroll_results = await page.evaluate('''() => ({
                      finalScrollPosition: window.pageYOffset,
                      totalHeight: document.body.scrollHeight,
                      lazyImagesLoaded: document.querySelectorAll('img[data-src]').length
                  })''')
                  print(f'Scroll results: {scroll_results}')
                  
                  # Inject custom CSS
                  await page.add_style_tag(content='''
                      .product { border: 2px solid red !important; }
                      .price { background-color: yellow !important; }
                  ''')
                  
                  print('Custom styles applied')
                  
              finally:
                  await browser.close()

      if __name__ == "__main__":
          asyncio.run(custom_javascript_scraper())
      ```
    </CodeGroup>
  </Tab>
</Tabs>

### JavaScript Execution Capabilities

* **Data extraction and processing:** Transform raw data within the browser context
* **Page statistics collection:** Gather comprehensive page metrics and analytics
* **Dynamic content interaction:** Trigger JavaScript events and handle dynamic updates
* **Custom styling injection:** Modify page appearance for testing or visual enhancement
* **Scroll automation:** Navigate through infinite scroll or lazy-loaded content
* **Complex calculations:** Perform mathematical operations on extracted data

<Tip>Custom JavaScript execution runs within the browser's security context, providing access to all DOM APIs and browser features available to the target website.</Tip>

## PDF Generation and Document Export

Generate PDF documents from web pages for archival, reporting, or documentation purposes. This capability proves valuable for creating snapshots of dynamic content or generating reports from scraped data.

<Tabs>
  <Tab title="Puppeteer">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const puppeteer = require('puppeteer-core');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const pdfGenerationScraper = async () => {
          const browser = await puppeteer.connect({ browserWSEndpoint: connectionURL });
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/', { 
                  waitUntil: 'domcontentloaded' 
              });
              
              console.log('Page loaded:', await page.title());
              
              // Generate basic PDF
              console.log('Generating basic PDF...');
              await page.pdf({
                  path: 'basic-page.pdf',
                  format: 'A4',
                  printBackground: true
              });
              
              // Generate custom PDF with options
              console.log('Generating custom PDF...');
              await page.pdf({
                  path: 'custom-page.pdf',
                  format: 'A4',
                  printBackground: true,
                  margin: {
                      top: '20mm',
                      bottom: '20mm',
                      left: '20mm',
                      right: '20mm'
                  },
                  displayHeaderFooter: true,
                  headerTemplate: '<div style="font-size: 10px; text-align: center; width: 100%;">E-commerce Scraping Report</div>',
                  footerTemplate: '<div style="font-size: 10px; text-align: center; width: 100%;">Page <span class="pageNumber"></span> of <span class="totalPages"></span></div>'
              });
              
              // Generate PDF of specific content area
              console.log('Generating product grid PDF...');
              const productGrid = await page.$('.products');
              if (productGrid) {
                  const boundingBox = await productGrid.boundingBox();
                  await page.pdf({
                      path: 'product-grid.pdf',
                      format: 'A4',
                      printBackground: true,
                      clip: boundingBox
                  });
              }
              
              // Generate landscape PDF
              console.log('Generating landscape PDF...');
              await page.pdf({
                  path: 'landscape-page.pdf',
                  format: 'A4',
                  landscape: true,
                  printBackground: true,
                  scale: 0.8
              });
              
              console.log('All PDFs generated successfully');
              
          } finally {
              await browser.close();
          }
      };

      pdfGenerationScraper();
      ```

      ```python Python theme={"dark"}
      # pip install pyppeteer
      import asyncio
      from pyppeteer import connect

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def pdf_generation_scraper():
          browser = await connect(browserWSEndpoint=connection_url)
          page = await browser.newPage()
          
          try:
              print('Navigating to target page...')
              await page.goto('https://www.scrapingcourse.com/ecommerce/', 
                          {'waitUntil': 'domcontentloaded'})
              
              print(f'Page loaded: {await page.title()}')
              
              # Generate basic PDF
              print('Generating basic PDF...')
              await page.pdf({
                  'path': 'basic-page.pdf',
                  'format': 'A4',
                  'printBackground': True
              })
              
              # Generate custom PDF with options
              print('Generating custom PDF...')
              await page.pdf({
                  'path': 'custom-page.pdf',
                  'format': 'A4',
                  'printBackground': True,
                  'margin': {
                      'top': '20mm',
                      'bottom': '20mm',
                      'left': '20mm',
                      'right': '20mm'
                  },
                  'displayHeaderFooter': True,
                  'headerTemplate': '<div style="font-size: 10px; text-align: center; width: 100%;">E-commerce Scraping Report</div>',
                  'footerTemplate': '<div style="font-size: 10px; text-align: center; width: 100%;">Page <span class="pageNumber"></span> of <span class="totalPages"></span></div>'
              })
              
              # Generate landscape PDF
              print('Generating landscape PDF...')
              await page.pdf({
                  'path': 'landscape-page.pdf',
                  'format': 'A4',
                  'landscape': True,
                  'printBackground': True,
                  'scale': 0.8
              })
              
              print('All PDFs generated successfully')
              
          finally:
              await page.close()
              await browser.disconnect()

      if __name__ == "__main__":
          asyncio.run(pdf_generation_scraper())
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Playwright">
    <CodeGroup>
      ```javascript Node.js theme={"dark"}
      const { chromium } = require('playwright');
      const connectionURL = 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY';

      const pdfGenerationScraper = async () => {
          const browser = await chromium.connectOverCDP(connectionURL);
          const page = await browser.newPage();
          
          try {
              console.log('Navigating to target page...');
              await page.goto('https://www.scrapingcourse.com/ecommerce/');
              
              console.log('Page loaded:', await page.title());
              
              // Generate basic PDF
              console.log('Generating basic PDF...');
              await page.pdf({
                  path: 'basic-page.pdf',
                  format: 'A4',
                  printBackground: true
              });
              
              // Generate custom PDF with options
              console.log('Generating custom PDF...');
              await page.pdf({
                  path: 'custom-page.pdf',
                  format: 'A4',
                  printBackground: true,
                  margin: {
                      top: '20mm',
                      bottom: '20mm',
                      left: '20mm',
                      right: '20mm'
                  },
                  displayHeaderFooter: true,
                  headerTemplate: '<div style="font-size: 10px; text-align: center; width: 100%;">E-commerce Scraping Report</div>',
                  footerTemplate: '<div style="font-size: 10px; text-align: center; width: 100%;">Page <span class="pageNumber"></span> of <span class="totalPages"></span></div>'
              });
              
              // Generate landscape PDF
              console.log('Generating landscape PDF...');
              await page.pdf({
                  path: 'landscape-page.pdf',
                  format: 'A4',
                  landscape: true,
                  printBackground: true,
                  scale: 0.8
              });
              
              console.log('All PDFs generated successfully');
              
          } finally {
              await browser.close();
          }
      };

      pdfGenerationScraper();
      ```

      ```python Python theme={"dark"}
      # pip install playwright
      import asyncio
      from playwright.async_api import async_playwright

      connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

      async def pdf_generation_scraper():
          async with async_playwright() as p:
              browser = await p.chromium.connect_over_cdp(connection_url)
              page = await browser.new_page()
              
              try:
                  print('Navigating to target page...')
                  await page.goto('https://www.scrapingcourse.com/ecommerce/')
                  
                  print(f'Page loaded: {await page.title()}')
                  
                  # Generate basic PDF
                  print('Generating basic PDF...')
                  await page.pdf(
                      path='basic-page.pdf',
                      format='A4',
                      print_background=True
                  )
                  
                  # Generate custom PDF with options
                  print('Generating custom PDF...')
                  await page.pdf(
                      path='custom-page.pdf',
                      format='A4',
                      print_background=True,
                      margin={
                          'top': '20mm',
                          'bottom': '20mm',
                          'left': '20mm',
                          'right': '20mm'
                      },
                      display_header_footer=True,
                      header_template='<div style="font-size: 10px; text-align: center; width: 100%;">E-commerce Scraping Report</div>',
                      footer_template='<div style="font-size: 10px; text-align: center; width: 100%;">Page <span class="pageNumber"></span> of <span class="totalPages"></span></div>'
                  )
                  
                  # Generate landscape PDF
                  print('Generating landscape PDF...')
                  await page.pdf(
                      path='landscape-page.pdf',
                      format='A4',
                      landscape=True,
                      print_background=True,
                      scale=0.8
                  )
                  
                  print('All PDFs generated successfully')
                  
              finally:
                  await browser.close()

      if __name__ == "__main__":
          asyncio.run(pdf_generation_scraper())
      ```
    </CodeGroup>
  </Tab>
</Tabs>

### PDF Generation Features

* **Multiple format support:** Generate A4, Letter, Legal, and custom page sizes
* **Custom headers and footers:** Add branding, page numbers, and metadata
* **Background preservation:** Include CSS backgrounds and styling in PDFs
* **Margin control:** Configure precise spacing and layout
* **Orientation options:** Create portrait or landscape documents
* **Scale adjustment:** Optimize content size for better readability

<Tip>PDF generation works seamlessly with the cloud browser, automatically transferring generated files to your local environment while maintaining high quality and formatting.</Tip>

## Conclusion

Zenrows' Browser Sessions transforms complex web automation challenges into straightforward solutions. These practical use cases demonstrate the platform's versatility in handling everything from basic content extraction to sophisticated browser automation workflows.

## Next Steps for Implementation

Start with the basic navigation and content extraction patterns to establish your foundation, then progressively incorporate advanced features like form interactions and network monitoring as your requirements evolve. The modular nature of these examples allows you to combine techniques for sophisticated automation workflows.

Consider implementing error handling and retry logic around these patterns for production deployments. Browser Sessions's consistent cloud environment reduces many common failure points, but robust error handling ensures reliable operation at scale.

## Frequently Asked Questions (FAQ)

<Accordion title="Can I combine multiple use cases in a single scraping session?">
  Absolutely! These use cases are designed to work together. For example, you can navigate to a page, take screenshots, extract data, and generate PDFs all within the same browser session. This approach is more efficient and maintains session state across operations.

  ```javascript Node.js theme={"dark"}
  // Example combining multiple use cases
  await page.goto('https://example.com');
  await page.screenshot({ path: 'before.png' });
  await page.fill('input[name="search"]', 'query');
  await page.click('button[type="submit"]');
  await page.screenshot({ path: 'after.png' });
  const data = await page.$$eval('.result', elements => /* extract data */);
  await page.pdf({ path: 'results.pdf' });
  ```
</Accordion>

<Accordion title="How do I handle dynamic content that loads after page navigation?">
  Use explicit waiting mechanisms to ensure content is fully loaded before interaction:

  ```javascript Node.js theme={"dark"}
  // Wait for specific elements
  await page.waitForSelector('.dynamic-content');
  // Wait for network to be idle
  await page.goto(url, { waitUntil: 'domcontentloaded' });

  // Wait for custom conditions
  await page.waitForFunction(() => document.querySelectorAll('.product').length > 0);
  ```

  Browser Sessions handles JavaScript rendering automatically, making these waiting strategies highly effective.
</Accordion>

<Accordion title="How do I optimize performance when scraping large amounts of data?">
  Several strategies can significantly improve performance:

  * **Block unnecessary resources:** Use request interception to block images, fonts, and other non-essential content
  * **Reuse browser instances:** Keep browsers open for multiple operations instead of creating new connections
  * **Implement concurrent processing:** Use multiple browser instances for parallel scraping
  * **Optimize waiting strategies:** Use specific selectors instead of generic timeouts

  The network monitoring examples demonstrate resource blocking techniques that can improve scraping speed.
</Accordion>

<Accordion title="What's the difference between using the WebSocket URL directly versus the SDK?">
  Both approaches provide identical functionality, but the SDK offers several advantages:

  * **Simplified configuration:** No need to manually construct WebSocket URLs
  * **Better error handling:** Built-in error messages and debugging information
  * **Future compatibility:** Automatic updates to connection protocols
  * **Additional utilities:** Helper methods for common tasks

  For production applications, the SDK is recommended for better maintainability and error handling, while direct WebSocket connections work well for simple scripts and testing.
</Accordion>

<Accordion title="How do I troubleshoot issues when these examples don't work as expected?">
  Follow this systematic troubleshooting approach:

  1. **Verify API key:** Ensure your Zenrows API key is correct and active
  2. **Check element selectors:** Use browser developer tools to verify CSS selectors
  3. **Add debugging output:** Include console.log statements to track execution flow
  4. **Implement error handling:** Wrap operations in try-catch blocks
  5. **Test with simpler examples:** Start with basic navigation before adding complexity

  The network monitoring examples are particularly valuable for debugging, as they reveal exactly what requests are being made and their responses.
</Accordion>


# Introduction to Browser Sessions
Source: https://docs.zenrows.com/browser-sessions/introduction

Learn what Zenrows Browser Sessions offers for dynamic website scraping, including residential proxy rotation, geo-targeting, and CDP support.

Zenrows' Browser Sessions (formerly, Scraping Browser) is the perfect solution for effortlessly extracting data from dynamic websites, especially if you're already using tools like Puppeteer or Playwright. **With just one line of code, you can integrate Browser Sessions into your existing setup** and immediately benefit from our vast network of over 55 million residential IPs across 190+ countries, offering 99.9% uptime for uninterrupted scraping.

Whether you're dealing with complex JavaScript content or interacting with web elements, Zenrows makes it easy to scrape without worrying about managing proxies or handling dynamic user interactions.

## Ideal for Puppeteer and Playwright Users

If you already have web scraping code using [Puppeteer](https://www.zenrows.com/blog/puppeteer-web-scraping) or [Playwright](https://www.zenrows.com/blog/playwright-scraping), integrating Zenrows' Browser Sessions is incredibly simple.

By adding just one line, you can unlock the power of our residential proxy network and browser simulation, without needing to rewrite your entire scraping logic. Zenrows enhances your scraping capabilities with minimal effort:

```bash Browser URL theme={"dark"}
wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY
```

This integration ensures that your existing scraping operations become more robust, efficient, and scalable.

## Key Advantages

Browser Sessions goes beyond traditional scraping techniques, offering solutions for the most challenging web scraping tasks:

* **One Line of Code:** Start scraping instantly with Zenrows' Browser Sessions. No complicated setup. Just a simple API call that does all the heavy lifting for you.
* **Dynamic Web Content Extraction:** Many websites rely on JavaScript to display content dynamically. Browser Sessions simulates a real user session, allowing you to extract data even from single-page applications (SPAs) or websites that load content asynchronously.
* **Handling User Interactions:** Simulate user actions like clicking, scrolling, or waiting for elements to load, all while avoiding IP blocks and ensuring the data is extracted seamlessly.
* **Geolocation Targeting:** Access localized content by selecting from millions of IPs across 190+ countries, bypassing geo-restrictions to scrape the data you need.
* **IP Auto-Rotation:** Rotate IPs with every request to avoid bans and blocks, ensuring your scraping remains undetected.

## Parameter Overview

Customize your Browser Sessions requests using the following parameters:

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **apikey** `required` | `string` | [**Get Your Free API Key**](https://app.zenrows.com/register?p=free) | Your unique API key for authentication |
| [**proxy\_region**](https://docs.zenrows.com/browser-sessions/features/world-region) | `string` | `global` | Focus your scraping on a specific geographic region (e.g., eu for Europe). **When set to global, no parameter is needed**. |
| [**proxy\_country**](https://docs.zenrows.com/browser-sessions/features/country) | `string` | | Target a specific country for geo-restricted data (e.g., `es` for Spain). Available only when `proxy_region` is set to global. |
| [**session\_ttl**](https://docs.zenrows.com/browser-sessions/features/session-ttl) | `number` | `180` | The time to live for the browser session, in seconds. Minimum: 60, Maximum: 900 (1-15 minutes). Defaults to 3 minutes. |

## Pricing

Browser Sessions bill in credits, on the same shared balance as every other Zenrows product. Bandwidth is metered at a flat **25,000 credits per GB** on every plan, plus a flat **5 credits per minute** of session time.

**With Zenrows, you only pay for the data and time you use**, and the Free plan (5,000 credits/month, no credit card required) lets you try it at no cost. Explore our flexible plans and find more details on our [pricing page](/first-steps/pricing).

## Tips for Optimal Performance

1. **Manage Session Duration**: Keep session durations as short as possible since session time is billed by the minute.

2. **Reuse Browser Sessions**: For multiple operations on the same site, reuse the browser session rather than creating a new one each time.

3. **Use Proper Page Closure**: Always close pages when done to free up resources:
   ```javascript theme={"dark"}
   // Close individual pages when done
   await page.close();
   ```

4. **Set Reasonable Timeouts**: Use appropriate timeouts for your operations to prevent long-running sessions:
   ```javascript theme={"dark"}
   // Set navigation timeout
   await page.goto(url, { timeout: 30000 });

   // Set selector timeout
   await page.waitForSelector('.element', { timeout: 15000 });
   ```

5. **Optimize Resource Usage**: Block unnecessary resources when they're not needed:
   ```javascript theme={"dark"}
   // Block images, fonts, etc.
   await page.setRequestInterception(true);
   page.on('request', (request) => {
     if (request.resourceType() === 'image' || request.resourceType() === 'font') {
       request.abort();
     } else {
       request.continue();
     }
   });
   ```

6. **Target Specific Countries Wisely**: Only specify a country when needed for your use case, as more specific targeting might have different success rates.


# Browser Sessions
Source: https://docs.zenrows.com/browser-sessions/sdk-overview

Browser Sessions doesn't need a separate SDK. It integrates into your existing browser automation setup with one line of code.

Browser Sessions (formerly, Scraping Browser) doesn't need a separate SDK. It integrates into your existing browser automation setup with one line: point your existing browser connection at the Zenrows Browser Sessions endpoint instead of a local browser. It works with Puppeteer, Playwright, or any CDP-compatible tool.

```bash Browser URL theme={"dark"}
wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY
```

Everything else, your scraping logic, selectors, and page interactions, stays exactly as it already is. See the [Browser Sessions introduction](/browser-sessions/introduction) for the full picture: residential proxy rotation, geo-targeting, and CDP support.

## Get Started by Language

<CardGroup>
  <Card title="Puppeteer" icon="node-js" href="/browser-sessions/get-started/puppeteer">
    Connect Puppeteer (Node.js) or Pyppeteer (Python) to Browser Sessions.
  </Card>

  <Card title="Playwright" icon="code" href="/browser-sessions/get-started/playwright">
    Connect Playwright, for Python or Node.js, to Browser Sessions.
  </Card>
</CardGroup>


# Setting Up Browser Sessions
Source: https://docs.zenrows.com/browser-sessions/setup

Quick setup guide for Zenrows Browser Sessions with Puppeteer or Playwright, including API key configuration and connection parameters.

Zenrows Browser Sessions (formerly, Scraping Browser) works seamlessly with your existing Puppeteer or Playwright code. Just one line of code is needed to integrate, making it ideal for developers looking for fast and easy solutions.

## Initial Setup

To start using Browser Sessions, follow these steps:

<Steps>
  <Step title="Sign Up for Zenrows">
    If you haven't already, create a Zenrows account by visiting the [Registration Page](https://app.zenrows.com/register).
  </Step>

  <Step title="Head to your Browser Sessions dashboard">
    Once registered, log in to your dashboard to access your [Browser Sessions page](https://app.zenrows.com/browser).

    <Frame>
      <img alt="scraping-browser-dashboard" />
    </Frame>
  </Step>
</Steps>

## Browser Sessions Parameters

Zenrows Browser Sessions is designed for ease of use with minimal configuration. However, you can still customize some options to suit your needs, including World Region, Country and Session TTL.

<Note>Auto-rotate and Residential IPs are pre-configured and enabled by default for all users.</Note>

### Available Parameter Options

Browser Sessions offers several customizable parameters to enhance your scraping experience:

* [World Region](/browser-sessions/features/world-region) - Target specific geographic regions for your scraping operations
* [Country Selection](/browser-sessions/features/country) - Choose from our extensive list of premium proxy countries for geo-restricted content
* [Session TTL](/browser-sessions/features/session-ttl) - Control browser session duration from 1 to 15 minutes

Each parameter page provides both SDK and direct WebSocket URL examples to help you implement the settings that best fit your scraping requirements.


# Agent Toolkit
Source: https://docs.zenrows.com/cli/agent-toolkit-overview

MCP, Skills, Templates, Workflows, Recipes, and Evals: the pieces that make Zenrows installable and usable by an AI agent, not just callable by one.

A tool being callable over an API isn't the same as a tool being usable by an agent. An agent that can call `fetch` still has to work out, on its own, whether `fetch` is even the right call for this page, how to start a real project instead of a one-off snippet, how to script something that runs the same way every time, and how to tell whether a change quietly broke something. The Agent Toolkit is the part of the Zenrows CLI that answers those questions directly, so an agent (or the developer directing one) doesn't have to reconstruct them from scratch on every project.

Each piece solves one specific part of that problem:

<CodeGroup>
  ```bash Connect and install everything theme={"dark"}
  zenrows init --all
  ```

  ```bash Just wire up an agent client theme={"dark"}
  zenrows plugin install claude-code
  ```
</CodeGroup>

`init --all` sets up a full workspace: every skill, template, workflow, recipe, and eval, plus MCP configured for your agent clients, in one command. `plugin install <client>` is the narrower path when you only need an agent client wired up: it configures MCP for that client and installs every skill, nothing else. Check what's already in place at any point with `zenrows status`.

## The Shape of the Problem

<img alt="Six pieces, each independent: Connect, Decide, Start, Automate, Verify, Benchmark, mapping to MCP, Skills, Templates, Workflows, Recipes, and Evals." />

Each piece answers a different question, and none of them require the others. The CLI authenticates and runs `fetch`, `extract`, `batch`, and `browser` entirely on its own, no agent client, no MCP:

* **Connect** an MCP-native agent client, Claude Desktop, Cursor, and similar, so the model can discover and call tools directly instead of a human typing CLI commands. [MCP](/cli/mcp) sets this up. Only relevant if you're wiring in that kind of client, an agent that already runs shell commands (like one driving the CLI directly) doesn't need it at all.
* **Decide** which primitive fits the task, without guessing. [Skills](/cli/skills) are short, agent-readable playbooks, a master decision tree plus one playbook per primitive, so an agent picks `fetch`, `extract`, `batch`, or a browser session correctly, without extra prompting from you.
* **Start** a real project instead of an empty file. [Templates](/cli/templates) scaffold a minimal, working project for a specific job, like calling Fetch from Node.js or building a Batch job spec.
* **Automate** something that has to run the same way every time. [Workflows](/cli/workflows) chain multiple Zenrows commands into one named, repeatable process, like keeping a competitor's pricing data fresh on a schedule.
* **Verify** that a single command actually works, right now, against a known-good target. [Recipes](/cli/recipes) are small, runnable examples that double as a smoke test before you build anything on top of them.
* **Benchmark** that it keeps working. [Evals](/cli/evals) run the same check reproducibly and write a scored report, so a regression shows up as a failed eval instead of a silent surprise in production.

## The Pieces

| Piece | Answers | Catalog today | Primary command |
| - | - | - | - |
| [MCP](/cli/mcp) | How does an agent client connect at all? | 8 supported clients | `zenrows plugin install <client>` |
| [Skills](/cli/skills) | Which primitive should I use, and why? | 8 skills | `zenrows skill install --all` |
| [Templates](/cli/templates) | How do I start this kind of project? | 3 templates | `zenrows template create <name>` |
| [Workflows](/cli/workflows) | How do I automate a multi-step process? | 1 workflow | `zenrows workflow run <name>` |
| [Recipes](/cli/recipes) | Does this one command actually work? | 2 recipes | `zenrows recipe run <name>` |
| [Evals](/cli/evals) | Does it still work, with a written report? | 3 evals | `zenrows eval run <name>` |

Every piece follows the same shape: `list` to see what's available, `install` (or `create`, for templates) to bring it into your project, `explain` to read its full docs, and `run` (for workflows, recipes, and evals) to actually execute it. Learn one, and you already know the other five.

## Next Steps

<CardGroup>
  <Card title="MCP" icon="plug-connected" href="/cli/mcp">
    Connect Claude Code, Cursor, and other clients.
  </Card>

  <Card title="Skills" icon="message-chatbot" href="/cli/skills">
    See the full catalog of agent playbooks.
  </Card>

  <Card title="Templates" icon="code" href="/cli/templates">
    Scaffold a working project.
  </Card>

  <Card title="Workflows" icon="route" href="/cli/workflows">
    Automate a multi-step process.
  </Card>

  <Card title="Recipes" icon="flask" href="/cli/recipes">
    Run one example end to end.
  </Card>

  <Card title="Evals" icon="chart-bar" href="/cli/evals">
    Benchmark a command with a written report.
  </Card>
</CardGroup>


# Batch
Source: https://docs.zenrows.com/cli/batch/introduction

Process thousands of URLs reliably, in one job, and check the cost before you spend a single credit.

`zenrows batch` runs the same fetch or extract job across a list of URLs as a single, trackable job: submit once, then check status or collect results whenever you want.

```bash theme={"dark"}
zenrows batch <estimate|create|status|results|cancel|wait|retry-failed> [flags]
```

`jobs.jsonl` is one JSON object per line, each with a `url` and optional per-line overrides (`js_render`, `premium_proxy`, `proxy_country`).

## Local, no key needed

```bash theme={"dark"}
zenrows batch estimate jobs.jsonl
```

Validates the spec and estimates credit cost with no cloud call. Run this before submitting a real job.

## Cloud

```bash theme={"dark"}
# Submit a job and wait for it to finish
zenrows batch create jobs.jsonl --wait

# Check status
zenrows batch status <job-id>

# Collect results (paginated)
zenrows batch results <job-id> --out results.jsonl

# Rerun only the failed tasks
zenrows batch retry-failed <job-id>

# Stop an in-flight run
zenrows batch cancel <job-id>
```

## Flags

`--json` (print structured output) works on every subcommand. The rest are subcommand-specific:

| Subcommand | Flags |
| - | - |
| `create <file.jsonl>` | `--js-render`, `--premium-proxy`, `--proxy-country <cc>` (needs `--premium-proxy`), `--output <fmt>` (`markdown`\|`plaintext`\|`pdf`), `--wait` |
| `results <id>` | `--status <successful\|failed\|all>`, `--out <file>`, `--download <dir>` (fetches each result body, plus a `_manifest.jsonl`) |
| `wait <id>` | `--timeout <ms>` |


# Browser Sessions
Source: https://docs.zenrows.com/cli/browser/introduction

Click, log in, navigate, and interact like a real user, only when simpler methods can't do the job.

`zenrows browser` gives you a real, remote browser session for the things Fetch and Extract can't do: logging in, clicking through a flow, or waiting on content that only appears after interaction.

```bash theme={"dark"}
zenrows browser info
```

Disabled by default. Enable it first:

```bash theme={"dark"}
zenrows policy set allow_browser true
```

<Warning>
  Escalation-only. Use [`zenrows fetch`](/cli/fetch/introduction) or [`zenrows extract`](/cli/extract/introduction) first, reach for a browser only when they genuinely can't do the job (logins, clicks, multi-step flows, persistent state).
</Warning>

Two ways to drive a session:

* Connect Playwright or Puppeteer directly to the Browser Sessions API
* Drive it through [`zenrows mcp`](/cli/mcp)'s browser tools (navigate, click, fill, screenshot, and more)


# Evals
Source: https://docs.zenrows.com/cli/evals

A reproducible benchmark bundled with the Zenrows CLI, with explicit pass or fail criteria.

An eval is a scripted benchmark against a known target. Run it to check that a command still behaves as expected, and get a written report back.

```bash theme={"dark"}
zenrows eval list
zenrows eval run <name>
```

| Command | Does |
| - | - |
| `zenrows eval list` | List all evals |
| `zenrows eval run <name>` | Run an eval and write a report |
| `zenrows eval report <name>` | Print a previous eval's report |
| `zenrows eval explain <name>` | Show an eval's full docs |
| `zenrows eval install <name>` | Copy an eval into `.zenrows/` |
| `zenrows eval update [name]` | Refresh installed eval(s) |
| `zenrows eval remove <name>` | Remove an installed eval |

## Catalog

| Eval | Description |
| - | - |
| `protected-fetch-smoke` | `fetch` against a known-good URL, expects HTTP 2xx and content |
| `extract-smoke` | `extract --autoparse` against a known-good ecommerce demo page |
| `batch-spec-validation` | A JSONL spec parses cleanly and estimates cost, no cloud call |

Each run writes `input.json`, `results.json`, `report.md`, `failures.jsonl`, `cost.json`, and `traces/` under `.zenrows/evals/<run-id>/`.


# Extract
Source: https://docs.zenrows.com/cli/extract/introduction

Turn any messy web page into structured JSON, exact fields you pick, or Markdown, from the terminal.

`zenrows extract` fetches a page and converts it into structured data: automatic JSON, fields you choose with CSS selectors, or Markdown.

```bash theme={"dark"}
zenrows extract <url> [flags]
```

## Examples

```bash theme={"dark"}
# Automatic structured JSON (default method)
zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse

# CSS selector map
zenrows extract <url> --css '{"title":"h1","price":".price"}' --validate

# Built-in output filters
zenrows extract <url> --outputs emails,links

# Convert to Markdown
zenrows extract <url> --output markdown
```

## Flags

Pick exactly one method per call: `--autoparse`, `--css`, `--outputs`, or `--output`.

| Flag | Description |
| - | - |
| `--autoparse` | Automatic structured JSON (default method) |
| `--css <json>` | CSS selector map, e.g. `'{"title":"h1","price":".price"}'` |
| `--outputs <list>` | Built-in output filters as JSON: `emails`, `phone_numbers`, `headings`, `images`, `audios`, `videos`, `links`, `menus`, `hashtags`, `metadata`, `tables`, `favicon`, comma-separated, or `*` for all |
| `--output <fmt>` | `markdown` or `text` conversion instead of extraction |
| `--validate` | Fail if the result is not valid JSON |
| `--manual` / `--js-render` / `--premium-proxy` | Same manual fetch controls as [`zenrows fetch`](/cli/fetch/introduction) |
| `--proxy-country <cc>` | Geo-target |
| `--wait <ms>` / `--wait-for <selector>` | Same wait controls as `zenrows fetch` |
| `--out <file>` | Write output to a file |
| `--no-signup` | Do not auto-create a Free-plan account if no key exists |
| `--json` | Print a structured result |


# Fetch
Source: https://docs.zenrows.com/cli/fetch/introduction

Get any page as clean, usable content from the terminal, reliably, even if it's dynamic, JavaScript-heavy, or protected.

`zenrows fetch` retrieves a single page and returns its HTML, Markdown, or plain text. It handles JavaScript rendering and anti-bot protection automatically, so most calls need no extra flags.

```bash theme={"dark"}
zenrows fetch <url> [flags]
```

## Examples

```bash theme={"dark"}
# Just works, no flags needed
zenrows fetch https://httpbin.io/html

# Markdown output
zenrows fetch https://example.com --output markdown

# Take manual control of rendering and proxies
zenrows fetch https://example.com --manual --js-render --premium-proxy

# Geo-target and wait for an element
zenrows fetch https://example.com --proxy-country us --wait-for ".price"

# Save the response to a file
zenrows fetch https://example.com --out page.html
```

<Tip>
  Start with the defaults. Only add `--manual --js-render --premium-proxy` after a run's trace (`zenrows trace explain <run-id>`) shows it's actually needed, each flag raises cost.
</Tip>

## Flags

| Flag | Description |
| - | - |
| `--manual` | Take manual control of rendering and proxies instead of the automatic default |
| `--js-render` | Render JavaScript (manual mode only) |
| `--premium-proxy` | Use residential IPs (manual mode only) |
| `--proxy-country <cc>` | Geo-target. Works alone by default; in manual mode requires `--premium-proxy` |
| `--wait <ms>` | Fixed wait after load |
| `--wait-for <selector>` | Wait for a CSS selector |
| `--js-instructions <json>` | JSON instructions to run on the page |
| `--session-id <n>` | Keep the same IP across requests |
| `--original-status` | Return the target's original HTTP status |
| `--output <fmt>` | `html` (default), `markdown`, `text`, or `pdf` |
| `--screenshot` | Capture an above-the-fold screenshot |
| `--out <file>` | Write the response body to a file |
| `--no-signup` | Do not auto-create a Free-plan account if no key exists |
| `--json` | Print a structured result instead of the raw body |


# Zenrows CLI
Source: https://docs.zenrows.com/cli/introduction

Install the Zenrows CLI, authenticate, and get your first result from the terminal in under five minutes.

Zenrows gives your terminal, and your AI agents, reliable access to the protected web: sites that are dynamic, JavaScript-heavy, or built to block ordinary tools. The `zenrows` CLI puts fetching, extracting, and running jobs at scale behind one command, plus the skills, templates, and MCP wiring that make it work naturally with an agent, not just a script.

It's [open source](https://github.com/ZenRows/cli).

## Installation

```bash theme={"dark"}
npx -y @zenrows/cli init
```

<Note>
  No API key needed to start. The first cloud call auto-provisions a Free-plan account and stores the key in `.zenrows/secrets.json`. Claim it later to keep your usage history, the same key keeps working:

  ```bash theme={"dark"}
  zenrows account claim
  ```
</Note>

To use an existing key instead of the auto-provisioning flow:

```bash theme={"dark"}
zenrows login --api-key <your-key>
# or
export ZENROWS_API_KEY=<your-key>
zenrows login --env
```

## Quickstart

```bash theme={"dark"}
npx -y @zenrows/cli init
zenrows fetch https://httpbin.io/html
zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse
```

## Full setup

One command instead of many: creates your workspace, installs every skill, template, workflow, recipe, and eval, configures MCP for your agent clients, and runs a health check.

```bash theme={"dark"}
zenrows init --all
```

Target specific agent clients instead of the defaults:

```bash theme={"dark"}
zenrows init --agents claude-code,cursor,windsurf
```

<Note>
  If a client can't be auto-configured, `init` prints the manual steps instead of skipping it silently.
</Note>

## Commands

### Primitives

The actual scraping work.

| Command | Does |
| - | - |
| [`fetch`](/cli/fetch/introduction) | Get any page as clean, usable content, even if it's protected |
| [`extract`](/cli/extract/introduction) | Turn messy pages into clean, structured JSON |
| [`batch`](/cli/batch/introduction) | Process thousands of URLs reliably, in one job |
| [`browser`](/cli/browser/introduction) | Click, log in, navigate, and interact, like a real user |

### Agent Toolkit

What makes Zenrows easy for an agent, or a developer, to actually use.

| Command | Does |
| - | - |
| [`mcp`](/cli/mcp) | Connect an AI agent, Claude Code, Cursor, and more, to Zenrows |
| [`skill`](/cli/skills) | Install agent-readable playbooks for choosing the right tool |
| [`template`](/cli/templates) | Scaffold a minimal, working project |
| [`workflow`](/cli/workflows) | Run a multi-step, end-to-end process |
| [`recipe`](/cli/recipes) | Run a single, ready-made example |
| [`eval`](/cli/evals) | Run a reproducible benchmark |

Check your setup, and what's already installed:

```bash theme={"dark"}
zenrows status
```

## Next steps

<CardGroup>
  <Card title="Connect an agent" icon="plug" href="/cli/mcp">
    Wire Claude Code, Cursor, Windsurf, and more into Zenrows with one command.
  </Card>

  <Card title="Troubleshooting" icon="life-buoy" href="/cli/troubleshooting">
    Read a failed run's trace and find the specific next action.
  </Card>
</CardGroup>


# MCP
Source: https://docs.zenrows.com/cli/mcp

Connect Zenrows to Claude Code, Cursor, and other AI agents through MCP, using the Zenrows CLI.

`zenrows mcp` and `zenrows plugin` are Zenrows CLI commands that wire the Zenrows MCP server into an agent client for you. They generate the config your client needs, they don't run a server themselves.

<Note>
  This page covers the CLI commands only. For the MCP server itself, its tools, and authentication, see [MCP](/mcp/overview).
</Note>

```bash theme={"dark"}
zenrows mcp status
zenrows mcp config --client <client>
```

## Clients

`claude-code`, `cursor`, `vscode`, `windsurf`, `gemini`, `codex`, `opencode`, `generic`.

<Note>
  Codex and OpenCode need manual configuration. The other six auto-configure.
</Note>

## Examples

```bash theme={"dark"}
# Print the exact config snippet for a client
zenrows mcp config --client claude-code

# Target the hosted server instead of the local one
zenrows mcp config --client cursor --remote

# Remove a configured server
zenrows mcp uninstall --client vscode
```

## `zenrows plugin`: MCP config + skills in one command

`zenrows plugin install <client>` wraps `mcp config` and also installs every skill, one command instead of two.

```bash theme={"dark"}
zenrows plugin list
zenrows plugin install claude-code
zenrows plugin status
```


# Recipes
Source: https://docs.zenrows.com/cli/recipes

A single, ready-to-run example bundled with the Zenrows CLI that doubles as a smoke test.

A recipe is one small, runnable example with a known-good target. Run it to confirm a Zenrows command works before you build on it.

```bash theme={"dark"}
zenrows recipe list
zenrows recipe run <name>
```

| Command | Does |
| - | - |
| `zenrows recipe list` | List all recipes |
| `zenrows recipe run <name>` | Run a recipe end to end |
| `zenrows recipe explain <name>` | Show a recipe's full docs |
| `zenrows recipe install <name>` | Copy a recipe into `.zenrows/` |
| `zenrows recipe update [name]` | Refresh installed recipe(s) |
| `zenrows recipe remove <name>` | Remove an installed recipe |

## Catalog

| Recipe | Description |
| - | - |
| `fetch-protected-page` | Fetch a single anti-bot-protected page |
| `batch-estimate` | Validate a JSONL batch spec and estimate its credit cost, locally, no key needed |


# Skills
Source: https://docs.zenrows.com/cli/skills

List, install, and manage playbooks that teach an AI agent which tool to use, bundled with the Zenrows CLI.

A skill is a short, agent-readable playbook. Install one and an AI agent knows when and how to use a Zenrows tool, no extra prompting needed.

```bash theme={"dark"}
zenrows skill list
zenrows skill install --all
```

| Command | Does |
| - | - |
| `zenrows skill list` | List all skills, installed or not |
| `zenrows skill install <name>` | Install a skill |
| `zenrows skill install --all` | Install every skill |
| `zenrows skill explain <name>` | Show a skill's full docs |
| `zenrows skill validate <name>` | Check an installed skill is well-formed |
| `zenrows skill generate --from-run <run-id>` | Scaffold a new skill from a past run |
| `zenrows skill update [name]` | Refresh installed skill(s) |
| `zenrows skill remove <name>` | Remove an installed skill |

## Catalog

| Skill | Description |
| - | - |
| `zenrows` | Master playbook: the decision tree, credit costs, escalation rules |
| `protected-fetch` | Use Protected Fetch for anti-bot-protected page retrieval |
| `extract` | Turn protected pages into structured data |
| `batch-jobs` | Scale fetch/extract over many URLs via Batch |
| `interact-browser` | Escalate to a browser only when fetch/extract can't do the job |
| `cost-control` | Pick the cheapest reliable option; understand the credit costs |
| `trace-debug` | Read run traces and choose the next action after a failure |
| `compliance-policy` | Domain allow/deny rules, credit limits, secret handling |


# Templates
Source: https://docs.zenrows.com/cli/templates

Scaffold a minimal, working project for Zenrows in one command.

A template is a ready-made project scaffold. Instantiate one into a folder and you get a minimal, working starting point instead of an empty file.

```bash theme={"dark"}
zenrows template list
zenrows template create <name> --output <dir>
```

| Command | Does |
| - | - |
| `zenrows template list` | List all templates |
| `zenrows template create <name> --output <dir>` | Instantiate a template into `<dir>` |
| `zenrows template install <name>` | Copy a template into `.zenrows/` |
| `zenrows template explain <name>` | Show a template's full docs |
| `zenrows template update [name]` | Refresh installed template(s) |
| `zenrows template remove <name>` | Remove an installed template |

## Catalog

| Template | Description |
| - | - |
| `protected-fetch-node` | Minimal Node.js project calling Fetch directly |
| `extract-pricing-pages` | Extract structured pricing fields from product pages |
| `batch-jsonl-pipeline` | JSONL job-spec scaffold for Batch |


# Troubleshooting
Source: https://docs.zenrows.com/cli/troubleshooting

Fix a failed run: read its trace and find the exact next step for every error code.

Every `fetch`/`extract`/`batch` run writes a secret-free record under `.zenrows/runs/<run-id>/`. Read it before reflexively adding more flags.

```bash theme={"dark"}
zenrows trace inspect <run-id>    # raw run record
zenrows trace explain <run-id>    # what happened and why
zenrows trace replay <run-id>     # print the equivalent command to re-run
zenrows trace export <run-id>     # JSON, for piping elsewhere
```

## Error codes

| Code | When it happens | Next action |
| - | - | - |
| `AUTH_MISSING` | No API key is configured | `zenrows login --api-key <your-key>` |
| `AUTH_INVALID` | Zenrows rejected the key | Re-check the key in the dashboard, then `zenrows login --api-key <your-key>` again |
| `DOMAIN_FORBIDDEN` | The target domain is blocked by Zenrows policy | Permanent, the same request fails again with any flags. Use a different source |
| `FETCH_EMPTY_RESPONSE` | The page returned no usable content | Retry with `--js-render` and a `--wait-for <selector>` or `--wait <ms>` |
| `PARAM_CONFLICT_AUTO_MANUAL` | `--js-render` / `--premium-proxy` passed without `--manual` | Drop those flags to keep the automatic default, or add `--manual` |
| `PARAM_PROXY_COUNTRY_REQUIRES_PREMIUM` | `--proxy-country` used in manual mode without `--premium-proxy` | Add `--premium-proxy` (10 credits), or drop `--manual` to use auto mode |
| `POLICY_MAX_CREDITS_EXCEEDED` | The **account** has spent its credit allowance | Credits renew at the end of the billing period — run `zenrows usage` for the date. To carry on now, claim the Free-plan account (if unclaimed) or add credits/upgrade in the dashboard |
| `POLICY_LIMIT_EXCEEDED` | The run exceeds a cap **you** set in `policy.json` (`max_credits_per_run`, `max_pages_per_run`) | Split the run, drop `--premium-proxy` / `--js-render` to lower the estimate, or raise the cap with `zenrows policy set` |
| `POLICY_EXPERIMENTAL_DISABLED` | `zenrows browser` used before enabling it | `zenrows policy set allow_browser true`, only once Fetch/Extract genuinely can't do the job |
| `MCP_CLIENT_UNSUPPORTED` | `--client` isn't a known MCP client id | Use one of the supported client ids, or generate a generic config snippet |
| `CAPABILITY_UNAVAILABLE` | The command's backend capability isn't enabled yet | Use a local spec/validation path, or escalate to Protected Fetch/Extract |


# Workflows
Source: https://docs.zenrows.com/cli/workflows

A named, multi-step process bundled with the Zenrows CLI, like keeping competitor data fresh automatically.

A workflow is a multi-step, end-to-end process built from Zenrows commands. Install one and run it as a single command instead of scripting each step yourself.

```bash theme={"dark"}
zenrows workflow list
zenrows workflow run <name>
```

| Command | Does |
| - | - |
| `zenrows workflow list` | List all workflows |
| `zenrows workflow run <name>` | Run a workflow end to end |
| `zenrows workflow explain <name>` | Show a workflow's full docs |
| `zenrows workflow install <name>` | Copy a workflow into `.zenrows/` |
| `zenrows workflow update [name]` | Refresh installed workflow(s) |
| `zenrows workflow remove <name>` | Remove an installed workflow |

## Catalog

| Workflow | Description |
| - | - |
| `competitor-intelligence` | Gather competitor pages, extract structured data, and keep it fresh |


# Extract Endpoints
Source: https://docs.zenrows.com/extract/endpoints

Reference for the Zenrows Extract endpoints: read which domains Extract serves today, ask for a new domain to be prepared, and track a preparation through to ready.

Extract's own endpoints live under `https://api.zenrows.com/v1/extract/`. They answer two questions that `extract=auto` cannot: **which domains work today**, and **how to get one that doesn't yet**.

None of them fetches a page, so none of them costs credits.

Authenticate the same way as any Zenrows request, with either the `apikey` query parameter or the `X-API-Key` header. Both work on every endpoint below; the examples alternate to show each.

## Which domains Extract serves

### `GET /v1/extract/domains`

Every domain Extract can handle right now. This is the list behind [Domain coverage](/extract/setup#domain-coverage), read live instead of from a page, so an integration can check coverage before sending a request rather than handling an error after one.

<CodeGroup>
  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/extract/domains?apikey=YOUR_ZENROWS_API_KEY"
  ```

  ```python Python theme={"dark"}
  # pip install requests
  import requests

  response = requests.get(
      'https://api.zenrows.com/v1/extract/domains',
      headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'},
  )
  supported = {entry['domain'] for entry in response.json()['domains']}
  print('amazon.com' in supported)
  ```

  ```javascript Node.js theme={"dark"}
  const response = await fetch('https://api.zenrows.com/v1/extract/domains', {
      headers: { 'X-API-Key': 'YOUR_ZENROWS_API_KEY' },
  });
  const { domains } = await response.json();
  const supported = new Set(domains.map((entry) => entry.domain));
  console.log(supported.has('amazon.com'));
  ```
</CodeGroup>

```json Response theme={"dark"}
{
  "domains": [
    {
      "domain": "amazon.com",
      "verticals": [
        "Automotive/Auto Parts",
        "Books and Literature/Fiction",
        "Shopping",
        "Technology & Computing/Consumer Electronics"
      ],
      "warm": true
    },
    {
      "domain": "americanas.com.br",
      "verticals": ["Shopping", "Technology & Computing/Consumer Electronics"],
      "warm": true
    }
  ]
}
```

| Field | Description |
| - | - |
| `domain` | The registrable domain. It covers every subdomain: `amazon.com` also means `www.amazon.com`. |
| `verticals` | The kinds of page Extract has prepared for this domain. A domain serving several verticals is one entry, not several. Roughly a quarter of entries have none, which does not mean the domain works any less. |
| `warm` | Always `true` today. The field exists for a future "declared but not prepared yet" state. |

A domain in this list is a domain you can send `extract=auto` to. A domain missing from it returns [`REQS007`](/api-error-codes#REQS007), naming the domain and pointing at the endpoint below.

## Getting a domain prepared

Extract works on a domain once Zenrows has prepared it: learned its page shapes and verified that extraction actually returns the right data. You can ask for a domain to be prepared yourself, within a monthly allowance.

<Note>
  Preparing a domain is not instant. A submission is accepted immediately and then works through discovery, preparation and verification, typically minutes rather than seconds. Poll the status endpoint rather than blocking on the submit call.
</Note>

### `POST /v1/extract/prepared-domains`

Ask for a domain to be prepared. The body takes the domain and, optionally, example URLs.

**Send example URLs when you can.** They are authoritative: Zenrows prepares exactly the page kinds you point at, and skips the discovery step that guesses them. One URL per page kind you care about is enough: a product page, a search results page.

| Field | Required | Description |
| - | - | - |
| `domain` | yes | The domain to prepare, without scheme. |
| `example_urls` | no | URLs on that domain, one per page kind. When present, they replace discovery. |

Send an `Idempotency-Key` header to make a retry safe: the same key returns the original submission instead of spending a second allowance slot.

<CodeGroup>
  ```bash cURL theme={"dark"}
  curl -X POST "https://api.zenrows.com/v1/extract/prepared-domains?apikey=YOUR_ZENROWS_API_KEY" \
    -H "Content-Type: application/json" \
    -H "Idempotency-Key: 8f14e45f-ea2c-4f2b-9c1a-7d3b6a0e5c91" \
    -d '{
      "domain": "scrapingcourse.com",
      "example_urls": [
        "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie",
        "https://www.scrapingcourse.com/ecommerce/"
      ]
    }'
  ```

  ```python Python theme={"dark"}
  # pip install requests
  import requests

  response = requests.post(
      'https://api.zenrows.com/v1/extract/prepared-domains',
      headers={
          'X-API-Key': 'YOUR_ZENROWS_API_KEY',
          'Idempotency-Key': '8f14e45f-ea2c-4f2b-9c1a-7d3b6a0e5c91',
      },
      json={
          'domain': 'scrapingcourse.com',
          'example_urls': [
              'https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie',
              'https://www.scrapingcourse.com/ecommerce/',
          ],
      },
  )
  print(response.status_code, response.json()['state'])
  ```
</CodeGroup>

A submission returns **`202 Accepted`** and its starting state, in the same shape the status endpoint returns:

```json Response theme={"dark"}
{
  "domain": "scrapingcourse.com",
  "status": "preparing",
  "state": "received",
  "submitted_at": "2026-09-14T08:02:38.686126Z",
  "updated_at": "2026-09-14T08:02:38.686126Z"
}
```

Submitting a domain that is already prepared returns `202` with its current record rather than starting again, so a retry is never wasted.

### `GET /v1/extract/prepared-domains/{domain}`

How far a submission has got.

<CodeGroup>
  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/extract/prepared-domains/scrapingcourse.com?apikey=YOUR_ZENROWS_API_KEY"
  ```

  ```python Python theme={"dark"}
  # pip install requests
  import requests, time

  while True:
      response = requests.get(
          'https://api.zenrows.com/v1/extract/prepared-domains/scrapingcourse.com',
          headers={'X-API-Key': 'YOUR_ZENROWS_API_KEY'},
      )
      body = response.json()
      if body['status'] != 'preparing':
          break
      time.sleep(30)

  print(body['status'])
  ```
</CodeGroup>

A domain that was never submitted returns `404`:

```json Response theme={"dark"}
{
  "code": "not_found",
  "detail": "This domain has never been submitted.",
  "status": 404,
  "title": "Not found (not_found)",
  "type": "https://docs.zenrows.com/api-error-codes#not_found"
}
```

#### Reading the response

Two fields describe progress, at two levels of detail. **Branch on `status`; show `state` if you are drawing a progress bar.**

`status` is the answer to "can I use this domain yet":

| `status` | Meaning |
| - | - |
| `preparing` | Still working. Keep polling. |
| `ready` | Done. `extract=auto` works on this domain now. |
| `failed` | Stopped. See `failure`. |

`state` is the current step, and will gain steps over time:

| `state` | Meaning |
| - | - |
| `received` | Accepted, not started. |
| `discovering` | Finding the domain's page kinds. Skipped when you sent `example_urls`. |
| `preparing` | Building the extraction for each page kind. |
| `verifying` | Checking that extraction returns the right data. |
| `ready` / `failed` | Terminal, matching `status`. |

`roles` breaks the same progress down per page kind, each with its own `status`, the `url` it is working from, and a `detail` sentence if it stopped. It is only present for the account that submitted the domain, because example URLs disclose what you scrape and are not shown to other accounts polling it.

#### When a preparation fails

A `failed` submission carries a `failure` block with a `code`, a `detail` sentence, and `slot_consumed`:

```json Response theme={"dark"}
{
  "domain": "scrapingcourse.com",
  "status": "failed",
  "state": "failed",
  "roles": [
    {
      "role": "",
      "status": "failed",
      "url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie",
      "detail": "the target could not be fetched; the gateway rejected this request"
    }
  ],
  "failure": {
    "code": "unpreparable",
    "detail": "we could not prepare this domain: 1 of 1 pages could not be prepared: https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie (the target could not be fetched; the gateway rejected this request)",
    "slot_consumed": false
  },
  "submitted_at": "2026-09-14T08:02:38.686126Z",
  "updated_at": "2026-09-14T08:02:39.927245Z"
}
```

`slot_consumed` tells you whether the attempt cost you a slot, but **it does not tell you what to do next** on its own. Two failures hand the slot back and still need different responses: one is ours to fix and a plain retry works, the other will fail again with the same URLs.

Read the code, and use `slot_consumed` for the billing half of the sentence:

| `code` | What happened | What to do | `slot_consumed` |
| - | - | - | - |
| `interrupted` | Processing stopped on our side. | Resubmit exactly as you sent it. | `false` |
| `preparation_exhausted` | We could not produce a verifiable extraction in two independent attempts. Ours, not yours. | Resubmit exactly as you sent it. | `false` |
| `discovery_failed` | We could not work out the domain's page kinds on our own. | Resubmit with `example_urls`. | `false` |
| `unpreparable` | None of the pages you sent gave us usable HTML: blocked, a challenge page, or not HTML at all. | Try different example URLs from the same site. The same ones will fail again. | `false` |
| `preparation_failed` | Extraction could not be built for a page kind. | Try other example URLs, or contact support. | `true` |
| `verification_failed` | Extraction was built but did not meet the quality bar. | Contact support. | `true` |
| `unsupported` | The domain cannot be handled. | Contact support before spending more slots on it. | `true` |

### `GET /v1/extract/prepared-domains`

Every domain your account has submitted, with your allowance alongside it.

<CodeGroup>
  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/extract/prepared-domains?apikey=YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

```json Response theme={"dark"}
{
  "allowance": {
    "used": 0,
    "limit": 10,
    "remaining": 10,
    "month": "2026-09"
  },
  "domains": []
}
```

This lists what **you** submitted, which is a different question from [which domains Extract serves](#which-domains-extract-serves). That one lists every prepared domain, however it got there.

### `GET /v1/extract/prepared-domains/allowance`

How many domains you can still ask for this month, on its own.

<CodeGroup>
  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/extract/prepared-domains/allowance?apikey=YOUR_ZENROWS_API_KEY"
  ```
</CodeGroup>

```json Response theme={"dark"}
{
  "used": 0,
  "limit": 10,
  "remaining": 10,
  "month": "2026-09"
}
```

`month` is the UTC calendar month the count applies to; the allowance resets when it rolls over. A failure that returns your slot (`slot_consumed: false`) is not counted in `used`.

## Errors

Alongside the [standard Zenrows error codes](/api-error-codes), these endpoints return:

| Code | Status | Meaning |
| - | - | - |
| `invalid_argument` | `400` | The request body is missing or malformed. `detail` says what, and `reason` carries a machine-readable tag such as `invalid_domain`. |
| `not_found` | `404` | This account has never submitted that domain. |
| `in_progress` | `409` | That domain already has a preparation running, possibly someone else's. Poll its status instead of resubmitting. |
| `idempotency_conflict` | `409` | The same `Idempotency-Key` was reused with a different body. |
| `out_of_allowance` | `429` | No preparation slots left this month. Check `GET /v1/extract/prepared-domains/allowance`. |

An invalid submission is rejected before it costs anything:

```json Response theme={"dark"}
{
  "code": "invalid_argument",
  "detail": "Domain is required.",
  "reason": "invalid_domain",
  "status": 400,
  "title": "Invalid argument (invalid_argument)",
  "type": "https://docs.zenrows.com/api-error-codes#invalid_argument"
}
```


# Introduction to Extract
Source: https://docs.zenrows.com/extract/introduction

Turn any web page into structured JSON in a single request with Zenrows Extract, which builds a tailored extraction for each website and keeps it working when the site's layout changes.

<Note>
  Extract is in **Beta**. It works on any domain once that domain has been prepared, and you can ask for a new one to be prepared yourself. Any active Zenrows API key can use it, with no extra setup needed.
</Note>

Zenrows Extract turns a web page into structured JSON in a single request. Add `extract=auto` to your Fetch call, and Zenrows identifies the meaningful data on the page and returns it as clean JSON, together with the raw HTML. There are no selectors to write, no schema to define, and no parser to maintain, and the extraction adapts automatically when a website changes its layout.

## How It Works

Unlike general-purpose parsing rules that apply the same logic to every website, Extract builds and maintains an extraction that's tailored to each individual site. Zenrows analyzes a domain's pages, identifies its meaningful data points, and keeps that extraction working as the site evolves, so you get richer and more complete fields than a one-size-fits-all approach can offer.

## Basic Usage

Enable Extract by adding the `extract=auto` parameter to your Zenrows request:

```bash cURL theme={"dark"}
curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto"
```

The response returns `Content-Type: application/json` with a `parsed` field containing the structured data, and an `html` field with the raw page for validation during Beta.

Two optional parameters shape what comes back: [`extract_fields`](/extract/setup#selecting-the-fields-you-need) narrows `parsed` to the fields you name, and [`extract=standard`](/extract/setup#extraction-modes) swaps each website's own field names for a canonical schema shared across a vertical, on the API and currently for real estate only.

## Next Steps

<CardGroup>
  <Card title="Extract Reference" icon="book" href="/extract/setup">
    Full parameter reference: response shape, domain coverage, pricing, and FAQ.
  </Card>

  <Card title="Autoparse (Legacy)" icon="wand" href="/fetch/features/autoparse">
    The deprecated general-purpose parser, and how to migrate an integration off it.
  </Card>
</CardGroup>


# Extract
Source: https://docs.zenrows.com/extract/sdk-overview

Extract doesn't need a separate SDK. Use the same Fetch client for your language and pass extract=auto as a request parameter.

Extract has no dedicated client or package. It's the same SDK client you already use for Fetch, with `extract=auto` set as a request parameter.

```python Python theme={"dark"}
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://www.scrapingcourse.com/ecommerce/"

response = client.get(url, params={"extract": "auto"})

print(response.json())
```

The response comes back as JSON with a `parsed` field containing the structured data Zenrows identified on the page, plus the raw `html` for validation.

Extract's other parameters ride in the same `params` map: `{"extract": "auto", "extract_fields": "products.name,products.price"}` returns only the fields you name, and `{"extract": "standard"}` picks the [canonical schema](/extract/setup#extraction-modes) instead of the website's own field names.

<CardGroup>
  <Card title="Extract Introduction" icon="wand" href="/extract/introduction">
    What Extract does and how it works.
  </Card>

  <Card title="Extract Reference" icon="book" href="/extract/setup">
    Full parameter reference: response shape, domain coverage, pricing, and FAQ.
  </Card>
</CardGroup>


# Extract
Source: https://docs.zenrows.com/extract/setup

Turn any web page into structured JSON in a single request with Zenrows Extract, which builds a tailored extraction for each website and keeps it working when the site's layout changes.

Extract turns a web page into structured JSON in a single request. Add `extract=auto` to your Fetch call, and Zenrows identifies the meaningful data on the page and returns it as clean JSON, together with the raw HTML. There are no selectors to write, no schema to define, and no parser to maintain. The extraction adapts automatically when a website changes its layout.

<Note>
  Extract is in **Beta** and works on domains Zenrows has prepared. Any active Zenrows API key can use it on those, with no extra setup needed, and you can ask for a new domain to be prepared yourself.
</Note>

## How Extract works

Unlike general-purpose parsing rules that apply the same logic to every website, Extract builds and maintains an extraction that's tailored to each individual site. Zenrows analyzes a domain's pages, identifies its meaningful data points, and keeps that extraction working as the site evolves, so you get richer and more complete fields than a one-size-fits-all approach can offer.

That tailoring is per domain, so a domain has to be prepared before Extract can serve it. Any domain can be: see [Domain coverage](#domain-coverage).

## Basic usage

Enable Extract by adding the `extract=auto` parameter to your Zenrows request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'extract': 'auto',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.json())
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'extract': 'auto',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto"
  ```
</CodeGroup>

This example extracts structured data from the Scraping Course eCommerce page. Instead of raw HTML alone, you get a JSON response with the meaningful data Zenrows identified on the page, plus the original HTML for reference.

## Parameters

Extract adds two parameters to a [Fetch](/fetch/api-reference) request. Every other Fetch parameter keeps working alongside them.

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **extract** | `string` | | Turns Extract on and picks the extraction mode. Accepted values: `auto`, `native`, `standard`.<br />*Use-case: [Get the same field names across websites](#extraction-modes)* |
| **extract\_fields** | `string` | | Comma-separated list of the fields to return inside `parsed`, dot-notated for fields inside a group. Omit it to receive every field.<br />*Use-case: [Return only the fields you need](#selecting-the-fields-you-need)* |

## Response

A request with `extract=auto` returns `Content-Type: application/json` with two fields:

```json JSON theme={"dark"}
{
  "html": "<!DOCTYPE html><html>...</html>",
  "parsed": {
    "page_context": {
      "current_page": 1,
      "results_per_page": 69,
      "total_pages": 12,
      "total_results": 188
    },
    "products": [
      {
        "name": "Abominable Hoodie",
        "price": 69,
        "currency": "USD",
        "availability": "in_stock",
        "sku": "MH09",
        "product_id": "246",
        "product_type": "variable",
        "image_url": "https://www.scrapingcourse.com/.../mh09-blue_main.jpg",
        "product_url": "https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"
      }
    ]
  }
}
```

* **`parsed`** contains the structured data extracted from the page.
* **`html`** contains the raw HTML of the scraped page. It's included during the beta so you can validate the extracted data against the source.

Field names are in plain English and reflect the data each website actually exposes, so different websites can return different fields for similar pages. The set of fields inside `parsed` stays stable across requests for a given website, and new fields may be added over time as extraction improves, so treat unknown fields as additive rather than breaking. The `html` field is included for validation during the beta, and the overall response shape may evolve once the beta ends, so build your data pipeline around `parsed` and treat `html` as a validation aid rather than a permanent contract.

<Warning>Extraction is fully automatic, and a field is only returned when it's present on the page. Always run test requests on your target pages and verify all the data you need is extracted before using Extract in production.</Warning>

## Selecting the fields you need

`parsed` carries every field Extract found on the page. Add `extract_fields` with a comma-separated list to get back only the ones your pipeline reads:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'extract': 'auto',
      'extract_fields': 'products.name,products.price',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.json())
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'extract': 'auto',
          'extract_fields': 'products.name,products.price',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto&extract_fields=products.name%2Cproducts.price"
  ```
</CodeGroup>

```json Response theme={"dark"}
{
  "html": "<!DOCTYPE html><html>...</html>",
  "parsed": {
    "products": [
      { "name": "Abominable Hoodie", "price": 69 },
      { "name": "Adrienne Trek Jacket", "price": 57 }
    ]
  }
}
```

How the list is read:

* **A bare name selects a whole group.** `extract_fields=products` returns every product with all of its fields.
* **A dot path selects inside a group, element by element.** `products.name` returns the `name` of every product in the list, not only the first one.
* **Spaces after commas are ignored.** `products.name, products.price` selects the same fields as `products.name,products.price`.
* **Only `parsed` is filtered.** The `html` field always comes back in full.
* **Filtering never adds a field.** Naming something the page doesn't provide doesn't produce an empty field, it just isn't there.

It works the same way in every extraction mode, so a `standard` request can be narrowed with the canonical field names: `extract=standard&extract_fields=listings.title,listings.price`.

<Note>Field names come from each website, so they differ from site to site in the default mode. Send one request without `extract_fields` first, read the names in `parsed`, then pin the ones you want.</Note>

### When none of the requested fields exist

A name that matches nothing is ignored as long as at least one other name in the list does match, so a typo costs you that field quietly. When **no** name matches, the request fails with `400` ([`REQS004`](/api-error-codes#REQS004)) rather than returning an empty object, and the message lists the fields the page does offer:

```json Response theme={"dark"}
{
  "code": "REQS004",
  "detail": "None of the fields requested in 'extract_fields' are available for this page (requested: prize). Available fields: page_context, page_context.current_page, page_context.results_per_page, page_context.total_pages, page_context.total_results, products, products.availability, products.currency, products.image_url, products.name, products.price, products.product_id, products.product_type, products.product_url, products.sku. Adjust 'extract_fields', or omit it to receive every field this page provides.",
  "instance": "/v1",
  "status": 400,
  "title": "None of the requested 'extract_fields' are available for this page (REQS004)",
  "type": "https://docs.zenrows.com/api-error-codes#REQS004"
}
```

## Extraction modes

The value of `extract` also picks which extraction resolves for the page:

| Value | Extraction |
| - | - |
| `auto` | The default, and the same extraction as `native`. |
| `native` | The website's own extraction. Field names follow what that site exposes, so they are as rich and as specific as the page itself. |
| `standard` | A canonical schema shared by every website in the same vertical. The same field names come back from every site, so pages from different websites line up without mapping. |

Stay on `auto` unless you're merging pages from several websites into one table, which is the job `standard` exists for. The trade is richness for uniformity: a native extraction returns everything a given site exposes, and a standard one returns the agreed set.

<Note>
  The mode is an API-level parameter. The playground in your [dashboard](https://app.zenrows.com) runs the default extraction and lists `standard` as coming soon, so pick a mode from a direct API request or from an SDK, which passes the parameter straight through.
</Note>

The difference is visible on two real-estate search pages from different websites. In the default mode, each site names things its own way, one calling floor area `construction_area_m2` and the other `area_sqm`, one splitting the price into `price` and `price_currency`, the other into `price_amount` and `price_currency`. In `standard`, both return the same field names:

<CodeGroup>
  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.casasyterrenos.com%2Fyucatan%2Fmerida%2Fcasas%2Fventa&extract=standard"
  ```

  ```python Python theme={"dark"}
  # pip install requests
  import requests

  apikey = 'YOUR_ZENROWS_API_KEY'
  for url in [
      'https://www.casasyterrenos.com/yucatan/merida/casas/venta',
      'https://www.lamudi.com.mx/yucatan/merida/for-sale/',
  ]:
      response = requests.get('https://api.zenrows.com/v1/', params={
          'url': url,
          'apikey': apikey,
          'extract': 'standard',
      })
      listings = response.json()['parsed']['listings']
      print(url, sorted(listings[0]))  # the same field names on both sites
  ```
</CodeGroup>

```json Response theme={"dark"}
{
  "html": "<!DOCTYPE html><html>...</html>",
  "parsed": {
    "search": {
      "location": "merida",
      "total_results": 169
    },
    "listings": [
      {
        "title": "Venta de Casa en San jeronimo en Conkal",
        "listing_type": "venta",
        "property_type": "casa",
        "price": { "value": 4500000, "currency": "MXN" },
        "bedrooms": 3,
        "bathrooms": 3,
        "living_area": 215,
        "area_unit": "m²",
        "address": "Conkal, Yucatán",
        "city": "Conkal",
        "postal_code": null,
        "latitude": null,
        "longitude": null,
        "agent_name": null,
        "image": "https://cdn.casasyterrenos.com/img_casas/26-Casas-Renta-Pueblo-Conkal-Conkal-Yucatan-3879000-SQzkaHpu.jpg",
        "listing_id": "3879000",
        "url": "/propiedad/casa-venta-conkal-pueblo-conkal-conkal-yuc-3879000"
      }
    ]
  }
}
```

<Warning>
  `standard` currently covers **real estate** only, both search results and listing pages. A page in any other vertical is rejected with `400` ([`REQS004`](/api-error-codes#REQS004)), so use `auto` or `native` everywhere else.
</Warning>

Any value other than the three above is rejected before the page is even fetched, with `400` ([`REQS004`](/api-error-codes#REQS004)) and the message `Invalid value "..." for 'extract' parameter. Supported values: auto, native, standard`.

## Domain coverage

Extract works on **any domain**, once that domain has been prepared: Zenrows learns the shapes of its pages and checks that extraction returns the right data. A prepared domain then works for every Zenrows API key, so coverage is per domain and never per account, and there is no allowlist to manage on your side.

Many domains are prepared already, and the set grows as people ask for the sites they need.

### Checking whether a domain is ready

`GET /v1/extract/domains` returns everything Extract serves right now:

```bash cURL theme={"dark"}
curl "https://api.zenrows.com/v1/extract/domains?apikey=YOUR_ZENROWS_API_KEY"
```

```json Response theme={"dark"}
{
  "domains": [
    {
      "domain": "zillow.com",
      "verticals": ["Real Estate/Houses", "Real Estate/Real Estate Buying and Selling"],
      "warm": true
    },
    {
      "domain": "amazon.com",
      "verticals": ["Shopping", "Technology & Computing/Consumer Electronics"],
      "warm": true
    }
  ]
}
```

An integration can check this before sending a request instead of handling an error after one. See the [endpoint reference](/extract/endpoints#which-domains-extract-serves) for the full response.

### Asking for a domain

If the domain you need isn't there, ask for it:

```bash cURL theme={"dark"}
curl -X POST "https://api.zenrows.com/v1/extract/prepared-domains?apikey=YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example-shop.com",
    "example_urls": ["https://example-shop.com/product/blue-shirt"]
  }'
```

Preparation usually takes minutes. Poll [`GET /v1/extract/prepared-domains/{domain}`](/extract/endpoints#getting-a-domain-prepared) until `status` is `ready`, then send your `extract=auto` request as normal. Example URLs are worth including: they tell Zenrows exactly which kinds of page you care about.

Each account can prepare a set number of new domains per month; the [endpoint reference](/extract/endpoints#getting-a-domain-prepared) covers how to check what's left.

If you send `extract=auto` for a domain that isn't prepared, the request fails with a `403` error ([`REQS007`](/api-error-codes#REQS007)) naming the domain, and the message tells you how to get it prepared. Nothing about your plan or your account changes that: preparation is per domain.

## Combining with other features

Extract works on the final HTML of the page, so it combines with the scraping features you already use. For JavaScript-heavy websites, add JavaScript Rendering with a short wait so dynamic content finishes loading before extraction. For protected websites, add Premium Proxy:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'extract': 'auto',
      'js_render': 'true',
      'wait': '3000',
      'premium_proxy': 'true',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.json())
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&extract=auto&js_render=true&wait=3000&premium_proxy=true"
  ```
</CodeGroup>

<Note>When `extract=auto` is set, it takes precedence over other output formats. `autoparse`, `css_extractor`, `response_type`, and `outputs` are ignored on that request.</Note>

## Extract vs. Autoparse vs. CSS Extractor

| Method | Best for | Pros | Cons |
| - | - | - | - |
| **Extract** (`extract=auto`) | Any site where you want the richest data | Tailored to each website, adapts when the site's layout changes, richer fields | The domain has to be prepared first |
| **Autoparse** (`autoparse=true`, deprecated) | Integrations already built on it | Works on any site, no setup, included at no extra cost | General-purpose rules, often returns page scaffolding rather than the data you want. [Migrate to Extract](/fetch/features/autoparse#migrate-to-extract) |
| **CSS Extractor** (`css_extractor`) | Specific data points, single site, custom requirements | Full control, precise targeting | Requires HTML knowledge, selectors are site-specific |

## Pricing

Extract is **free during Beta**. You only pay the standard Fetch cost of the underlying request, including JavaScript Rendering or Premium Proxy if you use them.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<AccordionGroup>
  <Accordion title="How is Extract different from Autoparse?">
    Autoparse (`autoparse=true`) applies general-purpose extraction rules that work the same way on every website, and it is deprecated. Extract (`extract=auto`) builds an extraction tailored to each website, which returns richer, more complete fields and adjusts itself when the website's layout changes. Extract needs a domain to be prepared first, which you can request; see [Migrate to Extract](/fetch/features/autoparse#migrate-to-extract) for the parameter swap.
  </Accordion>

  <Accordion title="What happens on a domain that isn't prepared yet?">
    The request fails with a `403` error ([`REQS007`](/api-error-codes#REQS007)) naming the domain, and the message tells you how to get it prepared: [`POST /v1/extract/prepared-domains`](/extract/endpoints#getting-a-domain-prepared), then retry once it reports `ready`. Nothing about your plan or your account changes the outcome, since preparation is per domain.
  </Accordion>

  <Accordion title="Can I get only some of the fields back?">
    Yes. Add [`extract_fields`](#selecting-the-fields-you-need) with a comma-separated list, using dot paths for fields inside a group: `extract_fields=products.name,products.price`. A dot path applies to every element of a list, so that example returns the name and price of every product. Omit the parameter to receive every field the page provides.
  </Accordion>

  <Accordion title="What is the difference between the native and standard extraction modes?">
    `extract=auto` (the default) and `extract=native` return the website's own extraction, so field names follow what that particular site exposes and differ from site to site. `extract=standard` returns a canonical schema shared by every website in the same vertical, so pages from different websites line up without mapping. `standard` currently covers real estate only, and modes are available on the API, not yet in the dashboard playground. See [Extraction modes](#extraction-modes).
  </Accordion>

  <Accordion title="Will the extracted fields change over time?">
    The response shape for an enabled website stays stable. Your integration keeps receiving the same fields even as Zenrows improves extraction behind the scenes. New fields may be added over time, so treat unknown fields as additive rather than breaking.
  </Accordion>

  <Accordion title="Do I need to sign up separately to use Extract?">
    No. Access is per domain, not per account. Any active Zenrows API key can use `extract=auto` on a domain once that domain is prepared, and any key can ask for a domain to be prepared. There's no separate signup or plan change required.
  </Accordion>

  <Accordion title="Can I rely on the html field long term?">
    Treat `html` as a validation aid rather than a permanent part of the contract. It's included during the beta so you can check the extracted data against the source page. Build your data pipeline around `parsed`, since the surrounding response shape may evolve once the beta ends.
  </Accordion>

  <Accordion title="Can I combine Extract with other Zenrows features?">
    Yes. Extract works alongside JavaScript Rendering, Premium Proxy, and other request options. It takes precedence over other output formats, so `autoparse`, `css_extractor`, `response_type`, and `outputs` are ignored on a request that sets `extract=auto`.
  </Accordion>
</AccordionGroup>


# Introduction to Fetch
Source: https://docs.zenrows.com/fetch/api-reference

Explore the full Zenrows Fetch reference, including endpoint parameters, authentication, pricing tiers, and feature documentation.

Zenrows Fetch (formerly, Universal Scraper API) is a versatile tool designed to simplify and enhance the process of extracting data from websites. Whether you're dealing with static or dynamic content, our API provides a range of features to meet your scraping needs efficiently.

With Premium Proxies, Zenrows gives you access to over 55 million residential IPs from 190+ countries, ensuring 99.9% uptime and highly reliable scraping sessions. Our system also handles advanced fingerprinting, header rotation, and IP management, **enabling you to scrape even the most protected sites without needing to manually configure these elements**.

Zenrows makes it easy to bypass complex anti-bot measures, handle JavaScript-heavy sites, and interact with web elements dynamically (all with the right features enabled).

## Key Features

### Adaptive Stealth Mode

Let Zenrows automatically choose the proper configuration for each request. Set `mode=auto`, and the API starts with the cheapest viable setup, then escalates to JavaScript rendering or Premium Proxies only when needed. So you get reliable extraction without manual tuning or overpaying.

**When to use:** Use Adaptive Stealth Mode when scraping multiple or unknown sites, when you want to minimize maintenance as sites change, or when reliability matters more than fine-grained control over parameters.

**Real-world scenarios:**

* Production workloads across many different domains
* Sites whose anti-bot or rendering requirements change over time
* Ensuring successful extraction by starting with simple requests and automatically escalating to JS rendering or premium proxies only when needed

**Additional options:**

* You are billed only for the configuration that succeeds; failed attempts are not charged
* Managed automatically: `js_render` and `premium_proxy` (you are also allowed to choose the `proxy_country`)
* Other parameters (e.g. `js_instructions`, `custom_headers`) work alongside Adaptive Stealth Mode

### JavaScript Rendering

Render JavaScript on web pages using a headless browser to scrape dynamic content that traditional methods might miss.

**When to use:** Use this feature when targeting modern websites built with JavaScript frameworks (React, Vue, Angular), single-page applications (SPAs), or any site that loads content dynamically after the initial page load.

**Real-world scenarios:**

* E-commerce product listings that load items as you scroll
* Dashboards and analytics platforms that render charts/data with JavaScript
* Social media feeds that dynamically append content
* Sites that hide certain content until JavaScript is rendered

**Additional options:**

* Wait times to ensure elements are fully loaded
* Interaction with the page to click buttons, fill forms, or scroll
* Screenshot capabilities for visual verification
* CSS-based extraction of specific elements

### Premium Proxies

Leverage a vast network of residential IP addresses across 190+ countries, ensuring a 99.9% uptime for uninterrupted scraping.

**When to use:** Essential for accessing websites with sophisticated anti-bot systems, geo-restricted content, or when you consistently encounter blocks with standard datacenter IPs.

**Real-world scenarios:**

* Scraping major e-commerce platforms (Amazon, Walmart)
* Accessing real estate listings (Zillow, Redfin)
* Gathering pricing data from travel sites (Expedia, Booking.com)
* Collecting data from financial or investment platforms

**Additional options:**

* Geolocation selection to access region-specific content
* Automatic IP rotation to prevent detection

### Custom Headers

Add custom HTTP headers to your requests for more control over how your requests appear to target websites.

**When to use:** When you need to mimic specific browser behavior, set cookies, or a referer.

**Real-world scenarios:**

* Setting language preferences to get content in specific languages
* Adding a referer to avoid being blocked by bot detection systems

### Session Management

Use a session ID to maintain the same IP address across multiple requests for up to 10 minutes.

**When to use:** When scraping multi-page flows or processes that require maintaining the same IP across multiple requests.

**Real-world scenarios:**

* Multi-step forms processes
* Maintaining consistent session for search results and item visits

### Advanced Data Extraction

Extract only the data you need with CSS selectors or automatic parsing.

**When to use:** When you need specific information from pages and want to reduce bandwidth usage or simplify post-processing.

**Real-world scenarios:**

* Extracting pricing information from product pages
* Gathering contact details from business directories
* Collecting specific metrics from analytics pages

### Language agnostic

While Python examples are provided, the API works with any programming language that can make HTTP requests.

## Parameter Overview

Customize your scraping requests using the following parameters:

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **apikey** `required` | `string` | [**Get Your Free API Key**](https://app.zenrows.com/register?p=free) | Your unique API key for authentication |
| **url** `required` | `string` | | The URL of the page you want to scrape |
| [**mode**](/fetch/features/adaptive-stealth-mode) | `string` | | Enables Adaptive Stealth Mode when set to `auto`.<br />*Use-case: [Adaptive Stealth Mode vs Manual Parameters](/fetch/features/adaptive-stealth-mode#adaptive-stealth-mode-vs-manual-configuration)* |
| [**js\_render**](/fetch/features/js-rendering) | `boolean` | false | Enable JavaScript rendering with a headless browser. Essential for modern web apps, SPAs, and sites with dynamic content.<br />*Use-case: [Load dynamic pages (SPAs, dashboards, infinite scroll)](/fetch/features/js-rendering#when-to-use-javascript-rendering)* |
| [**js\_instructions**](/fetch/features/js-instructions) | `string` | | Execute custom JavaScript on the page to interact with elements, scroll, click buttons, or manipulate content. Use when you need to perform actions before the content is returned.<br />*Use-case: [Submit forms or simulate user actions](/fetch/features/js-instructions#common-use-cases-and-workflows)* |
| [**custom\_headers**](/fetch/features/headers) | `boolean` | false | Enables you to add custom HTTP headers to your request, such as cookies or referer, to better simulate real browser traffic or provide site-specific information.<br />*Use-case: [Simulate browser behavior for reliable access](/fetch/features/headers#referrer-simulation)* |
| [**premium\_proxy**](/fetch/features/premium-proxy) | `boolean` | false | Route through residential IPs for protected access to sites with anti-bot protection.<br />*Use-case: [Access protected sites using residential IPs](/fetch/features/premium-proxy#basic-usage)* |
| [**proxy\_country**](/fetch/features/proxy-country) | `string` | | Set the country of the IP used for the request (requires Premium Proxies). Use for accessing geo-restricted content or seeing region-specific content.<br />*Use-case: [Access geo-restricted content](/fetch/features/proxy-country#common-use-cases)* |
| [**session\_id**](/fetch/features/other#session-id) | `integer` | | Maintain the same IP for multiple requests for up to 10 minutes. Essential for multi-step processes.<br />*Use-case: [Keep session/IP across requests](/fetch/features/other#session-id)* |
| [**original\_status**](/fetch/features/other#original-http-code) | `boolean` | false | Return the original HTTP status code from the target page. Useful for debugging in case of errors.<br />*Use-case: [Debug failed requests](/fetch/features/other#original-http-code)* |
| [**allowed\_status\_codes**](/fetch/features/other#return-content-on-error) | `string` | | Returns the content even if the target page fails with specified status codes. Useful for debugging or when you need content from error pages.<br />*Use-case: [Debug failed requests and return the failed page](/fetch/features/other#return-content-on-error)* |
| [**wait\_for**](/fetch/features/wait-for) | `string` | | Wait for a specific CSS Selector to appear in the DOM before returning content. Essential for elements that load asynchronously.<br />*Use-case: [Capture elements that load at unpredictable times](/fetch/features/wait-for#basic-usage)* |
| [**wait**](/fetch/features/wait) | `integer` | 0 | Wait a fixed amount of milliseconds after page load. Use for sites that load content in stages or have delayed rendering.<br />*Use-case: [Pause the request process after the initial page load](/fetch/features/wait#basic-usage)* |
| [**block\_resources**](/fetch/features/block-resources) | `string` | | Block specific resources (images, fonts, etc.) from loading to speed up scraping and reduce bandwidth usage. Enabled by default, careful when changing it.<br />*Use-case: [Optimize performance and minimize bandwidth](/fetch/features/block-resources#basic-usage)* |
| [**json\_response**](/fetch/features/json-response) | `string` | false | Capture network requests in JSON format, including XHR or Fetch data. Ideal for intercepting API calls made by the web page.<br />*Use-case: [Capture API calls and background requests](/fetch/features/json-response#basic-usage)* |
| [**css\_extractor**](/fetch/features/css-extractor) | `string (JSON)` | | Extract specific elements using CSS selectors. Perfect for targeting only the data you need from complex pages.<br />*Use-case: [Extract only specific data fields](/fetch/features/css-extractor)* |
| [**extract**](/extract/setup) | `string` | | Extract structured JSON data from the page automatically, with no selectors to write. Beta. Accepted values: `auto`, `native`, `standard`.<br />*Use-case: [Get structured data from a page](/extract/setup)* |
| [**extract\_fields**](/extract/setup#selecting-the-fields-you-need) | `string` | | Comma-separated list of the fields `extract` should return, dot-notated for fields inside a group. Omit it to receive every field.<br />*Use-case: [Return only the fields you need](/extract/setup#selecting-the-fields-you-need)* |
| [**autoparse**](/fetch/features/autoparse) *(deprecated)* | `boolean` | false | General-purpose structured data from HTML. Deprecated in favor of `extract`, which is tailored per website. Still works for existing integrations.<br />*Use-case: [Migrate to Extract](/fetch/features/autoparse#migrate-to-extract)* |
| [**response\_type**](/fetch/features/markdown) | `string` | | Convert HTML to other formats (Markdown, Plaintext, PDF). Useful for content readability, storage, or to train AI models.<br />*Use-case: [Extract only specific formats](/fetch/features/markdown)* |
| [**screenshot**](/fetch/features/screenshot) | `boolean` | false | Capture an above-the-fold screenshot of the page. Helpful for visual verification or debugging.<br />*Use-case: [Visual verification of page rendering](/fetch/features/screenshot)* |
| [**screenshot\_fullpage**](/fetch/features/screenshot) | `boolean` | false | Capture a full-page screenshot. Useful for content that extends below the fold.<br />*Use-case: [Capture complete page content](/fetch/features/screenshot#capture-mode)* |
| [**screenshot\_selector**](/fetch/features/screenshot) | `string` | | Capture a screenshot of a specific element using CSS Selector. Perfect for capturing specific components.<br />*Use-case: [Capture specific page elements](/fetch/features/screenshot#capture-mode)* |
| [**screenshot\_format**](/fetch/features/screenshot#image-format-and-quality) | `string` | | Choose between `png` (default) and `jpeg` formats for screenshots.<br />*Use-case: [Choose appropriate image format](/fetch/features/screenshot#image-format-and-quality)* |
| [**screenshot\_quality**](/fetch/features/screenshot#image-format-and-quality) | `integer` | | For JPEG format, set quality from `1` to `100`. Lower values reduce file size but decrease quality.<br />*Use-case: [Control image compression levels](/fetch/features/screenshot#image-format-and-quality)* |
| [**outputs**](/fetch/features/output-filters) | `string` | | Specify which data types to extract from the scraped HTML.<br />*Use-case: [Extract only specific data fields](/fetch/features/output-filters)* |

## Pricing

Zenrows uses credit-based pricing. Every plan gives you a monthly credit allowance shared across Fetch, Extract, Batch, and Browser Sessions. The Free plan starts at 5,000 credits/month; the first paid tier, Build B1, is \$19/month for 45,000 credits, enough for 45,000 basic requests a month. Enterprise plans scale beyond 45M credits/month.

For complex or highly protected websites, enabling advanced features like JavaScript rendering (`js_render`) and Premium Proxies unlocks Zenrows' full potential, ensuring the best success rate possible.

The credit cost depends on the complexity of the request. You only spend the credits the scraping tech you need actually requires. With Adaptive Stealth Mode (`mode=auto`), the same credit weights apply; you're billed only for the configuration that succeeds. If multiple internal attempts are needed before success, only the successful request is charged.

* **Standard request:** 1 credit
* **JS rendering:** 5 credits
* **Premium proxies:** 10 credits
* **Both (JS & proxies, protected):** 25 credits

<Tip>For the full plan table, credit costs per tier, and top-up pricing, see our [pricing documentation page](/first-steps/pricing).</Tip>

### Concurrency

Concurrency determines how many requests can run simultaneously, and scales by plan tier:

| Tier | Concurrency Limit |
| - | - |
| Free | 5 |
| Build | 20 |
| Launch | 50 |
| Growth | 100 |
| Scale | 200 (S1 to S3), 300 (S4), 400 (S5) |
| Enterprise | Custom (500 to 1,000+) |

**Important notes about concurrency:**

* Canceling requests on the client side does NOT immediately free up concurrency slots
* The server continues processing canceled requests until completion
* If you exceed your concurrency limit, you'll receive a `429 Too Many Requests` error

**If response size is exceeded:**

* You'll receive a `413 Content Too Large` error
* No partial data will be returned when a size limit is hit

**Strategies for handling large pages:**

1. **Use CSS selectors**: Target only the specific data you need with `css_extractor` parameter
2. **Use `response_type`**: Convert to markdown or plaintext to reduce size
3. **Disable screenshots**: If using `screenshot` features, these can significantly increase response size
4. **Segment your scraping**: Break down large pages into smaller, more manageable sections

## Response Headers

Zenrows provides useful information through response headers:

| Header | Description | Example Value | Usage |
| - | - | - | - |
| **Concurrency-Limit** | Maximum concurrent requests allowed by your plan | `20` | Monitor your plan's capacity |
| **Concurrency-Remaining** | Available concurrent request slots | `17` | Adjust request rate dynamically |
| **X-Request-Cost** | Cost of this request | `0.001` | Track balance consumption |
| **X-Key-Credits-Remaining** | Lowest remaining credits across the API key's [credit caps](/first-steps/api-key-credit-caps). Only sent for keys with at least one cap | `12500` | Slow down before a key cap stops requests |
| **X-Request-Id** | Unique identifier for this request | `67fa4e35647515d8ad61bb3ee041e1bb` | Include when contacting support |
| **Zr-Final-Url** | The final URL after any redirects occurred during the request | `https://example.com/page?id=123` | Track redirects |

Why these headers matter:

* **Monitoring usage**: Track your concurrent usage and stay within limits
* **Support requests**: When reporting issues, always include the `X-Request-Id` for faster troubleshooting
* **Cost tracking**: The `X-Request-Cost` helps you monitor your usage per request
* **Redirection tracking**: `Zr-Final-Url` shows where you ended up after any redirects

## Additional Considerations

Beyond the core features and limits, these additional aspects are important to consider when using Fetch:

### Cancelled Request Behavior

When you cancel a request on the client side:

* The server **continues processing** the request until completion
* The concurrency slot remains occupied for up to 3 minutes
* This can result in unexpected `429 Too Many Requests` errors
* Implement request timeouts carefully to avoid depleting concurrency slots

### Security Best Practices

To keep your Zenrows integration secure:

* Store API keys as environment variables, never hardcode them
* Monitor usage patterns to detect unauthorized use
* Rotate API keys periodically for critical applications

### Regional Performance Optimization

To optimize performance based on target website location:

* Consider the geographical distance between your servers and the target website
* For global applications, distribute scraping across multiple regions - the system does it by default
* Monitor response times by region to identify optimization opportunities
* For region-specific content, use the appropriate `proxy_country` parameter

### Compression Support

Zenrows API supports response compression to optimize bandwidth usage and improve performance. Enabling compression offers several benefits for your scraping operations:

* **Reduced latency**: Smaller response sizes mean faster data transfer times
* **Lower bandwidth consumption**: Minimize data transfer costs and usage
* **Improved client performance**: Less data to process means reduced memory usage

Zenrows supports the following compression encodings: `gzip`, `deflate`, `br`.

To use compression, include the appropriate `Accept-Encoding` header in your requests. Most HTTP clients already compress the request automatically. But you can also provide simple options to enable it:

<CodeGroup>
  ```python Python (Requests) theme={"dark"}
  import requests

  response = requests.get(
      "https://api.zenrows.com/v1/",
      params={
          "apikey": "YOUR_API_KEY",
          "url": "https://example.com"
      },
      # Python Requests uses compression by default
      # headers={"Accept-Encoding": "gzip,  deflate"}
  )
  ```

  ```javascript Javascript (Axios) theme={"dark"}
  const axios = require('axios');

  const response = await axios.get('https://api.zenrows.com/v1/', {
    params: {
      apikey: 'YOUR_API_KEY',
      url: 'https://example.com'
    },
    headers: {
      'Accept-Encoding': 'gzip, deflate', // axios does not support br by default
    },
    decompress: true, // Enables automatic handling of compression
  });
  ```

  ```bash cURL theme={"dark"}
  curl --compressed "https://api.zenrows.com/v1/?apikey=YOUR_API_KEY&url=https://example.com"
  ```
</CodeGroup>

<Tip>Most modern HTTP clients automatically handle decompression, so you'll receive the uncompressed content in your response object without any additional configuration.</Tip>


# Common Use Cases & Recipes
Source: https://docs.zenrows.com/fetch/common-use-cases

Browse ready-to-use web scraping recipes for anti-bot bypass, JavaScript rendering, data extraction, and other common scraping scenarios.

This guide shows you the most common web scraping scenarios and the exact parameters you need for each one. Use these ready-to-implement recipes to quickly solve your specific scraping challenges with Fetch.

## Use Case Overview

| Use Case | Key Parameters | When You Need This |
| - | - | - |
| Load dynamic pages (SPAs, dashboards, infinite scroll) | `js_render`, `wait_for`, `block_resources`, `js_instructions` | When content loads via JavaScript or requires user interaction |
| Keep session/IP across requests | `session_id`, `custom_headers` | For multi-step workflows like account access → search → checkout |
| Submit forms or simulate user actions | `js_instructions`, `js_render`, `wait_for` | When you need to click buttons, fill forms, or navigate pages |
| Access geo-restricted content | `premium_proxy`, `proxy_country` | When content varies by location or is region-locked |
| Handle pagination and infinite scroll | `js_render`, `js_instructions`, `wait_for` | When content loads dynamically as you scroll or click "Load More" |
| Extract only specific data fields | `extract`, `css_extractor`, `outputs` | When you want structured data instead of full HTML |
| Capture API calls and background requests | `json_response`, `js_render`, `wait_for` | When the data you need comes from AJAX/XHR requests |
| Bypass anti-bot protection | `premium_proxy`, `js_render`, `custom_headers`, `session_id` | When sites use Cloudflare or other bot detection |
| Debug failed requests | `original_status`, `allowed_status_codes`, `screenshot` | When requests fail and you need to understand why |
| Optimize performance and costs | `block_resources`, `outputs`, `wait_for` | When you want faster responses and lower bandwidth usage |

## Code Recipes

### 1. Search for products using the search form

This example shows how to automatically fill and submit a search form, then wait for the search results to load.

```python Python theme={"dark"}
import requests

url = "https://api.zenrows.com/v1/"
params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "js_render": "true",
    "js_instructions": """[
        {"wait_for":"#wp-block-search__input-1"},
        {"fill":["#wp-block-search__input-1","hoodie"]},
        {"wait":500},
        {"click":"#block-2 > form > div > button"},
        {"wait_for":"#main"}
    ]""",
    "premium_proxy": "true",
}
response = requests.get(url, params=params)
print(response.text)
```

The `js_instructions` parameter fills the search field with "hoodie", clicks the search button, and waits for the main content area to load with search results before returning the content.

<Tip>
  Head to the [JS Instructions Documentation](/fetch/features/js-instructions) for more details.
</Tip>

### 2. Maintain a Session Across Requests

Use the same `session_id` value to maintain cookies and session state across multiple requests.

```python Python theme={"dark"}
import requests

# Define session ID and API endpoint
session_id = "1234"
url = "https://api.zenrows.com/v1/"

# First request establishes a session
params1 = {
  "apikey": "<YOUR_ZENROWS_API_KEY>",
  "url": "https://www.scrapingcourse.com/",
  "session_id": session_id
}
response1 = requests.get(url, params=params1)

# Second request reuses the same session
params2 = {
  "apikey": "<YOUR_ZENROWS_API_KEY>",
  "url": "https://www.scrapingcourse.com/pagination",
  "session_id": session_id
}
response2 = requests.get(url, params=params2)
```

This approach is essential for workflows where you need to maintain the same IP across requests.

<Tip>
  Learn more about the `session_id` parameter [here](/fetch/features/other#session-id).
</Tip>

### 3. Extract Specific Fields Only

Instead of getting the full HTML, extract only the data you need using CSS selectors.

```python Python theme={"dark"}
params = {
  "apikey": "<YOUR_ZENROWS_API_KEY>",
  "url": "https://www.scrapingcourse.com/ecommerce/",
  "css_extractor": """{"product_name":".product-name","price":".price"}""",
}
```

Returns clean, structured data:

```json JSON expandable theme={"dark"}
{
  "price": [
    "$69.00",
    "$57.00",
    "$48.00",
    "$24.00",
    "$74.00",
    "$7.00",
    "$45.00",
    "$69.00",
    "$40.00",
    "$42.00",
    "$34.00",
    "$32.50",
    "$20.00",
    "$22.00",
    "$39.00",
    "$45.00"
  ],
  "product_name": [
    "Abominable Hoodie",
    "Adrienne Trek Jacket",
    "Aeon Capri",
    "Aero Daily Fitness Tee",
    "Aether Gym Pant",
    "Affirm Water Bottle",
    "Aim Analog Watch",
    "Ajax Full-Zip Sweatshirt",
    "Ana Running Short",
    "Angel Light Running Short",
    "Antonia Racer Tank",
    "Apollo Running Short",
    "Arcadio Gym Short",
    "Argus All-Weather Tank",
    "Ariel Roll Sleeve Sweatshirt",
    "Artemis Running Short"
  ]
}
```

<Tip>
  Check the [CSS Extractor Documentation](/fetch/features/css-extractor) for more details.
</Tip>

### 4. Scrape Localized Content

Access geo-restricted content or get localized pricing by routing your request through specific countries.

```python Python theme={"dark"}
params = {
  "apikey": "<YOUR_ZENROWS_API_KEY>",
  "url": "https://www.scrapingcourse.com/ecommerce/",
  "premium_proxy": "true",
  "proxy_country": "de"
}
```

This example routes your request through Germany, allowing you to see content as it appears to German users.

<Tip>
  Find more details about geolocation on the [Proxy Country Documentation](/fetch/features/proxy-country).
</Tip>

### 5. Debugging a Blocked Request

When requests fail, use these parameters to understand what's happening without losing the response data.

#### Check the actual HTTP status

Use the `original_status` parameter to see the real HTTP status code the target page returns.

```python Python theme={"dark"}
params = {
  "apikey": "<YOUR_ZENROWS_API_KEY>",
  "url": "https://example.com/blocked",
  "original_status": "true",
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response)
```

<Tip>
  Check more about the `original_status` parameter [here](/fetch/features/other#original-http-code).
</Tip>

#### Allow specific error status codes

Use the `allowed_status_codes` parameter to receive the actual HTML content even when the page returns error status codes.

```python Python theme={"dark"}
params = {
  "apikey": "<YOUR_ZENROWS_API_KEY>",
  "url": "https://example.com/blocked",
  "allowed_status_codes": "403,404,500",
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
```

<Tip>
  Find out more on the [Allowed Status Codes Documentation](/fetch/features/other#return-content-on-error).
</Tip>

#### Add custom headers

Use custom headers to simulate browser behavior for more reliable access.

```python Python theme={"dark"}
params = {
  "apikey": "<YOUR_ZENROWS_API_KEY>",
  "url": "https://example.com/blocked",
}
headers = {
    'cookie': 'your-cookie',
    'referer': 'https://www.google.com',
}
response = requests.get('https://api.zenrows.com/v1/', params=params, headers=headers)
print(response.text)
```

<Tip>
  Zenrows handles all browser-based headers. To find out more about it, check our [Headers Documentation](/fetch/features/headers).
</Tip>

## What's Next

Ready to implement these recipes? Here are your next steps:

* Explore the [Parameter Reference](/fetch/api-reference#parameter-overview)

* Follow the [First Request Guide](/fetch/first-request)

* Jump back to [Welcome](/first-steps/welcome)


# Fetch FAQ: Cookies, Proxies, Output
Source: https://docs.zenrows.com/fetch/faq

Find answers to frequently asked questions about Zenrows API features, including cookies, session management, proxies, output formats and other common questions.

<Accordion title="Can I Get Cookies from the Responses?">
  Headers, including cookies, returned by the target website are prefixed with Zr- and included in all our responses.

  Suppose you are scraping a website that requires session cookies for authentication. By capturing the Zr-Cookies header from the initial response, you can include these cookies in your subsequent requests to maintain the session and access authenticated content.

  ```plaintext theme={"dark"}
  Zr-Content-Encoding: gzip
  Zr-Content-Type: text/html
  Zr-Cookies: _pxhd=Bq7P4CRaW1B...
  Zr-Final-Url: https://www.example.com/
  ```

  You could send those cookies in a subsequent request as [Custom Headers](/fetch/features/headers) and also use [session\_id](https://docs.zenrows.com/fetch/features/other#session-id) to keep the same IP for up to 10 minutes.

  By following this process, you can handle sessions and access restricted areas of the website seamlessly.
</Accordion>

<Accordion title="Can I Log In/Register and Access Content Behind a Login?">
  If you need to scrape data from a website that requires login authentication, you can log in or register and access content behind a login. However, due to privacy and legal reasons, we offer limited support for these cases.

  Login and registration work like regular forms and can be treated as such. There are two main methods to send forms:

  1. Send [POST requests](/fetch/faq#how-do-i-send-post-requests).
  2. Fill in and submit a form using [JavaScript Instructions](/fetch/features/js-instructions).

  ```bash theme={"dark"}
  {"fill": [".input-selector", "website_username"]} // Fill the username input
  {"fill": [".input-selector", "website_password"]} // Fill the password input
  ```

  All requests will return headers, including the [session cookies](/fetch/faq#can-i-get-cookies-from-the-responses). By using these cookies in subsequent requests, you can operate as a logged-in user. Additionally, you can include a [Session ID](/fetch/features/other#session-id) to maintain the same IP address for up to 10 minutes.

  <Warning>Zenrows is a scraping tool, not a VPN. If your goal is to log in once and browse the internet with the same IP, you may need a different service.</Warning>
</Accordion>

<Accordion title="Can I Maintain Sessions/IPs Between Requests">
  Suppose you need to perform multiple actions on a website that requires maintaining the same session/IP. You can use the [Session ID](/fetch/features/other#session-id) parameter to maintain the same IP between requests. Zenrows will store the IP for 10 minutes from the first request with that ID. All subsequent requests with that ID will use the same IP.

  However, session\_id will not store any other request data, such as session cookies. You will receive those cookies as usual and can decide which ones to send on the next request.

  Multiple Session IDs can run concurrently, with no limit to the number of sessions.
</Accordion>

<Accordion title="Can I Run the API/Proxy in Multiple Threads to Improve Speed?">
  Each plan comes with a concurrency limit that scales with your plan tier. See the [Concurrency, Support, and Rollover table](/first-steps/pricing#concurrency-support-and-rollover) for exact numbers. Requests up to that limit can run simultaneously, significantly improving speed.

  Sending requests above that limit will result in a 429 Too Many Requests error.

  We wrote a [guide on using concurrency](/fetch/features/concurrency#using-concurrency) that provides more details, including examples in Python and JavaScript. The same principles apply to other languages and libraries.
</Accordion>

<Accordion title="Can I Send/Submit Forms?">
  There are different ways to approach submitting forms on a website when you need to retrieve data.

  POST Requests:
  The most straightforward way for non-secured endpoints is to send a [POST request](/fetch/faq#how-do-i-send-post-requests) as the page does. You can examine and replicate the requests in the browser DevTools.

  Imitate User Behavior Using JavaScript Instructions:
  Use [JavaScript Instructions](/fetch/features/js-instructions) to visit pages protected by anti-bot solutions and interact with them. This includes filling in inputs, clicking buttons, and performing other actions.
</Accordion>

<Accordion title="CSS Selectors Do Not Work or 'Parser is Not Valid'">
  ## Common Issues with CSS Selectors

  One of the most common issues users encounter when working with CSS Selectors in web scraping is improper encoding. CSS Selectors need to be correctly encoded to be recognized and processed by the API.

  You can use Zenrows' [Playground](https://app.zenrows.com/fetch) or an <a href="https://www.url-encode-decode.com/">online tool</a> to properly encode your CSS Selectors before sending them in a request.

  ## Example of Using a CSS Selector

  Let's say you want to extract content from the `.my-class` CSS selector and store it in a property named `test`. You would encode the selector and include it in your request like this:

  ```bash theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=YOUR_URL&css_extractor=%257B%2522test%2522%253A%2520%2522.my-class%2522%257D"
  ```

  ## Troubleshooting CSS Selector Issues

  If you're still getting empty responses or the parser reports an error:

  1. **Check the Raw HTML:** Request the plain HTML to see if the content served by the website differs from what you see in your browser. Some websites serve different content based on the user's location, device, or other factors.

  2. **Verify the Selector:** Ensure the selector you're using is correct by testing it in your browser's Developer Tools (e.g., using Chrome's Console with `document.querySelectorAll(".my-class")`).

  3. **Review the Documentation:** Refer to the Zenrows [documentation](/fetch/features/css-extractor) for detailed information on using CSS Selectors with the API.

  If the HTML looks correct, the selector works in the browser, but the parser still fails, <a href="mailto:success@zenrows.com">contact us</a>, and we'll help you troubleshoot the issue.

  ## See Also

  For comprehensive examples of working with complex layouts and advanced selector techniques, check out our [Advanced CSS Selector Examples](/fetch/troubleshooting/advanced-css-selector-examples) guide.
</Accordion>

<Accordion title="Does session_id Remember Session Data?">
  `session_id` won't store any request data, such as session cookies. You will get those back as usual and decide which ones to send on the next request.
</Accordion>

<Accordion title="How do I Export Data to CSV using Fetch?">
  Once you've extracted data using Zenrows, you might want to store it in CSV format. For simplicity, we'll focus on a single URL and save the data to one file. In real-world scenarios, you might need to handle multiple URLs and aggregate the results.

  To start, we'll explore how to export data to CSV using both Python and JavaScript.

  ## From JSON using Python

  If you've obtained JSON output from Zenrows, with [Extract](/extract/setup) (`extract=auto`) or with the deprecated `autoparse` parameter, you can use Python to convert this data into a CSV file.

  <Note>The example below uses `autoparse`, which is deprecated. For new work use `extract=auto`; the flattening code is the same, only the JSON shape changes. See [Migrate to Extract](/fetch/features/autoparse#migrate-to-extract).</Note>

  The Pandas library will help us flatten nested JSON attributes and save the data as a CSV file.

  Here's a sample Python script:

  ```python scraper.py theme={"dark"}
  # pip install requests pandas
  import requests
  import json
  import pandas as pd

  url = "https://www.zillow.com/san-francisco-ca/"
  apikey = "YOUR_ZENROWS_API_KEY"
  params = {"autoparse": True, "url": url, "apikey": apikey}
  response = requests.get("https://api.zenrows.com/v1/", params=params)

  content = json.loads(response.text)

  data = pd.json_normalize(content)
  data.to_csv("result.csv", index=False)
  ```

  You can also adjust the `json_normalize` function to control how many nested levels to flatten and rename fields. For instance, to flatten only one inner level and remove `latLong` from latitude and longitude fields:

  ```python theme={"dark"}
  data = pd.json_normalize(content, max_level=1).rename(
  	columns=lambda x: x.replace("latLong.", ""))
  ```

  ## From HTML using Python

  When dealing with HTML output without the `autoparse` feature, you can use BeautifulSoup to parse the HTML and extract data. We'll use the example of an eCommerce site from [Scraping Course](https://www.scrapingcourse.com/ecommerce/). Create a dictionary for each product with essential details, then use Pandas to convert this list of dictionaries into a DataFrame and save it as a CSV file.

  Here's how to do it:

  ```python scraper.py theme={"dark"}
  # pip install requests beautifulsoup4 pandas
  import requests
  from bs4 import BeautifulSoup
  import pandas as pd

  url = "https://www.scrapingcourse.com/ecommerce/"
  apikey = "YOUR_ZENROWS_API_KEY"
  params = {"url": url, "apikey": apikey}
  response = requests.get("https://api.zenrows.com/v1/", params=params)
  soup = BeautifulSoup(response.content, "html.parser")

  content = [{
  	"product_name": product.select_one(".product-name").text.strip(),
  	"price": product.select_one(".price").text.strip(),
  	"rating": product.select_one(".rating").text.strip() if product.select_one(".rating") else "N/A",
  	"link": product.select_one(".product-link").get("href"),
  } for product in soup.select(".product")]

  data = pd.DataFrame(content)
  data.to_csv("result.csv", index=False)
  ```

  ## From JSON using JavaScript

  For JavaScript and Node.js, you can use the `json2csv` library to handle the JSON to CSV conversion.

  After getting the data, we will parse it with a `flatten` transformer. As the name implies, it will flatten the nested structures inside the JSON. Then, save the file using `writeFileSync`.

  Here's an example using the Zenrows Fetch with Node.js:

  ```javascript scraper.js theme={"dark"}
  // npm install zenrows json2csv
  const fs = require("fs");
  const {
  	Parser,
  	transforms: { flatten },
  } = require("json2csv");
  const { ZenRows } = require("zenrows");

  (async () => {
  	const client = new ZenRows("YOUR_ZENROWS_API_KEY");
  	const url = "https://www.scrapingcourse.com/ecommerce/";

  	const { data } = await client.get(url, { autoparse: "true" });

  	const parser = new Parser({ transforms: [flatten()] });
  	const csv = parser.parse(data);

  	fs.writeFileSync("result.csv", csv);
  })();
  ```

  ## From HTML using JavaScript

  For extracting data from HTML without `autoparse` you can use the cheerio library to parse the HTML and extract relevant information. We'll use the [Scraping Course eCommerce](https://www.scrapingcourse.com/ecommerce/) example for this task:

  As with the Python example, we will use AutoScout24 to extract data from HTML without the autoparse feature. For that, we will get the plain result and load it into cheerio. It will allow us to query elements as we would in the browser or with jQuery. We will return an object with essential data for each car entry in the list. Parse that list into CSV using `json2csv`, and no flatten is needed this time. And lastly, store the result. These last two steps are similar to the autoparse case.

  ```javascript scraper.js theme={"dark"}
  // npm install zenrows json2csv cheerio
  const fs = require("fs");
  const cheerio = require("cheerio");
  const { Parser } = require("json2csv");
  const { ZenRows } = require("zenrows");

  (async () => {
  	const client = new ZenRows("YOUR_ZENROWS_API_KEY");
  	const url = "https://www.scrapingcourse.com/ecommerce/";

  	const { data } = await client.get(url);
  	const $ = cheerio.load(data);

  	const content = $(".product").map((_, product) => ({
  		product_name: $(product).find(".product-name").text().trim(),
  		price: $(product).find(".price").text().trim(),
  		rating: $(product).find(".rating").text().trim() || "N/A",
  		link: $(product).find(".product-link").attr("href"),
  	}))
  	.toArray();

  	const parser = new Parser();
  	const csv = parser.parse(content);

  	fs.writeFileSync("result.csv", csv);
  })();
  ```

  If you encounter any issues or need further assistance with your scraper setup, please <a href="mailto:success@zenrows.com">contact us</a>, and we'll be happy to help!
</Accordion>

<Accordion title="Extract Data from Lists, Tables, and Grids">
  We'll explore popular use cases for scraping, such as lists, tables, and product grids. Use these as inspiration and a guide for your scrapers.

  ## Scraping from Lists

  We will use the <a href="https://en.wikipedia.org/wiki/Web_scraping">Wikipedia page on Web scraping</a> for testing. A section at the bottom, "See also", contains links in a list. We can get the content by using the [CSS selector](/fetch/features/css-extractor) for the list items: `{"items": ".div-col > ul li"}`.

  That will get the text, but what of the links? To access attributes, we need a non-standard syntax for the selector: `@href`. It won't work with the previous selector since the last item is the `li` element, which does not have an `href` attribute. So we must change it for the link element: `{"links": ".div-col > ul a @href"}`.

  CSS selectors, in some languages, must be [encoded](/fetch/faq#how-to-encode-urls) to avoid problems with URLs.

  ```bash theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FWeb_scraping&css_extractor=%257B%2522items%2522%253A%2520%2522.div-col%2520%253E%2520ul%2520li%2522%252C%2520%2522links%2522%253A%2520%2522.div-col%2520%253E%2520ul%2520a%2520%2540href%2522%257D"
  ```

  Our [Playground](https://app.zenrows.com/fetch) can help you write and test the selectors and output code in several languages.

  <Frame>
    <img alt="Extract Data from Wikipedia Lists" />
  </Frame>

  ## Scraping from Tables

  Assuming regular tables (no empty cells, rows with fewer items, and others), we can extract table data with CSS selectors. We'll use <a href="https://en.m.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population">a list of countries</a>, the first table on the page, the one with the class `wikitable`.

  To extract the rank, which is the first column, we can use `"table.wikitable tr > :first-child"`. It will return an array with 243 items, 2 header lines, and 241 ranks. For the country name, second column, something similar but adding an `a` to avoid capturing the flags: `"table.wikitable tr > :nth-child(2) a"`. In this case, the array will have one less item since the second heading has no link. That might be a problem if we want to match items by array index.

  ```bash theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fen.m.wikipedia.org%2Fwiki%2FList_of_countries_and_dependencies_by_population&css_extractor=%257B%2522rank%2522%253A%2520%2522table.wikitable%2520tr%2520%253E%2520%253Afirst-child%2522%252C%2520%2522countries%2522%253A%2520%2522table.wikitable%2520tr%2520%253E%2520%253Anth-child%282%29%2520a%2522%257D"
  ```

  Outputs:

  ```json theme={"dark"}
  {
  	"countries": ["Country or dependent territory", "China", "India", ...],
  	"rank": ["Rank", "-", "1", "2", ...]
  }
  ```

  As stated above, this might prove difficult for non-regular tables. For those, we might prefer to get the Plain HTML and [scrape the content with a tool or library](/zenrows-academy/how-to-extract-data#using-external-libraries) so we can add conditionals and logic.

  This example lists items by column, not row, which might prove helpful in various cases. However, there are no easy ways to extract structured data from tables using CSS Selectors and group it by row.

  ## Scraping from Product Grids

  As with the tables, non-regular grids might cause problems. We'll scrape the price, product name, and link from an <a href="https://www.scrapingcourse.com/ecommerce/">online store</a>. By manually searching the page's content, we arrive at cards with the class `.product`. Those contain all the data we want.

  It is essential to avoid duplicates, so we have to use some precise selectors. For example, `".product-item .product-link @href"` for the links. We added the `.product-link` class because it is unique to the product cards. The same goes for name and price, which also have unique classes.
  All in all, the final selector would be:

  ```json theme={"dark"}
  {
  	"links": ".product-item .product-link @href",
  	"names": ".product-item .product-name",
  	"prices": ".product-item .product-price"
  }
  ```

  Several items are on the page at the time of this writing. And each array has the same number of elements, so everything looks fine. If we were to group them, we could zip the arrays.

  For example, in python, taking advantage of the auto-encoding that `requests.get` does to parameters. Remember to [encode](/fetch/faq#how-to-encode-urls) the URL and CSS extractor for different scenarios when that is not available.

  ```python scraper.py theme={"dark"}
  # pip install requests
  import requests
  import json

  zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY"
  url = "https://www.scrapingcourse.com/ecommerce/"

  css_extractor = """{
  	"links": ".product .product-link @href",
  	"names": ".product .product-name",
  	"prices": ".product .product-price"
  }"""

  response = requests.get(zenrows_api_base, params={
  						"url": url, "css_extractor": css_extractor})
  parsed_json = json.loads(response.text)
  result = zip(parsed_json["links"], parsed_json["names"], parsed_json["prices"])
  print(list(result))

  # [('/products/product1', 'Product 1', '$10.00'), ... ]
  ```

  Remember that this approach won't work properly if, for example, some products have no price. Not all the arrays would have the same length, and the zipping would misassign data. Getting the Plain HTML and parsing the content with a library and custom logic is a better solution for those cases.

  If you encounter any problems or cannot correctly set up your scraper, <a href="mailto:success@zenrows.com">contact us</a>, and we'll help you.
</Accordion>

<Accordion title="How Can I Set Specific Headers?">
  Zenrows allows you to send [Custom Headers](/fetch/features/headers) on requests in case you need to scrape a website that requires specific headers.

  However, it's important to test the success rate when changing them. Zenrows automatically manages certain headers, especially those related to the browser environment, such as User-Agent.

  Defensive systems inspect headers as a whole, and not all browsers use the same ones. If you choose to send custom headers, ensure the rest of the headers match accordingly.
</Accordion>

<Accordion title="How Do I Send POST Requests with JSON Data?">
  By default, POST requests use `application/x-www-form-urlencoded`. To send JSON data, you need to add the `Content-Type: application/json` header manually, though some software/tools may do this automatically.

  Before trying on your target site, we recommend using a testing site like <a href="https://httpbin.io/anything">httpbin.io</a> to verify that the parameters are sent correctly.

  Ensure that the parameters are sent and the format is correct. If in doubt, switch between both modes to confirm that the changes are applied correctly.

  For more info on POST requests, see [How do I send POST requests?](/fetch/faq#how-do-i-send-post-requests).
</Accordion>

<Accordion title="How do I Send POST Requests?">
  Send [POST requests](https://docs.zenrows.com/fetch/features/other#post-/-put-requests) using your chosen programming language. Zenrows will transparently forward the data to the target site.

  Before trying on your target site, we recommend using a testing site like <a href="https://httpbin.io/anything">httpbin.io</a> to verify that the parameters are sent correctly.

  Testing is important because not all languages and tools handle POST requests the same way. Ensure that the parameters and format are correct. By default, browsers send content as `application/x-www-form-urlencoded`, but many sites expect JSON content, requiring the `Content-Type: application/json header`.
</Accordion>

<Accordion title="How to encode URLs?">
  When working with the Zenrows Fetch, it's crucial to encode your target URLs, especially if they contain query parameters. Encoding ensures that your URLs are correctly interpreted by the API, avoiding potential conflicts between the target URL's parameters and those used in the API request.

  Consider the following URL example:

  `https://www.scrapingcourse.com/ecommerce/?course=web-scraping&section=advanced`

  If you were to send this URL directly as part of your API request without encoding, and you also include the `premium_proxy` parameter, the request might look something like this:

  ```bash theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https://www.scrapingcourse.com/ecommerce/?course=web-scraping&section=advanced&premium_proxy=true"
  ```

  In this scenario, the API might incorrectly interpret the `course` and `section` parameters as part of the API's query string rather than the target URL. This could lead to errors or unintended behavior.

  To avoid such issues, you should encode your target URL before including it in the API request. URL encoding replaces special characters (like `&`, `?`, and `=`) with a format that can be safely transmitted over the internet.

  Here's how you can encode the URL in Python:

  ```python encoder.py theme={"dark"}
  import urllib.parse
  encoded_url = urllib.parse.quote("https://www.scrapingcourse.com/ecommerce/?course=web-scraping&section=advanced")
  ```

  After encoding, your Fetch request would look like this:

  ```bash theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F%3Fcourse%3Dweb-scraping%26section%3Dadvanced&premium_proxy=true"
  ```

  Many HTTP clients, such as `axios` (JavaScript) and `requests` (Python), automatically encode URLs for you. However, if you are manually constructing requests or using a client that doesn't handle encoding, you can use programming language functions or online tools to encode your URLs.

  For quick manual encoding, you can use <a href="https://www.url-encode-decode.com/">an online tool</a>, but remember that this method is not scalable for automated processes.
</Accordion>

<Accordion title="Using Premium Proxies + JS Render and still blocked">
  If you are scraping a site with high-security measures and encounter blocks even with Premium Proxies and JS Render enabled, try these steps to unblock your requests:

  <Steps>
    <Step title="Add Geotargeting">
      Use [geotargeting](/fetch/features/proxy-country) by selecting a country for the proxy, e.g., `proxy_country=us`. Many sites respond better to proxies close to their operation centers.
    </Step>

    <Step title="Use Wait For Selector">
      Implement [Wait For Selector](/fetch/features/wait-for) to have the scraper look for specific content before returning. This feature can change how the system interacts with the site and might help unblock the request.
    </Step>

    <Step title="Change Default Block Resources">
      Adjust the [Block Resources](/fetch/features/block-resources) settings. Zenrows blocks certain resources by default, such as CSS or images, to speed up scraping. Use your browser's DevTools to identify other resources to block, such as media or xhr (`block_resources=stylesheet,image,media,xhr`). Alternatively, disable blocking by setting it to false (`block_resources=none`).
    </Step>

    <Step title="Add Custom Headers">
      Many websites inspect the headers of a request to determine if it is coming from a legitimate browser. Adding custom headers to mimic normal browser behavior can help bypass these checks.

      Refer to our documentation on [Custom Headers](/fetch/features/headers) for more details.
    </Step>
  </Steps>

  Combining these methods may yield the expected results.

  For high-security endpoints or inner pages, you may need to simulate a typical user session to avoid detection. First, obtain session cookies from a less protected page on the same site. This step mimics the initial user interaction with the site. Then, use these session cookies to access the more secure target page.

  Additionally, you can use the [Session ID](/fetch/features/other#session-id) feature to maintain the same IP address for up to 10 minutes, ensuring consistency in your requests and reducing the likelihood of being blocked.
</Accordion>

<Accordion title="What are Residential IPs?">
  ## Understanding Proxy Types: Data Center vs. Residential IPs

  When it comes to web scraping proxies, there are two main types of IPs you can use: data center and residential.

  1. **Data Center IPs:** These are IP addresses provided by cloud service providers or hosting companies. They are typically fast and reliable, but because they are easily recognizable as belonging to data centers, they are more likely to be blocked by websites that have anti-scraping measures in place.
  2. **Residential IPs:** These IP addresses are assigned by Internet Service Providers (ISPs) to real residential users. Since they appear as regular users browsing the web, they are much harder to detect and block. This makes residential IPs particularly valuable when scraping sites with strong anti-bot protections, like Google or other heavily guarded domains.

  ## How Zenrows Uses Residential IPs

  By default, Zenrows uses data center connections for your requests. However, if you're facing blocks or need to scrape highly protected websites, you can opt for residential IPs by setting the `premium_proxy` parameter to `true`. This will route your request through a residential IP, significantly increasing your chances of success.

  It's important to note that using residential IPs comes with an additional cost due to the higher value and lower detection rate of these proxies.

  <Tip>You can check out more about Premium Proxies [here!](/fetch/features/premium-proxy)</Tip>

  ## Example of a Request with Residential IPs

  Here's how you can make a request using a residential IP:

  ```bash theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=YOUR_URL&premium_proxy=true"
  ```

  In cases where you're also targeting content localized to specific regions, Zenrows supports [geotargeting](/first-steps/faq#what-is-geolocation-and-what-are-all-the-premium-proxy-countries) with residential IPs, allowing you to specify the country of the IP.

  ## Troubleshooting Blocks

  If you continue to experience blocks even with residential IPs, feel free to <a href="mailto:success@zenrows.com">contact us</a>, and we'll work with you to find a solution.
</Accordion>

<Accordion title="What is Autoparse?">
  <Warning>Autoparse is deprecated. Use [Extract](/extract/setup) (`extract=auto`) instead, which builds a tailored extraction per website. The parameter still works for existing integrations, see [Migrate to Extract](/fetch/features/autoparse#migrate-to-extract).</Warning>

  ## Simplifying Data Extraction with Autoparse

  Zenrows offers a powerful feature called Autoparse, designed to simplify the process of extracting structured data from websites. This feature leverages custom parsers allowing you to easily retrieve data in a structured JSON format rather than raw HTML.

  ## How It Works

  By default, when you call the Zenrows API, the response will be in Plain HTML. However, when you activate the `autoparse` parameter, the API will automatically parse the content of supported websites and return the data as a JSON object. This makes it much easier to work with the data, especially when dealing with complex websites that require extensive parsing logic.

  ## Example of a Request with Autoparse

  Here's how you can make an API call with the Autoparse feature enabled:

  ```bash theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=YOUR_URL&autoparse=true"
  ```

  ## Limitations and Troubleshooting

  1. **Supported Domains:** Only a limited set of sites has a dedicated parser. On every other site, Autoparse returns the JSON objects embedded in the page, which may be empty, incomplete, or not the data you asked for. [Extract](/extract/setup) covers any prepared domain and returns named fields instead.

  2. **Fallback to HTML:** If you find that Autoparse doesn't return the desired results, you can simply remove the `autoparse` parameter and try the request again. This will return the plain HTML response, allowing you to manually parse the data as needed.
</Accordion>

<Accordion title="What Are the Benefits of JavaScript Rendering?">
  Enabling JavaScript Rendering not only allows you to scrape content that would otherwise be inaccessible, but it also unlocks advanced scraping features. For example, with JavaScript Rendering, you can use the `wait_for` parameter to delay scraping until a specific element is present on the page, ensuring you capture the content you need.

  <Tip>Check out more about JavaScript Rendering [here!](/fetch/features/js-rendering)</Tip>
</Accordion>

<Accordion title="Why Some Headers are Managed by Zenrows?">
  Browser-based headers are crucial for ensuring that requests appear legitimate to target websites. Zenrows manages these headers to mimic real user behavior, which significantly reduces the risk of being blocked. By preventing customers from manually setting these headers, Zenrows can optimize the success rate and avoid common pitfalls associated with improper header configurations.

  ## Example of Sending Custom Headers

  Here's an example using `cURL` to send custom headers that are permitted along with your Zenrows request:

  ```bash bash theme={"dark"}
  curl \
  -H "Accept: application/json" \
  -H "Referer: https://www.google.com" \
  "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=YOUR_URL&custom_headers=true"
  ```
</Accordion>


# Adaptive Stealth Mode
Source: https://docs.zenrows.com/fetch/features/adaptive-stealth-mode

Zenrows intelligent scraping mode that automatically selects optimal configurations while minimizing costs and maintenance overhead.

Adaptive Stealth Mode is Zenrows' automated execution mode that analyzes each request and selects the most cost-effective configuration needed for successful data extraction. Instead of manually configuring parameters, Adaptive Stealth Mode starts with basic settings and escalates only when necessary, eliminating ongoing maintenance as websites change their protection mechanisms.

**When you enable Adaptive Stealth Mode, Zenrows:**

* **Prioritizes successful extraction** above all else
* **Starts with the cheapest viable configuration** for each request
* **Automatically escalates** to more powerful features only when necessary
* **Bills only for the configuration that succeeds** - failed attempts incur no charges
* **Adapts automatically** as website behavior changes over time
* **Reduces maintenance overhead** by eliminating the need to monitor and update scraping configurations

Adaptive Stealth Mode eliminates the need for manual parameter tuning in production environments.

## How Adaptive Stealth Mode Works

Adaptive Stealth Mode executes each request using a progressive strategy:

1. **Target Analysis:** Evaluates the target URL using known patterns and runtime signals to predict the best starting configuration.

2. **Lowest-Cost Attempt:** Executes the request with the least expensive configuration likely to succeed based on the analysis.

3. **Automatic Escalation:** Retries with stronger features if the initial attempt fails, progressively adding capabilities as needed.

4. **Success-Only Billing:** You are billed only for the configuration that completes successfully. Failed attempts incur no charges.

This approach ensures reliable data extraction without overpaying for unnecessary features.

## Basic Usage

### API Usage

Add `mode=auto` to your request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'mode': 'auto',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'mode': 'auto',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&mode=auto";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&mode=auto');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&mode=auto", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&mode=auto')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?url=https://www.scrapingcourse.com/ecommerce/&apikey=YOUR_ZENROWS_API_KEY&mode=auto"
  ```
</CodeGroup>

## When to Use Adaptive Stealth Mode

**Recommended when:**

* Scraping multiple websites with different behaviors
* Website requirements are unknown or change frequently
* Reliability matters more than manual optimization
* You're running production workloads at scale
* You want to minimize maintenance overhead from website changes

**Consider manual configuration for:**

* Single, stable domains where you know the optimal setup
* Scenarios requiring predictable, consistent feature usage
* Cost-sensitive applications with well-understood requirements

## Managed Parameters

Adaptive Stealth Mode automatically manages these API parameters:

| Parameter | Purpose | When Applied |
| - | - | - |
| `js_render` | JavaScript rendering for dynamic content | Sites using React, Vue, Angular, or AJAX loading |
| `premium_proxy` | Residential IP addresses | Sites blocking datacenter IPs or requiring geo-location |

<Warning>
  The managed parameters above are disabled when using Adaptive Stealth Mode to prevent conflicting configurations. Other parameters remain available for use.

  If you set the parameters manually, Adaptive Stealth Mode will return a 400 error:

  ```json theme={"dark"}
  {
    "code": "REQ_INVALID_PARAMS",
    "message": "js_render and premium_proxy parameters cannot be used with automatic mode"
  }
  ```
</Warning>

## Adaptive Stealth Mode vs Manual Configuration

| Aspect | Adaptive Stealth Mode (`mode=auto`) | Manual Parameters |
| - | - | - |
| Setup time | Single parameter | Requires tuning |
| Success optimization | Automatic | User-managed |
| Cost optimization | Automatic, per request | Fixed by chosen config |
| Adaptation to site changes | Automatic | Requires updates |
| Failed attempts billed | No | No |
| Best for | Production, multi-domain workloads | Stable, well-known targets |

## Troubleshooting

### Common Issues

**Adaptive Stealth Mode not working as expected**

* Verify the `mode=auto` parameter is correctly set
* Check that other conflicting parameters aren't overriding Adaptive Stealth Mode
* Review the response headers for cost and feature information

**Higher costs than expected**

* Monitor the `X-Request-Cost` header to understand feature usage
* Consider manual configuration if cost patterns are predictable
* Use the Playground UI to preview costs before running requests

**Inconsistent results**

* Adaptive Stealth Mode adapts to website changes, which may affect feature selection
* Use `session_id` to maintain consistency across related requests
* Switch to manual configuration for completely predictable behavior

## Pricing

Adaptive Stealth Mode uses transparent, credit-based pricing:

| Configuration | Credits per request | When Used |
| - | - | - |
| Basic request | 1 | Simple static websites |
| JavaScript rendering | 5 | Dynamic content requiring browser processing |
| Premium Proxies | 10 | Sites blocking datacenter IPs |
| Combined features | 25 | Highly protected sites requiring both JS and proxies |

<Info>You are billed only once, for the configuration that succeeds. If Adaptive Stealth Mode requires multiple internal attempts before success, Zenrows bills only for the successful configuration.</Info>

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="Will Adaptive Stealth Mode increase my costs?">
  No. Adaptive Stealth Mode minimizes costs by avoiding unnecessary features and charging only for successful requests. It often reduces costs compared to manually over-configuring parameters.
</Accordion>

<Accordion title="How much visibility do I have into Adaptive Stealth Mode decisions?">
  You retain full visibility through response headers (including request cost) and the <a href="https://app.zenrows.com/overview#performance">Analytics</a> dashboard, without needing to manage configuration decisions manually.
</Accordion>

<Accordion title="What happens if a website changes its anti-bot measures?">
  Adaptive Stealth Mode adapts automatically without requiring parameter updates on your end. The system learns from new blocking patterns and adjusts accordingly.
</Accordion>

<Accordion title="Can I use Adaptive Stealth Mode with other parameters?">
  Yes, but with limitations. When you use `mode=auto`, only the managed parameters (`js_render`, and `premium_proxy`) are disabled to prevent conflicts. You can still use other API parameters like `css_extractor`, `session_id`, or custom request settings alongside Adaptive Stealth Mode.
</Accordion>


# Autoparse
Source: https://docs.zenrows.com/fetch/features/autoparse

Autoparse returns general-purpose JSON from a page. It is deprecated in favor of Zenrows Extract (extract=auto), which builds a tailored extraction for each website.

<Warning>
  **Autoparse is deprecated. Use [Extract](/extract/setup) (`extract=auto`) instead.**

  The `autoparse` parameter still works and existing integrations keep running, so nothing breaks today. It is no longer the recommended way to get JSON from a page, and new integrations should start with Extract. Any sunset will be announced separately, with notice.
</Warning>

Autoparse returns structured JSON instead of raw HTML. A small set of sites has a dedicated parser behind the parameter; on every other site, Autoparse returns the JSON objects it finds embedded in the page's `<script>` tags, which may or may not hold the data you are after.

## Migrate to Extract

Swap one parameter. Everything else about the request stays the same:

<CodeGroup>
  ```python Python theme={"dark"}
  # before
  params = {'url': url, 'apikey': apikey, 'autoparse': 'true'}

  # after
  params = {'url': url, 'apikey': apikey, 'extract': 'auto'}
  ```

  ```javascript Node.js theme={"dark"}
  // before
  params: { url: url, apikey: apikey, autoparse: 'true' }

  // after
  params: { url: url, apikey: apikey, extract: 'auto' }
  ```

  ```bash cURL theme={"dark"}
  # before
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=YOUR_URL&autoparse=true"

  # after
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=YOUR_URL&extract=auto"
  ```
</CodeGroup>

Two things change, so test before you switch a production job over:

* **The response shape.** Autoparse returns whatever JSON it can find on the page, with no guaranteed keys. Extract returns named fields for the entities on the page, so any code that reads Autoparse output by position or by a key it discovered by inspection needs updating.
* **Domain coverage.** Extract serves [prepared domains](/extract/setup#domain-coverage). If yours is not prepared yet, [request it](/extract/endpoints#getting-a-domain-prepared); preparation usually takes minutes and then applies to every Zenrows API key.

On the eCommerce page used throughout this guide, the difference is concrete:

| Parameter | What comes back |
| - | - |
| `autoparse=true` | The JSON embedded in the page: WordPress and WooCommerce configuration objects and a breadcrumb list. The product listing is not part of it. |
| `extract=auto` | The 188 products, each with `name`, `price`, `currency`, `sku`, `availability`, `image_url` and `product_url`, plus a `page_context` object with the pagination. |

## How Autoparse works

Autoparse takes one of two paths, depending on the site:

* **A dedicated parser**, where Zenrows ships one for that site. These cover a limited set of well-known sites and return named fields such as product details, job listings or search results.
* **Embedded JSON**, everywhere else. Autoparse collects the JSON objects it finds in the page's `<script>` tags and returns them as an array. On sites that render their data into those objects this is useful; on sites that do not, it returns page configuration and little else.

Neither path is tailored to the site you are requesting, which is the gap [Extract](/extract/setup) closes: it builds and maintains an extraction per domain and returns named fields for the entities on the page.

<Warning>Remember that Autoparse is an automatic feature designed for general-purpose extraction. It may not capture all fields on every website. Always test requests to verify that all required data is present before implementing in production.</Warning>

## Basic usage

Enable Autoparse by adding the `autoparse=true` parameter to your Zenrows request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'autoparse': 'true',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'autoparse': 'true',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true"
  ```
</CodeGroup>

This returns JSON instead of raw HTML. What that JSON holds depends on the site: a dedicated parser where one exists, otherwise the JSON objects embedded in the page. On this eCommerce page it is WordPress and WooCommerce configuration rather than the product listing, which is what [Extract](/extract/setup) returns instead.

## Where Autoparse is still used

New integrations should start with [Extract](/extract/setup). These are the cases the parameter covers today:

**Content extraction needs:**

* **E-commerce scraping** - Product catalogs, pricing data, reviews, and specifications
* **News and media** - Article content, headlines, author information, and publication dates
* **Job board aggregation** - Job listings, company details, requirements, and salary information
* **Real estate data** - Property listings, prices, descriptions, and location details
* **Event information** - Event details, dates, venues, and other event information

**Development scenarios:**

* **Rapid prototyping** - Quick data extraction without writing custom parsers
* **Multi-site scraping** - Extracting similar data from different website layouts
* **Unknown site structures** - When you need to explore what data is available
* **Proof of concept projects** - Testing data availability before building custom solutions

**For dynamic content that loads via JavaScript, combine Autoparse with JavaScript Rendering:**

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'autoparse': 'true',
      'js_render': 'true',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'autoparse': 'true',
          'js_render': 'true',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true&js_render=true";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true&js_render=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true&js_render=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true&js_render=true')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&autoparse=true&js_render=true"
  ```
</CodeGroup>

This combination ensures that dynamically loaded content is available for parsing while still providing structured JSON output. For more information about JavaScript Rendering, see the [JavaScript Rendering](/fetch/features/js-rendering) documentation.

## Comparing extraction methods

| Method | Best for | Pros | Cons |
| - | - | - | - |
| **Extract** (`extract=auto`, Beta) | Any site where you want structured data | Tailored to each website, adapts when the site's layout changes, richer fields | The domain has to be prepared first |
| **Autoparse** (deprecated) | Integrations already built on it | No coding required, works across sites, JSON output | General-purpose rules, often returns page scaffolding rather than the data you want |
| **CSS Extractor** | Specific data, single site, custom requirements | Full control, precise targeting, efficient | Requires HTML knowledge, site-specific |
| **Custom Parsing** | Complex logic, data transformation | Maximum flexibility, custom processing | Time-intensive, maintenance overhead |

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| **Missing expected data** | Content not in standard format | Contact support for analysis or switch to custom parsing |
| **Empty or incomplete extraction** | JavaScript-loaded content | Add `js_render=true` and `wait` parameters |
| **Page blocked or captcha** | Site protection systems | Combine `js_render=true` + `premium_proxy=true` |
| **Unexpected data structure** | Site uses non-standard markup | Test with manual CSS Extractor instead of Autoparse |

### Improving extraction accuracy

When Autoparse doesn't capture all the data you need:

<Steps>
  <Step title="Test if the content loads dynamically">
    ```python Python theme={"dark"}
    # For empty or missing content, enable JavaScript rendering
    params = {
        'autoparse': 'true',
        'js_render': 'true',
        'wait': '3000',  # Wait for content to load
    }
    ```
  </Step>

  <Step title="Bypass protection if the site is blocked">
    ```python Python theme={"dark"}
    # For blocked pages or captchas
    params = {
        'autoparse': 'true',
        'js_render': 'true',
        'premium_proxy': 'true',
    }
    ```
  </Step>

  <Step title="Wait for specific elements to appear">
    ```python Python theme={"dark"}
    # Wait for specific elements to appear
    params = {
        'autoparse': 'true',
        'js_render': 'true',
        'wait_for': '.product-price',
    }
    ```
  </Step>

  <Step title="Contact support or switch to manual parsing">
    If Autoparse consistently misses the specific fields you need, contact Zenrows support for analysis, or consider switching to the manual CSS Extractor for precise control. [Extract](/extract/setup) may also capture more complete fields once the domain is prepared.

    <Warning>Remember that Autoparse is an automatic feature designed for general-purpose extraction. It may not capture all fields on every website. Always run test requests to verify that all required data is present before implementing in production.</Warning>
  </Step>
</Steps>

## Pricing

The `autoparse=true` parameter is included at **no additional cost** with all Zenrows requests. You only pay extra for JavaScript Render and Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="What types of websites work best with Autoparse?">
  Autoparse works best with structured content sites like e-commerce stores, news websites, job boards, real estate listings, and social media platforms. Sites with clear content hierarchy and semantic markup provide the most accurate results.
</Accordion>

<Accordion title="Can I combine Autoparse with other Zenrows features?">
  Yes, Autoparse works with all Zenrows features, except other output features like JSON Response or Markdown Response. If you also set `extract=auto` (Beta) on the same request, Extract takes precedence and Autoparse is ignored for that request.
</Accordion>

<Accordion title="What happens if Autoparse doesn't find the data I need?">
  If Autoparse misses specific data points feel free to contact Zenrows support for analysis or consider switching to manual CSS Extractor for precise control.
</Accordion>

<Accordion title="Does Autoparse work with JavaScript-heavy websites?">
  Autoparse processes whatever HTML is available. For JavaScript-heavy sites, combine it with `js_render=true` to ensure dynamic content is loaded before parsing. This combination provides comprehensive extraction for modern web applications.
</Accordion>

<Accordion title="What replaces Autoparse?">
  [Extract](/extract/setup) (`extract=auto`). Instead of applying the same general-purpose rules to every website, it builds a dedicated extraction for each individual site, which returns richer fields and adapts automatically when the site's layout changes. See [Migrate to Extract](#migrate-to-extract) for the parameter swap and what changes in the response.
</Accordion>

<Accordion title="Is the autoparse parameter going away?">
  Not today. It is deprecated, which means it still works and existing integrations keep running, but it is no longer recommended and new work should start with [Extract](/extract/setup). If a removal is scheduled, it will be announced separately with notice.
</Accordion>


# Block Resources
Source: https://docs.zenrows.com/fetch/features/block-resources

Block images, stylesheets, fonts, and other unnecessary resources during web scraping to reduce bandwidth and speed up page load times.

The Block Resources parameter prevents your headless browser from downloading specific types of content that aren't essential for your scraping task. By blocking unnecessary resources, such as images, stylesheets, fonts, and media files, you can significantly improve scraping efficiency, reduce loading times, optimize performance, and minimize bandwidth usage.

When you block resources, Zenrows instructs the browser to ignore requests for the specified content types, allowing pages to load faster and consume less data. This is particularly valuable when scraping content-heavy sites where you only need the text data or specific elements.

<Warning>Zenrows automatically blocks certain resource types by default, such as stylesheets and images, to optimize scraping speed and reduce unnecessary data load. **We recommend only using this parameter when you need to customize the default blocking behavior or when troubleshooting specific issues.**</Warning>

## How Block Resources works

Block Resources operates at the network level within the browser, intercepting requests for specified resource types before they're downloaded. When the browser encounters a request for a blocked resource type, it prevents the download entirely, reducing both loading time and bandwidth consumption.

This process affects:

* Page loading speed by eliminating unnecessary downloads
* Bandwidth usage by preventing large file transfers
* Memory consumption by reducing the amount of data processed
* Rendering time by focusing only on essential content
* Network request volume by filtering out non-critical resources

The blocking happens transparently during page rendering, ensuring that essential functionality remains intact while removing unnecessary overhead.

## Basic usage

Add the `block_resources` parameter with comma-separated resource types to your request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'block_resources': 'image,media,font',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'js_render': 'true',
          'block_resources': 'image,media,font',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&block_resources=image%2Cmedia%2Cfont";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&block_resources=image%2Cmedia%2Cfont');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&block_resources=image%2Cmedia%2Cfont", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&block_resources=image%2Cmedia%2Cfont')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&block_resources=image%2Cmedia%2Cfont"
  ```
</CodeGroup>

This example blocks images, media files, and fonts while allowing other resources to load normally, resulting in faster page loading and reduced data usage.

## Available resource types

Zenrows supports blocking the following resource types:

### Content resources

* `stylesheet` - CSS files that define visual styling and layout
* `image` - All image formats including JPG, PNG, GIF, SVG, and WebP
* `media` - Audio and video files of all formats
* `font` - Web fonts including WOFF, WOFF2, TTF, and OTF files

### Script and dynamic content

* `script` - JavaScript files and inline scripts
* `xhr` - XMLHttpRequest calls used for AJAX functionality
* `fetch` - Modern Fetch API requests for dynamic content loading
* `eventsource` - Server-sent events for real-time data streams
* `websocket` - WebSocket connections for bidirectional communication

### Application resources

* `texttrack` - Video subtitle and caption files
* `manifest` - Web app manifests containing application metadata
* `other` - Any resource types not specifically categorized above

### Special values

* `none` - Disables all resource blocking, allowing everything to load

## When to use Block Resources

Block Resources is essential for these scenarios:

### Performance optimization:

* **Large-scale scraping** - Reduce bandwidth and processing time across many requests
* **Content-heavy sites** - Skip images and media when extracting text data
* **Slow connections** - Minimize data transfer in bandwidth-limited environments
* **High-volume operations** - Optimize resource usage for continuous scraping tasks
* **Cost reduction** - Lower bandwidth costs for cloud-based scraping operations

### Content-specific extraction:

* **Text-only scraping** - Extract articles, reviews, or descriptions without visual elements
* **Data mining** - Focus on structured data while ignoring presentation layers
* **API monitoring** - Allow dynamic requests while blocking static resources
* **Search engine optimization** - Analyze content structure without styling interference
* **Accessibility testing** - Test content availability without visual dependencies

### Troubleshooting and debugging:

* **JavaScript conflicts** - Isolate issues by selectively blocking script types
* **Loading problems** - Identify problematic resources causing page failures
* **Performance analysis** - Measure impact of different resource types on loading speed
* **Content validation** - Verify that essential content doesn't depend on blocked resources

## Disabling resource blocking

To turn off Zenrows' default resource blocking and allow all content to load:

```python Python theme={"dark"}
params = {
    'js_render': 'true',
    'block_resources': 'none',  # Disable all resource blocking
}
```

Use `block_resources=none` when:

* You need complete visual fidelity
* Debugging layout or styling issues
* Testing how pages appear to real users
* Analyzing complete resource loading patterns
* Ensuring no content dependencies are missed

## Best practices

### Monitor content integrity

Verify that blocked resources don't affect essential content:

```python Python theme={"dark"}
def validate_content_integrity(url, target_selectors):
    # Compare content availability with and without resource blocking

    from bs4 import BeautifulSoup
    
    # Test with default blocking
    default_response = requests.get('https://api.zenrows.com/v1/', params={
        'url': url,
        'apikey': 'YOUR_ZENROWS_API_KEY',
        'js_render': 'true',
    })
    
    # Test with aggressive blocking
    aggressive_response = requests.get('https://api.zenrows.com/v1/', params={
        'url': url,
        'apikey': 'YOUR_ZENROWS_API_KEY',
        'js_render': 'true',
        'block_resources': 'image,media,font,stylesheet,script',
    })
    
    default_soup = BeautifulSoup(default_response.text, 'html.parser')
    aggressive_soup = BeautifulSoup(aggressive_response.text, 'html.parser')
    
    integrity_report = {}
    
    for selector in target_selectors:
        default_elements = default_soup.select(selector)
        aggressive_elements = aggressive_soup.select(selector)
        
        integrity_report[selector] = {
            'default_count': len(default_elements),
            'aggressive_count': len(aggressive_elements),
            'content_preserved': len(default_elements) == len(aggressive_elements),
        }
    
    return integrity_report

# Validate that important content is preserved
selectors = ['.product-title', '.price', '.description', '.reviews']
integrity = validate_content_integrity('https://ecommerce.com/product/123', selectors)

for selector, data in integrity.items():
    status = "✓" if data['content_preserved'] else "✗"
    print(f"{status} {selector}: {data['default_count']} → {data['aggressive_count']} elements")
```

### Combine with other parameters

Use Block Resources with complementary features for optimal results:

```python Python theme={"dark"}
params = {
    'js_render': 'true',
    'block_resources': 'image,media,font,stylesheet',  # Block visual resources
    'wait_for': '.main-content',                       # Wait for essential content
    'json_response': 'true',                          # Capture API calls
    'premium_proxy': 'true',                          # For protected sites
}
```

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Missing content | Essential scripts blocked | Remove `script` from blocked resources |
| Incomplete data | XHR/Fetch requests blocked | Allow `xhr` and `fetch` resources |
| Layout issues | CSS dependencies | Remove `stylesheet` from blocking |
| Slow loading | Too many resources allowed | Add more resource types to blocking |
| JavaScript errors | Font or media dependencies | Test with individual resource types |

### Debugging missing content

When content disappears after blocking resources:

<Steps>
  <Step title="Compare HTML outputs">
    Compare the HTML obtained with and without resource blocking to identify missing elements.

    ```python Python theme={"dark"}
    # Get baseline without custom blocking
    baseline = requests.get('https://api.zenrows.com/v1/', params={
        'url': url,
        'apikey': 'YOUR_ZENROWS_API_KEY',
        'js_render': 'true',
    })
    # Get result with blocking
    blocked = requests.get('https://api.zenrows.com/v1/', params={
        'url': url,
        'apikey': 'YOUR_ZENROWS_API_KEY',
        'js_render': 'true',
        'block_resources': 'image,media,font,script',
    })

    print(f"Baseline length: {len(baseline.text)}")
    print(f"Blocked length: {len(blocked.text)}")
    ```
  </Step>

  <Step title="Test individual resource types">
    Block one resource type at a time to identify the problematic category.

    ```python Python theme={"dark"}
    resource_types = ['image', 'media', 'font', 'stylesheet', 'script', 'xhr']

    for resource_type in resource_types:
        response = requests.get('https://api.zenrows.com/v1/', params={
            'url': url,
            'apikey': 'YOUR_ZENROWS_API_KEY',
            'js_render': 'true',
            'block_resources': resource_type,
        })
        
        # Check if target content exists
        has_content = 'target-selector' in response.text
        print(f"Blocking {resource_type}: Content present = {has_content}")
    ```
  </Step>

  <Step title="Adjust blocking strategy">
    Remove problematic resource types from your blocking configuration.

    ```python Python theme={"dark"}
    # If script blocking causes issues, exclude it

    safe_blocking = 'image,media,font,stylesheet'  # Exclude script, xhr, fetch
    response = requests.get('https://api.zenrows.com/v1/', params={
        'url': url,
        'apikey': 'YOUR_ZENROWS_API_KEY',
        'js_render': 'true',
        'block_resources': safe_blocking,
    })
    ```
  </Step>
</Steps>

### Performance vs. content trade-offs

Balance optimization with content completeness:

```python Python theme={"dark"}
def find_optimal_blocking(url, required_selectors):
    # Find the most aggressive blocking that preserves required content

    blocking_configurations = [
        'image',
        'image,media',
        'image,media,font',
        'image,media,font,stylesheet',
        'image,media,font,stylesheet,other',
    ]
    
    optimal_config = None
    
    for config in blocking_configurations:
        response = requests.get('https://api.zenrows.com/v1/', params={
            'url': url,
            'apikey': 'YOUR_ZENROWS_API_KEY',
            'js_render': 'true',
            'block_resources': config,
        })
        
        from bs4 import BeautifulSoup
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Check if all required content is present
        all_content_present = all(
            soup.select(selector) for selector in required_selectors
        )
        
        if all_content_present:
            optimal_config = config
            print(f"✓ Configuration '{config}' preserves all content")
        else:
            print(f"✗ Configuration '{config}' missing content")
            break
    
    return optimal_config

# Find optimal blocking for specific content requirements
required_elements = ['.product-title', '.price', '.description']
best_config = find_optimal_blocking('https://shop.com/product/123', required_elements)
print(f"Recommended blocking: {best_config}")
```

## Pricing

The `block_resources` parameter doesn't increase the request cost on its own. Cost still depends on `js_render` and `premium_proxy`, exactly as it would for a standard request.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="What resources does Zenrows block by default?">
  Zenrows automatically blocks stylesheets and images by default to optimize scraping speed and reduce data usage. You can override this behavior by setting `block_resources=none` to allow all resources, or by specifying custom blocking for specific resource types.
</Accordion>

<Accordion title="Will blocking JavaScript break dynamic content loading?">
  Yes, blocking scripts can prevent dynamic content from loading properly. If you need content that's loaded via JavaScript, avoid blocking the `script`, `xhr`, and `fetch` resource types. Focus on blocking visual resources like `image`, `media`, `font`, and `stylesheet` instead.
</Accordion>

<Accordion title="How much bandwidth can I save by blocking resources?">
  Bandwidth savings vary significantly depending on the website's content and design. Image-heavy sites can significantly reduce data transfer by blocking images and media. Text-focused sites may achieve savings by blocking stylesheets and fonts.
</Accordion>

<Accordion title="Can I block resources when not using js_render?">
  No, the `block_resources` parameter only works with `js_render=true` because resource blocking operates within the browser environment during JavaScript rendering. For static HTML requests, resource blocking isn't applicable.
</Accordion>

<Accordion title="What happens if I block XHR or Fetch requests?">
  Blocking XHR or Fetch requests prevents the page from making AJAX calls, which can result in missing dynamic content. Only block these if you're certain the content you need is available in the initial HTML without requiring additional API calls.
</Accordion>

<Accordion title="Does blocking resources affect the JSON Response feature?">
  Yes, if you block `xhr` or `fetch` resources, those requests won't appear in the JSON Response XHR array. However, blocking visual resources, such as images and stylesheets, won't affect the capture of API calls and can reduce the overall response size.
</Accordion>

<Accordion title="Can I use wildcards or patterns in resource blocking?">
  No, you must specify exact resource type names from the supported list. You cannot use wildcards or custom patterns. To block multiple types, separate them with commas, such as `image,media,font`.
</Accordion>

<Accordion title="How do I know which resources are safe to block for my use case?">
  Start by blocking only visual resources (`image,media,font,stylesheet`) and test if your target content is still available. If content is missing, remove resource types one by one until you find the minimal blocking configuration that preserves your required data.
</Accordion>


# Concurrency
Source: https://docs.zenrows.com/fetch/features/concurrency

Learn how concurrent request limits work in Zenrows, including plan-based rate limits, parallel scraping, and managing simultaneous API calls.

Concurrency in web scraping is essential for efficient data extraction, especially when dealing with multiple URLs. Managing the number of concurrent requests helps prevent overwhelming the target server and ensures you stay within rate limits. Depending on your subscription plan, you can perform anywhere from 5 to 200 or more <a href="https://en.wikipedia.org/wiki/Concurrency_(computer_science)">concurrent requests</a>. See the [Concurrency, Support, and Rollover table](/first-steps/pricing#concurrency-support-and-rollover) for exact numbers by tier.

<Note>In short, concurrency refers to the number of API requests you can have in progress (or *running*) simultaneously. If your plan supports 5 concurrent requests, you can process up to 5 requests simultaneously. You'll get an error if you send a sixth request while five are already processing.</Note>

## Understanding Concurrency

Concurrency is a fundamental concept in web scraping, referring to the ability to handle multiple tasks simultaneously. In the context of Zenrows, it defines how many scraping requests can be processed at the same time.

Think of concurrency like a team of workers in a factory. Each worker represents a "concurrent request slot." If you have 5 workers, you can assign them 5 tasks (or requests) simultaneously. If you try to assign a 6th task while all workers are occupied, you will need to wait until one of them finishes their current task before the new one can be started.

In Zenrows, each "task" is an API request, and each "worker" is a concurrent request slot available to you based on your subscription.

### Impact of Request Duration on Throughput

The duration that each request takes to complete significantly influences how many requests you can process in a given timeframe. This concept is crucial for optimizing your scraping efficiency and maximizing throughput. Here's how it works:

* **Fast Requests:** If each request takes 1 second to complete and you have 5 concurrent slots available, you can process 5 requests every second. Over a 60-second period, this means you can handle 300 requests (5 requests/second × 60 seconds).
* **Slow Requests:** Conversely, if each request takes 10 seconds to complete, you can process 5 requests every 10 seconds. Over the same 60-second period, you'll only manage 30 requests (5 requests/10 seconds × 60 seconds).

This demonstrates that reducing the duration of each request increases the number of requests you can process in the same amount of time.

### Example Scenario

To better understand this, consider a situation where your plan allows 5 concurrent requests:

Scenario:

* **1st Request**: Takes 10 seconds to finish.
* **2nd Request**: Takes 7 seconds to finish.
* **3rd Request**: Takes 8 seconds to finish.
* **4th Request**: Takes 9 seconds to finish.
* **5th Request**: Takes 14 seconds to finish.

You start all 5 requests simultaneously. Each request occupies one of the 5 available slots. If you then attempt to send a:

* **6th & 7th Request**: Since all 5 slots are occupied, you will receive "[429 Too Many Requests](https://docs.zenrows.com/api-error-codes#auth008-rate-limit-exceeded)" errors. The system can only process additional requests once one of the initial 5 requests finishes. In this example, the quickest request (the 2nd request) completes in 7 seconds, freeing up a slot for new requests.

<Frame>
  <img alt="Concurrency example" />
</Frame>

## Concurrency Headers

To help you manage and optimize your API usage, each response from our API includes two important HTTP headers related to concurrency:

1. `Concurrency-Limit`: Indicates the total number of concurrent requests allowed by your current plan. This header helps you understand the maximum concurrency capacity available to you.
2. `Concurrency-Remaining`: Shows the number of available concurrency slots at the time the request was received by the server. This provides insight into how many slots are still free.

For example, if your plan supports 20 concurrent requests and you send 3 requests simultaneously, the response headers might be:

* `Concurrency-Limit: 20`
* `Concurrency-Remaining: 17`

This means that at the time of the request, 17 slots were available, while 3 were occupied by the requests you had in progress.

<Frame>
  <img alt="Concurrency headers" />
</Frame>

### Using Concurrency Headers for Optimization

These headers are valuable tools for optimizing your scraping tasks. By monitoring and interpreting these headers in real time, you can adjust your request patterns to make the most efficient use of your concurrency slots.

Optimization Tips:

1. Before sending a batch of requests, inspect the `Concurrency-Remaining` header of the most recent response.
2. Based on the value of this header, adjust the number of parallel requests you send. For example, if `Concurrency-Remaining` is 5, avoid sending more than 5 simultaneous requests.

By adapting your request strategy based on these headers, you can reduce the likelihood of encountering "429 Too Many Requests" errors and ensure a smoother, more efficient interaction with the API.

## Using Concurrency

<Tip>Most programming languages and clients do not natively support concurrency, so you may need to implement your solution. Alternatively, you can use the solutions provided below. 😉</Tip>

### Zenrows SDK for Python

To run the examples, ensure you have <a href="https://www.python.org/downloads/">Python 3</a> installed. Install the necessary libraries with:

```bash theme={"dark"}
pip install zenrows
```

<a href="https://pypi.org/project/zenrows/">Zenrows Python SDK</a> comes with built-in concurrency and retries. You can set these parameters in the constructor. Keep in mind that each client instance has its own limit, so running multiple scripts might lead to 429 Too Many Requests errors.

The `asyncio.gather` function will wait for all the calls to finish and store all the responses in an array. Afterward, you can loop over the array and extract the necessary data. Each response will include the status, request, response content, and other values. Remember to run the scripts with `asyncio.run` to avoid a `coroutine 'main' was never awaited` error.

```python scraper.py theme={"dark"}
from zenrows import ZenRowsClient
import asyncio
from urllib.parse import urlparse, parse_qs

client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1)

urls = [
	# ...
]

async def main():
	responses = await asyncio.gather(*[client.get_async(url) for url in urls])

	for response in responses:
		original_url = parse_qs(urlparse(response.request.url).query)["url"]
		print({
			"response": response,
			"status_code": response.status_code,
			"request_url": original_url,
		})

asyncio.run(main())
```

### Python with `requests`

If you prefer using the `requests` library and want to handle multiple requests concurrently, Python's multiprocessing package can be an effective solution. This approach is particularly useful when you're dealing with a large list of URLs and need to speed up the data collection process by sending multiple requests simultaneously.

```bash theme={"dark"}
pip install requests
```

The `multiprocessing` package in Python includes a `ThreadPool` class, which allows you to manage a pool of worker threads. Each thread can handle a separate task, enabling multiple HTTP requests to be processed in parallel. This is particularly beneficial when scraping data from a large number of websites, as it reduces the overall time required.

```python scraper.py theme={"dark"}
import requests
from multiprocessing.pool import ThreadPool

apikey = "YOUR_ZENROWS_API_KEY"
concurrency = 10
urls = [
	# ... your URLs here
]

def scrape_with_zenrows(url):
	response = requests.get(
		url="https://api.zenrows.com/v1/",
		params={
			"url": url,
			"apikey": apikey,
		},
	)

	return {
		"content": response.text,
		"status_code": response.status_code,
		"request_url": url,
	}

pool = ThreadPool(concurrency)
results = pool.map(scrape_with_zenrows, urls)
pool.close()
pool.join()

[print(result) for result in results]
```

### Zenrows SDK for JavaScript

When working with JavaScript for web scraping, managing concurrency and handling retries can be challenging.

The <a href="https://www.npmjs.com/package/zenrows">Zenrows JavaScript SDK</a> simplifies these tasks by providing built-in concurrency and retry options. This is particularly useful for developers who need to scrape multiple URLs efficiently while avoiding rate limits.

To get started, install the Zenrows SDK using npm:

```bash theme={"dark"}
npm i zenrows
```

Zenrows allows you to control the concurrency level by passing a number in the constructor. It's important to set this according to your subscription plan's limits to prevent 429 (Too Many Requests) errors. Remember, each client instance has its own concurrency limit, so running multiple scripts won't share this limit.

```javascript theme={"dark"}
const { ZenRows } = require('zenrows');

const apiKey = 'YOUR_ZENROWS_API_KEY';

(async () => {
	const client = new ZenRows(apiKey, { concurrency: 5, retries: 1 });

	const urls = [
		// ...
	];
	const promises = urls.map(url => client.get(url));

	const results = await Promise.allSettled(promises);
	console.log(results);
	/*
	[
		{
			status: 'fulfilled',
			value: {
				status: 200,
				statusText: 'OK',
				data: ...
		...
	*/

	// separate results list into rejected and fulfilled for later processing
	const rejected = results.filter(({ status }) => status === 'rejected');
	const fulfilled = results.filter(({ status }) => status === 'fulfilled');
})();
```

In this example, we use <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled">Promise.allSettled()</a> to handle multiple asynchronous requests. This method is available in Node.js 12.9 and later. It waits for all the promises to settle, meaning it doesn't stop if some requests fail. Instead, it returns an array of objects, each with a status of either fulfilled or rejected.

This approach makes your scraping more robust, as it ensures that all URLs in your list are processed, even if some requests encounter issues. You can then handle the `fulfilled` and `rejected` responses separately, allowing you to log errors or retry failed requests as needed.

## Frequently Asked Questions (FAQ)

<Accordion title="How does concurrency affect the speed of scraping tasks?">
  Concurrency directly influences how many tasks can be handled at the same time, meaning higher concurrency results in faster scraping as multiple requests can be processed simultaneously.
</Accordion>

<Accordion title="What happens if I exceed my concurrency limit?">
  If you exceed your concurrency limit, you will receive a `429 Too Many Requests` error. The system won't accept additional requests until current tasks complete and free up slots.
</Accordion>

<Accordion title="How can I monitor my available concurrency slots?">
  Use the `Concurrency-Remaining` header in your API responses to check how many slots are still available for sending new requests.
</Accordion>

<Accordion title="What is the difference between concurrency and rate limiting?">
  Concurrency refers to the number of tasks handled at once, while rate limiting controls the total number of requests allowed in a set time period. Both can impact the efficiency of your scraping tasks.
</Accordion>

<Accordion title="How can I handle slow requests and optimize concurrency usage?">
  To improve efficiency, consider breaking down larger tasks into smaller chunks or optimizing the time each request takes to complete. Faster requests will free up slots more quickly.
</Accordion>

<Accordion title="If I cancel a request, will it free up the concurrency right away?">
  No, when you cancel a request, it can take up to 3 minutes for the associated concurrency slot to be freed. If you cancel multiple requests at once, you will need to wait a few minutes before you can fully utilize those freed slots again.
</Accordion>


# CSS Extractor
Source: https://docs.zenrows.com/fetch/features/css-extractor

Use CSS selectors and XPath expressions to extract specific data points from HTML pages and return structured JSON with Zenrows CSS Extractor.

The CSS Extractor parameter transforms Zenrows' standard HTML output into structured JSON data containing only the specific elements you need. Instead of receiving the full HTML content and parsing it yourself, you get clean, organized data extracted using CSS selectors and XPath expressions.

This feature is particularly useful when you need to:

* Extract specific data points like product prices, titles, or links
* Transform unstructured HTML into structured JSON for easy processing
* Reduce response size by getting only relevant information
* Automate data collection from consistent page structures
* Build data pipelines that require predictable JSON output

<Note>The CSS Extractor works with both standard scraping and JavaScript rendering. For dynamic content that loads via AJAX, combine it with `js_render=true` for complete data extraction.</Note>

## How CSS Extractor works

CSS Extractor processes the rendered HTML content using CSS selectors or XPath expressions to identify and extract specific elements. The browser parses the page content, locates elements matching your selectors, and returns the extracted data in a structured JSON format.

This process captures:

* Text content from matching elements
* Attribute values (href, src, data attributes, etc.)
* Multiple elements as arrays when selectors match several items
* Complex data structures using nested extraction rules

The extraction happens after the page is fully loaded, ensuring you capture all content including dynamically generated elements when used with JavaScript rendering.

## Basic usage

Enable CSS Extractor by adding the `css_extractor` parameter with a JSON object defining your extraction rules:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'css_extractor': """{"links":"a @href","images":"img @src"}""",
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'css_extractor': `{"links":"a @href","images":"img @src"}`,
      },
  })
      .then(response => {
          console.log(response.data.tables[0]); // Access the first table directly
      })
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%7B%22links%22%3A%22a%20%40href%22%2C%20%22images%22%3A%22img%20%40src%22%7D";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%7B%22links%22%3A%22a%20%40href%22%2C%20%22images%22%3A%22img%20%40src%22%7D');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%7B%22links%22%3A%22a%20%40href%22%2C%20%22images%22%3A%22img%20%40src%22%7D", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%7B%22links%22%3A%22a%20%40href%22%2C%20%22images%22%3A%22img%20%40src%22%7D')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%7B%22links%22%3A%22a%20%40href%22%2C%20%22images%22%3A%22img%20%40src%22%7D"
  ```
</CodeGroup>

This example extracts the page title, price elements, and all link URLs, returning them as a structured JSON object instead of raw HTML.

## Extraction patterns

The CSS Extractor supports various extraction patterns to handle different types of content and data structures.

### Basic text extraction

Extract text content from elements using standard CSS selectors:

| Extraction Rule | Sample HTML | Description | JSON Output |
| - | - | - | - |
| \{"title":"h1"} | \<h1>Welcome to Our Store\</h1> | Extract text from h1 element | \{"title": "Welcome to Our Store"} |
| \{"description":"p.intro"} | \<p class="intro">Best products here\</p> | Extract text from paragraph with intro class | \{"description": "Best products here"} |
| \{"content":"#main-content"} | \<div id="main-content">Page content\</div> | Extract text from element with specific ID | \{"content": "Page content"} |

### Attribute extraction

Extract specific attributes from elements by adding `@attribute_name` to your selector:

| Extraction Rule | Sample HTML | Description | JSON Output |
| - | - | - | - |
| \{"links":"a @href"} | \<a href="/products">Products\</a> | Extract *href* attribute from links | \{"links": "/products"} |
| \{"images":"img @src"} | \<img src="photo.jpg" alt="Product" /> | Extract *src* attribute from images | \{"images": "photo.jpg"} |
| \{"form\_token":"input\[name=\_token] @value"} | \<input name="\_token" value="abc123" /> | Extract *value* attribute from hidden input | \{"form\_token": "abc123"} |

### Multiple elements

When your selector matches multiple elements, CSS Extractor automatically returns an array:

| Extraction Rule | Sample HTML | Description | JSON Output |
| - | - | - | - |
| \{"products":"h2.product-title"} | \<h2 class="product-title">Product 1\</h2>\<h2 class="product-title">Product 2\</h2> | Extract text from multiple elements | \{"products": \["Product 1", "Product 2"]} |
| \{"prices":".price"} | \<span class="price">\$19.99\</span>\<span class="price">\$29.99\</span> | Extract text from multiple price elements | \{"prices": \["\$19.99", "\$29.99"]} |
| \{"all\_links":"a @href"} | \<a href="/page1">Link 1\</a>\<a href="/page2">Link 2\</a> | Extract href attributes from multiple links | \{"all\_links": \["/page1", "/page2"]} |

### Advanced selectors

Use complex CSS selectors for precise targeting:

| Extraction Rule | Sample HTML | Description | JSON Output |
| - | - | - | - |
| \{"emails":"a\[href^='mailto:'] @href"} | \<a href="mailto:[contact@example.com](mailto:contact@example.com)">Email us\</a> | Extract *href* attribute for mailto links | \{"emails": "mailto:[contact@example.com](mailto:contact@example.com)"} |
| \{"hidden\_values":"input\[type=hidden] @value"} | \<input type="hidden" value="secret123" /> | Extract *value* attribute from hidden inputs | \{"hidden\_values": "secret123"} |
| \{"data\_attrs":"button @data-product-id"} | \<button data-product-id="12345">Buy Now\</button> | Extract custom data attribute | \{"data\_attrs": "12345"} |

### XPath expressions

For more complex extractions, use XPath expressions. XPath is a query language for selecting nodes in XML/HTML documents, offering more flexibility than CSS selectors:

| Extraction Rule | Sample HTML | Description | JSON Output |
| - | - | - | - |
| \{"heading":"//h1"} | \<h1>Page Title\</h1> | Extract text using XPath | \{"heading": "Page Title"} |
| \{"image\_src":"//img @src"} | \<img src="banner.png" alt="Banner" /> | Extract *src* attribute using XPath | \{"image\_src": "banner.png"} |
| \{"text\_content":"//div\[@class='content']//text()"} | \<div class="content">Hello \<span>World\</span>\</div> | Extract all text content using XPath | \{"text\_content": "Hello World"} |

### Complex extraction example

Here's a comprehensive example showing how to extract structured product data from an e-commerce page:

```json JSON theme={"dark"}
{
  "products": "article.product",
  "product_titles": "article.product h3.title",
  "product_prices": "article.product .price @data-price",
  "product_images": "article.product img @src",
  "product_links": "article.product a.product-link @href",
  "availability": "article.product .stock-status",
  "ratings": "article.product .rating @data-rating",
  "categories": "nav.breadcrumb a",
  "page_title": "//title",
  "meta_description": "//meta[@name='description'] @content"
}
```

This extraction rule would return a structured JSON object with all the specified product information, making it easy to process and analyze the data.

## When to use CSS Extractor

CSS Extractor is essential for these scenarios:

**E-commerce data collection**

* Product information - Extract prices, titles, descriptions, and availability
* Inventory monitoring - Track stock levels and price changes
* Competitor analysis - Collect product data from multiple sources
* Review aggregation - Extract customer reviews and ratings
* Category browsing - Collect product listings from category pages

**Content aggregation**

* News articles - Extract headlines, authors, publication dates, and content
* Blog posts - Collect titles, excerpts, and metadata
* Job listings - Collect job titles, companies, locations, and requirements
* Real estate - Extract property details, prices, and contact information

**Data monitoring and analysis**

* Price tracking - Monitor price changes across multiple retailers
* Content changes - Track updates to specific page elements
* SEO analysis - Extract meta tags, headings, and structured data
* Form data - Collect form fields and validation tokens
* API endpoint discovery - Extract AJAX endpoints and data sources

**Development and testing**

* Quality assurance - Verify that specific elements appear correctly
* A/B testing - Extract different page variants for comparison
* Performance monitoring - Track loading of specific page components
* Integration testing - Verify data consistency across different pages

<Note>For pages with dynamic content that loads via JavaScript, combine CSS Extractor with `js_render=true` to ensure all content is captured before extraction.</Note>

## Best practices

### Combine with appropriate Zenrows parameters

Maximize your extraction success by strategically combining CSS Extractor with other Zenrows features. While CSS Extractor works independently with static content, pairing it with complementary parameters ensures reliable data extraction across different website types and protection levels.

#### For dynamic content that loads via JavaScript

When targeting websites that render content dynamically, enable JavaScript rendering and use timing controls to ensure all elements are present before extraction:

```python Python theme={"dark"}
# Dynamic content extraction
params = {
    'url': url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'js_render': 'true',  # Enable JavaScript rendering
    'wait_for': '.product-item',  # Wait for specific elements to appear
    'css_extractor': '{"products":".product-item","prices":".price"}',
}
```

<Tip>
  You can find more information about the `wait_for` parameter [here](/fetch/features/wait-for).
</Tip>

#### For protected or geo-restricted websites

Combine with proxy features to access content that may be blocked or restricted by location:

```python Python theme={"dark"}
params = {
    'url': url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'premium_proxy': 'true',
    'proxy_country': 'US',  # Specify country
    'css_extractor': '{"content":"main","links":"a @href"}',
}
```

<Tip>
  You can find more information about the proxy features on the [Premium Proxy Documentation](/fetch/features/premium-proxy).
</Tip>

#### For complex interactive websites

Use JavaScript Instructions to simulate user interactions before extracting data:

```python Python theme={"dark"}
# Interactive content extraction
params = {
    'url': url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'js_render': 'true',
    'js_instructions': '[{"click": ".load-more"}, {"wait": 2000}]',  # Simulate user actions
    'css_extractor': '{"products":".product-item","total_count":".results-count"}',
}
```

<Tip>
  You can find more information about the JavaScript Instructions Parameter [here](/fetch/features/js-instructions).
</Tip>

### Choose stable and reliable selectors

The foundation of successful CSS extraction is using selectors that remain consistent over time. Prioritize semantic and stable attributes over auto-generated or fragile ones:

```python Python theme={"dark"}
# Excellent - semantic and stable selectors
params = {
    'css_extractor': '{"title":"h1","price":"[data-price]","description":".product-description"}',
}

# Good - stable class names and IDs
params = {
    'css_extractor': '{"content":"#main-content","items":".product-item"}',
}

# Avoid - auto-generated or fragile selectors
params = {
    'css_extractor': '{"title":"._titleComponent_1a2b3c","price":"div:nth-child(3) > span"}',
}
```

**Selector stability hierarchy (most to least stable):**

1. `data-*` attributes (e.g., `[data-testid="product"]`)
2. Semantic IDs (e.g., `#product-title`)
3. Semantic class names (e.g., `.product-description`)
4. Element types with attributes (e.g., `img[alt="product"]`)
5. Complex descendant selectors (use sparingly)

### Test selectors before implementation

Always verify your CSS selectors work correctly on the target website before deploying them in production. This prevents extraction failures and ensures reliable data collection.

<Steps>
  <Step title="Open the target website">
    Navigate to the page you want to scrape in your browser
  </Step>

  <Step title="Access DevTools console">
    1. Right-click on the page and select "Inspect" or press F12
    2. Navigate to the "Console" tab
    3. Test your selector using JavaScript:

    ```javascript theme={"dark"}
    // Test if your selector finds elements
    document.querySelectorAll('.your-selector');

    // Check specific attributes
    document.querySelectorAll('a').forEach(link => console.log(link.href));

    // Verify text content
    document.querySelectorAll('.product-title').forEach(title => console.log(title.textContent));
    ```
  </Step>

  <Step title="Validate results">
    * Ensure the selector returns the expected number of elements
    * Verify the content matches what you want to extract
    * Test attribute extraction (href, src, data attributes)
  </Step>
</Steps>

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Empty or null values | Selector doesn't match any elements | Verify selector syntax and element existence |
| Missing dynamic content | Content loads after page render | Add `js_render=true` and increase `wait` time |
| Incorrect attribute extraction | Wrong attribute name or syntax | Check attribute exists and use correct `@attribute` syntax |
| Partial data extraction | Elements load asynchronously | Use `wait_for` parameter to wait for specific elements |
| Selector too specific | Overly complex selector breaks easily | Use more general, stable selectors |
| Large response size | Extracting too much data | Focus on essential data points only |

### Handling selector failures

If Zenrows cannot find matching elements for your CSS selectors, it will retry internally several times. If selectors still don't match after the timeout period, you may receive incomplete data or empty results. This typically means your selectors don't exist in the final HTML or are too fragile to be reliable.

#### Selector not present in final HTML

<Steps>
  <Step title="Inspect the site using browser DevTools">
    1. Open the target page in your browser
    2. Right-click the target content and choose "Inspect"
    3. Check if your selector exists after the page fully loads
  </Step>

  <Step title="Verify your selector">
    1. Run `document.querySelectorAll('your_selector')` in the browser console
    2. If it returns no elements, your selector is incorrect

    <img alt="ScrapingCourse DevTools Selector Debug" />
  </Step>

  <Step title="Optimization tips">
    1. Use simple selectors like `.class` or `#id`
    2. Prefer stable attributes like `[data-testid="item"]`
    3. Avoid overly specific or deep descendant selectors
  </Step>
</Steps>

#### Dynamic or fragile selectors

Some websites use auto-generated class names that change frequently. These are considered dynamic and unreliable for consistent data extraction.

* Re-check the page in DevTools if a previously working selector fails
* Look for stable attributes like `data-*` attributes
* Use attribute-based selectors, which are more stable over time

**Instead of fragile selectors:**

```python Python theme={"dark"}
# Avoid - auto-generated or fragile selectors
params = {
    'css_extractor': '{"products":".xY7zD1"}',  # Google-style auto-generated
}

params = {
    'css_extractor': '{"items":".product_list__V9tjod"}',  # Mix of readable and random
}
```

**Use stable alternatives:**

```python Python theme={"dark"}
# Better - stable, semantic selectors
params = {
    'css_extractor': '{"products":"[data-testid=\\"product-list\\"]"}',
}

params = {
    'css_extractor': '{"images":"img[src$=\\".jpg\\"]"}',
}

params = {
    'css_extractor': '{"items":"[data-products=\\"item\\"]"}',
}
```

<Tip>Track your CSS selectors over time. When the target website changes its structure, you'll likely need to update your selectors to maintain reliable data extraction.</Tip>

#### Content is conditional or missing

When scraping at scale, it's common to encounter pages where expected content is missing or appears under certain conditions.

**Common scenarios where selectors might fail:**

* **Inexistent elements** - The product exists, but elements like price or "Add to cart" button are missing
* **Deleted or unavailable pages** - Product URLs may be valid, but the product has been removed
* **Failed page loads** - The page might fail to load properly, causing selectors to miss content
* **Conditional rendering** - Content only renders based on user location, browser behavior, or interactions

**How to handle missing content:**

Use these Zenrows parameters to identify and handle these cases:

1. Monitor original status codes
   ```python Python theme={"dark"}
   params = {
       'css_extractor': '{"title":"h1","price":".price"}',
       'original_status': 'true',  # Returns original HTTP status
   }

   response = requests.get('https://api.zenrows.com/v1/', params=params)
   print(response)
   ```
   <Info>For more details check the [original\_status documentation](/fetch/features/other#original-http-code)</Info>

2. Allow error status codes

   ```python Python theme={"dark"}
   params = {
       'css_extractor': '{"error_message":".error-text","content":"main"}',
       'allowed_status_codes': '404,500,503',  # Capture error pages
   }
   ```

   <Info>For more details check the [allowed\_status\_codes documentation](/fetch/features/other#return-content-on-error)</Info>

3. Best practices for handling missing content
   * Anticipate that some selectors may not match if content is missing
   * Include fallback selectors for critical data points
   * Check for error indicators in your extraction rules
   * Monitor extraction success rates to detect site changes

#### Selector exists but extraction still fails

Sometimes your CSS selector is correct but still doesn't extract the expected data:

**Common causes and solutions:**

* **Element is hidden** (`display: none`) - CSS Extractor can still extract hidden content. If you need visible elements only, target child elements or wrappers that appear when content is shown.

  <Info>You can find more information about advanced CSS selectors [here](/fetch/troubleshooting/advanced-css-selectors).</Info>

* **Content appears after user interaction** - Use `js_instructions` to simulate clicks or scrolls before extraction:

  ```python Python theme={"dark"}
  params = {
      'js_render': 'true',
      'js_instructions': '[{"click": ".load-more-button"}]',
      'css_extractor': '{"products": ".product-item"}',
  }
  ```

* **Page relies on slow external scripts** - Try waiting for different selectors that appear earlier, or increase wait times

  ```python Python theme={"dark"}
  params = {
      'js_render': 'true',
      'wait_for': '.initial-content',  # Wait for early-loading content
      'css_extractor': '{"data": ".late-loading-content"}',
  }
  ```

## Pricing

The `css_extractor` parameter is included at no additional cost with all Zenrows requests - you only pay extra for JavaScript Render and Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="Can I use CSS Extractor without JavaScript rendering?">
  Yes, CSS Extractor works with both standard scraping and JavaScript rendering. Use `js_render=true` only when you need to extract content that loads dynamically via JavaScript.
</Accordion>

<Accordion title="What's the difference between CSS selectors and XPath?">
  CSS selectors are simpler and more familiar to web developers, while XPath offers more powerful querying capabilities. CSS selectors are sufficient for most use cases, but XPath is useful for complex document traversal and text manipulation.
</Accordion>

<Accordion title="How many extraction rules can I include in one request?">
  There's no strict limit on the number of extraction rules, but keep in mind that more complex extractions may increase processing time and response size. Focus on extracting only the data you actually need.
</Accordion>

<Accordion title="Can I extract nested or hierarchical data structures?">
  CSS Extractor returns flat JSON structures. For complex nested data, you may need to make multiple requests or use different selectors to extract related data points separately.
</Accordion>

<Accordion title="What happens if my selector matches no elements?">
  If a selector doesn't match any elements, that field will be null or omitted from the JSON response. This won't cause an error, but you should validate your results to ensure critical data was extracted.
</Accordion>

<Accordion title="Can I combine CSS Extractor with other Zenrows features?">
  Yes, CSS Extractor works seamlessly with all Zenrows features including Premium Proxy, JavaScript rendering, Screenshots, and Block Resources. This allows you to handle complex scraping scenarios while getting structured data output.
</Accordion>

<Accordion title="How do I extract data from elements that appear after user interactions?">
  Use JavaScript Instructions to simulate user interactions (clicks, scrolls, form submissions) before extraction. The CSS Extractor will then process the updated page content after these interactions complete.
</Accordion>

<Accordion title="Is there a way to extract only the first match when multiple elements exist?">
  CSS Extractor automatically returns arrays for multiple matches. To get only the first match, you can either make your selector more specific or process the results in your code to take only the first item from arrays.
</Accordion>


# Download Files and Pictures
Source: https://docs.zenrows.com/fetch/features/downloadfile

Download images, PDFs, and other files directly from web pages through the Zenrows API.

Zenrows lets you download images, PDFs, and other binary files directly from web pages as part of your scraping workflow. Instead of scraping HTML and then fetching assets separately, you can point the API at a file URL and receive the binary content ready to save.

This is useful for collecting product images, downloadable reports, manuals, or any non-text content that lives at a direct URL.

## How it works

There are two scenarios where Zenrows can return a file download:

<Steps>
  <Step title="Direct file response">
    If the target URL returns the file directly in the HTTP response (for example, a direct link to a PDF or image), Zenrows fetches the file and returns it as binary content. This is the most reliable and straightforward method.
  </Step>

  <Step title="Triggered download via JS Instructions">
    If the download is triggered by a user action (such as clicking a button or link), you can use Zenrows JS Instructions to simulate that interaction. If the file download starts automatically after the action without prompting the user to choose a save location, Zenrows can capture and return the file.

    <Tip>Find more details on our [JS Instructions Documentation](/fetch/features/js-instructions).</Tip>
  </Step>
</Steps>

<Note>
  Downloads are only possible when the file is delivered directly in the HTTP response. If the site asks the user to choose a download location or requires additional interaction after the trigger, Zenrows cannot capture the file. In those cases, use [Browser Sessions](/browser-sessions/introduction) for more control over the browser session.
</Note>

## Basic usage

Pass the direct file URL as the `url` parameter. Zenrows returns the binary content of the file, which you can then save to disk:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  apikey = "YOUR_ZENROWS_API_KEY"
  target_url = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"

  params = {
      "url": target_url,
      "apikey": apikey,
  }

  response = requests.get("https://api.zenrows.com/v1/", params=params)

  if response.status_code == 200:
      with open("output.pdf", "wb") as f:
          f.write(response.content)
      print("File downloaded and saved to output.pdf")
  else:
      print(f"Download failed. Status: {response.status_code}")
  ```

  ```javascript Node.js theme={"dark"}
  import fs from "fs";

  const apiKey = "YOUR_ZENROWS_API_KEY";
  const targetUrl = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf";

  const url = `https://api.zenrows.com/v1/?apikey=${apiKey}&url=${encodeURIComponent(targetUrl)}`;

  const res = await fetch(url);

  if (!res.ok) throw new Error(`Download failed. Status: ${res.status}`);

  const buffer = await res.arrayBuffer();
  fs.writeFileSync("output.pdf", Buffer.from(buffer));
  console.log("File downloaded and saved to output.pdf");
  ```

  ```java Java theme={"dark"}
  import java.io.*;
  import java.net.*;

  public class DownloadFile {
      public static void main(String[] args) throws IOException {
          String targetUrl = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf";
          String apiKey = "YOUR_ZENROWS_API_KEY";

          String requestUrl = "https://api.zenrows.com/v1/?apikey=" + apiKey + "&url=" + URLEncoder.encode(targetUrl, "UTF-8");

          URL url = new URL(requestUrl);
          HttpURLConnection connection = (HttpURLConnection) url.openConnection();

          if (connection.getResponseCode() == 200) {
              InputStream in = connection.getInputStream();
              FileOutputStream out = new FileOutputStream("output.pdf");

              byte[] buffer = new byte[4096];
              int n;
              while ((n = in.read(buffer)) != -1) {
                  out.write(buffer, 0, n);
              }

              out.close();
              in.close();
              System.out.println("File downloaded and saved to output.pdf");
          } else {
              System.out.println("Download failed. Status code: " + connection.getResponseCode());
          }

          connection.disconnect();
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $targetUrl = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf";
  $apiKey = "YOUR_ZENROWS_API_KEY";

  $apiUrl = "https://api.zenrows.com/v1/?apikey=" . $apiKey . "&url=" . urlencode($targetUrl);

  $response = file_get_contents($apiUrl);

  if ($response !== false) {
      file_put_contents("output.pdf", $response);
      echo "File downloaded and saved to output.pdf";
  } else {
      echo "Download failed.";
  }
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "fmt"
      "io"
      "net/http"
      "net/url"
      "os"
  )

  func main() {
      apiKey := "YOUR_ZENROWS_API_KEY"
      targetUrl := "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"

      requestUrl := fmt.Sprintf("https://api.zenrows.com/v1/?apikey=%s&url=%s", apiKey, url.QueryEscape(targetUrl))

      resp, err := http.Get(requestUrl)
      if err != nil {
          fmt.Println("Error:", err)
          return
      }
      defer resp.Body.Close()

      if resp.StatusCode == 200 {
          out, err := os.Create("output.pdf")
          if err != nil {
              fmt.Println("Error creating file:", err)
              return
          }
          defer out.Close()

          _, err = io.Copy(out, resp.Body)
          if err != nil {
              fmt.Println("Error saving file:", err)
              return
          }

          fmt.Println("File downloaded and saved to output.pdf")
      } else {
          fmt.Println("Download failed. Status code:", resp.Status)
      }
  }
  ```

  ```ruby Ruby theme={"dark"}
  require "net/http"
  require "uri"

  api_key = "YOUR_ZENROWS_API_KEY"
  target_url = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"

  uri = URI("https://api.zenrows.com/v1/")
  uri.query = URI.encode_www_form({ "apikey" => api_key, "url" => target_url })

  response = Net::HTTP.get_response(uri)

  if response.is_a?(Net::HTTPSuccess)
      File.open("output.pdf", "wb") { |file| file.write(response.body) }
      puts "File downloaded and saved to output.pdf"
  else
      puts "Download failed. Status code: #{response.code}"
  end
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.w3.org%2FWAI%2FER%2Ftests%2Fxhtml%2Ftestfiles%2Fresources%2Fpdf%2Fdummy.pdf" -o output.pdf
  ```
</CodeGroup>

<Warning>
  Always save the response as binary. Use `response.content` in Python (not `response.text`) and binary write mode (`"wb"`) when opening the output file. Writing binary content as text will corrupt the file.
</Warning>

## When to use file downloads

File download is the right approach when the content you need is a binary asset rather than scraped HTML. It works well for:

* **Product images**: download images directly from e-commerce product pages or media libraries.
* **PDFs and documents**: collect downloadable reports, whitepapers, manuals, or invoices.
* **Data exports**: fetch CSV, JSON, or XML files served at direct URLs.
* **Media assets**: download audio or video files linked directly on a page.

<Note>For files that require navigating a page and triggering a download interaction, use [JS Instructions](/fetch/features/js-instructions) to simulate the user action before the file is returned.</Note>

## File size limits

Zenrows enforces a maximum file size per request to ensure stable performance across all plans. If the file you are trying to download exceeds your plan's limit, the API returns a `413 Content Too Large` error.

<Info>You can find the file size limits for each plan on our [Pricing Documentation](https://docs.zenrows.com/first-steps/pricing).</Info>

## Best practices

**Save as binary, not text:** <br />
All file downloads return binary content. Always write the response body in binary mode (for example, `"wb"` mode in Python and Ruby, `Buffer` in Node.js, or `FileOutputStream` in Java). Writing binary content as text will produce a corrupt file.

**Enable `js_render` for pages that load content dynamically:** <br />
If the page loads content dynamically after the initial render, enable `js_render` to ensure the full page is visible before the file is downloaded.

<Tip>Find more details on our [JS Rendering Documentation](/fetch/features/js-rendering).</Tip>

**Enable `premium_proxy` for pages that are blocked by anti-bot systems:** <br />
If the page is blocked by an anti-bot system, enable `premium_proxy` to bypass the block.

<Tip>Find more details on our [Premium Proxy Documentation](/fetch/features/premium-proxy).</Tip>

**Check the status code before saving:** <br />
Always verify that the response returned a `200` status code before writing to disk. Saving an error response as a file will produce an unreadable or corrupt output.

**Use direct file URLs where possible:** <br />
Direct file URLs (ending in `.pdf`, `.jpg`, `.csv`, etc.) are the most reliable download method. If you need to navigate to a page first to trigger the download, use JS Instructions to simulate the click and capture the resulting file.

**Credits are charged on successful responses:** <br />
File download requests are charged when the API returns a `200` status code, regardless of what the file contains. Test with a small set of URLs before running large download batches.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The downloaded file is corrupt or cannot be opened">
    You are likely writing the response as text instead of binary. Make sure you are using binary write mode in your language of choice (`"wb"` in Python and Ruby, `Buffer` in Node.js, `FileOutputStream` in Java) and writing the raw response body rather than a text representation of it. This applies to all binary file types including PDFs, images, and archives.
  </Accordion>

  <Accordion title="The API returns HTML instead of the file">
    This usually means the target URL returns a webpage rather than the file directly. Check that the URL you are passing points directly to the file (for example, ending in `.pdf` or `.jpg`) rather than to a page that contains a download button or link. For page-triggered downloads, use JS Instructions to simulate the interaction.
  </Accordion>

  <Accordion title="I receive a 413 Content Too Large error">
    The file exceeds your plan's maximum file size limit. Check the [Pricing Documentation](https://docs.zenrows.com/first-steps/pricing) for the limit on your plan. For large files, consider whether you can access a compressed version or a lower-resolution asset from the same source.
  </Accordion>

  <Accordion title="The download is triggered by a button click and Zenrows returns nothing">
    If the file download is initiated by a user action (a button or link click), you need to use JS Instructions to simulate that interaction. If the download then prompts the user to choose a save location or requires additional confirmation, Zenrows cannot capture it. In that case, use [Browser Sessions](/browser-sessions/introduction) for full browser session control.
  </Accordion>
</AccordionGroup>

## Pricing

File download requests are included at **no additional cost** with all Zenrows requests. You only pay extra for JS Render and Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## FAQ (Frequently Asked Questions)

<Accordion title="What file types can Zenrows download?">
  Zenrows can download any file type that is returned directly in the HTTP response body, including PDFs, images (PNG, JPEG, WebP, GIF), CSVs, ZIPs, audio files, and video files. The API returns the raw binary content regardless of file type.
</Accordion>

<Accordion title="Do I need js_render=true to download files?">
  No, not for direct file URLs. If the URL points directly to a file, Zenrows fetches and returns it without needing a browser render. You only need `js_render=true` if the file download is triggered by a JavaScript interaction on a page.
</Accordion>

<Accordion title="Can I download multiple files in one request?">
  No. Each Zenrows request targets a single URL and returns a single response. To download multiple files, send a separate request for each file URL. For high-volume downloads, use concurrent requests within your plan's concurrency limits.
</Accordion>

<Accordion title="Can I download files that require authentication?">
  Yes. You can pass custom headers (such as `Authorization` or session cookies) using the `custom_headers` parameter to authenticate with the target server before the file is returned. Another method is to use the JS Instructions to simulate the user action that triggers the download.

  <Tip>Find more details on our [JS Instructions Documentation](/fetch/features/js-instructions).</Tip>
</Accordion>

<Accordion title="What happens if the file exceeds the size limit?">
  The API returns a `413 Content Too Large` error. The request is still counted toward your usage. Check your plan's file size limit in the [Pricing Documentation](https://docs.zenrows.com/first-steps/pricing) before downloading large files at scale.
</Accordion>


# Custom Headers
Source: https://docs.zenrows.com/fetch/features/headers

Set custom HTTP headers in your Zenrows scraping requests to manage cookies, user agents, authorization tokens, and browser fingerprinting.

Custom headers let you modify HTTP request headers (the information your browser sends to websites) to control how target servers respond to your scraping requests. You can simulate different browser behaviors, maintain session continuity, and bypass website restrictions.

<Note>Set `custom_headers=true` in your request parameters to enable custom header functionality while maintaining Zenrows' automatic header optimization.</Note>

## How custom headers work

When you make a request to a website, your browser automatically includes dozens of HTTP headers that provide information about the request context, browser capabilities, and user preferences. These headers help servers understand:

* **What type of content to return** (HTML, JSON, XML)
* **Where the request originated from** (referrer information)
* **What the browser can handle** (encoding, language preferences)
* **Authentication credentials** (cookies, tokens)
* **User preferences** (language, timezone)

## Enabling custom headers

To use custom headers, set the `custom_headers` parameter to `true` in your API request. This enables your custom headers while Zenrows continues to manage sensitive browser-specific ones automatically.

### Basic custom header usage

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'custom_headers': 'true',
  }
  headers = {
      'Referer': 'https://google.com',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params, headers=headers)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      headers: {
          'Referer': 'https://google.com',
      },
      params: {
          'url': url,
          'apikey': apikey,
          'custom_headers': 'true',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&custom_headers=true";
          String response = Request.get(apiUrl)
                  .addHeader("Referer", "https://google.com")
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&custom_headers=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_HTTPHEADER, [
      'Referer: https://google.com',
  ]);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&custom_headers=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Add("Referer", "https://google.com")
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&custom_headers=true')
  headers = {
      "Referer": "https://google.com",
  }
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, headers)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl -H "Referer: https://google.com" "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&custom_headers=true"
  ```
</CodeGroup>

<Warning>
  **Zenrows automatically manages browser-environment headers** to ensure consistency, high success rates, and protection against anti-bot detection systems. These automatically managed headers include:

  **Browser fingerprinting headers**

  * **`User-Agent`** - Browser identification and capabilities
  * **`Sec-Ch-Ua`** - Client hints for browser information
  * **`Sec-Ch-Ua-Mobile`** - Mobile device indication
  * **`Sec-Ch-Ua-Platform`** - Operating system information

  **Request context headers**

  * **`Accept-Encoding`** - Supported compression methods
  * **`Accept-Language`** - Language preferences
  * **`Sec-Fetch-Mode`** - Request mode (navigate, cors, same-origin)
  * **`Sec-Fetch-Site`** - Request site relationship
  * **`Sec-Fetch-User`** - User activation indication
  * **`Sec-Fetch-Dest`** - Request destination type

  **Connection headers**

  * **`Connection`** - Connection management preferences
  * **`Upgrade-Insecure-Requests`** - HTTPS upgrade preferences
  * **`Cache-Control`** - Caching behavior

  These headers are tightly coupled with browser behavior and cannot be customized. **Attempts to override them will be ignored** to maintain optimal success rates and prevent anti-bot detection.
</Warning>

## Common header use cases

### Referrer simulation

Control where the request appears to originate from by setting the `Referer` header.

```python Python theme={"dark"}
# pip install requests
import requests

url = 'https://www.amazon.com/dp/B00HPAIY6A'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
    'custom_headers': 'true',
}
headers = {
    'Referer': 'https://www.amazon.com/s?k=brazilian+coffee',  # Simulate coming from search results
}

response = requests.get('https://api.zenrows.com/v1/', params=params, headers=headers)
print(response.text)
```

**Benefits of referrer simulation:**

* Access content that's restricted to specific traffic sources
* Bypass anti-bots that allow access from search engines
* Receive personalized content based on traffic source
* Avoid bot detection that checks for missing referrers

### Session management with cookies

Maintain authentication state across requests by including session cookies in the `Cookie` header.

```python Python theme={"dark"}
# pip install requests
import requests

url = 'https://protected-site.com/dashboard'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
    'custom_headers': 'true',
}
headers = {
    'Cookie': 'session_id=abc123; user_token=xyz789; preferences=theme=dark', # Format cookies as a proper cookie string
}

response = requests.get('https://api.zenrows.com/v1/', params=params, headers=headers)
print(response.text)
```

<Warning>
  Session cookies often contain encrypted IP addresses and browser fingerprinting data that must match the actual request environment. Since Zenrows rotates IP addresses by default and automatically manages browser fingerprinting, using cookies obtained outside of Zenrows may cause authentication failures or trigger anti-bot detection.

  When websites detect mismatched session data (such as a cookie containing one IP address while the request comes from a different IP), they typically invalidate the session or block the request entirely.

  For reliable session management, either:

  * Obtain cookies through Zenrows by checking the response headers to ensure consistency (`Zr-Cookies`)
  * Use the [`session_id`](/fetch/features/other#session-id) parameter to maintain the same IP across requests

  Learn more about session management in our [Can I Maintain Session/IP Between Requests](/fetch/faq#can-i-maintain-sessions-ips-between-requests) FAQ section.
</Warning>

### Language, localization and currency

Request content in specific languages or regions using website-specific cookies.

Zenrows automatically manages the `Accept-Language` header, but some websites use specific cookies to control currency, language, or localization settings.

Consider the website `stockx.com`, which shows different currencies based on the `stockx_selected_region` cookie. Here's how to find and use such cookies:

<Steps>
  <Step title="Find localization cookies in DevTools">
    1. Open the target website in an incognito browser tab
    2. Open DevTools and go to the Application tab
    3. Navigate to Cookies and select the website URL
    4. Look for cookies that indicate a relation to language, localization, or currency
    5. Test by changing the cookie value in DevTools and refreshing the page

    <img alt="Stockx cookie currency example" />
  </Step>

  <Step title="Include the cookie in your request">
    Add the localization cookie to your `Cookie` header. Use geolocation that matches your localization settings for consistency.

    ```python Python theme={"dark"}
    # pip install requests
    import requests

    url = 'https://stockx.com/asics-gel-nyc-cream'
    apikey = 'YOUR_ZENROWS_API_KEY'
    params = {
        'url': url,
        'apikey': apikey,
        'js_render': 'true',
        'premium_proxy': 'true',
        'proxy_country': 'de', # Match geolocation to cookie setting
        'custom_headers': 'true',
    }
    headers = {
        'Cookie': 'stockx_selected_region=DE', # Set currency to EUR and Germany locale
    }
    response = requests.get('https://api.zenrows.com/v1/', params=params, headers=headers)
    print(response.text)
    ```
  </Step>
</Steps>

## Best practices

### Start with minimal headers

Only add custom headers when necessary. Unnecessary headers can increase detection risk, especially with high-volume scraping. Start with no custom headers and add them only when you encounter specific blocking or need particular functionality.

### Analyze target behavior first

Before adding custom headers, check what the target website expects by examining real browser requests:

<Note>
  **Research workflow:**

  1. Open the target URL in an **incognito/private browser tab**
  2. Open **DevTools** (`F12`) and go to the **Network tab**
  3. Reload the page and examine the request headers
  4. For cookie-based functionality, check the **Application tab** in DevTools
  5. Only include headers that are essential for your use case
</Note>

### Header rotation for multiple requests

Vary headers across requests to avoid creating detectable patterns. Websites can detect if all requests come from the same referrer and may start blocking those requests. Rotating referrers helps simulate more natural browsing behavior from different traffic sources.

```python Python expandable theme={"dark"}
import random
import time
import requests

def get_random_headers():
    # Generate randomized headers for each request

    referrers = [
        'https://www.google.com',
        'https://www.bing.com',
        'https://duckduckgo.com',
        'https://www.yahoo.com',
        'https://twitter.com',
    ]
    
    return {
        'Referer': random.choice(referrers),
    }

def scrape_with_rotation(urls):
    # Scrape multiple URLs with randomized headers

    results = []
    
    for url in urls:
        headers = get_random_headers()
        
        response = requests.get('https://api.zenrows.com/v1/',
            params={
                'url': url,
                'apikey': 'YOUR_ZENROWS_API_KEY',
                'custom_headers': 'true',
            },
            headers=headers
        )
        
        results.append({
            'url': url,
            'content': response.text,
            'headers_used': headers,
        })
        
        # Add delay between requests to avoid rate limiting
        time.sleep(random.uniform(1, 3))
    
    return results

# Example usage
urls_to_scrape = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3',
]

results = scrape_with_rotation(urls_to_scrape)
for result in results:
    print(f"Scraped {result['url']} using referrer: {result['headers_used']['Referer']}")
```

## Troubleshooting

### Common header issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Headers ignored | `custom_headers=true` not set | Add `custom_headers=true` to request parameters |
| Authentication fails | Incorrect cookie format | Format cookies as proper cookie string: `key1=value1; key2=value2` |
| 403 Forbidden errors | Missing required headers | Check target website's requirements and include necessary headers |
| Localization not working | Wrong cookie format or value | Use DevTools to find correct cookie names and values |
| Session expired quickly | IP mismatch in cookies | Use `session_id` or obtain cookies through Zenrows and reuse them |
| Headers not taking effect | Conflicting header values | Ensure headers are consistent and don't contradict each other |
| Website rate limiting triggered | Same headers for all requests | Rotate headers across requests to simulate natural browsing |

### Debugging header problems

When headers aren't working as expected:

<Steps>
  <Step title="Check for referrer-based access control">
    Some websites only allow access to specific pages if you first visit another page. This is because the initial page sets cookies necessary for subsequent pages.

    Example: Amazon product page accessed from search results

    ```python Python theme={"dark"}
    # pip install requests
    import requests

    url = 'https://www.amazon.com/dp/B00HPAIY6A'
    apikey = 'YOUR_ZENROWS_API_KEY'
    params = {
        'url': url,
        'apikey': apikey,
        'custom_headers': 'true',
    }
    headers = {
        'Referer': 'https://www.amazon.com/s?k=brazilian+coffee', # Simulate search page origin
    }
    response = requests.get('https://api.zenrows.com/v1/', params=params, headers=headers)
    print(response.text)
    ```

    Alternatively, use `https://www.google.com` as a referrer to simulate traffic from search engines.
  </Step>

  <Step title="Check for header conflicts">
    Ensure headers don't contradict each other:

    ```python theme={"dark"}
    # BAD: Conflicting content type expectations
    conflicting_headers = {
        'Accept': 'application/json',
        'Content-Type': 'text/html',  # Conflicts with Accept
    }

    # GOOD: Consistent headers
    consistent_headers = {
        'Accept': 'application/json',
        'Content-Type': 'application/json',
    }
    ```
  </Step>
</Steps>

## Pricing

The `custom_headers` parameter doesn't increase the request cost. The only features who increase request costs are the Premium Proxy and JavaScript Render.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="What are custom headers and why would I use them?">
  Custom headers allow you to modify HTTP request headers like `Referer`, `Accept`, `Cookie`, or `Authorization` to control how the target server perceives your request. This is useful for handling authentication, simulating specific browser behaviors, requesting particular content types, bypassing certain restrictions, and maintaining session continuity across requests.
</Accordion>

<Accordion title="Which headers are automatically managed by Zenrows?">
  Zenrows automatically manages browser-environment headers, including `User-Agent`, `Accept-Encoding`, `Sec-Ch-Ua`, `Sec-Fetch-Mode`, `Sec-Fetch-Site`, `Sec-Fetch-User`, and other browser fingerprinting headers. These are optimized for high success rates and anti-bot protection, and cannot be customized to maintain consistency and reliability.
</Accordion>

<Accordion title="Why can't I override certain headers like User-Agent?">
  Headers like `User-Agent`, `Sec-Ch-Ua`, and `Accept-Encoding` are closely tied to browser behavior and fingerprinting. Zenrows manages these automatically to ensure they match the browser environment being simulated, preventing detection by anti-bot systems that check for inconsistencies between headers and actual browser capabilities.
</Accordion>

<Accordion title="What happens if I don't set custom_headers=true?">
  Without `custom_headers=true`, any custom headers you include in your request will be ignored. Zenrows will use only its automatically managed headers. You must explicitly enable custom headers to have your additional headers included in the request to the target website.
</Accordion>

<Accordion title="Can I use custom headers with JavaScript rendering?">
  Yes, custom headers work with both static requests and JavaScript rendering (`js_render=true`). The headers are applied to the initial page request and any subsequent requests made during JavaScript execution, providing consistent header behavior throughout the rendering process.
</Accordion>

<Accordion title="What should I do if my custom headers aren't working?">
  First, verify that `custom_headers=true` is set in your parameters. Then check that your header format is correct and matches what the target website expects. Use debugging tools like `httpbin.io/anything` to verify that your headers are being sent correctly. Compare your headers with those sent by a real browser using developer tools, and ensure you're not attempting to set headers that Zenrows manages automatically.
</Accordion>

<Accordion title="Can I rotate headers across multiple requests?">
  Yes, you can vary headers across requests to avoid creating detectable patterns. Generate different combinations of referrers, language preferences, and other allowed headers for each request. This helps simulate more natural browsing behavior and can improve success rates when scraping multiple pages or making repeated requests to the same site.
</Accordion>


# JavaScript Instructions
Source: https://docs.zenrows.com/fetch/features/js-instructions

Automate page interactions like clicking buttons, filling forms, scrolling, and running custom JavaScript during headless browser rendering.

JavaScript Instructions enable you to interact with web pages dynamically by automating user actions like clicking buttons, filling forms, waiting for content to load, and executing custom JavaScript code. This powerful feature allows you to scrape content that requires user interaction or appears after specific actions are performed.

These instructions are essential for modern web scraping because many websites load content dynamically, hide information behind user interactions, or require form submissions to access data. With JavaScript Instructions, you can automate these interactions to extract the complete content you need.

<Note>JavaScript Instructions require `js_render=true` to function, as they operate within the browser environment during page rendering.</Note>

## How JavaScript Instructions work

JavaScript Instructions execute sequentially within a real browser environment, simulating genuine user interactions. Each instruction waits for the previous one to complete before executing, ensuring reliable automation of complex user workflows.

The process works as follows:

1. **Page loads** - The browser renders the initial page content
2. **Wait/Wait\_for parameters execute** - Global wait conditions are processed first
3. **JavaScript Instructions execute** - Each instruction runs in order, waiting for completion
4. **DOM updates** - The page responds to interactions, potentially loading new content
5. **Final capture** - The complete rendered page is captured after all instructions finish

This sequential execution ensures that dynamic content has time to load and that user interactions trigger the expected page changes.

<Warning>JavaScript Instructions have a **40-second timeout limit**. If your instruction sequence takes longer than 40 seconds to complete (due to many interactions, slow page responses, or redirects), the execution will stop and return whatever content is currently visible on the page. For complex workflows that may exceed this limit, consider using our [Browser Sessions](/browser-sessions/introduction) which provides up to 15 minutes of execution time and more advanced automation capabilities.</Warning>

## Basic usage

JavaScript Instructions are provided as a JSON array where each instruction is an object specifying the action and its parameters:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.example.com'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'js_instructions': """[{"click":".button-selector"},{"wait":500}]""",
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.example.com';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'js_render': 'true',
          'js_instructions': `[{"click":".button-selector"},{"wait":500}]`,
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.example.com&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.button-selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%255D";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.example.com&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.button-selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%255D');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.example.com&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.button-selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%255D", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.example.com&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.button-selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%255D')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```csharp C# theme={"dark"}
  using RestSharp;
  namespace TestApplication {
      class Test {
          static void Main(string[] args) {
              var client = new RestClient("https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.example.com&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.button-selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%255D");
              var request = new RestRequest();

              var response = client.Get(request);
              Console.WriteLine(response.Content);
          }
      }
  }
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.example.com&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.button-selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%255D"
  ```
</CodeGroup>

<Tip>Use the [Zenrows Playground](https://app.zenrows.com/fetch) to create and test your JavaScript Instructions visually, or use an <a href="https://www.url-encode-decode.com/">Online URL Encoder</a> to properly encode your instructions for API requests.</Tip>

## Available instructions

### Click interactions

Simulates clicking on page elements like buttons, links, or interactive areas. This is one of the most fundamental interactions for web automation, enabling you to trigger actions that reveal hidden content, navigate through interfaces, or activate dynamic functionality.

```json JSON theme={"dark"}
[
    {"click": ".read-more-button"},
    {"click": "#submit-btn"},
    {"click": "button[data-action='load-more']"}
]
```

**Common use cases**

* Expanding collapsed content sections to access full article text
* Navigating through pagination to scrape multiple pages of results
* Triggering modal dialogs or popups that contain additional information
* Activating dropdown menus to access navigation options
* Loading additional content dynamically (infinite scroll triggers)
* Accepting cookie consent banners or terms of service

**Best practices**

* Always wait for elements to be clickable before clicking
* Use specific selectors to avoid clicking wrong elements
* Combine with `wait_for` to handle dynamic content loading

  ```json JSON theme={"dark"}
  # Example: Click through pagination to scrape multiple pages

  [
      {"wait_for": ".product-list"},           # Wait for initial content
      {"click": ".pagination .next-page"},     # Click next page
      {"wait_for": ".product-list"},           # Wait for new content to load
      {"click": ".pagination .next-page"},     # Click next page again
      {"wait": 1000}                           # Final wait for stability
  ]
  ```

### Wait for selector

Pauses execution until a specific element appears in the DOM. This instruction is crucial for handling asynchronous content loading, which is common in modern web applications that use AJAX, React, Vue, or other dynamic frameworks.

```json JSON theme={"dark"}
[
    {"wait_for": ".dynamic-content"},
    {"wait_for": "#ajax-loaded-section"},
    {"wait_for": "[data-loaded='true']"}
]
```

**When to use**

* Content loads asynchronously via AJAX or fetch requests
* Elements appear after animations or CSS transitions
* Waiting for user-triggered content to become available
* Ensuring forms are fully rendered before interaction
* After clicking buttons that trigger content loading
* When dealing with single-page applications (SPAs)

<Warning>If the selector doesn't appear within the timeout period (default: 10 seconds), the instruction will fail and execution will continue to the next instruction. This contributes to the overall 40-second timeout limit for all JavaScript Instructions.</Warning>

### Wait for specific duration

Pauses execution for a fixed amount of time in milliseconds. While less precise than `wait_for`, this instruction is useful when you need to accommodate processes that don't have visible indicators or when you need to ensure stability after rapid interactions.

```json JSON theme={"dark"}
[
    {"wait": 1000},    # Wait 1 second
    {"wait": 5000},    # Wait 5 seconds
    {"wait": 500}      # Wait 0.5 seconds
]
```

**Use cases**

* Allowing animations to complete before taking screenshots
* Giving time for slow-loading content when no loading indicator exists
* Preventing rate limiting by spacing requests appropriately
* Ensuring page stability after rapid-fire interactions
* Waiting for third-party widgets or ads to load
* Accommodating server processing time for form submissions

```json JSON theme={"dark"}
# Example: Handle slow-loading content with strategic waits

[
    {"click": ".load-data-button"},    # Trigger data loading
    {"wait": 3000},                    # Wait for server response
    {"wait_for": ".data-table"},       # Wait for table to appear
    {"wait": 1000}                     # Additional stability wait
]
```

<Warning>The maximum allowed wait time for a single `wait` instruction is 10 seconds, and the total combined duration of all `wait` instructions cannot exceed 30 seconds. For example, you could use three `{"wait": 10000}` instructions (10 seconds each) but not four. This 30-second limit contributes to the overall 40-second timeout for the entire JavaScript Instructions sequence.</Warning>

### Wait for browser events

Waits for specific browser events to occur, providing more sophisticated timing control based on actual browser state rather than arbitrary time delays. This is particularly useful for ensuring that all network activity has completed or that the page has reached a stable state.

```json JSON theme={"dark"}
[
    {"wait_event": "networkidle"},        # Wait until network is idle
    {"wait_event": "networkalmostidle"},  # Wait until network is almost idle
    {"wait_event": "load"},               # Wait for page load event
    {"wait_event": "domcontentloaded"}    # Wait for DOM to be ready
]
```

**Event descriptions**

* `networkidle` - No network requests for 500ms (ideal for SPAs with API calls)
* `networkalmostidle` - No more than 2 network requests for 500ms (less strict timing)
* `load` - Page load event fired (all resources including images loaded)
* `domcontentloaded` - DOM parsing completed (faster than load event)

**Use cases**

* Ensure all API calls and resource loading have completed
* Wait for specific browser lifecycle events
* Avoid unnecessary waiting by responding to actual browser state
* Use browser events as more accurate timing signals than fixed delays

### Fill input fields

Populates form fields with specified values, enabling automation of data entry tasks. This instruction is essential for logging into websites, submitting forms, or providing input data that triggers dynamic content loading.

```json JSON theme={"dark"}
[
    {"fill": ["#username", "john_doe"]},
    {"fill": ["input[name='email']", "user@example.com"]},
    {"fill": [".search-box", "search query"]}
]
```

**Supported input types**

* Text inputs (single-line text fields)
* Email fields (with built-in validation)
* Password fields (securely handled)
* Search boxes (often trigger autocomplete)
* Textarea elements (multi-line text)
* Number inputs (numeric data entry)

**Common applications**

* Login automation for accessing protected content
* Search form submission to find specific content
* Filter application to narrow down results
* Contact form completion for lead generation
* Registration form automation for account creation
* Configuration forms for customizing page content

```json JSON theme={"dark"}
# Example: Complete a contact form

[
    {"wait_for": "#contact-form"},                   # Wait for form to load
    {"fill": ["#name", "John Smith"]},               # Fill name field
    {"fill": ["#email", "john@example.com"]},        # Fill email field
    {"fill": ["#message", "Hello, I need help..."]}, # Fill message
    {"click": "#submit-button"},                     # Submit form
    {"wait": 10000}                                  # Wait 10 seconds
]
```

### Checkbox interactions

Check or uncheck checkbox and radio button elements, allowing you to select options, agree to terms, or configure settings. These interactions are crucial for forms that require user consent or option selection.

```json JSON theme={"dark"}
[
    {"check": "#agree-terms"},           # Check a checkbox
    {"uncheck": "#newsletter-signup"},   # Uncheck a checkbox
    {"check": "input[name='payment'][value='credit']"}  # Select radio button
]
```

**Use cases**

* Accepting cookie consent or privacy policies
* Selecting payment methods or shipping options
* Configuring notification preferences
* Filtering product catalogs by features
* Agreeing to terms of service during registration
* Enabling optional services or add-ons

<Warning>Using `check` on an already checked element will not uncheck it. Use `uncheck` specifically to deselect elements. This prevents accidental state changes in your automation.</Warning>

### Select dropdown options

Choose options from dropdown menus by their value, enabling selection from predefined lists of options. This is essential for forms that use select elements for categories, locations, or other structured data.

```json JSON theme={"dark"}
[
    {"select_option": ["#country-select", "USA"]},
    {"select_option": [".size-dropdown", "large"]},
    {"select_option": ["select[name='category']", "electronics"]}
]
```

**Important notes**

* The second parameter must match the value attribute of the option, not the displayed text
* Use the actual value from the HTML, which may differ from what users see
* Works with both single and multiple select elements
* Triggers change events that may load additional content

**Common scenarios**

* Selecting countries for shipping calculations
* Choosing product categories for filtered browsing
* Setting language preferences for localized content
* Selecting time zones or date formats
* Choosing sorting options for search results
* Configuring display preferences (items per page, view type)

### Vertical scrolling

Scroll the page vertically by a specified number of pixels, essential for triggering content that loads based on scroll position or for ensuring elements are visible before interaction.

```json JSON theme={"dark"}
[
    {"scroll_y": 1000},    # Scroll down 1000 pixels
    {"scroll_y": -500},    # Scroll up 500 pixels
    {"scroll_y": 2000}     # Scroll down 2000 pixels
]
```

**Common applications**

* Social media feeds with scroll loading
* E-commerce sites with lazy-loaded product images
* News sites with continuous article loading
* Search results that load more items on scroll
* Long-form content with progressive disclosure
* Image galleries with scroll-triggered loading

### Horizontal scrolling

Scroll the page horizontally by a specified number of pixels, useful for content that extends beyond the viewport width or for navigating horizontal carousels and galleries.

```json JSON theme={"dark"}
[
    {"scroll_x": 800},     # Scroll right 800 pixels
    {"scroll_x": -400},    # Scroll left 400 pixels
    {"scroll_x": 1200}     # Scroll right 1200 pixels
]
```

**Use cases**

* Product image carousels on e-commerce sites
* Data tables with many columns
* Horizontal navigation menus
* Timeline or calendar interfaces
* Wide charts or graphs
* Panoramic image viewers

### Execute custom JavaScript (`evaluate`)

Run arbitrary JavaScript code within the page context, providing unlimited flexibility for complex interactions or page modifications that aren't covered by standard instructions.

```json JSON theme={"dark"}
[
    {"evaluate": "document.querySelector('.modal').style.display = 'none';"},
    {"evaluate": "window.scrollTo(0, document.body.scrollHeight);"},
    {"evaluate": "document.querySelector('.load-more').scrollIntoView();"}
]
```

**Common JavaScript patterns**

```javascript JavaScript theme={"dark"}
// Scroll to specific element
"document.querySelector('.target-element').scrollIntoView();"

// Modify page styling
"document.body.style.backgroundColor = '#ffffff';"

// Trigger custom events
"document.querySelector('.button').dispatchEvent(new Event('click'));"

// Access page data
"window.dataLayer = window.dataLayer || [];"

// Remove overlays
"document.querySelectorAll('.overlay').forEach(el => el.remove());"

// Trigger infinite scroll
"window.scrollTo(0, document.body.scrollHeight);"

// Click multiple elements
"document.querySelectorAll('.expand-button').forEach(btn => btn.click());"
```

**Advanced use cases**

* Removing cookie banners or overlay advertisements
* Triggering complex JavaScript functions specific to the site
* Modifying page state to reveal hidden content
* Collecting data from JavaScript variables
* Simulating complex user interactions
* Bypassing client-side restrictions

### CAPTCHA solving

Automatically solve CAPTCHAs using integrated solving services, enabling automation of forms and processes that are protected by CAPTCHA challenges.

CAPTCHA solving requires a [2Captcha](https://www.zenrows.com/go/2captcha) API key configured in your [Zenrows Integration Settings](https://app.zenrows.com/settings/integrations).

```json JSON theme={"dark"}
[
    {"solve_captcha": {"type": "recaptcha"}},
    {"solve_captcha": {"type": "cloudflare_turnstile"}},
    {"solve_captcha": {"type": "recaptcha", "options": {"solve_inactive": true}}}
]
```

**Supported CAPTCHA types**

* **reCAPTCHA v2** - Standard checkbox and image challenges
* **reCAPTCHA v3** - Invisible background verification
* **Cloudflare Turnstile** - Cloudflare's CAPTCHA system

<Tip>
  To ensure the CAPTCHA is solved before proceeding, add `wait` instructions **before** and **after** the CAPTCHA-solving step, allowing time for the CAPTCHA to load and be resolved.

  ```javascript theme={"dark"}
  [
      {"wait": 3000}, // Wait for 3 seconds to allow the page to load
      {"solve_captcha": {"type": "recaptcha"}},
      {"wait": 2000} // Wait 2 seconds to confirm CAPTCHA resolution
  ]
  ```
</Tip>

### Working with iframes

Standard instructions don't work inside iframe elements due to browser security restrictions. Iframes create isolated contexts that require special handling to access their content and interact with their elements.

#### Frame instructions

All frame instructions require specifying the iframe selector as the first parameter

```json JSON theme={"dark"}
[
    {"frame_click": ["#payment-iframe", ".submit-button"]},
    {"frame_wait_for": ["#content-iframe", ".loaded-content"]},
    {"frame_fill": ["#form-iframe", "#email", "user@example.com"]},
    {"frame_check": ["#options-iframe", "#agree-checkbox"]},
    {"frame_uncheck": ["#settings-iframe", "#notifications"]},
    {"frame_select_option": ["#dropdown-iframe", "#country", "USA"]},
    {"frame_evaluate": ["iframe-name", "document.body.style.color = 'red';"]}
]
```

#### Revealing iframe content

Extract content from iframes for processing:

```json JSON theme={"dark"}
[
    {"frame_reveal": "#payment-iframe"}
]
```

This creates a hidden div element with the iframe content encoded in base64:

```php HTML theme={"dark"}
<div class="iframe-content-element" style="display: none;" data-id="#payment-iframe">
    <!-- Base64 encoded iframe content -->
</div>
```

<Warning>For security reasons, iframe content isn't included in the standard response. Use frame\_reveal to explicitly extract iframe content when needed.</Warning>

### Using XPath selectors

In addition to CSS selectors, you can use XPath expressions for more precise element targeting:

```json JSON theme={"dark"}
[
    {"click": "//button[text()='Submit']"},
    {"wait_for": "//div[@class='content' and @data-loaded='true']"},
    {"fill": ["//input[@placeholder='Enter email']", "user@example.com"]}
]
```

**XPath advantages**

* **Text-based selection** - Select elements by their text content
* **Complex conditions** - Use logical operators and functions
* **Hierarchical navigation** - Navigate parent/child relationships easily
* **Attribute matching** - Complex attribute-based selection

## Debugging JavaScript Instructions

Enable detailed execution reporting by adding `json_response=true` to your request:

```python Python theme={"dark"}
params = {
    'js_render': 'true',
    'js_instructions': """[{"click":".button-selector"},{"wait":500}]""",
    'json_response': 'true'  # Enable debugging information
}
```

### Understanding the debug report

The debug report provides comprehensive execution details:

```json JSON expandable theme={"dark"}
{
    "instructions_duration": 5041,
    "instructions_executed": 4,
    "instructions_succeeded": 3,
    "instructions_failed": 1,
    "instructions": [
        {
            "instruction": "wait_for",
            "params": {"selector": ".content"},
            "success": true,
            "duration": 1200
        },
        {
            "instruction": "click",
            "params": {"selector": ".button"},
            "success": true,
            "duration": 150
        },
        {
            "instruction": "wait_for",
            "params": {"selector": ".missing-element"},
            "success": false,
            "duration": 30000
        },
        {
            "instruction": "fill",
            "params": {"selector": "#input", "value": "test"},
            "success": true,
            "duration": 80
        }
    ]
}
```

## Common use cases and workflows

<Tabs>
  <Tab title="Multi-step form automation">
    <CodeGroup>
      ```python Python theme={"dark"}
      def automate_registration_form(signup_url, user_data):
      # Complete multi-step registration process

      instructions = [
          # Step 1: Basic information
          {"wait_for": "#registration-form"},
          {"fill": ["#first-name", user_data['first_name']]},
          {"fill": ["#last-name", user_data['last_name']]},
          {"fill": ["#email", user_data['email']]},
          {"click": ".next-step"},
          
          # Step 2: Account details
          {"wait_for": "#account-details"},
          {"fill": ["#username", user_data['username']]},
          {"fill": ["#password", user_data['password']]},
          {"fill": ["#confirm-password", user_data['password']]},
          {"click": ".next-step"},
          
          # Step 3: Preferences
          {"wait_for": "#preferences"},
          {"check": "#newsletter-signup"},
          {"select_option": ["#country", user_data['country']]},
          {"click": ".submit-registration"},
          
          # Step 4: Confirmation
          {"wait_for": ".registration-success"},
      ]

      return requests.get('https://api.zenrows.com/v1/', params={
          'url': signup_url,
          'apikey': 'YOUR_ZENROWS_API_KEY',
          'js_render': 'true',
          'js_instructions': json.dumps(instructions),
      })
      ```

      ```javascript Node.js theme={"dark"}
      async function automateRegistrationForm(signupUrl, userData) {
          /**
           * Complete multi-step registration process
           */
          const instructions = [
              // Step 1: Basic information
              {"wait_for": "#registration-form"},
              {"fill": ["#first-name", userData.firstName]},
              {"fill": ["#last-name", userData.lastName]},
              {"fill": ["#email", userData.email]},
              {"click": ".next-step"},
              
              // Step 2: Account details
              {"wait_for": "#account-details"},
              {"fill": ["#username", userData.username]},
              {"fill": ["#password", userData.password]},
              {"fill": ["#confirm-password", userData.password]},
              {"click": ".next-step"},
              
              // Step 3: Preferences
              {"wait_for": "#preferences"},
              {"check": "#newsletter-signup"},
              {"select_option": ["#country", userData.country]},
              {"click": ".submit-registration"},
              
              // Step 4: Confirmation
              {"wait_for": ".registration-success"},
          ];
          
          try {
              const response = await axios({
                  url: 'https://api.zenrows.com/v1/',
                  method: 'GET',
                  params: {
                      'url': signupUrl,
                      'apikey': 'YOUR_ZENROWS_API_KEY',
                      'js_render': 'true',
                      'js_instructions': JSON.stringify(instructions),
                  }
              });
              
              return response.data;
          } catch (error) {
              console.error('Error automating registration:', error);
              throw error;
          }
      }
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Search and pagination">
    <CodeGroup>
      ```python Python theme={"dark"}
      def scrape_search_results(search_url, query, max_pages=3):
      # Search and navigate through multiple result pages

      instructions = [
          # Perform search
          {"wait_for": "#search-form"},
          {"fill": ["#search-input", query]},
          {"click": "#search-button"},
          {"wait_for": ".search-results"},
      ]

      # Navigate through pagination
      for page in range(2, max_pages + 1):
          instructions.extend([
              {"click": f".pagination a[data-page='{page}']"},
              {"wait_for": ".search-results"},
              {"wait": 1000},  # Allow page to stabilize
          ])

      return requests.get('https://api.zenrows.com/v1/', params={
          'url': search_url,
          'apikey': 'YOUR_ZENROWS_API_KEY',
          'js_render': 'true',
          'js_instructions': json.dumps(instructions),
      })
      ```

      ```javascript Node.js theme={"dark"}
      async function scrapeSearchResults(searchUrl, query, maxPages = 3) {
          /**
           * Search and navigate through multiple result pages
           */
          const instructions = [
              // Perform search
              {"wait_for": "#search-form"},
              {"fill": ["#search-input", query]},
              {"click": "#search-button"},
              {"wait_for": ".search-results"},
          ];
          
          // Navigate through pagination
          for (let page = 2; page <= maxPages; page++) {
              instructions.push(
                  {"click": `.pagination a[data-page='${page}']`},
                  {"wait_for": ".search-results"},
                  {"wait": 1000}  // Allow page to stabilize
              );
          }
          
          try {
              const response = await axios({
                  url: 'https://api.zenrows.com/v1/',
                  method: 'GET',
                  params: {
                      'url': searchUrl,
                      'apikey': 'YOUR_ZENROWS_API_KEY',
                      'js_render': 'true',
                      'js_instructions': JSON.stringify(instructions),
                  }
              });
              
              return response.data;
          } catch (error) {
              console.error('Error scraping search results:', error);
              throw error;
          }
      }

      // Usage examples
      async function runExamples() {
          try {
              // E-commerce example
              const productData = await scrapeProductWithShipping('https://shop.example.com/product/123');
              console.log('Product data scraped successfully');
              
              // Social media example
              const feedData = await scrapeSocialFeed('https://social.example.com/user/profile', 5);
              console.log('Social feed data scraped successfully');
              
              // Registration example
              const userData = {
                  firstName: 'John',
                  lastName: 'Doe',
                  email: 'john.doe@example.com',
                  username: 'johndoe',
                  password: 'SecurePass123',
                  country: 'USA'
              };
              const registrationResult = await automateRegistrationForm('https://app.example.com/signup', userData);
              console.log('Registration automated successfully');
              
              // Search example
              const searchResults = await scrapeSearchResults('https://marketplace.example.com', 'wireless headphones', 3);
              console.log('Search results scraped successfully');
              
          } catch (error) {
              console.error('Example execution failed:', error);
          }
      }

      // Run examples
      runExamples();
      ```
    </CodeGroup>
  </Tab>

  <Tab title="E-commerce product scraping">
    <CodeGroup>
      ```python Python theme={"dark"}
      def scrape_product_with_shipping(product_url):
       #Scrape product page with shipping calculation by zip code

      instructions = [
          {"wait_for": ".product-details"},                    # Wait for product to load
          {"click": ".size-option[data-size='large']"},       # Select size
          {"wait": 500},                                      # Wait for price update
          {"click": ".color-option[data-color='blue']"},      # Select color
          {"wait": 500},                                      # Wait for price update
          {"click": ".shipping-calculator"},                  # Open shipping calculator
          {"wait_for": "#zip-code-input"},                    # Wait for shipping form
          {"fill": ["#zip-code-input", "90210"]},            # Enter zip code
          {"click": ".calculate-shipping"},                   # Calculate shipping
          {"wait_for": ".shipping-options"},                  # Wait for shipping options
      ]

      return requests.get('https://api.zenrows.com/v1/', params={
          'url': product_url,
          'apikey': 'YOUR_ZENROWS_API_KEY',
          'js_render': 'true',
          'js_instructions': json.dumps(instructions),
      })
      ```

      ```javascript Node.js theme={"dark"}
      const axios = require('axios');

      async function scrapeProductWithShipping(productUrl) {
          /**
           * Scrape product page with shipping calculation by zip code
           */
          const instructions = [
              {"wait_for": ".product-details"},                    // Wait for product to load
              {"click": ".size-option[data-size='large']"},       // Select size
              {"wait": 500},                                      // Wait for price update
              {"click": ".color-option[data-color='blue']"},      // Select color
              {"wait": 500},                                      // Wait for price update
              {"click": ".shipping-calculator"},                  // Open shipping calculator
              {"wait_for": "#zip-code-input"},                    // Wait for shipping form
              {"fill": ["#zip-code-input", "90210"]},            // Enter zip code
              {"click": ".calculate-shipping"},                   // Calculate shipping
              {"wait_for": ".shipping-options"},                  // Wait for shipping options
          ];
          
          try {
              const response = await axios({
                  url: 'https://api.zenrows.com/v1/',
                  method: 'GET',
                  params: {
                      'url': productUrl,
                      'apikey': 'YOUR_ZENROWS_API_KEY',
                      'js_render': 'true',
                      'js_instructions': JSON.stringify(instructions),
                  }
              });
              
              return response.data;
          } catch (error) {
              console.error('Error scraping product:', error);
              throw error;
          }
      }
      ```
    </CodeGroup>
  </Tab>
</Tabs>

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Element not found | Selector doesn't match any elements | Verify selector in browser DevTools |
| Click not working | Element not clickable or covered | Use `wait_for` to ensure element is ready |
| Form submission fails | Missing required fields | Fill all required fields before submitting |
| Timeout errors | Content takes too long to load | Increase wait times or use `wait_for` |
| Instructions skip | Previous instruction failed | Check debug report for failed instructions |
| Iframe content missing | Using standard instructions on iframe | Use `frame_*` instructions for iframe elements |

### Debugging selector issues

When selectors aren't working as expected:

<Steps>
  <Step title="Verify selector in browser">
    Open the target page in your browser and test the selector in DevTools console:

    ```javascript JavaScript theme={"dark"}
    // Test CSS selector
    document.querySelector('.your-selector')

    // Test XPath
    document.evaluate('//your/xpath', document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue
    ```
  </Step>

  <Step title="Check element timing">
    Ensure the element exists when the instruction runs:

    ```json JSON theme={"dark"}
    [
        {"wait_for": ".parent-container"},  // Wait for parent first
        {"click": ".child-element"}         // Then interact with child
    ]
    ```
  </Step>

  <Step title="Use alternative selectors">
    Try different selector strategies:

    ```json JSON theme={"dark"}
    [
        {"click": "#button-id"},                    // Try ID first
        {"click": "button[data-action='submit']"},  // Try attribute
        {"click": "//button[text()='Submit']"}      // Try XPath with text
    ]
    ```
  </Step>

  <Step title="Check for dynamic content">
    Some elements are created dynamically:

    ```json JSON theme={"dark"}
    [
        {"click": ".load-content"},         // Trigger content creation
        {"wait_for": ".dynamic-element"},   // Wait for element to appear
        {"click": ".dynamic-element"}       // Then interact with it
    ]
    ```
  </Step>
</Steps>

## Pricing

The `js_instructions` parameter doesn't increase the request cost on its own. Cost still depends on `js_render` and `premium_proxy`, exactly as it would for a standard request.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="What happens if an instruction fails?">
  When an instruction fails, execution continues with the next instruction. Use `json_response=true` to get detailed failure information and adjust your instructions accordingly. Critical failures may require adding `wait_for` instructions to ensure elements are available.
</Accordion>

<Accordion title="Can I use JavaScript Instructions without js_render?">
  No, JavaScript Instructions require `js_render=true` because they operate within a browser environment. Static HTML requests cannot execute dynamic interactions or JavaScript code.
</Accordion>

<Accordion title="How long do JavaScript Instructions take to execute?">
  Execution time varies based on the complexity of instructions and page responsiveness. Simple clicks take 50-200ms, while waiting for elements can take several seconds. Use the debug report to analyze timing and optimize your instruction sequences.
</Accordion>

<Accordion title="Can I interact with elements inside iframes?">
  Standard instructions don't work with iframe content. Use frame-specific instructions like `frame_click`, `frame_fill`, and `frame_wait_for` to interact with elements inside iframes. You'll need to specify the iframe selector as the first parameter.
</Accordion>

<Accordion title="What's the maximum number of instructions I can use?">
  There's no hard limit on instruction count, but longer sequences increase execution time and potential failure points. For complex workflows, consider breaking them into smaller, focused instruction sets and making multiple requests if needed.
</Accordion>

<Accordion title="How do I handle CAPTCHAs in my instructions?">
  Use the `solve_captcha` instruction with a configured [2Captcha](https://www.zenrows.com/go/2captcha) API key. Add wait instructions before and after CAPTCHA solving to allow time for loading and verification. Different CAPTCHA types require specific configuration options.
</Accordion>

<Accordion title="Can I modify the page content with JavaScript Instructions?">
  Yes, use the `evaluate` instruction to run custom JavaScript code that can modify page content, styling, or behavior. This is useful for removing overlays, triggering custom events, or preparing the page for data extraction.
</Accordion>


# JavaScript Rendering (Headless Browser)
Source: https://docs.zenrows.com/fetch/features/js-rendering

Enable headless browser rendering in Zenrows to scrape JavaScript-heavy websites and capture dynamically loaded content behind anti-bot systems.

JavaScript Rendering processes web pages through a headless browser - a browser without a graphical interface that can execute JavaScript and render dynamic content. Many modern websites load content dynamically after the initial page load, making this data invisible to standard HTTP requests that only capture the initial HTML.

When you enable JavaScript Rendering, Zenrows simulates a real browser environment, executing all JavaScript code and waiting for dynamic content to load before returning the fully rendered page. This browser simulation also helps bypass sophisticated anti-bot protections that analyze browser behavior, JavaScript execution patterns, and other browser-specific characteristics that Premium Proxy alone cannot address.

## How JavaScript Rendering Works

JavaScript Rendering launches a headless browser instance that navigates to your target URL just like a regular browser would. The browser executes all JavaScript code, processes CSS, loads additional resources, and waits for the page to fully render before extracting the HTML content.

This process captures content that appears after the initial page load, such as:

* AJAX-loaded product listings
* Single-page application (SPA) navigation
* Dynamic pricing information
* User-generated content loaded via JavaScript

Additionally, the browser simulation helps bypass advanced anti-bot measures that detect:

* Missing browser APIs and properties
* Unusual JavaScript execution patterns
* Absence of browser-specific behaviors
* Automated request signatures

## Basic usage

Enable JavaScript Rendering by adding the `js_render=true` parameter to your Zenrows request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
  	url: 'https://api.zenrows.com/v1/',
  	method: 'GET',
  	params: {
  		'url': url,
  		'apikey': apikey,
  		'js_render': 'true',
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true"
  ```
</CodeGroup>

This example enables JavaScript Rendering for your request. Zenrows processes the page through a headless browser, executing all JavaScript and returning the fully rendered HTML content, rather than just the initial server response.

## When to use JavaScript Rendering

### Content-related needs:

* **Single-page applications (SPAs)** - React, Vue, Angular applications that load content dynamically
* **E-commerce sites** - Product listings, prices, and reviews loaded via JavaScript
* **Search results** - Dynamic search results and pagination
* **Infinite scroll content** - Content that loads as users scroll down
* **AJAX-heavy websites** - Sites that rely heavily on asynchronous data loading
* **Progressive web apps** - Modern web applications with dynamic content updates

### Protection bypass needs:

* **Advanced anti-bot systems** - Sites that analyze browser fingerprints and JavaScript execution
* **Behavioral detection** - Websites that monitor mouse movements, timing patterns, and user interactions
* **Browser API validation** - Sites that check for the presence of browser-specific APIs and properties
* **CloudFlare challenges** - Advanced protection that requires JavaScript execution to pass
* **Captcha systems** - Some captchas that rely on browser behavior analysis

For the highest success rate, especially on heavily protected websites, combine JavaScript Rendering with Premium Proxy. This combination provides both residential IP addresses and realistic browser behavior:

```python Python theme={"dark"}
# Maximum protection: JS Rendering + Premium Proxy
params = {
    'url': 'https://httpbin.io/anything',
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'js_render': 'true',        # Browser simulation
    'premium_proxy': 'true',    # Residential IP
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
```

This combination addresses multiple layers of protection:

* **Premium Proxy:** Provides residential IP addresses that are harder to detect and block
* **JavaScript Rendering:** Simulates genuine browser behavior and executes anti-bot detection scripts

## Identifying when you need JavaScript Rendering

You can determine if a website requires JavaScript Rendering by comparing the initial HTML with what you see in the browser:

```python Python theme={"dark"}
import requests

# Test without JavaScript Rendering
url = 'https://httpbin.io/anything'
response_standard = requests.get('https://api.zenrows.com/v1/', params={
    'url': url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
})

# Test with JavaScript Rendering
response_js = requests.get('https://api.zenrows.com/v1/', params={
    'url': url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'js_render': 'true',
})

# Compare content lengths
print(f"Standard HTML length: {len(response_standard.text)}")
print(f"JS-rendered HTML length: {len(response_js.text)}")

# If JS-rendered content is significantly longer, you need JavaScript Rendering
```

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Content still missing or incomplete | Page needs more time to load | Increase `wait` time or use `wait_for` parameter |
| Still getting blocked | Need residential IPs | Add `premium_proxy=true` |
| Slow response times | Browser processing overhead | Use JavaScript Rendering only when necessary |
| Higher costs than expected | Using both features unnecessarily | Implement progressive enhancement strategy |
| Advanced bot detection | Sophisticated fingerprinting | Combine JS Rendering + Premium Proxy + Custom Headers |

### Debugging protection bypassing

When you're still getting blocked despite using JavaScript Rendering:

<Steps>
  <Step title="Add Premium Proxy for residential IPs">
    ```python Python theme={"dark"}
    params = {
        'js_render': 'true',
        'premium_proxy': 'true',
    }
    ```
  </Step>

  <Step title="Increase wait time or wait for a specific selector for detection scripts">
    ```python Python theme={"dark"}
    params = {
        'js_render': 'true',
        'premium_proxy': 'true',
        'wait': '5000',
    }
    ```

    <Tip>
      See more on the [Wait](/fetch/features/wait) and [Wait\_for](/fetch/features/wait-for) documentation pages
    </Tip>
  </Step>

  <Step title="Add realistic headers">
    ```python Python theme={"dark"}
    params = {
        'js_render': 'true',
        'premium_proxy': 'true',
        'wait': '5000',
    }
    headers = {
        'referer': 'https://www.google.com'
    }
    ```
  </Step>

  <Step title="Return the original status from the website">
    ```python Python theme={"dark"}
    params = {
        'js_render': 'true',
        'original_status': 'true',
    }
    ```

    Returning the original status helps you identify what HTML status code the website is returning.

    <Tip>
      See more about the `original_status` parameter [here](/fetch/features/other#original-http-code)
    </Tip>
  </Step>
</Steps>

## Pricing

JavaScript Rendering costs 5 times the standard request rate due to the additional computational resources required for browser processing.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="Which features require the JavaScript Rendering?">
  Several features rely on `js_render` being set to true. These include:

  * [Wait:](/fetch/features/wait) Introduces a delay before proceeding with the request. Useful for scenarios where you need to allow time for JavaScript to load content.
  * [Wait For:](/fetch/features/wait-for) Waits for a specific element to appear on the page before proceeding. When used with `js_render`, this parameter will cause the request to fail if the selector is not found.
  * [JSON Response:](/fetch/features/json-response) Retrieves the rendered page content in JSON format, including data loaded dynamically via JavaScript.
  * [Block Resources:](/fetch/features/block-resources) Block specific types of resources from being loaded.
  * [JavaScript Instructions:](/fetch/features/js-instructions) Allows you to execute custom JavaScript code on the page. This includes additional parameters.
  * [Screenshot:](/fetch/features/screenshot) Capture an above-the-fold screenshot of the target page by adding `screenshot=true` to the request.
</Accordion>

<Accordion title="When should I use JavaScript Rendering vs standard requests?">
  Use JavaScript Rendering when the content you need is loaded dynamically via JavaScript, when dealing with single-page applications, or when facing advanced anti-bot protection that analyzes browser behavior. If the content is present in the initial HTML response and the site has no protection, standard requests are more cost-effective.
</Accordion>

<Accordion title="Should I always combine JavaScript Rendering with Premium Proxy?">
  For the highest success rate on protected websites, yes. However, this combination costs 25 credits per request. Start with individual features and combine them only when necessary. Use progressive enhancement to find the most cost-effective approach for each target.
</Accordion>

<Accordion title="Why does JavaScript Rendering help bypass protections?">
  JavaScript Rendering simulates a real browser environment, executing anti-bot detection scripts and providing browser-specific APIs that automated tools typically lack. This makes requests appear more like genuine user traffic, helping bypass sophisticated behavioral analysis systems.
</Accordion>

<Accordion title="What happens if both features still don't work?">
  If the combination of JavaScript Rendering and Premium Proxy doesn't work, the website likely uses very advanced protection. Try adding longer wait times, custom headers, or contact Zenrows support for assistance with particularly challenging targets.
</Accordion>

<Accordion title="Can I use different countries with this combination?">
  Yes, you can specify proxy countries when using both features by adding the `proxy_country` parameter. This can help access geo-restricted content while maintaining the protection benefits of both features.
</Accordion>


# JSON Response
Source: https://docs.zenrows.com/fetch/features/json-response

Return scraped page content along with captured XHR and Fetch network requests as structured JSON for debugging and data analysis in Zenrows.

The JSON Response feature transforms Zenrows' standard HTML output into a structured JSON object containing the page content, network requests, and optional execution reports. This format provides comprehensive insights into page behavior, making it ideal for debugging, monitoring network activity, and analyzing dynamic content loading patterns.

When you enable JSON Response with `json_response=true`, Zenrows captures not only the final HTML content but also all XHR, Fetch, and AJAX requests made during page rendering. This gives you complete visibility into how modern web applications load and update their content.

<Note>The JSON Response parameter requires `js_render=true` to function, as it monitors network activity and JavaScript execution within the browser environment.</Note>

## How JSON Response works

JSON Response intercepts and records most network activity during JavaScript rendering, creating a comprehensive report of the page's behavior. The browser monitors every HTTP request made by the page, including background API calls, resource loading, and dynamic content updates.

This process captures:

* All XHR and Fetch requests with full request/response details
* The final rendered HTML content
* Optional JavaScript instruction execution reports
* Optional screenshot data of the rendered page
* Complete request and response headers for network analysis

The data is structured in a JSON format making it easy to programmatically analyze page behavior and extract specific information from network requests.

## Basic usage

Enable JSON Response by adding the `json_response=true` parameter to your JavaScript rendering request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'json_response': 'true',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'js_render': 'true',
          'json_response': 'true',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&json_response=true";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&json_response=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&json_response=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&json_response=true')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&json_response=true"
  ```
</CodeGroup>

This example returns a JSON object containing the HTML content and any network requests made during page rendering, instead of just the raw HTML string.

## JSON Response structure

The JSON Response contains several key fields that provide different types of information. Here's a complete example of what a typical JSON response looks like:

```json JSON expandable theme={"dark"}
{
  "html": "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n    <meta charset=\"UTF-8\">\n    <title>Product Page</title>\n</head>\n<body>\n    <div class=\"product-container\">\n        <h1 class=\"product-title\">Wireless Headphones</h1>\n        <div class=\"price-display\">$99.99</div>\n        <div class=\"stock-status\">In Stock</div>\n        <div class=\"reviews-section\">\n            <div class=\"review-item\">Great sound quality!</div>\n            <div class=\"review-item\">Comfortable to wear</div>\n        </div>\n    </div>\n</body>\n</html>",
  "xhr": [
    {
      "url": "https://api.example.com/product/123/price",
      "body": "{\"price\": 99.99, \"currency\": \"USD\", \"discount\": 0.15}",
      "status_code": 200,
      "method": "GET",
      "headers": {
        "content-type": "application/json",
        "content-length": "54",
        "cache-control": "no-cache"
      },
      "request_headers": {
        "accept": "application/json",
        "user-agent": "Mozilla/5.0 (compatible; Zenrows)"
      }
    },
    {
      "url": "https://api.example.com/product/123/inventory",
      "body": "{\"in_stock\": true, \"quantity\": 25, \"warehouse\": \"US-EAST\"}",
      "status_code": 200,
      "method": "GET",
      "headers": {
        "content-type": "application/json",
        "content-length": "67"
      },
      "request_headers": {
        "accept": "application/json",
        "authorization": "Bearer token123"
      }
    },
    {
      "url": "https://api.example.com/product/123/reviews",
      "body": "{\"reviews\": [{\"rating\": 5, \"comment\": \"Great sound quality!\"}, {\"rating\": 4, \"comment\": \"Comfortable to wear\"}], \"average_rating\": 4.5}",
      "status_code": 200,
      "method": "GET",
      "headers": {
        "content-type": "application/json",
        "content-length": "156"
      },
      "request_headers": {
        "accept": "application/json"
      }
    }
  ],
  "js_instructions_report": {
    "instructions_duration": 1041,
    "instructions_executed": 2,
    "instructions_succeeded": 2,
    "instructions_failed": 0,
    "instructions": [
      {
        "instruction": "wait_for_selector",
        "params": {
          "selector": ".price-display"
        },
        "success": true,
        "duration": 40
      },
      {
        "instruction": "wait",
        "params": {
          "timeout": 1000
        },
        "success": true,
        "duration": 1001
      }
    ]
  },
  "screenshot": {
    "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg==",
    "type": "image/png",
    "width": 1920,
    "height": 1080
  }
}
```

### Response fields

* **`html`** - The complete rendered HTML content of the page as a string. This content is JSON-encoded and contains the final DOM state after all JavaScript execution and dynamic loading.

* **`xhr`** - An array containing all XHR, Fetch, and AJAX requests made during page rendering. Each request object includes:
  * `url` - The complete URL of the request
  * `body` - The response body content
  * `status_code` - HTTP status code (200, 404, 500, etc.)
  * `method` - HTTP method used (GET, POST, PUT, DELETE, etc.)
  * `headers` - Response headers from the server
  * `request_headers` - Headers sent with the original request

* **`js_instructions_report`** *(Optional - only present when using JavaScript Instructions)* - Detailed execution report including:
  * `instructions_duration` - Total execution time in milliseconds
  * `instructions_executed` - Number of instructions processed
  * `instructions_succeeded` - Number of successful instructions
  * `instructions_failed` - Number of failed instructions
  * `instructions` - Array of individual instruction details with parameters, success status, and timing

* **`screenshot`** *(Optional - only present when using the Screenshot feature)* - Screenshot data containing:
  * `data` - Base64-encoded image data
  * `type` - Image format (typically "image/png")
  * `width` - Screenshot width in pixels
  * `height` - Screenshot height in pixels

## When to use JSON Response

JSON Response is essential for these scenarios:

### Dynamic content loading:

* **AJAX-loaded content** - When page content is loaded dynamically via XHR calls, JSON Response captures these API requests so you can access the raw data directly from the network calls rather than parsing it from the rendered HTML
* **API-dependent data** - Content that appears after external API calls
* **Progressive loading** - Pages that load content in stages
* **Conditional content** - Elements that appear based on user state or preferences
* **Real-time applications** - Capture live data feeds and WebSocket-like communications
* **Search platforms** - Monitor search API calls and result loading patterns

### Debugging and development:

* **Network troubleshooting** - Identify failed requests or slow API calls
* **Content loading analysis** - Understand how content loads in stages
* **Performance monitoring** - Track request timing and response sizes
* **Integration testing** - Verify that all expected API calls are made
* **Security analysis** - Monitor for unexpected network activity
* **JavaScript Instructions debugging** - When using JavaScript Instructions, JSON Response provides detailed execution reports to help debug instruction failures and timing issues

<Note>For comprehensive guidance on debugging JavaScript Instructions using JSON Response, see our [JavaScript Instructions Debugging documentation](/fetch/features/js-instructions#debugging-javascript-instructions).</Note>

## Best practices

### Combine with appropriate parameters

Use JSON Response with other features such as `wait` or `wait_for` to guarantee the content will be loaded:

```python Python theme={"dark"}
params = {
  'url': url,
  'apikey': 'YOUR_ZENROWS_API_KEY',
  'js_render': 'true',
  'json_response': 'true',
  'premium_proxy': 'true',  # For protected sites
  'wait': '3000',           # Allow time for API calls
  'screenshot': 'true',     # Include visual verification
}
```

### Filter and process network requests

Focus on relevant requests to avoid information overload

```python Python theme={"dark"}
def filter_relevant_requests(xhr_requests, filters):
    # Filter XHR requests based on specified criteria
    
    relevant_requests = []
    
    for request in xhr_requests:
        include_request = True
        
        # Apply filters
        if 'methods' in filters:
            if request['method'] not in filters['methods']:
                include_request = False
        
        if 'status_codes' in filters:
            if request['status_code'] not in filters['status_codes']:
                include_request = False
        
        if 'url_contains' in filters:
            if not any(pattern in request['url'] for pattern in filters['url_contains']):
                include_request = False
        
        if 'exclude_extensions' in filters:
            if any(request['url'].endswith(ext) for ext in filters['exclude_extensions']):
                include_request = False
        
        if include_request:
            relevant_requests.append(request)
    
    return relevant_requests

# Filter for API calls only
api_filters = {
    'methods': ['GET', 'POST'],
    'status_codes': [200, 201, 202],
    'url_contains': ['/api/', '.json', '/v1/', '/v2/'],
    'exclude_extensions': ['.css', '.js', '.png', '.jpg', '.gif', '.svg'],
}

api_requests = filter_relevant_requests(json_data['xhr'], api_filters)
print(f"Found {len(api_requests)} relevant API requests")
```

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Empty XHR array | No network requests made | Verify the page actually makes AJAX calls |
| Missing expected requests | Requests happen after page load | Increase `wait` time or use `wait_for` |
| Large response size | Many network requests captured | Filter requests to focus on relevant ones |
| JSON parsing errors | Malformed response data | Add error handling for JSON parsing |
| Incomplete request data | Requests still in progress | Ensure adequate wait time for completion |
| Response size exceeds plan limits | Too much data captured | Upgrade plan or use Block Resources parameter |

### Handling oversized responses

When JSON responses become too large and exceed your plan's limits:

**Response size exceeded error**: If your response exceeds the maximum size allowed by your plan, you'll receive an error indicating the limit has been reached.

**Solutions**:

1. **Upgrade your plan** - Higher-tier plans support larger response sizes
2. **Use Block Resources parameter** - Remove unnecessary content like images, stylesheets, and scripts:
   ```python theme={"dark"}
   params = {
       'json_response': 'true',
       'block_resources': 'image,stylesheet,script,font',  # Block heavy resources
   }
   ```
3. **Filter XHR requests** - Focus on specific API endpoints rather than capturing all network activity
4. **Use shorter wait times** - Reduce the time window to capture fewer requests

<Tip>The Block Resources parameter can significantly reduce response sizes by preventing the loading of images, CSS files, JavaScript files, and fonts that aren't essential for your scraping needs.</Tip>

### Debugging missing network requests

When expected API calls don't appear in the XHR array:

<Steps>
  <Step title="Verify if XHR requests are actually made">
    ```python Python theme={"dark"}
    # Check if any requests were captured
    if not json_data.get('xhr'):
        print("No network requests captured - verify page makes AJAX calls")
    else:
        print(f"Captured {len(json_data['xhr'])} requests")
    ```
  </Step>

  <Step title="Increase wait time for slow requests">
    ```python Python theme={"dark"}
    params = {
        'js_render': 'true',
        'json_response': 'true',
        'wait': '10000',  # Wait 10 seconds for slow API calls
    }
    ```
  </Step>

  <Step title="Use `wait_for` to ensure content loads">
    ```python Python theme={"dark"}
    params = {
        'js_render': 'true',
        'json_response': 'true',
        'wait_for': '.api-loaded-content',  # Wait for specific content
    }
    ```
  </Step>
</Steps>

## Pricing

The `json_response` parameter doesn't increase the request cost on its own. Cost still depends on `js_render` and `premium_proxy`, exactly as it would for a standard request.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="Do I need js_render=true to use json_response?">
  Yes, the `json_response` parameter requires `js_render=true` because it monitors network activity that occurs during JavaScript execution. Without JavaScript rendering, there would be no XHR/Fetch requests to capture.
</Accordion>

<Accordion title="Will json_response capture all network requests made by the page?">
  JSON Response captures XHR, Fetch, and AJAX requests made during the page rendering process. It does not capture initial resource loading (like CSS, images, or the main HTML request), but focuses on dynamic API calls and background requests.
</Accordion>

<Accordion title="How large can the JSON response become?">
  The size depends on the number of network requests and their response sizes. Pages with many API calls can generate large JSON responses. Consider filtering requests or processing data in chunks for very active pages.
</Accordion>

<Accordion title="Can I get only the XHR data without the HTML content?">
  No, the JSON Response always includes both HTML and XHR data. However, you can ignore the HTML field in your processing and focus only on the XHR array if that's all you need.
</Accordion>

<Accordion title="What happens if a network request fails during page loading?">
  Failed requests are still captured in the XHR array with their actual status codes (e.g., 404, 500). This allows you to analyze both successful and failed network activity.
</Accordion>

<Accordion title="Does json_response work with Premium Proxy?">
  Yes, JSON Response works perfectly with Premium Proxy. Network monitoring occurs within the browser environment, regardless of the type of proxy used.
</Accordion>

<Accordion title="Can I use json_response to monitor WebSocket connections?">
  No, JSON Response only captures HTTP-based requests (XHR, Fetch, AJAX). WebSocket connections use a different protocol and are not captured in the XHR array.
</Accordion>

<Accordion title="How do I handle pages that make requests after user interactions?">
  Use JavaScript Instructions to simulate user interactions (such as clicks and scrolls) before capturing the page. This will trigger additional network requests that will be included in the JSON response.
</Accordion>


# Markdown Response
Source: https://docs.zenrows.com/fetch/features/markdown

Get scraped page content returned as clean Markdown instead of raw HTML using the `response_type=markdown` parameter.

The `response_type=markdown` parameter tells Zenrows to convert the scraped page's HTML into clean Markdown before returning the response. Instead of receiving raw HTML that you need to parse yourself, you get structured, readable content that's ready to process or display.

This is particularly useful for AI pipelines, content extraction workflows, and any use case where you need human-readable text with preserved structure (headings, lists, links, tables) without the noise of HTML tags.

<Warning>
  `response_type=markdown` cannot be combined with the `outputs` parameter. Use one or the other depending on whether you need structured data extraction or a full Markdown conversion of the page.
</Warning>

## How it works

Zenrows fetches the target page, renders it if needed, and converts the resulting HTML into Markdown. The conversion preserves the semantic structure of the page: headings map to `#` through `######`, lists remain as `-` items, links keep their text and URL, and tables are converted to Markdown table syntax.

For example, given the following HTML:

```html HTML theme={"dark"}
<h1>Product Title</h1>
<p>This is a great product that does many things.</p>
<ul>
    <li>Feature 1</li>
    <li>Feature 2</li>
    <li>Feature 3</li>
</ul>
```

The Markdown response returns:

```markdown Markdown theme={"dark"}
# Product Title

This is a great product that does many things.

- Feature 1
- Feature 2
- Feature 3
```

## Basic Usage

Add `response_type=markdown` to your request parameters:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  url = "https://www.scrapingcourse.com/ecommerce/"
  apikey = "YOUR_ZENROWS_API_KEY"

  params = {
      "url": url,
      "apikey": apikey,
      "response_type": "markdown",
  }

  response = requests.get("https://api.zenrows.com/v1/", params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  import axios from "axios";

  const url = "https://www.scrapingcourse.com/ecommerce/";
  const apikey = "YOUR_ZENROWS_API_KEY";

  const response = await axios.get("https://api.zenrows.com/v1/", {
  params: {
      url,
      apikey,
      response_type: "markdown",
  },
  });

  console.log(response.data);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=markdown";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=markdown");
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=markdown", nil)
      if err != nil {
          log.Fatalln(err)
      }

      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  require "faraday"

  url = URI.parse("https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=markdown")
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=markdown"
  ```
</CodeGroup>

## When to use Markdown response

The Markdown response is the right choice when you need the full text content of a page with its structure preserved, but don't want to parse HTML yourself. It works well for:

* **AI and LLM pipelines**: feed scraped content directly into language models without pre-processing. Markdown is cleaner and more token-efficient than raw HTML.
* **Content extraction and archiving**: capture articles, documentation pages, or product descriptions in a portable, readable format.
* **Knowledge base ingestion**: convert web pages into Markdown for ingestion into vector databases, RAG pipelines, or internal wikis.
* **Developer tooling**: use Markdown output to power documentation crawlers, content diff tools, or search indexers.

<Note>If you only need specific data types from a page (like emails, links, or tables), use the `outputs` parameter instead. Markdown response converts the entire page, which may include content you don't need.</Note>

## Best practices

**Enable `js_render` for dynamic pages:** <br />
If the page loads content via JavaScript, combine `response_type=markdown` with `js_render=true`. Without it, the Markdown output will only reflect the initial HTML and may be missing key content.

**Use `wait` or `wait_for` when content loads after a delay:** <br />
For pages where content appears after a delay or user interaction, add `wait` (milliseconds) or `wait_for` (CSS selector) to ensure the full content is present before conversion.

<Tip>Find more details on our [Wait Documentation](/fetch/features/wait) or [Wait For Selector Documentation](/fetch/features/wait-for).</Tip>

**Expect navigation and boilerplate in the output:** <br />
The Markdown response converts the full page, including headers, footers, navigation menus, and sidebars. If you need only the main body content, consider post-processing the Markdown to strip boilerplate, or evaluate whether `css_extractor` with a targeted selector better fits your use case.

<Tip>Find more details on our [CSS Extractor Documentation](/fetch/features/css-extractor).</Tip>

**Credits are charged on successful responses:** <br />
A request using `response_type=markdown` is charged when the API returns a `200` status code, regardless of whether the Markdown output contains the content you expected. Test on a small set of URLs before running at scale.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The Markdown output is missing most of the page content">
    The page likely renders its content via JavaScript. Add `js_render=true` to your request to ensure Zenrows renders the page fully before converting it. If content still appears incomplete, combine with `wait_for` targeting a CSS selector that's only present once the main content has loaded.
  </Accordion>

  <Accordion title="The output includes a lot of navigation, ads, or boilerplate">
    The Markdown response converts the full HTML of the page, not just the main body. This is expected behavior. To target only a specific section of the page, use `css_extractor` with a selector pointing to the main content container (for example, `article` or `main`) instead of using `response_type=markdown`.
  </Accordion>

  <Accordion title="Tables are not converting correctly">
    Markdown table conversion works on standard HTML `table` elements. If the site uses `div`-based layouts to simulate tables, they won't convert cleanly. In that case, use the `outputs=tables` filter or `css_extractor` to target the relevant elements directly.
  </Accordion>

  <Accordion title="I'm getting an error when combining response_type=markdown with outputs">
    These two parameters are mutually exclusive. `response_type=markdown` converts the entire page to Markdown, while `outputs` performs targeted data extraction. You can only use one per request. Choose `outputs` when you need specific data types, and `response_type=markdown` when you need the full page content as structured text.
  </Accordion>
</AccordionGroup>

## Pricing

The `response_type=markdown` parameter is included at **no additional cost** with all Zenrows requests. You only pay extra for JavaScript Render and Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## FAQ (Frequently Asked Questions)

<Accordion title="Does response_type=markdown work with js_render?">
  Yes, you can combine `response_type=markdown` with `js_render=true` to render JavaScript before the conversion. This is recommended for any page that loads content dynamically.
</Accordion>

<Accordion title="Does it cost more credits to use response_type=markdown?">
  No, the `response_type=markdown` parameter does not add extra credit cost on its own. Credits are calculated based on the other parameters you use, such as `js_render` or `premium_proxy`, exactly as they would be for a standard request.
</Accordion>

<Accordion title="Can I use response_type=markdown with Browser Sessions?">
  No, `response_type=markdown` is a parameter of Fetch. It is not available in Browser Sessions (CDP/Playwright) sessions. In a Browser Sessions session, you control the page directly and can extract and convert content using your own script logic.
</Accordion>

<Accordion title="What Markdown flavor does the conversion follow?">
  Zenrows converts to standard CommonMark-compatible Markdown. Most elements (headings, lists, bold, italic, links, code blocks, tables) are supported. Custom HTML components, embedded scripts, and non-semantic markup are stripped from the output.
</Accordion>

<Accordion title="Can I use response_type=markdown to feed content into an LLM?">
  Yes, and this is one of the most common use cases. Markdown is significantly more compact and cleaner than raw HTML, which reduces token usage and improves the quality of LLM outputs. Zenrows strips tags, scripts, and boilerplate during conversion, leaving structured prose that models can process directly.
</Accordion>


# Other Features
Source: https://docs.zenrows.com/fetch/features/other

Configure advanced Zenrows API parameters including Session ID for persistent IPs, original HTTP status codes, and POST request handling.

## Session ID

The Session ID feature allows you to use the same IP address across multiple API requests. By setting the `session_id=12345` parameter in your requests, Zenrows will ensure that all requests made with the same Session ID are routed through the same IP address for 10 minutes.

This feature is useful for web scraping sites that track sessions or limit IP rotation. It helps simulate a persistent session and avoids triggering anti-bot systems that flag frequent IP changes.

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
  	'session_id': '12345',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
  	url: 'https://api.zenrows.com/v1/',
  	method: 'GET',
  	params: {
  		'url': url,
  		'apikey': apikey,
  		'session_id': '12345',
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&session_id=12345";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&session_id=12345');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&session_id=12345", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&session_id=12345')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&session_id=12345"
  ```
</CodeGroup>

<Tip>You can use any number from `1` to `99999` for `session_id`.</Tip>

## Original HTTP Code

By default, the Zenrows API returns standard HTTP codes based on the success or failure of your request. However, websites may provide their own status codes, which can give you valuable information about how the site handled the request (e.g., `403 Forbidden` or `500 Internal Server Error`).

If you need to retrieve the original HTTP status code returned by the target website, simply enable the `original_status=true` parameter. This is particularly useful for debugging, understanding site-specific responses, or adapting scraping strategies based on the target site's behavior.

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
  	'original_status': 'true',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
  	url: 'https://api.zenrows.com/v1/',
  	method: 'GET',
  	params: {
  		'url': url,
  		'apikey': apikey,
  		'original_status': 'true',
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&original_status=true";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&original_status=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&original_status=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&original_status=true')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&original_status=true"
  ```
</CodeGroup>

## Return content on error

When working with web scraping, you may encounter errors such as `404 Not Found`, `500 Internal Server Error`, or `503 Service Unavailable`. Even in these cases, the response might contain valuable information, such as error messages or partial content that can aid in debugging.

To access this content despite the error status, you can use the `allowed_status_codes` parameter. This feature enhances the API's debugging capabilities by allowing you to retrieve the content associated with specified error codes.

You can specify multiple status codes in a single request by separating them with commas.

<Warning>Requests using the param `allowed_status_codes` will incur charges!</Warning>

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'allowed_status_codes': '404,500,503',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
  	url: 'https://api.zenrows.com/v1/',
  	method: 'GET',
  	params: {
  		'url': url,
  		'apikey': apikey,
          'allowed_status_codes': '404,500,503',
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&allowed_status_codes=404,500,503";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&allowed_status_codes=404,500,503');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&allowed_status_codes=404,500,503", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&allowed_status_codes=404,500,503')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&allowed_status_codes=404,500,503"
  ```
</CodeGroup>

## POST / PUT Requests

You can send POST and PUT requests using your preferred programming language, and Zenrows will handle the data forwarding to the target site seamlessly. This allows you to interact with APIs or web forms that require data submission without worrying about the underlying mechanics of HTTP requests.

The response you receive will contain the original content from the target site, including any headers and cookies. This can be particularly useful for maintaining sessions or tracking state during your interactions with web applications.

Accessing headers and cookies will vary based on the method you use to make the request, so refer to the relevant documentation for your programming language or library to handle these elements appropriately.

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
  }
  data = {
  	'key1': 'value1',
  	'key2': 'value2',
  }
  response = requests.post('https://api.zenrows.com/v1/', params=params, data=data)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
  	url: 'https://api.zenrows.com/v1/',
  	method: 'POST',
  	data: 'key1=value1&key2=value2',
  	params: {
  		'url': url,
  		'apikey': apikey,
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;
  import org.apache.hc.client5.http.fluent.Form;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything";
          String response = Request.post(apiUrl)
  				.bodyForm(Form.form()
  					.add("key1", "value1")
  					.add("key2", "value2")
  					.build())
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $postBody = [
  	'key1' => 'value1',
  	'key2' => 'value2',
  ];
  curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($postBody));
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  	"net/url"
  	"strings"
  )

  func main() {
  	form := url.Values{}
  	form.Set("key1", "value1")
  	form.Set("key2", "value2")

      client := &http.Client{}
      req, err := http.NewRequest("POST", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything", strings.NewReader(form.Encode()))
      if err != nil {
          log.Fatalln(err)
      }
  	req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything')
  data = {
  	"key1" => "value1",
  	"key2" => "value2",
  }
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.post(url, URI.encode_www_form(data), nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything" -X "POST" --data "key1=value1&key2=value2"
  ```
</CodeGroup>

## Plan Usage

To monitor your subscription plan's usage programmatically, you can call the endpoint `/v1/subscriptions/self/details`. This allows you to track your API usage in real time, enabling better management of your resources and ensuring you stay within your plan's limits.

Calls made to this endpoint do not count against your concurrency limits, making it an efficient way to gather usage statistics without impacting your scraping activities.

To use this feature, include your API Key in the request header using `X-API-Key`. This is necessary for authentication and to ensure that the usage data corresponds to your account.

<Note>The previous endpoint `/usage` is deprecated, so please use the updated endpoint for all future requests to avoid disruptions.</Note>

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  headers = {
  	'X-API-Key': 'YOUR_ZENROWS_API_KEY',
  }
  response = requests.get('https://api.zenrows.com/v1/subscriptions/self/details', headers=headers)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  axios({
  	url: 'https://api.zenrows.com/v1/subscriptions/self/details',
  	method: 'GET',
      headers: {
  		'X-API-Key': 'YOUR_ZENROWS_API_KEY',
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/subscriptions/self/details";
          String response = Request.get(apiUrl)
  				.addHeader("X-API-Key", "YOUR_ZENROWS_API_KEY")
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/subscriptions/self/details');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_HTTPHEADER, [
  	'X-API-Key: YOUR_ZENROWS_API_KEY',
  ]);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/subscriptions/self/details", nil)
      if err != nil {
          log.Fatalln(err)
      }
      req.Header.Add("X-API-Key", "YOUR_ZENROWS_API_KEY")
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/subscriptions/self/details')
  headers = {
  	"X-API-Key": "YOUR_ZENROWS_API_KEY",
  }
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, headers)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl -H "X-API-Key: YOUR_ZENROWS_API_KEY" "https://api.zenrows.com/v1/subscriptions/self/details"
  ```
</CodeGroup>

The output will be something similar to this:

```json theme={"dark"}
{
    "status": "ACTIVE",
    "period_starts_at": "202X-09-02T00:00:00Z",
    "period_ends_at": "202X-10-02T00:00:00Z",
    "usage": 103.815320296,
    "usage_percent": 34,
    "plan": {
        "name": "Business",
        "price": 299.99,
        "recurrence": "MONTHLY",
        "products": {
            "api": {
                "concurrency": {
                    "limit": 100,
                    "usage": 1
                },
                "name": "WebScraping API",
                "usage": 103.7470762335
            },
            "proxy_residential": {
                "name": "Residential Proxies",
                "usage": 0.033588062450952
            },
            "scraping_browser": {
                "name": "Browser Sessions",
                "usage": 0.034656
            }
        }
    },
    "top_ups": []
}
```

### API key credit caps

The response also includes an `api_key` object with the [credit caps](/first-steps/api-key-credit-caps) of the key you call with. It describes only that key, never the other keys on your account, and it does not repeat the key itself.

This endpoint keeps working after the key reaches one of its caps, so you can always check which cap was reached and when it resets.

```json theme={"dark"}
"api_key": {
    "caps": [
        {
            "window": "day",
            "credits": 50000,
            "used_credits": 1234,
            "remaining_credits": 48766,
            "resets_at": "202X-10-02T00:00:00Z"
        }
    ]
}
```

| Field | Description |
| - | - |
| `caps` | One entry per cap on the key. An empty list means the key has no caps. |
| `window` | The period the cap covers: `day`, `week`, `month` or `billing_period`. |
| `credits` | The cap, in credits. |
| `used_credits` | Credits this key has used in the current window, rounded up to a whole credit and never more than `credits`. |
| `remaining_credits` | Credits left before the cap is reached, never below 0. |
| `resets_at` | When the current window ends and usage starts again from 0, in UTC. |
| `unavailable` | Present and `true` when the cap's usage could not be read right now. `used_credits` and `remaining_credits` are left out, and the cap does not block the key. |

If `api_key` is missing from the response, the cap information could not be loaded for this request. Treat it as unknown, not as uncapped, and try again later.

## Frequently Asked Questions (FAQ)

<Accordion title="What is the Session ID feature?">
  The Session ID feature allows you to maintain the same IP address across multiple API requests. By using the `session_id` parameter (ranging from 1 to 99999), Zenrows ensures that all requests with the same Session ID are routed through the same IP for up to 10 minutes. This helps avoid triggering anti-bot mechanisms on websites that track sessions, making it particularly useful for web scraping tasks where consistency is important.
</Accordion>

<Accordion title="What should I do if I encounter errors during requests?">
  If you encounter errors, use the `allowed_status_codes` parameter to access content even when error responses (like 404 or 500) are returned. This can provide valuable data for debugging and understanding the response from the server.
</Accordion>

<Accordion title="How can I check my plan usage programmatically?">
  To check your plan usage, call the endpoint `/v1/subscriptions/self/details`. This will return real-time usage data without counting against your concurrency limits. Include your API Key in the `X-API-Key` header for authentication.
</Accordion>


# Output Filters
Source: https://docs.zenrows.com/fetch/features/output-filters

Extract only the data types you need from any scraped page using the outputs parameter.

The `outputs` parameter lets you specify exactly which data types to extract from a scraped page. Instead of parsing raw HTML yourself, Zenrows does the extraction for you and returns a clean, structured JSON response.

This is useful when you only need specific elements (like emails, links, or headings) and want to skip the overhead of processing full HTML responses.

<Tip>The parameter accepts a comma-separated list of filter names. You can also use `outputs=*` to retrieve all available data types at once.</Tip>

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  url = "https://www.scrapingcourse.com/ecommerce/"
  apikey = "YOUR_ZENROWS_API_KEY"

  params = {
      "url": url,
      "apikey": apikey,
      "outputs": "emails,headings,menus",
  }

  response = requests.get("https://api.zenrows.com/v1/", params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  import axios from "axios";

  const url = "https://www.scrapingcourse.com/ecommerce/";
  const apikey = "YOUR_ZENROWS_API_KEY";

  const response = await axios.get("https://api.zenrows.com/v1/", {
  params: {
      url,
      apikey,
      outputs: "emails,headings,menus",
  },
  });

  console.log(response.data);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&outputs=emails,headings,menus";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&outputs=emails,headings,menus");
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&outputs=emails,headings,menus", nil)
      if err != nil {
          log.Fatalln(err)
      }

      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  require "faraday"

  url = URI.parse("https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&outputs=emails,headings,menus")
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&outputs=emails,headings,menus"
  ```
</CodeGroup>

## Available filters

<AccordionGroup>
  <Accordion title="emails" icon="envelope">
    Extracts email addresses from the page using CSS selectors and regular expressions. This includes standard formats like `example@example.com` as well as obfuscated versions like `example[at]example.com` and `support at support dot com`.

    Example: `outputs=emails`

    ```json theme={"dark"}
    {
      "emails": [
        "example@example.com",
        "info@website.com",
        "contact[at]domain.com",
        "support at support dot com"
      ]
    }
    ```
  </Accordion>

  <Accordion title="phone_numbers" icon="phone">
    Extracts phone numbers by targeting `tel:` protocol links and applying regular expressions to detect common formats.

    Example: `outputs=phone_numbers`

    ```json theme={"dark"}
    {
      "phone_numbers": [
        "+1-800-555-5555",
        "(123) 456-7890",
        "+44 20 7946 0958"
      ]
    }
    ```
  </Accordion>

  <Accordion title="headings" icon="heading">
    Extracts all heading text from `h1` through `h6` elements, giving you a quick structural outline of the page.

    Example: `outputs=headings`

    ```json theme={"dark"}
    {
      "headings": [
        "Welcome to Our Website",
        "Our Services",
        "Contact Us",
        "FAQ"
      ]
    }
    ```
  </Accordion>

  <Accordion title="images" icon="image">
    Extracts image sources from `img` tags. Only the `src` attribute is returned.

    Example: `outputs=images`

    ```json theme={"dark"}
    {
      "images": [
        "https://example.com/image1.jpg",
        "https://example.com/image2.png"
      ]
    }
    ```
  </Accordion>

  <Accordion title="audios" icon="volume">
    Extracts audio sources from `source` elements nested inside `audio` tags. Only the `src` attribute is returned.

    Example: `outputs=audios`

    ```json theme={"dark"}
    {
      "audios": [
        "https://example.com/audio1.mp3",
        "https://example.com/audio2.wav"
      ]
    }
    ```
  </Accordion>

  <Accordion title="videos" icon="video">
    Extracts video sources from `source` elements nested inside `video` tags. Only the `src` attribute is returned.

    Example: `outputs=videos`

    ```json theme={"dark"}
    {
      "videos": [
        "https://example.com/video1.mp4",
        "https://example.com/video2.webm"
      ]
    }
    ```
  </Accordion>

  <Accordion title="links" icon="link">
    Extracts all URLs from `a` tags on the page. Only the `href` attribute is returned.

    Example: `outputs=links`

    ```json theme={"dark"}
    {
      "links": [
        "https://example.com/page1",
        "https://example.com/page2"
      ]
    }
    ```
  </Accordion>

  <Accordion title="menus" icon="bars">
    Extracts navigation menu items from `li` elements inside `menu` tags.

    Example: `outputs=menus`

    ```json theme={"dark"}
    {
      "menus": [
        "Home",
        "About Us",
        "Services",
        "Contact"
      ]
    }
    ```
  </Accordion>

  <Accordion title="hashtags" icon="hashtag">
    Extracts hashtags from page content using regular expressions, matching standard formats like `#example`.

    Example: `outputs=hashtags`

    ```json theme={"dark"}
    {
      "hashtags": [
        "#vacation",
        "#summer2024",
        "#travel"
      ]
    }
    ```
  </Accordion>

  <Accordion title="metadata" icon="tag">
    Extracts `name` and `content` attributes from `meta` tags in the `head` section. Each entry is returned in `name: content` format.

    Example: `outputs=metadata`

    ```json theme={"dark"}
    {
      "metadata": [
        "description: This is an example webpage.",
        "keywords: example, demo, website",
        "author: John Doe"
      ]
    }
    ```
  </Accordion>

  <Accordion title="tables" icon="table">
    Extracts data from `table` elements and returns it as structured JSON, including the table dimensions, column headings, and row content.

    Example: `outputs=tables`

    ```json theme={"dark"}
    {
      "dimensions": {
        "rows": 4,
        "columns": 4,
        "heading": true
      },
      "heading": ["A", "B", "C", "D"],
      "content": [
        {"A": "1", "B": "1", "C": "1", "D": "1"},
        {"A": "2", "B": "2", "C": "2", "D": "2"},
        {"A": "3", "B": "3", "C": "3", "D": "3"},
        {"A": "4", "B": "4", "C": "4", "D": "4"}
      ]
    }
    ```
  </Accordion>

  <Accordion title="favicon" icon="star">
    Extracts the favicon URL from the `link` element in the `head` section.

    Example: `outputs=favicon`

    ```json theme={"dark"}
    {
      "favicon": "https://example.com/favicon.ico"
    }
    ```
  </Accordion>
</AccordionGroup>

## When to use output filters

Output filters are the right choice when you know exactly which data types you need and don't want to deal with raw HTML. They're particularly well-suited for:

* **Lead generation and outreach**: use `emails` and `phone_numbers` to collect contact information from directories, company pages, or listings.
* **SEO and content audits**: combine `headings`, `metadata`, `links`, and `favicon` to analyze page structure and on-page signals across multiple URLs.
* **E-commerce and media monitoring**: use `images`, `videos`, and `tables` to track product listings, pricing tables, or media assets.
* **Social and community research**: use `hashtags` to monitor trending topics or branded tags across public pages.
* **Navigation and site mapping**: use `menus` and `links` to map out a site's structure without scraping full page content.

<Note>Output filters work on the final rendered HTML. If the data you need is loaded dynamically via JavaScript, enable `js_render=true` alongside your `outputs` parameter.</Note>

## Best practices

**Only request what you need:** <br />Each filter adds a small processing overhead. Requesting `outputs=*` is convenient during exploration, but in production you should specify only the filters relevant to your use case.

**Combine filters in a single request:** <br />You can pass multiple filters as a comma-separated list (`outputs=emails,links,metadata`) rather than making separate requests. This keeps your credit usage efficient.

**Use `js_render` for dynamic content:** <br />If a page loads its content via JavaScript (single-page apps, lazy-loaded sections), the filters will only capture what's present in the initial HTML unless you also pass `js_render=true`.

**Use `wait` or `wait_for` for delayed content:** <br />If the content is loaded after a delay or via JavaScript, use `wait` or `wait_for` to wait for the content to load before extracting it.

<Tip>Find more details on our [Wait Documentation](/fetch/features/wait) or [Wait For Selector Documentation](/fetch/features/wait-for).</Tip>

**Validate output shape before parsing:** <br />Some filters return a flat array (like `emails` or `links`) while others return structured objects (like `tables`). Check the response shape for each filter before building your parser.

**Handle empty results gracefully:** <br />If a filter finds no matching elements on the page, it returns an empty array. Credits are still charged for any request that returns a `200` status code, regardless of whether the filters matched anything. Make sure your pipeline verifies that the target page actually contains the data you need before running it at scale.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The response is empty or missing expected data">
    This usually means the target content is loaded dynamically via JavaScript. Add `js_render=true` to your request to render the page before extraction. If the data still doesn't appear, inspect the raw HTML response first (without `outputs`) to confirm the elements are present in the DOM.

    Also, enable `js_render=true` and `wait` or `wait_for` to wait for the content to load before extracting it. You can find more details on our [Troubleshooting Partial or Empty Responses](/fetch/troubleshooting/empty-or-partialresponses) guide.
  </Accordion>

  <Accordion title="emails returns obfuscated or partial addresses">
    Zenrows detects common obfuscation patterns like `[at]` and `at domain dot com`, but some sites use custom JavaScript-based obfuscation that only resolves after rendering. Enable `js_render=true` to capture those cases. If the email is injected purely via client-side script with no HTML representation, it may not be extractable with output filters alone.
  </Accordion>

  <Accordion title="tables returns unexpected or malformed data">
    The `tables` filter targets standard HTML `table` elements. If the site renders its tables as `div` grids or uses CSS to simulate table layout, the filter won't pick them up. In that case, use `css_extractor` with a custom CSS selector instead.
  </Accordion>

  <Accordion title="links includes relative or irrelevant URLs">
    The `links` filter returns all `href` values from `a` tags as-is, including relative paths, anchor links (`#section`), and `mailto:` links. Filter the results in your own code to keep only the URLs relevant to your use case.
  </Accordion>

  <Accordion title="menus returns an empty array on a page that clearly has navigation">
    The `menus` filter targets `li` elements inside semantic `menu` tags. Many modern sites use `nav > ul > li` structures instead. If `menus` returns empty, use `css_extractor` with a selector like `nav a` to capture navigation links directly.
  </Accordion>
</AccordionGroup>

## Pricing

The `outputs` parameter is included at **no additional cost** with all Zenrows requests. You only pay extra for JavaScript Render and Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## FAQ (Frequently Asked Questions)

<Accordion title="Can I use output filters alongside other parameters like js_render or premium_proxy?">
  Yes, output filters work alongside the other Zenrows parameters. You can combine `outputs` with `js_render`, `premium_proxy`, `custom_headers`, and any other parameter your request needs. The exceptions are the other output formats: `extract`, `autoparse`, `css_extractor` and `response_type` each decide the response on their own, so only one of them applies per request.
</Accordion>

<Accordion title="Do output filters count as additional credits?">
  No, using the `outputs` parameter does not add extra credit cost. You pay the same credit rate as a standard request for that URL, regardless of how many filters you include.
</Accordion>

<Accordion title="What happens if I request a filter and the page has no matching elements?">
  The filter returns an empty array for that key, and the rest of your requested filters are still returned normally. Keep in mind that credits are charged based on a successful `200` response, not on whether the filters returned data. If you're running filters at scale, validate that the target pages actually contain the elements you're looking for before sending large batches.
</Accordion>

<Accordion title="Can I use outputs=* in production?">
  You can, but it's not recommended for high-volume pipelines. Requesting all filters adds unnecessary processing for data types you don't need. Use `outputs=*` during development to explore what's available, then narrow it down to the specific filters your use case requires before going to production.
</Accordion>

<Accordion title="Do output filters work with Browser Sessions?">
  Output filters are a feature of Fetch and are passed as query parameters. They are not available in Browser Sessions (CDP/Playwright) sessions, where you handle data extraction directly in your script using standard browser APIs.
</Accordion>

<Accordion title="Is there a limit to how many filters I can combine in one request?">
  There is no hard limit on the number of filters you can combine. You can pass all available filters in a single request using `outputs=*` or by listing them individually. In practice, the response size grows with each filter, so only request what you actually need.
</Accordion>


# PDF Response
Source: https://docs.zenrows.com/fetch/features/pdf

Get a pixel-perfect PDF render of any scraped page using the `response_type=pdf` parameter.

The `response_type=pdf` parameter tells Zenrows to render the target page and return it as a PDF file instead of HTML. The result is a pixel-perfect snapshot of the page as it appears in a browser, including styles, images, and JavaScript-rendered content.

This is useful when you need to archive pages, generate reports, produce shareable snapshots, or feed visual page content into document processing pipelines.

<Warning>
  `response_type=pdf` cannot be combined with the `outputs` parameter. Use one or the other depending on whether you need a rendered PDF or targeted data extraction.
</Warning>

## How it works

PDF generation always requires a full browser render. When you add `response_type=pdf` to your request, Zenrows automatically enables JavaScript rendering and captures the fully rendered page as a PDF. You must include `js_render=true` explicitly in your request for the PDF response to work.

<Tip>Find more details on our [JS Rendering Documentation](/fetch/features/js-rendering).</Tip>

The returned binary content is a standard PDF file that you can save to disk, upload to storage, or pass directly to a document processing tool.

## Basic usage

Add `response_type=pdf` and `js_render=true` to your request parameters:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  url = "https://www.scrapingcourse.com/ecommerce/"
  apikey = "YOUR_ZENROWS_API_KEY"

  params = {
      "url": url,
      "apikey": apikey,
      "js_render": "true",
      "response_type": "pdf",
  }

  response = requests.get("https://api.zenrows.com/v1/", params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  import axios from "axios";

  const url = "https://www.scrapingcourse.com/ecommerce/";
  const apikey = "YOUR_ZENROWS_API_KEY";

  const response = await axios.get("https://api.zenrows.com/v1/", {
    params: {
      url,
      apikey,
      js_render: "true",
      response_type: "pdf",
    },
  });

  console.log(response.data);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&response_type=pdf";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&response_type=pdf");
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&response_type=pdf", nil)
      if err != nil {
          log.Fatalln(err)
      }

      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  require "faraday"

  url = URI.parse("https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&response_type=pdf")
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&response_type=pdf"
  ```
</CodeGroup>

## Saving the PDF to disk

The API returns the PDF as binary content. Use `response.content` (not `response.text`) to write it correctly to a file:

```python Python theme={"dark"}
import requests

url = "https://www.scrapingcourse.com/ecommerce/"
apikey = "YOUR_ZENROWS_API_KEY"

params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",
    "response_type": "pdf",
}

response = requests.get("https://api.zenrows.com/v1/", params=params)

with open("output.pdf", "wb") as file:
    file.write(response.content)

print("PDF saved to output.pdf")
```

<Warning>
  Always write the response as binary (`"wb"` mode in Python). Writing it as text will corrupt the PDF file.
</Warning>

## When to use PDF response

PDF response is the right choice when you need a visual, printable, or archivable snapshot of a page. It works well for:

* **Page archiving**: capture a timestamped visual record of a page's content and layout.
* **Report generation**: convert data-rich pages, dashboards, or invoices into portable PDF documents.
* **Compliance and auditing**: preserve the rendered state of a page as it appeared at a specific point in time.
* **Document pipelines**: feed rendered page snapshots into PDF processing tools, OCR engines, or document management systems.
* **Sharing and distribution**: generate a shareable, print-ready version of a page without requiring the recipient to visit the URL.

<Note>If you need to extract text or structured data from the page rather than a visual snapshot, use `response_type=markdown`, `response_type=plaintext`, or the `outputs` parameter instead.</Note>

## Best practices

**Always include `js_render=true`:** <br />
PDF generation requires a full browser render. Requests without `js_render=true` will not return a valid PDF.

**Use `wait` or `wait_for` for pages with delayed content:** <br />
If the page loads content after the initial render (lazy-loaded images, deferred scripts), use `wait` (milliseconds) or `wait_for` (CSS selector) to ensure the full page is visible before the PDF is captured.

<Tip>Find more details on our [Wait Documentation](/fetch/features/wait) or [Wait For Selector Documentation](/fetch/features/wait-for).</Tip>

**Save as binary, not text:** <br />
PDF files are binary. Always write the response body in binary mode (for example, `"wb"` mode in Python and Ruby, `Buffer` in Node.js, `FileOutputStream` in Java). Writing binary content as text will produce a corrupt file.

**Credits are charged on successful responses:** <br />
A request using `response_type=pdf` is charged when the API returns a `200` status code, regardless of whether the PDF contains the content you expected. Because `js_render=true` is required, the JS rendering credit cost also applies. Test on a small set of URLs before running at scale.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The PDF is blank or missing content">
    The page likely loads content after the initial render. Add `wait` (in milliseconds) to give the page time to finish loading, or use `wait_for` with a CSS selector that is only present once the main content is visible. Also confirm that `js_render=true` is included in the request.
  </Accordion>

  <Accordion title="The saved PDF file is corrupted or unreadable">
    You are likely writing the response as text instead of binary. Always use binary write mode in your language of choice (`"wb"` in Python and Ruby, `Buffer` in Node.js, `FileOutputStream` in Java) and write the raw response body. PDF is a binary format and writing it as text will corrupt the file.
  </Accordion>

  <Accordion title="I'm getting an error when combining response_type=pdf with outputs">
    These two parameters are mutually exclusive. `response_type=pdf` returns a rendered PDF of the full page, while `outputs` performs targeted data extraction from the HTML. You can only use one per request.
  </Accordion>

  <Accordion title="The PDF does not include images or styles">
    This usually happens when external resources (stylesheets, images, fonts) fail to load during the render. Make sure the target URL is publicly accessible and that `js_render=true` is included. If the page uses resources that require authentication, those assets may not render correctly.
  </Accordion>
</AccordionGroup>

## Pricing

The `response_type=pdf` parameter is included at **no additional cost** beyond the standard JS rendering rate. Because PDF generation requires `js_render=true`, all PDF requests are billed at the JS rendering credit rate. You only pay extra for Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## FAQ (Frequently Asked Questions)

<Accordion title="Is js_render=true required for PDF response?">
  Yes. PDF generation requires a full browser render. You must include `js_render=true` in every request that uses `response_type=pdf`. Requests without it will not return a valid PDF.
</Accordion>

<Accordion title="Does response_type=pdf cost more credits?">
  PDF requests are billed at the JS rendering credit rate because `js_render=true` is required. The `response_type=pdf` parameter itself does not add an extra cost beyond that.
</Accordion>

<Accordion title="Can I use response_type=pdf with Browser Sessions?">
  No, `response_type=pdf` is a parameter of Fetch. It is not available in Browser Sessions (CDP/Playwright) sessions. In a Browser Sessions session, you can generate PDFs using Playwright's built-in `page.pdf()` method directly in your script.
</Accordion>

<Accordion title="Can I control the PDF page size or orientation?">
  Not currently. The PDF is generated using the browser's default page settings. If you need specific page dimensions or orientation, consider post-processing the PDF with a library like `pypdf` or `reportlab` after saving it.
</Accordion>

<Accordion title="Can I use response_type=pdf to capture paginated content?">
  The PDF captures a single page render. For paginated content, you would need to make a separate request for each URL or page state you want to capture, then combine the resulting PDFs if needed.
</Accordion>


# Plain Text Response
Source: https://docs.zenrows.com/fetch/features/plaintext

Get scraped page content returned as plain text, stripped of all HTML tags and formatting, using the `response_type=plaintext` parameter.

The `response_type=plaintext` parameter tells Zenrows to strip all HTML tags and formatting from the scraped page and return only the raw text content. You get the words on the page, nothing else.

This is useful when you don't need structure or markup. Just the text itself. It's a good fit for keyword analysis, sentiment analysis, full-text search indexing, or any pipeline where clean, unformatted content is easier to work with than HTML or Markdown.

<Warning>
  `response_type=plaintext` cannot be combined with the `outputs` parameter. Use one or the other depending on whether you need targeted data extraction or a plain text version of the full page.
</Warning>

## How it works

Zenrows fetches the target page, renders it if needed, and strips all HTML tags from the resulting content. The output preserves line breaks between block-level elements but removes all formatting: no headings, no bold, no links, no list syntax.

For example, given the following HTML:

```html HTML theme={"dark"}
<h1>Product Title</h1>
<p>This is a great product that does many things.</p>
<ul>
    <li>Feature 1</li>
    <li>Feature 2</li>
    <li>Feature 3</li>
</ul>
```

The plain text response returns:

```text Plain text theme={"dark"}
Product Title

This is a great product that does many things.

Feature 1
Feature 2
Feature 3
```

## Basic usage

Add `response_type=plaintext` to your request parameters:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  url = "https://www.scrapingcourse.com/ecommerce/"
  apikey = "YOUR_ZENROWS_API_KEY"

  params = {
      "url": url,
      "apikey": apikey,
      "response_type": "plaintext",
  }

  response = requests.get("https://api.zenrows.com/v1/", params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  import axios from "axios";

  const url = "https://www.scrapingcourse.com/ecommerce/";
  const apikey = "YOUR_ZENROWS_API_KEY";

  const response = await axios.get("https://api.zenrows.com/v1/", {
    params: {
      url,
      apikey,
      response_type: "plaintext",
    },
  });

  console.log(response.data);
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=plaintext";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();
          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=plaintext");
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=plaintext", nil)
      if err != nil {
          log.Fatalln(err)
      }

      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  require "faraday"

  url = URI.parse("https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=plaintext")
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&response_type=plaintext"
  ```
</CodeGroup>

## When to use plain text response

Plain text is the right choice when structure doesn't matter and you just need the words. It works well for:

* **NLP and text analysis**: run sentiment analysis, keyword extraction, or topic modeling directly on the output without pre-processing HTML.
* **Full-text search indexing**: feed clean text into search engines like Elasticsearch or Typesense without stripping tags in your pipeline.
* **LLM pipelines with minimal context**: when you want to reduce token usage further than Markdown allows and don't need headings or list structure preserved.
* **Content comparison and diffing**: compare page text across time or across URLs without formatting noise affecting the diff.
* **Readability and summarization tools**: pass clean prose to summarization models or readability scoring systems.

<Note>If you need the page structure preserved (headings, lists, links), use `response_type=markdown` instead. If you only need specific data types like emails or tables, use the `outputs` parameter.</Note>

## Best practices

**Enable `js_render` for dynamic pages:** <br />
If the page loads content via JavaScript, combine `response_type=plaintext` with `js_render=true`. Without it, the output will only reflect the initial HTML and may be missing key content.

**Use `wait` or `wait_for` when content loads after a delay:** <br />
For pages where content appears after a delay or user interaction, add `wait` (milliseconds) or `wait_for` (CSS selector) to ensure the full content is present before the text is extracted.

<Tip>Find more details on our [Wait Documentation](/fetch/features/wait) or [Wait For Selector Documentation](/fetch/features/wait-for).</Tip>

**Expect navigation and boilerplate in the output:** <br />
Plain text response strips tags but converts the full page, including headers, footers, navigation menus, and sidebars. If you need only the main body content, post-process the output to remove boilerplate, or evaluate whether `css_extractor` with a targeted selector fits your use case better.

<Tip>Find more details on our [CSS Extractor Documentation](/fetch/features/css-extractor).</Tip>

**Credits are charged on successful responses:** <br />
A request using `response_type=plaintext` is charged when the API returns a `200` status code, regardless of whether the output contains the content you expected. Test on a small set of URLs before running at scale.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The plain text output is missing most of the page content">
    The page likely renders its content via JavaScript. Add `js_render=true` to your request to ensure Zenrows renders the page fully before extracting the text. If content still appears incomplete, combine with `wait_for` targeting a CSS selector that's only present once the main content has loaded.
  </Accordion>

  <Accordion title="The output contains a lot of navigation, menu items, or footer text">
    Plain text response converts the full HTML of the page, not just the main body. This is expected behavior. To target only a specific section, use `css_extractor` with a selector pointing to the main content container (for example, `article` or `main`) and process the extracted HTML as needed.
  </Accordion>

  <Accordion title="I'm getting an error when combining response_type=plaintext with outputs">
    These two parameters are mutually exclusive. `response_type=plaintext` converts the entire page to plain text, while `outputs` performs targeted data extraction. You can only use one per request. Choose `outputs` when you need specific data types, and `response_type=plaintext` when you need clean, unformatted text from the full page.
  </Accordion>

  <Accordion title="The text output has inconsistent whitespace or blank lines">
    Plain text conversion preserves line breaks between block-level elements, which can result in multiple blank lines between sections depending on how the page is structured. Post-process the output with a simple whitespace normalization step in your code to clean it up before further processing.
  </Accordion>
</AccordionGroup>

## Pricing

The `response_type=plaintext` parameter is included at **no additional cost** with all Zenrows requests. You only pay extra for JavaScript Render and Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## FAQ (Frequently Asked Questions)

<Accordion title="Does response_type=plaintext work with js_render?">
  Yes, you can combine `response_type=plaintext` with `js_render=true` to render JavaScript before the text extraction. This is recommended for any page that loads content dynamically.
</Accordion>

<Accordion title="What is the difference between plaintext and markdown response types?">
  Both strip HTML tags, but `response_type=markdown` preserves the semantic structure of the page by converting headings, lists, links, and tables into Markdown syntax. `response_type=plaintext` removes all formatting entirely and returns only the raw text. Use Markdown when structure matters, and plain text when you just need the words.
</Accordion>

<Accordion title="Does it cost more credits to use response_type=plaintext?">
  No, the `response_type=plaintext` parameter does not add extra credit cost on its own. Credits are calculated based on the other parameters you use, such as `js_render` or `premium_proxy`, exactly as they would be for a standard request.
</Accordion>

<Accordion title="Can I use response_type=plaintext with Browser Sessions?">
  No, `response_type=plaintext` is a parameter of Fetch. It is not available in Browser Sessions (CDP/Playwright) sessions. In a Browser Sessions session, you control the page directly and can extract text content using standard browser APIs like `document.body.innerText`.
</Accordion>

<Accordion title="Can I use response_type=plaintext to feed content into an LLM?">
  Yes. Plain text is the most token-efficient format for LLM input since it contains no markup at all. It works well when the structure of the page is irrelevant to your task. If your LLM prompt benefits from knowing which text was a heading or a list item, use `response_type=markdown` instead.
</Accordion>


# Premium Proxy (Residential IPs)
Source: https://docs.zenrows.com/fetch/features/premium-proxy

Bypass anti-bot detection by routing scraping requests through 55 million residential IP addresses with Zenrows Premium Proxy for higher success.

Premium Proxy routes your requests through residential IP addresses instead of standard datacenter IPs. These residential IPs come from real Internet Service Provider (ISP) connections assigned to actual households, making them significantly harder for websites to detect and block.

When websites implement anti-scraping measures, they typically target datacenter IP ranges first. Premium Proxy addresses this challenge by utilizing IP addresses that appear as regular user traffic, thereby dramatically improving your scraping success rates on protected websites.

## How Premium Proxy works

Premium Proxy automatically selects residential IP addresses from Zenrows' global pool when the feature is enabled. Each request routes through a different residential connection, mimicking natural user behavior and avoiding the IP-based blocking that affects datacenter proxies.

The system handles IP rotation automatically, so you don't need to manage proxy lists or worry about IP exhaustion. Zenrows maintains thousands of residential IPs across multiple countries to ensure reliable access to your target websites.

## Basic usage

Enable Premium Proxy by adding the `premium_proxy=true` parameter to your Zenrows request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'premium_proxy': 'true',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'premium_proxy': 'true',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true"
  ```
</CodeGroup>

This example enables Premium Proxy for your request. Zenrows automatically selects an available residential IP and routes your request through it, improving your chances of successful data extraction.

## When to use Premium Proxy

Premium Proxy is essential in these scenarios:

* **High-security websites** - Sites protected by CloudFlare, Akamai, or similar anti-bot systems
* **E-commerce platforms** - Online stores that actively block datacenter IPs
* **Social media sites** - Platforms with sophisticated bot detection
* **Financial websites** - Banking and trading sites with strict access controls
* **Streaming services** - Video platforms that block automated access
* **Government websites** - Official sites with enhanced security measures

## Best practices

### Combine with other features

Premium Proxy works best when combined with other Zenrows features:

```python Python theme={"dark"}
params = {
    'url': target_url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'premium_proxy': 'true',
    'js_render': 'true',  # For JavaScript-heavy sites
    'wait_for': '.content',  # Wait for specific elements
}
```

### Monitor success rates

Track your scraping success to optimize your approach:

```python Python theme={"dark"}
successful_requests = 0
total_requests = 0

for url in url_list:
    total_requests += 1
    response = requests.get('https://api.zenrows.com/v1/', params={
        'url': url,
        'apikey': 'YOUR_ZENROWS_API_KEY',
        'premium_proxy': 'true',
    })
    
    if response.status_code == 200:
        successful_requests += 1

success_rate = (successful_requests / total_requests) * 100
print(f"Success rate: {success_rate}%")
```

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Still getting blocked | Website uses advanced fingerprinting | Add `js_render=true` and custom headers |
| Inconsistent results | Different residential IPs may see different content | Use session management or sticky sessions ([session\_id](/fetch/features/other#session-id)), or test using [Geolocation](/fetch/features/proxy-country) |
| Higher costs than expected | Premium Proxy costs 10 credits per request | Monitor usage and optimize request frequency |

### Debugging blocked requests

If you're still getting blocked with Premium Proxy enabled:

<Steps>
  <Step title="Add JavaScript rendering">
    ```python Python theme={"dark"}
    params = {
        'premium_proxy': 'true',
        'js_render': 'true',
    }
    ```

    <Tip>
      See more about the `js_render` parameter [here](./js-rendering)
    </Tip>
  </Step>

  <Step title="Return the original status from the website">
    ```python Python theme={"dark"}
    params = {
        'premium_proxy': 'true',
        'original_status': 'true',
    }
    ```

    <Tip>
      See more about the `original_status` parameter [here](/fetch/features/other#original-http-code)
    </Tip>
  </Step>

  <Step title="Include realistic headers">
    ```python Python theme={"dark"}
    params = {
        'premium_proxy': 'true',
        'js_render': 'true',
        'custom_headers': 'true',
    }
    headers = {
        'referer': 'https://www.google.com'
    }
    ```

    <Tip>
      See more about the `custom_headers` parameter [here](/fetch/features/headers)
    </Tip>
  </Step>

  <Step title="Add wait times for dynamic content">
    ```python Python theme={"dark"}
    params = {
        'premium_proxy': 'true',
        'js_render': 'true',
        'wait': '3000',
    }
    ```
  </Step>
</Steps>

## Pricing

Premium Proxy requests cost 10 times the standard rate. This pricing reflects the higher cost and limited availability of residential IP addresses compared to datacenter proxies.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="How does Premium Proxy differ from standard proxies?">
  Premium Proxy utilizes residential IP addresses from genuine ISP connections, whereas standard proxies employ data center IPs. Residential IPs are much harder for websites to detect and block, resulting in higher success rates for scraping protected websites.
</Accordion>

<Accordion title="Can I specify which residential IP to use?">
  No, Zenrows automatically selects residential IPs from its pool to ensure optimal performance and avoid IP exhaustion. Manual IP selection could lead to blocking and reduced success rates.
</Accordion>

<Accordion title="Do I need Premium Proxy for all websites?">
  No, Premium Proxy is primarily needed for websites with anti-scraping measures. Simple websites without bot protection can often be scraped successfully using standard datacenter proxies.
</Accordion>

<Accordion title="Can I use Premium Proxy with custom headers?">
  Yes, Premium Proxy works with all other Zenrows features, including custom headers, JavaScript rendering, and wait parameters. Combining features often provides the best results.
</Accordion>


# Proxy Country (Geolocation)
Source: https://docs.zenrows.com/fetch/features/proxy-country

Route web scraping requests through residential IPs in 190+ countries to access geo-restricted content and localized data with Zenrows.

Proxy Country allows you to specify the geographic location from which your scraping requests originate. This feature enables access to geo-restricted content, region-specific pricing, and localized search results by routing your requests through residential IP addresses in your chosen country.

Many websites serve different content based on the visitor's location. Proxy Country addresses this challenge by allowing your requests to appear to originate from any of 190+ supported countries, thereby providing access to region-specific data that would otherwise be unavailable.

## How Proxy Country works

When you specify a country code with the `proxy_country` parameter, Zenrows routes your request through a residential IP address located in that country. The target website recognizes your request as coming from a real user in the specified location, enabling you to access geo-restricted content and view localized results.

Proxy Country requires Premium Proxy to be enabled, as only residential IP addresses can provide accurate geolocation. Datacenter proxies cannot reliably represent specific geographic locations.

## Basic usage

Enable Proxy Country by adding both `premium_proxy=true` and `proxy_country` parameters to your request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'premium_proxy': 'true',
      'proxy_country': 'es',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'premium_proxy': 'true',
          'proxy_country': 'es',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true&proxy_country=es";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true&proxy_country=es');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true&proxy_country=es", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true&proxy_country=es')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&premium_proxy=true&proxy_country=es"
  ```
</CodeGroup>

## Common use cases

### E-commerce price monitoring

Monitor pricing across different regions:

```python Python theme={"dark"}
countries = ['us', 'gb', 'de', 'fr', 'jp']
product_url = 'https://example-store.com/product/123'

for country in countries:
    response = requests.get('https://api.zenrows.com/v1/', params={
        'url': product_url,
        'apikey': 'YOUR_ZENROWS_API_KEY',
        'premium_proxy': 'true',
        'proxy_country': country,
    })
    
    # Extract and compare prices for each region
    print(f"Price in {country}: {extract_price(response.text)}")
```

### Search engine localization

Get search results as they appear in different countries:

```python Python theme={"dark"}
search_query = 'best restaurants'
search_url = f'https://www.google.com/search?q={search_query}'

# Get US results
us_response = requests.get('https://api.zenrows.com/v1/', params={
    'url': search_url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'premium_proxy': 'true',
    'proxy_country': 'us',
})

# Get ES results  
uk_response = requests.get('https://api.zenrows.com/v1/', params={
    'url': search_url,
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'premium_proxy': 'true',
    'proxy_country': 'es',
})
```

### Content availability testing

Check if content is accessible from different regions:

```python Python theme={"dark"}
def test_content_availability(url, countries):
    results = {}
    
    for country in countries:
        response = requests.get('https://api.zenrows.com/v1/', params={
            'url': url,
            'apikey': 'YOUR_ZENROWS_API_KEY',
            'premium_proxy': 'true',
            'proxy_country': country,
        })
        
        results[country] = {
            'accessible': response.status_code == 200,
            'status_code': response.status_code,
        }
    
    return results

# Test video availability across regions
video_url = 'https://streaming-service.com/video/123'
availability = test_content_availability(video_url, ['us', 'ca', 'gb', 'au'])
```

## Best practices

### Choose appropriate countries

Select countries based on your specific needs:

* US (`us`) - For American content
* GB (`gb`) - For UK-specific results
* DE (`de`) - For German/EU content
* JP (`jp`) - For Japanese content data
* CA (`ca`) - For Canadian content

<Tip>See our complete list of countries [here](/first-steps/faq#what-is-geolocation-and-what-are-all-the-premium-proxy-countries)!</Tip>

### Handle region-specific layouts

Some websites serve different HTML structures by region:

```python Python theme={"dark"}
def extract_data_by_region(html_content, country):
    if country in ['us', 'ca']:
        # North American layout
        return extract_with_us_selectors(html_content)
    elif country in ['gb', 'de', 'fr']:
        # European layout
        return extract_with_eu_selectors(html_content)
    else:
        # Default extraction
        return extract_with_default_selectors(html_content)
```

### Implement country fallbacks

If a specific country doesn't work, try alternatives:

```python Python theme={"dark"}
def scrape_with_fallback(url, preferred_countries):
    for country in preferred_countries:
        try:
            response = requests.get('https://api.zenrows.com/v1/', params={
                'url': url,
                'apikey': 'YOUR_ZENROWS_API_KEY',
                'premium_proxy': 'true',
                'proxy_country': country,
            })
            
            if response.status_code == 200:
                return response, country
                
        except Exception as e:
            print(f"Failed with {country}: {e}")
            continue
    
    return None, None

# Try US first, then Canada, then UK
response, used_country = scrape_with_fallback(
    'https://example.com', 
    ['us', 'ca', 'gb']
)
```

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Content still geo-blocked | Website blocks entire regions | Try neighboring countries or different continents |
| Different layout than expected | Region-specific website versions | Update CSS selectors for each region |
| Slower response times | Distance from target servers | Choose countries closer to the website's servers |
| Inconsistent results | Regional content variations | Implement region-specific data extraction logic |

### Debugging geo-restrictions

When content remains blocked despite using Proxy Country:

<Steps>
  <Step title="Try multiple countries in the same region">
    ```python Python theme={"dark"}
    eu_countries = ['de', 'fr', 'it', 'es', 'nl']
    for country in eu_countries:
        # Test each country
    ```
  </Step>

  <Step title="Check if the entire region is blocked">
    ```python Python theme={"dark"}
    test_countries = ['us', 'gb', 'de', 'jp', 'au', 'br']
    # Test diverse geographic locations
    ```
  </Step>

  <Step title="Combine with additional features">
    * [js\_render](/fetch/features/js-rendering): Makes the request act like a headles browser
    * [wait\_for](/fetch/features/wait-for): Waits for a specific CSS element on the target url
    * [custom\_headers](/fetch/features/headers): Enable using custom headers on the request

    ```python Python theme={"dark"}
    params = {
        'premium_proxy': 'true',
        'proxy_country': 'us',
        'js_render': 'true', # For JavaScript-heavy sites
        'wait_for': '.content',  # Wait for specific elements
        'custom_headers': 'true', # Enable using custom headers on the request
    }
    headers = {
        'referer': 'https://www.google.com'
    }
    ```
  </Step>
</Steps>

## Pricing

Proxy Country doesn't increase the request cost. You pay the Premium Proxy (10 times the standard price) regardless of the country you choose.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="Can I use Proxy Country without Premium Proxy?">
  No, Proxy Country requires Premium Proxy to be enabled. Only residential IP addresses can provide accurate geolocation, and Premium Proxy is necessary to access residential IPs.
</Accordion>

<Accordion title="How accurate is the geolocation?">
  Zenrows utilizes genuine residential IP addresses from ISPs in each country, ensuring highly accurate geolocation that websites recognize as legitimate traffic from those specific regions.
</Accordion>

<Accordion title="Can I specify a city or state within a country?">
  Currently, Zenrows supports country-level geolocation only. You cannot specify specific cities or states within a country.
</Accordion>

<Accordion title="What happens if a country is unavailable?">
  If no residential IPs are available for your specified country, the request will fail. Implement fallback logic to try alternative countries in such cases.
</Accordion>

<Accordion title="Do different countries have different success rates?">
  Yes, success rates can vary by country depending on the target website's blocking policies and the quality of residential IP pools in each region. Monitor your success rates and adjust accordingly.
</Accordion>

<Accordion title="Can I use the same country for all requests in a session?">
  Yes, you can use the same country consistently. However, some websites may still detect patterns, so consider rotating between countries in the same region if needed.
</Accordion>


# Page Screenshot
Source: https://docs.zenrows.com/fetch/features/screenshot

Capture a screenshot of any page (above the fold, full page, or a specific element) using the `screenshot=true` parameter.

The `screenshot=true` parameter tells Zenrows to capture a screenshot of the target page and return it as a PNG image. You get a pixel-accurate visual snapshot of the page as it appears in a browser, including styles, fonts, and JavaScript-rendered content.

This is useful for visual monitoring, page archiving, UI testing, or any workflow that needs a rendered image of a page rather than its raw content.

<Warning>
  Screenshot parameters cannot be combined with the `outputs` parameter. Use one or the other depending on whether you need a visual capture or targeted data extraction.
</Warning>

## How it works

Screenshot generation always requires a full browser render. You must include `js_render=true` in every screenshot request. Zenrows renders the page in a headless browser and captures the result as an image before returning it as binary content.

By default, the screenshot captures only the above-the-fold visible area of the page. You can expand this to the full page or narrow it to a specific element using the additional parameters described below.

<Tip>Find more details on our [JS Rendering Documentation](/fetch/features/js-rendering).</Tip>

## Basic usage

Add `screenshot=true` and `js_render=true` to your request parameters:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  url = "https://www.scrapingcourse.com/ecommerce/"
  apikey = "YOUR_ZENROWS_API_KEY"

  params = {
      "url": url,
      "apikey": apikey,
      "js_render": "true",
      "screenshot": "true",
  }

  response = requests.get("https://api.zenrows.com/v1/", params=params)

  with open("screenshot.png", "wb") as file:
      file.write(response.content)

  print("Screenshot saved to screenshot.png")
  ```

  ```javascript Node.js theme={"dark"}
  import axios from "axios";
  import fs from "fs";

  const url = "https://www.scrapingcourse.com/ecommerce/";
  const apikey = "YOUR_ZENROWS_API_KEY";

  const response = await axios.get("https://api.zenrows.com/v1/", {
  params: {
      url,
      apikey,
      js_render: "true",
      screenshot: "true",
  },
  responseType: "stream",
  });

  const file = fs.createWriteStream("screenshot.png");
  response.data.pipe(file);
  ```

  ```java Java theme={"dark"}
  import java.io.File;
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          File file = new File("screenshot.png");
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&screenshot=true";
          Request.get(apiUrl).execute().saveContent(file);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&screenshot=true");
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  file_put_contents("screenshot.png", $response);
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
      "os"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&screenshot=true", nil)
      if err != nil {
          log.Fatalln(err)
      }

      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      img, _ := os.Create("screenshot.png")
      defer img.Close()

      io.Copy(img, resp.Body)
  }
  ```

  ```ruby Ruby theme={"dark"}
  require "faraday"

  url = URI.parse("https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&screenshot=true")
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  File.open("screenshot.png", "wb") { |file| file.write(res.body) }
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&screenshot=true" > screenshot.png
  ```
</CodeGroup>

## Screenshot options

### Capture mode

Three capture modes are available depending on how much of the page you need:

| Parameter | Description |
| - | - |
| `screenshot=true` | Captures the above-the-fold visible area only. Default behavior. |
| `screenshot_fullpage=true` | Captures the entire page by scrolling and stitching the full content. |
| `screenshot_selector=<CSS Selector>` | Captures only the element matched by the given CSS selector. |

<Warning>
  `screenshot_fullpage` and `screenshot_selector` are mutually exclusive. Use one or the other in a single request.
</Warning>

### Image format and quality

By default, screenshots are returned as PNG. You can switch to JPEG for smaller file sizes using the following parameters:

| Parameter | Values | Description |
| - | - | - |
| `screenshot_format` | `png`, `jpeg` | Sets the output format. Defaults to `png`. |
| `screenshot_quality` | `1` to `100` | Sets the JPEG compression quality. Only applies when `screenshot_format=jpeg`. |

<Tip>Full-page screenshots can exceed 10MB as PNG. If you are hitting size limits, switch to `screenshot_format=jpeg` and tune `screenshot_quality` to find the right balance between file size and visual fidelity.</Tip>

### Combining with other parameters

Screenshot requests are compatible with `wait`, `wait_for`, and `js_instructions`, which lets you control exactly when the capture happens. Use these when the page loads content after the initial render or requires user interaction before the target content is visible.

<Tip>Find more details on our [Wait Documentation](/fetch/features/wait), [Wait For Selector Documentation](/fetch/features/wait-for), and [JS Instructions Documentation](/fetch/features/js-instructions).</Tip>

When using `json_response=true`, the screenshot data is returned base64-encoded inside a JSON object instead of as raw binary, which makes it easier to handle in API workflows that expect structured responses.

<Warning>
  `screenshot` is not compatible with `proxy_country`. Screenshot requests always route through a random IP, so you cannot target a specific country's proxy for the capture. You can still add `premium_proxy=true` to use Residential IPs instead of datacenter ones, but you cannot control their geolocation with `proxy_country`.
</Warning>

## When to use page screenshots

Screenshots are the right choice when you need a visual record of a page rather than its text or data. They work well for:

* **Visual monitoring**: detect layout changes, broken pages, or UI regressions across time.
* **Page archiving**: preserve a timestamped visual snapshot of how a page looked at a specific moment.
* **Element capture**: isolate and capture specific UI components, charts, or widgets using `screenshot_selector`.
* **QA and testing**: validate that pages render correctly across different URLs or conditions.
* **Reporting and sharing**: generate visual page previews for dashboards, reports, or stakeholder communication.

<Note>If you need to extract text or structured data from the page rather than a visual snapshot, use `response_type=markdown`, `response_type=plaintext`, or the `outputs` parameter instead.</Note>

## Best practices

**Always include `js_render=true`:** <br />
Screenshot generation requires a full browser render. Requests without `js_render=true` will not return a valid image.

**Use `wait` or `wait_for` for pages with delayed content:** <br />
If the page loads content after the initial render, use `wait` (milliseconds) or `wait_for` (CSS selector) to ensure everything is visible before the screenshot is captured.

<Tip>Find more details on our [Wait Documentation](/fetch/features/wait) or [Wait For Selector Documentation](/fetch/features/wait-for).</Tip>

**Switch to JPEG for large full-page screenshots:** <br />
Full-page screenshots of long pages can exceed 10MB as PNG, which causes errors. Use `screenshot_format=jpeg` with `screenshot_quality` set between `60` and `85` for a good balance of clarity and file size.

**Save as binary, not text:** <br />
Screenshot files are binary. Always write the response body in binary mode (for example, `"wb"` mode in Python and Ruby, `Buffer` in Node.js, `FileOutputStream` in Java). Writing binary content as text will corrupt the image file.

**Credits are charged on successful responses:** <br />
A screenshot request is charged when the API returns a `200` status code, regardless of whether the image contains the content you expected. Because `js_render=true` is required, the JS rendering credit cost also applies. Test on a small set of URLs before running at scale.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The screenshot is blank or shows a loading state">
    The page likely loads content after the initial render. Add `wait` (in milliseconds) to give the page time to finish loading, or use `wait_for` with a CSS selector that is only present once the main content is visible.
  </Accordion>

  <Accordion title="The screenshot file is corrupted or cannot be opened">
    You are likely writing the response as text instead of binary. Always use binary write mode in your language of choice (`"wb"` in Python and Ruby, `Buffer` in Node.js, `FileOutputStream` in Java) and write the raw response body. Image files are binary and writing them as text will produce a corrupt file.
  </Accordion>

  <Accordion title="The full-page screenshot is too large or returns an error">
    Full-page screenshots of long pages can exceed 10MB as PNG. Switch to `screenshot_format=jpeg` and set `screenshot_quality` to a value between `60` and `85` to reduce the file size significantly while keeping the image readable.
  </Accordion>

  <Accordion title="screenshot_selector is not capturing the right element">
    Make sure the CSS selector you are passing targets a single, visible element. If the element is loaded dynamically, use `wait_for` with the same selector to ensure it is present in the DOM before the screenshot is taken. Avoid auto-generated or highly dynamic class names, as these can change between requests.
  </Accordion>

  <Accordion title="screenshot_fullpage and screenshot_selector are conflicting">
    These two parameters are mutually exclusive. Use `screenshot_fullpage=true` when you want the entire page, and `screenshot_selector` when you want a specific element. You cannot use both in the same request.
  </Accordion>

  <Accordion title="proxy_country has no effect on my screenshot request">
    `proxy_country` is not supported for screenshot requests. Screenshots always use a random IP, regardless of the country you request. If you need Residential IPs for a screenshot, use `premium_proxy=true` on its own, without `proxy_country`.
  </Accordion>
</AccordionGroup>

## Pricing

Screenshot requests are included at **no additional cost** beyond the standard JS rendering rate. Because screenshot generation requires `js_render=true`, all screenshot requests are billed at the JS rendering credit rate. You only pay extra for Premium Proxy when used.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## FAQ (Frequently Asked Questions)

<Accordion title="Is js_render=true required for screenshots?">
  Yes. Screenshot generation requires a full browser render. You must include `js_render=true` in every screenshot request. Requests without it will not return a valid image.
</Accordion>

<Accordion title="Do screenshots cost more credits?">
  Screenshot requests are billed at the JS rendering credit rate because `js_render=true` is required. The screenshot parameters themselves do not add an extra cost beyond that.
</Accordion>

<Accordion title="Can I capture a screenshot of a specific element on the page?">
  Yes. Use `screenshot_selector=<CSS Selector>` to capture only the element matched by the selector. Note that `screenshot_selector` and `screenshot_fullpage` are mutually exclusive and cannot be used together.
</Accordion>

<Accordion title="What image formats are supported?">
  PNG and JPEG are supported via the `screenshot_format` parameter. PNG is the default and is best for high-fidelity captures. JPEG offers smaller file sizes and is recommended for full-page screenshots or high-volume pipelines where storage and bandwidth matter.
</Accordion>

<Accordion title="Can I use screenshots with Browser Sessions?">
  No, the `screenshot` parameters are features of Fetch. In Browser Sessions (CDP/Playwright) sessions, you can capture screenshots directly using Playwright's built-in `page.screenshot()` method.
</Accordion>

<Accordion title="Can I get the screenshot as base64 instead of binary?">
  Yes. Add `json_response=true` to your request and the screenshot will be returned base64-encoded inside a JSON object. This is useful for workflows that expect structured API responses rather than raw binary streams.
</Accordion>

<Accordion title="Can I target a specific country's proxy for a screenshot?">
  No. `proxy_country` is not supported with `screenshot`, `screenshot_fullpage`, or `screenshot_selector`. Screenshot requests always use a random IP. You can still add `premium_proxy=true` to capture the screenshot through a Residential IP instead of a datacenter one, but you cannot choose its geolocation.
</Accordion>


# Wait Parameter
Source: https://docs.zenrows.com/fetch/features/wait

Add a fixed millisecond delay during headless browser rendering to ensure slow-loading dynamic content is fully captured before HTML extraction.

The Wait parameter introduces a fixed delay during JavaScript rendering, allowing web pages additional time to load dynamic content before Zenrows fully captures the HTML. Many websites load content progressively through JavaScript, AJAX calls, or other asynchronous processes that occur after the initial page has been rendered.

When you add the `wait` parameter to your request, Zenrows pauses for the specified duration (in milliseconds) after the page initially loads, ensuring that slow-loading elements, animations, and dynamically generated content have time to appear before the HTML is extracted. The maximum `wait` value acceptable is 30 seconds (30,000 milliseconds).

<Note>The Wait parameter requires `js_render=true` to function, as it operates within the browser environment during JavaScript rendering.</Note>

## How the Wait parameter works

The Wait parameter creates a pause in the scraping process after the initial page load completes. During this waiting period, the headless browser remains active, allowing JavaScript to continue executing, AJAX requests to complete, and dynamic content to render.

This process ensures you capture:

* Content loaded through delayed AJAX calls
* Elements that appear after animations complete
* Data populated by slow third-party APIs
* Progressive loading sequences
* Time-sensitive dynamic updates

The wait occurs after the initial DOM load but before the final HTML extraction, giving websites the necessary time to complete their loading processes.

## Basic usage

Add the `wait` parameter with a value in milliseconds to your JavaScript rendering request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'wait': '5000',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'js_render': 'true',
          'wait': '5000',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&wait=5000";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&wait=5000');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&wait=5000", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&wait=5000')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&js_render=true&wait=5000"
  ```
</CodeGroup>

This example waits 5 seconds (5000 milliseconds) after the page loads before capturing the HTML content. During this time, any JavaScript-driven content loading processes can complete.

## When to use the Wait parameter

The Wait parameter is essential in these scenarios:

### Content loading delays:

* **Slow API responses** - Pages that fetch data from slow external APIs
* **Progressive content loading** - Sites that load content in multiple stages
* **Heavy JavaScript processing** - Pages with complex calculations or data processing
* **Third-party widgets** - External content like maps, charts, or embedded media
* **Animation sequences** - Content that appears after CSS or JavaScript animations

### Timing-dependent content:

* **Real-time data** - Stock prices, live scores, or updating counters
* **Lazy loading** - Images or content that loads as needed
* **Scroll preparation** - Initial content that loads before scroll triggers
* **Form validation** - Dynamic form elements that appear based on user input simulation

## Best practices

### Start with reasonable defaults

Begin with moderate wait times and adjust based on results:

```python Python theme={"dark"}
def smart_wait_scraping(url, target_indicators):
    # Use progressive wait times to find the minimum necessary delay

    wait_times = [2000, 4000, 6000, 8000]  # Progressive wait times
    
    for wait_time in wait_times:
        response = requests.get('https://api.zenrows.com/v1/', params={
            'url': url,
            'apikey': 'YOUR_ZENROWS_API_KEY',
            'js_render': 'true',
            'wait': str(wait_time),
        })
        
        # Check if all target content is present
        content_found = all(indicator in response.text for indicator in target_indicators)
        
        if content_found:
            print(f"Optimal wait time found: {wait_time}ms")
            return response, wait_time
    
    print("Content not found even with maximum wait time")
    return None, None

# Usage
indicators = ['product-price', 'customer-reviews', 'stock-status']
result, optimal_wait = smart_wait_scraping('https://httpbin.io/anything', indicators)
```

### Combine with other parameters for maximum effectiveness

Use Wait with Premium Proxy for protected sites:

```python Python theme={"dark"}
params = {
    'url': 'https://httpbin.io/anything',
    'apikey': 'YOUR_ZENROWS_API_KEY',
    'js_render': 'true',
    'premium_proxy': 'true',
    'wait': '6000',
}
```

## Troubleshooting

### Common issues and solutions

| Issue | Cause | Solution |
| - | - | - |
| Content still missing | Wait time too short | Increase wait duration incrementally |
| Inconsistent results | Variable loading times | Use longer wait time, use `wait_for`, or implement retry logic |
| REQS004 | Invalid value provided for `wait` parameter; value is too big (REQS004) | Keep wait under 30 seconds total or use `wait_for` |
| Unnecessary delays | Fixed wait for fast-loading content | Use `wait_for` parameter for dynamic waiting |

### Debugging missing content with Wait

When content is still missing despite using the Wait parameter:

<Steps>
  <Step title="Check browser behavior manually">
    * Compare with manual browser testing
    * Open the page in a browser and time how long content takes to appear
    * Use that timing as your baseline wait duration
  </Step>

  <Step title="Increase wait time gradually">
    ```python Python theme={"dark"}
    # Try progressively longer wait times
    for wait_time in [3000, 6000, 9000, 12000]:
        response = requests.get('https://api.zenrows.com/v1/', params={
            'url': url,
            'apikey': 'YOUR_ZENROWS_API_KEY',
            'js_render': 'true',
            'wait': str(wait_time),
        })
        
        if 'expected-content' in response.text:
            print(f"Content found with {wait_time}ms wait")
            break
    ```
  </Step>

  <Step title="Switch to the element-specific waiting (`wait_for`)">
    ```python Python theme={"dark"}
    params = {
        'js_render': 'true',
        'wait_for': '.content',   # Wait for specific element
    }
    ```

    <Tip>
      See more on [Wait\_for](/fetch/features/wait-for) documentation page
    </Tip>
  </Step>
</Steps>

## Understanding wait time limits

Zenrows has built-in limits for wait times to ensure service stability:

* **Maximum total wait time**: 30 seconds (30,000 milliseconds)
* **Recommended range**: 2,000 - 10,000 milliseconds (2-10 seconds)
* **Minimum practical wait**: 1,000 milliseconds (1 second). Values below that are acceptable but impractical

## Pricing

The `wait` parameter doesn't increase the request cost. You pay the JavaScript Render (5 times the standard price) regardless of the wait value you choose.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="What's the difference between wait and wait_for parameters?">
  The `wait` parameter introduces a fixed delay in milliseconds, while `wait_for` waits for a specific element to appear on the page. Use `wait` when you know content takes a particular amount of time to load, and `wait_for` when you want to wait for specific elements to appear.
</Accordion>

<Accordion title="Can I use wait without js_render=true?">
  No, the `wait` parameter only works with JavaScript rendering enabled. Without `js_render=true`, the wait parameter will be ignored since there's no browser environment to pause within.
</Accordion>

<Accordion title="How do I know if my wait time is too short or too long?">
  Test different wait times and monitor content length, as well as the presence of target elements. If content is missing, increase the wait time. If you're getting complete content with shorter waits, you can reduce the time to improve efficiency.
</Accordion>

<Accordion title="Should I use the same wait time for all pages on a website?">
  Not necessarily. Different pages may have different loading characteristics. Product pages may require longer wait times than category pages. Consider implementing adaptive wait times based on page type or previous successful timings.
</Accordion>

<Accordion title="Can I use wait with other features?">
  Yes, the `wait` parameter works perfectly with any other feature.
</Accordion>

<Accordion title="What happens if I use both wait and wait_for?">
  When using both `wait` and `wait_for`, the `wait_for` parameter takes precedence and overrides the `wait` parameter.
</Accordion>


# Wait For Parameter
Source: https://docs.zenrows.com/fetch/features/wait-for

Use the Zenrows wait_for parameter with CSS selectors to pause scraping until specific page elements load during JavaScript rendering.

The Wait For parameter instructs Zenrows to pause JavaScript rendering until a specific element appears on the page, identified by a CSS selector. Unlike fixed delays, Wait For provides dynamic waiting that adapts to actual page loading conditions, making your scraping more reliable and efficient.

When you specify a CSS selector with the `wait_for` parameter, Zenrows continuously monitors the page during JavaScript rendering until the target element becomes visible in the DOM. This ensures that critical content has loaded before capturing the HTML, regardless of the duration of the loading process.

<Note>The Wait For parameter requires `js_render=true` to function, as it operates within the browser environment during JavaScript rendering.</Note>

## How the Wait For parameter works

The Wait For parameter actively monitors the page's DOM structure during JavaScript rendering, checking repeatedly for the presence of your specified CSS selector. Once the target element appears and becomes visible, Zenrows immediately captures the HTML content.

This process ensures you capture:

* Elements that load at unpredictable times
* Content dependent on API response timing
* Elements that appear after user interactions
* Dynamically generated content with variable loading speeds
* Critical page components that indicate full loading completion

The monitoring continues until either the element appears or the maximum timeout of 3 minutes is reached. If the selector is not found within this timeframe, Zenrows returns a 422 error indicating that the element could not be located.

## Basic usage

Add the `wait_for` parameter with a CSS selector to your JavaScript rendering request:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'wait_for': '.price',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'js_render': 'true',
          'wait_for': '.price',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&wait_for=.price";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&wait_for=.price');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&wait_for=.price", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&wait_for=.price')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&wait_for=.price"
  ```
</CodeGroup>

This example waits until an element with the class `price` appears on the page before capturing the HTML content. The waiting time automatically adapts to the actual loading speed, up to a maximum of 3 minutes.

## CSS selector examples

Use various CSS selectors to target different types of elements:

```python Python theme={"dark"}
# Wait for specific classes
params = {'wait_for': '.content-loaded'}

# Wait for IDs
params = {'wait_for': '#main-content'}

# Wait for specific attributes
params = {'wait_for': '[data-loaded="true"]'}

# Wait for nested elements
params = {'wait_for': '.product-container .price-display'}

# Wait for elements with specific text content
params = {'wait_for': '.status:contains("Available")'}

# Wait for form elements
params = {'wait_for': 'input[name="search"]'}

# Wait for multiple class combinations
params = {'wait_for': '.product.loaded.visible'}
```

## When to use the Wait For parameter

The Wait For parameter is essential in these scenarios:

### Dynamic content loading:

* **AJAX-loaded content** - Elements populated by asynchronous requests
* **API-dependent data** - Content that appears after external API calls
* **Progressive loading** - Pages that load content in stages
* **Conditional content** - Elements that appear based on user state or preferences
* **Real-time updates** - Content that updates based on live data feeds

### User interface elements:

* **Interactive components** - Buttons, forms, and controls that load dynamically
* **Navigation elements** - Menus and links that appear after initialization
* **Modal dialogs** - Pop-ups and overlays that appear programmatically
* **Data visualizations** - Charts, graphs, and tables that render after data loading
* **Search results** - Results that appear after query processing

## Troubleshooting

If Zenrows cannot find a matching element for the CSS selector, it will retry internally several times. If it still doesn't match after the 3-minute timeout, the request will return a 422 error. This means your selector likely does not exist in the final HTML, or is too fragile to be reliable.

### Common reasons for the 422 error when using wait\_for

### Selector Not Present in Final HTML

<Steps>
  <Step title="Inspect the site using browser DevTools">
    1. Open the page
    2. Right-click the target content and choose “Inspect”
    3. Check if your selector exists after the page fully loads
  </Step>

  <Step title="Verify your selector">
    1. Run `document.querySelectorAll('your_selector')` in the browser console
    2. If it returns no elements, your selector is incorrect

    <img alt="ScrapingCourse DevTools Selector Debug" />
  </Step>

  <Step title="Tips">
    1. Use simple selectors like `.class` or `#id`
    2. Prefer stable attributes like `[data-testid="item"]`
    3. Avoid overly specific or deep descendant selectors
  </Step>
</Steps>

### Dynamic or Fragile Selectors

Some websites use auto-generated class names that change frequently. These are considered **dynamic** and unreliable.

* Re-check the page in DevTools if a previously working selector fails.
* Look for stable attributes like `data-*`.
* Use attribute-based selectors, which are more stable.

Instead of this:

```python Python theme={"dark"}
params = {'wait_for': '.xY7zD1'} # e.g., Google Search
params = {'wait_for': '.product_list__V9tjod'} # A mix of readable and random
```

Use stable alternatives:

```python Python theme={"dark"}
params = {'wait_for': '[data-testid="product-list"]'}
params = {'wait_for': 'img[src$=".jpg"]'}
params = {'wait_for': '[data-products="item"]'}
```

You can also combine multiple fallback selectors:

```python Python theme={"dark"}
params = {'wait_for': '.product, .listing, [data-products="item"]'}
```

<Tip>Track your CSS selectors over time. When the target website changes its structure, you'll likely need to update your selectors.</Tip>

### Content Is Conditional or Missing

When scraping at scale, it's common to encounter pages where the expected content is missing or appears under certain conditions.

#### Common Scenarios Where Selectors Might Fail

* **Out-of-stock products**: The product is valid, but some elements like the price or “Add to cart” button are missing.

* **Deleted or unavailable pages**: You may be accessing product URLs directly, but the product has been removed. The site may return a 404 error or a custom error page without updating the URL.

* **Failed pages**: The page might fail to load properly causing your selector to not match any on the HTML.

* **Conditional rendering**: Some content is only rendered based on user location, browser behavior, or scrolling. Especially on JavaScript-heavy websites.

#### How to Handle It

Use the following Zenrows parameters to help identify these cases:

1. `original_status=true` <br />Returns the original HTTP status from the target site. Helps distinguish between a bad selector and a broken page.

   ```python Python theme={"dark"}
   params = {
       'original_status': 'true'
   }
   ```

   <Info>For more details check the [Original Status Documentation](/fetch/features/other#original-http-code)</Info>

2. `allowed_status_codes=404,500` <br />Lets you capture and analyze error pages instead of discarding them.

   ```python Python theme={"dark"}
   params = {
       'allowed_status_codes': '404,500,503'
   }
   ```

   <Info>For more details check the [Allowed Status Codes Documentation](/fetch/features/other#return-content-on-error)</Info>

3. **Best practices:**
   * Anticipate that some selectors may not match if content is missing or the page structure changes.
   * Consider checking for fallback selectors or error indicators (like a 404 message or error class).
   * Monitor your scraping jobs for unexpected increases in 422 errors, which may indicate site changes, missing data, or blocking.

### The CSS Selector Exists but Still Fails

Sometimes, your CSS selector is correct but still triggers a 422 error. Here are possible causes:

* **CSS selector is present but hidden (`display: none`)** <br />Zenrows considers it valid. If you need a visible element, consider using a child or wrapper that only appears when the content is shown.
  <Info>You can find more information about advanced CSS selectors [here](/fetch/troubleshooting/advanced-css-selectors).</Info>
* **CSS selector appears after user interaction** <br />Use the [`js_instructions`](/fetch/features/js-instructions) to simulate a click or scroll action first.
* **The page relies on external scripts (slow loading)** <br />Try a different `wait_for` selector that appears earlier in the loading process. Alternatively, switch to our Browser Sessions, which offers longer session times and allows you to manipulate requests more deeply through Puppeteer or Playwright.
* **CSS selector Typos:** <br />Double-check for spelling errors, missing dots (.) for classes, or missing hashes (#) for IDs.

### Alternative: Manual Wait

Instead of waiting for a selector, you can add a fixed delay using the `wait` parameter:

```python Python theme={"dark"}
params = {'wait': 10000,} # Wait of 10 seconds
```

Useful when dynamic elements take time to appear but don't have consistent selectors. The maximum wait time is 30 seconds.

### Combine with Premium Proxy for Protected Sites

Use Wait For with Premium Proxy for maximum effectiveness:

```python Python theme={"dark"}
params = {
    'url': url,
    'apikey': apikey,
	'js_render': 'true',
	'wait_for': '.price',
	'premium_proxy': 'true',
}
```

## Real-World Case Example

Let's say you're targeting the product grid on `scrapingcourse.com/ecommerce/`, and your request fails with a 422 error while your `wait_for` CSS selector is:

```python Python theme={"dark"}
params = {'wait_for': '.lists-product',}
```

This selector is syntactically valid, but it does **not match any element in the HTML** of the target page, so Zenrows cannot proceed and returns a 422 error.

Inspecting the product section reveals the correct selectors:

```php HTML theme={"dark"}
<ul class="products columns-4" id="product-list" data-testid="product-list" data-products="list">
    <li data-products="item" class="product ...">
        <a href="..." class="woocommerce-LoopProduct-link ...">
        <img src="..." class="product-image ..." />
        <h2 class="product-name woocommerce-loop-product__title">Abominable Hoodie</h2>
        <span class="price" data-testid="product-price"><span class="product-price woocommerce-Price-amount amount"><bdi><span class="woocommerce-Price-currencySymbol">$</span>69.00</bdi></span></span>
        </a>
        <a class="button product_type_variable" data-product_id="246" data-product_sku="MH09">Select options</a>
    </li>
    <!-- more li.product ... -->
</ul>
```

**Valid alternatives:**

```python Python theme={"dark"}
params = {'wait_for': '#product-list'} # Waits for the ID 'product-list'
params = {'wait_for': '.products .product'} # Waits for any product in the product list
params = {'wait_for': '[data-products="item"]'} # Waits for any product item using a data attribute
params = {'wait_for': 'h2.product-name'} # Waits for product names
params = {'wait_for': '[data-testid="product-list"]'} # Waits for the entire product list container
```

## CSS Selector Cheat Sheet

| Selector | Example | Use Case |
| - | - | - |
| `.class` | `.price` | Wait for an elements with class “price” |
| `#id` | `#main-content` | Wait for an element with ID “main-content” |
| `[data-attr]` | `[data-loaded="true"]` | Wait for attribute presence |
| `[attr^="val"]` | `[id^="item-"]` | Attribute starts with “item-” |
| `[attr$="val"]` | `[src$=".png"]` | Attribute ends with “.png” |
| `A > B` | `.list > .item` | Direct child of a parent |
| `A B` | `.list .item` | Any descendant item inside a parent |
| `A, B` | `.price, .discount` | Match either `.price` or `.discount` |
| `:nth-child(n)` | `li:nth-child(2)` | Select the 2nd child (or any Nth) of its parent |
| `:first-child` | `div:first-child` | First child of a parent element |
| `:last-child` | `div:last-child` | Last child of a parent element |

## Pricing

The `wait_for` parameter doesn't increase the request cost. You pay the JavaScript Render (5 times the standard price) regardless of the wait value you choose.

<Tip>
  You can monitor your Zenrows usage in multiple ways to stay informed about your account activity and prevent unexpected overages.

  **Dashboard monitoring**: View real-time usage statistics, remaining requests, success rates, and request history on your [Analytics Page](https://app.zenrows.com/overview#performance). You can also set up usage alerts in your [notification settings](https://app.zenrows.com/settings/notifications) to receive notifications when you approach your limits.

  **Programmatic monitoring**: For automated monitoring in your applications, call the `/v1/subscriptions/self/details` endpoint with your API key in the `X-API-Key` header. This returns real-time usage data that you can integrate into your monitoring systems. [Learn more about the usage endpoint](https://docs.zenrows.com/fetch/features/other#plan-usage).

  **Response header monitoring**: Track your concurrency usage through response headers included with each request:

  * `Concurrency-Limit`: Your maximum concurrent requests
  * `Concurrency-Remaining`: Available concurrent request slots
  * `X-Request-Cost`: Cost of the current request
  * `X-Key-Credits-Remaining`: Lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps), sent only for keys with at least one cap
</Tip>

## Frequently Asked Questions (FAQ)

<Accordion title="Can I use XPath instead of CSS selectors?">
  Yes! Everything in this guide also applies to XPath. Zenrows fully supports XPath in the `wait_for` parameter. Please ensure that you provide a valid expression.

  Here's an example based on the HTML below:

  ```php HTML theme={"dark"}
  <ul class="products columns-4" id="product-list" data-testid="product-list">
  <li data-products="item" class="product">
      <a href="/product/abominable-hoodie" class="woocommerce-LoopProduct-link">
      <img src="hoodie.jpg" class="product-image" />
      <h2 class="product-name">Abominable Hoodie</h2>
      <span class="price">$69.00</span>
      </a>
  </li>
  </ul>
  ```

  **Working XPath Examples:**

  ```python Python theme={"dark"}
      params = {'wait_for': '//ul[@id="product-list"]/li'} # Waits for any <li> inside the product list.
      params = {'wait_for': '//h2[contains(@class, "product-name")]'} # Waits for a heading with a class containing product-name.
      params = {'wait_for': '//span[@class="price" and contains(text(), "69")]'} # Waits for price elements that include the number 69.
  ```

  <Tip>To test XPath in your browser, open DevTools and use the console command: `$x('//your/xpath')`.</Tip>

  Just as with CSS selectors, ensure that your XPath expressions are accurate and match the final rendered HTML. If not found, you'll still get a 422 error.
</Accordion>

<Accordion title="What happens if the element I'm waiting for never appears?">
  If the specified element doesn't appear within the 3-minute timeout limit, Zenrows will return a 422 error. This indicates that the selector could not be found in the rendered HTML. You should verify that your selector is correct and that the element exists on the page.
</Accordion>

<Accordion title="How does wait_for differ from wait when both are specified?">
  When both parameters are present, `wait_for` takes precedence and completely overrides the `wait` parameter. Zenrows will ignore the fixed timing and only wait for the specified element to appear.
</Accordion>

<Accordion title="What CSS selectors are supported?">
  Zenrows supports standard CSS selectors including classes, IDs, attributes, pseudo-classes, and complex combinations. However, some advanced CSS4 selectors or browser-specific extensions might not be supported.
</Accordion>

<Accordion title="How do I know if my selector is too specific or too general?">
  Test your selector in the browser's DevTools console using `document.querySelector('your-selector')`. A good selector should reliably match the element you want without being so specific that minor page changes break it.
</Accordion>

<Accordion title="Does wait_for work with Premium Proxy?">
  Yes, `wait_for` works perfectly with Premium Proxy. Element monitoring occurs within the browser environment, regardless of the proxy type used, although residential connections may take slightly longer to load content.
</Accordion>

<Accordion title="What's the maximum time wait_for will wait?">
  The `wait_for` parameter will wait up to 3 minutes (180 seconds) for the specified element to appear. This is the ultimate timeout limit for any Zenrows request when using Fetch. If the element doesn't appear within this time, you'll receive a 422 error.
</Accordion>


# Make Your First Request with Zenrows' Fetch
Source: https://docs.zenrows.com/fetch/first-request

Follow this step-by-step guide to make your first web scraping API call with Zenrows in Python, Node.js, Go, Java, PHP, Ruby, or cURL.

Learn how to extract data from any website using Zenrows' Fetch. This guide walks you through creating your first scraping request that can handle sites at any scale.

Zenrows' Fetch is designed to simplify web scraping. Whether you're dealing with static content or dynamic JavaScript-heavy sites, you can get started in minutes with any programming language that supports HTTP requests.

## 1. Set Up Your Project

### Set Up Your Development Environment

Before diving in, ensure you have the proper development environment and required HTTP client libraries for your preferred programming language. Zenrows works with any language that can make HTTP requests.

<Note>While previous versions may work, we recommend using the latest stable versions for optimal performance and security.</Note>

<Tabs>
  <Tab title="Python">
    Python 3 is recommended, preferably the latest version.
    Consider using an IDE like <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> or <a href="https://code.visualstudio.com/docs/languages/python">Visual Studio Code</a> with the Python extension.

    ```bash theme={"dark"}
    # Install Python (if not already installed)
    # Visit https://www.python.org/downloads/ or use package managers:

    # macOS (using Homebrew)
    brew install python

    # Ubuntu/Debian
    sudo apt update && sudo apt install python3 python3-pip

    # Windows (using Chocolatey)
    choco install python

    # Install the requests library
    pip install requests
    ```

    <Tip>
      If you need help setting up your environment, check out our detailed [Python web scraping setup guide](https://www.zenrows.com/blog/web-scraping-python#setup).
    </Tip>
  </Tab>

  <Tab title="Node.js">
    Node.js 18 or higher is recommended, preferably the latest LTS version.
    Consider using an IDE like <a href="https://code.visualstudio.com/">Visual Studio Code</a> or <a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a> to enhance your coding experience.

    ```bash theme={"dark"}
    # Install Node.js (if not already installed)
    # Visit https://nodejs.org/ or use package managers:

    # macOS (using Homebrew)
    brew install node

    # Ubuntu/Debian (using NodeSource)
    curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
    sudo apt-get install -y nodejs

    # Windows (using Chocolatey)
    choco install nodejs

    # Install the axios library
    npm install axios
    ```

    <Tip>
      If you need help setting up your environment, check out our detailed [Node.js scraping guide](https://www.zenrows.com/blog/web-scraping-javascript-nodejs)
    </Tip>
  </Tab>

  <Tab title="Java">
    Java 8 or higher is recommended, preferably the latest LTS version, and a build tool like Maven or Gradle.
    IDEs like <a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a> or <a href="https://www.eclipse.org/">Eclipse</a> provide excellent Java development support.

    ```bash theme={"dark"}
    # Install Java (if not already installed)
    # Visit https://adoptium.net/ or use package managers:

    # macOS (using Homebrew)
    brew install openjdk

    # Ubuntu/Debian
    sudo apt update && sudo apt install openjdk-17-jdk

    # Windows (using Chocolatey)
    choco install openjdk
    ```

    For Maven projects, add this dependency to your `pom.xml`:

    ```xml xml theme={"dark"}
    <dependency>
        <groupId>org.apache.httpcomponents.client5</groupId>
        <artifactId>httpclient5-fluent</artifactId>
        <version>5.2.1</version>
    </dependency>
    ```
  </Tab>

  <Tab title="PHP">
    PHP 7.4 or higher with cURL extension enabled is recommended, preferably the latest stable version.
    Consider editors like <a href="https://www.jetbrains.com/phpstorm/">PhpStorm</a> or <a href="https://code.visualstudio.com/">Visual Studio Code</a> with PHP extensions.

    ```bash theme={"dark"}
    # Install PHP (if not already installed)
    # Visit https://www.php.net/downloads or use package managers:

    # macOS (using Homebrew)
    brew install php

    # Ubuntu/Debian
    sudo apt update && sudo apt install php php-curl

    # Windows (using Chocolatey)
    choco install php
    ```

    PHP comes with cURL built-in, so no additional packages are needed for basic HTTP requests.
  </Tab>

  <Tab title="Go">
    Go 1.16 or higher is recommended, preferably the latest stable version.
    <a href="https://code.visualstudio.com/">Visual Studio Code</a> with the Go extension or <a href="https://www.jetbrains.com/go/">GoLand</a> provide excellent Go development environments.

    ```bash theme={"dark"}
    # Install Go (if not already installed)
    # Visit https://golang.org/dl/ or use package managers:

    # macOS (using Homebrew)
    brew install go

    # Ubuntu/Debian
    sudo apt update && sudo apt install golang-go

    # Windows (using Chocolatey)
    choco install golang
    ```

    Go's standard library includes the `net/http` package, so no additional dependencies are needed for HTTP requests.
  </Tab>

  <Tab title="Ruby">
    Ruby 2.7 or higher is recommended, preferably the latest stable version.
    Consider editors like <a href="https://www.jetbrains.com/ruby/">RubyMine</a> or <a href="https://code.visualstudio.com/">Visual Studio Code</a> with Ruby extensions.

    ```bash theme={"dark"}
    # Install Ruby (if not already installed)
    # Visit https://www.ruby-lang.org/en/downloads/ or use package managers:

    # macOS (using Homebrew)
    brew install ruby

    # Ubuntu/Debian
    sudo apt update && sudo apt install ruby-full

    # Windows (using Chocolatey)
    choco install ruby

    # Install the faraday gem
    gem install faraday
    ```
  </Tab>

  <Tab title="cURL">
    Typically pre-installed on most systems. If not available, install using your system's package manager.

    ```bash theme={"dark"}
    # cURL is typically pre-installed on most systems. If not, install it:

    # macOS (using Homebrew)
    brew install curl

    # Ubuntu/Debian
    sudo apt update && sudo apt install curl

    # Windows (using Chocolatey)
    choco install curl
    ```

    No additional packages needed - cURL works directly from the command line.
  </Tab>
</Tabs>

### Get Your API Key

[Sign up](https://app.zenrows.com/register?prod=universal_scraper) for a free Zenrows account and get your API key from the [Playground dashboard](https://app.zenrows.com/fetch). You'll need this key to authenticate your requests.

## 2. Make Your First Request

Start with a simple request to understand how Zenrows works. We'll use the <a href="https://httpbin.io/get">HTTPBin.io/get</a> endpoint to demonstrate how Zenrows processes requests and returns data.

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests

  url = 'https://httpbin.io/get'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'mode': 'auto',
  }

  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/get';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'mode': 'auto',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fget&mode=auto";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fget&mode=auto');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fget&mode=auto", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fget&mode=auto')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fget&mode=auto"
  ```
</CodeGroup>

Replace `YOUR_ZENROWS_API_KEY` with your actual API key and run the script:

<CodeGroup>
  ```bash Python theme={"dark"}
  python your_script.py
  ```

  ```bash Node.js theme={"dark"}
  node your_script.js
  ```

  ```bash Java theme={"dark"}
  javac YourScript.java && java YourScript
  ```

  ```bash PHP theme={"dark"}
  php your_script.php
  ```

  ```bash Go theme={"dark"}
  go run your_script.go
  ```

  ```bash Ruby theme={"dark"}
  ruby your_script.rb
  ```

  ```bash cURL theme={"dark"}
  # The cURL command runs directly in terminal
  ```
</CodeGroup>

### Expected Output

The script will print the contents of the website, for `HTTPBin.io/get` it's something similar to this:

```json Response theme={"dark"}
{
    "args": {},
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
        // additional headers omitted for brevity...
    },
    "origin": "38.154.5.224:6693",
    "url": "http://httpbin.io/get"
}
```

Perfect! You've just made your first web scraping request with Zenrows.

## 3. Scrape More Complex Websites

Modern websites often use JavaScript to load content dynamically and employ sophisticated anti-bot protection. Zenrows provides powerful features to handle these challenges automatically.

### Use Adaptive Stealth Mode

The simplest way to scrape complex or protected sites in code is to use **Adaptive Stealth Mode** (`mode=auto`). Zenrows analyzes each request and automatically applies the right combination of JavaScript rendering and Premium Proxies only when needed, so you get reliable results without having to tune parameters yourself. Add `mode=auto` to your request:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  url = 'https://www.scrapingcourse.com/antibot-challenge'
  apikey = 'YOUR_ZENROWS_API_KEY'

  params = {
      'url': url,
      'apikey': apikey,
      'mode': 'auto',
  }

  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/antibot-challenge';
  const apikey = 'YOUR_ZENROWS_API_KEY';

  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'mode': 'auto',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto"
  ```
</CodeGroup>

**Optional parameters with Adaptive Stealth Mode:** When you use `mode=auto`, only the managed parameters (`js_render` and `premium_proxy`) are disabled to prevent conflicts. You can still use other API parameters alongside Adaptive Stealth Mode, such as:

* **Proxy country** (`proxy_country`): For region-restricted or geo-specific content, set the country of the IP (e.g., `proxy_country=us`). When a proxy country is set, Zenrows automatically applies Premium Proxies to test the most reliable configuration. If our system detects that another proxy type or country delivers better results for your request, we may switch it automatically to maximize success rates.
* **JavaScript rendering wait** (`wait` or `wait_for`): For slow-loading or dynamic content, add `wait` (milliseconds after page load) or `wait_for` (CSS selector). When a wait time is set, Zenrows automatically applies JavaScript rendering on the configuration.
* **Other parameters**: `css_extractor`, `session_id`, custom headers, and other API parameters remain available for use with `mode=auto`.

Example using optional parameters with Adaptive Stealth Mode:

```python Python theme={"dark"}
params = {
    'url': url,
    'apikey': apikey,
    'mode': 'auto',
    'proxy_country': 'us',  # Optional: specify proxy location
    'wait': '3000',         # Optional: wait 3 seconds when JS rendering is used
}
```

### Use the Request Playground (easy mode)

The [Request Playground](https://app.zenrows.com/fetch) in your Zenrows dashboard offers an **easy mode** that uses Adaptive Stealth Mode by default. Enter your target URL and run a request. There's no need to choose JavaScript rendering or Premium Proxies yourself. Zenrows picks the proper configuration for each URL.

**Benefits:**

* Test any URL visually and see the result immediately
* Get a ready-to-use cURL command or code snippet with `mode=auto` included
* Reliable results on complex or protected sites without manual parameter tuning

Enter the target URL (for this demonstration, the <a href="https://www.scrapingcourse.com/antibot-challenge">Anti-bot Challenge</a> page) in the **URL to Scrape** field and use easy mode to get started.

<img alt="Zenrows Request Playground Easy Mode" />

### Use the Request Playground (custom mode)

For fine-grained control when you know exactly which features a site needs, use the Request Playground's **custom mode** to set parameters yourself. You can manually configure JavaScript rendering, Premium Proxies, and other options, or select Adaptive Stealth Mode (`mode=auto`) to let Zenrows choose automatically.

<img alt="Zenrows Request Playground Custom Mode with Adaptive Stealth Mode" />

### Use Premium Proxies

[Premium Proxies](/fetch/features/premium-proxy) provide access to over 55 million residential IP addresses from 190+ countries with 99.9% uptime, ensuring the ability to bypass sophisticated anti-bot protection.

### Enable JavaScript Rendering

[JavaScript Rendering](/fetch/features/js-rendering) uses a real browser to execute JavaScript and capture the fully rendered page. This is essential for modern web applications, single-page applications (SPAs), and sites that load content dynamically.

### Combine Features for Maximum Success

For the most protected sites, enable both **JavaScript Rendering** and **Premium Proxies**. This provides the highest success rate for challenging targets.

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  url = 'https://www.scrapingcourse.com/antibot-challenge'
  apikey = 'YOUR_ZENROWS_API_KEY'

  params = {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'premium_proxy': 'true',
  }

  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/antibot-challenge';
  const apikey = 'YOUR_ZENROWS_API_KEY';

  axios({
      url: 'https://api.zenrows.com/v1/',
      method: 'GET',
      params: {
          'url': url,
          'apikey': apikey,
          'js_render': 'true',
          'premium_proxy': 'true',
      },
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true"
  ```
</CodeGroup>

This code sends a GET request to the Zenrows API endpoint with your target URL and authentication. The `js_render` parameter enables JavaScript processing, while `premium_proxy` routes your request through residential IP addresses.

#### Use-case recipes

Here are a few quick recipes you can adapt:

* **Form submission** → Use `js_render` + `js_instructions`.
* **Keep session across requests** → Add `session_id` to maintain session state and IP consistency.
* **Extract structured fields only** → Use `css_extractor` to return just the fields you need.

<Info>
  See the full [Common Use Cases & Recipes](/fetch/common-use-cases) guide.
</Info>

### Run Your Application

Execute your script to test the scraping functionality and verify that your setup works correctly.

<CodeGroup>
  ```bash Python theme={"dark"}
  python your_script.py
  ```

  ```bash Node.js theme={"dark"}
  node your_script.js
  ```

  ```bash Java theme={"dark"}
  javac YourScript.java && java YourScript
  ```

  ```bash PHP theme={"dark"}
  php your_script.php
  ```

  ```bash Go theme={"dark"}
  go run your_script.go
  ```

  ```bash Ruby theme={"dark"}
  ruby your_script.rb
  ```

  ```bash cURL theme={"dark"}
  # The cURL command runs directly in terminal
  ```
</CodeGroup>

### Example Output

Run the script, and Zenrows will handle the heavy lifting by rendering the page's JavaScript and routing your request through premium residential proxies. The response will contain the entire HTML content of the page:

```html HTML theme={"dark"}
<html lang="en"><head>
    <title>Antibot Challenge - ScrapingCourse.com</title>
</head><body>
    <h2>You bypassed the Antibot challenge! :D</h2>
</body></html>
```

Congratulations! You now have a Zenrows integration that can scrape websites at any scale while bypassing anti-bot protection. You're ready to tackle more advanced scenarios and [customize the API](/fetch/api-reference#parameter-overview) to fit your scraping needs.

## Troubleshooting

Request failures can happen for various reasons. While some issues can be resolved by adjusting Zenrows parameters, others are beyond your control, such as the target server being temporarily down.

Below are some quick troubleshooting steps you can take:

<Steps>
  <Step title="Check the Error Code and Error Message">
    When faced with an error, it's essential first to check the error code and message for indications of the error. The most common error codes are:

    * **401 Unauthorized** <br />Your API key is missing, incorrect, or improperly formatted. Double-check that you are sending the correct API key in your request headers.

    * **429 Too Many Requests** <br />You have exceeded your concurrency limit. Wait for ongoing requests to finish before sending new ones, or consider upgrading your plan for higher limits.

    * **413 Content Too Large** <br />The response size exceeds your plan's limit. Use CSS selectors to extract only the needed data, reducing the response size.

    * **422 Unprocessable Entity** <br />Your request contains invalid parameter values, or anti-bot protection is blocking access. Review the API documentation to ensure all parameters are correct and supported.

    <Tip>Get more information on the [API Error Codes page](/api-error-codes).</Tip>
  </Step>

  <Step title="Check if the Site is Publicly Accessible">
    Some websites may require a session, so verifying if the site can be accessed without logging in is a good idea. Open the target page in an incognito browser to check this.

    You must handle session management in your requests if authentication credentials are required. You can learn how to scrape a website that requires authentication in our guide: [Web scraping with Python](https://www.zenrows.com/blog/web-scraping-login-python).
  </Step>

  <Step title="Verify the Site is Accessible in Your Country">
    Sometimes, the target site may be region-restricted and only accessible to specific locations. Zenrows automatically selects the best proxy, but if the site is only available in concrete regions, specify a geolocation using `proxy_country`.

    Here's how to choose a proxy in the US:

    <CodeGroup>
      ```python Python theme={"dark"}
      params = {
          'premium_proxy': 'true',
          'proxy_country': 'us' # <- choose a premium proxy in the US
          # other configs...
      }
      response = requests.get('https://api.zenrows.com/v1/', params=params)

      ```

      ```javascript Node.js theme={"dark"}
      params: {
          'premium_proxy': 'true',
          'proxy_country': 'us' // <- choose a premium proxy in the US
          // other configs...
      },
      ```
    </CodeGroup>

    If the target site requires access from a specific region, adding the `proxy_country` parameter will help.

    <Tip>Check out more about it on our [Geolocation Documentation Page](/fetch/features/proxy-country).</Tip>
  </Step>

  <Step title="Add Pauses to Your Request">
    You can also enhance your request by adding options like `wait` or `wait_for` to ensure the page fully loads before extracting data, improving accuracy.

    <CodeGroup>
      ```python Python theme={"dark"}
      # Use a wait parameter or a wait_for parameter
      params = {
          'wait': '3000', # <- Adds a delay of 3 seconds
          # other configs...
      }
      response = requests.get('https://api.zenrows.com/v1/', params=params)

      ```

      ```javascript Node.js theme={"dark"}
      params: {
          'wait': '3000', // <- Adds a delay of 3 seconds
          // other configs...
      },
      ```
    </CodeGroup>

    <Tip>Find more details on our [Wait Documentation](/fetch/features/wait) on [Wait For Selector Documentation](/fetch/features/wait-for).</Tip>
  </Step>

  <Step title="Retry the Request">
    Network issues or temporary failures can cause your request to fail. Implementing retry logic can solve this by automatically repeating the request.

    <CodeGroup>
      ```python Python theme={"dark"}
      import requests
      from requests.adapters import HTTPAdapter
      from urllib3.util.retry import Retry

      # Define the retry strategy
      retry_strategy = Retry(
          total=4,  # Maximum number of retries
          status_forcelist=[429, 500, 502, 503, 504],  # HTTP status codes to retry on
      )
      # Create an HTTP adapter with the retry strategy and mount it to session
      adapter = HTTPAdapter(max_retries=retry_strategy)

      # Create a new session object
      session = requests.Session()
      session.mount('http://', adapter)
      session.mount('https://', adapter)

      # Make a request using the session object
      response = session.get('https://scrapingcourse.com/ecommerce/')

      if response.status_code == 200:
          print(f'SUCCESS: {response.text}')
      else:
          print("FAILED")
      ```

      ```javascript Node.js theme={"dark"}
      // npm install axios-retry
      import axiosRetry from 'axios-retry';
      import axios from 'axios';

      // Pass the axios instance to the retry function and call it
      axiosRetry(axios, {
          retries: 3, // Number of retries (Defaults to 3)
      });

      // Make Axios requests below
      axios.get('https://scrapingcourse.com/ecommerce/') // The request is retried if it fails
          .then((response) => {
              console.log('Data: ', response.data);
          }).catch((error) => {
              console.log('Error: ', error);
          });
      ```
    </CodeGroup>

    <Tip>You can learn more on our [Python requests retry guide](https://www.zenrows.com/blog/python-requests-retry) and [Node.js Axios retry guide](https://www.zenrows.com/blog/axios-retry).</Tip>
  </Step>

  <Step title="Get Help From Zenrows Experts">
    Our support team can assist you if the issue persists despite following these tips. Use the [Playground page](https://app.zenrows.com/fetch) or <a href="mailto:success@zenrows.com">contact us via email</a> to get personalized help from Zenrows experts.

    <Frame>
      <img alt="Zenrows Request Playground Page" />
    </Frame>
  </Step>
</Steps>

<Tip>
  For more solutions and detailed troubleshooting steps, see our [Troubleshooting Guides](/fetch/troubleshooting/troubleshooting-guide).
</Tip>

## Next Steps

You now have a solid foundation for web scraping with Zenrows. Here are some recommended next steps to take your scraping to the next level:

* **[Complete API Reference](/fetch/api-reference)**: <br />Explore all available parameters and advanced configuration options to customize Zenrows for your specific use cases.
* **[JavaScript Instructions Guide](/fetch/features/js-instructions)**: <br />Learn how to perform complex page interactions like form submissions, infinite scrolling, and multi-step workflows.
* **[Output Formats and Data Extraction](/fetch/features/markdown)**: <br />Learn advanced data extraction with CSS selectors, output formats including Markdown and PDF conversion, and screenshot configurations.
* **[Pricing and Plans](/first-steps/pricing)**: <br />Understand how request costs are calculated and choose the plan that best fits your scraping volume and requirements.

## Frequently Asked Questions (FAQ)

<Accordion title="How can I bypass CloudFlare and other protections?">
  To reliably access sites behind CloudFlare or similar security mechanisms, enable both `js_render` and `premium_proxy` in your requests. These features simulate a full browser environment and route through high-quality residential proxies for protected access.
  You can also enhance your request by adding options like `wait` or `wait_for` to ensure the page fully loads before extracting data, improving accuracy.

  <Tip>Check out our documentation about the [wait](/fetch/features/wait) and [wait\_for](/fetch/features/wait-for) params.</Tip>
</Accordion>

<Accordion title="How can I ensure my requests don't fail?">
  You can configure retry logic to handle failed HTTP requests. Learn more in our [guide on retrying requests](/zenrows-academy/retry-failed-requests).
</Accordion>

<Accordion title="How do I extract specific content from a page?">
  You can use the `css_extractor` parameter to directly extract content from a page using CSS selectors. Find out more in our [tutorial on data parsing](/zenrows-academy/how-to-extract-data).
</Accordion>

<Accordion title="Can I integrate Zenrows with Python's Requests and BeautifulSoup?">
  Yes! You can use Zenrows alongside Python Requests and BeautifulSoup for HTML parsing. Learn how in our guide on [Python Requests and BeautifulSoup integration](/zenrows-academy/how-to-extract-data#python-with-beautifulsoup).
</Accordion>

<Accordion title="Can I integrate Zenrows with Node.js and Cheerio?">
  Yes! You can integrate Zenrows with Node.js and Cheerio for efficient HTML parsing and web scraping. Check out our guide to learn how to combine these tools: [Node.js and Cheerio integration](/zenrows-academy/how-to-extract-data#javascript-with-cheerio).
</Accordion>

<Accordion title="How can I simulate user interactions on the target page?">
  Use the `js_render` and `js_instructions` features to simulate actions such as clicking buttons or filling out forms. Discover more about interacting with web pages in our [JavaScript instructions guide](/fetch/features/js-instructions).
</Accordion>

<Accordion title="How can I scrape faster using Zenrows?">
  You can scrape multiple URLs simultaneously by making concurrent API calls. Check out our guide on [using concurrency](/fetch/features/concurrency#using-concurrency) to boost your scraping speed.
</Accordion>


# JavaScript, NodeJS and Cheerio Integration
Source: https://docs.zenrows.com/fetch/help/javascript-nodejs-and-cheerio-integration

How to integrate Zenrows with Node.js, Axios, and Cheerio for scalable web scraping with built-in concurrency and auto-retry support.

Learn how to integrate Zenrows API with Axios and Cheerio to scrape any website. From the most basic calls to advanced features such as concurrency and auto-retry. From installation to the final code, we will go step-by-step, explaining everything we code.

To just grab the code, go to the final snippet and copy it. It is commented with the parts that must be filled and helpful remarks for the complicated details.

For the code to work, you will need <a href="https://nodejs.org/en/">Node</a> (or <a href="https://github.com/nvm-sh/nvm">nvm</a>) and npm installed. Some systems have it pre-installed. After that, install all the necessary libraries by running `npm install`.

```bash theme={"dark"}
npm install axios axios-retry cheerio
```

You will also need to [register to get your API Key](https://app.zenrows.com/register?p=free).

## Using Axios to Get a Page

The first library we will see is Axios, a "promise based HTTP client for the browser and node.js". It exposes a get method that will call a URL and return a response with the HTML. For the moment, we won't be using any parameters, just as a demo to see how it works.

**Careful! This script will run without any proxy, and the server will see your real IP.** You don't need to execute this snippet.

```javascript theme={"dark"}
const axios = require("axios");

const url = ""; // ... your URL here
axios.get(url).then((response) => {
	console.log(response.data); // pages's HTML
});
```

## Calling Zenrows API with Axios

Connecting Axios to Zenrows API is straightforward. `axios.get`'s target will be the API base, and a second parameter is an object with `params`: `apikey` for authentication and `url`. URLs must be [encoded](/fetch/faq#how-to-encode-urls), but Axios will handle that when using params.

With this simple change, we will handle all the hassles of scraping, such as proxies rotation, bypassing CAPTCHAs and anti-bot solutions, setting correct headers, and many more. However, there are still some challenges that we will address now. Continue reading.

```javascript theme={"dark"}
const axios = require("axios");

const url = ""; // ... your URL here
const apikey = "YOUR_ZENROWS_API_KEY"; // paste your API Key here
const zenrowsApiBase = "https://api.zenrows.com/v1/";

axios
	.get(zenrowsApiBase, {
		params: { apikey, url },
	})
	.then((response) => {
		console.log(response.data); // pages's HTML
	});
```

## Extracting Basic Data with Cheerio

We will now parse the page's HTML with <a href="https://cheerio.js.org/">Cheerio</a> and extract some data. We'll create a simple function `extractContent` to return URL, title, and h1 content. Your custom extracting logic goes there.

Cheerio offers a "jQuery-like" syntax, and it is designed to work on the server. Its `load` method receives a plain HTML and creates a querying function that will allow us to find elements. Then you can query with CSS Selectors and navigate, manipulate, or extract content as a browser would. The resulting selector exposes `text`, which will give us the content in plain text, without tags. Check the docs for more advanced features.

```javascript theme={"dark"}
const axios = require("axios");
const cheerio = require("cheerio");

const url = ""; // ... your URL here
const apikey = "YOUR_ZENROWS_API_KEY"; // paste your API Key here
const zenrowsApiBase = "https://api.zenrows.com/v1/";

const extractContent = (url, $) => ({
	// extracting logic goes here
	url,
	title: $("title").text(),
	h1: $("h1").text(),
});

axios
	.get(zenrowsApiBase, {
		params: { apikey, url },
	})
	.then((response) => {
		const $ = cheerio.load(response.data);
		const content = extractContent(url, $);
		console.log(content); // custom scraped content
	});
```

## List of URLs with Concurrency

We've seen how to scrape a single URL. Instead, we will now introduce a list of URLs closer to an actual use case. We'll also set up concurrency so we don't have to wait for the sequential process to finish. It will allow the script to process several URLs simultaneously, always with a maximum. That number depends on the plan you are in.

<a href="https://www.npmjs.com/package/zenrows">Zenrows JavaScript SDK</a> provides full concurrency support, as JavaScript's support is limited.

```bash theme={"dark"}
npm i zenrows
```

It will enqueue and execute all our requests. And it will do so by handling the parallelism for us and the maximum number of requests going on simultaneously, but never over the limit (10 in the example). Once all the requests finish, we will print the results. In a real case, for example, store them in a database.

```javascript theme={"dark"}
const { ZenRows } = require("zenrows");
const cheerio = require("cheerio");

const apikey = "YOUR_ZENROWS_API_KEY"; // paste your API Key here
const urls = [
	// ... your URLs here
];



(async () => {
	const client = new ZenRows(apiKey, { concurrency: 10 });

	const extractContent = (url, $) => ({
		// extracting logic goes here
		url,
		title: $("title").text(),
		h1: $("h1").text(),
	});

	const scrapeUrl = async (url) => {
		try {
			const response = await client.get(url);
			const $ = cheerio.load(response.data);

			return extractContent(url, $);
		} catch (error) {
			return { url, error: error.message };
		}
	};

	const promises = urls.map((url) => scrapeUrl(url));

	const results = await Promise.allSettled(promises);
	console.log(results);
	/*
		[
			{
				status: "fulfilled",
				value: {
					url: "YOUR_FIRST_URL",
					title: "First Title",
					h1: "Some Important H1"
				}
			},
			...
		]
	*/
})();
```

## Auto-Retry Failed Requests

The last step to having a robust scraper is to retry failed requests. We could use <a href="https://github.com/softonic/axios-retry">axios-retry</a>, but the [SDK already does that](/fetch/features/concurrency#zenrows-sdk-for-javascript).

The basic idea goes like this:

1. Identify the failed requests based on the return status code.
2. Wait an arbitrary amount of time. Using the library's `exponentialDelay` will increment exponentially plus a random margin between attempts.
3. Retry the request until it succeeds or reaches a maximum amount of retries.

Keep in mind that all the retries will take place on the same concurrency thread, effectively blocking it. Some errors are temporary, so retrying might not solve the issue. For those cases, a better strategy would be to store the URL as failed and enqueue it again after some minutes.

Passing an integer value on the SDK constructor is enough to set the number of retries you want. Visit the article on [Retry Failed Requests](/zenrows-academy/retry-failed-requests) for more info.

```javascript theme={"dark"}
const { ZenRows } = require("zenrows");

// same snippet as above

const apikey = "YOUR_ZENROWS_API_KEY"; // paste your API Key here
const client = new ZenRows(apiKey, { concurrency: 10, retries: 3 });
```

If the implementation does not work for your use case or you have any problem, <a href="mailto:success@zenrows.com">contact us</a> and we'll gladly help you.


# Python Requests and BeautifulSoup Integration
Source: https://docs.zenrows.com/fetch/help/python-requests-and-beautifulsoup-integration

How to use Zenrows with Python Requests and BeautifulSoup for reliable web scraping, including concurrency, auto-retry, and setup.

Learn how to integrate Zenrows API with Python Requests and BeautifulSoup to extract the data you want. From basic calls to advanced features such as auto-retry and concurrency. We will walk over each stage of the process, from installation to final code, explaining everything we code.

For a short version, go to the final code and copy it. It is commented with the parts that must be completed and helpful suggestions for the more challenging details.

For the code to work, you will need <a href="https://www.python.org/downloads/">python3 installed</a>. Some systems have it pre-installed. After that, install all the necessary libraries by running `pip install`.

```bash theme={"dark"}
pip install requests beautifulsoup4
```

You will also need to [register to get your API Key](https://app.zenrows.com/register?p=free).

## Using Requests to Get a Page

The first library we will see is `requests`, an HTTP library for Python. It exposes a `get` method that will call a URL and return its HTML. For the time being, we won't be utilizing any parameters; this is simply a demo to see how it works.

**Careful! This script will execute without any proxy so that the server will see your actual IP.** You don't need to run this snippet.

```python theme={"dark"}
import requests

url = "" # ... your URL here
response = requests.get(url)

print(response.text)  # pages's HTML
```

## Calling Zenrows API with Requests

Connecting requests to Zenrows API is straightforward. `get`'s target will be the API base and then two params: `apikey` for authentication and `url`. URLs must be [encoded](/fetch/faq#how-to-encode-urls); however, `requests` will handle that when using `params`.

With this simple update, we will manage most scraping problems, such as proxy rotation, setting correct headers, avoiding CAPTCHAs and anti-bot solutions, and many more. But there are a few issues that we will address now. Keep on reading.

```python theme={"dark"}
import requests

url = "" # ... your URL here
apikey = "YOUR_ZENROWS_API_KEY" # paste your API Key here
zenrows_api_base = "https://api.zenrows.com/v1/"

response = requests.get(zenrows_api_base, params={
	"apikey": apikey,
	"url": url,
})

print(response.text)  # pages's HTML
```

## Extracting Basic Data with BeautifulSoup

We'll now use BeautifulSoup to parse the HTML on the page and extract some data. We will write a simple function called `extract_content` that returns URL, title, and h1 content. There is where you can put your custom extracting logic.

```python theme={"dark"}
import requests
from bs4 import BeautifulSoup

url = "" # ... your URL here
apikey = "YOUR_ZENROWS_API_KEY" # paste your API Key here
zenrows_api_base = "https://api.zenrows.com/v1/"

def extract_content(url, soup):
	# extracting logic goes here
	return {
		"url": url,
		"title": soup.title.string,
		"h1": soup.find("h1").text,
	}

response = requests.get(zenrows_api_base, params={
	"apikey": apikey,
	"url": url,
})
soup = BeautifulSoup(response.text, "html.parser")
content = extract_content(url, soup)

print(content)  # custom scraped content
```

## List of URLs with Concurrency

Up until now, we were scraping a single URL. Instead, we will now introduce a list of URLs more relevant to a real-world use case. In addition, we will set up concurrency, so we don't have to wait for the sequential process to complete. It will allow the script to process multiple URLs simultaneously, always with a maximum. That number is determined by the plan you are in.

In short, `multiprocessing` package implements a `ThreadPool` that will queue and execute all our requests. And it will do so by handling the parallelism for us and the maximum number of requests going on simultaneously, but never over the limit (10 in the example). Once all the requests finish, it will group all the results in a single variable, and we will print them. In a real case, for example, store them in a database.

Note that this is not a queue; we can add no new URLs once the process initiates. If that is your use case, check out our guide on how to [Scrape and Crawl from a Seed URL](/zenrows-academy/scrape-and-crawl-from-a-seed-url).

```python theme={"dark"}
import requests
from bs4 import BeautifulSoup
from multiprocessing.pool import ThreadPool

apikey = "YOUR_ZENROWS_API_KEY" # paste your API Key here
zenrows_api_base = "https://api.zenrows.com/v1/"

concurrency = 10
urls = [
	# ... your URLs here
]

def extract_content(url, soup):
	# extracting logic goes here
	return {
		"url": url,
		"title": soup.title.string,
		"h1": soup.find("h1").text,
	}

def scrape_with_zenrows(url):
	response = requests.get(zenrows_api_base, params={
		"apikey": apikey,
		"url": url,
	})
	soup = BeautifulSoup(response.text, "html.parser")
	return extract_content(url, soup)

pool = ThreadPool(concurrency)
results = pool.map(scrape_with_zenrows, urls)
pool.close()
pool.join()

[print(result) for result in results]  # custom scraped content
```

## Auto-Retry Failed Requests

The final step in creating a robust scraper is to retry on failed requests. We will be using `Retry` from urllib3 and `HTTPAdapter` from requests.

The basic idea is as follows:

1. Using the return status code, identify the failed requests.
2. Wait an arbitrary amount of time. In our example, it will grow exponentially between tries.
3. Retry the request until it succeeds or reaches a maximum number of retries.

Fortunately, we can use these two libraries to implement that behavior. We must first configure `Retry` and then mount the `HTTPAdapter` for a requests session. Unlike the previous ones, we won't be calling `requests.get` directly but `requests_session.get`. Once created the session, it will use the same adapter for all subsequent calls.

For more information, visit the article on [Retry Failed Requests](/zenrows-academy/retry-failed-requests).

```python theme={"dark"}
import requests
from bs4 import BeautifulSoup
from multiprocessing.pool import ThreadPool
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

apikey = "YOUR_ZENROWS_API_KEY" # paste your API Key here
zenrows_api_base = "https://api.zenrows.com/v1/"
urls = [
	# ... your URLs here
]
concurrency = 10  # maximum concurrent requests, depends on the plan

requests_session = requests.Session()
retries = Retry(
	total=3,  # number of retries
	backoff_factor=1,  # exponential time factor between attempts
	status_forcelist=[429, 500, 502, 503, 504]  # status codes that will retry
)

requests_session.mount("http://", HTTPAdapter(max_retries=retries))
requests_session.mount("https://", HTTPAdapter(max_retries=retries))

def extract_content(url, soup):
	# extracting logic goes here
	return {
		"url": url,
		"title": soup.title.string,
		"h1": soup.find("h1").text,
	}

def scrape_with_zenrows(url):
	try:
		response = requests_session.get(zenrows_api_base, params={
			"apikey": apikey,
			"url": url,
		})

		soup = BeautifulSoup(response.text, "html.parser")
		return extract_content(url, soup)
	except Exception as e:
		print(e)  # will print "Max retries exceeded"

pool = ThreadPool(concurrency)
results = pool.map(scrape_with_zenrows, urls)
pool.close()
pool.join()

[print(result) for result in results if result]  # custom scraped content
```

If you have any problem with the implementation or it does not work for your use case, <a href="mailto:success@zenrows.com">contact us</a> and we'll help you.


# Get Started with Zenrows Fetch in Node.js
Source: https://docs.zenrows.com/fetch/nodejs

Quick start guide for Zenrows Fetch with Node.js and Axios, covering installation, setup, and your first API request.

This guide will walk you through the steps to get started with Zenrows in Node.js, from installing the necessary packages to performing your first API request. Zenrows simplifies web scraping by handling anti-bot measures and rendering JavaScript-heavy sites, allowing you to focus on data extraction without worrying about site protection mechanisms. Let's dive in!

## How to Use Zenrows with Node.js

Before starting, ensure you have Node.js 18+ installed on your machine. Using an IDE like <a href="https://code.visualstudio.com/docs/languages/python">Visual Studio Code</a> or <a href="https://www.jetbrains.com/idea/">IntelliJ IDEA</a> will also enhance your coding experience.

We'll create a Node.js script named `scraper.js` inside a `/scraper` directory. If you need help setting up your environment, check out our [Node.js scraping guide](https://www.zenrows.com/blog/web-scraping-javascript-nodejs) for detailed instructions on preparing everything.

### Install Node.js's Axios Library

To interact with the Zenrows API in a Node.js environment, you can use the popular HTTP client library, Axios. Axios simplifies making HTTP requests and handling responses, making it an ideal choice for integrating with web services such as Zenrows.

To install `axios` in your Node.js project, run the following command in your terminal:

```bash theme={"dark"}
npm install axios
```

This command will install the Axios library, which allows you to easily send HTTP requests from your Node.js applications. Once installed, you can start making API calls to Zenrows, retrieving data efficiently while managing different aspects of the request and response cycle.

### Perform Your First API Request

In this step, you will send your first request to Zenrows using the Axios library to scrape content from a simple URL. We will use the <a href="https://httpbin.io/get">HTTPBin.io/get</a> endpoint to demonstrate how Zenrows processes the request and returns the data.

Here's an example:

```javascript scraper.js theme={"dark"}
// npm install axios
const axios = require('axios');

const url = 'https://httpbin.io/get';
const apikey = 'YOUR_ZENROWS_API_KEY';
axios({
	url: 'https://api.zenrows.com/v1/',
	method: 'GET',
	params: {
		'url': url,
		'apikey': apikey,
	},
})
    .then(response => console.log(response.data))
    .catch(error => console.log(error));
```

Replace `YOUR_ZENROWS_API_KEY` with your actual API key and run the script:

```bash theme={"dark"}
node scraper.js
```

The script will print something similar to this:

```json theme={"dark"}
{
    "args": {},
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
        // additional headers omitted for brevity...
    },
    "origin": "38.154.5.224:6693",
    "url": "http://httpbin.io/get"
}
```

The response includes useful information such as the origin, which shows the IP address from which the request was made. Zenrows automatically rotates your IP address and adjusts the User-Agent for each request, ensuring anonymity and preventing blocks.

Congratulations! You've just made your first web scraping request with Node.js and Axios.

### Scrape More Complex Web Pages

While scraping simple sites like HTTPBin is straightforward, many websites, especially those with dynamic content or strict anti-scraping measures, require additional features. Zenrows allows you to bypass these defenses by enabling [JavaScript Rendering](/fetch/features/js-rendering) and using [Premium Proxies](/fetch/features/premium-proxy).

For example, if you try to scrape a page like <a href="https://www.g2.com/products/jira/reviews">G2's Jira reviews</a> without any extra configurations, you'll encounter an error:

```json theme={"dark"}
{
    "code":"REQS002",
    "detail":"The requested URL domain needs JavaScript rendering and/or Premium Proxies due to its high-level security defenses. Please retry by adding 'js_render' and/or 'premium_proxy' parameters to your request.",
    "instance":"/v1",
    "status":400,
    "title":"Www.g2.com requires javascript rendering and premium proxies enabled (REQS002)",
    "type":"https://docs.zenrows.com/api-error-codes#REQS002"
}
```

This error happens because G2 employs advanced security measures that block basic scraping attempts.

Here's how you can modify the request to enable both:

```javascript scraper.js theme={"dark"}
// npm install axios
const axios = require('axios');

const url = 'https://www.g2.com/products/jira/reviews';
const apikey = 'YOUR_ZENROWS_API_KEY';
axios({
	url: 'https://api.zenrows.com/v1/',
	method: 'GET',
	params: {
		'url': url,
		'apikey': apikey,
		'js_render': 'true',
		'premium_proxy': 'true',
	},
})
    .then(response => console.log(response.data))
    .catch(error => console.log(error));
```

Run the script, and this time, Zenrows will handle the heavy lifting by rendering the page's JavaScript and routing your request through premium residential proxies. The response will contain the entire HTML content of the page:

```html theme={"dark"}
<!DOCTYPE html>
<head>
    <meta charset="utf-8">
    <title>Jira Reviews 2024: Details, Pricing, & Features | G2</title>
    <!-- omitted for brevity -->
</head>
<body>
    <!-- page content -->
</body>
</html>
```

This demonstrates how you can scrape more advanced websites that rely on JavaScript or have stringent anti-bot mechanisms in place.

## Troubleshooting

Request failures can happen for various reasons. While some issues can be resolved by adjusting Zenrows parameters, others are beyond your control, such as the target server being temporarily down.

Below are some quick troubleshooting steps you can take:

<Steps>
  <Step title="Retry the Request">
    Network issues or temporary failures can cause your request to fail. Implementing retry logic can solve this by automatically repeating the request. Learn how to add retries in our [Node.js Axios retry guide](https://www.zenrows.com/blog/axios-retry).

    Example of retry logic using requests:

    ```javascript scraper.js theme={"dark"}
    // Retry interceptor function
    import axiosRetry from 'axios-retry';
    // Default axios instance
    import axios from 'axios';

    // Pass the axios instance to the retry function  and call it
    axiosRetry(axios, {
      retries: 3, // Number of retries (Defaults to 3)
    });

    // Make Axios requests below
    axios.get('https://scrapingcourse.com/ecommerce/') // The request is retried if it fails
      .then((response) => {
        console.log('Data: ', response.data);
      }).catch((error) => {
        console.log('Error: ', error);
      });
    ```
  </Step>

  <Step title="Verify the Site is Accessible in Your Country">
    Sometimes, the target site might be region-restricted and only available to some proxies. Zenrows automatically selects the best proxy, but if the site is available only in specific regions, specify a geolocation using `proxy_country`.

    Here's how to choose a proxy in the US:

    ```javascript theme={"dark"}
    params: {
        'premium_proxy': 'true',
        'proxy_country': 'us' // <- choose a premium proxy in the US
        // other configs...
    },
    ```

    If the target site requires access from a specific region, adding the `proxy_country` parameter will help.

    <Tip>Check out more about it on our [Geolocation Documentation Page](/fetch/features/proxy-country).</Tip>
  </Step>

  <Step title="Check if the Site is Publicly Accessible">
    Some websites may require a session, so verifying if the site can be accessed without logging in is a good idea. Open the target page in an incognito browser to check this.

    You must handle session management in your requests if authentication credentials are required. You can learn how to scrape a website that requires authentication in our guide: [Web scraping with Python](https://www.zenrows.com/blog/web-scraping-login-python).
  </Step>

  <Step title="Get Help From Zenrows Experts">
    If the issue persists despite following these tips, our support team is available to assist you. Use the [Playground page](https://app.zenrows.com/fetch) or <a href="mailto:success@zenrows.com">contact us via email</a> to get personalized help from Zenrows experts.

    <Frame>
      <img alt="Zenrows Request Playground Page" />
    </Frame>
  </Step>
</Steps>

## Frequently Asked Questions (FAQ)

<Accordion title="How can I bypass CloudFlare and other protections?">
  To reliably access sites behind CloudFlare or similar security mechanisms, enable both `js_render` and `premium_proxy` in your requests. These features simulate a full browser environment and route through high-quality residential proxies for protected access.

  You can also enhance your request by adding options like `wait` or `wait_for` to ensure the page fully loads before extracting data, improving accuracy.

  <Tip>Check out our documentation about the [wait](/fetch/features/wait) and [wait\_for](/fetch/features/wait-for) params</Tip>
</Accordion>

<Accordion title="How can I ensure my requests don't fail?">
  You can configure retry logic to handle failed HTTP requests. Learn more in our [guide on retrying requests](/zenrows-academy/retry-failed-requests).
</Accordion>

<Accordion title="How do I extract specific content from a page?">
  You can use the `css_extractor` parameter to directly extract content from a page using CSS selectors. Find out more in our [tutorial on data parsing](/zenrows-academy/how-to-extract-data).
</Accordion>

<Accordion title="Can I integrate Zenrows with Node.js and Cheerio?">
  Yes! You can integrate Zenrows with Node.js and Cheerio for efficient HTML parsing and web scraping. Check out our guide to learn how to combine these tools: [Node.js and Cheerio integration](/zenrows-academy/how-to-extract-data#javascript-with-cheerio).
</Accordion>

<Accordion title="How can I simulate user interactions on the target page?">
  Use the `js_render` and `js_instructions` features to simulate actions such as clicking buttons or filling out forms. Discover more about interacting with web pages in our [JavaScript instructions guide](/fetch/features/js-instructions).
</Accordion>

<Accordion title="How can I scrape faster using Zenrows?">
  You can scrape multiple URLs simultaneously by making concurrent API calls. Check out our guide on [using concurrency](/fetch/features/concurrency#using-concurrency) to boost your scraping speed.
</Accordion>


# Zenrows Fetch with Python: Getting Started
Source: https://docs.zenrows.com/fetch/python

Set up Zenrows Fetch with Python and Requests. Covers installation, API key configuration, and first scraping call.

Want to scrape the web effortlessly using Python? With Zenrows, you can handle complex scraping tasks, bypass anti-bot measures, and extract data from almost any website. This tutorial will guide you step-by-step in setting up a Zenrows scraper with Python.

## How to Use Zenrows with Python

Before we dive in, ensure you have Python 3 installed. If you're new to Python or web scraping, using an IDE such as <a href="https://www.jetbrains.com/pycharm/">PyCharm</a> or <a href="https://code.visualstudio.com/docs/languages/python">Visual Studio Code</a> with the Python extension is recommended for a smoother experience.

We'll create a Python script named `scraper.py` inside a `/scraper` directory. If you need help setting up your environment, check out our [Python web scraping guide](https://www.zenrows.com/blog/web-scraping-python#setup) for detailed instructions on preparing everything.

### Install Python's requests Library

To interact with the Zenrows API, you can use Python's `requests` library. It's a widely used Python library for making HTTP requests, and it simplifies sending requests and handling responses, making it a great tool for integrating with web services like Zenrows.

This approach allows you to manage your API requests directly, allowing greater control over the web scraping process.

To install the `requests` library, use the following command in your terminal:

```bash theme={"dark"}
pip install requests
```

This will install the library to send HTTP requests from your Python scripts.

### Make Your First Request

In this step, you will send your first request to Zenrows using the requests library to scrape content from a simple URL. We will use <a href="https://httpbin.io/get">HTTPBin.io/get</a> endpoint to see how Zenrows processes the request and returns the data.

Here's an example:

```python scraper.py theme={"dark"}
# pip install requests
import requests

url = 'https://httpbin.io/get'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
```

Replace `YOUR_ZENROWS_API_KEY` with your actual API key and run the script:

```bash theme={"dark"}
python scraper.py
```

It'll print something similar to this:

```json theme={"dark"}
{
    "args": {},
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
        // additional headers omitted for brevity...
    },
    "origin": "38.154.5.224:6693",
    "url": "http://httpbin.io/get"
}
```

The response includes useful information, like the `origin`, which shows the IP address from which the request was made. Zenrows automatically rotates your IP address and changes the `User-Agent` for each request, helping to maintain anonymity and avoid blocks.

Perfect, you just learned how to make scraping requests with Python!

### Scrape More Complex Web Pages

While scraping simple sites like HTTPBin is straightforward, many websites, especially those with dynamic content or strict anti-scraping measures, require additional features. Zenrows allows you to bypass these defenses by enabling [JavaScript Rendering](/fetch/features/js-rendering) and using [Premium Proxies](/fetch/features/premium-proxy).

For example, if you try to scrape a page like <a href="https://www.g2.com/products/asana/reviews">G2's Asana reviews</a> without any extra configurations, you'll encounter an error:

```json theme={"dark"}
{
    "code":"REQS002",
    "detail":"The requested URL domain needs JavaScript rendering and/or Premium Proxies due to its high-level security defenses. Please retry by adding 'js_render' and/or 'premium_proxy' parameters to your request.",
    "instance":"/v1",
    "status":400,
    "title":"Www.g2.com requires javascript rendering and premium proxies enabled (REQS002)",
    "type":"https://docs.zenrows.com/api-error-codes#REQS002"
}
```

This error happens because G2 employs advanced security measures that block basic scraping attempts.

Here's how you can modify the request to enable both:

```python scraper.py theme={"dark"}
# pip install requests
import requests

url = 'https://www.g2.com/products/asana/reviews'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
	'js_render': 'true',
	'premium_proxy': 'true',
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
```

Run the script, and this time, Zenrows will handle the heavy lifting by rendering the page's JavaScript and routing your request through premium residential proxies. The response will contain the entire HTML content of the page:

```html theme={"dark"}
<!DOCTYPE html>
<head>
    <meta charset="utf-8">
    <title>Asana Reviews 2024: Details, Pricing, & Features | G2</title>
    <!-- omitted for brevity -->
</head>
<body>
    <!-- page content -->
</body>
</html>
```

This demonstrates how you can scrape more advanced websites that rely on JavaScript or have stringent anti-bot mechanisms in place.

## Troubleshooting

Request failures can happen for various reasons. While some issues can be resolved by adjusting Zenrows parameters, others are beyond your control, such as the target server being temporarily down.

Below are some quick troubleshooting steps you can take:

<Steps>
  <Step title="Retry the Request">
    Network issues or temporary failures can cause your request to fail. Implementing retry logic can solve this by automatically repeating the request. Learn how to add retries in our [Python requests retry guide](https://www.zenrows.com/blog/python-requests-retry).

    Example of retry logic using requests:

    ```python scraper.py theme={"dark"}
    import requests
    from requests.adapters import HTTPAdapter
    from urllib3.util.retry import Retry

    # Define the retry strategy
    retry_strategy = Retry(
        total=4,  # Maximum number of retries
        status_forcelist=[429, 500, 502, 503, 504],  # HTTP status codes to retry on
    )
    # Create an HTTP adapter with the retry strategy and mount it to session
    adapter = HTTPAdapter(max_retries=retry_strategy)

    # Create a new session object
    session = requests.Session()
    session.mount('http://', adapter)
    session.mount('https://', adapter)

    # Make a request using the session object
    response = session.get('https://scrapingcourse.com/ecommerce/')

    if response.status_code == 200:
        print(f'SUCCESS: {response.text}')
    else:
        print("FAILED")
    ```
  </Step>

  <Step title="Verify the Site is Accessible in Your Country">
    Sometimes, the target site might be region-restricted and only available to some proxies. Zenrows automatically selects the best proxy, but if the site is available only in specific regions, specify a geolocation using `proxy_country`.

    Here's how to choose a proxy in the US:

    ```python theme={"dark"}
    params = {
        'premium_proxy': 'true',
        'proxy_country': 'us' # <- choose a premium proxy in the US
        # other configs...
    }
    response = client.get(url, params=params)
    ```

    If the target site requires access from a specific region, adding the `proxy_country` parameter will help.

    <Tip>Check out more about it on our [Geolocation Documentation Page](/fetch/features/proxy-country).</Tip>
  </Step>

  <Step title="Check if the Site is Publicly Accessible">
    Some websites may require a session, so verifying if the site can be accessed without logging in is a good idea. Open the target page in an incognito browser to check this.

    You must handle session management in your requests if authentication credentials are required. You can learn how to scrape a website that requires authentication in our guide: [Web scraping with Python](https://www.zenrows.com/blog/web-scraping-login-python).
  </Step>

  <Step title="Get Help From Zenrows Experts">
    If the issue persists despite following these tips, our support team is available to assist you. Use the [Playground page](https://app.zenrows.com/fetch) or <a href="mailto:success@zenrows.com">contact us via email</a> to get personalized help from Zenrows experts.

    <Frame>
      <img alt="Zenrows Request Playground Page" />
    </Frame>
  </Step>
</Steps>

## Frequently Asked Questions (FAQ)

<Accordion title="How can I bypass CloudFlare and other protections?">
  To reliably access sites behind CloudFlare or similar security mechanisms, enable both `js_render` and `premium_proxy` in your requests. These features simulate a full browser environment and route through high-quality residential proxies for protected access.

  You can also enhance your request by adding options like `wait` or `wait_for` to ensure the page fully loads before extracting data, improving accuracy.

  <Tip>Check out our documentation about the [wait](/fetch/features/wait) and [wait\_for](/fetch/features/wait-for) params</Tip>
</Accordion>

<Accordion title="How can I ensure my requests don't fail?">
  You can configure retry logic to handle failed HTTP requests. Learn more in our [guide on retrying requests](/zenrows-academy/retry-failed-requests).
</Accordion>

<Accordion title="How do I extract specific content from a page?">
  You can use the `css_extractor` parameter to directly extract content from a page using CSS selectors. Find out more in our [tutorial on data parsing](/zenrows-academy/how-to-extract-data).
</Accordion>

<Accordion title="Can I integrate Zenrows with Python's Requests and BeautifulSoup?">
  Yes! You can use Zenrows alongside Python Requests and BeautifulSoup for HTML parsing. Learn how in our guide on [Python Requests and BeautifulSoup integration](/zenrows-academy/how-to-extract-data#python-with-beautifulsoup).
</Accordion>

<Accordion title="How can I simulate user interactions on the target page?">
  Use the `js_render` and `js_instructions` features to simulate actions such as clicking buttons or filling out forms. Discover more about interacting with web pages in our [JavaScript instructions guide](/fetch/features/js-instructions).
</Accordion>

<Accordion title="How can I scrape faster using Zenrows?">
  You can scrape multiple URLs simultaneously by making concurrent API calls. Check out our guide on [using concurrency](/fetch/features/concurrency#using-concurrency) to boost your scraping speed.
</Accordion>


# Concurrency with the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/async-concurrency

Scrape multiple URLs at once with the Go SDK's built-in concurrency limiter.

Go doesn't need a separate async API the way Python or JavaScript do, every SDK call is a regular blocking function you run inside a goroutine. The SDK adds one thing on top of that: an optional internal semaphore that caps how many requests are in flight at once, regardless of how many goroutines you launch.

## Basic usage

<Steps>
  <Step title="Set a concurrency limit">
    Pass `WithMaxConcurrentRequests` when creating the client to cap how many requests run at the same time:

    ```go theme={null} theme={"dark"}
    client := scraperapi.NewClient(
        scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
        scraperapi.WithMaxConcurrentRequests(5),
    )
    ```

    <Note>
      Set `WithMaxConcurrentRequests` to match your Zenrows plan's concurrent request limit. The default is 5. Each client instance enforces its own limit. Two scripts (or two client instances) don't share it, so running both at once can still trigger `429 Too Many Requests` errors. See [Concurrency](/fetch/features/concurrency) for how plan limits work.
    </Note>
  </Step>

  <Step title="Scrape a list of URLs concurrently">
    Launch a goroutine per URL and use a `sync.WaitGroup` to track completion:

    ```go theme={null} theme={"dark"}
    urls := []string{
        "https://www.scrapingcourse.com/ecommerce/",
        "https://www.scrapingcourse.com/ecommerce/page/2/",
        "https://www.scrapingcourse.com/ecommerce/page/3/",
    }

    var wg sync.WaitGroup
    for i, url := range urls {
        wg.Add(1)
        go func(i int, url string) {
            defer wg.Done()

            response, err := client.Get(context.Background(), url, nil)
            if err != nil {
                fmt.Println(i, err)
                return
            }

            fmt.Printf("[#%d]: %s\n", i, response.Status())
        }(i, url)
    }
    ```

    Each goroutine that acquires a slot from the internal semaphore runs its request, then releases the slot, so no more than `WithMaxConcurrentRequests` requests from this client are in flight at once, no matter how many goroutines you've launched.
  </Step>

  <Step title="Wait for everything to finish">
    ```go theme={null} theme={"dark"}
    wg.Wait()
    fmt.Println("done")
    ```
  </Step>
</Steps>

## When to think about concurrency

You don't need to do anything special to run requests concurrently. Launching goroutines already benefits from the internal semaphore. The default limit is 5. Set `WithMaxConcurrentRequests` explicitly when you want to match a higher plan limit, or lower it if you're seeing `429` errors.

## POST and PUT in parallel

The same pattern works with `client.Post()` and `client.Put()`:

<Tabs>
  <Tab title="POST in parallel">
    ```go theme={null} theme={"dark"}
    go func(i int, url string) {
        defer wg.Done()

        response, err := client.Post(context.Background(), url, nil, map[string]string{"key": "value"})
        if err != nil {
            fmt.Println(i, err)
            return
        }

        fmt.Printf("[#%d]: %s\n", i, response.Status())
    }(i, url)
    ```
  </Tab>

  <Tab title="PUT in parallel">
    ```go theme={null} theme={"dark"}
    go func(i int, url string) {
        defer wg.Done()

        response, err := client.Put(context.Background(), url, nil, map[string]string{"key": "value"})
        if err != nil {
            fmt.Println(i, err)
            return
        }

        fmt.Printf("[#%d]: %s\n", i, response.Status())
    }(i, url)
    ```
  </Tab>
</Tabs>

## How it works under the hood

<Info>
  `WithMaxConcurrentRequests(n)` creates a buffered channel of size `n` inside the client. Every call to `Get`, `Post`, `Put`, or `Scrape` acquires a slot from that channel before sending the request, and releases it once the response comes back. This applies uniformly across methods, there's no separate limit for GET versus POST/PUT. Retry behavior configured via `WithMaxRetryCount` applies the same way regardless of how many requests run in parallel. See [Error Handling and Retries](/fetch/sdk/go/error-handling-and-retries) for details.
</Info>

<Warning>
  The SDK's default concurrency is 5. If your plan allows more concurrent requests and you don't set `WithMaxConcurrentRequests` higher, you may be leaving throughput on the table. If you launch more goroutines than your plan allows, you'll get `429 Too Many Requests` errors.
</Warning>

## Troubleshooting

* **`429 Too Many Requests` even with a `WithMaxConcurrentRequests` value that matches your plan**: Another script or process is using the same API key at the same time. Concurrency limits are per plan, not per client instance, so simultaneous scripts share the same real ceiling even though each `*scraperapi.Client` only tracks its own local limit.
* **Goroutines seem to run one at a time**: Confirm you're not accidentally blocking inside the goroutine before calling `wg.Done()`, and that you're not reusing a single `context.Context` with an already-expired deadline.
* **Program exits before all goroutines finish**: Make sure `wg.Wait()` runs before `main()` returns; a goroutine that hasn't called `Done()` yet is silently dropped when the program exits.
* **Cancelling a request doesn't free a concurrency slot right away**: Zenrows keeps the slot occupied for up to a few minutes after a client-side cancellation while it finishes processing server-side.

## Pricing

Requests are billed identically whether they run one at a time or concurrently. Cost depends on the parameters you send per request, not on how many run in parallel:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `JSRender: true` | 5 |
| `UsePremiumProxies: true` | 10 |
| `JSRender: true` + `UsePremiumProxies: true` | 25 |

Running requests concurrently doesn't add cost by itself. It only changes how fast the same requests complete. See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="What's the default concurrency limit if I don't set one?">
    The default is 5. Even without passing `WithMaxConcurrentRequests` explicitly, the SDK creates an internal semaphore capped at 5. Set it explicitly if your plan allows more, or lower it if you're seeing `429 Too Many Requests` errors.
  </Accordion>

  <Accordion title="Do GET, POST, and PUT share the same concurrency limit?">
    Yes. Every call from the same `*scraperapi.Client`, regardless of method, goes through one shared internal semaphore.
  </Accordion>

  <Accordion title="Do I need a worker pool library to do this?">
    No. A `sync.WaitGroup` plus goroutines, as shown above, is enough. Reach for a worker pool library only if you need more advanced scheduling, like rate limiting independent of the SDK's own semaphore.
  </Accordion>
</AccordionGroup>


# Custom Headers with the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/custom-headers

Send custom headers to the target website through the Zenrows Go SDK.

By default, Zenrows sends an optimized set of headers to the target site on your behalf. If you need to forward specific headers instead, like a `Referer` or a session `Cookie`, set the `CustomHeaders` field on `RequestParameters`.

```go theme={null} theme={"dark"}
package main

import (
    "context"
    "fmt"
    "log"
    "net/http"

    scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"
)

func main() {
    client := scraperapi.NewClient(
        scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
    )

    response, err := client.Get(
        context.Background(),
        "https://www.scrapingcourse.com/ecommerce/",
        &scraperapi.RequestParameters{
            CustomHeaders: http.Header{
                "Referer": []string{"https://www.google.com"},
            },
        },
    )
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(response.String())
}
```

## What happens when you set CustomHeaders

As soon as `CustomHeaders` is non-empty, the SDK automatically sets `custom_headers=true` as a query parameter for you, so you don't need to add that yourself. This enables your custom headers for the target request, while Zenrows continues to manage browser-fingerprinting headers automatically.

## Headers you can and can't override

Not every header can be customized. Zenrows always manages browser-fingerprinting headers itself to keep success rates high and avoid anti-bot detection, even when `custom_headers` is enabled:

| Header | Customizable? |
| - | - |
| `Referer` | Yes |
| `Cookie` | Yes |
| `Authorization` | Yes |
| Most other headers | Yes, once `custom_headers=true` is set |
| `User-Agent` | No, always managed by Zenrows |
| `Sec-Ch-Ua`, `Sec-Ch-Ua-Mobile`, `Sec-Ch-Ua-Platform` | No |
| `Accept-Encoding`, `Accept-Language` | No |
| `Sec-Fetch-Mode`, `Sec-Fetch-Site`, `Sec-Fetch-User`, `Sec-Fetch-Dest` | No |
| `Connection`, `Upgrade-Insecure-Requests`, `Cache-Control` | No |

<Warning>
  Setting a `User-Agent` (or any other browser-fingerprinting header from the table above) in `CustomHeaders` has no effect on the request sent to the target site. Zenrows ignores it and keeps using its own managed value to maintain a consistent, realistic browser fingerprint. See [Custom Headers](https://docs.zenrows.com/fetch/features/headers) in Fetch docs for the full explanation.
</Warning>

<Note>
  The SDK does send its own `User-Agent` (`zenrows-go/<version>`) as part of the HTTP request it makes to Zenrows' API endpoint. This identifies the request as coming from the Go SDK for Zenrows' own logging, it is not the header Zenrows sends to the target website.
</Note>

## Combining with other parameters

`CustomHeaders` is just one field on `RequestParameters`, so it combines naturally with everything else:

```go theme={null} theme={"dark"}
response, err := client.Get(
    context.Background(),
    url,
    &scraperapi.RequestParameters{
        UsePremiumProxies: true,
        CustomHeaders: http.Header{
            "Referer": []string{"https://www.google.com"},
        },
    },
)
```

<Note>
  If a request keeps failing after you add custom headers, try removing them first to confirm whether they're the cause. Zenrows' defaults are usually the safer starting point.
</Note>

## Troubleshooting

* **Success rate drops after adding custom headers**: This is expected per the warning above. Remove `CustomHeaders` first to confirm whether it's the cause; Zenrows' defaults are usually the safer starting point.
* **Header doesn't seem to reach the target site**: Confirm you're setting `CustomHeaders` on `RequestParameters` (forwarded to the target), not adding it somewhere else, like a header on the outgoing request to Zenrows' own API.
* **Cookies aren't persisting across requests**: Each SDK call is a separate, stateless request. Set the `Cookie` header explicitly in `CustomHeaders` on every call if you need to maintain a session.
* **`custom_headers=true` shows up unexpectedly in logs**: This is set automatically whenever `CustomHeaders` is non-empty, it's not something you need to remove.

## Pricing

Custom headers don't change the credit cost on their own. Credit cost still depends on `JSRender` and `UsePremiumProxies`:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `JSRender: true` | 5 |
| `UsePremiumProxies: true` | 10 |
| `JSRender: true` + `UsePremiumProxies: true` | 25 |

See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Do I need to set custom_headers myself?">
    No. Setting a non-empty `CustomHeaders` field on `RequestParameters` sets `custom_headers=true` automatically.
  </Accordion>

  <Accordion title="Will my custom headers always be used exactly as sent?">
    Non-fingerprinting headers like `Referer`, `Cookie`, and `Authorization` are forwarded exactly as you set them. Browser-fingerprinting headers like `User-Agent` are never forwarded as sent, Zenrows always manages those regardless of what you pass. This is why overriding headers you can customize can still lower success rates on protected sites, since you lose Zenrows' tuned defaults for that request.
  </Accordion>

  <Accordion title="Can I set headers meant for Zenrows itself, like an API version header?">
    No. `CustomHeaders` is forwarded to the target website, not to Zenrows' API. Zenrows-specific options belong in the other `RequestParameters` fields, or in `CustomParams` for anything not yet exposed as a typed field.
  </Accordion>
</AccordionGroup>


# Error Handling and Retries with the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/error-handling-and-retries

Configure automatic retries in the Zenrows Go SDK and handle its custom error types.

The Go SDK separates two kinds of failure: errors returned before a request is even sent (bad configuration, invalid parameters, malformed URLs), and unsuccessful HTTP responses that still come back as a valid `*Response`. Automatic retries only apply to the second kind.

## Basic usage

Retries are **off by default** (`maxRetryCount` is `0`). Pass `WithMaxRetryCount` to turn them on:

```go theme={null} theme={"dark"}
client := scraperapi.NewClient(
    scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
    scraperapi.WithMaxRetryCount(3),
)

response, err := client.Get(context.Background(), "https://www.scrapingcourse.com/ecommerce/", nil)
```

With `WithMaxRetryCount(3)`, a failed request is retried up to 3 additional times before the SDK returns the last response or error it received.

## When to enable retries

Enable retries (2 to 3 is a reasonable starting point) whenever you're running requests at scale or in production, anywhere a transient `429` or `500` shouldn't fail your whole job. Leave retries off (the default) for quick one-off scripts or debugging, where you'd rather see the raw failure immediately.

## Which failures get retried

The SDK retries a request when either of these is true:

* The request failed at the network level (DNS failure, connection reset, timeout), regardless of status code.
* The response came back with one of these status codes:

| Status code | Meaning |
| - | - |
| `422` | Unprocessable Entity |
| `429` | Too Many Requests |
| `500` | Internal Server Error |

<Warning>
  `502`, `503`, and `504` are **not** retried automatically by the Go SDK. If your use case needs to retry those too, check `response.StatusCode()` yourself and re-issue the request in a loop.
</Warning>

Between attempts, the SDK waits using `WithRetryWaitTime` as a starting point (default 5 seconds), doubling on each subsequent attempt, capped at `WithRetryMaxWaitTime` (default 30 seconds):

```go theme={null} theme={"dark"}
client := scraperapi.NewClient(
    scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
    scraperapi.WithMaxRetryCount(5),
    scraperapi.WithRetryWaitTime(20*time.Second),
    scraperapi.WithRetryMaxWaitTime(25*time.Second),
)
```

<Tip>
  You aren't charged for retried attempts that fail. Only successful responses count toward your usage. See the [Retry Failed Requests](/zenrows-academy/retry-failed-requests) guide for more on how this works.
</Tip>

## Handling failures

<Tabs>
  <Tab title="HTTP error status">
    Even after retries are exhausted, the SDK returns a `*Response` with `err == nil`. It doesn't turn a non-2xx status into a Go `error` on its own. Always check the response before trusting its content:

    ```go theme={null} theme={"dark"}
    response, err := client.Get(context.Background(), url, nil)
    if err != nil {
        log.Fatal(err)
    }

    if response.IsSuccess() {
        fmt.Println(response.String())
    } else {
        fmt.Println("Failed after retries:", response.StatusCode())
        if prob := response.Problem(); prob != nil {
            fmt.Println("Detail:", prob.Detail)
        }
    }
    ```
  </Tab>

  <Tab title="SDK and network errors">
    The other four cases return a non-`nil` `error` instead of a response, before any HTTP call is attempted (except for genuine network failures, which happen mid-request):

    ```go theme={null} theme={"dark"}
    response, err := client.Get(ctx, targetURL, params)
    if err != nil {
        var notConfigured scraperapi.NotConfiguredError
        var invalidMethod scraperapi.InvalidHTTPMethodError
        var invalidURL scraperapi.InvalidTargetURLError
        var invalidParam scraperapi.InvalidParameterError

        switch {
        case errors.As(err, &notConfigured):
            log.Fatal("missing API key")
        case errors.As(err, &invalidMethod):
            log.Fatal("unsupported HTTP method")
        case errors.As(err, &invalidURL):
            log.Fatal("bad target URL: ", err)
        case errors.As(err, &invalidParam):
            log.Fatal("bad request parameters: ", err)
        default:
            log.Fatal("network or unexpected error: ", err)
        }
    }
    ```

    <Note>
      Use `context.WithTimeout(ctx, 3*time.Minute)` if you want a request canceled after a fixed duration. We recommend not going below 3 minutes, since canceling a request client-side doesn't immediately free up the concurrency slot on Zenrows' side.
    </Note>
  </Tab>
</Tabs>

## The four SDK error types

| Error type | When it happens |
| - | - |
| `NotConfiguredError` | No API key was set (neither `WithAPIKey` nor `ZENROWS_API_KEY`). |
| `InvalidHTTPMethodError` | You called `Scrape` with a method other than `GET`, `POST`, or `PUT`. |
| `InvalidTargetURLError` | The target URL is empty or fails to parse; wraps the underlying `url.Parse` error via `Unwrap()`. |
| `InvalidParameterError` | A `RequestParameters` field fails validation, for example, `ScreenshotQuality` outside 1 to 100, or `Screenshot: false` combined with `ScreenshotFullPage: true`. |

All four are returned as values (not pointers), and `InvalidTargetURLError` supports `errors.Unwrap` so you can inspect the underlying parse error with `errors.As` or `errors.Is`.

## Troubleshooting

* **Response still fails after retries are exhausted**: Check `response.StatusCode()` and look it up in [API Error Codes](/api-error-codes). Some failures, like an invalid API key, aren't fixed by retrying.
* **Getting `InvalidParameterError` for a parameter you didn't set**: Several `RequestParameters` fields depend on each other, for example, `ScreenshotFullPage` requires `Screenshot: true`, and `WaitForSelector` requires `JSRender: true`. Check `Validate()`'s rules in the SDK source if the message isn't specific enough.
* **`429` errors even with retries enabled**: You're likely exceeding your plan's concurrency limit, not hitting a one-off rate limit. See [Concurrency](/fetch/sdk/go/async-concurrency).
* **Retries seem to take a long time**: That's the doubling backoff by design, capped at `WithRetryMaxWaitTime`. Lower `WithMaxRetryCount` or `WithRetryMaxWaitTime` if latency matters more than success rate.

## Pricing

Retried attempts that fail are **not** billed. Only the final successful response counts:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `JSRender: true` | 5 |
| `UsePremiumProxies: true` | 10 |
| `JSRender: true` + `UsePremiumProxies: true` | 25 |

<Note>`404` and `410` responses count as successful and are billed, since the request itself completed correctly.</Note>

## FAQ

<AccordionGroup>
  <Accordion title="What's the default number of retries?">
    Zero. Retries are disabled unless you explicitly pass `WithMaxRetryCount(n)` with `n > 0`.
  </Accordion>

  <Accordion title="Which status codes trigger an automatic retry?">
    `422`, `429`, and `500`, plus any network-level failure. `502`, `503`, and `504` are not retried automatically.
  </Accordion>

  <Accordion title="Does the SDK ever return a Go error for a non-2xx response?">
    No. A non-2xx HTTP response is still a successful round trip from the SDK's point of view, it returns a `*Response` with `err == nil`. Check `response.IsError()` or `response.StatusCode()` yourself.
  </Accordion>
</AccordionGroup>

## Look up a specific error code

For a full breakdown of Zenrows error codes (like `AUTH002` or `RESP001`) and how to resolve each one, see the [API Error Codes](/api-error-codes) reference.


# Frequently Asked Questions
Source: https://docs.zenrows.com/fetch/sdk/go/faq

Answers to common questions about the Zenrows Go SDK.

### Setup and compatibility

<AccordionGroup>
  <Accordion title="What Go versions does the SDK support?" icon="golang">
    Go 1.23.1 or later, matching the version pinned in the SDK's `go.mod`.
  </Accordion>

  <Accordion title="What's the correct import path?" icon="download">
    `github.com/zenrows/zenrows-go-sdk/service/api`, singular `service`. The package name you import it as is `scraperapi`.
  </Accordion>

  <Accordion title="Does the SDK support PATCH or DELETE requests?" icon="ban">
    No. Fetch only accepts `GET`, `POST`, and `PUT`. Calling `Scrape` with any other method returns an `InvalidHTTPMethodError` immediately.
  </Accordion>

  <Accordion title="Does the SDK support Proxy Mode?" icon="route">
    No. The SDK talks to Fetch's endpoint directly (API Mode). If you want to use Zenrows as a proxy in your own HTTP client instead, see the [Proxy Mode documentation](/fetch/setup#connection-modes) rather than the SDK.
  </Accordion>
</AccordionGroup>

### Requests, errors, and concurrency

<AccordionGroup>
  <Accordion title="How do I set a request timeout?" icon="clock">
    Pass a `context.Context` created with `context.WithTimeout` to `Get`, `Post`, or `Put`. We recommend not setting a timeout below 3 minutes, since canceling a request client-side doesn't immediately free up the concurrency slot on Zenrows' side.
  </Accordion>

  <Accordion title="Why doesn't a failed request return a Go error?" icon="shield-halved">
    By design, a non-2xx HTTP response is still a successful round trip as far as the SDK is concerned, it returns a `*Response` with `err == nil`. Check `response.IsSuccess()`, `response.IsError()`, or `response.StatusCode()` yourself. See [Error Handling and Retries](/fetch/sdk/go/error-handling-and-retries).
  </Accordion>

  <Accordion title="What's the default concurrency limit?" icon="gauge-high">
    The default is 5. The SDK creates an internal semaphore capped at 5 even without an explicit `WithMaxConcurrentRequests` call. Set it higher if your plan allows more concurrent requests, or lower it if you're seeing `429 Too Many Requests` errors.
  </Accordion>

  <Accordion title="Which status codes are retried automatically?" icon="rotate">
    `422`, `429`, and `500`, plus network-level failures. `502`, `503`, and `504` are not retried by default.
  </Accordion>
</AccordionGroup>

### Project and support

<AccordionGroup>
  <Accordion title="Is there a Python or Node.js version of the SDK?" icon="node-js">
    Yes. See the [Zenrows Python SDK](/fetch/sdk/python/introduction), published as <a href="https://pypi.org/project/zenrows/">`zenrows` on PyPI</a>, and the [Zenrows Node.js SDK](/fetch/sdk/nodejs/introduction), published as <a href="https://www.npmjs.com/package/zenrows">`zenrows` on npm</a>.
  </Accordion>

  <Accordion title="Where can I see the SDK's source code or report a bug?" icon="github">
    The SDK is open source on <a href="https://github.com/ZenRows/zenrows-go-sdk">GitHub</a>, under the `service/api` directory. Open an issue there for bugs, or a pull request if you'd like to contribute.
  </Accordion>

  <Accordion title="What license is the SDK released under?" icon="scale-balanced">
    MIT. You can use, modify, and distribute it freely; see the <a href="https://github.com/ZenRows/zenrows-go-sdk/blob/main/LICENSE">LICENSE</a> file in the repository.
  </Accordion>
</AccordionGroup>

<Card title="Still stuck?" icon="life-ring" href="mailto:success@zenrows.com">
  Check the [API Error Codes](/api-error-codes) reference, or contact support directly.
</Card>


# Installing the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/installation

Install the Zenrows Go SDK and get your API key ready to make requests.

## Requirements

<CardGroup>
  <Card title="Go 1.23+" icon="golang">
    Required by the SDK's `go.mod`.
  </Card>

  <Card title="API key" icon="key" href="https://app.zenrows.com/register?p=free">
    Create a free account to get yours, no credit card required.
  </Card>
</CardGroup>

<Steps>
  <Step title="Install the package">
    Add the SDK to your module with `go get`:

    ```bash theme={null} theme={"dark"}
    go get github.com/zenrows/zenrows-go-sdk/service/api
    ```

    <Warning>
      Use the singular path `service/api`. Some copy-pasted snippets show `services/api` (plural), which doesn't match the module's actual directory layout and won't resolve.
    </Warning>

    The package name is `scraperapi`, not `zenrows`, so import it with an explicit alias:

    ```go theme={null} theme={"dark"}
    import (
        scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"
    )
    ```
  </Step>

  <Step title="Verify the install">
    Create a client and print its type to confirm everything resolves:

    ```go theme={null} theme={"dark"}
    package main

    import (
        "fmt"

        scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"
    )

    func main() {
        client := scraperapi.NewClient(
            scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
        )
        fmt.Printf("%T\n", client) // *scraperapi.Client
    }
    ```

    If it prints `*scraperapi.Client` without a compile error, you're ready to go.
  </Step>

  <Step title="Get your API key">
    Your API key authenticates every request you send through the SDK. Grab it from the [Zenrows Request Playground](https://app.zenrows.com/fetch) after signing up.

    <Warning>
      Keep your API key private. Load it from the `ZENROWS_API_KEY` environment variable (read automatically by `scraperapi.NewClient()`) instead of hardcoding it in scripts you commit to version control.
    </Warning>
  </Step>
</Steps>

<Card title="Next: Quickstart" icon="rocket" href="/fetch/sdk/go/quickstart">
  Make your first request with the SDK.
</Card>


# Introduction to the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/introduction

Official Zenrows Go SDK for Fetch. Handles authentication, retries, and concurrency so you can scrape with less boilerplate.

The <a href="https://github.com/ZenRows/zenrows-go-sdk">Zenrows Go SDK</a> (package `scraperapi`) is a lightweight client for [Fetch](/fetch/setup). It wraps the same `https://api.zenrows.com/v1` endpoint you'd otherwise call with `net/http`, adding automatic retries, concurrency limiting, and a typed request builder on top.

<Info>
  Every request parameter is a typed field on a `RequestParameters` struct instead of a loose map or dictionary, so the compiler catches a typo'd parameter name before your code ever makes a network call. Every method also takes a `context.Context`, so cancellation and timeouts work the same way they do with any other Go HTTP call.
</Info>

## What the SDK handles for you

* **Authentication** - your API key is set once in the client constructor, not repeated on every call.
* **Retries** - optionally retry failed requests (network errors, `422`, `429`, `500`) with exponential backoff, without writing retry logic yourself.
* **Concurrency** - cap how many requests run in parallel with an internal semaphore, to match your Zenrows plan's concurrency limit.
* **Typed parameters** - every Fetch parameter (`js_render`, `premium_proxy`, `css_extractor`, and more) is a typed field on `RequestParameters`, not a hand-built query string.

## What you'll need

<CardGroup>
  <Card title="Go 1.23+" icon="golang">
    Required by the SDK's `go.mod`.
  </Card>

  <Card title="API key" icon="key" href="https://app.zenrows.com/register?p=free">
    Create a free account to get yours, no credit card required.
  </Card>
</CardGroup>

<Note>
  The SDK only covers Fetch's GET, POST, and PUT modes; it doesn't expose PATCH or DELETE. For the full list of request parameters (`premium_proxy`, `css_extractor`, `extract`, and so on), see the [Fetch setup guide](/fetch/setup). Most parameters work the same way through `RequestParameters` regardless of method, with one exception: `JSRender` only works on GET requests, not POST or PUT. See [POST and PUT Requests](/fetch/sdk/go/post-put-requests).
</Note>

<Tip>
  Looking for a different language? Zenrows also maintains a <a href="https://pypi.org/project/zenrows/">Python SDK</a> and a <a href="https://www.npmjs.com/package/zenrows">Node.js SDK</a>.
</Tip>

## Explore the Go SDK

<CardGroup>
  <Card title="Installation" icon="download" href="/fetch/sdk/go/installation">
    Install the package and get your API key.
  </Card>

  <Card title="Quickstart" icon="rocket" href="/fetch/sdk/go/quickstart">
    Make your first request in a few lines of code.
  </Card>

  <Card title="POST and PUT Requests" icon="send" href="/fetch/sdk/go/post-put-requests">
    Submit forms or data payloads.
  </Card>

  <Card title="Concurrency" icon="bolt" href="/fetch/sdk/go/async-concurrency">
    Scrape many URLs at once with goroutines.
  </Card>

  <Card title="Error Handling and Retries" icon="shield" href="/fetch/sdk/go/error-handling-and-retries">
    Configure automatic retries and handle failures.
  </Card>

  <Card title="Response Object" icon="file-text" href="/fetch/sdk/go/response-object">
    What you get back from every request.
  </Card>

  <Card title="Custom Headers" icon="tags" href="/fetch/sdk/go/custom-headers">
    Forward specific headers to the target site.
  </Card>

  <Card title="FAQ" icon="help-circle" href="/fetch/sdk/go/faq">
    Answers to common questions.
  </Card>
</CardGroup>


# POST and PUT Requests with the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/post-put-requests

Send POST and PUT requests through the Zenrows Go SDK, including form data payloads.

Besides `client.Get()`, the SDK exposes `client.Post()` and `client.Put()` for targets that expect a form submission or a data payload. Both take the same `context.Context`, target URL, and `*RequestParameters` as `Get`, plus a `body any` argument for the request payload.

## Basic usage

```go theme={null} theme={"dark"}
package main

import (
    "context"
    "fmt"
    "log"

    scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"
)

func main() {
    client := scraperapi.NewClient(
        scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
    )

    body := map[string]string{"key": "value"}

    response, err := client.Post(context.Background(), "https://httpbin.io/anything", nil, body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(response.String())
}
```

`client.Put()` works identically, replacing `Post` with `Put`:

```go theme={null} theme={"dark"}
response, err := client.Put(context.Background(), "https://httpbin.io/anything", nil, body)
```

## When to use POST or PUT

* **POST**: submitting a form on the target site, triggering a search or filter that requires a form submission, or calling a write endpoint on a target API.
* **PUT**: updating a resource on a target API that expects `PUT` semantics.
* **Neither**: most scraping tasks (reading page content) only need `client.Get()`. Reach for POST/PUT specifically when the target requires it, not by default.

<Tabs>
  <Tab title="POST">
    ```go theme={null} theme={"dark"}
    response, err := client.Post(
        context.Background(),
        "https://httpbin.io/anything",
        &scraperapi.RequestParameters{UsePremiumProxies: true},
        map[string]string{"key": "value"},
    )
    ```
  </Tab>

  <Tab title="PUT">
    ```go theme={null} theme={"dark"}
    response, err := client.Put(
        context.Background(),
        "https://httpbin.io/anything",
        &scraperapi.RequestParameters{UsePremiumProxies: true},
        map[string]string{"key": "value"},
    )
    ```
  </Tab>
</Tabs>

## Combining with request parameters

`RequestParameters` still works the same way it does for GET requests, so you can enable `UsePremiumProxies`, or other Fetch parameters, alongside `body`:

```go theme={null} theme={"dark"}
response, err := client.Post(
    context.Background(),
    "https://httpbin.io/anything",
    &scraperapi.RequestParameters{UsePremiumProxies: true},
    map[string]string{"key": "value"},
)
```

<Warning>
  `JSRender` only works with GET requests (`client.Get()`). It is not supported on POST or PUT requests.
</Warning>

## Sending a struct instead of a map

Since `body` is typed as `any`, a struct works the same way as a map, as long as it has exported fields (or JSON tags), and the underlying HTTP client encodes it as JSON automatically:

```go theme={null} theme={"dark"}
type Payload struct {
    Key string `json:"key"`
}

response, err := client.Post(context.Background(), "https://httpbin.io/anything", nil, Payload{Key: "value"})
```

## No PATCH or DELETE

Fetch, and this SDK, only support `GET`, `POST`, and `PUT`. Calling the lower-level `Scrape` method with any other HTTP method string returns an `InvalidHTTPMethodError` without making a network call:

```go theme={null} theme={"dark"}
// This returns InvalidHTTPMethodError, not a network error
_, err := client.Scrape(context.Background(), "PATCH", "https://httpbin.io/anything", nil, nil)
```

## Troubleshooting

* **Target site doesn't reflect the submitted data**: Some forms require specific headers (like a `Referer`) to accept the submission. See [Custom Headers](/fetch/sdk/go/custom-headers).
* **Getting `REQS005 Method Not Allowed`**: The target endpoint doesn't accept the HTTP verb you're using. Zenrows only supports `GET`, `POST`, and `PUT`; confirm the target expects one of these.
* **Body arrives at the target site as `{}`**: You likely passed an unexported struct field (lowercase, no JSON tag) or a `nil` map. Confirm your struct fields are exported and tagged correctly.
* **Target site returns an error for a POST/PUT you know works with plain `curl`**: Confirm you haven't accidentally set `JSRender: true`, it's silently unsupported on non-GET requests and can cause the request to fail.

## Pricing

POST and PUT requests are billed the same way as GET requests. Cost depends on the parameters you use, not the HTTP method:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `UsePremiumProxies: true` | 10 |

`JSRender` isn't available on POST/PUT, so the 5-credit and 25-credit costs only apply to GET requests. See [Zenrows Pricing](/first-steps/pricing) for plan-specific rates.

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="What type does the body parameter need to be?">
    Any JSON-serializable Go value, a `struct`, a `map[string]string`, a `map[string]any`, or even a raw `[]byte` or `string` if you want full control over the encoding.
  </Accordion>

  <Accordion title="Can I send form-encoded data instead of JSON?">
    The SDK doesn't expose a dedicated form-encoding option. Encode your data as a `url.Values`-formatted string yourself and pass it as the `body` argument if the target site expects `application/x-www-form-urlencoded`.
  </Accordion>

  <Accordion title="Does Zenrows support DELETE or PATCH?">
    No. Only `GET`, `POST`, and `PUT` are supported by Fetch and this SDK. Calling `Scrape` with any other method returns an `InvalidHTTPMethodError`.
  </Accordion>
</AccordionGroup>


# Making your first request with the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/quickstart

Make your first request with the Zenrows Go SDK in a few lines of code.

This guide makes a single GET request with the Zenrows Go SDK: create a client, send the request, read the response.

<Note>Haven't installed the SDK yet? Follow [Installation](/fetch/sdk/go/installation) first.</Note>

## Basic usage

<Steps>
  <Step title="Create a client">
    Import the `scraperapi` package and instantiate a client with your API key:

    ```go theme={null} theme={"dark"}
    package main

    import (
        scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"
    )

    func main() {
        client := scraperapi.NewClient(
            scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
        )
    }
    ```
  </Step>

  <Step title="Send a GET request">
    Call `client.Get()` with a `context.Context` and the URL you want to scrape:

    ```go theme={null} theme={"dark"}
    package main

    import (
        "context"
        "fmt"
        "log"

        scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"
    )

    func main() {
        client := scraperapi.NewClient(
            scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
        )

        response, err := client.Get(context.Background(), "https://www.scrapingcourse.com/ecommerce/", nil)
        if err != nil {
            log.Fatal(err)
        }

        fmt.Println(response.String())
    }
    ```

    `response` is a `*scraperapi.Response`, a thin wrapper that also exposes the original `*http.Response`.

    <Tip>See [The Response Object](/fetch/sdk/go/response-object) for everything available on it.</Tip>
  </Step>

  <Step title="Add request parameters">
    Pass a `*scraperapi.RequestParameters` instead of `nil` to enable Fetch features like JavaScript rendering or Premium Proxies:

    ```go theme={null} theme={"dark"}
    package main

    import (
        "context"
        "fmt"
        "log"

        scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"
    )

    func main() {
        client := scraperapi.NewClient(
            scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
        )

        response, err := client.Get(context.Background(), "https://www.scrapingcourse.com/ecommerce/", &scraperapi.RequestParameters{
            JSRender:          true,
            UsePremiumProxies: true,
            ProxyCountry:      "us",
        })
        if err != nil {
            log.Fatal(err)
        }

        fmt.Println(response.String())
    }
    ```

    <Tip>
      `RequestParameters` exposes every parameter documented in the [Fetch setup guide](/fetch/setup), including `AutoParse`, `CSSExtractor`, `WaitForSelector`, `BlockResources`, and `Outputs`. They all work exactly the same way through the SDK.
    </Tip>
  </Step>

  <Step title="Check the result">
    Since the SDK doesn't turn a non-2xx response into a Go `error`, always check the status before using the content:

    ```go theme={null} theme={"dark"}
    if response.IsSuccess() {
        fmt.Println(response.String())
    } else {
        fmt.Println("Request failed with status", response.StatusCode())
    }
    ```

    See [Error Handling and Retries](/fetch/sdk/go/error-handling-and-retries) for how to configure automatic retries so you don't have to handle every transient failure manually.
  </Step>
</Steps>

## Troubleshooting

* **`response.StatusCode()` is `401` or similar auth error**: Double-check your API key was copied correctly from the [Zenrows Request Playground](https://app.zenrows.com/fetch) with no extra whitespace. See [`AUTH002`](/api-error-codes#auth002-invalid-api-key).
* **`response.String()` is empty or looks like an error page, not the target site**: The target likely needs `JSRender: true` and/or `UsePremiumProxies: true`. See [`REQS002`](/api-error-codes#reqs002-request-requirements-unsatisfied).
* **Request hangs for a long time**: Pass a `context.Context` created with `context.WithTimeout(...)`. We recommend not going below 3 minutes for protected sites using `JSRender`.
* **`429 Too Many Requests`**: You've hit your plan's concurrency or rate limit. See [Concurrency](/fetch/sdk/go/async-concurrency).

## Pricing

Cost depends on the parameters you send, not on the SDK itself:

| Configuration | Credits per request |
| - | - |
| Basic request (no `JSRender`/`UsePremiumProxies`) | 1 |
| `JSRender: true` | 5 |
| `UsePremiumProxies: true` | 10 |
| `JSRender: true` + `UsePremiumProxies: true` | 25 |

Only successful requests are billed. See [Zenrows Pricing](/first-steps/pricing) for exact per-plan rates.

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Do I need to pass params for every request?">
    No, passing `nil` is valid, it's the same as an empty `&scraperapi.RequestParameters{}`.
  </Accordion>

  <Accordion title="Why doesn't a failed request return a Go error?">
    By design, the SDK returns a `*Response` even for error status codes. Check `response.IsSuccess()` or `response.StatusCode()` yourself. See [Error Handling and Retries](/fetch/sdk/go/error-handling-and-retries).
  </Accordion>

  <Accordion title="Can I reuse the same client for multiple requests?">
    Yes. Create one `*scraperapi.Client` and call `.Get()`/`.Post()`/`.Put()` on it as many times as needed, there's no need to recreate it per request. It's also safe to share across goroutines.
  </Accordion>
</AccordionGroup>

## What's next

<CardGroup>
  <Card title="POST and PUT Requests" icon="send" href="/fetch/sdk/go/post-put-requests">
    Submit forms or data payloads.
  </Card>

  <Card title="Concurrency" icon="bolt" href="/fetch/sdk/go/async-concurrency">
    Scrape many URLs in parallel.
  </Card>

  <Card title="Custom Headers" icon="tags" href="/fetch/sdk/go/custom-headers">
    Forward specific headers to the target site.
  </Card>
</CardGroup>


# The Response Object with the Go SDK
Source: https://docs.zenrows.com/fetch/sdk/go/response-object

Understand the *scraperapi.Response type returned by every Zenrows Go SDK call.

Every method on `*scraperapi.Client` (`Get`, `Post`, `Put`, `Scrape`) returns a `*scraperapi.Response` on success. It wraps an underlying `go-resty` response, and also exposes the original `*http.Response` if you need it.

## Commonly used methods

| Method | Returns | Use it when |
| - | - | - |
| `response.String()` | Response body as a `string` | Reading HTML or plain text |
| `response.Body()` | Response body as `[]byte` | Handling binary data (PDFs, screenshots, images) |
| `response.StatusCode()` | HTTP status code, e.g. `200` | Checking exactly what happened |
| `response.Status()` | Status text, e.g. `"200 OK"` | Logging or display |
| `response.IsSuccess()` | `true` if status is 200 to 299 | Quick success check |
| `response.IsError()` | `true` if status is 400 or higher | Quick failure check |
| `response.Header()` | `http.Header` of response headers | Reading Zenrows monitoring headers |
| `response.RawResponse` | The original `*http.Response` | Anything the wrapper doesn't expose |

<Tabs>
  <Tab title="Text / HTML">
    ```go theme={null} theme={"dark"}
    response, err := client.Get(context.Background(), url, nil)
    html := response.String()
    ```
  </Tab>

  <Tab title="Binary (PDF, screenshot)">
    ```go theme={null} theme={"dark"}
    response, err := client.Get(context.Background(), url, &scraperapi.RequestParameters{
        JSRender:     true,
        ResponseType: scraperapi.ResponseTypePDF,
    })
    if err != nil {
        log.Fatal(err)
    }

    if err := os.WriteFile("page.pdf", response.Body(), 0o644); err != nil {
        log.Fatal(err)
    }
    ```

    <Tip>
      Use `response.Body()` (not `response.String()`) when the target returns binary data, like a PDF from `ResponseType: scraperapi.ResponseTypePDF` or a screenshot. `Body()` gives you the raw `[]byte`, ready to write straight to a file.
    </Tip>
  </Tab>

  <Tab title="JSON">
    ```go theme={null} theme={"dark"}
    response, err := client.Get(context.Background(), url, &scraperapi.RequestParameters{
        JSRender:     true,
        JSONResponse: true,
    })
    if err != nil {
        log.Fatal(err)
    }

    var data map[string]any
    if err := json.Unmarshal(response.Body(), &data); err != nil {
        log.Fatal(err)
    }
    ```
  </Tab>
</Tabs>

## Reading Zenrows-specific headers

Zenrows adds monitoring headers to every response, useful for debugging and tracking usage:

```go theme={null} theme={"dark"}
response, err := client.Get(context.Background(), url, nil)

fmt.Println(response.Header().Get("X-Request-Id"))          // unique ID for this request
fmt.Println(response.Header().Get("X-Request-Cost"))        // dollar cost of this request (e.g., 0.001)
fmt.Println(response.Header().Get("X-Request-Credits"))     // number of credits consumed by this request
fmt.Println(response.Header().Get("X-Key-Credits-Remaining")) // credits left before this key hits a credit cap (capped keys only)
fmt.Println(response.Header().Get("Concurrency-Remaining")) // concurrency slots left on your plan
fmt.Println(response.Header().Get("Zr-Final-Url"))          // final URL after redirects
```

<Tip>
  If you ever contact Zenrows support about a failed request, include the `X-Request-Id` header value. It lets the team trace the exact request in the logs.
</Tip>

Headers coming from the target website itself are prefixed with `Zr-` (for example, `Zr-Content-Encoding`, `Zr-Cookies`). See the full list in the [Fetch setup guide](/fetch/setup#response-headers).

<Note>
  The SDK also exposes `response.TargetHeaders()` and `response.TargetCookies()` as shortcuts for reading target-site headers and cookies. If they return fewer headers than you expect for a given response, fall back to filtering `response.Header()` yourself for keys with the `Zr-` prefix shown above, that always works regardless of which prefix the shortcut methods look for internally.
</Note>

## Checking success before using the body

Because the SDK returns a response as-is rather than turning HTTP errors into a Go `error`, always confirm the request succeeded first:

```go theme={null} theme={"dark"}
response, err := client.Get(context.Background(), url, nil)
if err != nil {
    log.Fatal(err)
}

if response.IsSuccess() {
    data := response.String()
} else {
    fmt.Println("Request failed:", response.StatusCode())
    if prob := response.Problem(); prob != nil {
        fmt.Println("Detail:", prob.Detail)
    }
}
```

`response.Problem()` parses a structured error body into a `*problem.Problem` when the response is an error and the `Content-Type` is JSON. `response.Error()` does the same thing but returns it as a plain Go `error`, so `if err := response.Error(); err != nil { ... }` works too.

<Card title="Error Handling and Retries" icon="shield-halved" href="/fetch/sdk/go/error-handling-and-retries">
  More on this pattern, plus how to configure automatic retries.
</Card>

## Troubleshooting

* **`json.Unmarshal` fails on the response body**: The body isn't JSON. Confirm you requested `JSONResponse: true` (with `JSRender: true`) or that the target actually returns JSON.
* **Saved file (PDF, screenshot, image) is corrupted**: You wrote `response.String()` instead of `response.Body()`. `String()` can mangle binary data; always use `Body()` for non-text responses.
* **`response.TargetHeaders()` or `TargetCookies()` return nothing**: Fall back to reading `response.Header()` directly and filtering for the `Zr-` prefix, as noted above.
* **`response.Problem()` returns `nil` on a failed request**: This method only parses a body when `IsError()` is true and the response `Content-Type` is JSON. A non-JSON error body (or a successful response) always returns `nil`.

## Pricing

The response itself doesn't add cost. Zenrows adds two headers to help you track usage:

* `X-Request-Cost`: the dollar cost of the request (e.g., `0.001`)
* `X-Request-Credits`: the number of credits consumed by the request

Both values are based on the parameters used:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `JSRender: true` | 5 |
| `UsePremiumProxies: true` | 10 |
| `JSRender: true` + `UsePremiumProxies: true` | 25 |

See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Is the Response type specific to Zenrows, or is it go-resty's type directly?">
    It's a thin Zenrows-specific wrapper around a `go-resty` response, plus the original `*http.Response` exposed as the `RawResponse` field. You don't need `go-resty` imported in your own code to use it.
  </Accordion>

  <Accordion title="How do I get the response as a Go struct instead of a map?">
    Define a struct matching the expected JSON shape and pass a pointer to it in `json.Unmarshal(response.Body(), &yourStruct)`, the same as unmarshaling any other JSON in Go.
  </Accordion>

  <Accordion title="Where do I find how much a request cost?">
    Check the response headers: `X-Request-Cost` gives you the dollar cost (e.g., `0.001`), read with `response.Header().Get("X-Request-Cost")`. `X-Request-Credits` gives you the number of credits consumed, read with `response.Header().Get("X-Request-Credits")`.
  </Accordion>
</AccordionGroup>


# Concurrency with the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/async-concurrency

Scrape multiple URLs in parallel with the Zenrows Node.js SDK's built-in concurrency queue.

Every call to `client.get()` or `client.post()` already returns a `Promise`, so there's no separate "async" method to learn. The SDK queues every request from a client instance internally and only runs up to `concurrency` of them at the same time.

## Basic usage

<Steps>
  <Step title="Set a concurrency limit">
    Pass `concurrency` when creating the client to cap how many requests run at the same time:

    ```javascript theme={null} theme={"dark"}
    const { ZenRows } = require("zenrows");

    const client = new ZenRows("YOUR_ZENROWS_API_KEY", { concurrency: 5, retries: 1 });
    ```

    <Note>
      Set `concurrency` to match your Zenrows plan's concurrent request limit. Each client instance enforces its own limit. Two scripts (or two client instances) don't share it, so running both at once can still trigger `429 Too Many Requests` errors. See [Concurrency](/fetch/features/concurrency) for how plan limits work.
    </Note>
  </Step>

  <Step title="Scrape a list of URLs in parallel">
    Map your URLs to `client.get()` calls and use `Promise.allSettled()` to wait for all of them, even if some fail:

    ```javascript theme={null} theme={"dark"}
    const { ZenRows } = require("zenrows");

    const client = new ZenRows("YOUR_ZENROWS_API_KEY", { concurrency: 5, retries: 1 });

    const urls = [
      "https://www.scrapingcourse.com/ecommerce/",
      // ... more URLs
    ];

    (async () => {
      const promises = urls.map((url) => client.get(url));

      const results = await Promise.allSettled(promises);
      console.log(results);
    })();
    ```

    `Promise.allSettled()` waits for every call to finish, even if some of them fail. Unlike `Promise.all()`, it doesn't reject as soon as one promise fails, so the rest of the batch still completes.
  </Step>

  <Step title="Split fulfilled and rejected results">
    ```javascript theme={null} theme={"dark"}
    const rejected = results.filter(({ status }) => status === "rejected");
    const fulfilled = results.filter(({ status }) => status === "fulfilled");
    ```

    Each fulfilled result's `.value` is a standard `Response`, read the same way as `client.get()`'s return value.
  </Step>
</Steps>

## When to think about concurrency

You don't need to do anything special to run requests concurrently, calling `.get()`/`.post()` in a `.map()` already benefits from the internal queue. Set `concurrency` explicitly when you want to match a specific plan limit, or lower it if you're seeing `429` errors.

## POST in parallel

The same pattern works with `client.post()`:

```javascript theme={null} theme={"dark"}
const promises = urls.map((url) =>
  client.post(url, {}, { data: new URLSearchParams({ key1: "value1" }).toString() }),
);

const results = await Promise.allSettled(promises);
```

## How it works under the hood

<Info>
  The SDK uses <a href="https://github.com/mcollina/fastq">fastq</a> internally to queue every `get()` and `post()` call from the same client instance and run at most `concurrency` of them at once (default `5`). This applies uniformly, there's no separate queue for GET versus POST. Retry behavior configured via `retries` applies the same way regardless of how many requests run in parallel. See [Error Handling and Retries](/fetch/sdk/nodejs/error-handling-and-retries) for details.
</Info>

## Troubleshooting

* **`429 Too Many Requests` even though `concurrency` is set correctly**: Another script or process is using the same API key at the same time. Concurrency limits are per plan, not per client instance, so simultaneous scripts share the same real ceiling even though each `ZenRows` client only tracks its own local limit.
* **Some responses in the list are much slower than others**: This is expected; total throughput depends on the slowest requests in each concurrent batch. See the [Concurrency](/fetch/features/concurrency#impact-of-request-duration-on-throughput) guide for the underlying math.
* **A `Promise.all()` call fails as soon as one request errors**: Use `Promise.allSettled()` instead, as shown above, so a single failure doesn't abort the whole batch.
* **Cancelling a request doesn't free a concurrency slot right away**: Zenrows keeps the slot occupied for up to a few minutes after a client-side cancellation while it finishes processing server-side.

## Pricing

Requests are billed identically whether they run one at a time or concurrently. Cost depends on the config you send per request, not on how many run in parallel:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render: true` | 5 |
| `premium_proxy: true` | 10 |
| `js_render: true` + `premium_proxy: true` | 25 |

Running requests concurrently doesn't add cost by itself. It only changes how fast the same requests complete. See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="What's the default concurrency if I don't set one?">
    5. You can set it to any number that matches your Zenrows plan's concurrent request limit.
  </Accordion>

  <Accordion title="Do GET and POST share the same concurrency limit?">
    Yes. Every call from the same `ZenRows` instance, GET or POST, goes through one shared internal queue.
  </Accordion>

  <Accordion title="Is there a get_async or similar method?">
    No. Every method already returns a `Promise`, so there's no separate async variant to call, unlike SDKs for languages without built-in async I/O.
  </Accordion>
</AccordionGroup>


# Custom Headers with the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/custom-headers

Send custom headers to the target website through the Zenrows Node.js SDK.

By default, Zenrows sends an optimized set of headers to the target site on your behalf. If you need to forward specific headers instead, like a `Referer` or a session `Cookie`, pass a `headers` object as part of the third argument to `get` or `post`.

```javascript theme={null} theme={"dark"}
const { ZenRows } = require("zenrows");

const client = new ZenRows("YOUR_ZENROWS_API_KEY");
const url = "https://www.scrapingcourse.com/ecommerce/";

const response = await client.get(url, {}, {
  headers: {
    Referer: "https://www.google.com",
  },
});

console.log(await response.text());
```

## What happens when you pass headers

As soon as you pass a non-empty `headers` object, the SDK automatically sets `custom_headers=true` on the request for you, so you don't need to add that yourself. This enables your custom headers for the target request while Zenrows continues to manage browser-fingerprinting headers automatically.

## Headers you can and can't override

Not every header can be customized. Zenrows always manages browser-fingerprinting headers itself to keep success rates high and avoid anti-bot detection, even when `custom_headers` is enabled:

| Header | Customizable? |
| - | - |
| `Referer` | Yes |
| `Cookie` | Yes |
| `Authorization` | Yes |
| Most other headers | Yes, once `custom_headers=true` is set |
| `User-Agent` | No, always managed by Zenrows |
| `Sec-Ch-Ua`, `Sec-Ch-Ua-Mobile`, `Sec-Ch-Ua-Platform` | No |
| `Accept-Encoding`, `Accept-Language` | No |
| `Sec-Fetch-Mode`, `Sec-Fetch-Site`, `Sec-Fetch-User`, `Sec-Fetch-Dest` | No |
| `Connection`, `Upgrade-Insecure-Requests`, `Cache-Control` | No |

<Warning>
  Passing a `User-Agent` (or any other browser-fingerprinting header from the table above) in the `headers` object has no effect on the request sent to the target site. Zenrows ignores it and keeps using its own managed value to maintain a consistent, realistic browser fingerprint. See [Custom Headers](https://docs.zenrows.com/fetch/features/headers) in Fetch docs for the full explanation.
</Warning>

<Note>
  The SDK does send its own `User-Agent` (`zenrows/<version> node`) as part of the HTTP request it makes to Zenrows' API endpoint. This identifies the request as coming from the Node.js SDK for Zenrows' own logging, it is not the header Zenrows sends to the target website. If you pass a `User-Agent` key in `headers`, it replaces this value on the request to Zenrows' API, but that still has no bearing on what the target site receives.
</Note>

## Combining with request parameters

The config object and `headers` are independent arguments, so you can use both together:

```javascript theme={null} theme={"dark"}
const response = await client.get(
  url,
  { premium_proxy: true },
  { headers: { Referer: "https://www.google.com" } },
);
```

<Note>
  If a request keeps failing after you add custom headers, try removing them first to confirm whether they're the cause. Zenrows' defaults are usually the safer starting point.
</Note>

## Troubleshooting

* **Success rate drops after adding custom headers**: Remove your custom `headers` first to confirm whether they're the cause; Zenrows' defaults are usually the safer starting point.
* **Header doesn't seem to reach the target site**: Confirm you're using the `headers` argument (forwarded to the target) and not accidentally modifying the config object meant for Zenrows itself.
* **Cookies aren't persisting across requests**: Each SDK call is a separate, stateless request. Pass the `Cookie` header explicitly on every call if you need to maintain a session.
* **`custom_headers` shows up unexpectedly in requests**: This is set automatically whenever you pass a non-empty `headers` object; it's not something you need to remove.

## Pricing

Custom headers don't change the credit cost on their own. Credit cost still depends on `js_render` and `premium_proxy`:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render: true` | 5 |
| `premium_proxy: true` | 10 |
| `js_render: true` + `premium_proxy: true` | 25 |

See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Do I need to set custom_headers myself?">
    No. Passing a non-empty `headers` object to `get` or `post` sets `custom_headers=true` automatically.
  </Accordion>

  <Accordion title="Can I override the User-Agent sent to the target site?">
    No. `User-Agent` is one of several browser-fingerprinting headers Zenrows always manages for the request to the target, regardless of what you pass in `headers`. See the table above for the full list.
  </Accordion>

  <Accordion title="Can I set headers meant for Zenrows itself (like an API version header)?">
    No. `headers` is forwarded to the target website, not to Zenrows' API. Zenrows-specific options belong in the config object (the second argument).
  </Accordion>
</AccordionGroup>


# Error Handling and Retries with the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/error-handling-and-retries

Configure automatic retries in the Zenrows Node.js SDK and handle failed requests reliably.

Web scraping requests occasionally fail for transient reasons, like a temporary server hiccup or a dropped connection. The SDK can retry these automatically so you don't have to write that logic yourself.

## Basic usage

Retries are **off by default** (`retries: 0`). Pass a `retries` count to the client constructor to turn them on:

```javascript theme={null} theme={"dark"}
const { ZenRows } = require("zenrows");

const client = new ZenRows("YOUR_ZENROWS_API_KEY", { retries: 2 });

const response = await client.get("https://www.scrapingcourse.com/ecommerce/");
```

With `retries: 2`, a failed request is attempted up to 2 additional times (3 attempts total) before the SDK resolves with the last response it received.

## When to enable retries

Enable `retries` (2-3 is a reasonable starting point) whenever you're running requests at scale or in production, anywhere a transient failure shouldn't fail your whole job. Leave `retries: 0` (the default) for quick one-off scripts or debugging, where you'd rather see the raw failure immediately than wait through automatic retry attempts.

## Which failures get retried

<Warning>
  The Node.js SDK retries a narrower set of status codes than some other Zenrows SDKs. Only these trigger a retry:
</Warning>

| Status code | Meaning |
| - | - |
| `422` | Unprocessable Entity |
| `503` | Service Unavailable |
| `504` | Gateway Timeout |

Network-level errors (a rejected `fetch`, like a DNS failure or dropped connection) also trigger a retry. `429` and `500`/`502` responses are **not** retried automatically; they're returned as-is on the first attempt.

Between attempts, the SDK waits using exponential backoff: `2^attempt * 1000` milliseconds, so 1s, then 2s, then 4s, and so on for each subsequent retry.

<Tip>
  You aren't charged for retried attempts that fail. Only successful responses count toward your usage. See the [Retry Failed Requests](/zenrows-academy/retry-failed-requests) guide for more on how this works.
</Tip>

## Handling failures

The SDK treats HTTP error status codes and network-level problems differently. Pick the tab that matches what you're dealing with:

<Tabs>
  <Tab title="HTTP error status">
    Even after all retry attempts are exhausted, the SDK resolves with the final `Response` object. It does **not** throw for a non-2xx status, matching how native `fetch` behaves. Always check the response before trusting its content:

    ```javascript theme={null} theme={"dark"}
    const response = await client.get(url, { js_render: true });

    if (response.ok) {
      console.log(await response.text());
    } else {
      console.log(`Failed after retries: ${response.status} - ${await response.text()}`);
    }
    ```

    Unlike some HTTP libraries, native `fetch` (and this SDK) has no built-in `raise_for_status()` equivalent. Throw manually if you want an exception:

    ```javascript theme={null} theme={"dark"}
    const response = await client.get(url);

    if (!response.ok) {
      throw new Error(`Request failed with status ${response.status}`);
    }
    ```
  </Tab>

  <Tab title="Network-level exceptions">
    Connection problems (DNS failures, dropped connections) reject the returned `Promise` instead of resolving with a response. Wrap calls in `try`/`catch` to handle those:

    ```javascript theme={null} theme={"dark"}
    try {
      const response = await client.get(url);
      console.log(await response.text());
    } catch (error) {
      console.log(`Request failed: ${error.message}`);
    }
    ```
  </Tab>
</Tabs>

## Troubleshooting

* **Response still fails after retries are exhausted**: Check `response.status` and look it up in [API Error Codes](/api-error-codes). Some failures (like an invalid `apikey`) won't be fixed by retrying and need a code change instead.
* **A `429` isn't being retried**: This is expected. The Node.js SDK's retry list only covers `422`, `503`, and `504`, plus network errors. Handle `429` by lowering your `concurrency` or adding your own backoff, see [Concurrency](/fetch/sdk/nodejs/async-concurrency).
* **Retries seem to take a long time**: That's the exponential backoff by design (`2^attempt * 1000` ms). Lower `retries` if latency matters more than success rate for your use case.
* **Getting a rejected promise instead of an error response**: That only happens for network-level failures (DNS errors, dropped connections), not HTTP error status codes. Wrap the call in `try`/`catch` as shown above.

## Pricing

Retried attempts that fail are **not** billed. Only the final successful response counts. This makes `retries` effectively free insurance against transient failures:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render: true` | 5 |
| `premium_proxy: true` | 10 |
| `js_render: true` + `premium_proxy: true` | 25 |

<Note>`404` and `410` responses count as successful and are billed, since the request itself completed correctly.</Note>

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="What's the default number of retries?">
    0. Retries are disabled unless you explicitly pass `retries` to the constructor.
  </Accordion>

  <Accordion title="Why doesn't the SDK retry 429 or 500 responses?">
    The current Node.js SDK's retry list only includes `422`, `503`, `504`, and network errors. This differs from some other Zenrows SDKs, so don't assume a `429` will be retried automatically here.
  </Accordion>

  <Accordion title="Am I charged for retried requests that eventually fail?">
    No. Only successful requests consume your Zenrows balance.
  </Accordion>
</AccordionGroup>

## Look up a specific error code

For a full breakdown of Zenrows error codes (like `AUTH002` or `RESP001`) and how to resolve each one, see the [API Error Codes](/api-error-codes) reference.


# Frequently Asked Questions
Source: https://docs.zenrows.com/fetch/sdk/nodejs/faq

Answers to common questions about the Zenrows Node.js SDK.

### Setup and compatibility

<AccordionGroup>
  <Accordion title="What Node.js versions does the SDK support?">
    Node.js 20 and later, per the package's `engines` field. The SDK has two dependencies, `fastq` and `fetch-retry`, installed automatically with `npm install zenrows`.
  </Accordion>

  <Accordion title="Does the SDK work with both CommonJS and ES modules?">
    Yes. `package.json` exposes both a CommonJS build (`require("zenrows")`) and an ES module build (`import { ZenRows } from "zenrows"`), plus TypeScript types.
  </Accordion>

  <Accordion title="Does the SDK support Proxy Mode?">
    No. The SDK talks to Fetch's endpoint directly (API Mode). If you want to use Zenrows as a proxy in your own HTTP client instead, see the [Proxy Mode documentation](/fetch/setup#connection-modes) rather than the SDK.
  </Accordion>
</AccordionGroup>

### Requests, errors, and concurrency

<AccordionGroup>
  <Accordion title="Does the SDK support PUT or DELETE requests?">
    No. Only `get()` and `post()` are exposed. See [Sending a PUT request](/fetch/sdk/nodejs/post-put-requests#sending-a-put-request) for a workaround using native `fetch`.
  </Accordion>

  <Accordion title="Why isn't my request throwing an exception on failure?">
    By design, the SDK resolves with the `Response` object even for non-2xx status codes, matching native `fetch` behavior. Check `response.ok` or `response.status` yourself. See [Error Handling and Retries](/fetch/sdk/nodejs/error-handling-and-retries).
  </Accordion>

  <Accordion title="Why doesn't a 429 get retried automatically?">
    The Node.js SDK's built-in retry logic only covers `422`, `503`, `504`, and network errors, not `429` or `500`/`502`. Lower your `concurrency` or add your own backoff for rate-limit errors. See [Concurrency](/fetch/sdk/nodejs/async-concurrency).
  </Accordion>

  <Accordion title="Can two scripts share the same concurrency limit?">
    No. Each `ZenRows` instance enforces its own `concurrency` setting independently. Running two scripts (or two client instances) at the same time can exceed your plan's actual limit and trigger `429 Too Many Requests` errors. Coordinate concurrency across processes yourself if you run multiple scripts against the same Zenrows plan.
  </Accordion>
</AccordionGroup>

### Project and support

<AccordionGroup>
  <Accordion title="Is there a Python version of the SDK?">
    Yes. See the <a href="https://pypi.org/project/zenrows/">Zenrows Python SDK</a> on PyPI, which additionally supports `put()` and explicit async methods.
  </Accordion>

  <Accordion title="Where can I see the SDK's source code or report a bug?">
    The SDK is open source on <a href="https://github.com/ZenRows/zenrows-node-sdk">GitHub</a>. Open an issue there for bugs, or a pull request if you'd like to contribute.
  </Accordion>

  <Accordion title="What license is the SDK released under?">
    MIT. You can use, modify, and distribute it freely; see the <a href="https://github.com/ZenRows/zenrows-node-sdk/blob/main/LICENSE">LICENSE</a> file in the repository.
  </Accordion>
</AccordionGroup>

<Card title="Still stuck?" href="mailto:success@zenrows.com">
  Check the [API Error Codes](/api-error-codes) reference, or contact support directly.
</Card>


# Installing the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/installation

Install the Zenrows Node.js SDK and get your API key ready to make requests.

## Requirements

<CardGroup>
  <Card title="Node.js 20+" icon="node-js">
    Required by the SDK's `engines` field in `package.json`.
  </Card>

  <Card title="API key" icon="key" href="https://app.zenrows.com/register?p=free">
    Create a free account to get yours, no credit card required.
  </Card>
</CardGroup>

<Steps>
  <Step title="Install the package">
    Install the SDK with your package manager of choice:

    <Tabs>
      <Tab title="npm">
        ```bash theme={null} theme={"dark"}
        npm install zenrows
        ```
      </Tab>

      <Tab title="yarn">
        ```bash theme={null} theme={"dark"}
        yarn add zenrows
        ```
      </Tab>

      <Tab title="pnpm">
        ```bash theme={null} theme={"dark"}
        pnpm install zenrows
        ```
      </Tab>

      <Tab title="bun">
        ```bash theme={null} theme={"dark"}
        bun install zenrows
        ```
      </Tab>
    </Tabs>

    The SDK has two dependencies, `fastq` (concurrency queue) and `fetch-retry` (retry logic on top of `fetch`), installed automatically.
  </Step>

  <Step title="Verify the install">
    Run this in a Node.js script or REPL to confirm the package is available:

    ```javascript theme={null} theme={"dark"}
    const { ZenRows } = require("zenrows");

    console.log(ZenRows);
    ```

    Or with ES modules / TypeScript:

    ```typescript theme={null} theme={"dark"}
    import { ZenRows } from "zenrows";

    console.log(ZenRows);
    ```

    If it prints the `ZenRows` class without raising an error, you're ready to go.
  </Step>

  <Step title="Get your API key">
    Your API key authenticates every request you send through the SDK. Grab it from the [Zenrows Request Playground](https://app.zenrows.com/fetch) after signing up.

    <Warning>
      Keep your API key private. Load it from an environment variable (like `process.env.ZENROWS_API_KEY`) instead of hardcoding it in scripts you commit to version control.
    </Warning>
  </Step>
</Steps>

<Card title="Next: Quickstart" icon="rocket" href="/fetch/sdk/nodejs/quickstart">
  Make your first request with the SDK.
</Card>


# Introduction to the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/introduction

Official Zenrows Node.js SDK for Fetch. Handles authentication, retries, and concurrency so you can scrape with less boilerplate.

The <a href="https://www.npmjs.com/package/zenrows">Zenrows Node.js SDK</a> (npm package `zenrows`) is a lightweight client for [Fetch](/fetch/setup). It wraps the native <a href="https://nodejs.org/api/globals.html#fetch">`fetch`</a> API, adding automatic retries and a built-in concurrency queue on top.

<Info>
  Every call returns a standard `Response` object, the same one the native `fetch` API returns. Since every method is already a `Promise`, there's no separate set of "async" methods to learn, unlike some other language SDKs.
</Info>

## What the SDK handles for you

* **Authentication** - your API key is set once in the client constructor, not repeated on every call.
* **Retries** - optionally retry failed requests (network errors, `422`, `503`, `504`) with exponential backoff, without writing retry logic yourself.
* **Concurrency** - every request from a client instance is automatically queued and limited to match your Zenrows plan's concurrency limit.
* **Native fetch** - no extra HTTP client dependency; the SDK builds directly on Node's built-in `fetch`.

## What you'll need

<CardGroup>
  <Card title="Node.js 20+" icon="node-js">
    Required by the SDK's `engines` field in `package.json`.
  </Card>

  <Card title="API key" icon="key" href="https://app.zenrows.com/register?p=free">
    Create a free account to get yours, no credit card required.
  </Card>
</CardGroup>

<Note>
  The SDK only covers Fetch's GET and POST modes; it doesn't expose a `put()` method. For the full list of request parameters (`premium_proxy`, `css_extractor`, `extract`, and so on), see the [Fetch setup guide](/fetch/setup). Most parameters work the same way through the SDK's `config` argument, with one exception: `js_render` only works on GET requests, not POST. See [POST Requests](/fetch/sdk/nodejs/post-put-requests).
</Note>

<Tip>
  Looking for a different language? Zenrows also maintains a <a href="https://pypi.org/project/zenrows/">Python SDK</a>.
</Tip>

## Explore the Node.js SDK

<CardGroup>
  <Card title="Installation" icon="download" href="/fetch/sdk/nodejs/installation">
    Install the package and get your API key.
  </Card>

  <Card title="Quickstart" icon="rocket" href="/fetch/sdk/nodejs/quickstart">
    Make your first request in a few lines of code.
  </Card>

  <Card title="POST Requests" icon="send" href="/fetch/sdk/nodejs/post-put-requests">
    Submit forms or data payloads.
  </Card>

  <Card title="Concurrency" icon="bolt" href="/fetch/sdk/nodejs/async-concurrency">
    Scrape many URLs in parallel.
  </Card>

  <Card title="Error Handling and Retries" icon="shield" href="/fetch/sdk/nodejs/error-handling-and-retries">
    Configure automatic retries and handle failures.
  </Card>

  <Card title="The Response Object" icon="file-text" href="/fetch/sdk/nodejs/response-object">
    What you get back from every request.
  </Card>

  <Card title="Custom Headers" icon="tags" href="/fetch/sdk/nodejs/custom-headers">
    Forward specific headers to the target site.
  </Card>

  <Card title="FAQ" icon="help-circle" href="/fetch/sdk/nodejs/faq">
    Answers to common questions.
  </Card>
</CardGroup>


# POST Requests with the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/post-put-requests

Send POST requests through the Zenrows Node.js SDK, including form data payloads.

<Note>
  The Node.js SDK currently only exposes `get()` and `post()`. There is no `put()` method. If you need to send a `PUT` request, see [Sending a PUT request](#sending-a-put-request) below.
</Note>

Besides `client.get()`, the SDK exposes `client.post()` for targets that expect a form submission or a data payload. It accepts the same config object as `get`, plus a third argument for `headers` and `data`.

## Basic usage

```javascript theme={null} theme={"dark"}
const { ZenRows } = require("zenrows");

const client = new ZenRows("YOUR_ZENROWS_API_KEY");
const url = "https://httpbin.org/anything";

(async () => {
  const response = await client.post(
    url,
    {},
    {
      data: new URLSearchParams({
        key1: "value1",
        key2: "value2",
      }).toString(),
    },
  );

  const data = await response.json();

  console.log(data);
})();
```

<Warning>
  `data` must be a string for form-encoded submissions. Encode a plain object with `new URLSearchParams(obj).toString()` first, as shown above. The SDK defaults the `Content-Type` header to `application/x-www-form-urlencoded`, so the body needs to match that format.
</Warning>

## When to use POST

* **POST**: submitting a form on the target site, triggering a search that requires a form submission, or calling a write endpoint on a target API.
* **GET** (most cases): reading page content only needs `client.get()`. Reach for POST specifically when the target requires it.

## Sending JSON instead of form data

Pass a plain object as `data` and override the `Content-Type` header to `application/json`. The SDK `JSON.stringify`s any object passed as `data`:

```javascript theme={null} theme={"dark"}
const response = await client.post(
  url,
  {},
  {
    headers: { "Content-Type": "application/json" },
    data: { key1: "value1" },
  },
);
```

## Combining with request parameters

The config object (first argument after the URL) still works the same way it does for GET requests, so you can enable `premium_proxy` or other Fetch parameters:

```javascript theme={null} theme={"dark"}
const response = await client.post(
  url,
  { premium_proxy: true },
  { data: new URLSearchParams({ key1: "value1" }).toString() },
);
```

<Warning>
  `js_render` only works with GET requests (`client.get()`). It is not supported on POST requests.
</Warning>

## Sending a PUT request

The SDK doesn't expose a `put()` method, so call the Zenrows API directly with native `fetch` for `PUT`:

```javascript theme={null} theme={"dark"}
const params = new URLSearchParams({
  url: "https://httpbin.org/anything",
  apikey: "YOUR_ZENROWS_API_KEY",
});

const response = await fetch(`https://api.zenrows.com/v1/?${params.toString()}`, {
  method: "PUT",
  headers: { "Content-Type": "application/x-www-form-urlencoded" },
  body: new URLSearchParams({ key1: "value1" }).toString(),
});
```

This loses the SDK's automatic retries and concurrency queue for that call. If you need those for PUT-heavy workloads, consider using the [Python SDK](/fetch/sdk/python/post-put-requests) instead, which supports `put()` natively.

## Troubleshooting

* **Target site doesn't reflect the submitted data**: Some forms require specific headers (like a `Referer`) to accept the submission. See [Custom Headers](/fetch/sdk/nodejs/custom-headers).
* **Getting `REQS005 Method Not Allowed`**: The target endpoint doesn't accept the HTTP verb you're using. Zenrows only supports `GET`, `POST`, and `PUT` at the API level; confirm the target expects one of these.
* **Body doesn't match Content-Type**: If `data` is a plain object, it's sent as JSON regardless of what `Content-Type` you set. Either encode it as a string with `URLSearchParams` for form submissions, or explicitly set `Content-Type: application/json` when passing an object.

## Pricing

POST requests are billed the same way as GET requests. Cost depends on the config you use, not the HTTP method:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `premium_proxy: true` | 10 |

See [Zenrows Pricing](/first-steps/pricing) for plan-specific rates.

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Does the SDK support PUT or DELETE?">
    No. Only `GET` and `POST` are exposed as methods on the `ZenRows` class. See [Sending a PUT request](#sending-a-put-request) for a workaround using native `fetch`.
  </Accordion>

  <Accordion title="Can I send both a config object and data in the same request?">
    Yes. The config object (Zenrows/API options like `premium_proxy`) and `data` (the payload sent to the target) are independent arguments and can be combined.
  </Accordion>

  <Accordion title="How do I send multipart/form-data (file uploads)?">
    The SDK doesn't support this directly. Any object passed as `data` is run through `JSON.stringify()`, which doesn't serialize a `FormData` instance correctly. For file uploads, call the Zenrows API directly with native `fetch` and a `FormData` body instead, the same way as the [PUT workaround](#sending-a-put-request) above.
  </Accordion>
</AccordionGroup>


# Making your first request with the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/quickstart

Make your first request with the Zenrows Node.js SDK in a few lines of code.

This guide makes a single GET request with the Zenrows Node.js SDK: create a client, send the request, read the response.

<Note>Haven't installed the SDK yet? Follow [Installation](/fetch/sdk/nodejs/installation) first.</Note>

## Basic usage

<Steps>
  <Step title="Create a client">
    Import `ZenRows` and instantiate it with your API key:

    ```javascript theme={null} theme={"dark"}
    const { ZenRows } = require("zenrows");

    const client = new ZenRows("YOUR_ZENROWS_API_KEY");
    ```
  </Step>

  <Step title="Send a GET request">
    Call `client.get()` with the URL you want to scrape:

    ```javascript theme={null} theme={"dark"}
    const { ZenRows } = require("zenrows");

    const client = new ZenRows("YOUR_ZENROWS_API_KEY");
    const url = "https://www.scrapingcourse.com/ecommerce/";

    (async () => {
      const response = await client.get(url);
      const html = await response.text();

      console.log(html);
    })();
    ```

    `response` is a standard `fetch` `Response` object.

    <Tip>See [The Response Object](/fetch/sdk/nodejs/response-object) for everything available on it.</Tip>
  </Step>

  <Step title="Add request parameters">
    Pass a config object as the second argument to enable Fetch features like JavaScript rendering or Premium Proxies:

    ```javascript theme={null} theme={"dark"}
    const { ZenRows } = require("zenrows");

    const client = new ZenRows("YOUR_ZENROWS_API_KEY");
    const url = "https://www.scrapingcourse.com/ecommerce/";

    (async () => {
      const response = await client.get(url, {
        js_render: true,
        premium_proxy: true,
        proxy_country: "us",
      });

      const html = await response.text();

      console.log(html);
    })();
    ```

    <Tip>
      The config object accepts every parameter documented in the [Fetch setup guide](/fetch/setup), including `extract`, `css_extractor`, `wait_for`, `block_resources`, and `outputs`. They all work exactly the same way through the SDK.
    </Tip>
  </Step>

  <Step title="Check the result">
    Since the SDK doesn't throw an exception on non-2xx responses, always check the status before using the content:

    ```javascript theme={null} theme={"dark"}
    if (response.ok) {
      const html = await response.text();
      console.log(html);
    } else {
      console.log(`Request failed with status ${response.status}: ${await response.text()}`);
    }
    ```

    See [Error Handling and Retries](/fetch/sdk/nodejs/error-handling-and-retries) for how to configure automatic retries so you don't have to handle every transient failure manually.
  </Step>
</Steps>

## Troubleshooting

* **`response.status` is `401` or similar auth error**: Double-check your API key was copied correctly from the [Zenrows Request Playground](https://app.zenrows.com/fetch) with no extra whitespace. See [`AUTH002`](/api-error-codes#auth002-invalid-api-key).
* **The response body is empty or looks like an error page, not the target site**: The target likely needs `js_render: true` and/or `premium_proxy: true`. See [`REQS002`](/api-error-codes#reqs002-request-requirements-unsatisfied).
* **`await response.text()` hangs or the script never finishes**: Make sure the call is wrapped in an `async` function and awaited; the SDK doesn't have a `timeout` option, so a hung request will wait indefinitely unless your own code aborts it (e.g. with `AbortController`).
* **`429 Too Many Requests`**: You've hit your plan's concurrency or rate limit. See [Concurrency](/fetch/sdk/nodejs/async-concurrency).

## Pricing

Cost depends on the config you send, not on the SDK itself:

| Configuration | Credits per request |
| - | - |
| Basic request (no `js_render`/`premium_proxy`) | 1 |
| `js_render: true` | 5 |
| `premium_proxy: true` | 10 |
| `js_render: true` + `premium_proxy: true` | 25 |

Only successful requests are billed. See [Zenrows Pricing](/first-steps/pricing) for exact per-plan rates.

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Do I need a config object for every request?">
    No, the config argument is optional. `client.get(url)` alone works for public pages with no anti-bot protection.
  </Accordion>

  <Accordion title="Why doesn't a failed request throw an exception?">
    By design, the SDK resolves with the `Response` object even for error status codes, matching how native `fetch` behaves. Check `response.ok` or `response.status` yourself. See [Error Handling and Retries](/fetch/sdk/nodejs/error-handling-and-retries).
  </Accordion>

  <Accordion title="Can I reuse the same client for multiple requests?">
    Yes. Create one `ZenRows` instance and call `.get()`/`.post()` on it as many times as needed, there's no need to recreate it per request.
  </Accordion>
</AccordionGroup>

## What's next

<CardGroup>
  <Card title="POST Requests" icon="send" href="/fetch/sdk/nodejs/post-put-requests">
    Submit forms or data payloads.
  </Card>

  <Card title="Concurrency" icon="bolt" href="/fetch/sdk/nodejs/async-concurrency">
    Scrape many URLs in parallel.
  </Card>

  <Card title="Custom Headers" icon="tags" href="/fetch/sdk/nodejs/custom-headers">
    Forward specific headers to the target site.
  </Card>
</CardGroup>


# The Response Object with the Node.js SDK
Source: https://docs.zenrows.com/fetch/sdk/nodejs/response-object

Understand the Response object returned by every Zenrows Node.js SDK call.

Every method on `ZenRows` (`get` and `post`) resolves with a standard <a href="https://nodejs.org/api/globals.html#response">`Response`</a> object, the same one native `fetch` returns. The SDK doesn't wrap or transform it, so everything you already know about the Fetch API applies directly.

## Commonly used properties and methods

| Property / method | Returns | Use it when |
| - | - | - |
| `await response.text()` | Response body as a string | Reading HTML or plain text |
| `await response.json()` | Parsed JSON | The body is JSON (e.g. `json_response: true`) |
| `await response.arrayBuffer()` | Response body as raw bytes | Handling binary data (PDFs, screenshots, images) |
| `response.status` | HTTP status code, e.g. `200` | Checking exactly what happened |
| `response.ok` | `true` if `status` is 200-299 | Quick success check |
| `response.headers` | `Headers` object | Reading Zenrows monitoring headers |
| `response.url` | The final URL that was requested | Debugging what was actually sent |

<Warning>
  `response.text()` and `response.json()` are asynchronous, they return a `Promise` and must be awaited. This is different from libraries where the body is already available as a plain property.
</Warning>

<Tabs>
  <Tab title="Text / HTML">
    ```javascript theme={null} theme={"dark"}
    const response = await client.get(url);
    const html = await response.text();
    ```
  </Tab>

  <Tab title="Binary (PDF, screenshot)">
    ```javascript theme={null} theme={"dark"}
    const fs = require("fs/promises");

    const response = await client.get(url, { response_type: "pdf", js_render: true });
    const buffer = Buffer.from(await response.arrayBuffer());

    await fs.writeFile("page.pdf", buffer);
    ```

    <Tip>
      Use `response.arrayBuffer()` (not `response.text()`) when the target returns binary data. For example, a PDF from `response_type: "pdf"` or a screenshot.
    </Tip>
  </Tab>

  <Tab title="JSON">
    ```javascript theme={null} theme={"dark"}
    const response = await client.get(url, { json_response: true, js_render: true });
    const data = await response.json();
    ```
  </Tab>
</Tabs>

## Reading Zenrows-specific headers

Zenrows adds monitoring headers to every response, useful for debugging and tracking usage:

```javascript theme={null} theme={"dark"}
const response = await client.get(url);

console.log(response.headers.get("X-Request-Id"));          // unique ID for this request
console.log(response.headers.get("X-Request-Cost"));        // dollar cost of this request (e.g., 0.001)
console.log(response.headers.get("X-Request-Credits"));     // number of credits consumed by this request
console.log(response.headers.get("X-Key-Credits-Remaining")); // credits left before this key hits a credit cap (capped keys only)
console.log(response.headers.get("Concurrency-Remaining"));  // concurrency slots left on your plan
console.log(response.headers.get("Zr-Final-Url"));           // final URL after redirects
```

<Tip>
  If you ever contact Zenrows support about a failed request, include the `X-Request-Id` header value. It lets the team trace the exact request in the logs.
</Tip>

Headers coming from the target website itself are prefixed with `Zr-` to distinguish them from Zenrows' own headers. See the full list in the [Fetch setup guide](/fetch/setup#response-headers).

## Checking success before using the body

Because the SDK resolves with the response as-is rather than throwing on error status codes, always confirm the request succeeded first:

```javascript theme={null} theme={"dark"}
const response = await client.get(url);

if (response.ok) {
  const data = await response.text();
} else {
  console.log(`Request failed: ${response.status}`);
}
```

<Card title="Error Handling and Retries" icon="shield" href="/fetch/sdk/nodejs/error-handling-and-retries">
  More on this pattern, plus how to configure automatic retries.
</Card>

## Troubleshooting

* **`await response.json()` throws a `SyntaxError`**: The body isn't JSON. Confirm you requested `json_response: true` or that the target actually returns JSON before calling `.json()`.
* **Saved file (PDF, screenshot, image) is corrupted**: You wrote `response.text()` instead of `response.arrayBuffer()`, or forgot to wrap it in `Buffer.from(...)` before writing to disk.
* **Calling `response.text()` twice throws an error**: A `Response` body can only be read once. Store the result of the first call in a variable and reuse it instead of calling `.text()`/`.json()` again.
* **`response.url` doesn't match the URL you passed**: This is expected. It's the full URL sent to Zenrows' API (including your `apikey` and config as query parameters), not the target URL by itself.

## Pricing

The response itself doesn't add cost. Zenrows adds two headers to help you track usage:

* `X-Request-Cost`: the dollar cost of the request (e.g., `0.001`)
* `X-Request-Credits`: the number of credits consumed by the request

Both values are based on the config used:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render: true` | 5 |
| `premium_proxy: true` | 10 |
| `js_render: true` + `premium_proxy: true` | 25 |

See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Is the response object specific to Zenrows?">
    No. It's the standard `Response` object from the Fetch API. Any Fetch API documentation or Stack Overflow answer about it applies here too.
  </Accordion>

  <Accordion title="How do I get the response as a JavaScript object?">
    Call `await response.json()` if the body is JSON. For example, when using `json_response: true` or `extract: 'auto'` on structured data.
  </Accordion>

  <Accordion title="Where do I find how much a request cost?">
    Check the response headers: `X-Request-Cost` gives you the dollar cost (e.g., `0.001`) and `X-Request-Credits` gives you the number of credits consumed. Read them with `response.headers.get("X-Request-Cost")` and `response.headers.get("X-Request-Credits")`.
  </Accordion>
</AccordionGroup>


# Zenrows SDK Overview
Source: https://docs.zenrows.com/fetch/sdk/overview

Official Zenrows SDKs for Python, Node.js, and Go. Pick your language and start scraping in minutes.

The Zenrows SDKs are lightweight, official clients for [Fetch](/fetch/setup). Each one wraps the same `https://api.zenrows.com/v1` endpoint, adding authentication, retries, and concurrency control so you write less boilerplate and spend more time on what you're actually building.

<Info>
  All three SDKs are open source and MIT licensed. You can inspect the source, report bugs, or contribute on [GitHub](https://github.com/ZenRows).
</Info>

## Pick your language

<CardGroup>
  <Card title="Python" icon="brand-python" href="/fetch/sdk/python/introduction">
    Built on `requests`. Returns `requests.Response`. Supports `async`/`await` via `asyncio`. GET, POST, and PUT.
  </Card>

  <Card title="Node.js" icon="brand-nodejs" href="/fetch/sdk/nodejs/introduction">
    Built on native `fetch`. Returns a standard `Response`. Every method is already a `Promise`. GET and POST.
  </Card>

  <Card title="Go" icon="golang" href="/fetch/sdk/go/introduction">
    Built on `net/http` via `go-resty`. Typed `RequestParameters` struct. Takes `context.Context`. GET, POST, and PUT.
  </Card>
</CardGroup>

## What every SDK gives you

All three SDKs share the same core behavior. You set your API key once in the client constructor, and every call handles the rest.

<AccordionGroup>
  <Accordion title="Authentication" icon="key">
    Your API key is passed once to the client constructor. Every subsequent request is authenticated automatically; you never repeat it per call.

    <Tabs>
      <Tab title="Python">
        ```python theme={null} theme={"dark"}
        from zenrows import ZenRowsClient

        client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
        ```
      </Tab>

      <Tab title="Node.js">
        ```javascript theme={null} theme={"dark"}
        const { ZenRows } = require("zenrows");

        const client = new ZenRows("YOUR_ZENROWS_API_KEY");
        ```
      </Tab>

      <Tab title="Go">
        ```go theme={null} theme={"dark"}
        import scraperapi "github.com/zenrows/zenrows-go-sdk/service/api"

        client := scraperapi.NewClient(
            scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
        )
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Automatic retries" icon="refresh">
    Retries are off by default (`0`). Set a retry count in the constructor and the SDK retries failed requests with exponential backoff automatically. Only successful responses are billed; failed retries are free.

    <Tabs>
      <Tab title="Python">
        ```python theme={null} theme={"dark"}
        client = ZenRowsClient("YOUR_ZENROWS_API_KEY", retries=2)
        ```

        Retries on: `422`, `429`, `500`, `502`, `503`, `504`
      </Tab>

      <Tab title="Node.js">
        ```javascript theme={null} theme={"dark"}
        const client = new ZenRows("YOUR_ZENROWS_API_KEY", { retries: 2 });
        ```

        Retries on: `422`, `503`, `504`, and network errors
      </Tab>

      <Tab title="Go">
        ```go theme={null} theme={"dark"}
        client := scraperapi.NewClient(
            scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
            scraperapi.WithMaxRetryCount(2),
        )
        ```

        Retries on: `422`, `429`, `500`, and network errors
      </Tab>
    </Tabs>

    <Warning>
      The retry status code list differs between SDKs. If you need consistent retry behavior across languages, check each SDK's [Error Handling and Retries](/fetch/sdk/python/error-handling-and-retries) page for the exact list.
    </Warning>
  </Accordion>

  <Accordion title="Concurrency control" icon="bolt">
    All three SDKs cap parallel requests via an internal queue or semaphore. The default concurrency limit is `5`. Set it to match your Zenrows plan.

    <Tabs>
      <Tab title="Python">
        ```python theme={null} theme={"dark"}
        client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5)
        ```
      </Tab>

      <Tab title="Node.js">
        ```javascript theme={null} theme={"dark"}
        const client = new ZenRows("YOUR_ZENROWS_API_KEY", { concurrency: 5 });
        ```
      </Tab>

      <Tab title="Go">
        ```go theme={null} theme={"dark"}
        client := scraperapi.NewClient(
            scraperapi.WithAPIKey("YOUR_ZENROWS_API_KEY"),
            scraperapi.WithMaxConcurrentRequests(5),
        )
        ```
      </Tab>
    </Tabs>

    <Note>
      Each client instance enforces its own concurrency limit. Two scripts using the same API key don't share limits, so running both at once can trigger `429 Too Many Requests` errors. See the [Concurrency](/fetch/features/concurrency) guide for plan-level limits.
    </Note>
  </Accordion>

  <Accordion title="Request parameters" icon="table">
    Every Fetch parameter (`js_render`, `premium_proxy`, `css_extractor`, and so on) is available through each SDK. Only the syntax differs.

    <Tabs>
      <Tab title="Python">
        ```python theme={null} theme={"dark"}
        response = client.get(url, params={
            "js_render": True,
            "premium_proxy": True,
            "proxy_country": "us",
        })
        ```
      </Tab>

      <Tab title="Node.js">
        ```javascript theme={null} theme={"dark"}
        const response = await client.get(url, {
          js_render: true,
          premium_proxy: true,
          proxy_country: "us",
        });
        ```
      </Tab>

      <Tab title="Go">
        ```go theme={null} theme={"dark"}
        response, err := client.Get(ctx, url, &scraperapi.RequestParameters{
            JSRender:          true,
            UsePremiumProxies: true,
            ProxyCountry:      "us",
        })
        ```
      </Tab>
    </Tabs>

    For the full parameter list, see the [Fetch setup guide](/fetch/setup).
  </Accordion>

  <Accordion title="Response headers" icon="code">
    Every response includes Zenrows monitoring headers. The two most useful for tracking usage are:

    | Header | Contains |
    | - | - |
    | `X-Request-Cost` | Dollar cost of the request (e.g. `0.001`) |
    | `X-Request-Credits` | Number of credits consumed |
    | `X-Key-Credits-Remaining` | Lowest remaining credits across the API key's [credit caps](/first-steps/api-key-credit-caps). Sent only for keys with at least one cap |
    | `X-Request-Id` | Unique request ID. Include this when contacting support |
    | `Concurrency-Remaining` | Concurrency slots remaining on your plan |
    | `Zr-Final-Url` | Final URL after any redirects |

    Headers from the target website are prefixed with `Zr-` to avoid collisions with Zenrows' own headers.
  </Accordion>
</AccordionGroup>

## SDK comparison

<Note>
  All SDKs target the same Fetch endpoint. The differences below are language-level constraints, not API-level ones.
</Note>

| Feature | Python | Node.js | Go |
| - | - | - | - |
| Install | `pip install zenrows` | `npm install zenrows` | `go get github.com/zenrows/zenrows-go-sdk/service/api` |
| Min version | Python 3.6+ | Node.js 20+ | Go 1.23+ |
| Response type | `requests.Response` | Fetch `Response` | `*scraperapi.Response` |
| GET | Yes | Yes | Yes |
| POST | Yes | Yes | Yes |
| PUT | Yes | No (use raw `fetch`) | Yes |
| Async / parallel | `get_async` + `asyncio` | Every method is a `Promise` | Goroutines + semaphore |
| Default concurrency | 5 | 5 | 5 |
| Retried status codes | `422`, `429`, `500`, `502`, `503`, `504` | `422`, `503`, `504` | `422`, `429`, `500` |
| Typed parameters | No (dict / object) | No (dict / object) | Yes (`RequestParameters` struct) |
| GitHub | <a href="https://github.com/ZenRows/zenrows-python-sdk">zenrows-python-sdk</a> | <a href="https://github.com/ZenRows/zenrows-node-sdk">zenrows-node-sdk</a> | <a href="https://github.com/ZenRows/zenrows-go-sdk">zenrows-go-sdk</a> |

## Pricing

Cost is determined by the parameters you send per request, not by which SDK you use. Retried attempts that fail are not billed.

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render=True` / `js_render: true` / `JSRender: true` | 5 |
| `premium_proxy=True` / `premium_proxy: true` / `UsePremiumProxies: true` | 10 |
| JS render + Premium Proxy | 25 |

See [Zenrows Pricing](/first-steps/pricing) for plan-specific rates and concurrency limits.

## Get started

<Steps>
  <Step title="Pick your SDK">
    Choose the SDK that matches your language. All three share the same API key and request parameters.
  </Step>

  <Step title="Install it">
    Follow the installation guide for your language: [Python](/fetch/sdk/python/installation), [Node.js](/fetch/sdk/nodejs/installation), or [Go](/fetch/sdk/go/installation).
  </Step>

  <Step title="Make your first request">
    Each SDK has a quickstart that gets you to a working response in under 10 lines of code: [Python](/fetch/sdk/python/quickstart), [Node.js](/fetch/sdk/nodejs/quickstart), [Go](/fetch/sdk/go/quickstart).
  </Step>

  <Step title="Scale up">
    Once the basics work, add retries and concurrency to match your production requirements. Each SDK's [Error Handling](/fetch/sdk/python/error-handling-and-retries) and [Concurrency](/fetch/sdk/python/async-concurrency) pages walk through the options.
  </Step>
</Steps>

<Card title="Need help?" icon="help-circle" href="mailto:success@zenrows.com">
  Check the [API Error Codes](/api-error-codes) reference for error details, or contact support with your `X-Request-Id` header value for faster triage.
</Card>


# Async Requests and Concurrency with the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/async-concurrency

Scrape multiple URLs in parallel with the Zenrows Python SDK's built-in concurrency and asyncio support.

When you need to scrape many URLs at once, calling `client.get()` in a loop wastes time waiting for each request to finish before starting the next one. The SDK's async methods let you fire off requests in parallel instead.

## Basic usage

<Steps>
  <Step title="Set a concurrency limit">
    Pass `concurrency` when creating the client to cap how many requests run at the same time:

    ```python theme={null} theme={"dark"}
    from zenrows import ZenRowsClient

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1)
    ```

    <Note>
      Set `concurrency` to match your Zenrows plan's concurrent request limit. Each client instance enforces its own limit. Two scripts (or two client instances) don't share it, so running both at once can still trigger `429 Too Many Requests` errors. See [Concurrency](/fetch/features/concurrency) for how plan limits work.
    </Note>
  </Step>

  <Step title="Scrape a list of URLs concurrently">
    Use `get_async` with `asyncio.gather` to run requests in parallel and wait for all of them to finish:

    ```python theme={null} theme={"dark"}
    from zenrows import ZenRowsClient
    import asyncio

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1)

    async def main():
        urls = [
            "https://www.scrapingcourse.com/ecommerce/",
            # ... more URLs
        ]
        responses = await asyncio.gather(*[client.get_async(url) for url in urls])

        for response in responses:
            print(response.status_code, response.text)

    asyncio.run(main())
    ```

    `asyncio.gather` waits for every call to finish, even if some of them fail. It won't raise until all results (or exceptions) are collected. Each item in `responses` is a full `requests.Response`, so you can inspect `status_code`, `text`, and `headers` exactly as you would with `client.get()`.
  </Step>

  <Step title="Run it with asyncio.run">
    <Warning>
      Always run async code with `asyncio.run(main())`. Calling `main()` directly returns a coroutine object without executing it, and Python will raise `RuntimeWarning: coroutine 'main' was never awaited`.
    </Warning>
  </Step>
</Steps>

## When to use async and concurrency

Use `get_async`/`post_async`/`put_async` with `asyncio.gather` when scraping more than a handful of URLs and you want them processed in parallel instead of one at a time. For a single URL, `client.get()` (see [Quickstart](/fetch/sdk/python/quickstart)) is simpler and does the same thing without the `asyncio` boilerplate.

## Async POST and PUT

`post_async` and `put_async` work the same way, accepting the same `data`, `params`, and `headers` arguments as their synchronous counterparts:

<Tabs>
  <Tab title="POST async">
    ```python theme={null} theme={"dark"}
    responses = await asyncio.gather(*[
        client.post_async(url, data={"key1": "value1"}) for url in urls
    ])
    ```
  </Tab>

  <Tab title="PUT async">
    ```python theme={null} theme={"dark"}
    responses = await asyncio.gather(*[
        client.put_async(url, data={"key1": "value1"}) for url in urls
    ])
    ```
  </Tab>
</Tabs>

## How it works under the hood

<Info>
  The SDK runs each async call on a thread pool sized to your `concurrency` value, then awaits the results, giving you the ergonomics of `asyncio` without needing an async HTTP client. This means the same retry behavior configured via `retries` applies equally to synchronous and async calls. See [Error Handling and Retries](/fetch/sdk/python/error-handling-and-retries) for details.
</Info>

## Troubleshooting

* **`RuntimeWarning: coroutine 'main' was never awaited`**: You called `main()` directly instead of `asyncio.run(main())`.
* **`429 Too Many Requests` even though `concurrency` is set correctly**: Another script or process is using the same API key at the same time. Concurrency limits are per plan, not per client instance, so simultaneous scripts share the same real ceiling even though each `ZenRowsClient` only tracks its own local limit.
* **Some responses in the list are much slower than others**: This is expected; total throughput depends on the slowest requests in each concurrent batch. See the [Concurrency](/fetch/features/concurrency#impact-of-request-duration-on-throughput) guide for the underlying math.
* **Cancelling a request doesn't free a concurrency slot right away**: Zenrows keeps the slot occupied for up to a few minutes after a client-side cancellation while it finishes processing server-side.

## Pricing

Async requests are billed identically to synchronous ones. Cost depends on the `params` you send per request, not on how many run in parallel:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render=True` | 5 |
| `premium_proxy=True` | 10 |
| `js_render=True` + `premium_proxy=True` | 25 |

Running requests concurrently doesn't add cost by itself. It only changes how fast the same requests complete. See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="What's the default concurrency if I don't set one?">
    The default concurrency is 5. You can set it to any number you want according to your Zenrows plan.
  </Accordion>

  <Accordion title="Do async and sync methods share the same concurrency limit?">
    No. The concurrency limit is per `ZenRowsClient` instance. Separate instances or separate scripts don't share limits with each other, even though your Zenrows plan enforces one real limit across all of them.
  </Accordion>

  <Accordion title="Can I use requests' own concurrency tools instead?">
    Yes. `multiprocessing.pool.ThreadPool` with plain `requests` calls works too. See the [Concurrency](/fetch/features/concurrency#python-with-requests) guide for that approach if you'd rather not use the SDK's async methods.
  </Accordion>
</AccordionGroup>


# Custom Headers with the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/custom-headers

Send custom headers to the target website through the Zenrows Python SDK.

By default, Zenrows sends an optimized set of headers to the target site on your behalf. If you need to forward specific headers instead (a `Referer`, a custom `User-Agent`, a cookie), pass a `headers` dictionary to `get`, `post`, or `put`.

```python theme={null} theme={"dark"}
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://www.scrapingcourse.com/ecommerce/"

response = client.get(url, headers={
    "Referer": "https://www.google.com",
    "User-Agent": "MyCustomUserAgent",
})

print(response.text)
```

## What happens when you pass headers

As soon as you pass a `headers` argument, the SDK automatically sets `custom_headers=True` in the request `params` for you, so you don't need to add that yourself. This tells Zenrows to use your headers for the target request instead of its own defaults.

<Warning>
  Sending custom headers overwrites Zenrows' own optimized defaults for that request. This can reduce success rates on sites with strong anti-bot protection, since Zenrows' default headers are tuned to look like real browser traffic. Only override headers when you have a specific reason to (for example, a required `Referer` or session cookie).
</Warning>

## Overriding the default User-Agent

The SDK sends its own `User-Agent` (`zenrows/<version> python`) on every request so Zenrows can identify SDK traffic. If you include a `User-Agent` key in your `headers` dictionary, it replaces this default:

```python theme={null} theme={"dark"}
response = client.get(url, headers={"User-Agent": "MyCustomUserAgent"})
```

## Combining with params

`headers` and `params` are independent arguments, so you can use both together:

```python theme={null} theme={"dark"}
response = client.get(
    url,
    params={"premium_proxy": True},
    headers={"Referer": "https://www.google.com"},
)
```

<Note>
  If a request keeps failing after you add custom headers, try removing them first to confirm whether they're the cause. Zenrows' defaults are usually the safer starting point.
</Note>

## Troubleshooting

* **Success rate drops after adding custom headers**: This is expected per the warning above. Remove your custom `headers` first to confirm whether they're the cause; Zenrows' defaults are usually the safer starting point.
* **Header doesn't seem to reach the target site**: Confirm you're using the SDK's `headers` argument (forwarded to the target) and not accidentally modifying request options meant for Zenrows itself, like `params`.
* **Cookies aren't persisting across requests**: Each SDK call is a separate, stateless request. Pass the `Cookie` header explicitly on every call if you need to maintain a session.
* **`custom_headers` shows up unexpectedly in logs/requests**: This is set automatically whenever you pass a non-empty `headers` dictionary; it's not something you need to remove.

## Pricing

Custom headers don't change the credit cost on their own. Credit cost still depends on `js_render` and `premium_proxy`:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render=True` | 5 |
| `premium_proxy=True` | 10 |
| `js_render=True` + `premium_proxy=True` | 25 |

See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Do I need to set custom_headers myself?">
    No. Passing a non-empty `headers` dictionary to `get`, `post`, or `put` sets `custom_headers=True` automatically.
  </Accordion>

  <Accordion title="Will my custom headers always be used exactly as sent?">
    Yes, Zenrows forwards them to the target as provided. This is precisely why overriding them can lower success rates on protected sites; you lose Zenrows' tuned defaults.
  </Accordion>

  <Accordion title="Can I set headers meant for Zenrows itself (like an API version header)?">
    No. `headers` is forwarded to the target website, not to Zenrows' API. Zenrows-specific options belong in `params`.
  </Accordion>
</AccordionGroup>


# Error Handling and Retries with the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/error-handling-and-retries

Configure automatic retries in the Zenrows Python SDK and handle failed requests reliably.

Web scraping requests occasionally fail for transient reasons, like a rate limit or a temporary server hiccup. The SDK can retry these automatically so you don't have to write that logic yourself.

## Basic usage

Retries are **off by default** (`retries=0`). Pass a `retries` count to the client constructor to turn them on:

```python theme={null} theme={"dark"}
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY", retries=2)

response = client.get("https://www.zenrows.com/")
```

With `retries=2`, a failed request is attempted up to 2 additional times (3 attempts total) before the SDK returns the last response it received.

## When to enable retries

Enable `retries` (2–3 is a reasonable starting point) whenever you're running requests at scale or in production. Anywhere a transient `429` or `5xx` shouldn't fail your whole job. Leave `retries=0` (the default) for quick one-off scripts or debugging, where you'd rather see the raw failure immediately than wait through automatic retry attempts.

## Which failures get retried

The SDK retries requests that come back with one of these status codes:

| Status code | Meaning |
| - | - |
| `422` | Unprocessable Entity |
| `429` | Too Many Requests |
| `500` | Internal Server Error |
| `502` | Bad Gateway |
| `503` | Service Unavailable |
| `504` | Gateway Timeout |

Between attempts, it waits using exponential backoff (starting around 0.5 seconds and increasing with each retry), so it doesn't hammer the API or the target site immediately after a failure.

<Tip>
  You aren't charged for retried attempts that fail. Only successful responses count toward your usage. See the [Retry Failed Requests](/zenrows-academy/retry-failed-requests) guide for more on how this works.
</Tip>

## Handling failures

The SDK treats HTTP error status codes and network-level problems differently. Pick the tab that matches what you're dealing with:

<Tabs>
  <Tab title="HTTP error status">
    Even after all retry attempts are exhausted, the SDK returns the final `requests.Response` object. It does **not** raise an exception for a non-2xx status. Always check the response before trusting its content:

    ```python theme={null} theme={"dark"}
    response = client.get(url, params={"js_render": True})

    if response.ok:
        print(response.text)
    else:
        print(f"Failed after retries: {response.status_code} - {response.text}")
    ```

    If you'd rather fail loudly, call `response.raise_for_status()`, which raises `requests.exceptions.HTTPError` for 4xx/5xx responses:

    ```python theme={null} theme={"dark"}
    response = client.get(url)
    response.raise_for_status()
    ```
  </Tab>

  <Tab title="Network-level exceptions">
    Connection problems (DNS failures, connection timeouts, dropped connections) are raised as exceptions by the underlying `requests` library, not returned as a response. Wrap calls in a `try`/`except` to handle those:

    ```python theme={null} theme={"dark"}
    import requests

    try:
        response = client.get(url, timeout=30)
    except requests.exceptions.RequestException as error:
        print(f"Request failed: {error}")
    ```

    <Note>
      `timeout` and other [requests](https://docs.python-requests.org/) keyword arguments can be passed directly to `get`, `post`, and `put`. They're forwarded to the underlying HTTP call.
    </Note>
  </Tab>
</Tabs>

## Troubleshooting

* **Response still fails after retries are exhausted**: Check `response.status_code` and look it up in [API Error Codes](/api-error-codes). Some failures (like an invalid `apikey`) won't be fixed by retrying and need a code change instead.
* **Retries seem to take a long time**: That's the exponential backoff by design. Lower `retries` if latency matters more than success rate for your use case.
* **Getting exceptions instead of error responses**: That only happens for network-level failures (timeouts, DNS errors), not HTTP error status codes. Wrap the call in `try`/`except requests.exceptions.RequestException` as shown above.
* **429 errors even with retries enabled**: You're likely exceeding your plan's concurrency limit, not hitting a one-off rate limit. See [Async Requests and Concurrency](/fetch/sdk/python/async-concurrency).

## Pricing

Retried attempts that fail are **not** billed. Only the final successful response counts. This makes `retries` effectively free insurance against transient failures:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render=True` | 5 |
| `premium_proxy=True` | 10 |
| `js_render=True` + `premium_proxy=True` | 25 |

<Note>`404` and `410` responses count as successful and are billed, since the request itself completed correctly.</Note>

## FAQ

<AccordionGroup>
  <Accordion title="What's the default number of retries?">
    The default number of retries is 0. Retries are disabled unless you explicitly pass `retries=N` to the constructor.
  </Accordion>

  <Accordion title="Am I charged for retried requests that eventually fail?">
    No. Only successful requests consume your Zenrows balance.
  </Accordion>

  <Accordion title="Does the SDK retry on connection timeouts automatically?">
    Connection-level failures raise a `requests` exception rather than returning a response, so they're not covered by the `retries` status-code list. Handle them with your own `try`/`except`, as shown above.
  </Accordion>
</AccordionGroup>

## Look up a specific error code

For a full breakdown of Zenrows error codes (like `AUTH002` or `RESP001`) and how to resolve each one, see the [API Error Codes](/api-error-codes) reference.


# Frequently Asked Questions
Source: https://docs.zenrows.com/fetch/sdk/python/faq

Answers to common questions about the Zenrows Python SDK.

### Setup and compatibility

<AccordionGroup>
  <Accordion title="What Python versions does the SDK support?">
    Python 3.6 and later. The SDK has no other runtime dependencies beyond `requests` and `urllib3`, which are installed automatically with `pip install zenrows`.
  </Accordion>

  <Accordion title="Do I need to install requests separately?">
    No. `requests` (and `urllib3`) are installed automatically as dependencies when you run `pip install zenrows`.
  </Accordion>

  <Accordion title="Does the SDK support Proxy Mode?">
    No. The SDK talks to Fetch's endpoint directly (API Mode). If you want to use Zenrows as a proxy in your own HTTP client instead, see the [Proxy Mode documentation](/fetch/setup#connection-modes) rather than the SDK.
  </Accordion>
</AccordionGroup>

### Requests, errors, and concurrency

<AccordionGroup>
  <Accordion title="How do I set a request timeout?">
    Pass `timeout` as an extra keyword argument to `get`, `post`, or `put`. It's forwarded to the underlying `requests` call: `client.get(url, timeout=30)`. We recommend not setting a timeout below 3 minutes, since canceling a request client-side doesn't immediately free up the concurrency slot on Zenrows' side.
  </Accordion>

  <Accordion title="Why isn't my request raising an exception on failure?">
    By design, the SDK returns the `requests.Response` object even for non-2xx status codes. It doesn't raise automatically. Check `response.ok` or `response.status_code`, or call `response.raise_for_status()` yourself if you want an exception. See [Error Handling and Retries](/fetch/sdk/python/error-handling-and-retries).
  </Accordion>

  <Accordion title="Can two scripts share the same concurrency limit?">
    No. Each `ZenRowsClient` instance enforces its own `concurrency` setting independently. Running two scripts (or two client instances) at the same time can exceed your plan's actual limit and trigger `429 Too Many Requests` errors. Coordinate concurrency across processes yourself if you run multiple scripts against the same Zenrows plan.
  </Accordion>
</AccordionGroup>

### Project and support

<AccordionGroup>
  <Accordion title="Is there a Node.js version of the SDK?">
    Yes. See the <a href="https://www.npmjs.com/package/zenrows">Zenrows Node.js SDK</a> on npm.
  </Accordion>

  <Accordion title="Where can I see the SDK's source code or report a bug?">
    The SDK is open source on <a href="https://github.com/ZenRows/zenrows-python-sdk">GitHub</a>. Open an issue there for bugs, or a pull request if you'd like to contribute.
  </Accordion>

  <Accordion title="What license is the SDK released under?">
    MIT. You can use, modify, and distribute it freely; see the <a href="https://github.com/ZenRows/zenrows-python-sdk/blob/main/LICENSE">LICENSE</a> file in the repository.
  </Accordion>
</AccordionGroup>

<Card title="Still stuck?" href="mailto:success@zenrows.com">
  Check the [API Error Codes](/api-error-codes) reference, or contact support directly.
</Card>


# Installing the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/installation

Install the Zenrows Python SDK and get your API key ready to make requests.

## Requirements

<CardGroup>
  <Card title="Python 3.6+" icon="python">
    Any Python version from 3.6 onward works.
  </Card>

  <Card title="API key" icon="key" href="https://app.zenrows.com/register?p=free">
    Create a free account to get yours, no credit card required.
  </Card>
</CardGroup>

<Steps>
  <Step title="Install the package">
    Install the SDK from PyPI with `pip`:

    ```bash theme={null} theme={"dark"}
    pip install zenrows
    ```

    This pulls in the SDK's only dependencies, `requests` and `urllib3`, so there's nothing else to install manually.

    <Tip>
      Working in a project with other dependencies? Add `zenrows` to your `requirements.txt` (or `pyproject.toml`) instead of installing it standalone, so the version stays pinned alongside the rest of your project.
    </Tip>
  </Step>

  <Step title="Verify the install">
    Run this in a Python shell or script to confirm the package is available:

    ```python theme={null} theme={"dark"}
    import zenrows

    print(zenrows.ZenRowsClient)
    ```

    If it prints something like `<class 'zenrows.client.ZenRowsClient'>` without raising an `ImportError`, you're ready to go.
  </Step>

  <Step title="Get your API key">
    Your API key authenticates every request you send through the SDK. Grab it from the [Zenrows Request Playground](https://app.zenrows.com/fetch) after signing up.

    <Warning>
      Keep your API key private. Load it from an environment variable or a `.env` file instead of hardcoding it in scripts you commit to version control.
    </Warning>
  </Step>
</Steps>

<Card title="Next: Quickstart" icon="rocket" href="/fetch/sdk/python/quickstart">
  Make your first request with the SDK.
</Card>


# Introduction to the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/introduction

Official Zenrows Python SDK for Fetch. Handles authentication, retries, and concurrency so you can scrape with less boilerplate.

The <a href="https://pypi.org/project/zenrows/">Zenrows Python SDK</a> is a lightweight client for [Fetch](/fetch/setup). It wraps the same HTTP endpoint you'd call directly with `requests`, but adds a few conveniences developers ask for on every scraping project: automatic retries, built-in concurrency control, and `async`/`await` support.

<Info>
  If you're already comfortable making requests with Python's `requests` library, the SDK will feel familiar. Every call still returns a standard `requests.Response` object, so nothing about how you read the response changes.
</Info>

## What the SDK handles for you

* **Authentication** – your API key is set once in the client constructor, not repeated on every call.
* **Retries** – optionally retry failed requests (`429`, `500`, `502`, `503`, `504`, and `422`) with exponential backoff, without writing retry logic yourself.
* **Concurrency** – cap how many requests run in parallel to match your Zenrows plan's concurrency limit.
* **Async support** – `get_async`, `post_async`, and `put_async` methods for running many requests concurrently with `asyncio`.

## What you'll need

<CardGroup>
  <Card title="Python 3.6+" icon="python">
    Any Python version from 3.6 onward works.
  </Card>

  <Card title="API key" icon="key" href="https://app.zenrows.com/register?p=free">
    Create a free account to get yours, no credit card required.
  </Card>
</CardGroup>

<Note>
  The SDK only covers Fetch's GET, POST, and PUT modes. For the full list of request parameters (`js_render`, `premium_proxy`, `css_extractor`, and so on), see the [Fetch setup guide](/fetch/setup). Every parameter documented there works the same way through the SDK's `params` argument.
</Note>

<Tip>
  Looking for a different language? Zenrows also maintains a <a href="https://www.npmjs.com/package/zenrows">Node.js SDK</a>.
</Tip>

## Explore the Python SDK

<CardGroup>
  <Card title="Installation" icon="download" href="/fetch/sdk/python/installation">
    Install the package and get your API key.
  </Card>

  <Card title="Quickstart" icon="rocket" href="/fetch/sdk/python/quickstart">
    Make your first request in a few lines of code.
  </Card>

  <Card title="POST and PUT Requests" icon="send" href="/fetch/sdk/python/post-put-requests">
    Submit forms or data payloads.
  </Card>

  <Card title="Async Requests and Concurrency" icon="bolt" href="/fetch/sdk/python/async-concurrency">
    Scrape many URLs in parallel with `asyncio`.
  </Card>

  <Card title="Error Handling and Retries" icon="shield" href="/fetch/sdk/python/error-handling-and-retries">
    Configure automatic retries and handle failures.
  </Card>

  <Card title="The Response Object" icon="file-text" href="/fetch/sdk/python/response-object">
    What you get back from every request.
  </Card>

  <Card title="Custom Headers" icon="tags" href="/fetch/sdk/python/custom-headers">
    Forward specific headers to the target site.
  </Card>

  <Card title="FAQ" icon="help-circle" href="/fetch/sdk/python/faq">
    Answers to common questions.
  </Card>
</CardGroup>


# POST and PUT Requests with the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/post-put-requests

Send POST and PUT requests through the Zenrows Python SDK, including form data payloads.

Besides `client.get()`, the SDK exposes `client.post()` and `client.put()` for targets that expect a form submission or a data payload. Both accept the same `params` and `headers` arguments as `get`, plus a `data` argument for the request body.

## Basic usage

```python theme={null} theme={"dark"}
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"

response = client.post(url, data={
    "key1": "value1",
    "key2": "value2",
})

print(response.text)
```

`client.put()` works identically, replacing `.post` with `.put`:

```python theme={null} theme={"dark"}
response = client.put(url, data={
    "key1": "value1",
    "key2": "value2",
})
```

## When to use POST or PUT

* **POST**: submitting a form on the target site, triggering a search or filter that requires a form submission, or calling a write endpoint on a target API.
* **PUT**: updating a resource on a target API that expects `PUT` semantics.
* **Neither**: most scraping tasks (reading page content) only need `client.get()`. Reach for POST/PUT specifically when the target requires it, not by default.

<Tabs>
  <Tab title="POST">
    ```python theme={null} theme={"dark"}
    from zenrows import ZenRowsClient

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
    url = "https://httpbin.io/anything"

    response = client.post(url, data={
        "key1": "value1",
        "key2": "value2",
    })

    print(response.text)
    ```
  </Tab>

  <Tab title="PUT">
    ```python theme={null} theme={"dark"}
    from zenrows import ZenRowsClient

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
    url = "https://httpbin.io/anything"

    response = client.put(url, data={
        "key1": "value1",
        "key2": "value2",
    })

    print(response.text)
    ```
  </Tab>
</Tabs>

## Combining with request parameters

`params` still works the same way it does for GET requests, so you can enable `premium_proxy`, or other Fetch parameters alongside `data`:

```python theme={null} theme={"dark"}
response = client.post(
    url,
    data={"key1": "value1"},
    params={"premium_proxy": True},
)
```

<Warning>
  `js_render` only works with GET requests (`client.get()`). It is not supported on POST or PUT requests.
</Warning>

In case you need to send a JSON body instead of form-encoded data, pass `json={...}` as an extra keyword argument. It's forwarded straight to the underlying `requests` call: `client.post(url, json={"key1": "value1"})`.

<Note>
  `post` and `put` also have `async` counterparts, `post_async` and `put_async`. See [Async Requests and Concurrency](/fetch/sdk/python/async-concurrency).
</Note>

## Troubleshooting

* **Target site doesn't reflect the submitted data**: Some forms require specific headers (like `Content-Type` or a `Referer`) to accept the submission. See [Custom Headers](/fetch/sdk/python/custom-headers).
* **Getting `REQS005 Method Not Allowed`**: The target endpoint doesn't accept the HTTP verb you're using. Zenrows only supports `GET`, `POST`, and `PUT`; confirm the target expects one of these.
* **`data` isn't being sent as expected**: `data` is form-encoded by default (like an HTML form submit). If the target expects JSON, use the `json=` keyword argument instead, not `data=`.

## Pricing

POST and PUT requests are billed the same way as GET requests. Cost depends on the `params` you use, not the HTTP method:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `premium_proxy=True` | 10 |

See [Zenrows Pricing](/first-steps/pricing) for plan-specific rates.

<Warning>
  `js_render` only works with GET requests (`client.get()`). It is not supported on POST or PUT requests.
</Warning>

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Can I send both params and data in the same request?">
    Yes. `params` (Zenrows/API options like `premium_proxy`) and `data` (the payload sent to the target) are independent arguments and can be combined.
  </Accordion>

  <Accordion title="Does Zenrows support DELETE or PATCH?">
    No. Only `GET`, `POST`, and `PUT` are supported by Fetch and this SDK.
  </Accordion>

  <Accordion title="How do I send multipart/form-data (file uploads)?">
    Pass `files={...}` as an extra keyword argument, forwarded directly to `requests`: `client.post(url, files={"file": open("photo.jpg", "rb")})`.
  </Accordion>
</AccordionGroup>


# Making your first request with the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/quickstart

Make your first request with the Zenrows Python SDK in a few lines of code.

This guide makes a single GET request with the Zenrows Python SDK: create a client, send the request, read the response.

<Note>Haven't installed the SDK yet? Follow [Installation](/fetch/sdk/python/installation) first.</Note>

## Basic usage

<Steps>
  <Step title="Create a client">
    Import `ZenRowsClient` and instantiate it with your API key:

    ```python theme={null} theme={"dark"}
    from zenrows import ZenRowsClient

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
    ```
  </Step>

  <Step title="Send a GET request">
    Call `client.get()` with the URL you want to scrape:

    ```python theme={null} theme={"dark"}
    from zenrows import ZenRowsClient

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
    url = "https://www.scrapingcourse.com/ecommerce/"

    response = client.get(url)

    print(response.text)
    ```

    `response` is a standard `requests.Response` object.

    <Tip>See [The Response Object](/fetch/sdk/python/response-object) for everything available on it.</Tip>
  </Step>

  <Step title="Add request parameters">
    Pass a `params` dictionary to enable Fetch features like JavaScript rendering or Premium Proxies:

    ```python theme={null} theme={"dark"}
    from zenrows import ZenRowsClient

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
    url = "https://www.scrapingcourse.com/ecommerce/"

    response = client.get(url, params={
        "js_render": True,
        "premium_proxy": True,
        "proxy_country": "us",
    })

    print(response.text)
    ```

    <Tip>
      `params` accepts every parameter documented in the [Fetch setup guide](/fetch/setup), including `extract`, `css_extractor`, `wait_for`, `block_resources`, and `outputs`. They all work exactly the same way through the SDK.
    </Tip>
  </Step>

  <Step title="Check the result">
    Since the SDK doesn't raise an exception on non-2xx responses, always check the status before using the content:

    ```python theme={null} theme={"dark"}
    if response.ok:
        print(response.text)
    else:
        print(f"Request failed with status {response.status_code}: {response.text}")
    ```

    See [Error Handling and Retries](/fetch/sdk/python/error-handling-and-retries) for how to configure automatic retries so you don't have to handle every transient failure manually.
  </Step>
</Steps>

## Troubleshooting

* **`response.status_code` is `401` or similar auth error**: Double-check your API key was copied correctly from the [Zenrows Request Playground](https://app.zenrows.com/fetch) with no extra whitespace. See [`AUTH002`](/api-error-codes#auth002-invalid-api-key).
* **`response.text` is empty or looks like an error page, not the target site**: The target likely needs `js_render=True` and/or `premium_proxy=True`. See [`REQS002`](/api-error-codes#reqs002-request-requirements-unsatisfied).
* **Request hangs for a long time**: Pass a `timeout` keyword argument, e.g. `client.get(url, timeout=60)`. We recommend not going below 3 minutes for protected sites using `js_render`.
* **`429 Too Many Requests`**: You've hit your plan's concurrency or rate limit. See [Async Requests and Concurrency](/fetch/sdk/python/async-concurrency).

## Pricing

Cost depends on the `params` you send, not on the SDK itself:

| Configuration | Credits per request |
| - | - |
| Basic request (no `js_render`/`premium_proxy`) | 1 |
| `js_render=True` | 5 |
| `premium_proxy=True` | 10 |
| `js_render=True` + `premium_proxy=True` | 25 |

Only successful requests are billed. See [Zenrows Pricing](/first-steps/pricing) for exact per-plan rates.

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Do I need params for every request?">
    No, `params` is optional. `client.get(url)` alone works for public pages with no anti-bot protection.
  </Accordion>

  <Accordion title="Why doesn't a failed request raise an exception?">
    By design, the SDK returns the `requests.Response` object even for error status codes. Check `response.ok` or call `response.raise_for_status()` yourself. See [Error Handling and Retries](/fetch/sdk/python/error-handling-and-retries).
  </Accordion>

  <Accordion title="Can I reuse the same client for multiple requests?">
    Yes. Create one `ZenRowsClient` instance and call `.get()`/`.post()`/`.put()` on it as many times as needed; there's no need to recreate it per request.
  </Accordion>
</AccordionGroup>

## What's next

<CardGroup>
  <Card title="POST and PUT Requests" icon="send" href="/fetch/sdk/python/post-put-requests">
    Submit forms or data payloads.
  </Card>

  <Card title="Async Requests and Concurrency" icon="bolt" href="/fetch/sdk/python/async-concurrency">
    Scrape many URLs in parallel.
  </Card>

  <Card title="Custom Headers" icon="tags" href="/fetch/sdk/python/custom-headers">
    Forward specific headers to the target site.
  </Card>
</CardGroup>


# The Response Object with the Python SDK
Source: https://docs.zenrows.com/fetch/sdk/python/response-object

Understand the requests.Response object returned by every Zenrows Python SDK call.

Every method on `ZenRowsClient` (`get`, `post`, `put`, and their `_async` counterparts) returns a standard <a href="https://docs.python-requests.org/en/latest/api/#requests.Response">`requests.Response`</a> object. The SDK doesn't wrap or transform it, so everything you already know about `requests` applies directly.

## Commonly used attributes

| Attribute | Returns | Use it when |
| - | - | - |
| `response.text` | Response body as a string | Reading HTML or plain text |
| `response.content` | Response body as raw bytes | Handling binary data (PDFs, screenshots, images) |
| `response.json()` | Parsed JSON | The body is JSON (e.g. `json_response=true`) |
| `response.status_code` | HTTP status code, e.g. `200` | Checking exactly what happened |
| `response.ok` | `True` if `status_code` is under 400 | Quick success check |
| `response.headers` | Dict-like object of response headers | Reading Zenrows monitoring headers |
| `response.request.url` | The final URL that was requested | Debugging what was actually sent |

<Tabs>
  <Tab title="Text / HTML">
    ```python theme={null} theme={"dark"}
    response = client.get(url)
    html = response.text
    ```
  </Tab>

  <Tab title="Binary (PDF, screenshot)">
    ```python theme={null} theme={"dark"}
    response = client.get(url, params={"response_type": "pdf", "js_render": True})

    with open("page.pdf", "wb") as f:
        f.write(response.content)
    ```

    <Tip>
      Use `response.content` (not `response.text`) when the target returns binary data. For example, a PDF from `response_type=pdf` or a screenshot. Writing `response.content` in binary mode (`"wb"`) avoids corrupting the file.
    </Tip>
  </Tab>

  <Tab title="JSON">
    ```python theme={null} theme={"dark"}
    response = client.get(url, params={"json_response": True, "js_render": True})
    data = response.json()
    ```
  </Tab>
</Tabs>

## Reading Zenrows-specific headers

Zenrows adds monitoring headers to every response, useful for debugging and tracking usage:

```python theme={null} theme={"dark"}
response = client.get(url)

print(response.headers.get("X-Request-Id"))          # unique ID for this request
print(response.headers.get("X-Request-Cost"))        # dollar cost of this request (e.g., 0.001)
print(response.headers.get("X-Request-Credits"))     # number of credits consumed by this request
print(response.headers.get("X-Key-Credits-Remaining"))  # credits left before this key hits a credit cap (capped keys only)
print(response.headers.get("Concurrency-Remaining"))  # concurrency slots left on your plan
print(response.headers.get("Zr-Final-Url"))           # final URL after redirects
```

<Tip>
  If you ever contact Zenrows support about a failed request, include the `X-Request-Id` header value. It lets the team trace the exact request in the logs.
</Tip>

Headers coming from the target website itself are prefixed with `Zr-` to distinguish them from Zenrows' own headers. See the full list in the [Fetch setup guide](/fetch/setup#response-headers).

## Checking success before using the body

Because the SDK returns the response as-is rather than raising on error status codes, always confirm the request succeeded first:

```python theme={null} theme={"dark"}
response = client.get(url)

if response.ok:
    data = response.text
else:
    print(f"Request failed: {response.status_code}")
```

<Card title="Error Handling and Retries" icon="shield" href="/fetch/sdk/python/error-handling-and-retries">
  More on this pattern, plus how to configure automatic retries.
</Card>

## Troubleshooting

* **`response.json()` raises `JSONDecodeError`**: The body isn't JSON. Confirm you requested `json_response=true` or that the target actually returns JSON before calling `.json()`.
* **Saved file (PDF, screenshot, image) is corrupted**: You wrote `response.text` instead of `response.content`, or opened the file in text mode (`"w"`) instead of binary (`"wb"`).
* **`response.headers.get("X-Request-Id")` returns `None`**: The header name is case-insensitive in `requests`, so this shouldn't normally happen; confirm the request actually reached Zenrows (check `response.status_code` first) rather than failing at the network layer.
* **`response.request.url` doesn't match the URL you passed**: This is expected. It's the full URL sent to Zenrows' API (including your `apikey` and `params` as query parameters), not the target URL by itself.

## Pricing

The response itself doesn't add cost. Zenrows adds two headers to help you track usage:

* `X-Request-Cost`: the dollar cost of the request (e.g., `0.001`)
* `X-Request-Credits`: the number of credits consumed by the request

Both values are based on the `params` used:

| Configuration | Credits per request |
| - | - |
| Basic request | 1 |
| `js_render=True` | 5 |
| `premium_proxy=True` | 10 |
| `js_render=True` + `premium_proxy=True` | 25 |

See [Zenrows Pricing](/first-steps/pricing).

## FAQ (Frequently Asked Questions)

<AccordionGroup>
  <Accordion title="Is the response object specific to Zenrows?">
    No. It's the standard `requests.Response` object from Python's `requests` library. Any `requests` documentation or Stack Overflow answer about it applies here too.
  </Accordion>

  <Accordion title="How do I get the response as a Python dict?">
    Call `response.json()` if the body is JSON. For example, when using `json_response=true` or `extract=auto` on structured data.
  </Accordion>

  <Accordion title="Where do I find how much a request cost?">
    Check the response headers: `X-Request-Cost` gives you the dollar cost (e.g., `0.001`), and `X-Request-Credits` gives you the number of credits consumed.
  </Accordion>
</AccordionGroup>


# Setting Up Fetch
Source: https://docs.zenrows.com/fetch/setup

Complete setup guide for Zenrows Fetch including API key configuration, authentication, and first scraping request.

Zenrows' Fetch enables fast, efficient, and hassle-free data extraction from web pages by providing versatile scraping modes. Whether you're new to scraping or already experienced, Zenrows adapts to your needs, making it easy to collect data from the web while overcoming the common challenges posed by modern websites, including CAPTCHAs and anti-bot mechanisms.

## Getting started

Zenrows' API offers a straightforward approach to web scraping with just two key components:

1. [API key](https://app.zenrows.com/register?p=free) – Your unique key to authenticate requests.
2. **The encoded URL**  – The target URL you want to scrape, properly encoded.

Once logged in, you'll be directed to the [Overview page](https://app.zenrows.com/overview), where you can locate your API key and start running your requests.

<Frame>
  <img alt="Zenrows Request Playground Page" />
</Frame>

Here's what you can do:

* Paste your target URL to configure a request.
* Enable Zenrows parameters like blocking unnecessary resources or specifying JavaScript instructions.
* Select a programming language (e.g., Python) and a connection mode. For general use, select "cURL" to generate the full API endpoint that works with any HTTP client.
* Test the request by clicking "Try It" to execute the request in the browser.
* Copy the generated code and integrate it into your Python script.

### Connection modes

Zenrows Fetch accommodates different needs through three flexible connection methods:

* **API Mode:** Ideal for quick and easy scraping with all the processing handled by Zenrows. Just pass your API key and URL, and get the data back.
* **Proxy Mode:** For users who want to maintain control over their scraping logic but need Zenrows' powerful proxy network, this mode allows you to use Zenrows as a proxy. You can integrate it into your existing system, enabling access to residential, datacenter, and rotating proxies to avoid IP bans and anti-bot systems.
* **SDK:** Zenrows offers **dedicated SDKs for Python and Node.js** to simplify integration for developers. These SDKs abstract many low-level details, like request handling, retries, and response processing, making it easy for both beginners and experienced developers to get started quickly.

## Response Headers

Zenrows provides several types of headers in API responses to help you monitor usage and understand the results:

### Target Website Headers

Headers from the target website are prefixed with `Zr-` for easy identification. For example:

```plaintext theme={"dark"}
Zr-Content-Encoding: gzip
Zr-Content-Type: text/html
Zr-Cookies: _pxhd=Bq7P4CRaW1B...
```

### Zenrows Monitoring Headers

Zenrows also includes valuable monitoring headers with each response:

```
Concurrency-Limit: 200
Concurrency-Remaining: 199
X-Request-Cost: 0.001
X-Request-Id: 67fa4e35647515d8ad61bb3ee041e1bb
Zr-Final-Url: https://www.example.com/
```

These headers provide important information:

* **Concurrency-Limit**: Your maximum concurrent request limit based on your plan
* **Concurrency-Remaining**: Number of additional concurrent requests you can make
* **X-Request-Cost**: The cost of this request
* **X-Key-Credits-Remaining**: The lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps). Sent only for keys with at least one cap
* **X-Request-Id**: Unique identifier for the request, crucial for troubleshooting
* **Zr-Final-Url**: The final URL after any redirects, ensuring you always know the last page accessed.

<Tip>When contacting support about any issues, always include the **X-Request-Id** from your response. This identifier allows our team to quickly trace your specific request in our logs.</Tip>

## API Key

Your API Key is the gateway to the Zenrows Fetch, functioning as both an authentication tool and an identifier for all your requests. Without it, you won't be able to access Zenrows' features or receive any scraped data.

You can easily [create your API Key](https://app.zenrows.com/register?p=free) by signing up for an account.

<Tip>Make sure to store your key securely and never share it publicly.</Tip>

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'mode': 'auto',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
  	url: 'https://api.zenrows.com/v1/',
  	method: 'GET',
  	params: {
  		'url': url,
  		'apikey': apikey,
  		'mode': 'auto',
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto"
  ```
</CodeGroup>

## URL

The URL you want to scrape must be URL-encoded to prevent issues with special characters like spaces, question marks, and ampersands. Encoding ensures that servers interpret the URL correctly, avoiding common issues like broken requests or misinterpretation of the data.

Always double-check your URLs for proper encoding. If you're using Zenrows SDKs or common HTTP libraries like `requests` in Python or `axios` in Node.js, they often handle this for you.


# Advanced CSS Selector Examples
Source: https://docs.zenrows.com/fetch/troubleshooting/advanced-css-selector-examples

Practical CSS selector examples for extracting structured data from tables, lists, and nested layouts using the Zenrows scraping API.

This guide walks you through advanced CSS selector strategies to help you extract structured data from a wide variety of web layouts using the Zenrows API.

## Basic API Call Structure

All examples use the following basic API call pattern with the Zenrows API:

```javascript theme={"dark"}
const axios = require('axios');

const response = await axios.get('https://api.zenrows.com/v1/', {
  params: {
    apikey: 'YOUR_ZENROWS_API_KEY',
    url: 'https://example.com',
    js_render: true, // Optional, needed for JavaScript-rendered content
    css_extractor: JSON.stringify({
      // Your selectors here
    })
  }
});
```

## Essential CSS Selector Techniques

### Basic Selectors

Target elements by their tag names, classes, or IDs:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  headings: "h1", // Select all <h1> elements
  products: ".product", // Select elements with the 'product' class
  mainContent: "#main-content", // Select the element with the 'main-content' ID
  productTitles: ".product h2.title", // Select <h2> elements with the 'title' class inside 'product' class
  topLevelNav: "nav > a" // Select direct child <a> elements of <nav>
})
```

### Attribute Selectors

Extract content based on HTML attributes:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  imageUrls: "img @src", // Extract 'src' attribute from <img> tags
  linkUrls: "a @href", // Extract 'href' attribute from <a> tags
  premiumItems: "[data-premium='true']", // Select elements with a specific attribute value
  externalLinks: "[href^='https://'] @href", // Extract 'href' starting with 'https://'
  pdfDownloads: "[href$='.pdf'] @href" // Extract 'href' ending with '.pdf'
})
```

### Positional Selectors

Target elements based on their position in the document:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  firstProduct: ".product:first-child", // Selects the first element with the class 'product'
  lastProduct: ".product:last-child", // Selects the last element with the class 'product'

  thirdProduct: ".product:nth-child(3)", // Selects the third element with the class 'product'

  evenProducts: ".product:nth-child(even)", // Selects all even-numbered 'product' elements
  oddProducts: ".product:nth-child(odd)", // Selects all odd-numbered 'product' elements

  firstHeading: "h2:nth-of-type(1)", // Selects the first <h2> element of its type

  tableHeaders: "table th", // Selects all <th> elements inside any <table>
  secondColumnCells: "tr td:nth-child(2)" // Selects the second <td> element in each table row
})
```

### Combining Multiple Selectors

Combine selectors for more specific targeting:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  headingsAndLinks: "h1, h2, a", // Multiple selectors with commas
  cardTitles: ".card .title", // Descendant combinator (space)
  directListItems: "ul > li", // Child combinator (>)
  labelValues: "label + input @value", // Adjacent sibling combinator (+)
  relatedItems: ".main-item ~ .related-item" // General sibling combinator (~)
})
```

## Practical Extraction Scenarios

### E-commerce Product Extraction

Extract product details from a listing page:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  productNames: ".product-item .product-title", // Selects the product title within each product item
  productPrices: ".product-item .price", // Selects the price element within each product item
  productRatings: ".product-item .rating @data-score", // Extracts the 'data-score' attribute from the rating element
  productImages: ".product-item img.product-image @src", // Extracts the 'src' attribute from the product image
  productUrls: ".product-item a.product-link @href", // Extracts the 'href' attribute from the product link
  productAvailability: ".product-item .availability-badge", // Selects the availability badge within each product item
  productDiscounts: ".product-item .discount-tag" // Selects the discount tag element within each product item
})
```

### Product Specification Tables

Extracting structured data from specification tables:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  specLabels: ".specs-table tr td:first-child", // Table headers
  specValues: ".specs-table tr td:last-child", // Table values
  processor: ".tech-specs .processor", // Processor details
  memory: ".tech-specs .memory", // Memory details
  storage: ".tech-specs .storage", // Storage details
  graphics: ".tech-specs .graphics" // Graphics details
})
```

### Real Estate Listings

Extracting property information:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  propertyAddresses: ".property-listing .address", // Property addresses
  propertyPrices: ".property-listing .price", // Prices
  propertyBedrooms: ".property-listing .bedrooms", // Number of bedrooms
  propertyBathrooms: ".property-listing .bathrooms", // Number of bathrooms
  propertyArea: ".property-listing .square-footage", // Square footage
  propertyTypes: ".property-listing .property-type", // Property type
  propertyAgents: ".property-listing .agent-name", // Agent names
  propertyImages: ".property-listing .property-image @src" // Image URLs
})
```

### News Articles and Blog Posts

Extracting content from articles:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  articleTitle: "article h1", // Article title
  articleSubtitle: "article h2", // Article subtitle
  articleDate: "article .publication-date", // Publication date
  articleAuthor: "article .author-name", // Author name
  articleContent: "article .content p", // Article content
  articleCategories: "article .category-tag", // Categories
  articleImages: "article .article-image @src", // Image URLs
  relatedArticles: ".related-articles .article-link @href" // Related article links
})
```

## Advanced Selection Techniques

### Working with Pagination Elements

Identify and extract pagination information:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  currentPage: ".pagination .current @data-page", // Current page number
  totalPages: ".pagination @data-total-pages", // Total number of pages
  nextPageUrl: ".pagination .next @href", // Next page URL
  prevPageUrl: ".pagination .prev @href", // Previous page URL
  pageNumbers: ".pagination .page-number", // All page numbers
  isLastPage: ".pagination .next @disabled" // Check if it's the last page
})
```

### Navigation Menu Extraction

Extract multi-level navigation structures:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  // Main navigation
  mainNavLinks: ".main-nav > li > a @href",
  mainNavText: ".main-nav > li > a",

  // Second level categories
  subNavLinks: ".main-nav > li > .dropdown > a @href",
  subNavText: ".main-nav > li > .dropdown > a",

  // Third level
  deepNavLinks: ".main-nav > li > .dropdown > .sub-dropdown > a @href"
})
```

### Social Media Content

Extract content from social media-style layouts:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  postAuthors: ".post .author-name",
  postTimestamps: ".post .timestamp",
  postContent: ".post .content-text",
  postImages: ".post .post-image @src",
  postLikes: ".post .like-count",
  postComments: ".post .comment-count",
  postShares: ".post .share-count",

  // Comments
  commentAuthors: ".comments .comment .author",
  commentContent: ".comments .comment .text",
  commentTimestamps: ".comments .comment .time"
})
```

### Data Table Extraction

Extract structured data from tables:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  // Table headers
  tableHeaders: "table thead th",

  // First column (often labels)
  rowLabels: "table tbody tr td:first-child",

  // Specific cells using nth-child
  secondColValues: "table tbody tr td:nth-child(2)",
  thirdColValues: "table tbody tr td:nth-child(3)",

  // Cell with specific data attributes
  highlightedCells: "table td[data-highlight='true']"
})
```

## Special Extraction Scenarios

### Form Fields

Extract form field values and attributes:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  formLabels: "form label", // Form labels
  inputValues: "form input @value", // Input field values
  inputPlaceholders: "form input @placeholder", // Input placeholders
  selectedOptions: "form select option[selected]", // Selected options in dropdowns
  checkboxStatus: "form input[type='checkbox'] @checked", // Checkbox status
  radioStatus: "form input[type='radio'] @checked", // Radio button status
  formActionUrl: "form @action", // Form action URL
  formMethod: "form @method" // Form method (GET/POST)
})
```

### Metadata Extraction

Extract metadata from the HTML `<head>`:

```javascript theme={"dark"}
css_extractor: JSON.stringify({
  pageTitle: "title", // Page title
  metaDescription: "meta[name='description'] @content", // Meta description
  canonicalUrl: "link[rel='canonical'] @href", // Canonical URL
  ogTitle: "meta[property='og:title'] @content", // Open Graph title
  ogImage: "meta[property='og:image'] @content", // Open Graph image
  ogDescription: "meta[property='og:description'] @content", // Open Graph description
  twitterCard: "meta[name='twitter:card'] @content" // Twitter card type
})
```

## Troubleshooting Selectors

When your selectors aren't working as expected, try these approaches:

1. **Make Selectors More Specific**
   ```javascript theme={"dark"}
   // Too general
   { title: ".title" }

   // More specific
   { title: "article .main-content .title" }
   ```

2. **Check for iframes**
   Content might be inside iframes that require additional handling.

3. **Handle Special Characters**
   ```javascript theme={"dark"}
   // For classes with special characters
   { price: ".price-\\$" }
   ```

4. **Use Developer Tools to Verify**
   Always test your selectors using the browser developer tools first.

## Testing Workflow

We recommend this workflow for developing and testing selectors:

1. **Test the selector in the browser using DevTools**

2. **Extract Full HTML First**
   ```javascript theme={"dark"}
   const htmlResponse = await axios.get('https://api.zenrows.com/v1/', {
     params: {
       apikey: 'YOUR_ZENROWS_API_KEY',
       url: 'https://example.com',
       js_render: true
     }
   });
   ```

3. **Test Selectors Locally with Cheerio**
   ```javascript theme={"dark"}
   const cheerio = require('cheerio');
   const $ = cheerio.load(htmlResponse.data);
   console.log($('h1.product-title').text()); // Test selector
   ```

4. **Refine and Apply CSS Selectors with Zenrows**
   ```javascript theme={"dark"}
   const extractedData = await axios.get('https://api.zenrows.com/v1/', {
     params: {
       apikey: 'YOUR_ZENROWS_API_KEY',
       url: 'https://example.com',
       js_render: true,
       css_extractor: JSON.stringify({
         // Your refined selectors
       })
     }
   });
   ```

By following these techniques, you can effectively extract data from even the most complex web layouts using Zenrows and CSS selectors.


# Advanced CSS Selectors
Source: https://docs.zenrows.com/fetch/troubleshooting/advanced-css-selectors

Learn advanced CSS selector techniques like pseudo-classes, combinators, and attribute selectors to scrape dynamic website structures.

This guide will teach you how to use advanced CSS selector techniques to extract specific data from challenging websites. Whether dealing with dynamic content or intricate page structures, these strategies will help you scrape data with precision.

## Beyond Basic Selectors

While simple selectors like `.class` and `#id` work well for simple tasks, complex websites often require more sophisticated approaches. Advanced CSS selectors allow you to:

* Target elements with specific attributes or patterns.
* Combine multiple conditions for greater accuracy.
* Extract data based on element relationships.

## Selector Types and Examples

Websites often don't provide convenient classes but use data attributes or dynamic IDs:

| Selector Type | Example | Description |
| - | - | - |
| Attribute Contains | `[attr*="value"]` | Selects elements with an attribute containing "value" |
| Attribute Starts With | `[attr^="value"]` | Selects elements with an attribute starting with "value" |
| Attribute Ends With | `[attr$="value"]` | Selects elements with an attribute ending with "value" |
| Not Selector | `:not(selector)` | Excludes elements that match the selector |
| Nth-child | `:nth-child(n)` | Selects the nth child of its parent |
| Nth-of-type | `:nth-of-type(n)` | Selects the nth sibling of its type |

## Attribute-Based Selection

Many websites use dynamic IDs or data attributes instead of simple classes. Here's how you can target these elements:

<CodeGroup>
  ```python Python theme={"dark"}
  # Import the necessary libraries
  import json
  import requests

  url = 'https://www.scrapingcourse.com/ecommerce/'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'premium_proxy': 'true',
      # Remove comments below for valid JSON string
      'css_extractor': """{
          # Selects product names within <a> tags that have a data-product_id attribute
          "products_with_data_id": "a[data-product_id] .product-name",

          # Selects the product name where SKU equals 'MS01'
          "products_with_sku_ms01": "a[data-product_sku='MS01'] .product-name",

          # Extracts image URLs that end with .jpg
          "image_jpg": "img[src$='.jpg'] @src",

          # Extracts the link for 'Add to Cart' buttons
          "add_to_cart_buttons": "a.button.add_to_cart_button @href",

          # Selects product names in list items belonging to the 'shorts' category
          "shorts_only": "li[class*='product_cat-shorts'] .product-name"
      }"""
  }
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');

  const url = 'https://www.scrapingcourse.com/ecommerce/';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  const params = {
      url: url,
      apikey: apikey,
      js_render: 'true',
      premium_proxy: 'true',
      // Remove comments below for valid JSON string
      css_extractor: `{
          # Selects product names within <a> tags that have a data-product_id attribute
          "products_with_data_id": "a[data-product_id] .product-name",

          # Selects the product name where SKU equals 'MS01'
          "products_with_sku_ms01": "a[data-product_sku='MS01'] .product-name",

          # Extracts image URLs that end with .jpg
          "image_jpg": "img[src$='.jpg'] @src",

          # Extracts the link for 'Add to Cart' buttons
          "add_to_cart_buttons": "a.button.add_to_cart_button @href",

          # Selects product names in list items belonging to the 'shorts' category
          "shorts_only": "li[class*='product_cat-shorts'] .product-name"
      }`
  };

  axios.get('https://api.zenrows.com/v1/', { params })
      .then(response => {
          // Prettify the JSON response
          const parsedResponse = response.data;
          console.log(JSON.stringify(parsedResponse, null, 4));  // Prettify JSON with indentation
      })
      .catch(error => {
          if (error.response) {
              // The request was made and the server responded with a status code
              // that falls out of the range of 2xx
              console.log("Error: Unable to parse JSON response");
              console.log(error.response.data);
          } else if (error.request) {
              // The request was made but no response was received
              console.log("No response received");
          } else {
              // Something happened in setting up the request that triggered an Error
              console.log("Error", error.message);
          }
      });
  ```
</CodeGroup>

<Note>You can find more details about the CSS extractor in the [CSS Extractor documentation](/fetch/features/css-extractor).</Note>

## Combinatorial Selectors

Combine multiple conditions to pinpoint specific elements:

```javascript theme={"dark"}
// Complex selector combinations
const response = await fetch('https://api.zenrows.com/v1/', {
  method: 'GET',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    apikey: 'YOUR_ZENROWS_API_KEY',
    url: 'https://example.com/search',
    css_extractor: {
      // Items that are on sale AND in stock
      available_sale_items: '.product[data-sale="true"]:not([data-inventory="0"])',
      // Every third featured product
      featured_third: '.featured-products > div:nth-child(3n)',
      // First paragraph in each section except the intro
      first_paragraphs: 'section:not(#intro) p:first-of-type'
    }
  })
});
```

## Selecting by Relationships

Use sibling and parent-child relationships to locate elements:

| Selector | Syntax | Description |
| - | - | - |
| Adjacent | `A + B` → `h2 + p` | Select `p` immediately after an `h2` |
| General Sibling | `A ~ B` → `.a ~ .b` | Select all `.b` siblings after `.a` |
| Direct Child | `A > B` → `ul > li` | Select `li` that is a direct child of `ul` |

```javascript theme={"dark"}
// Relationship-based selectors
const response = await fetch('https://api.zenrows.com/v1/', {
  method: 'GET',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    apikey: 'YOUR_ZENROWS_API_KEY',
    url: 'https://example.com/blog',
    css_extractor: {
      // Extract paragraph immediately following each heading
      heading_descriptions: 'h2 + p',
      // Get all list items after the "featured" item
      items_after_featured: '.featured ~ li',
      // Extract direct text labels within form fields
      form_labels: '.form-field > label'
    }
  })
});
```

## Dynamic Content Selection

When dealing with dynamic or JavaScript-rendered content, enable `js_render` and use flexible selectors:

```javascript theme={"dark"}
// Adapting to dynamic content
const response = await fetch('https://api.zenrows.com/v1/', {
  method: 'GET',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    apikey: 'YOUR_ZENROWS_API_KEY',
    url: 'https://example.com/dynamic-page',
    js_render: true,
    css_extractor: {
      // Multiple potential selectors for the same data type
      prices: [
        '.price-new',
        '[data-testid="price"]',
        '.product-price .amount',
        'span.current-price'
      ].join(', '),
      // Get all elements with specific text pattern
      availability_elements: [
        '[data-availability]',
        ':not(script):not(style):contains("In Stock")',
        ':not(script):not(style):contains("Available")'
      ].join(', ') // Use multiple selectors with .join(', ') to fall back if some fail.
    }
  })
});
```

## Debugging Selectors

When selectors don't work as expected:

1. **Inspect the Full HTML**: Use Zenrows with `js_render: true` to see what the DOM actually contains

2. **Start Broad, Then Narrow Down**:
   ```javascript theme={"dark"}
   // Start general and get more specific
   css_extractor: {
     all_elements: '*', // Get everything to inspect structure
     possible_containers: 'div, section, article', // All potential containers
     with_attributes: '[data-*]' // Elements with data attributes
   }
   ```

3. **Use Text-Based Debugging**: Find elements by their text content:
   ```javascript theme={"dark"}
   js_instructions: [
     'const findElementsByText = (text) => {',
     '  const walker = document.createTreeWalker(',
     '    document.body,',
     '    NodeFilter.SHOW_TEXT,',
     '    { acceptNode: node => node.textContent.includes(text) ? NodeFilter.FILTER_ACCEPT : NodeFilter.FILTER_REJECT }',
     '  );',
     '  const results = [];',
     '  let node;',
     '  while (node = walker.nextNode()) {',
     '    results.push({',
     '      text: node.textContent.trim(),',
     '      path: getNodePath(node.parentElement)',
     '    });',
     '  }',
     '  ',
     '  function getNodePath(el) {',
     '    const path = [];',
     '    while (el && el.nodeType === Node.ELEMENT_NODE) {',
     '      let selector = el.nodeName.toLowerCase();',
     '      if (el.id) selector += `#${el.id}`;',
     '      else if (el.className) selector += `.${Array.from(el.classList).join(".")}`;',
     '      path.unshift(selector);',
     '      el = el.parentElement;',
     '    }',
     '    return path.join(" > ");',
     '  }',
     '  ',
     '  // Create a visible element with results',
     '  const debugDiv = document.createElement("div");',
     '  debugDiv.id = "zenrows-debug-results";',
     '  debugDiv.setAttribute("data-results", JSON.stringify(results));',
     '  debugDiv.style.display = "none";',
     '  document.body.appendChild(debugDiv);',
     '};',
     'findElementsByText("Price");' // Replace with text you're looking for
   ]
   ```

## Selector Performance Tips

Optimize your selectors for both accuracy and performance:

1. **Avoid Universal Selectors**: `*` is slow; use more specific selectors. Use class (`.class`) and ID (`#id`) selectors over attribute selectors for speed.
2. **Minimize Selector Depth**: `.product-grid .product .title` is faster than `body div.container div.products div.product-grid div.product div.title`
3. **Prefer ID and Class Selectors**: `#product-123` is faster than `[data-product-id="123"]`
4. **Avoid Parent Selectors When Possible**: Child (`>`) and adjacent (`+`) selectors are faster than descendant selectors (space)

## CSS Selector Cheat Sheet

| Selector | Purpose | Example |
| - | - | - |
| `element` | Select by tag | `div`, `span`, `h1` |
| `.class` | Select by class | `.product`, `.price` |
| `#id` | Select by ID | `#main`, `#product-123` |
| `[attr]` | Has attribute | `[data-id]` |
| `[attr="val"]` | Exact attribute | `[type="submit"]` |
| `[attr*="val"]` | Contains value | `[href*="product"]` |
| `[attr^="val"]` | Starts with value | `[class^="product-"]` |
| `[attr$="val"]` | Ends with value | `[src$=".jpg"]` |
| `:nth-child(n)` | By position | `li:nth-child(2)` |
| `:first-child` | First child | `li:first-child` |
| `:last-child` | Last child | `li:last-child` |
| `:not(selector)` | Negation | `.item:not(.featured)` |
| `A > B` | Direct child | `.product > .title` |
| `A + B` | Adjacent sibling | `h2 + p` |
| `A ~ B` | General sibling | `.featured ~ .product` |
| `A, B` | Multiple selectors | `.price, .discount` |
| `A B` | Descendant | `.product .price` |

Use these advanced CSS selector techniques to create precise data extraction patterns for even the most complex websites.


# Troubleshooting Partial or Empty Responses
Source: https://docs.zenrows.com/fetch/troubleshooting/empty-or-partialresponses

Troubleshoot empty or partial scraping responses from JavaScript-rendered pages. Covers wait parameters, AJAX handling, and Zenrows fixes.

Sometimes, your requests may return empty or incomplete content. This is typically due to how modern websites load content dynamically using JavaScript or AJAX (XHR) calls. By default, Zenrows returns the HTML content once the DOM is initially loaded, which might not include content populated by JavaScript.

This guide provides diagnostic steps, common scenarios, and parameter configurations to help you resolve these issues and get complete, usable responses.

<Tip>These troubleshooting steps are applicable across all Zenrows features, including `screenshot`, `outputs`, `json_response`, and others.</Tip>

## Why Does This Happen?

Even if a page visually "loads," its content might still be missing because:

* The content is rendered by **JavaScript** after the DOM loads.
* The actual data is fetched via **XHR (AJAX) requests**.
* The page relies on **user interaction** or **delays** to render content.
* **Anti-bot measures** may serve minimal or obfuscated content.

## Quick Decision Tree

```plaintext theme={"dark"}
ERROR
  ↓
Is js_render enabled?
  ├─ No → Enable `js_render`
  └─ Yes
       ↓
   Is content dynamic (loaded after a delay or via JS)?
       ├─ Yes → Use `wait` or `wait_for`
       └─ No
            ↓
     Is content loaded via XHR?
            ├─ Yes → Use `json_response`
            └─ No → Enable screenshot to visually debug
                ↓
        Does the page require user interaction?
                ├─ Yes → Use `js_instructions`
                └─ No → Content may be static or blocked

```

## Parameter Reference

| Parameter | Description | Required When | Type |
| - | - | - | - |
| [js\_render](/fetch/features/js-rendering) | Enables JavaScript rendering in a headless browser | Site loads content via JS | Boolean |
| [wait](/fetch/features/wait) | Waits for a specified time in milliseconds before returning the response | Site loads slowly | Integer |
| [wait\_for](/fetch/features/wait-for) | Waits for a specific CSS selector to be present | Site depends on dynamic elements | String |
| [json\_response](/fetch/features/json-response) | Extracts data from XHR/Fetch calls loaded during page render | Site uses API calls for content | Boolean |
| [js\_instructions](/fetch/features/js-instructions) | Performs custom user interactions (clicks, scrolls, etc.) | Site requires interaction to reveal content | Array |
| [premium\_proxy](/fetch/features/premium-proxy) | Uses residential proxies for protected access to restricted content | Site blocks datacenter IPs or has bot detection | Boolean |

## Recommended Steps

### 1. Enable JavaScript Rendering

If the target website relies on JavaScript to display content, enable `js_render`. This ensures that Zenrows runs JavaScript in a headless browser to render the page fully.

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  }
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true"
  ```
</CodeGroup>

This parameter is required when using other advanced features such as `wait`, `wait_for`, `screenshot`, `json_response`, `outputs`, and others.

### 2. Use Wait Parameters

Some websites load content after a delay. Use the `wait` parameter to introduce a fixed delay or `wait_for` to wait for a specific CSS selector to appear.

#### Example with `wait`

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  	'wait': '15000', # Waits for 15 seconds
  }
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true&wait=15000"
  ```
</CodeGroup>

#### Example with `wait_for`

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  	'wait_for': '.content-loaded', # Waits for the specified CSS selector
  }
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true&wait_for=.content-loaded"
  ```
</CodeGroup>

<Tip>`wait_for` will cause the request to fail if the selector is not found. Use browser DevTools to find reliable CSS selectors on your target site.</Tip>

### 3. Use `json_response` for XHR-based Pages

If the data you're trying to scrape is loaded via JavaScript through AJAX calls, you can use the `json_response` parameter to access the response of those internal XHR requests.

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  	'wait_for': '.selector',
  	'json_response': 'true',
  }
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true&wait_for=.selector&json_response=true"
  ```
</CodeGroup>

<Tip>Combine `json_response` with `wait` or `wait_for` to ensure all background requests have completed before data is captured.</Tip>

### 4. Use Screenshot for Debugging

To understand what Zenrows actually "sees", enable the `screenshot` parameter. This can help determine whether content is missing due to rendering delays or protections.

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  	'wait_for': '.selector',
  	'screenshot': 'true',
  }   
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true&wait_for=.selector&screenshot=true"
  ```
</CodeGroup>

This feature provides a rendered image of the page after all scripts have run and delays have passed.

### 5. Use js\_instructions for Interactive Pages

If the page requires user interaction to load the content (like clicking a button, scrolling, or inputting text), use the `js_instructions` parameter. This allows you to simulate actions in the browser before the content is captured.

Example:

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  	'js_instructions': """[
          {"click":".selector"},
          {"wait":500},
          {"fill":[".input","value"]},
          {"wait_for":".slow_selector"}
      ]""",
  } 
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%252C%257B%2522fill%2522%253A%255B%2522.input%2522%252C%2522value%2522%255D%257D%252C%257B%2522wait_for%2522%253A%2522.slow_selector%2522%257D%255D"
  ```
</CodeGroup>

When using `js_instructions`, each instruction can have an optional `wait_for` step. If the selector is not found, Zenrows will attempt the next instruction after a short delay.

## Mixing Steps for Advanced Debugging

You can combine multiple steps for complex scenarios. For example:

* Use `js_instructions` to simulate user interactions, followed by `json_response` to capture XHR data.
* Enable `screenshot` alongside `wait_for` to visually confirm dynamic content.

Example:

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  	'js_instructions': """[
          {"click":".selector"},
          {"wait":500},
          {"fill":[".input","value"]},
          {"wait_for":".slow_selector"}
      ]""",
      'json_response': 'true',
  } 
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true&js_instructions=%255B%257B%2522click%2522%253A%2522.selector%2522%257D%252C%257B%2522wait%2522%253A500%257D%252C%257B%2522fill%2522%253A%255B%2522.input%2522%252C%2522value%2522%255D%257D%252C%257B%2522wait_for%2522%253A%2522.slow_selector%2522%257D%255D&json_response=true"
  ```
</CodeGroup>

## Summary Table

| Scenario | Recommended Parameters | Notes |
| - | - | - |
| Empty or partially loaded pages | `js_render=true` | Ensures JavaScript runs to render dynamic content |
| Content loads slowly | `wait=5000` or `wait_for=.selector` | Allows time for content to appear or specific element to load |
| Page requires user interaction | `js_instructions=[{"click":".selector"}...]` | Simulates user actions like dismissing modals or clicking buttons |
| Data comes from background requests | `json_response=true` + `wait` or `wait_for` | Captures XHR/Fetch data loaded after DOM is ready |
| Complex sites with strong protection | `premium_proxy=true` + `js_render=true` | Mimics real browser with rotating residential IPs |

## Still Stuck?

If you're still getting empty or partial responses after following the steps above:

1. Double-check your CSS selectors in DevTools
2. Use a screenshot to confirm what Zenrows is rendering
3. Reach out to support at [success@zenrows.com](mailto:success@zenrows.com) with:
   * Your full request URL.
   * Parameters used.
   * Expected vs actual response.
   * What you've already tested.
   * Any other useful information, such as specific errors or observations.

Our team will help you optimize your request for better results.


# IP Address Blocked or Connection Refused/Aborted
Source: https://docs.zenrows.com/fetch/troubleshooting/ip-address-blocked

Resolve IP address blocked and connection refused errors in Zenrows by adjusting rate limits, request patterns, and proxy configuration.

Zenrows has protective measures in place to safeguard the infrastructure from abusive behavior and malformed traffic. If too many faulty requests are made in a short period, our system will temporarily block your IP address to maintain service reliability and fairness for all users.

When a block is triggered, you may encounter one of the following errors:

* `BLK0001 IP Address Blocked`
* `Connection Refused/Aborted`
* `RemoteDisconnected('Remote end closed connection without response')`
* `ECONNRESET`

Understanding why your IP was blocked, and how to avoid it, is key to keeping your scraping processes stable and efficient.

**To resolve the issue, check the error code and error body from your previous request. These details will help you identify the cause of the block.**

## What Triggers an IP Block?

Faulty requests are the cause of an IP block. These include:

* Invalid or missing parameters
* Unauthorized or unverified access
* Requests to unsupported or blocked domains
* Too many concurrent requests beyond plan limits

Here are specific error types that can trigger a temporary IP block:

| **HTTP Code** | **Error Codes** | **Description** |
| - | - | - |
| 400 | REQS001, REQS002, REQS004, RESP004, REQS006, RESP008 | Malformed or incomplete requests |
| 401 | AUTH001, AUTH002, AUTH003 | Missing or invalid API key |
| 403 | AUTH009 | Unverified account access |
| 405 | REQS005 | Unsupported HTTP method |
| 429 | AUTH006, AUTH008 | Concurrency or rate limit exceeded |

## Common Errors and How to Resolve Them

Below are the most common reasons for blocks and their solutions:

### 400 Bad Request

#### REQS001 Requests to This Domain Are Forbidden

Zenrows proactively blocks scraping attempts to certain domains to comply with legal, ethical, or technical guidelines.

**Solution:** <br />Check the [Forbidden Domains Documentation Page](/forbidden-sites) for more details and avoid targeting these domains.

<Tip>If you believe a domain should be whitelisted, feel free to contact support at [success@zenrows.com](mailto:success@zenrows.com)</Tip>

#### REQS002 Request Requirements Unsatisfied

Some domains require the `js_render` and/or `premium_proxy` features to be active to bypass protections such as JavaScript-based content or regional blocks.

**Solution:** <br />Check the error body for details on which feature is needed and retry your request with them applied:

<CodeGroup>
  ```python Python theme={"dark"}
  params = {
      'url': url,
      'apikey': apikey,
  	'js_render': 'true',
  	'premium_proxy': 'true',
  }
  ```

  ```javascript Node.js theme={"dark"}
  params: {
      'url': url,
      'apikey': apikey,
      'js_render': 'true',
      'premium_proxy': 'true',
  },
  ```

  ```bash cURL theme={"dark"}
  curl "<...>&js_render=true&premium_proxy=true"
  ```
</CodeGroup>

#### REQS004 Invalid Params Provided

One or more parameters in your request are unrecognized or incorrectly formatted.

**Solution:**

* Double-check all query parameters
* Ensure URLs are properly encoded
* Refer to the [API Reference](/fetch/api-reference#parameter-overview) for supported parameters

#### RESP004 CSS Extractor Parameter is Not Valid

The `css_extractor` parameter is malformed.

**Solution:**

* Review your selector and consult the [CSS Selectors Documentation](/fetch/features/css-extractor).

#### REQS006 Invalid CAPTCHA Solver Key

An incorrect or missing key for 2Captcha was configured on the [Integration Page](https://app.zenrows.com/settings/integrations).

**Solution:** <br />Before using the `solve_captcha` JavaScript Instructions verify that the correct CAPTCHA solver key is entered in your account.

<Warning>Leaving the 2Captcha API key empty on the integration page and attempting to use the `solve_captcha` instruction will also result in errors and an IP block.</Warning>

#### RESP008 Non-Retryable Error not Related to Zenrows

This error indicates a domain-specific issue (e.g., configuration or certificate problems) that Zenrows can't fix by changing the request's parameters.

**Solution:**

* Review the full error message; these errors usually cannot be resolved by modifying the API request
* If the message reports a certificate problem but the certificate checks out fine when you open the site directly, the condition may be intermittent on the target's side. Zenrows doesn't retry RESP008 automatically, but retrying once or twice, a few minutes apart, can succeed. Don't retry it in a loop, because repeated 400 errors can trigger an IP block
* See [RESP008 in the API Error Codes reference](/api-error-codes#RESP008) for details

### 401 Unauthorized Errors

#### AUTH001 / AUTH002 / AUTH003 – API Key Issues

Common when:

* API key is missing
* API key is invalid or deleted

**Solution:** <br />Retrieve or regenerate your API key from the [Settings Page](https://app.zenrows.com/settings/api-keys) and update your requests accordingly.

### 403 Forbidden

#### AUTH009 User is Not Verified

Your account email must be verified to access the API.

**Solution:** <br />Check your inbox for a verification link. Didn't receive it? Contact [success@zenrows.com](mailto:success@zenrows.com).

### 405 Method Not Allowed

#### REQS005 Method Not Allowed

Zenrows only supports `GET`, `POST`, and `PUT`.

**Solution:** <br />Avoid using unsupported methods like `DELETE` or `PATCH`.

<Tip>If you're unsure which method to use, default to `GET`.</Tip>

### 429 Too Many Requests

#### AUTH006 Concurrency Exceeded

You're making more simultaneous requests than your plan allows.

**Solution:**

* Implement a queue system in your code to limit concurrent requests.
* Monitor the `Concurrency-Remaining` response header to adjust your request rate dynamically.

<Warning>Cancelling requests (e.g., `CTX0001 Context Cancelled`) does not release concurrency immediately. Zenrows requests can take up to 3 minutes to complete, and cancelling them prematurely may still impact your concurrency limits, potentially resulting in blocks.</Warning>

For more details, see the [Concurrency Documentation](/fetch/features/concurrency)

#### AUTH008 Rate Limit Exceeded

Similar to the concurrency error, this occurs when you exceed your rate limit. Monitor the `Concurrency-Remaining` response header and adjust your request rate to avoid blocks and apply the suggested solutions.

## Best Practices to Avoid IP Blocks

1. Validate your requests before sending them
2. Use error handling to detect and stop faulty retries
3. Limit concurrency based on your subscription tier
4. Monitor the `Concurrency-Remaining` response header
5. Use `js_render` and `premium_proxy` when necessary
6. Avoid retrying on 400 and 401 errors, as they are not recoverable by resending the same request. The exception is a RESP008 certificate error that turns out to be intermittent on the target's side (see RESP008 above)
7. Remove any timeout from your script to avoid cancelling your request prematurely causing issues on your concurrency

## Frequently Asked Questions (FAQ)

<Accordion title="How long does the IP block last?">
  IP blocks expire after 5 minutes.
</Accordion>

<Accordion title="Can I get my IP unblocked faster?">
  No. Blocks are automatic and expire on their own. If your IP is blocked frequently, review your request patterns and error codes.
</Accordion>

<Accordion title="Will using premium_proxy prevent IP blocks?">
  `premium_proxy` helps with access to protected websites but won't prevent IP blocks due to malformed or unauthorized requests. Always validate your inputs.
</Accordion>

<Accordion title="What should I do if I keep getting Connection Refused?">
  This usually indicates your IP is currently blocked. Wait 5 minutes, then retry after fixing any errors in your previous request.
</Accordion>

<Accordion title="What happens if I keep sending requests when my IP is blocked?">
  Nothing. The requests will be ignored.
</Accordion>


# Troubleshooting Response Size Too Large Errors
Source: https://docs.zenrows.com/fetch/troubleshooting/response-too-large

Resolve 413 Content Too Large errors in Zenrows by filtering HTML output, using CSS selectors, and adjusting response size parameters.

Zenrows enforces a maximum response size per request, which varies depending on your subscription plan. If the response exceeds your plan's limit (whether it's HTML content, JSON data, or a screenshot), you will receive a `413 Content Too Large` error and the response will not be delivered.

<Info>You can find detailed limits for each plan in the [Pricing Documentation](/first-steps/pricing#response-size-limits).</Info>

This error usually occurs when:

* You target pages with large amounts of HTML or dynamic content
* You use `json_response=true`, which includes all background network requests
* You capture high-resolution or full-page screenshots

Below are ways to reduce the response size and troubleshoot this issue.

## 1. **Check If the Response Exceeds Your Plan's Limit**

Start by confirming whether the requested content goes over your plan's response size allowance. This includes:

* HTML from the page
* JSON output (if `json_response=true` is used)
* Screenshots

Try the same request without optional parameters like `json_response=true` to compare the difference in response size.

## 2. **Use Targeted Extraction**

Instead of fetching the entire HTML, extract only what you need using:

* **CSS selectors** or **XPath** to select specific elements
* The **`outputs`** parameter to return items like emails, phone numbers, links, or headings

<Info>Learn more in the [CSS Selectors Documentation](/fetch/features/css-extractor) and [Outputs Documentation](/fetch/features/output-filters).</Info>

## 3. **Optimize Screenshot Requests**

Large screenshots can exceed the size limit. To avoid this:

* Use a CSS selector (like `screenshot_selector`) to capture a specific part of the page
* Avoid full-page screenshots unless strictly necessary
* Reduce file size with `screenshot_format` (for example, JPEG) and `screenshot_quality`

<Info>Learn more in the [Screenshot Feature Documentation](/fetch/features/screenshot).</Info>

## 4. **Avoid `json_response=true` if Not Needed**

This parameter includes all network activity such as AJAX and XHR requests, which can significantly increase the response size.

Remove it if you do not need this data. Instead, use targeted selectors to extract what matters.

## 5. **Block Unnecessary Resources**

You can use the `block_resources` parameter to prevent loading heavy assets like images, fonts, or videos. This reduces the total page weight.

<Warning>
  Zenrows blocks some resource types like stylesheets and images by default to improve performance.\
  Only override this behavior when necessary.\
  To disable all blocking, set `block_resources=none`.

  Learn more in the [Block Resources Documentation](/fetch/features/block-resources).
</Warning>

## Still Receiving the Error?

If the page is too large by nature and optimizations do not resolve it:

* Make multiple requests by dividing the page into sections
* Upgrade your plan for higher response size limits
* Contact us at [success@zenrows.com](mailto:success@zenrows.com) for support

A `413 Content Too Large` error means the request was successful but the final output could not be returned due to size restrictions. Use the suggestions above to avoid this limit and ensure efficient scraping.


# Basic Troubleshooting Guide
Source: https://docs.zenrows.com/fetch/troubleshooting/troubleshooting-guide

Solve common Zenrows web scraping problems like dynamic content loading, geo-restricted pages, session handling, and timeout errors.

This guide helps you resolve common web scraping issues by organizing solutions around specific use cases. Whether you're dealing with dynamic content, geo-restrictions, or session management, these targeted troubleshooting techniques will get you back on track quickly.

## Dynamic Content & Timing Issues

When scraping JavaScript-heavy sites, SPAs, or pages with delayed content loading, timing becomes critical.

### Multiple Wait-For Selectors

The `wait_for` parameter handles dynamic content, but edge cases like empty results require careful selector planning:

<CodeGroup>
  ```javascript Node.js theme={"dark"}
  const response = await axios.get('https://api.zenrows.com/v1/', {
    params: {
      apikey: YOUR_ZENROWS_API_KEY,
      url: TARGET_URL,
      js_render: true,
      // Wait for either product listings OR a "no results" message
      wait_for: '.product-listing, .no-results-message'
    }
  });
  ```

  ```python Python theme={"dark"}
  response = requests.get(
      'https://api.zenrows.com/v1/',
      params={
          'apikey': YOUR_ZENROWS_API_KEY,
          'url': TARGET_URL,
          'js_render': True,
          # Wait for either product listings OR a "no results" message
          'wait_for': '.product-listing, .no-results-message'
      }
  )
  ```
</CodeGroup>

This ensures your request completes when the expected content appears OR when a "no results" indicator is displayed, avoiding unnecessary timeouts.

### Testing CSS Selectors Before Scraping

Before using CSS selectors in your scraping code, test them directly in the browser console:

1. Open the target page in your browser
2. Open Developer Tools (F12 or ⌘+⌥+I)
3. In the Console tab, test your selector:

```javascript theme={"dark"}
document.querySelectorAll('.your-selector')
```

4. Verify it returns the expected elements across different page states

## Anti-Bot Protection & Access Issues

When sites block your requests or return unexpected errors, these debugging techniques help identify the root cause.

### Error 422: Unprocessable Entity

This typically means anti-bot protection is blocking access. Solutions include:

1. **Enable JavaScript rendering**: `js_render: true` to mimic real browser behavior
2. **Use premium proxies**: Set `premium_proxy: true` to route traffic through residential IPs
3. **Add a referer header**: Set `custom_headers=true` and send `Referer: https://www.google.com/` to mimic typical browser behavior
4. **Wait for content to load**: Use `wait_for` or `wait` parameters for slower sites

If adding parameters makes it worse, see [Still failing after stacking parameters](#still-failing-after-stacking-parameters-retest-with-the-minimal-config).

<Note>
  If your success rate suddenly drops, it's often due to new anti-bot challenges or region-specific blocks. Enabling `js_render`, switching proxy regions, or using `premium_proxy: true` typically resolves this.
</Note>

#### Still failing after stacking parameters? Retest with the minimal config

Extra parameters change how the request behaves, and on some protected sites they can lower your success rate instead of raising it. The `referer` value is a header, so it needs `custom_headers=true`.

If you still get 422 after adding `wait`, `wait_for`, or a custom `referer` header on top of `js_render=true` and `premium_proxy=true`, follow these steps:

1. Remove the extra parameters and retest with only `js_render=true` and `premium_proxy=true`.
2. Add the extras back one at a time, and keep only the ones that help.
3. Keep `proxy_country` if the site is region-restricted. See [Error 422 with Geo-Restrictions](#error-422-with-geo-restrictions).

The minimal request looks like this:

<CodeGroup>
  ```javascript Node.js theme={"dark"}
  const response = await axios.get('https://api.zenrows.com/v1/', {
    params: {
      apikey: YOUR_ZENROWS_API_KEY,
      url: TARGET_URL,
      js_render: true,
      premium_proxy: true
      // No wait, wait_for, or custom headers yet
    }
  });
  ```

  ```python Python theme={"dark"}
  response = requests.get(
      'https://api.zenrows.com/v1/',
      params={
          'apikey': YOUR_ZENROWS_API_KEY,
          'url': TARGET_URL,
          'js_render': True,
          'premium_proxy': True
          # No wait, wait_for, or custom headers yet
      }
  )
  ```
</CodeGroup>

If the minimal request still returns 422, contact support with the [`X-Request-Id` response header](/fetch/api-reference) so the team can check what the target site is doing.

### Retrieving Original Status Codes

When a target website returns an error, Zenrows handles it gracefully by default. Use `original_status` to see what's actually happening:

<CodeGroup>
  ```javascript Node.js theme={"dark"}
  const response = await axios.get('https://api.zenrows.com/v1/', {
    params: {
      apikey: YOUR_ZENROWS_API_KEY,
      url: TARGET_URL,
      original_status: true  // Return original status code
    }
  });
  console.log('Original status code:', response.status);
  ```

  ```python Python theme={"dark"}
  response = requests.get(
      'https://api.zenrows.com/v1/',
      params={
          'apikey': YOUR_ZENROWS_API_KEY,
          'url': TARGET_URL,
          'original_status': True  # Return original status code
      }
  )

  print(f'Original status code: {response.status_code}')
  ```
</CodeGroup>

This helps you identify if the target site is returning errors like `403` Forbidden or `429` Too Many Requests.

### Allowing Specific Status Codes

Examine error pages or handle specific status codes by allowing them to pass as successful responses:

<CodeGroup>
  ```javascript Node.js theme={"dark"}
  const response = await axios.get('https://api.zenrows.com/v1/', {
    params: {
      apikey: YOUR_ZENROWS_API_KEY,
      url: TARGET_URL,
      // Allow 404 and 500 responses to be treated as success
      allowed_status_codes: '404,500'
    }
  });
  ```

  ```python Python theme={"dark"}
  response = requests.get(
      'https://api.zenrows.com/v1/',
      params={
          'apikey': YOUR_ZENROWS_API_KEY,
          'url': TARGET_URL,
          # Allow 404 and 500 responses to be treated as success
          'allowed_status_codes': '404,500'
      }
  )
  ```
</CodeGroup>

## Region-Restricted Sites

Geographic restrictions require specific approaches to access localized content.

### Error 422 with Geo-Restrictions

When content is region-locked, adjust your geographic approach:

1. **Specify target country**: Use `proxy_country: 'us'` or another accessible region
2. **Enable premium proxies**: Set `premium_proxy: true` for residential IPs
3. **Test different regions**: Some content may be available in multiple countries

## Performance & Resource Management

Optimize your scraping efficiency and manage API limits effectively.

### Error 429: Too Many Requests

This indicates you've reached your concurrency limit. Solutions include:

1. **Request queue implementation**: Space out requests using a queue
2. **Concurrency monitoring**: Check the `concurrency-remaining` response header
3. **Backoff strategy**: Increase delays when 429 errors occur
4. **Complete requests fully**: Don't cancel requests - let them finish to release concurrency slots

<Note>
  If you begin to notice a lower-than-usual success rate, monitor your concurrency usage and ensure requests are being spaced out appropriately. Sudden spikes can trigger errors due to the concurrency limit being reached.
</Note>

### Error 413: Content Too Large

When responses exceed your plan's size limits:

1. **Use CSS selectors**: Extract only needed data with `css_extractor`
2. **Disable JSON response**: Remove `json_response` if not needed to reduce payload size
3. **Implement pagination**: Split large data sets into multiple requests

## Session-Dependent Sites

For multi-step workflows requiring consistent session state.

### Session management issues

When session state needs to persist across multiple requests:

1. **Use consistent session IDs**: Apply the same `session_id` across related requests
2. **Maintain custom headers**: Include necessary cookies and authentication headers
3. **Test session flow**: Verify each step maintains the required state

## Getting Support

If you're still encountering issues after applying these targeted solutions, Zenrows' support team is ready to help.

**Prepare Request Details:**

* Target URL and all parameters used
* Request ID from the `X-Request-Id` response header
* Expected vs. actual results

**Contact Support:**

Use live chat on the Zenrows website or [email](mailto:success@zenrows.com).


# Proxy and Geolocation Troubleshooting
Source: https://docs.zenrows.com/fetch/troubleshooting/troubleshooting-premium-proxies

Fix Zenrows Premium Proxy issues such as geolocation mismatches, residential IP blocks, and anti-bot bypass failures with proven steps.

Premium Proxies (residential IPs) help bypass sophisticated anti-bot systems. However, websites often employ additional layers of protection based on geolocation, JavaScript checks, and behavior analysis. If you're running into blocks, delays, or inconsistent content, this guide walks you through common problems and proven solutions.

## Troubleshooting Issues While using Premium Proxies

### Common Symptoms

* HTTP 403 (Forbidden) errors
* CAPTCHA or bot protection challenges
* Messages like “Access Denied” or “Your IP has been blocked”
* Page not loading

### Key Solutions:

#### 1. Enable JavaScript Rendering

Many modern websites require JavaScript to load content and check browser fingerprints as part of their security. Enabling JavaScript rendering can help bypass these checks.

```python theme={"dark"}
params = {
    "url": target_url,
    "apikey": "YOUR_ZENROWS_API_KEY",
    "premium_proxy": "true",
    "js_render": "true"  # Enable JavaScript rendering
}
```

<Info>You can find more information about the `js_render` parameter in the [JavaScript Rendering documentation](/fetch/features/js-rendering).</Info>

#### 2. Implement a Wait Strategy

Websites may detect bots based on speed. Introduce artificial delay with the `wait` parameter or wait for specific elements using `wait_for`.

```python theme={"dark"}
params = {
    "url": target_url,
    "apikey": "YOUR_ZENROWS_API_KEY",
    "premium_proxy": "true",
    "js_render": "true",
    "wait": "5000"  # Wait 5 seconds (5000ms)
}
```

<Info>See the [Wait for Selector documentation](/fetch/features/wait-for) for more information.</Info>

#### 3. Try Different Countries

Websites may block access differently based on your location. Switching to a proxy from another country can help.

```python theme={"dark"}
# Try multiple countries if one is getting blocked
countries = ["us", "gb", "ca", "de", "fr"]

for country in countries:
    params = {
        "url": target_url,
        "apikey": "YOUR_ZENROWS_API_KEY",
        "premium_proxy": "true",
        "js_render": "true",
        "proxy_country": country
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    if response.status_code == 200:
        print(f"Success with country: {country}")
        break
    else:
        print(f"Failed with country: {country}, status: {response.status_code}")
```

<Info>You can find more information about the `proxy_country` parameter in the [Proxy Country documentation](/fetch/features/proxy-country).</Info>

#### 4. Use Custom Referer Header

Some websites are easier to access when a custom referrer header is set. It can make your requests appear more legitimate.

```python theme={"dark"}
headers = {
    "Referer": "https://www.google.com/"
}

params = {
    "url": target_url,
    "apikey": "YOUR_ZENROWS_API_KEY",
    "premium_proxy": "true",
    "js_render": "true",
    "custom_headers": "true"
}
```

<Info>You can find more information about the `custom_headers` parameter in the [Custom Headers documentation](/fetch/features/headers).</Info>

## Troubleshooting Geolocation-Based Errors

### Still Blocked After Setting a Proxy Country?

Even when using Premium Proxies, access may be denied due to region-specific restrictions. Try:

1. Rotating countries using the `proxy_country` parameter
2. Combining it with `js_render=true` to simulate a full browser session
3. Waiting for dynamic content with `wait` or `wait_for`

### Requests Are Too Slow?

Long response times can stem from:

* Routing through distant proxy locations
* Heavy JavaScript rendering
* Waiting for elements that don't exist

**What to do:**

* Try a geographically closer proxy
* Use `wait_for` only for elements you're sure exist
* Check the page structure with DevTools to validate selectors

### Inconsistent Results Based on Region?

Some sites display different content or layouts depending on the IP's location. This can cause selectors to break or data to be missing.

**Solution:**

Inspect the HTML structure from different proxy countries and update your `css_extractor` or `wait_for` selectors accordingly.

### Content Completely Geo-Blocked?

If you're seeing region blocks despite using Premium Proxies:

1. Rotate through a wider range of countries
2. Test against regions that typically allow access (e.g., US or UK)

## When to Contact Support

If you've tried all the troubleshooting steps and still encounter issues, it might be time to contact Zenrows support. Here's what to include in your support ticket:

1. **Target URL** - The exact URL you're trying to access
2. **Configuration Used** - All parameters you've tried
3. **Error Messages** - Full error details, including status codes
4. **Response Headers** - If available, include response headers
5. **Success Patterns** - Note any patterns in successful vs. failed requests
6. **Business Use Case** - Explain what you're trying to accomplish

Contact [success@zenrows.com](mailto:success@zenrows.com) or use the live chat support available on the Zenrows website with these details to get expert assistance with your specific case.


# API Keys and Credit Caps
Source: https://docs.zenrows.com/first-steps/api-key-credit-caps

Use multiple Zenrows API keys and set per-key credit caps (per billing period, daily, weekly, monthly) to limit how many credits each key can use.

Your account can have up to 10 API keys, for example one per project, environment, or customer. Each key can have its own credit caps, so one workload can't use up the credits another one needs. Manage keys and caps on the [API keys page](https://app.zenrows.com/settings/api-keys).

## Set or change a cap

1. Open the [API keys page](https://app.zenrows.com/settings/api-keys) (**Settings** > **API keys**).
2. Open the menu of the key you want to limit and select **Credit caps**.
3. Click **Add a cap**, choose the **Window**, enter the cap in credits, and click **Add cap**.

To change or remove a cap, open that cap's actions menu in the same window.

## How caps work

A key can have any of these caps, alone or combined:

| Cap | Window |
| - | - |
| **Per billing period** | Your current billing period |
| **Daily** | Calendar day, UTC |
| **Weekly** | Monday to Sunday, UTC |
| **Monthly** | Calendar month, UTC |

* Keys are **uncapped by default**, including your main key.
* A key with several caps stops when it reaches **any** of them.
* All credits a key uses count toward its caps, overage included.
* [Browser Sessions](/browser-sessions/introduction) (formerly, Scraping Browser) count too, both their time and their data.
* Caps limit usage. They don't reserve credits and don't need to add up to your plan.
* Upgrades, downgrades and top-ups don't change your caps.

## When a key reaches its cap

The key stops working, and your other keys keep working. How the refusal looks depends on the product:

* **Fetch (formerly, Universal Scraper API) and [Extract](/extract/introduction) requests** return HTTP `402` with error code [AUTH014](/api-error-codes#AUTH014).
* **Batch requests** return HTTP `402` with the code [`api_key_cap_reached`](/api-error-codes#api_key_cap_reached). See [Batch jobs](#batch-jobs).
* **New [Browser Sessions](/browser-sessions/introduction)** are refused when they start, with error code [AUTH014](/api-error-codes#AUTH014).

The key works again when its cap window resets, or shortly after you raise or remove the cap.

### When the cap is checked

A cap is checked when something **starts**: each API request, and each [Browser Session](/browser-sessions/introduction). If the key is still under its cap at that moment, the request or session goes ahead.

What's already running isn't cut off:

* A request in progress when the cap is reached completes.
* A Browser Session that's already open keeps running until it ends. Everything it uses still counts toward the cap, and the next session started with that key is refused.

So usage can go over the cap by what was already running. For normal traffic that's a small amount. [Batch](/batch/introduction) runs many tasks in parallel, so a Batch run can go further over before it stops. A single session that moves a very large amount of data can take a key well past its cap before the cap stops the next one, so size caps with your heaviest sessions in mind.

If the usage of a cap can't be read for a moment, that cap doesn't block the key until its usage is available again. Any other caps on the key still apply.

<Note>
  AUTH014 means one key reached its own cap. [AUTH004](/api-error-codes#AUTH004) means your account has no credits left, which affects every key.
</Note>

<Warning>
  A key cap doesn't guarantee zero overage charges. Your account can run out of included credits before a key reaches its cap.
</Warning>

## Batch jobs

[Batch](/batch/introduction) jobs use the caps of the key they run with. A capped key can't start or add to a job, a run that reaches a cap stops, and scheduled runs are checked when they start. See [API key credit caps and Batch](/batch/developer-guide-restapi#api-key-credit-caps-and-batch).

## Tracking remaining credits

Responses from a key that has at least one cap include the `X-Key-Credits-Remaining` header: the lowest remaining credits across that key's caps. Uncapped keys don't get this header. If the usage of a cap can't be read for a moment, the header leaves that cap out, and it isn't sent at all when none of the key's caps can be read.

```
X-Key-Credits-Remaining: 12500
```

Read it to slow down or switch keys before a cap stops your requests. See all [response headers](/fetch/api-reference#response-headers).

## Alerts

Your account's [notification recipients](/first-steps/manage-notifications#api-key-credit-caps) get an email and an in-app notification when a key reaches 80% and 100% of a cap. They also get an email when a cap is set, changed, or removed. Turn these alerts on or off with the **API key credit caps** notification setting.

## Residential Proxies (Legacy)

Key caps don't apply to Residential Proxies. Residential proxy credentials keep their own limits.


# Your Billing page
Source: https://docs.zenrows.com/first-steps/billing

Manage your Zenrows subscription, add top-ups, and view and download past payments and invoices from the Billing page in your dashboard.

The **Billing** page is where you manage everything related to paying for Zenrows: your subscription, extra credits (top-ups), and your record of past payments and invoices. Open it from the **Billing** entry in the dashboard sidebar, or go directly to your <a href="https://app.zenrows.com/billing">Billing</a> page.

## Manage your subscription

From the Billing page you can change your plan at any time:

* **Upgrade** to a higher rung or tier for more monthly credits, concurrency, and support.
* **Downgrade** to a lower plan.
* **Cancel** your subscription.

Upgrades take effect immediately, while downgrades and cancellations take effect at the end of your current billing cycle. For exactly what happens to your credits and limits in each case, see [Managing your subscription](https://docs.zenrows.com/first-steps/pricing#managing-your-subscription) in the pricing guide.

## Top-ups

If you need more credits before your next monthly refresh, you can add **top-ups** (extra credits) from the Billing page without changing your plan. Top-up credits draw from the same shared balance as your plan credits.

<Tip>
  For top-up pack sizes and prices, per-period limits, and how unused top-up credits roll over, see [Managing your subscription](https://docs.zenrows.com/first-steps/pricing#managing-your-subscription) in the pricing guide.
</Tip>

## Payments & invoices

The **Payments & invoices** tab keeps a record of your past payments. Each invoice appears as its own row, and you can download any of them individually using the download control on that row.

<Note>
  Past invoices cannot be edited. To make sure future invoices carry the correct details, keep your profile information up to date in your [Account Settings](https://app.zenrows.com/settings/profile) and your payment methods on the Billing page. Changes apply to your next invoice onward.
</Note>


# Changing your account email address
Source: https://docs.zenrows.com/first-steps/change-email

Update your Zenrows account email address from your account settings, with steps for Google, GitHub, and password-based sign-in users.

You can update your account email address from your [Account Settings](https://app.zenrows.com/settings/profile). The process requires you to confirm your new address before the change takes effect.

<Warning>
  If you signed up with Google or GitHub, changing your email will unlink your account from that provider. You will need to set a password and sign in with email and password going forward. See [Google or GitHub users](#google-or-github-users) for details.
</Warning>

## How to change your email

1. Go to your [Account Settings](https://app.zenrows.com/settings/profile).
2. Click the edit icon next to your current email address.
3. Enter your new email address and click **Send verification email**.
4. Check your new email inbox for a confirmation link.
5. Click the link within **24 hours** to confirm the change.

Once confirmed, your account email is updated immediately.

After the change, you will receive a notification at both addresses:

* **New email**: a confirmation that your account is now linked to the new address.
* **Old email**: a notice that the change was made, with an option to undo it within 24 hours.

## Email address requirements

Your new email address must meet the following requirements:

* Valid email format, no longer than 255 characters.
* Not already in use by another Zenrows account.
* Different from your current email address.
* Not a reserved system address.

If your account belongs to an organisation, your new email must use the **same domain** as your current one (for example, `you@company.com` can only change to another `@company.com` address).

<Note>
  You can request an email change up to **3 times per hour**. If you reach this limit, wait before trying again.
</Note>

## What happens when you click the confirmation link

When you click the link in your confirmation email, a few checks are made before the change is applied:

* **Link expiry**: the link is valid for **24 hours**. If it expires, start the process again from your [Account Settings](https://app.zenrows.com/settings/profile).
* **Email taken**: if someone else registered with your new email address between your request and clicking the link, the change will be blocked. You will need to choose a different address and start over.
* **Account disabled**: if your account has been disabled in the meantime, the email change will be cancelled and you will not be able to proceed.

## Google or GitHub users

If you signed up using Google or GitHub, you can still change your email address. However, this will disconnect your account from that provider. After the change:

* You will sign in using your new email address and a password.
* You will be prompted to set a password if you don't already have one.
* You will no longer be able to sign in with Google or GitHub.

**If you have no password set**, clicking the confirmation link will log you in and hold the email change. You must set a password before the new address is finalised.

## Undoing an email change

After your email is updated, an undo link is sent to your **old** email address. You can use this link within **24 hours** to revert to your previous email address.

## Frequently Asked Questions (FAQ)

<Accordion title="What happens if my confirmation link expires?">
  Confirmation links are valid for **24 hours**. If yours has expired, go back to your [Account Settings](https://app.zenrows.com/settings/profile) and start the process again. A new confirmation email will be sent to your new address.
</Accordion>

<Accordion title="What if someone else registers my new email before I confirm?">
  If another user registers the email address you requested between the time you submitted the change and the time you clicked the confirmation link, the change will be blocked. You will need to choose a different address and start the process again.
</Accordion>

<Accordion title="I signed up with Google or GitHub. What changes after I update my email?">
  Changing your email disconnects your account from your Google or GitHub login. After the change, you will sign in using your new email address and a password. If you do not have a password set yet, you will be prompted to create one before the change is finalised. You will no longer be able to sign in with Google or GitHub.
</Accordion>

<Accordion title="Can I undo an email change?">
  Yes. After your email is updated, an undo link is sent to your **old** email address. You can use this link within **24 hours** to revert to your previous address. After 24 hours, the undo option is no longer available.
</Accordion>

<Accordion title="What if my account gets disabled after I request the change?">
  If your account is disabled between the time you request the change and when you click the confirmation link, the email change will be cancelled. You will not be able to proceed until your account is re-enabled. Contact [support@zenrows.com](mailto:support@zenrows.com) if you need help with your account status.
</Accordion>


# Deleting your Zenrows account
Source: https://docs.zenrows.com/first-steps/delete-account

Permanently delete your Zenrows account and data from your account settings, including what happens to your subscription, API keys, and data.

You can permanently delete your Zenrows account from your [Account Settings](https://app.zenrows.com/settings/profile). This removes your account and data from our systems.

<Warning>
  Deleting your account is immediate and cannot be undone. Once confirmed, you cannot restore your account or recover your data.
</Warning>

## How to delete your account

1. Go to your [Account Settings](https://app.zenrows.com/settings/profile).
2. Find the **Delete account** option.
3. Confirm in the dialog that appears.

Your account is deleted immediately, and you will receive a confirmation email.

## What happens when you delete your account

* Your API keys are disabled immediately, and any in-flight requests using them will fail.
* Any active subscription is canceled as part of the deletion.
* Your account data is removed from our systems.
* You will no longer receive emails from Zenrows, aside from the deletion confirmation.

<Note>
  If you are on a paid plan and want a refund before canceling, contact [support@zenrows.com](mailto:support@zenrows.com) first. Deleting your account cancels your subscription immediately and a completed deletion cannot be reversed to process a refund afterwards.
</Note>

## Frequently Asked Questions (FAQ)

<Accordion title="Can I undo an account deletion?">
  No. Account deletion is immediate and permanent. There is no way to restore your account or recover your data once it is deleted.
</Accordion>

<Accordion title="What happens to my active subscription?">
  Your subscription is canceled as part of the deletion. If you want a refund, contact [support@zenrows.com](mailto:support@zenrows.com) before deleting your account.
</Accordion>

<Accordion title="What happens to my API keys?">
  Your API keys are disabled immediately when your account is deleted. Any requests made with them will fail from that point on.
</Accordion>

<Accordion title="Will I get a confirmation that my account was deleted?">
  Yes. You will receive a confirmation email once your account has been deleted.
</Accordion>

<Accordion title="Can support delete my account for me?">
  Account deletion is a self-service action from your [Account Settings](https://app.zenrows.com/settings/profile). Contact [support@zenrows.com](mailto:support@zenrows.com) if you run into trouble with the process.
</Accordion>


# Zenrows General FAQ: Billing, Features, and Limits
Source: https://docs.zenrows.com/first-steps/faq

Find answers to frequently asked questions about Zenrows web scraping, including billing cycles, supported features, and usage limits.

<Accordion title="Does Usage Carry Over Months?">
  On paid plans (Build through Scale), yes: unused credits roll over for 1 additional billing period before they expire. The Free plan doesn't roll over; credits reset at the start of each month. Enterprise rollover terms are negotiated. Separately, if you upgrade your plan mid-cycle, any unused credits (including top-ups) carry over as bonus credits on top of your new plan's allowance. See [Upgrading Your Plan](/first-steps/pricing#upgrading-your-plan).
</Accordion>

<Accordion title="Can I Scrape Using Browser Extensions?">
  Zenrows does not offer a browser extension, and our products are designed to work via API requests, proxies, and automated browsers, not through browser extensions.

  ## Why Doesn't Zenrows Offer a Browser Extension?

  Browser extensions have significant limitations when it comes to web scraping:

  1. **Restricted Execution** – Extensions run in a browser's sandboxed environment, limiting their ability to bypass antibot protections.
  2. **Scalability Issues** – A browser extension would require manual interaction, making it impractical for large-scale or automated scraping.
  3. **Limited Customization** – Unlike our API and Browser Sessions, extensions lack the flexibility to integrate advanced scraping techniques, such as headless browsing, fingerprint evasion, and CAPTCHA handling.

  ## How to Scrape Without a Browser Extension

  Instead of using an extension, we recommend using one of our API-based solutions, which are optimized for web scraping at scale:

  ### Fetch

  * Designed for flexible, automated scraping with built-in antibot bypassing.
  * Supports JavaScript rendering, Premium Proxies, and CAPTCHA handling.
  * Ideal for scraping protected or dynamic pages.

  ### Browser Sessions

  * A headless browsing solution that allows full control over web automation.
  * Ideal for cases where manual browser behavior replication is needed.

  ### Residential Proxies

  * Provides rotating IPs to increase anonymity.
  * Best for handling IP-based blocks but requires custom browser automation for full scraping functionality.

  If you're looking for automated and scalable web scraping, our API solutions are the best fit. Let us know if you need help choosing the right approach! 🚀
</Accordion>

<Accordion title="Can I Scrape Data From a Mobile App?">
  Fetch, Extract, and Batch request HTTP(S) URLs, so they can reach an endpoint that a mobile app calls. Browser Sessions connects over a WebSocket and drives a browser, so it works with web pages, not app traffic. Results depend on the endpoint: some app APIs need signed headers or tokens that only the app can produce.

  ### Website or app endpoints

  If the same data is also on the website, such as ticket availability, prices, or listings, the website is usually the simplest target. Use [Premium Proxies](/fetch/features/premium-proxy) and [JS Render](/fetch/features/js-rendering) with Fetch for a protected page.

  If you target an app endpoint, you need to supply any headers, tokens, or session values the app generates. You can send them with [Custom Headers](/fetch/features/headers).

  ### No mobile or carrier IPs

  Fetch uses datacenter IPs by default, and [Premium Proxies](/fetch/features/premium-proxy) (`premium_proxy=true`) route requests through residential IPs. Standalone [Residential Proxies](/legacy/residential-proxies/introduction) are residential too. None of them are mobile or carrier network IPs.

  ### Login, authentication, and terms of service

  Data behind a login wall or restricted by a site's terms of service may require explicit authorization. See [Legal Considerations](/first-steps/getting-started-guide#legal-considerations).
</Accordion>

<Accordion title="Can I See the XHR / AJAX / Fetch Requests that Browsers Perform?">
  If you need to capture dynamic content loaded via AJAX requests, Zenrows offers different approaches depending on the product you're using. Some products provide built-in JSON responses, while others require custom configurations to extract network requests.

  More and more websites load content dynamically, meaning data is fetched via XHR, AJAX, or Fetch requests instead of being included in the initial HTML. Besides waiting for the content to load, you might want to capture these network requests, similar to how they appear in the Network tab in DevTools.

  ## How Each Zenrows Product Handles XHR / AJAX / Fetch Requests

  ### Fetch

  * The JSON Response feature (`json_response=true`) captures AJAX requests automatically.
  * Returns a structured JSON object containing two fields:
    * HTML – The final rendered page source.
    * XHR – An array of network requests (XHR, AJAX, Fetch), including URL, body, headers, and more.
  * This feature is exclusive to Fetch and is ideal for analyzing background requests.

  <Tip>Learn more: [JSON Response](/fetch/features/json-response)</Tip>

  ### Browser Sessions

  * Does not capture network requests automatically.
  * You'll need to configure custom JavaScript code to intercept and extract XHR/AJAX calls manually on Puppeteer or Playwright.

  <Tip>For a step-by-step guide on capturing network requests in Playwright, check out our comprehensive [Playwright guide](https://www.zenrows.com/blog/playwright-scraping#request-and-response-interception).</Tip>

  ### Residential Proxies

  * Acts as a proxy layer without modifying responses.
  * To capture XHR/AJAX requests, you must configure custom request logging in your own setup (e.g., Puppeteer, Playwright, or Selenium).

  By choosing the right Zenrows product and configuration, you can effectively capture network requests and analyze the data that websites load dynamically. Let us know if you need guidance on a specific use case! 🚀
</Accordion>

<Accordion title="Does Zenrows Bypass Antibot Solutions?">
  Zenrows is designed to bypass most modern antibot solutions out-of-the-box. We continuously test and optimize our systems to ensure a smooth scraping experience. However, antibot defenses vary by website, and different Zenrows products serve different purposes.

  Below is an overview of how each product handles antibot measures and what to expect when using them.

  ## Fetch

  Fetch, when combined with Premium Proxies and JS Render, effectively handles most antibot measures. This setup mimics real user behavior, helping bypass bot detection mechanisms.

  However, not all pages are protected equally. Many websites enforce stricter protections on internal APIs or login-restricted content. If you're targeting such endpoints, additional configurations might be needed.

  If you're experiencing blocks despite using Premium Proxies and JS Render, refer to this guide: [Using Premium + JS Render and still blocked](/fetch/faq#using-premium-proxies-js-render-and-still-blocked)

  ## Residential Proxies

  Residential Proxies prioritize anonymity rather than antibot bypassing. They provide IP rotation and geographic targeting but do not include built-in antibot or CAPTCHA-solving capabilities. For heavily protected websites, additional techniques may be required.

  ## Browser Sessions

  Browser Sessions is highly effective against antibot and anticaptcha solutions, using the same advanced algorithms as Fetch. However, if a website enforces a CAPTCHA challenge, we do not automatically bypass it. Solving CAPTCHAs currently requires implementing custom handling, such as integrating third-party CAPTCHA-solving services.

  By choosing the right combination of Zenrows tools, you can optimize your web scraping strategy to handle even the most complex antibot defenses. If you need further assistance, feel free to reach out to our support team.
</Accordion>

<Accordion title="Does Zenrows Have No-Code Options?">
  Zenrows supports a variety of **no-code platforms** to help you scrape data from websites without writing a single line of code. These integrations let you connect your scraping workflows with thousands of apps like **Google Sheets**, **Airtable**, **Notion**, **Amazon S3**, and more.

  <Note>
    These no-code integrations are ideal for marketers, analysts, product managers, and anyone looking to automate data collection without needing technical skills.
  </Note>

  ## When to Use No-Code Integrations

  Use Zenrows' no-code options when you:

  * Want to scrape data into a spreadsheet without writing code
  * Need to automate recurring data collection tasks
  * Prefer visual workflow builders over API requests
  * Are integrating web data into tools like CRMs, email platforms, or dashboards

  **Best Practice:** Start with pre-built Zenrows templates in platforms like Zapier or Make to set up your workflow in minutes.

  ## Next Steps

  Visit our [Integrations Page](/integrations/overview) to explore tutorials and real-world examples that walk you through setting up your first workflow.
</Accordion>

<Accordion title="Does Zenrows Integrate with External Tools?">
  Suppose you must scrape data from a website and automatically process it using a third-party tool. We offer various ways to integrate Zenrows with external software and tools. Currently, you can [integrate a captcha solver](https://docs.zenrows.com/integrations/2captcha) or [a no-code tool like Zapier/Make/Clay](/first-steps/faq#does-zenrows-have-no-code-options).

  Additionally, you can build your integrations using the Zenrows output, whether HTML or JSON. A good use case for this is [Extract](/extract/setup), which returns structured data from a page as JSON.
</Accordion>

<Accordion title="Does Zenrows offer Custom plans?">
  Yes! Our custom plans are available for high-volume cases. We cannot customize public plans, as they are standardized for all our clients.
</Accordion>

<Accordion title="How Can I Speed Up My Requests?">
  Optimizing your requests can significantly improve performance and reduce response times. Below are general best practices, followed by specific recommendations for each Zenrows product.

  ## Factors Affecting Request Speed

  1. **Concurrency**: Sending multiple requests simultaneously can increase throughput
  2. **Resource Usage**: Features like JavaScript rendering or waiting for specific conditions can impact speed
  3. **Response Size (optional)**: Pages with dynamic content will naturally take longer to load. Consider targeting only the necessary data or using output filters to minimize payload.
  4. **Success Rate**: The rate of successful requests. If the success rate is low, you may need to increase the number of requests or the concurrency.

  <Note>While Residential Proxies have no concurrency restrictions, other products have plan-specific limits. Monitor your performance when adjusting these settings.</Note>

  ### Monitoring Concurrency Usage

  Each response includes headers that help you manage and optimize your concurrency:

  ```text theme={"dark"}
  Concurrency-Limit: 200
  Concurrency-Remaining: 199
  ```

  These headers help you:

  * Monitor how many concurrent requests your plan allows
  * Track how many slots are currently available
  * Adjust request volume dynamically to avoid hitting limits that may delay or throttle requests

  <Warning>The Concurrency-Remaining header reflects the real-time state of your concurrency usage and is the primary value our system uses to enforce limits. If it reaches zero and more requests are sent, you may receive a `429 Too Many Requests` error and your IP can be temporarily blocked for 5 minutes. <br /><br />If you receive a `BLK0001` error (IP Address Blocked), it means your IP has exceeded the allowed error rate. The block will last for 5 minutes and will impact your ability to send new requests during that time, affecting your overall scraping speed. For more details, see our [API Error Codes documentation](https://docs.zenrows.com/api-error-codes#blk0001-ip-address-blocked).</Warning>

  Use these headers to adjust your request flow in real-time, scaling up when possible and backing off before hitting limits.

  ## Product-Specific Recommendations

  ### Fetch

  1. **Optimize JavaScript Rendering**:

  * Disable `js_render=true` for static content to improve speed
  * Only enable when necessary for dynamic content or accessing protected content
  * Consider the impact on response times

  2. **Minimize Wait Times**:

  * Use `wait` and `wait_for` only when required
  * Set the minimum necessary wait duration
  * Longer waits mean slower requests

  3. **Use Premium Proxies**:

  * Enable `premium_proxy=true` for faster, more consistent responses
  * Particularly useful for sites with anti-bot measures
  * Can reduce retries and improve overall speed

  ### Residential Proxies

  1. **Request Rate Optimization**:

  * Monitor response times at different request rates
  * Adjust based on target site performance
  * Implement backoff when responses slow down

  ### Browser Sessions

  1. **Resource Management**:

  * Disable unnecessary JavaScript execution
  * Block non-essential resources (images, media, etc.)
  * Optimize browser settings for faster loading

  2. **CAPTCHA Handling**:

  * Implement manual CAPTCHA solving to avoid automated delays
  * Consider the impact on overall request speed

  ## Speed Optimization Best Practices

  1. **Start with Baseline**: Begin with standard settings and measure response times
  2. **Monitor Performance**: Use response headers and timing metrics to track speed
  3. **Gradual Optimization**: Make incremental changes and measure their impact
  4. **Smart Retries**: Use exponential backoff for failed requests to maintain speed
  5. **Target-Specific Tuning**: Adjust settings based on the specific website's performance

  <Warning>While these optimizations can improve request speed, some features (like JavaScript rendering) might be necessary for your specific use case. If you need help optimizing for speed while maintaining functionality, our support team is here to assist.</Warning>
</Accordion>

<Accordion title="How Is Usage Counted?">
  Your dashboard displays usage as a **percentage** of your plan's total credit allowance. This percentage reflects spending across all Zenrows products, with different features consuming credits at different rates based on their [credit costs](/first-steps/pricing#cost-multipliers-for-fetch).

  <Note>
    Features like JavaScript Rendering (5 credits) or Premium Proxies (10 credits) consume more of your allowance per request than standard requests (1 credit). This means the number of API calls you make may differ from the usage percentage shown. The percentage reflects the actual credit cost of each request relative to your total allowance.
  </Note>

  Credit resets and rollover depend on your plan tier:

  ## Free

  * Credits reset each month.
    <Warning>Unused credits do not roll over.</Warning>

  ## Build, Launch, Growth, and Scale

  * Credits reset each billing period.
    <Note>Unused credits roll over for 1 additional billing period before they expire.</Note>

  ## Enterprise

  * Rollover terms are negotiated as part of your contract.

  <Note>If you upgrade your plan mid-cycle, unused credits carry over to your new plan. See our [Pricing page](/first-steps/pricing#upgrading-your-plan) for details.</Note>
</Accordion>

<Accordion title="How to Handle CAPTCHA-Protected Pages?">
  Zenrows is designed to bypass most modern antibot solutions out-of-the-box. We continuously test our service to ensure optimal performance.

  Handling CAPTCHAs depends on the type of CAPTCHA and the Zenrows product you're using.

  ## Handling CAPTCHAs on Forms

  CAPTCHAs on forms are not solved automatically. If you need to submit forms that trigger a CAPTCHA, we offer an integration with a CAPTCHA solver that might work for your use case. Learn more about it here: [Using JavaScript Instructions to Solve CAPTCHAs](/fetch/features/js-instructions#captcha-solving)

  ## CAPTCHA Handling by Product

  Each Zenrows product has its own approach to handling CAPTCHAs, depending on the level of antibot protection in place. While some products automatically bypass CAPTCHAs in most cases, others may require additional configurations or external solvers. Below, we outline how each product deals with CAPTCHAs and what you can do to improve your success rate.

  ### Fetch

  * Uses Premium Proxies and JS Render to bypass most antibot measures.
  * If a CAPTCHA appears, you can use JavaScript Instructions to interact with the page and solve it manually or through an external CAPTCHA-solving service.

  ### Residential Proxies

  * Residential Proxies provide anonymity but do not bypass CAPTCHAs automatically.
  * If CAPTCHA protection is strict, you'll need custom handling or an external solver.

  ### Browser Sessions

  * Uses the same bypassing techniques as Fetch.
  * Can handle most antibot solutions, but does not solve CAPTCHAs by default.
  * If a CAPTCHA is encountered, it requires custom handling, such as integrating a CAPTCHA solver.

  By choosing the right Zenrows product and implementing the appropriate CAPTCHA-handling techniques, you can minimize interruptions and improve your scraping success rate. If you need assistance with a specific case, feel free to contact our support team.
</Accordion>

<Accordion title="What is concurrency?">
  Concurrency refers to the number of ongoing requests that happen at any given time. By different means, computers and languages can call the API in parallel and wait for results while others are still running. You can use concurrency with any Zenrows plan; check out [pricing](https://www.zenrows.com/pricing) for more details.

  For more details, check our [how-to guide on concurrency](/fetch/features/concurrency#using-concurrency) to see details about implementation in Python and JavaScript.

  ## Important Behavior to Understand

  ### Canceled Requests Still Count Against Your Concurrency

  One crucial thing to understand is that **canceling requests on the client side does NOT immediately free up concurrency slots**. When you cancel a request:

  1. Your client stops waiting for a response
  2. But the Zenrows server continues processing the request
  3. The concurrency slot remains occupied until the server-side processing completes
  4. Only then is the concurrency slot released

  This can lead to unexpected `429` errors if you're canceling requests and immediately trying to make new ones, as your concurrency limit might still be reached.

  ### Security System for Failing Requests

  Zenrows implements a security system that may temporarily ban your API key if you send too many failing requests in a short period. Types of failing requests that can trigger this include:

  * `429 Too Many Requests` errors due to exceeding concurrency limits
  * Invalid API key errors
  * `400 Bad Request` errors due to invalid parameters
  * Other repeated API errors

  If your API key gets temporarily banned, you'll receive an error from the API. If the requests continue, the IP address might get banned for a few minutes and the requests will not even connect with the server.

  ## Monitoring Your Concurrency Usage

  You can monitor your concurrency usage through response headers:

  ```text theme={"dark"}
  Concurrency-Limit: 200
  Concurrency-Remaining: 199
  X-Request-Cost: 0.001
  X-Request-Id: 67fa4e35647515d8ad61bb3ee041e1bb
  Zr-Final-Url: https://httpbin.io/anything
  ```

  These headers provide valuable information about your request:

  * **Concurrency-Limit**: Your maximum concurrent request limit based on your plan
  * **Concurrency-Remaining**: Number of additional concurrent requests you can make
  * **X-Request-Cost**: The cost of this request (varies based on enabled features)
  * **X-Key-Credits-Remaining**: The lowest remaining credits across your API key's [credit caps](/first-steps/api-key-credit-caps). Sent only for keys with at least one cap
  * **X-Request-Id**: Unique identifier for this request - essential when reporting issues to support
  * **Zr-Final-Url**: The final URL after any redirects that occurred during the request

  ## Related questions

  ### How many concurrent requests are included in my plan?

  The concurrency limit scales by plan tier, with higher limits on the top Scale rungs:

  * **Free:** 5 concurrent requests
  * **Build:** 20 concurrent requests
  * **Launch:** 50 concurrent requests
  * **Growth:** 100 concurrent requests
  * **Scale:** 200 concurrent requests (S1 to S3), 300 (S4), 400 (S5)
  * **Enterprise:** Custom (500 to 1,000+) concurrent requests

  See the full [pricing table](/first-steps/pricing#concurrency-support-and-rollover) for details. Contact us for tailor-made Enterprise solutions.

  ### If I get a "429 Too Many Requests" error, do I lose that request or is it queued?

  You'll receive an error, and that request won't be queued or retried automatically. You'll need to manage retries on your end, ensuring you don't exceed your concurrency limit.

  ### Can I increase my concurrency limit?

  Absolutely! We offer [various plans](https://www.zenrows.com/pricing) with different concurrency limits to suit your needs. If you find yourself frequently hitting the concurrency limit, consider upgrading.

  ### How can I monitor my concurrency usage?

  When using Fetch, each response includes these helpful headers:

  * `Concurrency-Limit`: Shows your maximum concurrent request limit
  * `Concurrency-Remaining`: Shows how many free concurrency slots you have available

  ### I've been blocked by repeatedly exceeding my concurrency limit. Why?

  Whenever you exceed your plan concurrency limit, you'll start receiving "429 Too Many Requests" errors.
  If you keep sending more and more requests exceeding your plan concurrency limit in a short period of time, the service may temporarily block your IP address to prevent API misuse.

  The IP address ban will last only a few minutes, but repeatedly being blocked might end in a long-lasting block.
  Check the [concurrency optimization section](/fetch/features/concurrency#using-concurrency-headers-for-optimization) for more information on how to limit concurrent requests to prevent being blocked.

  ### Troubleshooting with Support

  When contacting Zenrows support for any issues with your requests, always include:

  1. The **X-Request-Id** from the response headers
  2. The exact error message you received
  3. The timestamp of when the error occurred
  4. Details about the parameters you used in the request

  This information, especially the Request ID, allows our support team to quickly locate your specific request in our logs and provide more effective assistance.
</Accordion>

<Accordion title="What is Geolocation and What are All the Premium Proxy Countries?">
  Many websites tailor their content based on the visitor's location. For example, Amazon displays different products and prices on its UK (`.co.uk`) and French (`.fr`) sites. If you're scraping data from these sites, using a regional IP ensures that you receive the correct localized content. Also some websites restrict access to their content based on geographic location.

  To avoid discrepancies caused by regional variations, such as different products being displayed on a retailer's website, you can send a specific country code with your request. This ensures that your request is localized to the desired country, allowing you to obtain consistent and replicable results.

  Zenrows supports proxies from numerous countries around the world. You can use any <a href="https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes#Current_ISO_3166_country_codes">country's ISO code</a> to configure your proxies.

  Here is the comprehensive list of premium proxy countries supported by Zenrows:

  ```json theme={"dark"}
  af => Afghanistan
  al => Albania
  dz => Algeria
  ad => Andorra
  ao => Angola
  ai => Anguilla
  ag => Antigua and Barbuda
  ar => Argentina
  am => Armenia
  au => Australia
  at => Austria
  az => Azerbaijan
  bs => Bahamas
  bh => Bahrain
  bd => Bangladesh
  bb => Barbados
  by => Belarus
  be => Belgium
  bz => Belize
  bj => Benin
  bm => Bermuda
  bt => Bhutan
  bo => Bolivia, Plurinational State of
  ba => Bosnia and Herzegovina
  bw => Botswana
  br => Brazil
  bn => Brunei Darussalam
  bg => Bulgaria
  bf => Burkina Faso
  bi => Burundi
  cv => Cabo Verde
  kh => Cambodia
  cm => Cameroon
  ca => Canada
  ky => Cayman Islands
  cf => Central African Republic
  td => Chad
  cl => Chile
  cn => China
  co => Colombia
  km => Comoros
  cg => Congo
  cd => Congo, The Democratic Republic o
  cr => Costa Rica
  ci => Cote D'ivoire
  hr => Croatia
  cu => Cuba
  cy => Cyprus
  cz => Czech Republic
  dk => Denmark
  dj => Djibouti
  do => Dominican Republic
  ec => Ecuador
  eg => Egypt
  sv => El Salvador
  gq => Equatorial Guinea
  er => Eritrea
  ee => Estonia
  et => Ethiopia
  fo => Faroe Islands
  fj => Fiji
  fi => Finland
  fr => France
  gf => French Guiana
  pf => French Polynesia
  ga => Gabon
  gm => Gambia
  ge => Georgia
  de => Germany
  gh => Ghana
  gi => Gibraltar
  gr => Greece
  gl => Greenland
  gp => Guadeloupe
  gu => Guam
  gt => Guatemala
  gn => Guinea
  gw => Guinea-Bissau
  gy => Guyana
  ht => Haiti
  hn => Honduras
  hk => Hong Kong
  hu => Hungary
  is => Iceland
  in => India
  id => Indonesia
  ir => Iran, Islamic Republic of
  iq => Iraq
  ie => Ireland
  im => Isle of Man
  il => Israel
  it => Italy
  jm => Jamaica
  jp => Japan
  jo => Jordan
  kz => Kazakhstan
  ke => Kenya
  kr => Korea, Republic of
  kw => Kuwait
  kg => Kyrgyzstan
  la => Lao People's Democratic Republic
  lv => Latvia
  lb => Lebanon
  ls => Lesotho
  lr => Liberia
  ly => Libya
  lt => Lithuania
  lu => Luxembourg
  mo => Macao
  mk => Macedonia, The Former Yugoslav Republic of
  mg => Madagascar
  mw => Malawi
  my => Malaysia
  mv => Maldives
  ml => Mali
  mt => Malta
  mq => Martinique
  mr => Mauritania
  mu => Mauritius
  mx => Mexico
  md => Moldova, Republic of
  mn => Mongolia
  me => Montenegro
  ma => Morocco
  mz => Mozambique
  mm => Myanmar
  na => Namibia
  np => Nepal
  nl => Netherlands
  nc => New Caledonia
  nz => New Zealand
  ni => Nicaragua
  ne => Niger
  ng => Nigeria
  no => Norway
  om => Oman
  pk => Pakistan
  ps => Palestine, State of
  pa => Panama
  pg => Papua New Guinea
  py => Paraguay
  pe => Peru
  ph => Philippines
  pl => Poland
  pt => Portugal
  pr => Puerto Rico
  qa => Qatar
  re => Reunion
  ro => Romania
  ru => Russia
  rw => Rwanda
  lc => Saint Lucia
  mf => Saint Martin (French Part)
  ws => Samoa
  sa => Saudi Arabia
  sn => Senegal
  rs => Serbia
  sc => Seychelles
  sl => Sierra Leone
  sg => Singapore
  sx => Sint Maarten (Dutch Part)
  sk => Slovakia
  si => Slovenia
  sb => Solomon Islands
  so => Somalia
  za => South Africa
  ss => South Sudan
  es => Spain
  lk => Sri Lanka
  sd => Sudan
  sr => Suriname
  sz => Swaziland
  se => Sweden
  ch => Switzerland
  sy => Syrian Arab Republic
  tw => Taiwan, Province of China
  tj => Tajikistan
  tz => Tanzania, United Republic of
  th => Thailand
  tl => Timor-Leste
  tg => Togo
  to => Tonga
  tt => Trinidad and Tobago
  tn => Tunisia
  tr => Turkey
  tm => Turkmenistan
  ug => Uganda
  ua => Ukraine
  ae => United Arab Emirates
  gb => United Kingdom
  us => United States
  uy => Uruguay
  uz => Uzbekistan
  vu => Vanuatu
  ve => Venezuela, Bolivarian Republic of
  vn => Viet Nam
  ye => Yemen
  zm => Zambia
  zw => Zimbabwe
  ```

  ## How to use it with each product:

  ### Fetch

  Incorporate the selected ISO code into your scraping script to route your requests through the chosen proxy.

  ```python scraper.py theme={"dark"}
  # pip install requests
  import requests

  # Example for setting a proxy for Canada
  params = {
      'premium_proxy': 'true',
      'proxy_country': 'ca',
  }

  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ### Residential Proxies

  Zenrows supports IPs from a wide variety of countries, allowing you to access geo-restricted data with ease. You can specify a country by using `country` followed by the country code in your proxy URL.

  Example for Spain:

  ```bash theme={"dark"}
  http://<YOUR_USERNAME>:<YOUR_PASSWORD>_country-es@superproxy.zenrows.com:1337
  ```

  ### Browser Sessions

  You basically have two ways to set a country geolocation on Browser Sessions, depending if you're using the SDK or not.

  Without using the SDK, select a specific country by adding the `proxy_country` parameter to the WebSocket URL:

  ```bash theme={"dark"}
  wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY&proxy_country=es
  ```

  In SDK mode, specify the country when generating the connection URL:

  ```bash theme={"dark"}
  const connectionURL = scrapingBrowser.getConnectURL({
      proxy: {
          location: ProxyCountry.ES
      },
  });
  ```

  By using the right proxy, you can ensure more reliable and geographically relevant data scraping while maintaining compliance with website policies.

  For further assistance or more detailed configuration, refer to the Zenrows documentation or contact their support team. Happy scraping!
</Accordion>

<Accordion title="Do you charge failed requests using Fetch?">
  Not at all. We only charge for successful requests :)

  <Warning>`404 Not Found` and `410 Gone` errors are charged</Warning>
</Accordion>

<Accordion title="I need to scrape millions of websites every month. Is it possible?">
  You can extract data from as many websites as you want. Throw us 1M requests or 50M; we can perfectly handle it!
</Accordion>

<Accordion title="Do I need to install anything to use the product?">
  Not at all. Our infrastructure is cloud-based, making our language-agnostic API easy and seamless to use.
</Accordion>

<Accordion title="Do you rotate IPs on every request?">
  Yes, each new request automatically uses a different IP to ensure anonymity and avoid detection.

  However, if you need to maintain the same IP for a specific period, this is also possible. Please refer to each primitive's documentation for detailed information on how to configure it.
</Accordion>

<Accordion title="Can I use Residential Proxies with my custom scraping tools?">
  Absolutely. Residential Proxies are designed to integrate seamlessly with any custom scraping setup, regardless of the tools or frameworks you use.
</Accordion>

<Accordion title="Do you return the fingerprints from the scraping process?">
  No, our fingerprints are not returned.
</Accordion>

<Accordion title="Do you return the page's IP or my own IP during scraping?">
  No, we do not return either the page's IP or our own IP.
</Accordion>

<Accordion title="Can I access the page headers during scraping?">
  No, you cannot access the page headers directly. However, you can access the response headers.
</Accordion>

<Accordion title="Do you return CAPTCHA tokens after bypassing them?">
  No, CAPTCHA tokens are not returned after bypassing.
</Accordion>

<Accordion title="Do you offer custom coding solutions?">
  No, we provide the tools for scraping, but you need to develop your own solutions.

  We also offer [No-Code options](/first-steps/faq#does-zenrows-have-no-code-options) for those who prefer to avoid coding.
</Accordion>


# Quick Setup and First Request
Source: https://docs.zenrows.com/first-steps/getting-started-guide

Step-by-step quickstart guide to create your Zenrows account, get your API key, and send your first web scraping API request today.

Start scraping any website in under 5 minutes. Zenrows handles anti-bot measures, JavaScript rendering, and proxy management automatically, so you can focus on collecting data instead of managing infrastructure.

## Quick Setup

<Steps>
  <Step title="Create Your Account">
    Visit the [Registration Page](https://app.zenrows.com/register) and sign up using Google, GitHub, or your email address. Account creation is free and provides immediate access to your dashboard.

    <img alt="Zenrows sign up page" />
  </Step>

  <Step title="Complete Onboarding">
    Select options that match your use case during the setup process. This helps Zenrows recommend the most suitable features for your scraping needs.
  </Step>

  <Step title="Get Your API Key">
    Get your API key from the dashboard and start your first request immediately. This key authenticates all your requests and tracks usage against your plan limits. Keep this key secure and never share it publicly.
  </Step>
</Steps>

## Your First Request

Test your Zenrows setup with this simple request:

<CodeGroup>
  ```python Python theme={"dark"}
  ## pip install requests
  import requests

  url = 'https://httpbin.io/anything'
  apikey = 'YOUR_ZENROWS_API_KEY'
  params = {
      'url': url,
      'apikey': apikey,
      'mode': 'auto',
  }
  response = requests.get('https://api.zenrows.com/v1/', params=params)
  print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  const url = 'https://httpbin.io/anything';
  const apikey = 'YOUR_ZENROWS_API_KEY';
  axios({
  	url: 'https://api.zenrows.com/v1/',
  	method: 'GET',
  	params: {
  		'url': url,
  		'apikey': apikey,
  		'mode': 'auto',
  	},
  })
      .then(response => console.log(response.data))
      .catch(error => console.log(error));
  ```

  ```java Java theme={"dark"}
  import org.apache.hc.client5.http.fluent.Request;

  public class APIRequest {
      public static void main(final String... args) throws Exception {
          String apiUrl = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto";
          String response = Request.get(apiUrl)
                  .execute().returnContent().asString();

          System.out.println(response);
      }
  }
  ```

  ```php PHP theme={"dark"}
  <?php
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  $response = curl_exec($ch);
  echo $response . PHP_EOL;
  curl_close($ch);
  ?>
  ```

  ```go Go theme={"dark"}
  package main

  import (
      "io"
      "log"
      "net/http"
  )

  func main() {
      client := &http.Client{}
      req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto", nil)
      if err != nil {
          log.Fatalln(err)
      }
      resp, err := client.Do(req)
      if err != nil {
          log.Fatalln(err)
      }
      defer resp.Body.Close()

      body, err := io.ReadAll(resp.Body)
      if err != nil {
          log.Fatalln(err)
      }

      log.Println(string(body))
  }
  ```

  ```ruby Ruby theme={"dark"}
  # gem install faraday
  require 'faraday'

  url = URI.parse('https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto')
  conn = Faraday.new()
  conn.options.timeout = 180
  res = conn.get(url, nil, nil)
  print(res.body)
  ```

  ```bash cURL theme={"dark"}
  curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fhttpbin.io%2Fanything&mode=auto"
  ```
</CodeGroup>

Replace `YOUR_ZENROWS_API_KEY` with your actual API key from the dashboard. A successful response confirms your setup is working correctly.

## Next Steps

### Choose your Zenrows solution

Select the approach that best fits your scraping requirements and technical expertise:

<CardGroup>
  <Card title="Fetch" icon={<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="#7C4FE8" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className="icon icon-tabler icons-tabler-outline icon-tabler-chisel"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M14 14l1.5 1.5" /><path d="M18.347 15.575l2.08 2.079a1.96 1.96 0 0 1 -2.773 2.772l-2.08 -2.079a1.96 1.96 0 0 1 2.773 -2.772z" /><path d="M3 6l3 -3l7.414 7.414a2 2 0 0 1 .586 1.414v2.172h-2.172a2 2 0 0 1 -1.414 -.586l-7.414 -7.414z" /></svg>} href="https://docs.zenrows.com/fetch/api-reference">
    Formerly Universal Scraper API. **Best for**: General web scraping with automatic optimization
  </Card>

  <Card title="Browser Sessions" icon={<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="#7C4FE8" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className="icon icon-tabler icons-tabler-outline icon-tabler-click"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M3 12l3 0" /><path d="M12 3l0 3" /><path d="M7.8 7.8l-2.2 -2.2" /><path d="M16.2 7.8l2.2 -2.2" /><path d="M7.8 16.2l-2.2 2.2" /><path d="M12 12l9 3l-4 2l-2 4l-3 -9" /></svg>} href="/browser-sessions/introduction">
    Formerly Scraping Browser. **Best for**: Complex automation requiring complete browser control
  </Card>

  <Card title="Residential Proxies" icon={<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="#7C4FE8" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className="icon icon-tabler icons-tabler-outline icon-tabler-home-bolt"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M19 10l-7 -7l-9 9h2v7a2 2 0 0 0 2 2h7.5" /><path d="M9 21v-6a2 2 0 0 1 2 -2h2c.661 0 1.248 .32 1.612 .815" /><path d="M19 14l-2 4h4l-2 4" /></svg>} href="/legacy/residential-proxies/introduction">
    **Best for**: High-quality proxies with existing infrastructure
  </Card>

  <Card title="Platform Integrations" icon={<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="#7C4FE8" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className="icon icon-tabler icons-tabler-outline icon-tabler-plug"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M9.785 6l8.215 8.215l-2.054 2.054a5.81 5.81 0 1 1 -8.215 -8.215l2.054 -2.054z" /><path d="M4 20l3.5 -3.5" /><path d="M15 4l-3.5 3.5" /><path d="M20 9l-3.5 3.5" /></svg>} href="/integrations/overview">
    **Best for**: No-code solutions and workflow automation
  </Card>
</CardGroup>

<Card title="MCP Integration" icon="box" href="/mcp/overview">
  **Best for**: AI-powered assistance in your code editor
</Card>

<Tip>Check our [Primitives Documentation](/first-steps/our-products#choosing-a-primitive) to compare features and find the best fit for your use case, or review our [Pricing Guide](/first-steps/pricing) to select the right plan.</Tip>

### Learn web scraping fundamentals

Build your knowledge with comprehensive guides and best practices:

* [Web Scraping Best Practices](https://www.zenrows.com/blog/web-scraping-best-practices) - Essential techniques for reliable data collection, rate limiting strategies, and ethical scraping guidelines.
* [Complete Python Web Scraping Tutorial](https://www.zenrows.com/blog/web-scraping-python) - Step-by-step implementation guide covering BeautifulSoup, requests, and data processing.
* [Web Scraping Use Cases Guide](https://www.zenrows.com/blog/7-use-cases-for-website-scraping) - Real-world applications including price monitoring, lead generation, market research, and competitive analysis.
* [Handling JavaScript-Heavy Websites](https://www.zenrows.com/blog/scraping-javascript-rendered-web-pages) - Advanced techniques for single-page applications and dynamic content.
* [E-commerce Data Collection](https://www.zenrows.com/blog/job-board-scraping) - Strategies for product information, pricing, and inventory tracking.

### Scale your operations

As your scraping needs grow, explore advanced features and optimization strategies:

* **Concurrent requests**: Optimize performance by running multiple requests simultaneously
* **Data storage**: Choose appropriate databases and storage solutions for your collected data
* **Automation**: Set up scheduled scraping jobs for regular data collection
* **Data processing**: Implement pipelines for cleaning, validating, and analyzing scraped data
* **Monitoring**: Track success rates, identify patterns, and optimize your scraping strategy

### Get help when you need it

Access support resources and community knowledge:

* **FAQs** - Find answers to common questions about API limits, billing, and technical issues
* **Troubleshooting Guides** - Resolve specific problems with blocked requests, parsing errors, and configuration issues
* **Support** - Contact our team for personalized assistance with complex scraping challenges
* **Documentation** - Explore detailed guides for advanced features and specific use cases

## Why Choose Zenrows?

Zenrows solves the most challenging aspects of web scraping through automated solutions:

**Anti-bot bypass & proxy rotation**: Detects and bypasses Cloudflare, reCAPTCHA, Akamai, Datadome, and other blocking mechanisms. Automatically switches between residential IPs across 190+ countries while rotating browser fingerprints to maintain access.

**JavaScript rendering**: Uses real browser engines to handle dynamic content, infinite scroll, and lazy-loaded elements that traditional HTTP requests cannot capture.

**Adaptive infrastructure**: Includes automatic retry mechanisms when requests fail, real-time analytics for monitoring performance, and maintains 99.9% uptime.

**Auto-parsing features**: Adapts automatically to structural changes on supported websites, reducing maintenance overhead.

You can start scraping immediately without technical expertise in proxy management or anti-detection techniques.

## Common Scraping Challenges

If you're new to web scraping, understanding these common challenges helps explain why Zenrows provides value and how it solves problems you might encounter with traditional scraping approaches:

<img alt="scraping-issues-infographic" />

### Anti-Bot Measures

Websites deploy various blocking mechanisms to prevent scraping, such as:

* Web application firewalls (e.g., **Cloudflare**, **Datadome**, **Akamai**).
* CAPTCHA services that require solving puzzles to verify human activity.
* Rate limiting restricts the number of requests from a single IP address within a given time.

<img alt="cloudflare_blocked_page_sample" />

Overcoming these measures often requires advanced techniques like CAPTCHA-solving services, IP rotation, and adapting to specific firewall rules.

### JavaScript Rendering

Many modern websites dynamically render content using JavaScript, which prevents data from loading immediately when the page is opened. For example:

* Dynamic content that appears only after JavaScript execution completes.
* Infinite scrolling websites only display initial content, loading more as users scroll.
* "Load more" buttons that reveal additional content through user interaction.

Traditional HTTP requests cannot capture this dynamically generated content.

<img alt="scrapingcourse_infinite_scrolling_page" />

### HTML Structural Changes

Websites frequently update page designs, layouts, and HTML structures to introduce new features or improve user experience. These changes often include:

* Modifications to **CSS selectors** or **HTML attribute names**.
* Adjustments to the overall structure of the page.

Such updates can render previously working scrapers ineffective, requiring constant monitoring and updates to your scraping logic.

### Legal Considerations

Scraping activities must comply with legal and ethical guidelines:

* Website **terms of service** that prohibit scraping certain types of data, especially sensitive or restricted information.
* **Data behind login walls** or private details is often protected by law and requires explicit authorization to access.
* Adhering to rules specified in a site's `robots.txt` file, which defines areas where bots are not allowed.

## Frequently Asked Questions (FAQ)

<Accordion title="Do I need technical experience to use Zenrows?">
  No technical expertise in proxy management or anti-detection techniques is required. Zenrows handles all the complex infrastructure automatically. If you can make HTTP requests, you can use Zenrows.
</Accordion>

<Accordion title="Can I test Zenrows before committing to a paid plan?">
  Yes, every new account starts on the recurring Free plan (5,000 credits/month, no credit card required). You can make requests immediately after signing up to evaluate Zenrows with your target websites.
</Accordion>

<Accordion title="How do I know if I need JavaScript rendering enabled?">
  Enable JavaScript rendering (`js_render=true`) when scraping websites that load content dynamically, have infinite scroll, use "Load More" buttons, or are single-page applications (SPAs) built with React, Vue, or Angular.
</Accordion>

<Accordion title="Is my API key secure and how should I handle it?">
  Your API key is unique to your account and should be kept secure. Never share it publicly or commit it to version control. Use environment variables in your code and regenerate your key if you suspect it's been compromised.
</Accordion>

<Accordion title="How do I switch between different Zenrows products?">
  Each Zenrows product has its own endpoint and documentation. You can use multiple products with the same API key. Simply change the endpoint URL and parameters based on which product you want to use for each request.
</Accordion>

<Accordion title="What programming languages can I use with Zenrows?">
  Zenrows works with any programming language that can make HTTP requests. We provide code examples for Python, JavaScript/Node.js, Java, PHP, Go, Ruby, and cURL. The API is language-agnostic and follows standard REST principles.
</Accordion>

<Accordion title="How do I monitor my usage and remaining usage?">
  Log into your Zenrows dashboard to view real-time usage statistics, remaining usage, success rates, and request history. You can also set up usage alerts to notify you when approaching your plan limits.
</Accordion>

<Accordion title="Why did I get 'Email domain not allowed, contact support for more information'?">
  This message means your company email domain has reached the limit on accounts that can register with Zenrows. Common email providers are not affected.

  Contact [success@zenrows.com](mailto:success@zenrows.com) or use the live chat on the Zenrows website with your company email address and a short description of your use case. If the team approves it, they add your domain to the allowed list and you can register.

  This is separate from [REQS001](/api-error-codes#REQS001) and the [forbidden sites list](/forbidden-sites), which cover target domains you scrape, not the email address you register with.
</Accordion>

<Accordion title="Why did I get 'Invalid email address' when signing up?">
  This message means the email address failed Zenrows' signup checks. Common causes:

  * The address contains a `+` alias, such as `name+test@company.com`.
  * The domain belongs to a disposable or temporary email provider.
  * The address, or an alias of it, is already registered.

  Sign up again with your primary address, without a `+` alias. If you use a valid company address and still see the error, contact [success@zenrows.com](mailto:success@zenrows.com) or use the live chat on the Zenrows website with the address you tried.
</Accordion>


# How to Use Zenrows
Source: https://docs.zenrows.com/first-steps/how-to-use-zenrows

Get your API key, then find the exact setup guide for the primitive and surface you need, Fetch, Extract, Batch, or Browser Sessions, over the API, CLI, SDK, or MCP.

This page is a shortcut: answer two questions and land on the exact setup guide for your use case, instead of reading through every primitive to find the one you need.

## Step 1: Get an API Key

Every primitive and surface uses the same key.

<Steps>
  <Step title="Create an account">
    [Sign up for free](https://app.zenrows.com/register?p=free), no credit card required.
  </Step>

  <Step title="Get your API key">
    Find it on your [dashboard](https://app.zenrows.com/) as soon as you sign up.
  </Step>
</Steps>

<Tip>Using the [CLI](/cli/introduction)? Skip both steps, its first cloud call auto-provisions a Free-plan key for you.</Tip>

## Step 2: Pick What You Need to Do

These aren't mutually exclusive: most real tasks combine two or three. Expand every accordion below that matches a step in your workflow, not just the first one that fits.

<AccordionGroup>
  <Accordion title="Get a page's content, HTML, Markdown, plain text, or a screenshot">
    Use **Fetch**. [Set it up](/fetch/setup) and make your first request.
  </Accordion>

  <Accordion title="Turn a page into structured JSON, without writing your own parser">
    Use **Extract**. Read the [introduction](/extract/introduction), then add `extract=auto` to a Fetch call.
  </Accordion>

  <Accordion title="Process thousands of URLs as one job, instead of scripting it yourself">
    Use **Batch**. [Set it up](/batch/setup) and submit your first job.
  </Accordion>

  <Accordion title="Click, log in, navigate, or interact with a page like a real user">
    Use **Browser Sessions**. [Set it up](/browser-sessions/setup) and start interacting with a page.
  </Accordion>
</AccordionGroup>

<Tip>
  **Combining primitives is normal, not an exception.** A typical pipeline: use Fetch or Extract to pull a page's data, Batch to run that same request across thousands of URLs, and Browser Sessions for the handful of pages that need a login or a click first. See [What is Zenrows?](/first-steps/what-is-zenrows#why-zenrows) for how the primitives fit together as a full workflow.
</Tip>

## Step 3: Pick How You Want to Build

<AccordionGroup>
  <Accordion title="Call an HTTP endpoint directly, from any language">
    Use the **API**. Fetch and Extract are parameters on one endpoint; Batch has its own dedicated REST API, and Browser Sessions connects over its own WebSocket endpoint.
  </Accordion>

  <Accordion title="Work from a terminal, or let an AI agent drive it">
    Use the **CLI**. [Install it](/cli/introduction) and run `zenrows fetch`, `extract`, `batch`, or `browser` directly.
  </Accordion>

  <Accordion title="Write typed code in Python, Node.js, or Go">
    Use an **SDK**. See the [SDK overview](/sdk/overview) for Fetch (Python, Node.js, Go), Extract (through that same client), Batch ([Python](/batch/sdk/python) or [Node.js](/batch/sdk/nodejs)), and Browser Sessions (your own Puppeteer or Playwright code, no separate client).
  </Accordion>

  <Accordion title="Give an AI assistant or agent client direct access, no code">
    Use **MCP**. [Connect your client](/mcp/overview) in a few lines of config.
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup>
  <Card title="Welcome to Zenrows" icon="rocket" href="/first-steps/welcome">
    See every primitive and surface at a glance, and how to build with AI.
  </Card>

  <Card title="What is Zenrows?" icon="help-circle" href="/first-steps/what-is-zenrows">
    The identity and reasoning behind the infrastructure.
  </Card>

  <Card title="Our Primitives" icon="sitemap" href="/first-steps/our-products">
    Compare every primitive in depth and find the right one for your use case.
  </Card>
</CardGroup>


# Managing Your Notifications and Usage Alerts
Source: https://docs.zenrows.com/first-steps/manage-notifications

Learn how to set up and customize Zenrows email notifications, usage alerts, and billing summaries to monitor your API consumption.

Stay in control of your account and data usage with our customizable notification system. Notifications help you track your subscription, monitor API usage, and avoid service disruptions without needing to log in daily. This guide outlines the types of notifications available, their default statuses, and how to manage them. [Manage your notifications](https://app.zenrows.com/settings/notifications).

<Tip>
  Your account email is always notified. You can add extra email recipients to any notification by separating them with commas. See [Sending Notifications to Additional Emails](#sending-notifications-to-additional-emails).
</Tip>

## Billing

### Invoice Paid

**Default: `Enabled`**

Receive a confirmation email with a receipt each time a payment is successfully processed. This is ideal for keeping your accounting records up to date and sharing invoices with your finance team.

## Alerts

### Subscription Utilization Threshold

**Default: `Enabled`**

Set a custom percentage threshold (e.g., 80%) to receive a warning when your usage approaches the plan's limit. This is useful for proactively managing high-volume workflows and preventing unexpected overages. Click **Manage** to configure your preferred threshold level.

### Subscription Limit Reached

**Default: `Enabled`**

Get notified the moment your monthly quota is fully consumed. This allows you to take immediate action, such as upgrading your plan, purchasing extra usage (Top-Ups), or optimizing your requests to maintain uninterrupted service.

### API Key Credit Caps

**Default: `Enabled`**

If an API key has [credit caps](/first-steps/api-key-credit-caps), get an email and an in-app notification when the key reaches 80% and 100% of a cap, and an email when a cap is set, changed, or removed.

## Activity

### Daily Usage Summary

**Default: `Disabled`**

Receive a quick daily snapshot of your API activity, showing the number of requests made and data consumed by product. This is ideal for users who want to keep a close eye on consumption or need to report usage daily.

### Weekly Usage Summary

**Default: `Enabled`**

Receive a consolidated report summarizing your usage for the past 7 days. This is great for identifying trends, evaluating API needs, and planning for scale. Recommended for teams and managers who track consumption over time.

### Fetch Daily Consumption Threshold

**Default: `Enabled`**

Stay informed when your daily Fetch usage exceeds a preset threshold. This is useful for preventing excessive daily usage or monitoring anomalies. Thresholds can be customized by clicking **Manage** in your notification settings.

## Sending Notifications to Additional Emails

You can have any notification sent to other email addresses, such as your finance team or teammates:

1. Go to your [notification settings](https://app.zenrows.com/settings/notifications).
2. Make sure the notification is enabled. The **Manage** button appears next to the toggle switch only while the notification is switched on.
3. Click **Manage** next to the toggle switch.
4. Add the extra email addresses, separated by commas.

<Note>
  Your account email is always notified. Add extra emails separated by commas.
</Note>

## Managing Your Notification Settings

You can manage all notifications from the [Notifications Settings page](https://app.zenrows.com/settings/notifications). To unsubscribe or pause alerts:

1. Visit your [notification settings](https://app.zenrows.com/settings/notifications).
2. Locate the notification type you want to adjust.
3. Use the toggle switch to enable or disable it.
4. If applicable, click **Manage** to configure thresholds or add recipients.

These settings apply instantly, giving you real-time control over how and when you're informed.

<Note>
  We recommend keeping critical alerts such as **Subscription Utilization Limit Reached** and **Invoice Paid** enabled at all times to avoid missed billing or service interruptions.
</Note>


# Zenrows' Primitives
Source: https://docs.zenrows.com/first-steps/our-products

Compare every Zenrows primitive, Fetch, Extract, Batch, and Browser Sessions, and find the right one for your use case.

Zenrows' web data infrastructure is built from a small set of primitives, each covering a different part of getting data off the live web, including the protected web.

## Primitives

### <Icon icon="chisel" /> Fetch (formerly, Universal Scraper API)

Zenrows' Fetch is a powerful tool for easily extracting data from websites. It handles everything from dynamic content and JavaScript rendering to IP rotation and fingerprint management, offering a hassle-free solution for web scraping. Its extensive customization options, residential proxy support, and protected access capabilities make it perfect for handling complex scraping tasks efficiently.

**Key Benefits:**

* Handles dynamic content and JavaScript rendering
* Extensive customization options
* Protected access to anti-bot-defended sites

<Tip>Check our [MCP Integration](/mcp/overview) for AI-powered assistance in your code editor.</Tip>

### <Icon icon="wand" /> Extract

Zenrows' Extract turns a web page into structured JSON in a single request. Add `extract=auto` to your Fetch call, and it builds a tailored extraction for each website, keeping it working automatically when the site's layout changes, no selectors or schema to maintain.

**Key Benefits:**

* Tailored, per-website structured JSON
* No selectors or schema to maintain
* Adapts automatically when a site's layout changes

<Note>Extract is currently in Beta. It works on any domain once that domain has been prepared, which you can request yourself. See the [Extract docs](/extract/setup) for coverage and details.</Note>

### <Icon icon="stack-2" /> Batch

Batch runs large lists of URLs as a single managed job. You submit the URLs and scraping options, and Zenrows handles concurrency, automatic retries, result storage, and progress tracking, so you don't have to build the orchestration yourself. It uses the same scraping engine and options as Fetch.

**Key Benefits:**

* Submit up to 100,000 URLs in a single job
* Automatic retries and per-URL result tracking
* Scheduling, webhooks, and CSV upload

### <Icon icon="click" /> Browser Sessions (formerly, Scraping Browser)

Browser Sessions gives you a cloud-hosted, remote browser for tasks that need real interaction: clicking, scrolling, filling forms, navigating multi-step flows, or maintaining a logged-in session. It works with Puppeteer, Playwright, or any CDP-compatible tool, not just those two, so it drops into almost any existing browser automation setup. Zenrows' residential proxy network and anti-bot handling run underneath it, so you get real browser behavior without managing proxies or fingerprinting yourself.

**Key Benefits:**

* Works with Puppeteer, Playwright, or any CDP-compatible tool
* Full browser interaction: clicking, scrolling, forms, multi-step navigation
* Backed by Zenrows' residential proxy network and anti-bot handling

## Choosing a Primitive

| Primitive | Best for | Setup |
| - | - | - |
| **Fetch** | General-purpose scraping: protected access, JS rendering, and proxy rotation, all handled automatically | [Setup](/fetch/setup) |
| **Extract** | Turning a page into ready-to-use structured JSON with no selectors to maintain | [Setup](/extract/setup) |
| **Batch** | Running thousands of URLs as one managed job instead of building your own queue | [Setup](/batch/setup) |
| **Browser Sessions** | Deep browser interaction via Puppeteer, Playwright, or any CDP-compatible tool: multi-step flows, logins, scrolling | [Setup](/browser-sessions/setup) |

Still not sure? [How to Use Zenrows](/first-steps/how-to-use-zenrows) walks through the same decision as a step-by-step guide.

### Evaluate Cost and Scale

When selecting the right primitive, evaluating cost and scalability is critical to ensure your solution meets your budget and growth requirements.

Understand your usage patterns by estimating your monthly data needs and consider how many requests or pages you need to scrape and the complexity of those requests. For instance, scraping heavily protected sites or performing complex browser interactions may require more resources, impacting costs.

<Tip>You can find detailed information about pricing for each primitive in our [Pricing Documentation](/first-steps/pricing)</Tip>

## <Icon icon="home-bolt" /> Residential Proxies (Legacy)

Zenrows' Residential Proxies provide access to a global network of over 55 million IPs across 190+ countries, ensuring reliable and anonymous connections. With features like IP auto-rotation and geo-targeting, they are perfect for scraping geo-restricted content and maintaining high performance while staying undetected.

**Key Benefits:**

* Access to a global network of residential IPs
* IP auto-rotation and geo-targeting
* Reliable and anonymous connections

For general product FAQs, see the [FAQ page](/first-steps/faq).


# Zenrows Pricing
Source: https://docs.zenrows.com/first-steps/pricing

Complete guide to Zenrows pricing plans, credit-based billing, cost multipliers, pay-per-success billing, and the shared credit balance.

Zenrows offers transparent, credit-based pricing for every technical level and scraping volume. Every plan includes all four Zenrows primitives: **Fetch**, **Extract**, **Batch**, and **Browser Sessions**, over all four interfaces: **API**, **SDK**, **CLI**, and **MCP**. Nothing is segmented by product or interface: what scales with your plan is your monthly credit allowance, concurrency, and support level.

## How Pricing Works

### Shared Credit Balance

Each plan gives you a monthly credit allowance that you spend across every Zenrows product from one shared balance. There's no separate limit per product.

### Pay Only for Success

You're charged **only for successful requests**. Failed or retried requests don't consume your balance. HTTP 404 and 410 responses count as successful because the request completed correctly and returned valid data.

### Cost Multipliers for Fetch

Different features multiply your base credit cost depending on the infrastructure required:

| Feature | Credits per successful request |
| - | - |
| Standard request | 1 |
| JavaScript Rendering | 5 |
| Premium Proxies | 10 |
| JS Rendering + Premium Proxies (protected) | 25 |

<Note>
  These credit weights are fixed and never change. A JavaScript-rendered request always costs 5 credits, a protected request always costs 25, on every plan. What changes between plans is how many credits you get each month, not what anything costs to run.
</Note>

### Residential Proxies and Browser Sessions

Residential Proxies, used standalone, meter at a flat **25,000 credits per GB** on every plan. Browser Sessions bill on bandwidth (the same 25,000 credits/GB) plus a flat **5 credits per minute** of session time. Everything runs off the same shared balance.

<Tip>Rule of thumb: **1 GB = 25,000 credits = 1,000 protected pages.**</Tip>

## Plan Comparison

| Plan | Rung | Price/mo | Credits/mo | GB max\* |
| - | - | - | - | - |
| Free | N/A | \$0 | 5,000 | 0.2 |
| Build | B1 | \$19 | 45,000 | 1.8 |
| Build | B2 | \$29 | 80,000 | 3.2 |
| Build | B3 | \$39 | 120,000 | 4.8 |
| Launch | L1 | \$69 | 250,000 | 10 |
| Launch | L2 | \$99 | 500,000 | 20 |
| Launch | L3 | \$129 | 700,000 | 28 |
| Growth | G1 | \$199 | 1,200,000 | 48 |
| Growth | G2 | \$279 | 2,000,000 | 80 |
| Growth | G3 | \$399 | 3,000,000 | 120 |
| Scale | S1 | \$549 | 5,000,000 | 200 |
| Scale | S2 | \$749 | 8,000,000 | 320 |
| Scale | S3 | \$999 | 12,500,000 | 500 |
| Scale | S4 | \$1,999 | 28,000,000 | 1,120 |
| Scale | S5 | \$2,999 | 45,000,000 | 1,800 |
| Enterprise | N/A | Custom | 45M+ | N/A |

<Info>
  **GB max** = the maximum Residential Proxies/Browser Sessions bandwidth if you spent your entire monthly allowance on it alone (credits ÷ 25,000).
</Info>

Every higher rung gives you more credits for your money. The credit weights themselves never change (see [Cost Multipliers for Fetch](#cost-multipliers-for-fetch) above). A standard request is always 1 credit and a protected request is always 25, on every plan. What changes as you move up is how many credits you have to spend, not what anything costs to run.

<Note>
  Enterprise plans offer custom credit pricing beyond Scale S5. [Contact our sales team](https://www.zenrows.com/contact-sales) if your usage exceeds 45M credits/month.
</Note>

## Top-Up Packs

Sized so one pack comfortably covers a shortfall and two packs are enough to make an upgrade the better deal. Up to 10 top-ups per billing period; unused top-up credits roll into the next period.

| Rung | Pack credits | Pack price |
| - | - | - |
| Build B1 | 11,000 | \$6.50 |
| Build B2 | 13,000 | \$6.50 |
| Build B3 | 43,000 | \$19.50 |
| Launch L1 | 50,000 | \$19.50 |
| Launch L2 | 70,000 | \$19.50 |
| Launch L3 | 175,000 | \$45.50 |
| Growth G1 | 224,000 | \$52 |
| Growth G2 | 400,000 | \$78 |
| Growth G3 | 525,000 | \$97.50 |
| Scale S1 | 845,000 | \$130 |
| Scale S2 | 1,240,000 | \$163 |
| Scale S3 | 1,800,000 | \$200 |
| Scale S4 | 4,000,000 | \$400 |
| Scale S5 | 6,000,000 | \$560 |

<Note>Free doesn't include top-ups. If you run out of credits on Free, wait for the next monthly refresh or upgrade to a paid plan.</Note>

## Concurrency, Support, and Rollover

Concurrency and support scale by **tier**, not by rung: moving up a rung within a tier buys more monthly credits, moving to a new tier buys more capability. The one exception is Scale, where S4 and S5 also raise concurrency.

| Tier | Concurrency | Support | Rollover |
| - | - | - | - |
| Free | 5 | Docs + community | None |
| Build | 20 | Email support | 1 month |
| Launch | 50 | Chat support | 1 month |
| Growth | 100 | Priority support | 1 month |
| Scale | 200 (S1 to S3), 300 (S4), 400 (S5) | Priority support | 1 month |
| Enterprise | Custom (500 to 1,000+) | Premium dedicated support + account manager | Negotiated |

<Info>
  **Concurrent requests** = how many pages you can scrape simultaneously. Higher concurrency enables faster data extraction. **Rollover** = unused monthly plan credits carry into the next billing period, once.
</Info>

## Response size limits

Each plan caps the maximum size of a single response. The cap is set per tier.

| Tier | Max response size |
| - | - |
| Free | 10 MB |
| Build | 5 MB |
| Launch | 10 MB |
| Growth | 10 MB |
| Scale | 20 MB |
| Enterprise | Custom |

<Info>
  The cap applies **per request** and covers the whole response body, whether that's HTML, JSON, or a screenshot. Sizes are in MB using the decimal convention, so 1 MB = 1,000,000 bytes. A response that exceeds your plan's cap returns HTTP `413` (`RESP005`) and isn't delivered. See [Response Too Large](/fetch/troubleshooting/response-too-large) for how to reduce a response below the cap.
</Info>

## Credit Calculation Examples

### Fetch and Extract

* **Available requests** = Plan credits ÷ credits per request (1 for standard, 5 for JavaScript rendering, 10 for Premium Proxies, 25 for protected)
* Extract runs through the same Fetch client and draws credits the same way: add `extract=auto` to a request and it draws exactly what the underlying Fetch call would.

### Batch

* Billed identically to Fetch: per successful task, using the same credit weights.

### Browser Sessions

* **Total credits** = (GB used × 25,000) + (session minutes × 5)

### Residential Proxies (Legacy)

* **Total credits** = GB used × 25,000

> **Remember:** you pay only for successful requests across all products.

## Choose Your Plan

1. **Estimate your monthly volume**: how many standard vs. protected pages will you scrape?
2. **Add Browser Sessions or standalone Residential Proxies usage**: estimate the GB and session minutes you'll need.
3. **Match your total against the plan table above**, with some buffer room for growth.

## Managing Your Subscription

Access all subscription changes from your <a href="https://app.zenrows.com/plans">Plans</a> or <a href="https://app.zenrows.com/billing">Billing</a> pages.

### Top-Ups

You'll receive usage alerts at 80% and 95% of your limit, if enabled from your [Notifications settings](https://app.zenrows.com/settings/notifications). Top-ups let you add credits without changing plans. See the pack prices for your rung in the [Top-Up Packs](#top-up-packs) table above.

You can buy a top-up from two places in the app:

* **<a href="https://app.zenrows.com/billing">Billing</a> page:** the **Extra credits** section shows your pack size and price, plus how many purchases you have left this billing period. Click **Add credits** to buy a pack, or **Enable automatic credit purchases** to purchase packs automatically when your usage hits a threshold.

* **<a href="https://app.zenrows.com/plans">Plans</a> page:** a top-up offer below the plan cards opens the same purchase flow.

* **Limit:** up to 10 top-ups per billing period

* **Rollover:** unused top-up credits roll into the next period

<Note>
  If you don't see the top-up option, your account can't purchase one right now. This happens on the Free plan, during a trial, while a payment is incomplete, or once you've used all 10 purchases for the billing period.
</Note>

<Tip>Set automatic top-ups to trigger at your chosen threshold (like 95%). To avoid surprises, enable invoice notifications in your [Notification Settings](https://app.zenrows.com/settings/notifications).</Tip>

### Upgrading Your Plan

When you upgrade your plan, the change takes effect immediately. Here's what happens to your credits and limits:

* **When it applies:** Immediately after purchase
* **Credit carry-over:** Any unused credits from your current plan, including base plan credits, top-ups, and any admin-granted credits, automatically carry over to your new plan. This carried-over amount is added on top of your new plan's allowance as bonus credits, giving you extra usage from day one.
* **Top-up limits reset:** The carried-over bonus credits don't count against your new plan's top-up purchase limit. You start the new billing period fresh and can purchase top-ups as usual.

<Info>
  **Example:** If you're on Launch L1 (250,000 credits/month) with 90,000 unused credits and upgrade to Launch L3 (700,000 credits/month), your new billing period starts with 700,000 + 90,000 = **790,000 available credits**.
</Info>

### Downgrading Your Plan

* **When it applies:** At the end of your current billing cycle
* **During transition:** You keep your current limits until renewal

### Cancelling Your Subscription

<Steps>
  <Step title="How to cancel">
    1. Go to <a href="https://app.zenrows.com/billing">Billing</a>
    2. Click **Cancel Subscription**
    3. Select your reason for cancelling (your feedback helps us improve!)
    4. Confirm to complete the cancellation
  </Step>

  <Step title="After cancellation">
    1. You keep access until your billing cycle ends
    2. You can use your remaining credits during this time
    3. No additional charges will occur
  </Step>
</Steps>

<Info>
  For complete terms and conditions, visit our <a href="https://www.zenrows.com/terms-and-conditions">Terms & Conditions</a> page.
</Info>

## Frequently Asked Questions (FAQ)

<Accordion title="How is pricing calculated?">
  Each plan includes a monthly allowance of credits, so you're not counting dollars per call. One credit is a standard successful page. Harder work draws more from the same balance: JavaScript rendering costs 5 credits, Premium Proxies 10, and both together (protected) 25. Residential Proxies, used standalone, draw 25,000 credits per GB. Browser Sessions bill on bandwidth plus session time: 5 credits per minute, all in credits. Every plan runs off one balance.
</Accordion>

<Accordion title="Do I need to pay separately for each product?">
  No. Your plan's credit allowance applies to all products from one shared balance, not allocated separately to each one. For example, on Launch L1 (250,000 credits/month), you could spend 150,000 credits on Fetch and 100,000 on Browser Sessions in the same month.
</Accordion>

<Accordion title="What payment methods do you accept?">
  We accept credit cards (Visa, MasterCard, American Express, etc.), Apple Pay, Google Pay and PayPal. Wire transfer is accepted for Enterprise customers. We don't accept crypto or debit cards at the moment.
</Accordion>

<Accordion title="Do you charge for failed requests?">
  Never. You only spend credits on successful requests. `404 Not Found` and `410 Gone` responses count as successful results.
</Accordion>

<Accordion title="Do you offer discounts?">
  Yes. Paying yearly gives you two months free (pay for 10, get 12) and your full year's credit allowance up front, as a single balance you draw down anytime over the year rather than monthly grants. Enterprise is priced on committed volume.
</Accordion>

<Accordion title="What happens if I hit my monthly limit?">
  We warn you before you reach it. On a paid plan, you can add extra credits (top-ups) any time to keep running, up to 10 times per billing period, and unused top-up credits roll into the next period. On Free, there are no top-ups: wait for the next monthly refresh or upgrade. Either way, no lost data or configuration.
</Accordion>

<Accordion title="Can I upgrade or downgrade at any time?">
  Yes. Upgrades take effect immediately, billing adjusted accordingly, with unused credits carried over as bonus credits on your new plan. Downgrades take effect at the end of the current billing cycle. No penalties, no hidden fees.
</Accordion>

<Accordion title="What are concurrent requests?">
  Concurrency is how many requests run at the same time. The limit scales with your plan tier (see the [Concurrency, Support, and Rollover](#concurrency-support-and-rollover) table above), so larger plans run more jobs in parallel. See our [concurrency guide](/fetch/features/concurrency#using-concurrency) for implementation details.
</Accordion>

<Accordion title="How does protected access work?">
  Same primitives, same API. Set the protection level (or let Adaptive Stealth Mode, `mode=auto`, choose it for you) and Zenrows routes through the right stack (residential network, fingerprinting, retries), so your code doesn't change between an open page and a hard-protected one.
</Accordion>

<Accordion title="Does Zenrows offer Pay-As-You-Go plans?">
  Not as a standalone option today. The Free plan (5,000 credits/month, no card required) covers light, ongoing use without a subscription.
</Accordion>

<Accordion title="Can I get extra concurrency, a plan for a single product, or a custom plan?">
  Custom plans are available for usage beyond Scale S5 (\$2,999/month, 45M credits/month). See the Enterprise row in the [Plan Comparison](#plan-comparison) table above. We don't offer the option to purchase additional concurrency separately or to subscribe to a plan for just one product at a lower price: every plan includes all four products and all four interfaces, with limits determined by your selected plan.
</Accordion>

<Accordion title="Can I pause or delay my subscription?">
  No, pausing or delaying your subscription isn't currently available. You can keep your subscription active or cancel it at any time. If you cancel, you keep access to all services until the end of your current billing cycle. Unused days or credits don't roll over if you reactivate a cancelled subscription later.
</Accordion>

<Accordion title="How can I access my subscription invoices?">
  You can view and download your subscription invoices on your [Billing page](https://app.zenrows.com/billing).
</Accordion>

<Accordion title="How can I change my invoice information?">
  It's not currently possible to update information on past invoices. To ensure your future invoices have the correct details, keep your profile information up to date in your [Account Settings](https://app.zenrows.com/settings/profile) and your payment methods on the [Billing page](https://app.zenrows.com/billing). Changes apply to your next invoice onward.
</Accordion>

<Accordion title="Does Zenrows work with Claude, Cursor, and Codex?">
  Yes. The Agent Toolkit ships an MCP server, a CLI, and SDKs that connect to Claude Code, Cursor, Codex, Windsurf, Gemini CLI, and any MCP-compatible client. Every primitive is exposed as a tool, so agents call them natively with auth and usage visibility built in. This is included on every plan, not a separate add-on.
</Accordion>

<Accordion title="Do you offer SLAs and enterprise security?">
  Yes, on Enterprise: an SLA on uptime and success rate, SOC 2 and ISO 27001 compliance, SSO/SAML, dedicated account management, a solutions engineer, and quarterly business reviews (QBRs).
</Accordion>


# Welcome to Zenrows
Source: https://docs.zenrows.com/first-steps/welcome

Get started with Zenrows' web data infrastructure, built for AI agents and applications: Fetch, Extract, Batch, and Browser Sessions, available over the API, CLI, MCP, and SDKs.

Zenrows is web data infrastructure built for agents and applications. It gives them reliable access to the live web, including the protected web. That means sites defended by anti-bot systems, rendered dynamically in JavaScript, or built to block ordinary tools.

It's for developers and AI agents alike. It's just as useful for data, product, and GTM teams running web data workflows in production.

A small set of primitives, Fetch, Extract, Batch, and Browser Sessions, handle the actual work of retrieving pages and extracting their data. Use them through whichever surface fits how you build: the API directly, the CLI, MCP, or an SDK.

**Simple by default. Deep when needed.**

## Primitives

<CardGroup>
  <Card title="Fetch" href="/fetch/api-reference">
    Get any page as clean, usable content, even if it's protected by anti-bot defenses.
  </Card>

  <Card title="Extract" href="/extract/setup">
    <Badge>New</Badge> Turn a messy page into clean, structured JSON with a single parameter.
  </Card>

  <Card title="Batch" href="/batch/introduction">
    <Badge>New</Badge> Process thousands of URLs reliably, as one managed job.
  </Card>

  <Card title="Browser Sessions" href="/browser-sessions/introduction">
    Click, log in, navigate, and interact with a page, like a real user, from your own Puppeteer or Playwright code.
  </Card>
</CardGroup>

## Surfaces

Every primitive above is available however you build.

<CardGroup>
  <Card title="API" href="/fetch/api-reference">
    One HTTP endpoint and a set of parameters. Call it directly from any language.
  </Card>

  <Card title="CLI" href="/cli/introduction">
    Fetch, extract, and run jobs at scale from your terminal, or an agent's.
  </Card>

  <Card title="MCP" href="/mcp/overview">
    Give an AI assistant the ability to scrape any page from inside your editor.
  </Card>

  <Card title="SDKs" href="/fetch/sdk/overview">
    Typed clients for Node.js, Python, and Go, wrapping the same endpoint.
  </Card>
</CardGroup>

## Build with AI

The CLI installs everything an agent needs to use Zenrows on its own: skills, templates, and MCP wiring, configured in one command.

```bash theme={"dark"}
npx -y @zenrows/cli init --all
```

<CardGroup>
  <Card title="Zenrows CLI" icon="terminal" href="/cli/introduction">
    Primitives, Agent Toolkit commands, and the full command reference.
  </Card>

  <Card title="Connect an agent" icon="plug" href="/cli/mcp">
    Wire Claude Code, Cursor, Windsurf, and more into Zenrows with one command.
  </Card>
</CardGroup>

### 💡 Need Help? You Can Ask Our AI-Powered Search!

<Tip>You can use the **search bar at the top of the page** to instantly get answers to your questions, find documentation, and explore Zenrows' features. Simply type what you're looking for, and our AI will guide you!</Tip>


# What is Zenrows?
Source: https://docs.zenrows.com/first-steps/what-is-zenrows

What Zenrows is, the primitives and surfaces it's built from, and why teams and AI agents use it for reliable access to the live web.

## What is Zenrows?

Zenrows is web data infrastructure built for agents and applications. It gives them reliable access to the live web, including the protected web. That means sites defended by anti-bot systems, rendered dynamically in JavaScript, or built to block ordinary tools. It's the reliability layer between AI systems, applications, and the human web.

Instead of shipping a single scraper, it exposes a small set of primitives: Fetch, Extract, Batch, and Browser Sessions. Together, they cover the actual lifecycle of getting data off a page: retrieving it, extracting its data, running it at scale, and interacting with it like a real user. Each primitive is available over whichever surface fits how you build: the API directly, the CLI, MCP, or an SDK.

**Simple by default. Deep when needed.**

This matters because most tools solve only part of the problem. A basic HTTP client can request a page but can't render JavaScript or get past anti-bot defenses. A browser automation library can render a page but leaves you to build proxy rotation, retries, and scale yourself.

Zenrows combines both: the reliability of production scraping infrastructure with the same small set of primitives, whether you're calling one URL or a hundred thousand.

## Why Zenrows?

**Reliable access to protected and dynamic pages.** Sites that block ordinary requests, that render their content in JavaScript, or that require a session to stay logged in, are the default case Zenrows is built for, not an edge case.

**Structured output, not just raw HTML.** Fetch and Extract return the format you actually need: clean HTML, Markdown, plain text, a PDF, or structured JSON, without you writing your own parser.

**A full workflow, not just single requests.** Fetch handles one page, Extract turns it into structured data, Batch runs the same work across thousands of URLs as a managed job, and Browser Sessions handles the pages that need real interaction. You reach for the same primitives at any scale.

**Built for agents and teams beyond just developers.** The CLI, MCP server, and SDKs give an AI agent the same reliable access a human developer gets, installable with one command instead of hand-written scraping code. The same primitives serve data, product, and GTM teams running web data workflows in production, not just engineers writing scrapers.

## Next steps

<CardGroup>
  <Card title="Welcome to Zenrows" icon="rocket" href="/first-steps/welcome">
    See every primitive and surface, and how to build with AI.
  </Card>

  <Card title="Our Primitives" icon="sitemap" href="/first-steps/our-products">
    Compare every primitive in depth and find the right one for your use case.
  </Card>

  <Card title="Pricing" icon="tag" href="/first-steps/pricing">
    See how each primitive is billed.
  </Card>
</CardGroup>


# How to Integrate 2Captcha with Zenrows
Source: https://docs.zenrows.com/integrations/2captcha

Learn how to integrate 2Captcha with Zenrows to solve interactive CAPTCHAs like reCAPTCHA and Cloudflare Turnstile while scraping protected websites.

Integrating 2Captcha with Zenrows gives you an extra layer of protection to handle interactive CAPTCHAs that appear on web pages. This guide shows you how to set up 2Captcha with both the Zenrows Fetch and the Zenrows Browser Sessions.

## What Is 2Captcha

2Captcha is a CAPTCHA-solving service that employs humans to solve various types of CAPTCHA challenges. Here's how 2Captcha works:

1. The scraper submits the CAPTCHA challenge as a unique key (e.g., sitekey for reCAPTCHA) via 2Captcha's API.
2. A human solver receives and solves the CAPTCHA challenge manually.
3. The human worker returns the CAPTCHA solution as a hashed token.
4. The scraper supplies the solution token to the site and solves the challenge.

The service is particularly useful for solving in-page CAPTCHAs, ensuring complete CAPTCHA evasion. These include CAPTCHAs attached to form fields or those that appear when you interact with page elements.

You'll learn to integrate 2Captcha with Zenrows in the next sections of this guide. To demonstrate the solution, we'll use the <a href="https://2captcha.com/demo/recaptcha-v2">reCAPTCHA demo page</a> as the target website.

## <svg><path /><path /><path /><path /></svg> Integrating 2Captcha With Fetch

Fetch automatically bypasses website protections, but you should combine it with 2Captcha for interactive or form-based CAPTCHAs. This approach requires minimal code changes and helps handle unexpected blocks.

### Step 1: Set up 2Captcha on Zenrows

1. Log in to your <a href="https://2captcha.com/enterpage">2Captcha dashboard</a> and copy your API key.
   <img alt="2Captcha Dashboard API Key" />
2. Go to the [Integrations Page](https://app.zenrows.com/settings/integrations) in your Zenrows account and click on `Integrations`.
3. Click `Manage` on the 2Captcha card.
   <img alt="Zenrows Integrations page with the 2Captcha integration" />
4. Enter your 2Captcha API key and click `Save`.
   <img alt="Zenrows Manage 2Captcha modal to enter the API key" />

### Step 2: Integrate 2Captcha in your scraping request

After adding the 2Captcha Zenrows integration, you need to specify its usage in your Fetch request via the [`js_instructions`](/fetch/features/js-instructions) parameter. Zenrows handles all communication between your script and 2Captcha.

The `js_instructions` parameter accepts a `solve_captcha` option that specifies which CAPTCHA type you want to solve.

Zenrows's Fetch supports these CAPTCHA types for 2Captcha integration:

* `recaptcha`: Google reCAPTCHA series, including invisible CAPTCHAs that don't require user interaction.
* `cloudflare_turnstile`: Cloudflare Turnstile CAPTCHAs are typically found on forms and interactive elements.

Here are the supported CAPTCHA types with their configuration options:

```javascript theme={"dark"}
[
    // Solve reCAPTCHA
    {"solve_captcha": {"type": "recaptcha"}},

    // Solve Cloudflare Turnstile
    {"solve_captcha": {"type": "cloudflare_turnstile"}},

    // Solve Invisible reCAPTCHA with inactive option
    {"solve_captcha": {"type": "recaptcha", "options": {"solve_inactive": true}}}
]
```

<Info>For more information on the available instructions, check the [JavaScript Instructions Documentation](https://docs.zenrows.com/fetch/features/js-instructions#captcha-solving) page.</Info>

Now, let's implement the `js_instructions` in your Zenrows request. In this example, the script waits for 3 seconds, solves the reCAPTCHA challenge using 2Captcha, waits 3 seconds for processing, clicks the submit button, and then waits for the success message to be present:

```python Python theme={"dark"}
# pip3 install requests
import requests

url = "https://2captcha.com/demo/recaptcha-v2"

def scraper(url):
    apikey = "YOUR_ZENROWS_API_KEY"
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "json_response": "true",
        "js_instructions": """[
            {"wait": 3000},
            {"solve_captcha":{"type":"recaptcha"}},
            {"wait": 3000},
            {"click": "button[type='submit']"},
            {"wait_for": "._successMessage_1ndnh_1"}
        ]""",
        "css_extractor": """{"success":"p._successMessage_1ndnh_1"}""",
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.json()
```

<Note>The CSS selectors provided in this example (`._successMessage_1ndnh_1`, `button[type='submit']`) are specific to the page used in this guide. Selectors may vary across websites. For guidance on customizing selectors, refer to the [CSS Extractor documentation](/fetch/features/css-extractor). If you're having trouble, the [Advanced CSS Selectors Troubleshooting Guide](/fetch/troubleshooting/advanced-css-selectors) can help resolve common issues.</Note>

### Step 3: Confirm you solved the CAPTCHA

You can confirm that the CAPTCHA was successfully solved by checking the response from both the `js_instructions` and `css_extractor`:

```python theme={"dark"}
# confirm if CAPTCHA is bypassed
html_content = scraper(url)
css_extractror_info = html_content.get("css_extractor")
js_instructions_info = html_content.get("js_instructions_report")

print(
    "CSS extractor info:",
    css_extractror_info,
    "\n",
    "JS instructions info:",
    js_instructions_info,
)
```

### Complete Example

Here's the full working code:

```python Python theme={"dark"}
# pip3 install requests
import requests

url = "https://2captcha.com/demo/recaptcha-v2"


def scraper(url):

    apikey = "YOUR_ZENROWS_API_KEY"
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "json_response": "true",
        "js_instructions": """[
            {"wait": 3000},
            {"solve_captcha":{"type":"recaptcha"}},
            {"wait": 3000},
            {"click": "button[type='submit']"},
            {"wait_for": "._successMessage_1ndnh_1"}
        ]""",
        "css_extractor": """{"success":"p._successMessage_1ndnh_1"}""",
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.json()

### confirm if CAPTCHA is bypassed
html_content = scraper(url)
css_extractror_info = html_content.get("css_extractor")
js_instructions_info = html_content.get("js_instructions_report")

print(
    "CSS extractor info:",
    css_extractror_info,
    "\n",
    "JS instructions info:",
    js_instructions_info,
)
```

When successful, you'll see a response like this, confirming the CAPTCHA was solved:

```python Response theme={"dark"}
CSS extractor info: {"data": {"success": "Captcha is passed successfully!"}}
JS instructions info: {
    "instructions": [
        # ...
        {
            "duration": 1456,
            "instruction": "solve_captcha",
            "params": {"options": None, "type": "recaptcha"},
            "result": [
                {
                    # ...
                    "response_element": True,
                    "solved": True,
                    # ...
                }
            ],
            "success": True,
        },
        # ...
    ],
    # ...
}
```

This response is equivalent to manually clicking the "Check" button on the reCAPTCHA demo page to verify the CAPTCHA was solved.

<Note>The [`json_response`](https://docs.zenrows.com/fetch/features/js-instructions#debugging-javascript-instructions) parameter is used here for debugging purposes. For production use, you can parse the `response.text` with a parser like BeautifulSoup since the request returns the HTML content that was previously locked behind the CAPTCHA.</Note>

## <svg><path /><path /><path /><path /><path /><path /><path /></svg> Integrating 2Captcha With the Zenrows Browser Sessions

The Zenrows Browser Sessions doesn't include direct 2Captcha integration, but you can achieve this using the 2Captcha SDK. Here's how to set it up using Playwright with Browser Sessions in Python.

### Step 1: Install the 2Captcha SDK

First, install the 2Captcha SDK:

```bash theme={"dark"}
pip3 install 2captcha-python
```

### Step 2: Connect to 2Captcha

Import the necessary libraries and create a connection to 2Captcha using your API key:

```python Python theme={"dark"}
# pip3 install 2captcha-python playwright
from playwright.async_api import async_playwright
from twocaptcha import TwoCaptcha

# connect to 2Captcha API
two_captcha_api_key = "2Captcha_API_KEY"
solver = TwoCaptcha(two_captcha_api_key)
```

The solver object handles all communication with the 2Captcha service.

### Step 3: Connect to Browser Sessions and obtain the CAPTCHA Token

Connect to Browser Sessions, navigate to your target page, and extract the reCAPTCHA sitekey (a unique identifier for the CAPTCHA):

```python Python theme={"dark"}
# ...
    
    # scraping browser connection URL
    connection_url = (
        "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"
    )

    async with async_playwright() as p:
        # connect to the browser over CDP (Chrome DevTools Protocol)
        browser = await p.chromium.connect_over_cdp(connection_url)

        # create a new page
        page = await browser.new_page()

        # navigate to the target URL
        await page.goto(url)

        # wait for the reCAPTCHA sitekey to load and extract it
        await page.wait_for_selector("#g-recaptcha")

        site_key = await page.evaluate(
            "document.querySelector('#g-recaptcha').getAttribute('data-sitekey')"
        )

        # solve CAPTCHA
        captcha_response = solver.recaptcha(sitekey=site_key, url=url)

        # extract the reCAPTCHA token from the response
        captcha_token = captcha_response["code"]
```

This code extracts the sitekey from the reCAPTCHA element and sends it to 2Captcha for solving.

### Step 4: Solve the CAPTCHA

When a user solves a CAPTCHA, the solution token gets placed in a hidden input field. You need to simulate this process by injecting the token into the appropriate field:

```python Python theme={"dark"}
# ...
async def scraper(url):

    async with async_playwright() as p:
        # ...

        # enter the CAPTCHA token to pass the block
        await page.evaluate(
            """
            (captchaToken) => {
                const responseField = document.getElementById("g-recaptcha-response");
                responseField.style.display = 'block';
                responseField.value = captchaToken;
                responseField.dispatchEvent(new Event('change', { bubbles: true }));
            }
            """,
            captcha_token,
        )
```

This JavaScript code finds the hidden response field, makes it visible, sets the token value, and triggers a change event to notify the page that the CAPTCHA was solved.

### Step 5: Confirm you solved the CAPTCHA

After applying the CAPTCHA solution, you can proceed with your scraping tasks. Let's verify the solution worked by clicking the submit button and checking the response:

```python Python theme={"dark"}
# ...
import time

# ...
async def scraper(url):

    async with async_playwright() as p:
        # ...

        # confirm if the CAPTCHA is passed
        await page.get_by_role("button", name="Check").click()

        time.sleep(10)

        try:
            success_text = await page.text_content("p._successMessage_1ndnh_1")

            if success_text:
                print(success_text)
            else:
                error_text = await page.text_content("div._alertBody_bl73y_16")
                print(error_text)
        except Exception as e:
            print(e)

        # close the browser
        await browser.close()
```

<Note>The CSS selectors provided in this example (`._successMessage_1ndnh_1`, `._alertBody_bl73y_16`) are specific to the page used in this guide. Selectors may vary across websites. For guidance on customizing selectors, refer to the [CSS Extractor documentation](/fetch/features/css-extractor). If you're having trouble, the [Advanced CSS Selectors Troubleshooting Guide](/fetch/troubleshooting/advanced-css-selectors) can help resolve common issues.</Note>

Run the code using `asyncio`:

```python Python theme={"dark"}
# ...
import asyncio

# ...

url = "https://2captcha.com/demo/recaptcha-v2"
# run the scraper
asyncio.run(scraper(url))
```

### Complete Example

Here's the full working code:

```python Python theme={"dark"}
# pip3 install 2captcha-python playwright
from playwright.async_api import async_playwright
from twocaptcha import TwoCaptcha
import time
import asyncio

# connect to 2Captcha API
two_captcha_api_key = "2Captcha_API_KEY"

solver = TwoCaptcha(two_captcha_api_key)

# scraping browser connection URL
connection_url = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"


async def scraper(url):

    async with async_playwright() as p:
        # connect to the browser over CDP (Chrome DevTools Protocol)
        browser = await p.chromium.connect_over_cdp(connection_url)

        # create a new page
        page = await browser.new_page()

        # navigate to the target URL
        await page.goto(url)

        # wait for the reCAPTCHA sitekey to load and extract it
        await page.wait_for_selector("#g-recaptcha")

        site_key = await page.evaluate(
            "document.querySelector('#g-recaptcha').getAttribute('data-sitekey')"
        )

        # solve CAPTCHA
        captcha_response = solver.recaptcha(sitekey=site_key, url=url)

        # extract the reCAPTCHA token from the response
        captcha_token = captcha_response["code"]

        # enter the CAPTCHA token to pass the block
        await page.evaluate(
            """
            (captchaToken) => {
                const responseField = document.getElementById("g-recaptcha-response");
                responseField.style.display = 'block';
                responseField.value = captchaToken;
                responseField.dispatchEvent(new Event('change', { bubbles: true }));
            }
            """,
            captcha_token,  # Pass the CAPTCHA token as an argument to the JavaScript function
        )

        # confirm if the CAPTCHA is passed
        await page.get_by_role("button", name="Check").click()

        time.sleep(10)

        try:
            success_text = await page.text_content("p._successMessage_1ndnh_1")

            if success_text:
                print(success_text)
            else:
                error_text = await page.text_content("div._alertBody_bl73y_16")
                print(error_text)
        except Exception as e:
            print(e)

        # close the browser
        await browser.close()


url = "https://2captcha.com/demo/recaptcha-v2"
# run the scraper
asyncio.run(scraper(url))
```

The above code extracts the following success message from the page:

```bash theme={"dark"}
Captcha is passed successfully!
```

Congratulations! 🎉 You've just integrated 2Captcha with the Zenrows Fetch and the Zenrows Browser Sessions.

## Conclusion

By integrating 2Captcha with Zenrows, you can reliably handle even the toughest CAPTCHAs and keep your scraping workflows running smoothly. Whether you use Fetch or Browser Sessions, this setup helps you automate CAPTCHA solving and reduce manual effort.

## Frequently Asked Questions (FAQ)

<Accordion title="Why Use 2Captcha With Zenrows?">
  At a success rate of 99.93%, Zenrows bypasses the CAPTCHAs and other anti-bot measures commonly encountered during web scraping. However, interactive or in-page CAPTCHAs, such as those tied to form fields, may occasionally appear after bypassing the primary CAPTCHA. While this is rare, integrating 2Captcha provides an additional layer of stealth, ensuring even higher reliability and enhancing the already impressive 99.93% success rate.
</Accordion>

<Accordion title="Does Fetch Integration Support All CAPTCHA Types?">
  Zenrows, when integrated with 2Captcha, natively supports solving reCAPTCHA and Cloudflare Turnstile, two of the most common CAPTCHA types.
</Accordion>

<Accordion title="Does Zenrows return CAPTCHA tokens?">
  No, Zenrows does not return CAPTCHA tokens.
</Accordion>


# How to Integrate Anthropic Claude API with Zenrows
Source: https://docs.zenrows.com/integrations/anthropic-claude-api

Give Claude real-time access to any website using Zenrows. Build web-aware AI applications that ground their answers in live data, even from protected, JavaScript-heavy sites.

Claude has a fixed knowledge cutoff and no built-in web access at the API layer. Pairing it with the [Zenrows Fetch](/fetch/api-reference) closes that gap: your application can fetch live content from any website, including JavaScript-heavy and bot-protected pages, and feed it directly into any Claude model.

This guide covers two implementation paths:

* **Direct API Call**: Best for request-level control over your AI application's scraping layer.
* **Model Context Protocol (MCP)**: Best for a plug-and-play setup with Zenrows' optimal anti-bot configurations built in.

<Tip>The guide uses Claude's Messages API, tool use (function calling), and structured outputs throughout. If you are building autonomous, multi-step agents with the Claude Agent SDK, see the [Claude Agent SDK integration guide](/integrations/claude-agent-sdk).</Tip>

## What is Anthropic Claude API?

The Anthropic API provides programmatic access to Claude through the <a href="https://platform.claude.com/docs/en/build-with-claude/working-with-messages">Messages API</a>, with built-in support for tool use (function calling), structured outputs, extended thinking, and the MCP connector for remote MCP servers.

Claude excels at reasoning over long documents, multi-step analysis, and structured generation, but it has a fixed knowledge cutoff and no built-in web access at the API layer. Pairing Claude with Zenrows for reliable scraping that bypasses anti-bot detection closes that gap.

### Key benefits

* **Real-time web grounding for any Claude model** <br />
  Zenrows feeds live, up-to-date web content into any response, so responses draw from real-time data instead of training knowledge. This eliminates stale or hallucinated answers about recent events, prices, or product details.

* **Anti-bot bypass out of the box** <br />
  [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode) (`mode=auto`) automatically handles JavaScript rendering, premium proxies, fingerprinting, and bot detection, so Claude can read pages that would block a regular scraper.

* **Token-efficient Markdown output** <br />
  Zenrows returns clean Markdown in addition to raw HTML. This pairs well with Claude's large context window, letting you fit more pages per request without burning through tokens on HTML noise.

* **Native tool use** <br />
  Expose Zenrows as a tool and let Claude decide when to scrape based on the user's question, with no orchestration code required.

* **Works with every Claude API surface** <br />
  Use Zenrows with the Messages API, structured outputs (`messages.parse`), and extended thinking. The integration is a plain HTTP call, so it fits into any pattern.

* **MCP-ready out of the box** <br />
  Plug the [Zenrows hosted MCP server](/mcp/overview) directly into the Messages API through Claude's MCP connector and give your AI application access to all of Zenrows' scraping capabilities.

### Use cases

The Claude and Zenrows combination unlocks a wide range of AI workflows:

<AccordionGroup>
  <Accordion title="Long-context document grounding" icon="file-lines">
    Scrape multiple pages with Zenrows and feed them into Claude's 200K+ context window for cross-document analysis, comparison, or synthesis.
  </Accordion>

  <Accordion title="Real-time competitive intelligence" icon="chart-line">
    Have Claude analyze competitor pricing pages, product launches, and changelogs as they happen.
  </Accordion>

  <Accordion title="Lead enrichment" icon="address-card">
    Scrape company websites and let Claude extract industry, headcount signals, tech stack hints, and product summaries into your CRM.
  </Accordion>

  <Accordion title="Web-aware support assistants" icon="headset">
    Build assistants that can read product docs, status pages, or knowledge bases the user references mid-conversation.
  </Accordion>

  <Accordion title="Structured data extraction" icon="table">
    Pull strongly-typed JSON (products, jobs, listings, reviews) from any page using Claude's structured outputs grounded in Zenrows-scraped content.
  </Accordion>
</AccordionGroup>

## Getting started: Basic usage

A simple example: scrape a JavaScript-heavy, anti-bot-protected demo page using Zenrows, then summarize it with `claude-sonnet-4-6` via the Messages API.

<Steps>
  <Step title="Install the Anthropic Python library and requests">
    ```shell theme={"dark"}
    pip install anthropic requests
    ```
  </Step>

  <Step title="Create a .env file with your API keys">
    ```shell theme={"dark"}
    ZENROWS_API_KEY=your_zenrows_key
    ANTHROPIC_API_KEY=your_anthropic_key
    ```

    Find your Zenrows API key in the [Zenrows dashboard](https://app.zenrows.com/register). Find your Anthropic API key in the <a href="https://console.anthropic.com">Claude Console</a> under **Settings > API keys**.
  </Step>

  <Step title="Run the following script">
    ```python Python expandable theme={"dark"}
    import os
    import requests
    from anthropic import Anthropic
    from dotenv import load_dotenv
    load_dotenv()
    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
    client = Anthropic()

    def scrape(url: str) -> str:
        """Scrape any URL with Zenrows and return clean Markdown."""
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "apikey": ZENROWS_API_KEY,
                "url": url,
                "mode": "auto",
                "response_type": "markdown",
            },
        )
        response.raise_for_status()
        return response.text

    # Scrape a JS-rendered, anti-bot-protected page
    markdown = scrape("https://www.scrapingcourse.com/antibot-challenge")

    # Summarize with Claude through the Messages API
    message = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system="You are a concise technical writer.",
        messages=[
            {
                "role": "user",
                "content": f"Summarize this page in 2 sentences:\n\n{markdown}",
            }
        ],
    )

    print(message.content[0].text)
    ```

    Two things are happening here:

    1. Zenrows handles the scrape. `mode=auto` tells the API to start with the cheapest viable configuration and automatically escalate to JavaScript rendering or premium proxies if the target site requires it. `response_type=markdown` returns clean Markdown instead of raw HTML, which is ideal for Claude's context window.
    2. Claude handles the reasoning. The Markdown is fed into `client.messages.create()`, with system-level guidance passed through the `system` parameter and the user prompt passed through `messages`. Claude's response comes back as a list of content blocks; for plain-text replies, `message.content[0].text` is the right call.
  </Step>
</Steps>

## Advanced usage: Building a web-aware AI assistant with tool use

Tool use is Anthropic's mechanism for letting Claude decide when to call an external tool. Instead of always scraping before each prompt, you expose Zenrows as a tool and let Claude call it only when needed.

This pattern is the foundation of any production-grade Claude application that needs web access.

<Steps>
  <Step title="Set up the environment">
    Install the Anthropic Python library and requests:

    ```shell theme={"dark"}
    pip install anthropic requests
    ```

    Create a `.env` file:

    ```shell theme={"dark"}
    ZENROWS_API_KEY=your_zenrows_key
    ANTHROPIC_API_KEY=your_anthropic_key
    ```
  </Step>

  <Step title="Define the scraping tool">
    Wrap the Zenrows REST call in a Python function and describe it to the Messages API using a JSON schema:

    ```python Python expandable theme={"dark"}
    import os
    import requests
    from anthropic import Anthropic
    from dotenv import load_dotenv
    load_dotenv()
    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
    client = Anthropic()

    def scrape_website(url: str) -> str:
        """Scrape any website with Zenrows and return Markdown content."""
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "apikey": ZENROWS_API_KEY,
                "url": url,
                "mode": "auto",
                "response_type": "markdown",
            },
        )
        response.raise_for_status()
        return response.text

    # JSON schema to tell Claude what scrape_website does and how to call it
    tools = [
        {
            "name": "scrape_website",
            "description": (
                "Scrape any public website and return its content as Markdown. "
                "Use this whenever you need up-to-date information from a specific URL."
            ),
            "input_schema": {
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The fully qualified URL of the website to scrape",
                    }
                },
                "required": ["url"],
            },
        }
    ]
    ```

    Each tool definition uses Anthropic's flat shape: `name`, `description`, and `input_schema` at the top level.
  </Step>

  <Step title="Let Claude decide when to scrape">
    Send the user's question along with the tool definition. Claude returns either a direct text response or a `tool_use` content block with `stop_reason: "tool_use"`:

    ```python Python expandable theme={"dark"}
    def ask(question: str) -> str:
        messages = [{"role": "user", "content": question}]

        # Send the question to Claude with the scraping tool available
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            tools=tools,
            messages=messages,
        )

        # Keep looping as long as Claude wants to call a tool
        while response.stop_reason == "tool_use":
            # Extract the tool call from Claude's response
            tool_use = next(block for block in response.content if block.type == "tool_use")
            # Execute the scrape and capture the result
            result = scrape_website(**tool_use.input)

            # Append Claude's response and the tool result to the conversation history
            messages.append({"role": "assistant", "content": response.content})
            messages.append(
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "tool_result",
                            "tool_use_id": tool_use.id,
                            "content": result,
                        }
                    ],
                }
            )

            # Send the updated conversation history back to Claude to continue
            response = client.messages.create(
                model="claude-sonnet-4-6",
                max_tokens=1024,
                tools=tools,
                messages=messages,
            )

        # Return Claude's final text response once no more tool calls are needed
        return next(block.text for block in response.content if block.type == "text")

    # Ask Claude a question that requires live web access to answer
    print(
        ask(
            "What does the Hugging Face blog cover? "
            "Visit https://huggingface.co/blog and tell me what topics they post about."
        )
    )
    ```

    The `while response.stop_reason == "tool_use"` loop is the standard Claude tool-use pattern. Claude can chain multiple tool calls in a single conversation, so the loop continues until Claude returns `stop_reason: "end_turn"`. Each iteration appends the assistant's tool use and your tool result to the messages list before the next API call.
  </Step>

  <Step title="Complete code example and output">
    ```python Python expandable theme={"dark"}
    import os
    import requests
    from anthropic import Anthropic
    from dotenv import load_dotenv

    load_dotenv()
    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
    client = Anthropic()

    def scrape_website(url: str) -> str:
        """Scrape any website with Zenrows and return Markdown content."""
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "apikey": ZENROWS_API_KEY,
                "url": url,
                "mode": "auto",
                "response_type": "markdown",
            },
        )
        response.raise_for_status()
        return response.text

    tools = [
        {
            "name": "scrape_website",
            "description": (
                "Scrape any public website and return its content as Markdown. "
                "Use this whenever you need up-to-date information from a specific URL."
            ),
            "input_schema": {
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The fully qualified URL of the website to scrape",
                    }
                },
                "required": ["url"],
            },
        }
    ]

    def ask(question: str) -> str:
        messages = [{"role": "user", "content": question}]

        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            tools=tools,
            messages=messages,
        )

        while response.stop_reason == "tool_use":
            tool_use = next(block for block in response.content if block.type == "tool_use")
            result = scrape_website(**tool_use.input)

            messages.append({"role": "assistant", "content": response.content})
            messages.append(
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "tool_result",
                            "tool_use_id": tool_use.id,
                            "content": result,
                        }
                    ],
                }
            )

            response = client.messages.create(
                model="claude-sonnet-4-6",
                max_tokens=1024,
                tools=tools,
                messages=messages,
            )

        return next(block.text for block in response.content if block.type == "text")

    print(
        ask(
            "What does the Hugging Face blog cover? "
            "Visit https://huggingface.co/blog and tell me what topics they post about."
        )
    )
    ```

    **Example output:**

    ```markdown Output expandable theme={"dark"}
    The **Hugging Face Blog** (https://huggingface.co/blog) is a community-driven publication covering a wide range of topics related to AI, machine learning, and open-source development. Here's a breakdown of what they post abo

    ### 🏷️ Main Topic Categories

    The blog uses tags to organize content, including:
    - **NLP** (Natural Language Processing)
    - ...truncated for brevity

    ### 📝 Recent Article Highlights

    Here are some of the specific topics covered in recent posts:
    1. **Robotics** – e.g.,...
    ...truncated for brevity

    ### 👥 Who Writes It?

    The blog is a **community blog**, meaning articles are contributed by a mix of:

    - **Hugging Face team members**
    - ...truncated for brevity

    In summary, the Hugging Face blog is a rich resource for anyone interested in **cutting-edge AI research, practical ML tutorials, model releases, open-source tooling, and industry insights**.
    ```

    Claude decides on its own that the question requires fresh information from `huggingface.co/blog`, calls `scrape_website` with the right URL, then synthesizes an answer from the scraped Markdown.
  </Step>
</Steps>

## Structured outputs

Anthropic's <a href="https://platform.claude.com/docs/en/build-with-claude/structured-outputs">structured outputs</a> let you guarantee that Claude returns JSON conforming to a specific schema. Combined with Zenrows, this is the cleanest way to extract typed data from any web page.

The Messages API exposes structured outputs through `client.messages.parse()`, which takes a Pydantic model and returns a parsed Python object directly through `response.parsed_output`.

The example below scrapes a company homepage and extracts a strongly-typed `CompanyInfo` object:

```python Python expandable theme={"dark"}
import os
import requests
from anthropic import Anthropic
from pydantic import BaseModel
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = Anthropic()

# Define the Pydantic schema
class CompanyInfo(BaseModel):
    name: str
    industry: str
    description: str

# Scrape the company website
response = requests.get(
    "https://api.zenrows.com/v1/",
    params={
        "apikey": ZENROWS_API_KEY,
        "url": "https://huggingface.co/",
        "mode": "auto",
        "response_type": "markdown",
    },
)
response.raise_for_status()
markdown = response.text

# Extract structured data through the Messages API
parsed = client.messages.parse(
    model="claude-opus-4-7",
    max_tokens=1024,
    system="Extract company information from the provided website content.",
    messages=[{"role": "user", "content": markdown}],
    output_format=CompanyInfo,
)

company = parsed.parsed_output
print(company.model_dump_json(indent=2))
```

**Example output:**

```json JSON theme={"dark"}
{
  "name": "Hugging Face",
  "industry": "Artificial intelligence / Machine learning platform",
  "description": "Hugging Face is an open-source AI community and platform that hosts machine learning models, datasets, and applications. It provides tooling for building, sharing, and deploying models, including the Transformers library and Inference Endpoints, and is widely used as the central hub for the open-source ML ecosystem."
}
```

The same pattern works for any structured extraction job: product listings, job postings, real estate, reviews, articles, and so on. Define the Pydantic schema, scrape the page with `mode=auto`, and let Claude return validated JSON.

<Tip>
  **Skip the schema with Zenrows Autoparse.** For common site types, you can skip LLM-based parsing entirely. Adding `autoparse=true` to your Zenrows request automatically identifies and extracts product details, article content, job listings, property data, and similar information into clean JSON, with no Pydantic schema or model call required. [Zenrows Autoparse](/fetch/features/autoparse) is included at no additional cost. Note that Autoparse cannot be combined with `response_type=markdown` in the same request.
</Tip>

## Using Zenrows MCP with the Messages API

The Messages API natively supports remote [Model Context Protocol (MCP)](/mcp/overview) servers through Claude's MCP connector. Zenrows has a hosted MCP server that exposes web scraping capabilities, so you can give Claude real-time web access without writing any tool use boilerplate.

```python Python expandable theme={"dark"}
import os
from anthropic import Anthropic
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = Anthropic()

response = client.beta.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Visit https://huggingface.co/blog and summarize the three most recent posts.",
        }
    ],
    mcp_servers=[
        {
            "type": "url",
            "url": "https://mcp.zenrows.com/mcp",
            "name": "zenrows",
            "authorization_token": ZENROWS_API_KEY,
        }
    ],
    tools=[
        {
            "type": "mcp_toolset",
            "mcp_server_name": "zenrows",
        }
    ],
    betas=["mcp-client-2025-11-20"],
)

# Print Claude's final text response
for block in response.content:
    if block.type == "text":
        print(block.text)
```

A few notes on this configuration:

* The MCP connector lives under `client.beta.messages.create()` (not the standard `messages.create()`), and requires the `betas=["mcp-client-2025-11-20"]` parameter.
* The configuration has two parts: `mcp_servers` defines the connection (URL, name, authorization token), and `tools` references that server with a `mcp_toolset` entry that controls which tools are enabled.
* `authorization_token` carries your Zenrows API key as the Bearer token. Anthropic does not retain the value between requests, so it must be present on every API call.
* Responses include new `mcp_tool_use` and `mcp_tool_result` content blocks alongside regular text blocks. Filter by `block.type == "text"` for the final answer, or inspect all blocks for full tool call traces.

For details on the Zenrows MCP server and its capabilities, see the [MCP overview](/mcp/overview).

## Troubleshooting

### Token limit exceeded

1. Use `response_type=markdown` (shown in every example above). Markdown reduces token usage significantly compared to raw HTML.
2. Use `css_extractor`, `autoparse` for supported sites, or `outputs` to scrape only the parts of the page you need (a product card, a pricing table, an article body) instead of the entire DOM.
3. Claude has one of the largest context windows in the industry (200K+ tokens for Sonnet 4.6 and Opus 4.7). For multi-page workflows, you can often fit all scraped content in a single request without chunking.

### API key errors

1. Confirm both `ZENROWS_API_KEY` and `ANTHROPIC_API_KEY` are set in your environment.
2. Verify your Zenrows API key in the [Zenrows dashboard](https://app.zenrows.com/settings/api-keys) and your Anthropic key in the <a href="https://console.anthropic.com/settings/keys">Anthropic Console</a>.
3. Check that your Zenrows subscription is active and has remaining quota on the [Analytics page](https://app.zenrows.com/overview#performance).

### Empty or incomplete tool responses

1. Confirm `mode=auto` is set. Without it, JavaScript rendering and premium proxies are off by default and protected sites return blocked or empty pages.
2. For sites that load content asynchronously, add `wait_for=<css-selector>` (waits for a specific element) or `wait=5000` (waits a fixed duration in milliseconds).
3. If Claude is calling the tool with a malformed URL, tighten the tool description and `input_schema`, or add validation in your `scrape_website` wrapper before sending the request to Zenrows.

### Infinite tool use loop

If your `while response.stop_reason == "tool_use"` loop runs more times than expected, Claude is asking to scrape the same URL repeatedly. Tighten the system prompt with a stop condition such as "Once you have the content of the requested URL, summarize and stop." For complex workflows, add an explicit iteration counter and break after a reasonable maximum.

### 424 failed dependency or 401 unauthorized from the MCP connector

1. Confirm the `authorization_token` field is set on the MCP server entry and that the value is your active Zenrows API key.
2. Confirm you are using `client.beta.messages.create()` (not `client.messages.create()`) and that `betas=["mcp-client-2025-11-20"]` is set.
3. Confirm the URL in the `mcp_servers` entry is exactly `https://mcp.zenrows.com/mcp`.

## FAQ (Frequently Asked Questions)

<Accordion title="Which Claude models work with this integration?">
  All current Claude 4 models. Tool use and structured outputs work best with `claude-opus-4-7` and `claude-sonnet-4-6`. For high-throughput, cost-sensitive workloads, `claude-haiku-4-5-20251001` is a faster, cheaper option. The MCP connector is currently available on the Claude API, Claude Platform on AWS, and Microsoft Foundry. It is not yet supported on Amazon Bedrock or Vertex AI.
</Accordion>

<Accordion title="How do I handle pages that exceed the model's token limit?">
  Three options, in order of preference: use `response_type=markdown` to keep token usage low; use `css_extractor`, `autoparse` (JSON output for supported sites), or `outputs` to scrape only the relevant section of the page; use Claude models with large context windows (Sonnet 4.6 and Opus 4.7 support 200K+ tokens) to fit more pages per request without chunking.
</Accordion>

<Accordion title="Does Zenrows handle JavaScript-heavy and anti-bot-protected sites automatically?">
  Yes. Setting `mode=auto` enables Adaptive Stealth Mode, which automatically activates JavaScript rendering, premium residential proxies, and stealth fingerprinting only when the target site requires them. You only pay for what succeeds.
</Accordion>

<Accordion title="Can I use this with the Claude Agent SDK?">
  Yes. The Claude Agent SDK is built on top of the Messages API and has native MCP support, so the same hosted MCP server URL works there. For agent-specific workflows including autonomous research, multi-step tool chains, and the SDK's built-in MCP support, see the [Claude Agent SDK integration guide](/integrations/claude-agent-sdk).
</Accordion>

<Accordion title="Can I let Claude browse multiple pages autonomously?">
  Yes. Tool use supports multi-turn loops, so Claude can call `scrape_website` repeatedly with different URLs, read what it gets back, and decide what to scrape next. For more complex agent behavior, pair this with the MCP connector, which exposes a richer scraping surface without requiring you to maintain the tool use loop yourself. For autonomous, multi-step research workflows, see the [Claude Agent SDK integration guide](/integrations/claude-agent-sdk).
</Accordion>

<Accordion title="How does this compare to Claude's built-in web search and web fetch tools?">
  Anthropic offers built-in `web_search` and `web_fetch` server tools, but they're limited: web search is a black-box query interface, and web fetch is a basic HTTP getter without anti-bot bypass, JavaScript rendering, or geo-routing. Zenrows gives you full control over which URL is fetched, how the page is rendered, what format the response comes back in, and which countries the request is routed through. It also handles pages that block standard scrapers, such as geo-restricted content, JS-gated sites, and heavily protected pages.
</Accordion>

<Accordion title="Is there a rate limit I should be aware of?">
  Zenrows has plan-based concurrency limits documented on the [Concurrency page](/fetch/features/concurrency). Anthropic also has <a href="https://platform.claude.com/docs/en/api/rate-limits">model-specific rate limits on the Messages API</a>. When using tool use loops or the MCP connector with many parallel scrapes, monitor your usage on both dashboards and consider adding retry logic with exponential backoff in your `scrape_website` wrapper.
</Accordion>


# How to Integrate Google Antigravity 2.0 with Zenrows
Source: https://docs.zenrows.com/integrations/antigravity-20

Add the Zenrows MCP to Google Antigravity 2.0 and give your autonomous agents reliable access to any website without getting blocked. Scrape live data, pull competitor pages, and run scheduled scraping tasks with results delivered directly to your Antigravity workspace.

## What Is Antigravity 2.0?

<a href="https://antigravity.google/docs/overview">Antigravity 2.0</a> is Google's dedicated platform for working with autonomous agents. It lets you orchestrate multiple agents working in parallel across independent projects, with support for dynamic subagents, scheduled tasks, MCP integrations, and customizable skills. Connecting Zenrows enables every agent in your workspace to access live web data from any site, regardless of bot protection.

## Why Connect Zenrows to Antigravity 2.0?

* **No more blocked agents:** When trying to access protected sites, Antigravity's agents encounter the same bot-protection walls as any other automated client. Zenrows automatically handles anti-bot evasion so your agents get the page every time.
* **Cleaner data, lower token cost:** Zenrows returns structured Markdown or JSON instead of raw HTML. Your agents process only what matters, with less noise and fewer tokens used per task.
* **Reliable scheduled tasks:** Antigravity's `/schedule` command lets agents run on a cron schedule for daily reports, price checks, and monitoring jobs. A blocked or blank page can break the entire run. Zenrows handles dynamic rendering and anti-bot bypass under the hood, enabling Antigravity to run scheduled tasks without interruptions.
* **No scraping code required:** Describe what you want in plain English. Antigravity instructs Zenrows' tools and returns structured results. No selectors, no HTTP clients, no infrastructure to maintain.
* **Works with Gemini, ChatGPT, or Claude:** Antigravity supports multiple models. Use Zenrows MCP with the LLM you're already using without switching vendors.

## What You Can Build with Antigravity 2.0 and Zenrows

* **SEO and competitive research:** Schedule an agent to scrape competitor landing pages, blog posts, and product pages daily. Have it compare what they're publishing against your own site and surface gaps in topics, pages, and angles.
* **Developer and API research:** Point an agent at documentation sites, GitHub READMEs, changelogs, and technical forums. It pulls the latest specs, summarizes breaking changes, and flags anything relevant to your current project.
* **Best deals and price monitoring:** Set up a scheduled agent that scrapes product listings from Amazon, Google Shopping, and niche retailers, then extracts prices and ratings and surfaces the top deals for your target items every day.
* **Live data pipelines without mocking:** When building dashboards or internal tools inside Antigravity, agents normally work with mock data. Connect Zenrows and the agent pulls real product listings, pricing, or public records directly from the web during development.

## Prerequisites

* <a href="https://antigravity.google">Google Antigravity 2.0</a> downloaded and installed for your OS.
* A Zenrows API key from the <a href="https://app.zenrows.com/fetch">Zenrows Playground</a>.

## Add Zenrows MCP to Antigravity

Antigravity has a built-in MCP Store where you can find and install supported servers with one click. Zenrows isn't in the store yet, so you'll need to add it manually via Antigravity's `mcp_config.json` file.

<Steps>
  <Step title="Step 1: Open the MCP config file">
    Open the MCP configuration file at the following path:

    ```
    ~/.gemini/config/mcp_config.json
    ```
  </Step>

  <Step title="Step 2: Add the Zenrows server">
    Add the Zenrows server block under `mcpServers`, replacing `YOUR_ZENROWS_API_KEY` with your actual key from the <a href="https://app.zenrows.com">Zenrows dashboard</a>:

    ```json JSON theme={"dark"}
    {
      "mcpServers": {
        "zenrows": {
          "command": "npx",
          "args": ["-y", "@zenrows/mcp"],
          "env": {
            "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
          }
        }
      }
    }
    ```

    Alternatively, if you prefer to connect via the remote HTTP server instead:

    ```json JSON theme={"dark"}
    {
      "mcpServers": {
        "zenrows": {
          "serverUrl": "https://mcp.zenrows.com/mcp",
          "headers": {
            "Authorization": "Bearer YOUR_ZENROWS_API_KEY"
          }
        }
      }
    }
    ```

    <Note>
      If you already have other MCP servers configured, add the `zenrows` block alongside them inside the same `mcpServers` object. Don't replace the whole file.
    </Note>
  </Step>

  <Step title="Step 3: Reload Antigravity">
    Reload Antigravity to pick up the new config. Zenrows tools will be available to the editor immediately after reload.

    <Tip>For best results with MCP tool calls, switch your model to Claude Opus or Claude Sonnet via the model selector in the Antigravity interface.</Tip>
  </Step>
</Steps>

## Try It Out: Basic Usage

With Zenrows connected, your Antigravity 2.0 agents can fetch and work with live web data from any site.

**Scrape an anti-bot-protected page:**

```text wrap theme={"dark"}
Scrape https://www.scrapingcourse.com/antibot-challenge and summarize the page
```

**Sample result:**

```text wrap theme={"dark"}
I have successfully scraped the Antibot Challenge page at https://www.scrapingcourse.com/antibot-challenge using Zenrows (with JS rendering and premium proxy settings) and bypassed the challenge.

Page Summary

1. Page Purpose

The Antibot Challenge page on scrapingcourse.com serves as a test environment designed to verify...
[Truncated for brevity]
```

**Get the top 5 stories from Y Combinator:**

```text wrap theme={"dark"}
Scrape https://news.ycombinator.com and summarize the top 5 stories.
```

**Scrape the top 5 deals from an Amazon product listing:**

```text wrap theme={"dark"}
Scrape https://www.amazon.com/s?k=mechanical+keyboard and return the top 5 deals based on product name, price, and rating.
```

Google Antigravity 2.0 will call Zenrows, receive clean Markdown or structured data, and work with it directly in the agent session to give you a response based on your request.

## Advanced: Scheduled Scraping

Antigravity 2.0 has a built-in scheduler that runs agents on a recurring or one-off basis. Pair it with Zenrows and you can set up scraping tasks that run automatically in the background and deliver results without you being present.

### Setting Up a Scheduled Task

You can schedule a task two ways:

* **Via the Schedule panel (recommended):** Open the main Antigravity interface, click **Scheduled Tasks**, then **+ New**, and fill in the prompt and schedule.
  <img alt="Antigravity 2.0 Schedule Panel" />
* **Via the `/schedule` command:** Type `/schedule` in any conversation to set it up inline. For example, to track competitor job postings daily at 8:00 AM:

```text wrap theme={"dark"}
/schedule
Prompt: Scrape https://www.indeed.com/q-london-uk-jobs.html and extract all job postings.
Identify roles that signal buying intent such as new engineering hires, scaling DevOps teams,
or data infrastructure roles. List new postings only. If nothing is new, respond with [SILENT].
Schedule: Daily at 8:00 AM
```

<Tip>Each scheduled task runs as a fresh agent session. Include everything the agent needs directly in the prompt, including the URL, what to extract, and how to format the output.</Tip>

From the **Scheduled Tasks** panel in the main interface, you can view, edit, pause, or delete any scheduled task.

## Disable Zenrows Tools Temporarily

If you want to keep the server configured but stop Antigravity from using specific tools, use `disabledTools`:

```json JSON theme={"dark"}
{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": {
        "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
      },
      "disabledTools": ["browser_navigate", "browser_click"]
    }
  }
}
```

To disable the server entirely without removing the config:

```json JSON theme={"dark"}
{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": {
        "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
      },
      "disabled": true
    }
  }
}
```

## Troubleshooting

### Zenrows Tools Not Appearing After Saving the Config

* Confirm the file is saved to `~/.gemini/config/mcp_config.json`.
* Restart Antigravity to force a config reload.
* Check that `"disabled"` is not set to `true` in the server block.

### Agent Not Using Zenrows MCP

Antigravity may reach for a built-in tool instead of Zenrows. Instruct it to use Zenrows explicitly in your prompt:

```text wrap theme={"dark"}
Use Zenrows to scrape https://www.scrapingcourse.com/antibot-challenge with JavaScript rendering.
```

### Scraping Returns Empty or Blocked Content

Check that your Zenrows API key is valid and active in the [Zenrows dashboard](https://app.zenrows.com). If the key is correct, try explicitly specifying JavaScript rendering in your prompt as shown above.

### Model Error Due to Quota or Rate Limit

If you encounter quota or rate-limit errors, the issue is with your LLM provider, not Zenrows. Switch to a model with a higher quota, or wait for the limit to reset. Claude Opus and Sonnet tend to handle MCP tool call sequences more reliably. You can switch models via the model selector in the Antigravity interface without restarting.

### JSON Syntax Error on Startup

The config file must be valid JSON with no trailing commas or comments. Run it through a validator if Antigravity fails to load the server.

## Frequently Asked Questions

<Accordion title="Is Zenrows in the Antigravity MCP Store?">
  No. The MCP Store lists Google-curated integrations. Zenrows is added manually via the config file as shown in this guide.
</Accordion>

<Accordion title="Can I use the remote HTTP version of Zenrows MCP instead?">
  Yes. If you prefer not to install the local Node.js package, use the remote `serverUrl` approach instead. Both options expose the same tools:

  ```json JSON theme={"dark"}
  {
      "mcpServers": {
      "zenrows": {
          "serverUrl": "https://mcp.zenrows.com/mcp",
          "headers": {
              "Authorization": "Bearer YOUR_ZENROWS_API_KEY"
          }
      }
    }
  }
  ```
</Accordion>

<Accordion title="Can I connect multiple custom MCP servers alongside Zenrows?">
  Yes. Add each server as a named block inside `mcpServers`:

  ```json JSON theme={"dark"}
  {
      "mcpServers": {
          "zenrows": {
              "command": "npx",
              "args": ["-y", "@zenrows/mcp"],
              "env": {"ZENROWS_API_KEY": "<YOUR_ZENROWS_API_KEY>"}
          },
          "github": {
              "command": "npx",
              "args": ["-y", "@modelcontextprotocol/server-github"],
              "env": {"GITHUB_PERSONAL_ACCESS_TOKEN": "<YOUR_GITHUB_TOKEN>"}
          },
      }
  }
  ```
</Accordion>

<Accordion title="Is there a cost to using Zenrows MCP with Google Antigravity 2.0?">
  The Zenrows MCP server is free to connect. Each tool call is billed against your Zenrows subscription. `scrape` calls use Fetch credits, and `browser_*` calls use Browser Sessions credits. Monitor usage via your <a href="https://app.zenrows.com/overview#performance">Zenrows Analytics page</a>.
</Accordion>

<Accordion title="Does Zenrows work with Gemini and ChatGPT models in Antigravity?">
  Yes. Zenrows MCP works regardless of which model you've selected. Antigravity supports Gemini, ChatGPT, and Claude models, and all of them can call Zenrows tools. That said, Claude Opus and Sonnet tend to handle MCP tool call sequences most reliably.
</Accordion>

<Accordion title="Can I use Zenrows MCP per project only?">
  Yes. In Antigravity, each project has its own isolated settings. Click the settings gear icon next to your project in the left navigation bar, go to **Project Settings**, and configure which MCP tools are allowed for that project. This lets you enable Zenrows only where you need it.
</Accordion>

<Accordion title="Will Zenrows work with Antigravity's /browser command?">
  They serve different purposes. `/browser` launches Antigravity's built-in Chrome debugging session for interacting with pages you control locally. Zenrows is for fetching external pages, especially those protected by anti-bot systems, without needing a local browser. Use `/browser` for local development testing and Zenrows for live web data extraction.
</Accordion>

<Accordion title="Does the Zenrows config carry over to the Antigravity IDE?">
  Yes. The Antigravity 2.0 desktop app and the Antigravity IDE share the same config file at `~/.gemini/config/mcp_config.json`. Adding Zenrows here makes it available in both without any additional configuration. The Antigravity CLI uses a separate config path and requires its own setup.
</Accordion>


# How to Integrate Zenrows with Antigravity IDE
Source: https://docs.zenrows.com/integrations/antigravity-ide

Connect Zenrows with Antigravity IDE to give your coding agent live web access directly inside your editor.

Antigravity IDE is Google's AI-powered code editor. It combines a fully featured editor with an agent that operates across your editor, terminal, and browser, handling tab autocompletion, natural-language code commands, and autonomous task execution.

With Zenrows connected, your coding agent can fetch live documentation, pull real API specs, and scrape technical references without getting blocked.

## Prerequisites

* <a href="https://antigravity.google/download#antigravity-ide">Antigravity IDE</a> installed.
* A Zenrows API key from the [Zenrows dashboard](https://app.zenrows.com/fetch).

## Setup

<Steps>
  <Step title="Step 1: Open the configuration file">
    Antigravity IDE shares its configuration file with Antigravity 2.0. Open the MCP configuration file:

    ```
    ~/.gemini/config/mcp_config.json
    ```

    If the file does not exist yet, create it.
  </Step>

  <Step title="Step 2: Add the Zenrows MCP server">
    Add the following configuration, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

    ```json JSON theme={"dark"}
    {
      "mcpServers": {
        "zenrows": {
          "command": "npx",
          "args": ["-y", "@zenrows/mcp"],
          "env": {
            "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
          }
        }
      }
    }
    ```
  </Step>

  <Step title="Step 3: Restart Antigravity IDE">
    Save the file and restart Antigravity IDE to load the new configuration. Zenrows tools will be available to the agent immediately.

    <Tip>For best results with MCP tool calls, use Claude Opus or Claude Sonnet as your model via the model selector in the IDE.</Tip>
  </Step>
</Steps>

## Fetch live data in your editor

With Zenrows connected, ask the agent to fetch and work with live web content directly in the editor.

To scrape a page and summarize its contents:

```text wrap theme={"dark"}
Scrape https://www.scrapingcourse.com/antibot-challenge and tell me what the page is all about.
```

For a more focused use case:

```text wrap theme={"dark"}
Use Zenrows to scrape https://github.com/vercel/next.js/issues and find open issues related to hydration errors. Summarize the most common causes and suggested fixes.
```

## Using autonomous agents with Zenrows

Antigravity IDE includes an agent that handles longer, multi-step tasks across your editor, terminal, and browser. For scheduled tasks, parallel research workflows, or autonomous data extraction, see the [Zenrows + Antigravity 2.0 integration guide](/integrations/antigravity-20) for the full agent setup.

## Troubleshooting

### Zenrows tools not appearing after saving the config

1. Confirm the file is saved to `~/.gemini/config/mcp_config.json`.
2. Restart Antigravity IDE to force a config reload.
3. Check that the JSON is valid, with no trailing commas or comments.

### Agent not using Zenrows tools

Antigravity IDE may use its built-in browser tool instead of Zenrows. Instruct the agent explicitly:

```text wrap theme={"dark"}
Use Zenrows to scrape https://www.scrapingcourse.com/antibot-challenge with JavaScript rendering.
```

### JSON syntax error on startup

The configuration file must be valid JSON. Run it through a validator if Antigravity IDE fails to load the server.

## FAQ (Frequently asked questions)

<Accordion title="Can I use Zenrows alongside Antigravity IDE's built-in browser tool?">
  Yes. Both tools can coexist in the same session. Antigravity IDE picks its built-in browser tool by default for general browsing tasks, so when you need Zenrows specifically (for anti-bot bypass, JavaScript rendering, or residential proxies), instruct the agent explicitly to use Zenrows. See the [troubleshooting section](https://docs.zenrows.com/integrations/antigravity-ide#agent-not-using-zenrows-tools) for the exact prompt pattern.
</Accordion>

<Accordion title="Does every scraping request consume Zenrows credits?">
  Yes. Each request the agent makes through Zenrows counts against your plan credits. JavaScript rendering (`js_render`) and Premium Proxies consume more credits per request than a standard fetch. You can monitor your usage in real time from the [Zenrows dashboard](https://app.zenrows.com).
</Accordion>

<Accordion title="Can I use this integration on Windows?">
  Yes. The `mcp_config.json` file path uses Unix-style notation in this guide, but on Windows the equivalent path is `%USERPROFILE%\.gemini\config\mcp_config.json`. The rest of the setup is identical.
</Accordion>

<Accordion title="Will this work if I have other MCP servers already configured?">
  Yes. Add the `zenrows` block inside the existing `mcpServers` object alongside any other servers already defined. Each server runs independently, so adding Zenrows does not affect your other MCP connections.

  ```json JSON theme={"dark"}
  {
      "mcpServers": {
          "your-existing-server": { ... },
          "zenrows": {
              "command": "npx",
              "args": ["-y", "@zenrows/mcp"],
              "env": {
                  "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
              }
          }
      }
  }
  ```
</Accordion>


# Claude Agent SDK
Source: https://docs.zenrows.com/integrations/claude-agent-sdk

Build autonomous, multi-step AI agents that automate and scrape any site via Zenrows' MCP and the Claude Agent SDK. No orchestration code.

Combine the <a href="https://code.claude.com/docs/en/agent-sdk/overview">Claude Agent SDK</a> with the [Zenrows hosted MCP server](/mcp/overview) to build autonomous, multi-step agents that can read any website in real time. The Agent SDK has native support for MCP servers through its `mcp_servers` option, so plugging in Zenrows requires a single block of code, with no tool-use boilerplate, no scraping infrastructure, and no anti-bot tuning.

This guide covers the Claude Agent SDK. If you want to use Zenrows with the standard Anthropic API (Messages API, tool use, structured outputs, MCP connector), use the [Zenrows MCP server](/mcp/overview) with Anthropic's MCP connector.

## What is Claude Agent SDK?

The <a href="https://code.claude.com/docs/en/agent-sdk/overview">Claude Agent SDK</a> is Anthropic's Python and TypeScript library for building production AI agents using the same agent loop, tools, and context management that power Claude Code. It exposes two main entry points: `query()` for stateless one-shot agent runs, and `ClaudeSDKClient` for stateful, interactive sessions. Both support MCP servers, custom Python tools, subagents, hooks, and Claude Code's built-in tools (`Read`, `Write`, `Edit`, `Bash`, `WebSearch`, `WebFetch`, and more).

Agents built with the SDK are well-suited to tasks that need multiple steps, autonomous tool selection, or long-running workflows. Adding Zenrows MCP gives every agent in your workflow real-time access to any website, including JavaScript-heavy and anti-bot-protected ones.

### Key benefits

* **Native MCP support, zero glue code** <br />
  The Agent SDK's `mcp_servers` option takes the Zenrows server URL and your API key. The agent then sees Zenrows' tools (`scrape` and 30+ browser automation tools) automatically, with no tool use loop to maintain.

* **Autonomous, multi-step web tasks** <br />
  Agents can scrape one page, decide what to scrape next based on what they read, and chain together browser interactions like form fills and clicks across multiple pages, without you writing the orchestration loop.

* **Anti-bot bypass on every tool call** <br />
  Premium residential proxies, JavaScript rendering, and stealth fingerprinting are built-in standards. Your agents reach JavaScript-heavy SPAs, Cloudflare-protected sites, and geo-restricted pages without configuration.

* **Composes with the full Agent SDK feature set** <br />
  Zenrows tools work alongside Claude Code's built-in tools, custom Python tools, subagents, hooks, and permission gates. Pass them to a single agent or many; the integration is the same.

* **Hosted execution, no local installation** <br />
  The hosted MCP server runs on Zenrows infrastructure. Your agent code stays clean, with no proxy management, scraping retries, or headless browser orchestration in your codebase.

### Use cases

The Agent SDK and Zenrows combination unlocks a wide range of agent workflows:

<AccordionGroup>
  <Accordion title="Autonomous research agents" icon="magnifying-glass">
    Agents that visit multiple URLs, follow citations, and synthesize findings from across the web into a single brief.
  </Accordion>

  <Accordion title="Lead enrichment pipelines" icon="chart-line">
    Agents that take a company name, scrape the company website, find pricing pages and product launches, and write enriched profiles into your CRM.
  </Accordion>

  <Accordion title="Competitive monitoring" icon="bell">
    Agents that periodically check competitor pricing pages, blog posts, and changelogs, then surface what changed since the last run.
  </Accordion>

  <Accordion title="Multi-agent web automation" icon="robot">
    A research subagent that gathers information and hands off to a writer subagent that produces the final report, both pulling from live web data.
  </Accordion>

  <Accordion title="Form-filling and extraction agents" icon="file-pen">
    Agents that navigate multi-step checkout flows, login-walled forms, or job applications using Zenrows' browser tools.
  </Accordion>
</AccordionGroup>

## Getting started: Basic usage

A minimal working example: a single agent equipped with Zenrows MCP that can answer questions by reading any website in real time.

<Steps>
  <Step title="Install the Claude Agent SDK">
    Python 3.10 or later is required.

    ```shell theme={"dark"}
    pip install claude-agent-sdk
    ```
  </Step>

  <Step title="Create a .env file with your API keys">
    ```shell theme={"dark"}
    ZENROWS_API_KEY=your_zenrows_key
    ANTHROPIC_API_KEY=your_anthropic_key
    ```

    Find your Zenrows API key in the [Zenrows dashboard](https://app.zenrows.com/register). Find your Anthropic API key in the <a href="https://console.anthropic.com">Claude Console</a> under **Settings > API keys**.
  </Step>

  <Step title="Run the following script">
    ```python Python expandable theme={"dark"}
    import asyncio
    import os
    from claude_agent_sdk import query, ClaudeAgentOptions
    from dotenv import load_dotenv
    load_dotenv()

    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

    async def main() -> None:
        options = ClaudeAgentOptions(
            model="claude-sonnet-4-6",
            system_prompt=(
                "You are a research assistant with live web access. "
                "Use the Zenrows tools to scrape pages when you need real-time information."
            ),
            mcp_servers={
                "zenrows": {
                    "type": "http",
                    "url": "https://mcp.zenrows.com/mcp",
                    "headers": {
                        "Authorization": f"Bearer {ZENROWS_API_KEY}",
                    },
                }
            },
            allowed_tools=["mcp__zenrows__*"],
            permission_mode="bypassPermissions",
        )

        async for message in query(
            prompt="Visit https://huggingface.co/blog and summarize the three most recent posts.",
            options=options,
        ):
            if hasattr(message, "result"):
                print(message.result)

    asyncio.run(main())
    ```

    Here's what each part does:

    * `ClaudeAgentOptions` configures the agent. The `mcp_servers` dictionary registers the Zenrows hosted MCP server with `type: "http"`, the server URL, and an `Authorization: Bearer` header carrying your Zenrows API key as the OAuth token.
    * `allowed_tools=["mcp__zenrows__*"]` pre-approves every tool from the Zenrows server. The Agent SDK names MCP tools using the pattern `mcp__<server-name>__<tool-name>`, so the wildcard enables `mcp__zenrows__scrape`, `mcp__zenrows__browser_navigate`, and so on.
    * `permission_mode="bypassPermissions"` lets the agent call tools without an interactive approval prompt. Use this for unattended scripts and production agents. For interactive use, leave it at the default and the SDK will prompt before each tool call.
    * `query()` runs the agent loop. The agent discovers Zenrows' tools, picks the right one for the prompt, calls it, and synthesizes the answer. Messages stream back asynchronously; the final answer arrives as a message with a `result` attribute.
  </Step>
</Steps>

## Advanced usage: Building an autonomous research agent

The real power of the Agent SDK is letting an agent decide for itself which tools to call across multiple steps. With Zenrows MCP, that means the agent can autonomously read several pages, follow relevant links, and synthesize findings, all from a single prompt.

The example below builds a research agent that visits a blog page, identifies the most relevant linked articles, scrapes each one, and produces a synthesized report:

```python Python expandable theme={"dark"}
import asyncio
import os
from claude_agent_sdk import query, ClaudeAgentOptions
from dotenv import load_dotenv
load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    options = ClaudeAgentOptions(
        model="claude-opus-4-7",
        system_prompt=(
            "You are a thorough web researcher. When given a topic and a starting URL, "
            "visit the URL, identify the 3 most relevant linked articles, scrape each one, "
            "and produce a synthesized report covering key findings, common themes, and "
            "any contradictions. Cite each source with its URL."
        ),
        mcp_servers={
            "zenrows": {
                "type": "http",
                "url": "https://mcp.zenrows.com/mcp",
                "headers": {
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            }
        },
        allowed_tools=["mcp__zenrows__*"],
        permission_mode="bypassPermissions",
        max_turns=20,
    )

    async for message in query(
        prompt=(
            "Research recent developments in open-source LLMs. "
            "Start at https://huggingface.co/blog and follow the most relevant links."
        ),
        options=options,
    ):
        if hasattr(message, "result"):
            print(message.result)

asyncio.run(main())
```

<Tip>
  * Don't write an orchestration loop in your code. The agent decides on its own when to scrape, which URLs to follow, and when it has enough information to write the report. The SDK runs the multi-step loop automatically.
  * Use a capable model for multi-step tasks. `claude-opus-4-7` is the right default for autonomous workflows because it plans its next action based on what it has learned so far. For lighter tasks, `claude-sonnet-4-6` is faster and cheaper.
  * `max_turns=20` caps the total number of agent turns. This is a useful safety net that prevents indefinite loops. Adjust upward for deeper research, downward for tighter workflows.
  * Same MCP config as the basic example. The only things that changed are the model, the system prompt, and the prompt itself.
</Tip>

## Stateful sessions with ClaudeSDKClient

For agents that need to maintain conversation state across multiple user inputs (chatbots, interactive research assistants, follow-up questions), use `ClaudeSDKClient` instead of `query()`. The client supports bidirectional, interactive conversations with the same MCP configuration.

```python Python expandable theme={"dark"}
import asyncio
import os
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions
from dotenv import load_dotenv
load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    options = ClaudeAgentOptions(
        model="claude-sonnet-4-6",
        system_prompt="You are a web research assistant with access to Zenrows scraping tools.",
        mcp_servers={
            "zenrows": {
                "type": "http",
                "url": "https://mcp.zenrows.com/mcp",
                "headers": {
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            }
        },
        allowed_tools=["mcp__zenrows__*"],
        permission_mode="bypassPermissions",
    )

    async with ClaudeSDKClient(options=options) as client:
        # First turn
        await client.query("Visit https://huggingface.co/blog and summarize the top post.")
        async for message in client.receive_response():
            if hasattr(message, "result"):
                print(message.result)

        # Follow-up turn, sharing conversation history
        await client.query("Now scrape the linked GitHub repo and tell me what license it uses.")
        async for message in client.receive_response():
            if hasattr(message, "result"):
                print(message.result)

asyncio.run(main())
```

The client preserves conversation context between turns, so the agent remembers what it scraped in the first turn when answering the follow-up. Use `query()` for one-shot scripts and `ClaudeSDKClient` whenever you need multi-turn conversation state.

## Combining with custom Python tools

The Agent SDK also supports custom Python tools defined as in-process MCP servers. These can be used alongside the Zenrows hosted MCP server in the same agent.

```python Python expandable theme={"dark"}
import asyncio
import os
from claude_agent_sdk import (
    query,
    tool,
    create_sdk_mcp_server,
    ClaudeAgentOptions,
)
from dotenv import load_dotenv
load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

# Custom in-process tool
@tool("save_summary", "Save a research summary to a file", {"filename": str, "content": str})
async def save_summary(args):
    with open(args["filename"], "w") as f:
        f.write(args["content"])
    return {
        "content": [
            {"type": "text", "text": f"Saved to {args['filename']}"}
        ]
    }

local_server = create_sdk_mcp_server(
    name="local",
    version="1.0.0",
    tools=[save_summary],
)

async def main() -> None:
    options = ClaudeAgentOptions(
        model="claude-sonnet-4-6",
        system_prompt=(
            "Research the user's topic with Zenrows, then save your final summary "
            "using the save_summary tool."
        ),
        mcp_servers={
            "zenrows": {
                "type": "http",
                "url": "https://mcp.zenrows.com/mcp",
                "headers": {
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            },
            "local": local_server,
        },
        allowed_tools=[
            "mcp__zenrows__*",
            "mcp__local__save_summary",
        ],
        permission_mode="bypassPermissions",
    )

    async for message in query(
        prompt=(
            "Scrape https://huggingface.co/blog, summarize the top three posts, "
            "and save the summary to hf_summary.txt."
        ),
        options=options,
    ):
        if hasattr(message, "result"):
            print(message.result)

asyncio.run(main())
```

This pattern composes two MCP servers in the same agent: Zenrows for live web access and an in-process Python server for local file writes. The agent autonomously chains them, scraping the blog first and then calling `save_summary` with the finished text.

## Troubleshooting

### 424 failed dependency or 401 unauthorized from the MCP server

1. Confirm the `Authorization: Bearer YOUR_ZENROWS_API_KEY` header is present in the `headers` dict on the MCP server entry.
2. Verify your API key in the [Zenrows dashboard](https://app.zenrows.com/settings/api-keys) and confirm your subscription has remaining quota.
3. Confirm the server URL is exactly `https://mcp.zenrows.com/mcp` with `type: "http"`.

### Agent connects but never calls Zenrows tools

1. Confirm `allowed_tools` includes the Zenrows tool names. Without an entry matching `mcp__zenrows__*` (or specific tool names), the SDK does not expose the tools to the agent.
2. If `permission_mode` is at the default, the SDK prompts before each tool call. For unattended scripts, set `permission_mode="bypassPermissions"`.
3. Strengthen the system prompt to explicitly point the agent at the Zenrows tools, for example, "Use the Zenrows tools to fetch any URL the user mentions."

### Agent repeatedly scrapes the same page

Add explicit guidance in the system prompt about what to do once it has the data, for example, "After scraping each URL once, summarize and stop." For complex flows, lower `max_turns` to bound runaway agents.

### Page blocked or data missing in scraped content

If the agent connects and invokes tools correctly but the scraped content comes back blocked or incomplete, the issue is from the target page rather than the MCP connection. Add explicit guidance to your prompt, for example "Scrape this page using JavaScript rendering" or "Use Premium Proxies." See the [Fetch troubleshooting guide](/fetch/troubleshooting/troubleshooting-guide) for parameter-level fixes.

### `ANTHROPIC_API_KEY` not found

The Agent SDK reads `ANTHROPIC_API_KEY` from the environment. Confirm the variable is set in your shell or `.env` file. For Bedrock, Vertex AI, or Azure routing, see the <a href="https://code.claude.com/docs/en/agent-sdk/overview">Claude Agent SDK authentication docs</a>.

## FAQ (Frequently Asked Questions)

<Accordion title="What is the difference between this and the standard Anthropic Claude integration?">
  The standard Anthropic Claude integration uses the Messages API, tool use, structured outputs, and the MCP connector. In this pattern, you make one or two model calls and process the results, with the tool-use loop running in your code.

  The Claude Agent SDK is for autonomous, multi-step agents. The SDK runs the orchestration loop, decides which tools to call, and chains tool calls without you having to write the loop. It also ships with Claude Code's built-in tools (file editing, shell, web search/fetch) and supports subagents and hooks. Use the standard integration when you need direct control. Use the Agent SDK when you want autonomy, multi-step behavior, and Claude Code's broader toolset.
</Accordion>

<Accordion title="Which Claude models work with the Agent SDK?">
  All current Claude 4 models. For autonomous multi-step workflows, `claude-opus-4-7` is the strongest. For balanced cost-performance, `claude-sonnet-4-6` is a good default. For high-throughput, cost-sensitive workloads, `claude-haiku-4-5` is a faster, cheaper option.
</Accordion>

<Accordion title="Can I use the local stdio version of Zenrows MCP with the Agent SDK?">
  Yes. Use the `command` and `args` form instead of `type: "http"`:

  ```python Python theme={"dark"}
  mcp_servers={
      "zenrows": {
          "command": "npx",
          "args": ["-y", "@zenrows/mcp"],
          "env": {"ZENROWS_API_KEY": ZENROWS_API_KEY},
      }
  }
  ```

  The hosted HTTP URL is recommended because it requires no local installation and runs the same way in development and production. See the [Zenrows MCP overview](/mcp/overview) for the local stdio configuration.
</Accordion>

<Accordion title="Can multiple agents share the same Zenrows MCP configuration?">
  Yes. Build the `mcp_servers` dict and `allowed_tools` list once and reuse them across as many `ClaudeAgentOptions` instances as you need. Each agent gets independent tool calls, and Zenrows handles them all on the same hosted server.
</Accordion>

<Accordion title="Is streaming supported?">
  Yes. Both `query()` and `ClaudeSDKClient.receive_response()` are async iterators that stream messages as the agent works. Each message can be a tool call, a tool result, a partial text response, or the final result. Use `if hasattr(message, "result")` to filter for the final answer, or inspect all messages for full traces.
</Accordion>

<Accordion title="Can I combine Zenrows MCP with Claude Code's built-in tools (Read, Write, Bash, etc.)?">
  Yes. Add the built-in tool names to `allowed_tools` alongside the Zenrows wildcards. This is the right pattern for agents that need to scrape the web, write to local files, run shell commands, or edit code. See the Combining with custom Python tools section above for a similar pattern with in-process MCP servers.
</Accordion>

<Accordion title="How does this compare to the OpenAI Agents SDK?">
  The Claude Agent SDK and OpenAI Agents SDK solve similar problems with different design philosophies. The Claude Agent SDK ships as "Claude Code as a library," so it includes a rich set of built-in tools (file editing, shell, web search/fetch) and is tightly coupled with the Claude model family. The OpenAI Agents SDK is a thinner orchestration layer optimized for the OpenAI Responses API and GPT models, with a strong handoff and tracing story. Zenrows MCP works with both: pick the SDK that fits your model and toolset preferences. See the [OpenAI Agents SDK integration guide](/integrations/openai-agents-sdk) for the OpenAI equivalent of this doc.
</Accordion>


# Zenrows MCP Integration for Claude Code
Source: https://docs.zenrows.com/integrations/claude-code

Learn how to set up the Zenrows MCP server in Claude Code to scrape and extract structured web data from your terminal or workspace.

## Before You Start

You need:

<Columns>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">A **Zenrows account**</Card>
  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">Your **Zenrows API key**</Card>
</Columns>

<Card icon="code" href="https://nodejs.org/en/download/" title="Node.js">Latest version of **Node.js** installed on your machine</Card>

## Setup

### Option A: Terminal command (recommended)

Run the following command in your terminal, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

```bash theme={"dark"}
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp
```

### Option B: Project configuration file

Add Zenrows to your project's `.mcp.json` file:

```json theme={"dark"}
{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": {
        "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
      }
    }
  }
}
```

## Start scraping

Ask Claude Code to scrape a webpage or automate a browser workflow in plain English:

```
Get the product names and prices from https://www.scrapingcourse.com/ecommerce/.
```

```
Scrape https://www.scrapingcourse.com/antibot-bypass using Premium Proxies.
```

```
Fetch https://www.scrapingcourse.com/javascript-rendering/ with JavaScript rendering enabled and return the page as Markdown.
```

```
Open https://scrapingcourse.com/login in a browser, fill in the credentials, click submit, then extract the account details.
```

## Troubleshooting

**The `scrape` tool is not available**

If you used Option A, confirm the command ran without errors. If you used Option B, confirm the `.mcp.json` file is in the root of your project and contains valid JSON.

**Page content is missing or incomplete**

The page likely loads content dynamically. Ask Claude to enable JavaScript rendering:

```
Scrape https://www.scrapingcourse.com/javascript-rendering/ with `js_render: true`.
```

**Requests return a 403 or bot detection page**

Ask Claude to use Premium Proxies:

```
Scrape https://www.scrapingcourse.com/antibot-bypass using `premium_proxy: true`.
```

For more troubleshooting options, see the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview#troubleshooting).


# Zenrows MCP Integration for Claude Desktop
Source: https://docs.zenrows.com/integrations/claude-desktop

Install and configure the Zenrows MCP server in Claude Desktop to scrape websites and extract structured content from conversations.

## Before You Start

You need:

<Columns>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">A **Zenrows account**</Card>
  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">Your **Zenrows API key**</Card>
</Columns>

<Card icon="code" href="https://nodejs.org/en/download/" title="Node.js">Latest version of **Node.js** installed on your machine</Card>

## Setup

<Steps>
  <Step title="Step 1: Open the configuration file">
    Open the Claude Desktop configuration file for your operating system:

    * **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
    * **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`

    If the file does not exist yet, create it.
  </Step>

  <Step title="Step 2: Add the Zenrows MCP server">
    Paste the following into the configuration file, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

    ```json theme={"dark"}
    {
        "mcpServers": {
            "zenrows": {
                "command": "npx",
                "args": ["-y", "@zenrows/mcp"],
                "env": {
                    "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
                }
            }
        }
    }
    ```
  </Step>

  <Step title="Step 3: Restart Claude Desktop">
    Save the file and fully quit Claude Desktop. Reopen it to load the new configuration.
  </Step>
</Steps>

## Verify the integration

Once Claude Desktop restarts, the Zenrows tools are available. You can confirm it loaded correctly by asking Claude:

```
Which tools do you have available?
```

Claude lists the `scrape` tool and the `browser_*` tools if the integration is active.

## Start scraping

Ask Claude to scrape a webpage or automate a browser workflow in plain English:

```
Get the product names and prices from https://www.scrapingcourse.com/ecommerce/.
```

```
Scrape https://www.scrapingcourse.com/antibot-bypass using Premium Proxies.
```

```
Fetch https://www.scrapingcourse.com/javascript-rendering/ with JavaScript rendering enabled and return the page as Markdown.
```

```
Open https://scrapingcourse.com/login in a browser, fill in the credentials, click submit, then extract the account details.
```

## Troubleshooting

**The `scrape` tool does not appear after restarting**

Confirm the configuration file is saved in the correct location and contains valid JSON. You can validate the JSON syntax at <a href="https://jsonlint.com">jsonlint.com</a>.

**Page content is missing or incomplete**

The page likely loads content dynamically. Ask Claude to enable JavaScript rendering:

```
Scrape https://www.scrapingcourse.com/javascript-rendering/ with `js_render: true`.
```

**Requests return a 403 or bot detection page**

Ask Claude to use Premium Proxies:

```
Scrape https://www.scrapingcourse.com/antibot-bypass using `premium_proxy: true`.
```

For more troubleshooting options, see the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview#troubleshooting).


# How to Integrate Zenrows with Claude Managed Agents
Source: https://docs.zenrows.com/integrations/claude-managed-agents

Connect Zenrows to Claude Managed Agents and let your autonomous agents execute long-running scraping tasks without getting blocked. Anthropic manages the agent loop, sandbox, and infrastructure, while Zenrows handles the scraping layer, feeding your agent with real-time web context.

## What is Claude Managed Agents?

Claude Managed Agents is Anthropic's fully managed agent harness, currently in beta. Instead of building your own agent loop, tool execution layer, and runtime infrastructure, you get a managed environment where Claude runs autonomously: reading files, executing bash commands, browsing the web, and calling external tools via MCP.

You define the agent once (model, system prompt, tools, MCP servers), create an environment (a cloud sandbox or your own infrastructure), and start sessions via API. Claude works through the task, streams events back in real time, and you can steer or interrupt mid-execution.

## Why connect Zenrows to Claude Managed Agents?

* **Get past bot protection the built-in fetch can't bypass.** Managed Agents has a native web fetch tool, but it won't get through site protections. Zenrows handles anti-bot evasion so the agent fetches the actual page every time.
* **Cleaner output for token-heavy sessions.** Zenrows returns structured Markdown instead of raw HTML. In long-running sessions with many tool calls, this keeps token usage lean and the model focused on what matters.
* **Interact with dynamic pages without writing scraping code.** Some sites only surface their data after scrolling, clicking through paginated results, selecting dropdown filters, or waiting for JavaScript to render. Zenrows' browser tools handle all of that. The agent describes the interaction in plain English, and Zenrows executes it.

## What you can build

* **Autonomous competitive intelligence reports.** Kick off a session that scrapes competitor pricing, product pages, and job boards across multiple sites, synthesizes findings, and writes a structured report to a file in the sandbox, all while you do other work.
* **Long-running data extraction pipelines.** Send the agent a list of URLs and instruct it to scrape each one, extract specific fields, and write structured output to a CSV. The session runs asynchronously and streams progress events as each page completes.
* **Research workflows with live web access.** Give the agent a research brief and let it autonomously decide which pages to visit, what to extract, and how to synthesize findings, using Zenrows to reliably fetch any source regardless of protection.
* **Scheduled monitoring via webhooks.** Combine sessions with Managed Agents' webhook support to build monitoring workflows that trigger on a schedule, scrape target pages via Zenrows, and push results to your system when done.

## Prerequisites

1. A Claude API key from the <a href="https://platform.claude.com/settings/keys">Anthropic Console</a>.
2. Access to Claude Managed Agents (enabled by default for all API accounts).
3. A Zenrows API key from the [Zenrows dashboard](https://app.zenrows.com/fetch).

## Getting started

<Steps>
  <Step title="Add your Zenrows API key to a Credential vault">
    Sign in to your <a href="https://platform.claude.com">Anthropic Console</a> dashboard, navigate to **Managed Agents** in the left sidebar, and click **Credential vaults**.

    <img alt="anthropic-dashboard" />

    Follow the on-screen instructions to create your vault:

    * Name your vault.
    * Set the **Type** to **Bearer token**.
    * Paste the following Zenrows MCP connection URL as a Custom Server in the **MCP server** field:

    ```
    https://mcp.zenrows.com/mcp
    ```

    * Paste your Zenrows API key in the **Token** field.
    * Acknowledge credential sharing and click **Add credential**.

    <img alt="anthropic-console-create-credentials" />

    Anthropic matches credentials to MCP servers by URL at session time. As long as the URL in your vault matches the URL in the agent definition, the key is injected automatically and doesn't need to appear in the agent config.

    Copy and save the **vault ID**. You'll need it when starting sessions via Anthropic's SDK or CLI.
  </Step>

  <Step title="Create the agent">
    Go to **Agents** and click **New agent**. Switch to the **JSON** tab and paste the following:

    ```json JSON theme={"dark"}
    {
      "name": "Second research agent",
      "description": "A web research agent",
      "model": "claude-sonnet-4-6",
      "system": "You are a research agent. You can fetch and extract content from any website on the internet, write code, run commands, and use connected tools to complete the user's task end to end.",
      "mcp_servers": [
        {
          "name": "zenrows",
          "url": "https://mcp.zenrows.com/mcp",
          "type": "url"
        }
      ],
      "tools": [
        {
          "type": "agent_toolset_20260401"
        },
        {
          "type": "mcp_toolset",
          "mcp_server_name": "zenrows",
          "default_config": {
            "permission_policy": {
              "type": "always_allow"
            }
          }
        }
      ],
      "skills": []
    }
    ```

    You can freely modify:

    * `name` and `description` to match your use case
    * `model` to any supported Claude model
    * `system` to tailor the agent's behavior and tone

    Don't change the `mcp_servers` block. The `name` and `url` fields must stay exactly as shown above so the vault credential you created in Step 1 can be matched and injected automatically. Any change to the URL will break the connection.

    `always_allow` removes the approval gate so Zenrows tools execute automatically. The default is `always_ask`, which pauses the session before every tool call.

    Click **Create agent** to save the agent, then copy and store the **agent ID** from the Console.
  </Step>

  <Step title="Create your environment">
    Go to **Environments** in the Console sidebar. If no environment exists yet, click **New environment**, name it, and save it. Then click **Edit** at the top-right of the environment, set **Type** to **Unrestricted**, and click **Save changes**.

    <img alt="anthropic-console-environment" />

    This allows the sandbox to reach the Zenrows MCP server at `mcp.zenrows.com` without any additional host configuration.

    Without this setting, you'll see this error when starting a session:

    ```text wrap theme={"dark"}
    MCP server host(s) blocked by environment network policy: "zenrows" (mcp.zenrows.com).

    Add these hosts to the environment's allowed_hosts, or set allow_mcp_servers=true.
    ```

    Copy the **environment ID**. Environments are reusable across sessions and agents. Each session gets its own isolated sandbox, but the environment configuration is shared.
  </Step>

  <Step title="Test in the Console UI">
    Before moving to code, confirm Zenrows is connected and working directly in the Console.

    Go to **Sessions** in the left sidebar and click **New session**. Select your agent and environment, then send a prompt in the message field:

    ```text wrap theme={"dark"}
    Scrape https://www.scrapingcourse.com/antibot-challenge and summarize the page.
    ```

    In the session transcript, you'll see:

    * Each Zenrows tool call listed inline as it executes.
    * The tool's parameters on the right panel, showing which options Zenrows used.
    * The agent's final response once the session goes idle.

    Since Zenrows is connected via the vault, there's no need to mention it in the prompt. The agent selects the right tool automatically.

    ```text Example output wrap expandable theme={"dark"}
    Here's a summary of the page at https://www.scrapingcourse.com/antibot-challenge:

    🛡️ Antibot Challenge - ScrapingCourse.com

    Page Title: Antibot Challenge

    Purpose: This page is designed as a bot-detection challenge for learning...Truncated for brevity...

    What Happens:

    When you first visit, Cloudflare intercepts the request...Truncated for brevity

    Once the verification passes (either automatically or via user interaction), the actual page content is revealed.

    Page Content (after bypass):

    The page simply displays a success message:

    "You bypassed the Antibot challenge! 😄"

    ...Truncated for brevity...

    In short, this is an educational demo page from ScrapingCourse.com that tests whether a scraping tool can bypass Cloudflare's anti-bot protection. 

    The Zenrows browser successfully passed the challenge and revealed the success message.
    ```

    Once the session returns a clean summary, your integration is working. Copy your **Vault ID** from **Credential vaults**, then start sessions via the SDK or CLI.
  </Step>

  <Step title="Start an agent session via SDK">
    With your agent ID, environment ID, and vault ID in hand, start a session and send a prompt programmatically. This guide uses the Python SDK. SDKs are also available for TypeScript, Go, Java, C#, PHP, and Ruby. You can also use the `ant` CLI for the same. See the <a href="https://platform.claude.com/docs/en/managed-agents/sessions">Claude API official docs</a> for CLI examples.

    Install the SDK:

    ```bash theme={"dark"}
    pip install anthropic
    ```

    Create a `.env` file in your project root:

    ```bash .env theme={"dark"}
    ANTHROPIC_API_KEY=<Your_Anthropic_API_Key>
    AGENT_ID=<Your_Agent_ID>
    ENVIRONMENT_ID=<Your_Environment_ID>
    VAULT_ID=<Your_Vault_ID>
    ```

    Then run the session:

    ```python theme={null} theme={"dark"}
    # pip install anthropic python-dotenv
    import os
    import anthropic
    from dotenv import load_dotenv

    load_dotenv()

    # Initialize the Anthropic client with your API key
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

    # Create a new session for the agent, specifying the agent ID, environment ID, and vault ID
    session = client.beta.sessions.create(
        agent=os.environ["AGENT_ID"],
        environment_id=os.environ["ENVIRONMENT_ID"],
        vault_ids=[os.environ["VAULT_ID"]],
        title="Website research",
        betas=["managed-agents-2026-04-01"],
    )

    # Stream events from the session and send a user message to the agent to start the task
    with client.beta.sessions.events.stream(
        session.id,
        betas=["managed-agents-2026-04-01"],
    ) as stream:
        client.beta.sessions.events.send(
            session.id,
            events=[
                {
                    "type": "user.message",
                    "content": [
                        {
                            "type": "text",
                            "text": "Scrape https://www.scrapingcourse.com/ecommerce/, get the top 3 best deals based on price and ratings, and return a summary of the findings.",
                        }
                    ],
                }
            ],
            betas=["managed-agents-2026-04-01"],
        )
        # Process the events from the stream, printing messages from the agent and tool usage
        for event in stream:
            match event.type:
                case "agent.message":
                    for block in event.content:
                        print(block.text, end="")
                case "agent.tool_use":
                    print(f"\n[Tool: {event.name}]")
                case "session.status_idle":
                    print("\n\nAgent finished.")
                    break
    ```

    <Tip>While your session is running, you can monitor it live in the Console. Go to **Sessions** in the left sidebar, and you'll see the active session with its transcript, tool calls, and status updating in real time.</Tip>
  </Step>
</Steps>

## Troubleshooting

### httpx.ReadError: An existing connection was forcibly closed

The streaming connection dropped mid-response due to network instability. The session itself completed successfully on Anthropic's side. Try on a more stable connection, or disable any VPN, firewall, or antivirus that may be cutting long-lived HTTP connections.

### MCP server host(s) blocked by environment network policy

Your environment's networking policy is blocking outbound access to `mcp.zenrows.com`. Go to **Environments** in the Console, open your environment, click **Edit**, and set the networking Type to **Unrestricted**. Save the environment and restart your session.

### mcp\_authentication\_failed\_error in session events

The vault credential URL doesn't match the `url` declared in `mcp_servers`. Both must be exactly `https://mcp.zenrows.com/mcp`. Check for trailing slashes or typos in either field.

### mcp\_connection\_failed\_error in session events

The Zenrows MCP server was unreachable. The session continues without Zenrows tools. Check your network settings and confirm the endpoint is reachable. The connection is retried automatically on the next session interaction.

### Session starts but agent doesn't call Zenrows tools

Confirm that the `mcp_toolset` entry references `mcp_server_name: zenrows` and that `zenrows` matches the `name` in your `mcp_servers` array exactly. A mismatch means the toolset is declared but not linked to any server.

### Agent pauses before every Zenrows tool call

The MCP toolset defaults to `always_ask`, which pauses the session and waits for your approval before each tool call. Set `always_allow` on the toolset to let Zenrows tools execute automatically:

```json JSON theme={"dark"}
{
  "type": "mcp_toolset",
  "mcp_server_name": "zenrows",
  "default_config": {
    "permission_policy": {
      "type": "always_allow"
    }
  }
}
```

### Tools.1.permission\_policy: Extra inputs are not permitted

The `permission_policy` field must be nested inside `default_config`, not placed directly on the toolset object. See the correct structure above.

### Missing beta header error

All Managed Agents requests require `managed-agents-2026-04-01` in the beta header. The Python and TypeScript SDKs accept this via the `betas` parameter on each call. If you're using raw HTTP, add `anthropic-beta: managed-agents-2026-04-01` to every request header.

## Further reading

* <a href="https://platform.claude.com/docs/en/managed-agents/overview">Claude Managed Agents overview</a>
* <a href="https://platform.claude.com/docs/en/managed-agents/mcp-connector">MCP connector</a>
* <a href="https://platform.claude.com/docs/en/managed-agents/vaults">Authenticate with vaults</a>
* <a href="https://platform.claude.com/docs/en/managed-agents/events-and-streaming">Session event stream</a>
* [Zenrows MCP documentation](/mcp/overview)

## FAQ (Frequently asked questions)

<Accordion title="Can I reuse the same environment across different agents?">
  Yes. Environments are created independently of agents and referenced by ID at session creation. Multiple sessions from different agents can all reference the same environment configuration. Each session still gets its own isolated sandbox (sessions do not share filesystem state), but the environment definition (networking, pre-installed packages, sandbox type) is shared.
</Accordion>

<Accordion title="Is Claude Managed Agents generally available?">
  It's currently in beta. All requests require the `managed-agents-2026-04-01` beta header. See the <a href="https://platform.claude.com/docs/en/managed-agents/overview">overview</a> for current access details.
</Accordion>

<Accordion title="Why store the Zenrows API key in a vault rather than in the agent definition?">
  Agent definitions are reusable and workspace-scoped. Putting credentials there would expose them to anyone with API access to the workspace. Vaults keep secrets separate: the agent definition stays clean, and each session injects the right credential at runtime.
</Accordion>

<Accordion title="Can I use Zenrows in multiagent sessions?">
  Yes. Vault credentials apply to every thread in a multiagent session. Any agent whose definition declares the Zenrows MCP server will authenticate with the vault credentials provided at session creation. See <a href="https://platform.claude.com/docs/en/managed-agents/multi-agent">Multiagent sessions</a>.
</Accordion>

<Accordion title="Can I restrict which Zenrows tools the agent can use?">
  Yes. Use the `default_config` and `configs` pattern on the `mcp_toolset` entry. Set `default_config.enabled` to `false` and explicitly enable only the tools you want:

  ```json JSON theme={"dark"}
  {
      "type": "mcp_toolset",
      "mcp_server_name": "zenrows",
      "default_config": {"enabled": false},
      "configs": [
          {"name": "zenrows_scrape", "enabled": true}
      ]
  }
  ```
</Accordion>

<Accordion title="What are the costs involved in running a Zenrows-powered Managed Agents session?">
  Connecting Zenrows via MCP is free. However, there are two separate costs from running an agent that calls the Zenrows MCP. Anthropic bills for the session itself based on token usage across all model calls. Zenrows bills separately against your subscription for each tool call: `scrape` calls use Fetch credits and `browser_*` calls use Browser Sessions credits. Monitor Zenrows usage on your [Zenrows Analytics page](https://app.zenrows.com/overview#performance).
</Accordion>

<Accordion title="Can I run sessions on my own infrastructure instead of Anthropic's cloud?">
  Yes. Claude Managed Agents supports self-hosted sandboxes to meet compliance or data residency requirements. The Zenrows MCP setup is identical: Zenrows connects via the remote HTTP server regardless of where the sandbox runs. See <a href="https://platform.claude.com/docs/en/managed-agents/self-hosted-sandboxes">Self-hosted sandboxes</a>.
</Accordion>


# How to Integrate Clay with Zenrows
Source: https://docs.zenrows.com/integrations/clay

Learn how to connect Zenrows with Clay for no-code web scraping. Extract website data directly into Clay spreadsheets for lead enrichment.

Zenrows provides powerful web scraping capabilities that integrate seamlessly with Clay, allowing you to extract data from websites without writing code. The integration enables the automated collection of structured data from any website directly into Clay's spreadsheet interface.

## What is Clay?

<a href="https://www.clay.com/">Clay</a> is a powerful no-code data platform that functions as an intelligent spreadsheet with built-in automation capabilities. It enables users to collect, transform, and analyze data from various sources without requiring programming knowledge, making complex data operations accessible to everyone.

With its extensive integration ecosystem, Clay allows users to connect to hundreds of data sources, including web scraping tools like Zenrows, CRM systems, and marketing platforms.

## Watch the Video Tutorial

Learn how to set up the Clay ↔ Zenrows integration step-by-step by watching this video tutorial:

<iframe title="Zenrows ↔ Clay" />

## Zenrows Configuration Options

Zenrows offers numerous configuration options to customize your web scraping experience. Here's a detailed breakdown of each option available in the Clay integration:

### Basic Configuration

* **Scrape URL:** The target webpage URL you want to scrape. In Clay, this is typically linked to a column containing URLs.
* **Autoparse:** When enabled, Zenrows automatically parses the HTML structure of the page, converting it into structured data.
* **Wait Time (ms):** Defines the delay in milliseconds after the initial page load before capturing the content. Essential for pages that load content dynamically.

### Advanced Configuration

* **Render JavaScript:** When enabled, Zenrows uses a headless browser to render JavaScript on the page, which is essential for modern websites where content is loaded dynamically.
* **Premium Proxy:** Enables the use of high-quality, residential IP addresses for protected access.
* **Proxy Country:** Specifies a two-letter country code (e.g., "US", "UK", "DE") to route requests through servers in that location. Enables access to region-specific content that may be unavailable from your location.
* **Auto-update:** Updates data automatically when your table changes.
* **Only run if:** Set conditions for when the scraper should run.

## Getting Started: Real-World Example

We'll use Zenrows to scrape property information from Zillow and organize it in Clay.

### Step 1: Create a New Workbook in Clay

1. Log in to your Clay account at <a href="https://www.clay.com/">[https://www.clay.com](https://www.clay.com)</a>.

2. Click **+ New Workbook** in the top right corner.

<Frame>
  <img alt="create new workbook clay" />
</Frame>

3. Name your workbook (e.g., "Real Estate Analysis") and click **+ New blank table** button.

<Frame>
  <img alt="name webhook and create new table on clay" />
</Frame>

### Step 2: Set Up Your Table Structure

1. Inside your new workbook, you'll see a blank table.

2. Create the following columns by clicking on the **+ Add column** icon to the right of the default column:
   * Property URL
   * City
   * State
   * Zipcode
   * Price
   * Description

3. Add these example Zillow property URLs in the "Property URL" column:
   * `https://www.zillow.com/homedetails/110-Rosewell-Mdw-Syracuse-NY-13214/31725398_zpid/`
   * `https://www.zillow.com/homedetails/27-Central-St-Greenlawn-NY-11740/59566735_zpid/`
   * `https://www.zillow.com/homedetails/6-Dean-St-Farmingdale-NY-11735/32614340_zpid/`
   * `https://www.zillow.com/homedetails/4848-Mount-Vernon-Blvd-Hamburg-NY-14075/30342319_zpid/`
   * `https://www.zillow.com/homedetails/251-Post-Ave-Rochester-NY-14619/30882345_zpid/`
   <Frame>
     <img alt="table with property url clay" />
   </Frame>

### Step 3: Add the Zenrows Integration

1. Click on the **+ Add Column** and select **Add Enrichment**.

2. In the search bar, type "ZenRows".

3. Under Integrations, select **Run Zenrows Scrape**.

<Frame>
  <img alt="zenrows clay integration" />
</Frame>

### Step 4: Configure the Zenrows Scraper

1. **Account Selection:**
   * By default, "Clay-managed Zenrows account" will be selected.
   * Alternatively, click **Add account** to use your own Zenrows API key.

2. **Configure Settings:**
   * **Scrape URL:** Select the "Property URL" column from your table.
   * **Autoparse:** Toggle ON to automatically parse the webpage content.
   * **Wait Time (ms):** Set to 5000 (5 seconds) to ensure the page loads fully.
   * **Render JavaScript:** Toggle ON as Zillow relies heavily on JavaScript.
   * **Premium Proxy:** Toggle ON for improved success rates on Zillow.

3. **Run Settings:**
   * **Auto-update:** Toggle ON to automatically update data when your table changes

4. Click **Save and run 5 rows in this view** button.

<Frame>
  <img alt="zenrows configuration on clay" />
</Frame>

### Step 5: Map the Scraped Data to Your Columns

After Zenrows scrapes the data, you need to map the relevant information to your specific columns:

1. **For the City column:**
   * Click on the header of your "City" column.
   * Select **Edit column**.
   * Ensure "Text" is selected as the Data type.
   * Click in the value field and type `/` to insert column.
   * From the dropdown that appears, select **Run Zenrows Scrape**.
   * Select "city" from the properties list (or click "Insert all properties" to see all available data).
   * Click **Save settings**.

2. **Repeat for other columns:**
   * **State:** Map to "state" property
   * **Zipcode:** Map to "zipcode" property
   * **Price:** Map to "price" property
   * **Description:** Map to "description" property

Once mapped, your columns will automatically populate with the corresponding data from Zillow.

<Frame>
  <img alt="zillow scraping on clay using zenrows" />
</Frame>

## Integration Benefits & Applications

This Zenrows-Clay integration enables professionals across industries, including real estate, e-commerce, recruitment, marketing, etc., to automate data collection processes that would otherwise require significant manual effort or technical expertise.

By eliminating development requirements and handling complex challenges like JavaScript rendering and anti-bot protections automatically, this integration allows you to focus on data analysis rather than acquisition.

## Frequently Asked Questions (FAQs)

<Accordion title="Can I use my existing Zenrows API key with Clay?">
  Yes, you can use your own Zenrows API key with Clay. When configuring the integration, select "Add account" instead of using the Clay-managed Zenrows account. This gives you more control over your scraping operations and allows you to use your existing subscription limits and settings.
</Accordion>

<Accordion title="How do I troubleshoot if I'm not getting any data from a website?">
  If you're not receiving data, first ensure JavaScript rendering is enabled, as most modern websites require it. Next, increase the Wait Time to 5000-8000ms to allow dynamic content to load fully.

  For websites with strong anti-bot measures, enable both Premium Proxy and Anti-Bot options. Finally, verify that the site's structure hasn't changed by testing the scrape on a single URL before scaling up.
</Accordion>

<Accordion title="How frequently can I update the scraped data in Clay?">
  Data update frequency depends on your configuration and account type. With Auto-update enabled, Clay will refresh the data whenever the source table changes. If you're using Clay's managed Zenrows account, be mindful of the shared usage limits.

  For high-frequency scraping needs, consider using your own Zenrows API key, which allows you to operate within your personal subscription limits. For websites that change frequently, you can set up scheduled updates in Clay to keep your data current without manual intervention.
</Accordion>


# How to Integrate Zenrows with CrewAI
Source: https://docs.zenrows.com/integrations/crewai

Connect the Zenrows MCP to CrewAI and give your agent crews reliable access to any website, including pages behind anti-bot protection and heavy JavaScript. No custom tool code required.

Connect <a href="https://www.crewai.com">CrewAI</a> to the Zenrows MCP server and every agent in your crew can read live web pages, including the ones that return a block page to an ordinary HTTP request. No tool class to write and nothing to publish, just an MCP connection your crew opens at startup.

## What Is CrewAI?

<a href="https://www.crewai.com">CrewAI</a> is an open-source Python framework for orchestrating teams of autonomous AI agents. You define agents with a role, a goal, and a backstory, give each one tools, and assign tasks. CrewAI handles delegation, sequencing, and the hand-off of one agent's output into the next agent's input.

Its `crewai-tools` package ships integrations for search, databases, file systems, and scraping, plus `MCPServerAdapter`, which pulls tools from any MCP server into a crew at runtime.

## Why Connect Zenrows to CrewAI?

* **Agents that don't get blocked:** a research agent calling `requests` or a plain HTTP tool gets a challenge page from any site behind Cloudflare or DataDome, then reasons confidently over the wrong content. Zenrows returns the real page.
* **No tool code to maintain:** `MCPServerAdapter` discovers the tools at runtime, so there's no `BaseTool` subclass, no Pydantic schema, and no package of your own to version.
* **Markdown instead of HTML:** the `scrape` tool returns clean Markdown by default, which costs far fewer tokens than raw HTML and gives the agent less noise to reason through.
* **Every agent in the crew shares one connection:** open the adapter once and pass `tools` to as many agents as you like.
* **Model-agnostic:** the tools arrive as ordinary CrewAI tools, so they work with whichever LLM your crew is configured to use.

## What You Can Build with CrewAI and Zenrows

* **A research crew with a scraper and an analyst:** one agent gathers pages, a second synthesizes them into a brief, and the scraping agent never returns a challenge page instead of content.
* **Competitive price monitoring:** a crew that walks a list of competitor product pages, extracts prices, and reports the deltas on a schedule.
* **Lead enrichment:** an agent reads a prospect's site, careers page, and pricing page, and hands a structured profile to a writer agent that drafts the outreach.
* **Content pipelines:** a crew that pulls source articles, extracts the substance, and produces a draft, with the sources fetched reliably rather than best-effort.

## Prerequisites

* Python 3.12 or 3.13, and an existing CrewAI project. `crewai-tools` requires
  `>=3.10,<3.14` and `mcpadapt` requires `>=3.12`, so the MCP path needs a
  version in that overlap. On Python 3.14, pip silently resolves
  `crewai-tools` down to a placeholder 0.0.1 release that has no `mcp` extra.
* An LLM provider key configured for CrewAI (`OPENAI_API_KEY` by default).
* A Zenrows API key from your <a href="https://app.zenrows.com/settings/api-keys">Zenrows dashboard</a>.

## Setup

<Steps>
  <Step title="Install the MCP extra">
    MCP support is an optional extra, not part of the base `crewai-tools` install:

    ```bash theme={"dark"}
    pip install 'crewai-tools[mcp]'
    ```

    The extra pulls in `mcp` and `mcpadapt`, which `MCPServerAdapter` needs. Without it, constructing the adapter offers to install the packages for you and raises an `ImportError` if you decline.
  </Step>

  <Step title="Connect to the Zenrows MCP server">
    Point `MCPServerAdapter` at `https://mcp.zenrows.com/mcp` and pass your key as a Bearer token.

    ```python Python theme={"dark"}
    import os
    from crewai_tools import MCPServerAdapter

    server_params = {
        "url": "https://mcp.zenrows.com/mcp",
        "transport": "streamable-http",
        "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
    }

    with MCPServerAdapter(server_params) as tools:
        print([tool.name for tool in tools])
    ```

    <Warning>
      **`"transport": "streamable-http"` is required.** `MCPServerAdapter` passes your dictionary to `mcpadapt`, which defaults to the deprecated SSE transport when no transport is given. Omit the line and the connection is attempted over SSE against an endpoint that speaks Streamable HTTP, and it fails. CrewAI's own docstring documents STDIO and SSE only, so this is not discoverable from the framework's documentation.
    </Warning>

    Use the adapter as a context manager, as above, and the server connection closes with the block. If you construct it directly instead, call `stop()` in a `finally` block once the crew has finished.
  </Step>

  <Step title="Give the tools to an agent and run the crew">
    The adapter yields ordinary CrewAI tools, so they go straight onto an agent.

    ```python Python theme={"dark"}
    import os
    from crewai import Agent, Crew, Task
    from crewai_tools import MCPServerAdapter

    server_params = {
        "url": "https://mcp.zenrows.com/mcp",
        "transport": "streamable-http",
        "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
    }

    with MCPServerAdapter(server_params, "scrape") as tools:
        researcher = Agent(
            role="Web Researcher",
            goal="Read live web pages and report exactly what they say",
            backstory=(
                "You read pages through Zenrows, which returns the real content "
                "even when a site blocks ordinary requests."
            ),
            tools=tools,
            verbose=True,
        )

        task = Task(
            description=(
                "Read https://www.scrapingcourse.com/ecommerce/ and list three "
                "products with their prices."
            ),
            expected_output="A Markdown list of three product names with prices.",
            agent=researcher,
        )

        result = Crew(agents=[researcher], tasks=[task]).kickoff()
        print(result)
    ```

    The second argument, `"scrape"`, filters the tool set. The next section explains why that matters.
  </Step>
</Steps>

## Filter to the Tools Your Crew Needs

The Zenrows MCP server exposes three families of tools: `scrape`, the `batch_*` job tools, and the `browser_*` session tools. See the [MCP overview](/mcp/overview) for the full list.

Every tool you load contributes its schema to the prompt on every request. A crew that only reads pages does not need the batch job lifecycle or browser session control, so pass the names you want as positional arguments:

```python Python theme={"dark"}
with MCPServerAdapter(server_params, "scrape") as tools:
    ...
```

Leave the names out and the crew receives everything the server offers.

## Raise the Timeout for Slow Pages

Keys in `server_params` other than `transport` are forwarded to the underlying Streamable HTTP client, whose request timeout defaults to 30 seconds. A page that needs JavaScript rendering or an anti-bot bypass can take longer than that, and the call fails on the client side while the request is still in flight.

```python Python theme={"dark"}
server_params = {
    "url": "https://mcp.zenrows.com/mcp",
    "transport": "streamable-http",
    "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
    "timeout": 120,
}
```

`MCPServerAdapter`'s own `connect_timeout` argument is separate and covers establishing the connection, not individual tool calls:

```python Python theme={"dark"}
MCPServerAdapter(server_params, connect_timeout=60)
```

## Troubleshooting

### ImportError or a Prompt to Install `mcp`

The MCP extra is not installed. Run `pip install 'crewai-tools[mcp]'`. Installing `mcp` alone is not enough, because `MCPServerAdapter` also needs `mcpadapt`.

### The Connection Hangs or Fails Immediately

The transport is missing. Confirm `"transport": "streamable-http"` is present in `server_params`. Without it the client attempts SSE, which the Zenrows MCP endpoint does not serve.

### Authentication Errors from the Server

Check the `Authorization` header is exactly `Bearer YOUR_ZENROWS_API_KEY`, with a single space and no trailing whitespace. Verify the key in your <a href="https://app.zenrows.com/settings/api-keys">Zenrows dashboard</a>.

### Tool Calls Time Out on Heavy Pages

Raise `timeout` in `server_params`, as described above. The default is 30 seconds, which is short for a rendered or protected page.

### The Agent Ignores the Scraping Tool

Make the task description name the URL explicitly and state that the page must be read rather than recalled. Filtering to `scrape` also helps, because a crew offered a dozen tools has more ways to choose wrongly.

### The Prompt Grows Too Large

Every loaded tool's schema is sent with each request, and page content is added on top. Filter the tool set to `scrape`, or move to a model with a larger context window.

## Further Reading

* <a href="https://docs.crewai.com/en/mcp/overview">CrewAI MCP Integration</a>
* <a href="https://docs.crewai.com/en/concepts/agents">CrewAI Agents</a>
* [Zenrows MCP Documentation](/mcp/overview)
* [Fetch API Reference](/fetch/api-reference)

## Frequently Asked Questions

<Accordion title="Do I need to write a custom CrewAI tool for Zenrows?">
  No. `MCPServerAdapter` discovers the Zenrows tools at runtime and adapts them into CrewAI tools, so there is no tool class to write, test, or keep in step with the API.
</Accordion>

<Accordion title="Why does the connection fail without the transport line?">
  `MCPServerAdapter` hands your dictionary to `mcpadapt`, which reads the transport key and falls back to SSE when it is absent. SSE is the deprecated MCP transport and is not what the Zenrows endpoint serves, so the connection cannot complete. Setting `"transport": "streamable-http"` selects the right client.
</Accordion>

<Accordion title="Can several agents in one crew share the same Zenrows tools?">
  Yes. Open the adapter once and pass the same `tools` value to each agent that needs web access. One connection serves the whole crew.
</Accordion>

<Accordion title="Can I use Zenrows alongside other MCP servers?">
  Yes. Create one `MCPServerAdapter` per server and combine the tool lists when constructing an agent.
</Accordion>

<Accordion title="Does this work with a local Zenrows MCP server instead?">
  Yes. `MCPServerAdapter` also accepts `StdioServerParameters` for a local subprocess. See the [MCP overview](/mcp/overview) for the local server configuration.
</Accordion>


# Zenrows MCP Integration for Cursor
Source: https://docs.zenrows.com/integrations/cursor

Add the Zenrows MCP server to Cursor so your AI coding assistant can scrape websites and extract structured data inside the editor.

## Before you start

You need:

<Columns>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">A **Zenrows account**</Card>
  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">Your **Zenrows API key**</Card>
</Columns>

<Card icon="code" href="https://nodejs.org/en/download/" title="Node.js">Latest version of **Node.js** installed on your machine</Card>
<Card icon="robot" href="https://www.cursor.com" title="Cursor">**Cursor** installed</Card>

## Setup

<Steps>
  <Step title="Step 1: Open the configuration file">
    Open the MCP configuration file for your setup:

    * **Global (all projects):** `~/.cursor/mcp.json`
    * **Project-specific:** `.cursor/mcp.json` in your project root

    If the file does not exist yet, create it.
  </Step>

  <Step title="Step 2: Add the Zenrows MCP server">
    Paste the following into the configuration file, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

    ```json theme={"dark"}
    {
    "mcpServers": {
        "zenrows": {
            "command": "npx",
            "args": ["-y", "@zenrows/mcp"],
            "env": {
                "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
                }
            }
        }
    }
    ```
  </Step>

  <Step title="Step 3: Restart Cursor">
    Save the file and restart Cursor to load the new configuration.
  </Step>
</Steps>

## Start scraping

Ask Cursor's AI assistant to scrape a webpage or automate a browser workflow in plain English:

```
Get the product names and prices from https://www.scrapingcourse.com/ecommerce/.
```

```
Scrape https://www.scrapingcourse.com/antibot-bypass using Premium Proxies.
```

```
Fetch https://www.scrapingcourse.com/javascript-rendering/ with JavaScript rendering enabled and return the page as Markdown.
```

```
Open https://scrapingcourse.com/login in a browser, fill in the credentials, click submit, then extract the account details.
```

## Troubleshooting

**The `scrape` tool is not available**

Confirm the configuration file is saved in the correct location and contains valid JSON. You can validate the JSON syntax at <a href="https://jsonlint.com">jsonlint.com</a>. Then restart Cursor.

**Page content is missing or incomplete**

The page likely loads content dynamically. Ask the assistant to enable JavaScript rendering:

```
Scrape https://www.scrapingcourse.com/javascript-rendering/ with `js_render: true`.
```

**Requests return a 403 or bot detection page**

Ask the assistant to use Premium Proxies:

```
Scrape https://www.scrapingcourse.com/antibot-bypass using `premium_proxy: true`.
```

For more troubleshooting options, see the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview#troubleshooting).


# How to Integrate Dify with Zenrows
Source: https://docs.zenrows.com/integrations/dify

Install the official Zenrows plugin from the Dify Marketplace and add Fetch, Extract, and Batch tools to your Dify workflows and agents, including pages behind anti-bot protection and heavy JavaScript.

Install the official Zenrows plugin from the <a href="https://marketplace.dify.ai/plugin/zenrows/zenrows">Dify Marketplace</a> and your Dify workflows and agents can read live web pages, including the ones that return a block page to an ordinary HTTP request. There's no custom code to write, no API schema to import, and no HTTP node to configure: the plugin adds five ready-made tools to the Dify tool picker.

Watch the plugin in action, from the marketplace listing to a workflow that summarizes a page:

<iframe title="Zenrows plugin for Dify" />

## What Is Dify?

<a href="https://dify.ai">Dify</a> is an open-source platform for building LLM applications. You assemble workflows, chatflows, and agents on a visual canvas by connecting nodes such as LLM calls, knowledge retrieval, code, and tools. It runs as a hosted service at `cloud.dify.ai` or self-hosted on your own infrastructure.

Dify's plugin system lets third parties ship tools that show up next to the built-in ones. The Zenrows plugin is one of them, published and maintained by Zenrows.

## Why Connect Zenrows to Dify?

* **Workflows that don't get blocked:** a plain HTTP request to a site behind Cloudflare, DataDome, or similar gets a challenge page, and the LLM node then reasons over the wrong content. Zenrows returns the real page.
* **Markdown by default:** the Fetch tool returns clean Markdown, which costs far fewer tokens than raw HTML when you pass it to an LLM node.
* **Structured data without parsing code:** the Extract tool returns JSON fields, so you don't need a Code node to pull prices or titles out of HTML.
* **Many URLs in one job:** the Batch tools submit up to 1,000 URLs per call as an asynchronous job (a plugin limit), instead of looping a Fetch node.
* **One credential for the workspace:** you authorize the plugin once, and every member of the workspace can use it in their apps.

## Available Tools

The plugin adds five tools under the **Zenrows** provider:

| Tool | What it does |
| - | - |
| **Fetch** | Retrieves one page as Markdown, HTML, plain text, or PDF. A separate **Screenshot** option returns an image instead. Supports JavaScript rendering, Premium Proxies, proxy country, wait conditions, and JavaScript instructions. |
| **Extract** | Returns structured JSON from a page. Pick a method: `auto` (site-tailored, falls back to autoparse), `autoparse` (any site), `css` (your own CSS selectors), or `outputs` (built-in filters such as emails, links, and tables). |
| **Batch Create** | Submits 1 to 1,000 URLs per call as one asynchronous job and returns a job ID. The 1,000 limit comes from the plugin; the Batch API itself accepts up to 100,000 URLs per job. It can also wait up to about 100 seconds for a small job to finish. |
| **Batch Status** | Reports a job's progress: completed, successful, and failed task counts, failure reasons, and credits spent. |
| **Batch Results** | Collects the scraped content of a finished job, 25 results by default and up to 200 per call. |

Use Fetch when you want the page, Extract when you want fields from the page, and Batch when you have more URLs than one request should carry.

## Prerequisites

* A Dify workspace, either on <a href="https://cloud.dify.ai">Dify Cloud</a> or self-hosted. Self-hosted instances need Dify 1.11.4 or later.
* A Zenrows API key from your <a href="https://app.zenrows.com/settings/api-keys">Zenrows dashboard</a>.
* For self-hosted Dify: outbound HTTPS access to `api.zenrows.com` and `async.api.zenrows.com`. The plugin needs no inbound connections.

## Set Up the Zenrows Plugin

<Steps>
  <Step title="Install the plugin from the Dify Marketplace">
    Open the <a href="https://marketplace.dify.ai/plugin/zenrows/zenrows">Zenrows plugin page</a> and click **Install**. Dify opens your workspace and installs the plugin. You can also search for **Zenrows** from **Marketplace** in the Dify sidebar.

    <Frame>
      <img alt="The Zenrows plugin page on the Dify Marketplace, showing the Install button and the five tools it provides" />
    </Frame>

    When installation finishes, Dify shows an **Installation successful** dialog, and Zenrows appears under **Integrations** > **Tools** > **Tool Plugin**.

    <Frame>
      <img alt="Dify's Installation successful dialog for the Zenrows plugin, version 0.1.0" />
    </Frame>
  </Step>

  <Step title="Authorize the plugin with your API key">
    In **Integrations** > **Tools** > **Tool Plugin**, click the **Zenrows** card, then click **API Key Authorization Configuration**. Fill in the dialog:

    * **Authorization Name**: any label, for example `Zenrows`.
    * **Zenrows API Key**: paste your key from the Zenrows dashboard.
    * **Who can use**: leave **All team members**, or restrict it.

    Click **Save**. Dify validates the key with Zenrows before saving it, and the check doesn't consume credits.

    <Frame>
      <img alt="The API Key Authorization Configuration dialog with Authorization Name, Zenrows API Key, and Who can use fields" />
    </Frame>

    The plugin panel now shows **1 Authorization** with a green indicator, and lists the five actions.

    <Frame>
      <img alt="The Zenrows plugin panel in Dify showing 1 Authorization and the Fetch, Extract, Batch Create, Batch Status, and Batch Results actions" />
    </Frame>
  </Step>
</Steps>

## Build a Workflow That Summarizes Any Web Page

This example builds a four-node workflow: it takes a URL, fetches the page through Zenrows as Markdown, has an LLM summarize it, and returns the summary.

<Steps>
  <Step title="Create a workflow with a URL input">
    In **Studio**, click **Create from Blank**, choose **Workflow**, name it, and click **Create**. Pick **User Input** as the start node, then click **+** next to **Input Field** and add a **Short Text** field with the variable name `url`.
  </Step>

  <Step title="Add the Zenrows Fetch tool">
    Under **Next Step**, click **Select Next Step**, open the **Tools** tab, search for `zenrows`, and click **Fetch**.

    <Frame>
      <img alt="The Dify node picker on the Tools tab, filtered to Zenrows, listing Fetch, Extract, Batch Create, Batch Status, and Batch Results" />
    </Frame>

    In the Fetch node's **URL** field, type `/` and select the **User Input** `url` variable. Leave **Response format** on **Markdown** and the other settings at their defaults.

    <Frame>
      <img alt="The Fetch node settings in Dify, with URL set to the User Input url variable and Response format set to Markdown" />
    </Frame>
  </Step>

  <Step title="Pass the page to an LLM node">
    From the Fetch node, add an **LLM** node. In its **System** prompt, write an instruction such as `Summarize the following web page in five bullet points. Use only what the page says.` On a new line, type `{` and select the Fetch node's **`text`** variable.

    <Frame>
      <img alt="The LLM node's system prompt in Dify, with the Fetch text variable inserted after the instruction" />
    </Frame>

    <Warning>
      **Use `text`, not `content`.** The Fetch node's output list in Dify also shows `content`, `url`, and `status_code`, but in plugin version 0.1.0 those aren't populated as variables. Referencing `content` fails the run with `Variable ... content# not found`. The page content arrives in `text`, and the same fields are available inside the `json` output.
    </Warning>
  </Step>

  <Step title="Return the summary and run it">
    From the LLM node, add an **Output** node with one variable, `summary`, set to the LLM node's `text`. The finished workflow looks like this:

    <Frame>
      <img alt="The finished Dify workflow: User Input, Fetch, LLM, and Output nodes connected left to right" />
    </Frame>

    Click **Test Run**, enter `https://www.scrapingcourse.com/ecommerce/` as the URL, and click **Start Run**. The **Result** tab shows the summary.

    <Frame>
      <img alt="The Test Run Result tab in Dify, showing a bullet-point summary of the ScrapingCourse ecommerce page with product names and prices" />
    </Frame>

    The **Tracing** tab shows each node's input and output. For Fetch, you can confirm the parameters sent to Zenrows and the Markdown that came back.

    <Frame>
      <img alt="The Tracing tab in Dify with the Fetch node expanded, showing its input parameters and the Markdown text output" />
    </Frame>
  </Step>
</Steps>

## Handle Protected and JavaScript-Heavy Pages

Fetch and Extract have **Adaptive stealth** turned on by default. With it on, Zenrows starts with a basic request and escalates to JavaScript rendering or Premium Proxies only when the target needs them, so most protected pages work without changing any settings. See [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode) for how escalation works and what it costs.

Leave the Fetch and Extract settings at their defaults unless you have a specific reason to change them. Adaptive stealth already decides when a page needs a browser or a residential IP.

## Work with Tool Outputs

Each tool returns its result in Dify's standard `text` and `json` outputs. Some also return named variables you can reference directly:

| Tool | Use downstream |
| - | - |
| **Fetch** | `text` holds the page content. `json` holds `content`, `url`, and `status_code`. A screenshot or PDF arrives in `files`. |
| **Extract** | `text` holds the extracted data as a JSON string, or a "No structured data found" message. `json` holds `data`, `method_used`, `fell_back_to_autoparse`, `empty`, and `url`. |
| **Batch Create** | `job_id`, `run_id`, `status`, `finished`, and `accepted_tasks` variables. |
| **Batch Status** | `job_id`, `status`, `finished`, `total`, `completed`, `successful`, `failed`, `failure_reasons`, and `spend` variables. |
| **Batch Results** | `results`, `returned`, and `truncated` variables. Each result has `url`, `status`, and, unless **Include page content** is off, `content`. |

For a batch, chain the tools: Batch Create, then Batch Status with the `job_id` until `finished` is `true`, then Batch Results. Leave **Wait for completion** off for more than a few dozen URLs. A tool call can't run longer than about 100 seconds, so a large job won't finish inside one call.

## Troubleshooting

### The Zenrows tools don't appear in the node picker

The plugin isn't installed in this workspace, or it has no authorization. Open **Integrations** > **Tools** > **Tool Plugin**, click **Zenrows**, and confirm the panel shows **1 Authorization**. If it shows **API Key Authorization Configuration** instead, add your key.

### Authorization fails when saving the key

Copy the key again from your <a href="https://app.zenrows.com/settings/api-keys">Zenrows dashboard</a> and check for leading or trailing spaces. On self-hosted Dify, confirm the instance can reach `api.zenrows.com` over HTTPS.

### `Variable ... not found` in the node after Fetch or Extract

The downstream node references `content`, `url`, `status_code`, or `data` from Fetch or Extract. Those aren't populated as variables in plugin version 0.1.0. Reference `text` instead, or read the field from the `json` output.

### "Turn off Adaptive stealth" error

The plugin rejects the request before it reaches Zenrows when Adaptive stealth is on together with **Render JavaScript**, **Premium proxy**, **Screenshot**, **PDF** as the response format, **Wait for selector**, **Wait (ms)**, or **JavaScript instructions**. Zenrows doesn't accept those combinations either.

* If you set **Render JavaScript** or **Premium proxy** to **True**, set it back to **False**. Adaptive stealth turns them on automatically when a page needs them.
* If you need a screenshot, a PDF, a wait condition, or JavaScript instructions, turn Adaptive stealth off for that node. Those options require a browser, so you also control rendering and proxies yourself.

### The page comes back empty or incomplete

Open the **Tracing** tab and check the Fetch node's output. If the text is a login wall or a cookie banner rather than the page you expected, the content isn't publicly available at that URL. If the target is consistently returning incomplete content, contact <a href="mailto:success@zenrows.com">Zenrows support</a> with the URL.

### Batch Create times out with Wait for completion on

Waiting gives up after about 100 seconds. Turn **Wait for completion** off and poll with Batch Status instead.

## Further Reading

* <a href="https://marketplace.dify.ai/plugin/zenrows/zenrows">Zenrows Plugin on the Dify Marketplace</a>
* <a href="https://github.com/ZenRows/dify-zenrows">Plugin Source Code on GitHub</a>
* <a href="https://docs.dify.ai/en/cloud/use-dify/nodes/tools">Dify Tool Node Documentation</a>
* [Fetch API Reference](/fetch/api-reference)
* [Batch Introduction](/batch/introduction)

## Frequently Asked Questions

<Accordion title="Is the Zenrows plugin for Dify official?">
  Yes. Zenrows publishes and maintains it, and its source is at `github.com/ZenRows/dify-zenrows`.
</Accordion>

<Accordion title="Does it work on self-hosted Dify?">
  Yes, on Dify 1.11.4 or later, as long as the instance can make outbound HTTPS requests to `api.zenrows.com` and `async.api.zenrows.com`.
</Accordion>

<Accordion title="Can I use the Zenrows tools in a Dify agent, not just a workflow?">
  Yes. The tools are available anywhere Dify offers tools, including agents. Each tool ships a model-facing description, so an agent can decide when to fetch a page, extract fields, or start a batch.
</Accordion>

<Accordion title="Does validating my API key cost credits?">
  No. Dify validates the key when you save the authorization, and that check doesn't consume Zenrows credits. Each tool call is billed like the equivalent Zenrows API request.
</Accordion>

<Accordion title="How is this different from calling the Zenrows API with an HTTP Request node?">
  The plugin handles authentication, parameter validation, and error messages for you, and exposes each option as a field in the node. An HTTP Request node works too, but you build the query string, handle errors, and parse the response yourself.
</Accordion>


# How to Integrate Flowise with Zenrows
Source: https://docs.zenrows.com/integrations/flowise

Integrate Zenrows with Flowise to build AI-powered web scraping workflows using a visual drag-and-drop builder for data collection pipelines.

Integrate Zenrows with Flowise to add reliable web scraping capabilities to your AI workflows. This guide shows you how to build data collection pipelines that combine Zenrows' scraping technology with Flowise's visual workflow builder.

## What is Flowise?

Flowise is an open-source, low-code platform for visually building, managing, and interacting with AI agents and large language model (LLM) workflows in a chat-based environment. Its drag-and-drop interface allows users to compose AI agents into interconnected, LLM-powered workflows **with little to no programming knowledge**.

## Use Cases

You can leverage Flowise-Zenrows integration to drive several business opportunities. Here are some of them:

* **Product research**: Discover the best marketing position for new products with insights from scraped data.
* **Demand forecasting**: Scrape products' data to predict the demand ahead of competitors. Make data-driven decisions on re-stocks and pricing.
* **Real-estate intelligence**: Collect real-estate data and analyze it to gain data-driven investment insights.
* **Competitor's research**: Gather competitor's data for information gain and predict their next moves.
* **Product optimization**: Collect customers' reviews, ratings, and other metrics from social platforms, review sites, and brand pages and fine-tune products to meet consumers' expectations.
* **Lead generation**: Collect quality lead contacts from various sources and automate captures, outreaches, and follow-ups.
* **Product recommendation**: Scrape and analyze relevant product data points to recommend the best deals to potential buyers.

## Basic Integration: Amazon Product Scraper

Let's start by building a simple Amazon product scraper workflow that fetches data using Zenrows and returns it as a response via the Flowise chat box. We'll use the Flowise Agentflows for this tutorial.

The Flowise platform is accessible locally or remotely via the <a href="https://cloud.flowiseai.com/">official Flowise website</a>. If you use Flowise remotely via the website, no installation steps are required. However, the local setup requires installing Flowise globally to run a local server.

In this guide, **we'll use the local setup**, starting with the installation and server startup steps.

### Step 1: Install Flowise and set up a server

This tutorial assumes you've set up Node.js on your machine. Otherwise, download and install it from the <a href="https://nodejs.org/en/download">official website</a>.

1. Install the `flowise` package globally:
   ```bash theme={"dark"}
   npm install -g flowise
   ```
2. Run the `flowise` server. The command below starts a server that defaults to `http://localhost:3000`:
   ```bash theme={"dark"}
   npx flowise start
   ```
3. Visit this URL to launch the Flowise interface.
4. Complete the initial registration steps, and you'll get to the platform's dashboard, as shown below:
   <img alt="flowise-main-dashboard" />

### Step 2: Add Zenrows API Key to Flowise environment variables

Adding your Zenrows API key to the Flowise environment variable makes it accessible across the platforms.

1. From the dashboard, go to **Variables**.
2. Click `+ Add Variable` at the top-right.
3. Fill in the **Variable Name** field with the name of your API key. Then, paste your Zenrows API key in the **Value** field.
4. Click `Add`.
   <img alt="zenrows-api-key-variable" />

### Step 3: Initiate a new Agentflow

1. Go to **Agentflows** and click `+ Add New` at the top right to open the Flowise canvas.
   <img alt="flowise-agentflows-section" />

2. Once in the Flowise Canvas, click the save icon at the top right.
   <img alt="flowise-empty-canvas" />

3. Enter a name for your new workflow and click `Save`.
   <img alt="flowise-flows-rename" />

### Step 4: Create a scraper flow with Zenrows

We'll integrate the Zenrows API using an HTTP agent node, which enables you to send an API request via Flowise.

1. Click the `+` icon at the top left and search for **"HTTP"** using the search bar.
   <img alt="flowise-http-agent" />

2. Drag and drop the **HTTP** node into your Canvas, placing it directly in front of the **Start** agent.
   <img alt="flowise-http-agent-dropped" />

3. Link the agents by dragging a line from the **Start** node to connect with the **HTTP** agent node.
   <img alt="flowise-agent-node-connection" />

4. Double-click the **HTTP** agent node.

5. Click the edit icon at the top of the modal to change the agent's name. Then, press **Enter** on your keyboard.
   <img alt="flowise-agent-name" />

6. Click `+ Add Query Params` to set the target URL parameter. We'll retrieve this URL dynamically from the chat box:
   * Type `url` in the **Key** field.
   * Type double braces (`{{`) in the **Value** field. This loads dynamic input options.
   * Select **question** from the options to load the URL from the chat box.
     <img alt="flowise-param-selection" />

7. Similarly, for the Zenrows API key, click `+ Add Query Params` and type `apikey` in the **Key** field. Type `{{` inside the **Value** and select `$vars.ZENROWS_API_KEY`.

8. Click `+ Add Query Params` repeatedly to add each of the following [Zenrows parameters](/fetch/api-reference#parameter-overview) in order:

   * `js_render` = `true`
   * `premium_proxy` = `true`
   * `wait` = `2500`
   * `css_extractor` =&#x20;
   * `original_status` = `true`

   <img alt="flowise-zenrows-http-request-setup" />

9. Add your [CSS selectors](/fetch/troubleshooting/advanced-css-selectors) as an array in the `css_extractor` value: We'll use the following `css_extractor` for this tutorial:
   ```json theme={"dark"}
   {
       "name":"span#productTitle",
       "price":"span.a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay",
       "ratings":"#acrPopover",
       "description":"ul.a-unordered-list.a-vertical.a-spacing-mini",
       "review summary":"#product-summary p span",
       "review count":"#acrCustomerReviewLink"
   }
   ```

10. Under **Response Type**, select **Text**.
    <img alt="flowise-zenrows-http-response-type" />

11. Click on the `Flowise Canvas`. Then, click the `Validate Nodes` icon at the top right and select `Validate Nodes` to confirm that your setup is correct.
    <img alt="flowise-validate-nodes" />

12. Finally, click the `save` icon at the top right to save your changes.

### Step 5: Run the flow

1. Click the chat icon at the top right.

2. Type an <a href="https://www.amazon.com/Sony-WH-CH520-Headphones-Bluetooth-Microphone/dp/B0BS1PRC4L">Amazon product page URL</a> in the box and press Enter to send your request.

3. Click the `Process Flow banner` in the chat box.

4. Click the expansion icon next to Scraper Agent.
   <img alt="chat-flow-flowise" />

5. You'll see the following response page, showing the scraped data:
   <img alt="flowise-response-page" />

However, we want the LLM to return the data as the sole response in the chat. Let's set this up using OpenAI.

### Step 6: Integrate an LLM flow

1. Click the `+` icon at the top left. Search **LLM**. Then, drag and drop it in the canvas next to the Scraper Agent.

2. Link the **Scraper Agent** to the **LLM** node.
   <img alt="flowise-zenrows-llm-linkage" />

3. Double-click the **LLM** agent and rename it.
   <img alt="flowise-llm-name" />

4. Under **Model**, search and select `ChatOpenAI`.

5. Click `ChatOpenAI` Parameters:

   * Click on `Connect Credentials`, then on `Create New` and set up your OpenAI API key and click `Add`.

   <img alt="flowise-openai-apikey" />

   * Select your desired model, temperature (preferably, 0.1 or 0.2 for increased accuracy), and other parameters.

   <img alt="flowise-openai-params" />

6. Click `+ Add Messages`:
   * Under **Role**, select `Assistant`.
   * Type `{{` inside the Content field and select `httpAgentflow_0`.
   <img alt="chatopenai-settings" />

7. Toggle on `Enable Memory`.
   <img alt="memory-activation-openai" />

8. Type the following prompt in the **Input Message** field. We've instructed the LLM to only show `{{ httpAgentflow_0 }}` as the output and not attempt to visit any URL provided in the chat:
   ```text theme={"dark"}
   You are a web scraping assistant. Only display the response provided by the Scraper Agent in the chat box. Do not attempt to access or retrieve information from external URLs yourself; rely solely on the Scraper Agent for all web data. Your output must be {{ httpAgentflow_0 }}.
   ```

9. Set the **Return Response As** field to **Assistant Message**. This allows the chat model to return the Scraper Agent's output in its response.
   <img alt="chatopenai-all-messages-settings" />

10. Click anywhere in the canvas. Then click the save icon at the top right.
    <img alt="flowise-zenrows-final-flow" />

11. To run the flow, click the chat icon and send the URL of an Amazon product page in the chat. The model now streams the scraped data as a response in the chat box:
    <img alt="flowise-chat-response" />

Nice! You just created your first scraping flow by integrating Zenrows with Flowise.

## Advanced Integration: Demand Analysis System

Besides the HTTP option, you can also integrate Zenrows with Flowise using custom functions. We'll see a real-life use case using the following agents:

* Create an agent that scrapes an Amazon listing page based on a search term using Zenrows.
* Build an agent node to handle data cleaning.
* Create an agent to analyze the data using LLM.

### Step 1: Create a custom web scraper function with Zenrows

1. Go to **Agentflows** and click `+ Add New` to start a new flow.

2. Click the save icon and name your flow.

3. Click `+` at the top left of the Flowise canvas.

4. Search for `Custom Function` and drag its card to the canvas.

5. Link the **Start** node with the **Custom Function** agent.
   <img alt="flowise-initial-canvas-with-function" />

6. Double-click the Custom Function agent and rename it.

7. Initiate the Zenrows scraping logic inside the **JavaScript Function** field. Import `axios`, configure the target URL to accept a search term, and load your Zenrows API key from the Flowise environment variables.
   ```javascript Node.js theme={"dark"}
   const axios = require("axios");

   // declare the search term variable
   const searchTerm = $searchTerm;

   // load your API key from the environment variables
   const apikey = $vars.ZENROWS_API_KEY;

   const url = `https://www.amazon.com/s?k=${encodeURIComponent(searchTerm)}`;
   ```

8. To create the `searchTerm` variable, click `+ Add Input Variables`.

9. Type `searchTerm` inside the **Variable Name** field.

10. We want the scraping logic to accept this search term dynamically from the user's question in the chat box. Type `{{` in the **Variable Value** and select `question`.
    <img alt="flowise-scraping-function-setup" />

11. Extend the scraping logic by adding the required Zenrows parameters. Scrape the data using the `css_extractor` and return the response data. See the complete code below:
    ```javascript Node.js theme={"dark"}
    const axios = require("axios");

    // declare the search term variable
    const searchTerm = $searchTerm;

    // load your API key from the environment variables
    const apikey = $vars.ZENROWS_API_KEY;

    const url = `https://www.amazon.com/s?k=${encodeURIComponent(searchTerm)}`;

    try {
    const response = await axios({
        url: "https://api.zenrows.com/v1/",
        method: "GET",
        params: {
        url: url,
        apikey: apikey,
        js_render: "true",
        wait: "2500",
        premium_proxy: "true",
        css_extractor: JSON.stringify({
            name: "div.s-title-instructions-style",
            url: "div.s-title-instructions-style a @href",
            listingPrice: "span.a-price",
            demandHistory:
            "div.a-section.a-section.a-spacing-none.a-spacing-top-micro span.a-size-base.a-color-secondary",
            reviewCount: "span.a-size-base.s-underline-text",
        }),
        },
    });

    return response.data;

    } catch (error) {
    console.error("Error fetching data:", error.message);
    throw error;
    }
    ```

12. Click the save icon at the top right to save your changes.
    <img alt="flowise-canvas-with-scraper-function" />

### Step 2: Create a data cleaning agent

1. Click the `+` icon.

2. Search for **Custom Function** and drag and drop it onto the canvas.

3. Link this new custom function agent with the **Scraper** agent.
   <img alt="flowise-custom-function-1" />

4. Double-click the `Custom Function 1` agent node and rename it (e.g., Data Cleaner).
   <img alt="flowise-custom-function-setup-1" />

5. We need to load the Scraper agent's output into the Data Cleaner agent node. Create this variable by clicking `+ Add Input Variables`. Then, type **scrapedData** in the **Variable Name** field. Then, enter `{{` in the **Variable Value** field and select `customFunctionAgentflow_0`.
   <img alt="flowise-zenrows-scraped-data-variable" />

6. Enter the following data cleaning code in the JavaScript Function field. This code first converts the scraped data into an object, merges it, and fixes anomalies, such as data inconsistencies.
   ```javascript Node.js theme={"dark"}
   // convert the scraped data to an object
   let data;
   try {
   data = JSON.parse($scrapedData);
   } catch (e) {
   return [];
   }

   // ensure all arrays exist
   const demand = Array.isArray(data.demandHistory) ? data.demandHistory : [];
   const name = Array.isArray(data.name) ? data.name : [];
   const listingPrice = Array.isArray(data.listingPrice) ? data.listingPrice : [];
   const reviewCount = Array.isArray(data.reviewCount) ? data.reviewCount : [];
   const url = Array.isArray(data.url) ? data.url : [];

   const baseUrl = "https://www.amazon.com";

   // check if the demand entry is a real monthly demand
   const isValidDemand = (d) =>
   typeof d === "string" && d.includes("bought in past month");

   // merge and clean the data
   const products = [];
   let demandIdx = 0;

   for (let i = 0; i < name.length; i++) {
   // filter out sponsored/ad names
   if (
       typeof name[i] !== "string" ||
       name[i].toLowerCase().startsWith("sponsored") ||
       name[i].toLowerCase().includes("ad feedback")
   ) {
       continue;
   }

   // find the next valid demand entry
   let demandValue = "Unavailable";
   while (demandIdx < demand.length) {
       if (isValidDemand(demand[demandIdx])) {
       demandValue = demand[demandIdx];
       demandIdx++;
       break;
       }
       demandIdx++;
   }

   // clean doubled price issues
   let price = listingPrice[i] || "Unavailable";
   if (typeof price === "string") {
       const match = price.match(/\$\d+(\.\d{2})?/);
       if (match) {
       price = match[0];
       }
   }

   // ensure URL is absolute
   let productUrl = url[i] || "Unavailable";
   if (
       typeof productUrl === "string" &&
       productUrl !== "Unavailable" &&
       !productUrl.startsWith("http")
   ) {
       productUrl = baseUrl + productUrl;
   }

   products.push({
       name: name[i] || "Unavailable",
       demand: demandValue,
       price: price,
       reviews: reviewCount[i] || "Unavailable",
       url: productUrl,
   });
   }

   return products;
   ```

7. Click anywhere on the canvas and click the save icon.
   <img alt="flowise-scraper-cleaner-flow" />

### Step 3: Create a Data Analyst Agent LLM

1. Click the `+` icon.

2. Search for **LLM** and drag and drop it into the canvas.

3. Link the LLM agent with the Data Cleaner agent. The LLM will act on the cleaned data returned from the Data Cleaner agent.
   <img alt="all-flows" />

4. Double-click the LLM agent and give it a descriptive name.

5. From the **Model** dropdown, search and select `ChatOpenAI` or your preferred model.

6. Set up your credentials, select your model, and choose a temperature.
   <img alt="flowise-openai-params" />

7. Click `+ Add Messages` under the **Messages** section.

8. Create a prompt that instructs the Analyst agent to get the most in-demand products from the cleaned data and establish a correlation between price and demand. We'll use the following prompt, where `customFunctionAgentflow_1` is the cleaned data from the Data Cleaner agent node.
   ```text theme={"dark"}
   Analyze the products in {{ customFunctionAgentflow_1 }}. Select the 4 most in-demand products based on the demand history (highest demand first). Also, tell if there is a correlation between price and demand across these products. For each product, format the product's URL as an HTML anchor tag with the text: Check product out.
   ```

9. Under Role, select `Assistant`. Then, enter the prompt into the **Content** field.
   <img alt="analyst-openai-prompt" />

10. Under **Return Response As**, select `Assistant Message`.
    <img alt="final-llm-setup" />

11. Finally, validate and save your workflow by clicking the validation and save icons at the top right, respectively.

### Step 4: Run the demand analysis workflow

Click the message icon to initiate a chat. Then, prompt the Analyst agent with a product search term (e.g., headphones, mouse, or teddy bears).

Here's a sample response for "mouse" product listings:

<img alt="sample-chat-response-from-analyst" />

Congratulations! 🎉 You just built a product research workflow with Zenrows and Flowise integration.

## Troubleshooting

### Error 401 or Zenrows authentication error

* **Solution 1**: Ensure you've entered the correct Zenrows API key in the Flowise environment variables.
* **Solution 2**: Ensure you load the Zenrows API key into your scraping workflow.

### Error 400

* **Solution 1**: Make sure the supplied Zenrows parameters are correct and don't contain unsupported strings.
* **Solution 2**: Use the correct CSS selectors and format the `css_extractor` array correctly.

### LLM agent fails

* **Solution 1**: Check your network connectivity and try your prompt again.
* **Solution 2**: Ensure you still have enough API credit for your chosen LLM.
* **Solution 3**: Make sure you've supplied the correct API key of your LLM agent.

### Custom function fails

* **Solution 1**: Execute the code in an external text editor, such as VS Code, and simulate the expected output for debugging.
* **Solution 2**: Make sure all variables in your code are listed as **Input Variables**.

### `localhost unavailable` or the Flowise interface doesn't load

* **Solution**: Stop and restart the running server. Then, reload the Flowise interface.

## Conclusion

You've learned to connect Zenrows with Flowise for a streamlined data flow across several agents, including data collection, cleaning, and analysis. While Flowise enables you to build agentic workflows for data automation and other purposes, Zenrows ensures the integrity of your data pipeline with a consistent data supply.

## Frequently Asked Questions (FAQ)

<Accordion title="Will Zenrows charge me extra for integrating Flowise?">
  Zenrows doesn't charge you extra for integrating with Flowise. Flowise is also an open-source tool with no initial cost implications. Although Flowise charges for advanced features like cloud hosting, the free plan provides most of what you need, including building Agentflows, Chatflows, Assistants, and more.
</Accordion>

<Accordion title="Are there alternatives to Flowise?">
  Yes, Zenrows integrates with other AI-powered low-code and no-code workflow tools, including Pipedream, Make, Clay, Zapier, and more. Check our [integration options](/integrations/overview) for more details.
</Accordion>

<Accordion title="Can I save the extracted data from Flowise to Google Sheets?">
  Although Flowise supports Google Sheets as a tool, connecting to it involves some technical steps. Currently, Flowise has limitations in terms of Google Sheets write operations compared to tools like Zapier, which offer easier and more seamless integration.
</Accordion>

<Accordion title="Does Zenrows bypass anti-bots on Flowise?">
  Yes, integrating Zenrows with Flowise enables your scraper agent to bypass anti-bots easily.
</Accordion>


# Google ADK
Source: https://docs.zenrows.com/integrations/google-adk

Build autonomous, multi-agent workflows that can read any website using Zenrows' hosted MCP server with Google's Agent Development Kit (ADK). Plug in scraping and browser automation as native ADK tools, with no orchestration code required.

## Overview

Combine <a href="https://google.github.io/adk-docs/">Google's Agent Development Kit (ADK)</a> with the <a href="https://docs.zenrows.com/mcp/overview">Zenrows MCP server</a> to build autonomous, multi-agent workflows that can read any website in real time. ADK has native MCP support via the `McpToolset` class, so connecting Zenrows takes a single block of code. No function-calling boilerplate, no scraping infrastructure, and no anti-bot tuning required.

<Note>
  This guide covers ADK with Python. ADK also supports TypeScript, Java, Go, and Kotlin. The same MCP integration pattern applies across all of them.
</Note>

## What is Google ADK?

<a href="https://google.github.io/adk-docs/">Google's Agent Development Kit (ADK)</a> is an open-source, code-first framework for building, evaluating, and deploying AI agents. It is model-agnostic and deployment-agnostic, and it is optimized for Google's Gemini models while supporting Claude, GPT, and 100+ other LLMs via LiteLLM.

ADK provides primitives for agent orchestration (`LlmAgent`, workflow agents, multi-agent collaboration), tools (function tools, MCP tools, OpenAPI tools), sessions, memory, and deployment to Vertex AI Agent Engine, Cloud Run, and GKE.

Adding Zenrows MCP gives every ADK agent real-time access to any website, including JavaScript-heavy and bot-protected ones.

### Key benefits

<CardGroup>
  <Card title="Native MCP support" icon="plug">
    ADK's `McpToolset` wrapper takes the Zenrows server URL and your API key. The agent discovers Zenrows' tools (`scrape` and 30+ browser automation tools) automatically, with no schema mapping to maintain.
  </Card>

  <Card title="Autonomous multi-step tasks" icon="robot">
    ADK agents can scrape one page, decide what to scrape next based on what they read, and chain browser interactions like form fills and clicks across multiple pages. ADK runs the orchestration loop for you.
  </Card>

  <Card title="Anti-bot bypass on every tool call" icon="shield-check">
    Premium residential proxies, JavaScript rendering, and stealth fingerprinting come standard. Your ADK agents reach JavaScript-heavy SPAs, Cloudflare-protected sites, and geo-restricted pages without configuration.
  </Card>

  <Card title="Composes with ADK's full primitive set" icon="arrows-split-up-and-left">
    Zenrows tools work alongside multi-agent workflows, sequential and parallel agents, sessions, memory, callbacks, evaluation, and tracing. Pass them to one agent or many; the integration is the same.
  </Card>
</CardGroup>

<Card title="Deploys cleanly to Google Cloud" icon="cloud">
  ADK has first-class deployment paths to Vertex AI Agent Engine, Cloud Run, and GKE. Because the Zenrows MCP server is hosted, your deployed ADK agent connects to it over HTTP, with no MCP server to package or run alongside your agent container.
</Card>

## Use cases

The Google ADK and Zenrows combination unlocks a wide range of agent workflows:

<AccordionGroup>
  <Accordion title="Autonomous research agents" icon="magnifying-glass">
    Agents that visit multiple URLs, follow citations, and synthesize findings from across the web into a single brief.
  </Accordion>

  <Accordion title="Lead enrichment pipelines on Vertex AI" icon="chart-line">
    ADK agents deployed to Agent Engine that take a company name, scrape the company website, find pricing pages and product launches, and write enriched profiles into BigQuery or your CRM.
  </Accordion>

  <Accordion title="Multi-agent web automation" icon="arrows-split-up-and-left">
    A research agent that gathers information and hands off to a writing agent that produces the final report, both pulling from live web data. ADK's sequential and collaborative workflows make this pattern trivial.
  </Accordion>

  <Accordion title="Competitive monitoring on Cloud Run" icon="bell">
    Scheduled ADK agents that periodically check competitor pricing pages, blog posts, and changelogs, then surface what has changed since the last run.
  </Accordion>

  <Accordion title="Form-filling and extraction agents" icon="file-pen">
    Agents that navigate multi-step checkout flows, login-walled forms, or job applications using Zenrows' browser tools.
  </Accordion>
</AccordionGroup>

## Getting started

Build a single ADK agent equipped with Zenrows MCP that can answer questions by reading any website in real time.

### Prerequisites

* Python 3.10 or higher
* A Zenrows API key (Find it in your <a href="https://app.zenrows.com/settings/api-keys">Zenrows dashboard</a>)
* A Google API key

### Setup

<Steps>
  <Step title="Install ADK">
    Install the ADK package with MCP support.

    ```bash theme={"dark"}
    pip install "google-adk[mcp]"
    ```
  </Step>

  <Step title="Create your environment file">
    Create a `.env` file in the parent folder of your agent directory with your API keys.

    ```bash .env theme={"dark"}
    ZENROWS_API_KEY=your_zenrows_key
    GOOGLE_API_KEY=your_google_api_key
    ```
  </Step>

  <Step title="Set up the directory structure">
    ADK uses a standard directory layout. Create the following structure:

    ```
    adk_agent_samples/
    └── zenrows_agent/
        ├── __init__.py
        └── agent.py
    ```
  </Step>

  <Step title="Create the agent">
    Add a re-export in `__init__.py`, then create the agent in `agent.py`.

    ```python __init__.py theme={"dark"}
    from . import agent
    ```

    ```python agent.py expandable theme={"dark"}
    # ./adk_agent_samples/zenrows_agent/agent.py
    import os

    from google.adk.agents import LlmAgent
    from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams

    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

    root_agent = LlmAgent(
       model="gemini-2.5-flash",
       name="web_research_assistant",
       instruction=(
           "You are a research assistant with live web access. "
           "Use the Zenrows tools to scrape pages when you need real-time information."
       ),
       tools=[
           MCPToolset(
               connection_params=StreamableHTTPConnectionParams(
                   url="https://mcp.zenrows.com/mcp",
                   headers={
                       "Authorization": f"Bearer {ZENROWS_API_KEY}",
                   },
               )
           )
       ],
    )
    ```
  </Step>

  <Step title="Run the agent">
    Start the ADK web UI from the parent directory.

    ```bash theme={"dark"}
    cd adk_agent_samples
    adk web
    ```

    <img alt="ADK Web UI" />

    Open the URL printed in your terminal, select `zenrows_agent` from the agent dropdown, and try a prompt like:

    ```
    Visit https://news.ycombinator.com/ and summarize the three most recent posts.
    ```
  </Step>
</Steps>

### How it works

* `MCPToolset` registers the Zenrows hosted MCP server as a tool source. `StreamableHTTPConnectionParams` points at the Zenrows MCP endpoint and carries your API key as an OAuth Bearer token in the `Authorization` header.
* On startup, `MCPToolset` queries the Zenrows server for its available tools and adapts each one to ADK's tool schema. The agent gets access to `scrape`, `browser_navigate`, `browser_click`, and 30+ other Zenrows tools with no extra configuration.
* `LlmAgent` equips the Gemini model with the toolset. The agent reads the prompt, picks the right tool, calls it through `MCPToolset`, gets the content back, and writes the answer.

**You've built a working web-aware agent in roughly 20 lines of code.** Let's look at more advanced patterns.

## Advanced usage

### Autonomous research agent

The real power of ADK is letting an agent decide for itself which tools to call across multiple steps. With Zenrows MCP, that means the agent can autonomously read several pages, follow links it finds interesting, and synthesize findings, all from a single prompt.

The example below builds a research agent that visits a topic landing page, identifies the most relevant linked articles, scrapes each one, and produces a synthesized report. This version uses ADK's programmatic `Runner` instead of `adk web`, which is the pattern to use in production code or when embedding ADK into your own application.

```python Python expandable theme={"dark"}
import asyncio
import os

from google.adk.agents import LlmAgent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams
from google.genai.types import Content, Part

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

research_agent = LlmAgent(
    model="gemini-2.5-pro",
    name="research_agent",
    instruction=(
        "You are a thorough web researcher. When given a topic and a starting URL, "
        "visit the URL, identify the 3 most relevant linked articles, scrape each one, "
        "and produce a synthesized report covering key findings, common themes, and "
        "any contradictions. Cite each source with its URL."
    ),
    tools=[
        MCPToolset(
            connection_params=StreamableHTTPConnectionParams(
                url="https://mcp.zenrows.com/mcp",
                headers={
                    "Authorization": f"Bearer {ZENROWS_API_KEY}",
                },
            )
        )
    ],
)


async def main() -> None:
    session_service = InMemorySessionService()
    runner = Runner(
        agent=research_agent,
        app_name="zenrows_research_app",
        session_service=session_service,
    )

    session = await session_service.create_session(
        app_name="zenrows_research_app",
        user_id="user_1",
    )

    prompt = (
        "Research recent developments in AI tools. "
        "Start at https://huggingface.co/blog and follow the most relevant links."
    )
    message = Content(role="user", parts=[Part(text=prompt)])

    async for event in runner.run_async(
        user_id=session.user_id,
        session_id=session.id,
        new_message=message,
    ):
        if event.content and event.content.parts:
            for part in event.content.parts:
                if part.text:
                    print(part.text)


asyncio.run(main())
```

<Tip>
  * Don't write an orchestration loop in your code. The agent decides on its own when to scrape, which URLs to follow, and when it has enough information to write the report. ADK runs the multi-step loop automatically.
  * `gemini-2.5-pro` for multi-step tasks. Reasoning-capable Gemini models work better for autonomous workflows because they can plan their next action based on what they have learned so far. For lighter tasks, `gemini-2.5-flash` is faster and cheaper.
  * Same MCP config as the basic example. The only things that changed are the agent's instructions, the model, and that you're driving the agent programmatically through `Runner` instead of `adk web`.
</Tip>

### Multi-agent workflows

Because Zenrows MCP is a standard ADK tool, you can share it across multiple agents and combine it with ADK's multi-agent patterns. A common setup is one agent for research (with Zenrows access) and another for writing (without web tools).

```python Python expandable theme={"dark"}
import os

from google.adk.agents import LlmAgent
from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

zenrows_toolset = MCPToolset(
    connection_params=StreamableHTTPConnectionParams(
        url="https://mcp.zenrows.com/mcp",
        headers={
            "Authorization": f"Bearer {ZENROWS_API_KEY}",
        },
    )
)

writer_agent = LlmAgent(
    model="gemini-2.5-flash",
    name="writer_agent",
    instruction="Take research findings and produce a polished article.",
)

researcher_agent = LlmAgent(
    model="gemini-2.5-pro",
    name="researcher_agent",
    instruction=(
        "Gather research from the web on the user's topic using Zenrows tools. "
        "Once you have enough material, transfer to the writer_agent for the final draft."
    ),
    tools=[zenrows_toolset],
    sub_agents=[writer_agent],
)
```

The researcher agent calls Zenrows tools to gather material, then transfers control to the writer agent (which has no web access) for the final draft. ADK also supports sequential, parallel, loop, and graph-based workflow patterns. See the <a href="https://google.github.io/adk-docs/workflows/">ADK multi-agent workflows documentation</a> for the full set of patterns.

## Troubleshooting

* **`424 Failed Dependency` or `401 Unauthorized` from the MCP server**
  1. Confirm the `Authorization: Bearer YOUR_ZENROWS_API_KEY` header is present in the `headers` dict on `StreamableHTTPConnectionParams`.
  2. Verify your key in the <a href="https://app.zenrows.com/settings/api-keys">Zenrows dashboard</a> and confirm your subscription has remaining quota.
  3. Confirm the URL is exactly `https://mcp.zenrows.com/mcp`.

* **Empty tool list when starting the agent**

  If `adk web` shows the agent but no Zenrows tools appear in tool calls, the connection to the MCP server failed silently.

  1. Check your API key, the URL, and your network connectivity.
  2. Look at the `adk web` console output. `MCPToolset` logs `list_tools` errors there.

* **Agent repeatedly scrapes the same page**

  Add explicit guidance in the agent's `instruction` about what to do once it has the data. For example, `"After scraping each URL once, summarize and stop."`

  <Tip>For complex flows, consider splitting the work across multiple agents using sub-agents.</Tip>

* **Deployment fails with Agent Definition Must Be Synchronous**

  ADK requires agents that use MCP tools to be defined at module load time. If you have an `async def get_agent()` factory, refactor so `root_agent` is a module-level variable instantiated synchronously.

  <Tip>See the <a href="https://adk.dev/tools-custom/mcp-tools/#critical-deployment-requirement-synchronous-agent-definition">ADK MCP deployment guide</a> for the full pattern.</Tip>

* **Page blocked or data missing in scraped content**

  If the agent connects and invokes tools correctly but the scraped content comes back blocked or incomplete, the issue is on the target page rather than the MCP connection.

  Add explicit guidance to your agent's instruction. For example, `"When scraping, ask Zenrows to use JavaScript rendering and Premium Proxies for protected sites."`

  <Tip>See the <a href="https://docs.zenrows.com/fetch/troubleshooting/troubleshooting-guide">Fetch troubleshooting guide</a> for parameter-level fixes.</Tip>

## FAQ (Frequently Asked Questions)

<Accordion title="What's the difference between using ADK with Zenrows MCP versus calling Fetch directly?">
  Calling Fetch directly works for any Python script. With ADK + Zenrows MCP, you get autonomous agent behavior on top: ADK decides which tool to call, chains multiple calls in a single run, handles multi-step workflows, and integrates with ADK's session, memory, and deployment infrastructure.

  Use direct API calls for simple deterministic scripts. Use ADK + MCP when you want agent autonomy.
</Accordion>

<Accordion title="Which Gemini model should I use?">
  For simple agents, `gemini-2.5-flash` is fast and cost-effective. For autonomous multi-step research and complex reasoning, `gemini-2.5-pro` is a better fit. You can also use Claude, GPT, or any LiteLLM-supported model through ADK's model routing if you want to mix providers within the same workflow.
</Accordion>

<Accordion title="Can I use ADK's TypeScript, Java, Go, or Kotlin SDKs with Zenrows MCP?">
  Yes. The `MCPToolset` class exists across all ADK language bindings, and the underlying transport (Streamable HTTP) is language-agnostic. Provide the URL `https://mcp.zenrows.com/mcp`, set the `Authorization: Bearer YOUR_ZENROWS_API_KEY` header, and the toolset discovers Zenrows' tools automatically. See the <a href="https://adk.dev/tools-custom/mcp-tools/">ADK MCP tools guide</a> for language-specific syntax.
</Accordion>

<Accordion title="Can I use the local stdio version of Zenrows MCP with ADK?">
  Yes. Use `StdioConnectionParams` with the `@zenrows/mcp` npm package instead of `StreamableHTTPConnectionParams`. The remote URL is recommended because it requires no local install, no Node.js dependency, and runs the same way in development and production. See the <a href="https://docs.zenrows.com/mcp/overview">Zenrows MCP overview</a> for the local stdio configuration.
</Accordion>

<Accordion title="Can multiple agents share the same Zenrows MCP toolset?">
  Yes. Build the `MCPToolset` once and pass it to as many agents as you need.

  ```python theme={"dark"}
  zenrows_toolset = MCPToolset(connection_params=...)

  agent_a = LlmAgent(name="A", tools=[zenrows_toolset])
  agent_b = LlmAgent(name="B", tools=[zenrows_toolset])
  ```

  Each agent gets independent tool calls. Zenrows handles them all on the same hosted server.
</Accordion>

<Accordion title="Are MCP tool calls captured in ADK's tracing?">
  Yes. ADK's <a href="https://google.github.io/adk-docs/observability/traces/">tracing</a> captures every MCP `list_tools` and `call_tool` invocation alongside the rest of the agent's events. You can view them in OpenTelemetry-compatible backends or in Vertex AI Agent Engine's built-in observability when deployed there.
</Accordion>


# Hermes MCP Integration for Zenrows
Source: https://docs.zenrows.com/integrations/hermes

Connect Hermes Agent with the Zenrows MCP to give your autonomous agent reliable access to any website without getting blocked. Scrape protected pages, extract live data, and run scheduled monitoring tasks with results delivered directly to your CLI or messaging app.

## What is Hermes Agent?

<a href="https://hermes-agent.nousresearch.com">Hermes Agent</a> is an open-source self-improving autonomous agent. Unlike a chatbot or a coding copilot, Hermes runs a closed learning loop by creating skills from experience, improving them during use, and building a persistent model of who you are across sessions. It runs wherever you put it: a VPS, a local machine, or serverless infrastructure. You can talk to it from your terminal, Telegram, or any supported messaging app you connect to it.

## Why connect Zenrows to Hermes?

* **Every scraping task becomes a skill:** When Hermes successfully scrapes a site through Zenrows, it can turn that workflow into a reusable skill. The next time you ask for competitor pricing or job board leads, it already knows how to execute your workflow.

* **Access any website data without getting blocked:** Proxy rotation, anti-bot evasion, and browser orchestration run on Zenrows' infrastructure. Hermes asks for the page and gets the content regardless of the anti-bot system protecting it.

* **Lower LLM costs per task:** Raw HTML boilerplate burns tokens. Zenrows returns clean, structured, LLM-friendly Markdown or JSON, so Hermes processes only what matters and your token budget is managed.

* **Reliable scheduled tasks:** When Hermes runs a cron job, such as a daily digest, a price check, or a lead monitor, a blocked page breaks the whole run. Zenrows keeps those tasks completing consistently.

* **Works across parallel subagents:** Hermes can spawn up to 3 concurrent subagents by default via `delegate_task`, each with its own isolated context and toolset. Pair that with Zenrows and you can run a multi-source research task, with one subagent pulling competitor pricing, another monitoring job boards, and another tracking industry news, and have all three deliver structured results to your chat simultaneously.

* **Full browser automation, described in plain English:** For sites that require login, pagination, or form interaction, Hermes can instruct Zenrows' browser tools in natural language. No selectors, no scraping code.

## What you can build with the Hermes-Zenrows integration

* **A self-improving competitive intelligence feed:** Schedule a daily cron job that scrapes competitors' pricing, changelogs, and job pages, compares them against the previous run, and delivers only what changed to your Telegram or Slack.

* **A parallel market research agent:** Give Hermes a research brief covering competitors, news sources, and job boards, and let it spawn concurrent subagents via `delegate_task`, each scraping a different source through Zenrows. Hermes then synthesizes everything into a single briefing.

* **An ad research and content brief pipeline:** Give Hermes a product URL and it scrapes the landing page, pulls winning hooks from Meta Ads Library and TikTok Creative Center, and writes a ready-to-use creative brief without you touching a single tab.

* **A daily news and industry digest:** Schedule Hermes to scrape Hacker News, RSS feeds, and industry blogs every morning, filter for what's actually relevant to your work, and push a tight summary to your Telegram or Discord before you start your day.

* **A pre-call research brief:** Give Hermes a company name before a sales or partnership call and it uses Zenrows to scrape their site, recent news coverage, job postings, and any public announcements, then assembles a one-page brief in your messaging app before the meeting starts.

* **A 24/7 monitoring agent on a remote server:** Deploy Hermes on a VPS or serverless instance, point a set of cron jobs at protected pages via Zenrows, and get messaged only when something is worth your attention.

## Connecting Zenrows to Hermes

This guide uses Anthropic as the LLM provider and the Zenrows MCP server. You can substitute any supported provider, but the setup steps shown here follow the Anthropic path.

Before continuing, you need a Zenrows account and API key. Obtain one from the [Zenrows dashboard](https://app.zenrows.com/register).

<Steps>
  <Step title="Install Hermes">
    The Hermes installer handles everything automatically: Python 3.11, Node.js 22, ripgrep, ffmpeg, and the `hermes` command itself. You don't need to install those manually.

    **Linux / macOS / WSL2:**

    ```bash theme={"dark"}
    curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
    ```

    **Windows (native PowerShell):**

    ```powershell theme={"dark"}
    iex (irm https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.ps1)
    ```

    Verify the installation:

    ```bash theme={"dark"}
    hermes doctor
    ```
  </Step>

  <Step title="Configure your LLM provider">
    Hermes requires a model with at least 64,000 tokens of context. Most hosted models meet this easily. Follow the on-screen prompts to choose your LLM provider, enter your API key, and select your preferred model.

    ```bash theme={"dark"}
    hermes model
    ```
  </Step>

  <Step title="Test the chat model">
    Verify that chat works by running the following command:

    ```bash theme={"dark"}
    hermes
    ```

    Once the CLI-based agent loads, send a test prompt to see if Hermes loads a response from the connected model:

    ```text theme={"dark"}
    Go to https://www.scrapingcourse.com/antibot-challenge and summarize the page.
    ```

    Hermes will attempt the page with its built-in search services and respond with a blocked page, showing its built-in tools aren't enough to access protected pages:

    ```text Output theme={"dark"}
    The site is hitting Cloudflare's anti-bot security check and blocking access.
    ```
  </Step>

  <Step title="Add Zenrows MCP to Hermes">
    Zenrows exposes its scraping and browser tools as an MCP server. Adding it to Hermes automatically gives the agent access to all Zenrows tools.

    Hermes has a built-in catalog of Nous-approved MCP servers that you can install with a single command (`hermes mcp catalog`). Zenrows isn't in the catalog yet, so add it manually to the config file instead.

    Hermes configuration lives in the `~/.hermes/config.yaml` file. Open it in any editor. The path depends on your OS:

    * **Linux / macOS / WSL2:** `~/.hermes/config.yaml`
    * **Windows:** `C:\Users\<username>\AppData\Local\hermes\config.yaml`

    Add Zenrows MCP to the `mcp_servers` block:

    ```yaml YAML theme={"dark"}
    mcp_servers:
        zenrows:
            url: "https://mcp.zenrows.com/mcp"
            headers:
                Authorization: "Bearer <YOUR_ZENROWS_API_KEY>"
    ```

    Save the file, then start Hermes:

    ```bash theme={"dark"}
    hermes chat
    ```

    Hermes discovers MCP servers at startup and registers their tools into the normal tool registry. Verify Zenrows loaded by asking:

    ```text theme={"dark"}
    Tell me which MCP-backed tools are available right now.
    ```

    The Hermes chat agent returns Zenrows and its exposed tools:

    ```text Output wrap theme={"dark"}
    Based on the tools loaded in this session, here are all the MCP-backed tools available right now:

    Zenrows MCP (mcp_zenrows_*)

    Scraping:
        - mcp_zenrows_scrape - Fetch any URL as markdown/plaintext/HTML/PDF, with optional JS rendering, proxies, CSS extraction

    Browser Automation:
    # ... list of exposed Zenrows tools truncated for brevity
    ```

    If you're adding Zenrows to a running session or made changes to the MCP configuration at runtime, use the reload command in that chat session instead of restarting:

    ```text theme={"dark"}
    /reload-mcp
    ```

    If the dynamic update doesn't work, close the current chat session and rerun `hermes chat` to reload the Hermes agent with your changes.
  </Step>
</Steps>

### Filtering Zenrows tools (optional)

If you want to restrict which Zenrows tools Hermes can use, for example allowing only the `scrape` tool and blocking browser automation, add a `tools.include` filter:

```yaml YAML theme={"dark"}
mcp_servers:
  zenrows:
    url: "https://mcp.zenrows.com/mcp"
    headers:
        Authorization: "Bearer <YOUR_ZENROWS_API_KEY>"
    tools:
        include: [scrape]
```

## Getting started: Basic usage

With Zenrows connected, Hermes can fetch and process any page without writing a single line of code. Start a chat session and try:

```text wrap theme={"dark"}
Scrape the top headlines from news.ycombinator.com and summarize all the top 5 stories in one paragraph.
```

Hermes will call the Zenrows scrape tool, receive clean Markdown back, and summarize the content for you:

```text Output wrap theme={"dark"}
Today's top Hacker News stories cover a mix of security, science, education, and programming. A researcher exposed...truncated for brevity...
```

Try a more targeted extraction:

```text wrap theme={"dark"}
Go to https://www.scrapingcourse.com/ecommerce and extract all product names and prices. Return them as a table.
```

Or test it against a protected site:

```text theme={"dark"}
Go to https://www.scrapingcourse.com/antibot-challenge and summarize the page.
```

With Zenrows, Hermes knows when to escalate to anti-bot bypass based on the response. You don't have to specify it unless you want to.

## Advanced usage: Autonomous Scheduled Scraping

Hermes has a built-in cron scheduler. Pair it with Zenrows, and you can set up scraping tasks that run on a schedule and deliver results to any platform without you having to intervene.

### Setting up a daily price monitor

From inside a chat session, use the `/cron` command:

```text wrap theme={"dark"}
/cron add "every day at 9 am" "Scrape https://www.amazon.com/s?k=headset, get the top 3 best deals based on price and ratings, and return a summary." --name "Headset deals."
```

The above prompt creates a cron job that runs every 24 hours. Hermes fires a fresh agent session, scrapes the page via Zenrows, and outputs a summary of the top deals.

<Tip>Cron jobs run in fresh agent sessions; they don't inherit the context of your current chat. They do have access to Hermes's persistent memory and skills, but anything specific to your current conversation needs to be included directly in the prompt. You set the prompt once when creating the job, and Hermes runs it on every cycle. Include everything it needs directly in that prompt: the URL, what to extract, and how to format the output.</Tip>

### Monitoring important responses using the \[SILENT] flag

For monitoring jobs, you only want to see when something changes. Include a `[SILENT]` flag to only get relevant alerts:

```text wrap theme={"dark"}
/cron add "1d" "Scrape https://www.indeed.com/q-london-uk-jobs.html and extract all job postings. Identify roles that signal buying intent such as new engineering hires, scaling teams, or data infrastructure roles. If nothing changed since the last run, respond with [SILENT]. Otherwise, list new postings added today." --name "Buying intent monitor."
```

When the agent's final response contains `[SILENT]`, delivery is suppressed. No notification noise on quiet days.

Manage your cron jobs with the following commands:

```bash theme={"dark"}
/cron list                              # List all active jobs
/cron run <job_id>                      # Trigger immediately for testing
/cron pause <job_id>                    # Pause without deleting
/cron edit <job_id> --schedule "4h"     # Change schedule
/cron remove <job_id>                   # Delete permanently
```

To learn how to run parallel scraping workstreams across multiple sources simultaneously, see <a href="https://hermes-agent.nousresearch.com/docs/guides/delegation-patterns">Delegation Patterns</a> in the Hermes docs.

## Set up a messaging channel (optional)

Once Zenrows and Hermes are working, you can connect a messaging platform to receive scraping results and cron job outputs directly in your preferred chat app.

The messaging gateway is a background process that handles all connected platforms and runs the cron scheduler. You need it running for message delivery to work.

<Steps>
  <Step title="Configure your platform">
    ```bash theme={"dark"}
    hermes gateway setup
    ```

    The above command starts an interactive chat channel setup. Select your preferred channel and follow the instructions to set it up. This connects Hermes to Telegram, Discord, Slack, WhatsApp, Signal, or any of 20+ supported platforms.

    This guide uses Telegram. Set up a Telegram bot by sending the `/newbot` command to `@BotFather`. Follow `@BotFather`'s instructions and enter the given bot ID into the interactive CLI. Once set up, scheduled cron jobs and agent results can be delivered straight to your preferred channel.
  </Step>

  <Step title="Install and start the gateway as a background service">
    ```bash theme={"dark"}
    hermes gateway install   # Registers as a system service
    hermes gateway start     # Starts the service
    hermes gateway status    # Confirms it's running
    hermes gateway stop      # Stops a running service
    ```

    Once installed, the gateway runs in the background and starts automatically whenever your machine boots.

    When the Hermes server starts, run a scraping prompt through the Telegram bot you created. It uses the Zenrows scrape tool to access your target and loads the response via the connected AI model.
  </Step>
</Steps>

## Troubleshooting

### No response is returned in Telegram

The message was delivered to the bot, but the gateway isn't running, so nothing is processing it.

1. Confirm the gateway is running:

   ```bash theme={"dark"}
   hermes gateway status
   ```

2. If it's stopped, restart it:

   ```bash theme={"dark"}
   hermes gateway start
   ```

Check that your Telegram bot token is correct and the bot was created via `@BotFather`. An invalid token means the gateway connects but cannot send messages back.

### Zenrows tools not loading after adding to config

* Confirm the config was saved to `~/.hermes/config.yaml`.
* Restart Hermes with the `hermes start` command.
* Verify your API key is correct and the Zenrows MCP endpoint is entered correctly.

### Scraping returns empty or blocked content

Zenrows handles anti-bot systems automatically, but Hermes may sometimes use its built-in browser tool instead. If you're seeing unexpected results, explicitly instruct Hermes to use Zenrows:

```text theme={"dark"}
Use Zenrows to scrape https://www.scrapingcourse.com/ and summarize the page.
```

### Cron job runs, but no message is delivered

* Confirm the gateway is running: `hermes gateway status`.
* Run the job manually with `/cron run <job_id>` to see the output immediately.

## FAQ (Frequently asked questions)

<Accordion title="Does Zenrows count as one of Hermes's 60+ built-in tools?">
  No. Zenrows tools are added via MCP and are registered separately under the `mcp_zenrows_` prefix. They sit alongside Hermes's native tools and the agent uses whichever is best for the task.
</Accordion>

<Accordion title="Can I use Zenrows alongside Hermes's built-in browser tool?">
  Yes. Hermes has a native browser tool (available via Nous Portal's Tool Gateway) and Zenrows runs as a separate MCP-backed toolset. They coexist cleanly. For heavy-duty scraping with anti-bot evasion, Zenrows is the better choice; for simple navigation tasks you control, the native browser may be sufficient.
</Accordion>

<Accordion title="How does Hermes decide when to use Zenrows vs. a direct fetch?">
  Hermes reasons about the task. For pages that are known to be protected or dynamic, it will reach for Zenrows tools. You can also instruct it explicitly: "use Zenrows to scrape this URL."
</Accordion>

<Accordion title="Can I run Hermes and Zenrows on a remote server?">
  Yes. Hermes is designed to run anywhere: a VPS, a cloud VM, Docker, or serverless infrastructure via Daytona or Modal. The setup is identical: install Hermes, add Zenrows to `~/.hermes/config.yaml`, configure your messaging platform, and your scraping agent runs 24/7 without you needing to SSH in.
</Accordion>

<Accordion title="Does Hermes store scraped content across sessions?">
  Hermes can save content to skills and memory if you instruct it to. Scraped data that's useful as a reference can be persisted as a skill or written to a file. By default, raw scrape output stays in the current session.
</Accordion>

<Accordion title="Can I use the local stdio version of Zenrows MCP instead of the remote URL?">
  Yes. Zenrows supports both a remote HTTP server and a local stdio server via the `@zenrows/mcp` npm package. Both options expose the same tools and capabilities. The remote URL (`https://mcp.zenrows.com/mcp`) is recommended for Hermes since it requires no local installation. If you prefer the local stdio approach, you need Node.js installed and use this config instead:

  ```yaml YAML theme={"dark"}
  mcp_servers:
      zenrows:
          command: "npx"
          args: ["-y", "@zenrows/mcp"]
          env:
              ZENROWS_API_KEY: "<YOUR_ZENROWS_API_KEY>"
  ```
</Accordion>

<Accordion title="Can I connect multiple MCP servers alongside Zenrows?">
  Yes. Hermes supports multiple MCP servers in the same config. Each gets its own named block under `mcp_servers`, and its tools are registered separately:

  ```yaml YAML theme={"dark"}
  mcp_servers:
      zenrows:
          url: "https://mcp.zenrows.com/mcp"
          headers:
              Authorization: "Bearer <YOUR_ZENROWS_API_KEY>"
      github:
          command: "npx"
          args: ["-y", "@modelcontextprotocol/server-github"]
          env:
              GITHUB_PERSONAL_ACCESS_TOKEN: "<YOUR_TOKEN>"
  ```
</Accordion>

<Accordion title="Is there a cost to using Zenrows MCP with Hermes?">
  The Zenrows MCP server itself is free to connect. Each tool call is billed against your Zenrows subscription: `scrape` calls use Fetch credits, and `browser_*` calls use Browser Sessions credits. You can monitor usage on your [Zenrows Analytics page](https://app.zenrows.com/overview#performance).
</Accordion>

<Accordion title="Can I install Zenrows through the Hermes MCP catalog?">
  The Hermes catalog (`hermes mcp catalog`) lists Nous-approved MCP servers. Zenrows isn't currently in the catalog, so this guide walks through adding it manually to `config.yaml`. You can check the latest catalog entries by running `hermes mcp catalog` in your terminal.
</Accordion>


# Zenrows MCP Integration for JetBrains IDEs
Source: https://docs.zenrows.com/integrations/jetbrains

Set up Zenrows MCP in JetBrains IDEs like IntelliJ, PyCharm, and WebStorm so the AI assistant can scrape any webpage from your editor.

## Before You Start

You need:

<Columns>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">A **Zenrows account**</Card>
  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">Your **Zenrows API key**</Card>
</Columns>

<Card icon="code" href="https://nodejs.org/en/download/" title="Node.js">Latest version of **Node.js** installed on your machine</Card>
<Card icon="robot" href="https://www.jetbrains.com" title="JetBrains IDE">A **JetBrains IDE** with the AI Assistant plugin enabled</Card>

## Setup

<Steps>
  <Step title="Step 1: Open the MCP settings">
    Go to **Settings > Tools > AI Assistant > Model Context Protocol**.
  </Step>

  <Step title="Step 2: Add the Zenrows MCP server">
    Add the following configuration, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

    ```json theme={"dark"}
    {
        "mcpServers": {
            "zenrows": {
                "command": "npx",
                "args": ["-y", "@zenrows/mcp"],
                "env": {
                    "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
                }
            }
        }
    }
    ```
  </Step>

  <Step title="Step 3: Apply and restart">
    Click **Apply** and restart your IDE to load the new configuration.
  </Step>
</Steps>

## Start scraping

Ask the AI assistant to scrape a webpage or automate a browser workflow in plain English:

```
Scrape https://www.zenrows.com/pricing and summarize the plans.
```

```
Get the product names and prices from https://www.scrapingcourse.com/ecommerce/.
```

```
Scrape https://www.scrapingcourse.com/antibot-bypass using Premium Proxies.
```

```
Fetch https://www.scrapingcourse.com/javascript-rendering/ with JavaScript rendering enabled and return the page as Markdown.
```

```
Open https://scrapingcourse.com/login in a browser, fill in the credentials, click submit, then extract the account details.
```

## Troubleshooting

**The `scrape` tool is not available**

Confirm the configuration was saved under **Settings > Tools > AI Assistant > Model Context Protocol** and that you restarted the IDE after applying.

**Page content is missing or incomplete**

The page likely loads content dynamically. Ask the assistant to enable JavaScript rendering:

```
Scrape https://www.scrapingcourse.com/javascript-rendering/ with js_render enabled.
```

**Requests return a 403 or bot detection page**

Ask the assistant to use Premium Proxies:

```
Scrape https://www.scrapingcourse.com/antibot-bypass using premium_proxy.
```

For more troubleshooting options, see the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview#troubleshooting).


# How to Integrate LangChain with Zenrows
Source: https://docs.zenrows.com/integrations/langchain

Install and configure the langchain-zenrows package to give LLM agents and AI chains real-time web scraping and data extraction capabilities.

Extract web data with AI agents using Zenrows' enterprise-grade scraping infrastructure. The <a href="https://pypi.org/project/langchain-zenrows/">`langchain-zenrows`</a> integration enables large language models (LLMs) to access real-time web data using Zenrows' robust scraping infrastructure. This guide covers how to scrape data with LLMs using the `langchain-zenrows` module.

## What is LangChain?

LangChain is a framework that connects large language models to external data sources and applications. It provides a composable architecture that enables you to create AI workflows by chaining LLM operations, from simple prompt-response patterns to autonomous agents.

One key advantage of LangChain is that it allows for easy swapping, coupling, and decoupling of LLMs.

### Key Benefits of Integrating LangChain With Zenrows

The `langchain-zenrows` integration brings the following benefits:

* **Integrate Zenrows with LLMs**: Easily integrate scraping capabilities into your desired LLM.
* **Build an agentic data pipeline**: Assign different data pipeline roles to each LLM agent based on its capabilities.
* **Real-time web access without getting blocked**: Fetch live web content without antibot or JavaScript rendering limitations.
* **Multiple output formats**: Fetch website data in various formats, including HTML, Markdown, Plaintext, PDF, or Screenshots.
* **Specific data point extraction**: Extract specific data from web pages, such as emails, tables, phone numbers, images, and more.
* **Support for custom parsing**: Fetch specific information from web elements using Zenrows' advanced CSS selector feature.

## Use Cases

Here are some use cases of the `langchain-zenrows` integration:

* **Real-time monitoring**: Develop an AI application that scrapes and monitors website content changes in real-time.
* **Market research and demand forecasting**: Scrape demand signals, such as reviews, social comments, engagement metrics, price trends, and more. Then, pass the data to an LLM model for forecasting.
* **Finding the best deals**: Spot the best deals for a specific product from several e-commerce websites using Zenrows.
* **Review summarization**: Summarize scraped reviews using a selected model.
* **Sentiment analysis**: Scrape and analyze sentiment in social comments or product reviews.
* **Product research and comparison**: Compare products across multiple retail websites and e-commerce platforms to identify the best options.
* **Consistent data pipeline update**: Keep your data pipeline up to date with fresh data by integrating `langchain-zenrows` into your pipeline operations.

## Getting Started: Basic Usage

Let's start with a simple example that uses the `langchain-zenrows` package to scrape the <a href="https://www.scrapingcourse.com/antibot-challenge">Antibot Challenge</a> page and return its content in Markdown format.

Install the `langchain-zenrows` package using `pip`:

```bash theme={"dark"}
pip3 install langchain-zenrows
```

Import the `ZenRowsUniversalScraper` class from the `langchain_zenrows` module, instantiate the universal scraper with your Zenrows API key, and specify Zenrows parameters with the `response_type` set to `markdown`:

```python Python theme={"dark"}
from langchain_zenrows import ZenRowsUniversalScraper

# Set your Zenrows API key
os.environ["ZENROWS_API_KEY"] = "YOUR_ZENROWS_API_KEY"

# Instantiate the universal scraper
scraper = ZenRowsUniversalScraper()

url = "https://www.scrapingcourse.com/antibot-challenge"

# Set Zenrows parameters
params = {
    "url": url,
    "js_render": "true",
    "premium_proxy": "true",
    "response_type": "markdown",
}

# Get content in markdown format
result = scraper.invoke(params)
print(result)
```

The integration bypasses the target site's antibot measure and returns its content as Markdown:

```html Output theme={"dark"}
[![](https://www.scrapingcourse.com/assets/images/logo.svg) Scraping Course](http://www.scrapingcourse.com/)

# Antibot Challenge

![](https://www.scrapingcourse.com/assets/images/challenge.svg)

## You bypassed the Antibot challenge! :D
```

You've successfully integrated Zenrows with LangChain and bypassed an antibot challenge. Let's build an AI research assistant with this integration.

## Advanced Usage: Building an AI Research Assistant

Let's take things a step further by building an AI-powered pricing research assistant for Etsy. Using the `langchain-zenrows` integration together with OpenAI's `gpt-4o-mini` model, our assistant will automatically visit Etsy's accessories category and extract key product details such as names, prices, and URLs.

Here's the prompt we'll use to guide the assistant:

### Example Prompt

```text Prompt theme={"dark"}
Go to the accessories category of https://www.etsy.com/ and return the names, prices, and URLs of the top 4 products in JSON format using the autoparse feature.
```

### Step 1: Install the packages

```bash theme={"dark"}
pip install langgraph langchain-openai langchain-zenrows
```

### Step 2: Add Zenrows as a scraping tool for the AI model

Import the necessary modules and define your Zenrows and OpenAI API keys. Instantiate OpenAI's chat model and `langchain-zenrows` integration with the relevant API keys. Configure the LLM agent to use Zenrows as a scraping tool:

```python Python theme={"dark"}
# pip install langgraph langchain-openai langchain-zenrows
from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

os.environ["ZENROWS_API_KEY"] = "YOUR_ZENROWS_API_KEY"
os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"

def scraper():
    # initialize the model
    llm = ChatOpenAI(model="gpt-4o-mini")

    # initialize the universal scraper
    zenrows_tool = ZenRowsUniversalScraper()

    # create an agent that uses Zenrows as a tool
    agent = create_react_agent(llm, [zenrows_tool])
```

### Step 3: Prompt the AI Agent

Invoke the AI agent with the research prompt and execute the scraper. As stated in the prompt, the agent uses Zenrows' [`markdown`](/fetch/features/markdown) response to scrape the target page in Markdown format. It then analyzes the result and returns the 4 cheapest products:

```python Python theme={"dark"}
# ...
def scraper():
    # ...

    try:
        # create a prompt
        result = agent.invoke(
            {
                "messages": "Go to the Accessories category page of https://www.etsy.com/. Scrape the page in markdown format and return the 4 cheapest products in JSON format."
            }
        )
        # extract the response
        for message in result["messages"]:
            print(f"{message.content}")

    except NameError:
        print(
            "⚠️  Agent not available."
        )
    except Exception as e:
        print(f"❌ Error running agent: {e}")

scraper()
```

The agent uses Zenrows to visit and scrape the product information. Once scraped, the agent returns the items in the desired format.

### Complete Code Example

Combine the snippets from the two steps, and you'll get the following code:

```python Python theme={"dark"}
# pip install langgraph langchain-openai langchain-zenrows
from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

os.environ["ZENROWS_API_KEY"] = "YOUR_ZENROWS_API_KEY"
os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"

def scraper():

    # initialize the model
    llm = ChatOpenAI(model="gpt-4o-mini")

    # initialize the universal scraper
    zenrows_tool = ZenRowsUniversalScraper()

    # create an agent that uses Zenrows as a tool
    agent = create_react_agent(llm, [zenrows_tool])

    try:
        # create a prompt
        result = agent.invoke(
            {
                "messages": "Go to the Accessories category page of https://www.etsy.com/. Scrape the page in markdown format and return the 4 cheapest products in JSON format."
            }
        )
        # extract the response
        for message in result["messages"]:
            print(f"{message.content}")

    except NameError:
        print(
            "⚠️  Agent not available."
        )
    except Exception as e:
        print(f"❌ Error running agent: {e}")


scraper()
```

The above code returns the names, prices, and URLs of the 4 cheapest products in JSON format as expected.

### Example Output

```json Output theme={"dark"}
[
    {
        "title": "Lovely Cat Keychain Gift For Pet Mom",
        "price": "$4.68",
        "url": "https://www.etsy.com/listing/1812260433/lovel...",
    },
    {
        "title": "Personalized slim leather keychain, key fob, custom keychain, leather initial keychain, quick shipping anniversary gift",
        "price": "$4.79",
        "url": "https://www.etsy.com/listing/876501930/personalized...",
    },
    {
        "title": "Custom OWALA Name Tag Back to School for daughter Owala Cup accessory for son waterbottle Tumbler Name Plate for sports tumbler athlete tag",
        "price": "$4.50",
        "url": "https://www.etsy.com/listing/1796331543/custom-...",
    },
    {
        "title": "Set of Blue and White Striped Hair Bows - 3-Inch Handmade Clips for Girls & Toddlers",
        "price": "$6.00",
        "url": "https://www.etsy.com/listing/4328846122/set...",
    },
]
```

Congratulations! 🎉 You've now integrated Zenrows as a web scraping tool for an AI agent using the `langchain-zenrows` module.

## API Reference

| Parameter | Type | Description |
| - | - | - |
| `zenrows_api_key` | `string` | Your Zenrows API key. If not provided, the setup looks for the `ZENROWS_API_KEY` environment variable. |
| `url` | `string` | **Required.** The URL to scrape. |
| `js_render` | `boolean` | Enable JavaScript rendering with a headless browser. Essential for modern web apps, SPAs, and sites with dynamic content (default: False). |
| `js_instructions` | `string` | Execute custom JavaScript on the page to interact with elements, scroll, click buttons, or manipulate content. |
| `premium_proxy` | `boolean` | Use residential IPs to bypass antibot protection. Essential for accessing protected sites (default: False). |
| `proxy_country` | `string` | Set the country of the IP used for the request. Use for accessing geo-restricted content. Two-letter country code. |
| `session_id` | `integer` | Maintain the same IP for multiple requests for up to 10 minutes. Essential for multi-step processes. |
| `custom_headers` | `boolean` | Include custom headers in your request to mimic browser behavior. |
| `wait_for` | `string` | Wait for a specific CSS Selector to appear in the DOM before returning content. |
| `wait` | `integer` | Wait a fixed amount of milliseconds after page load. |
| `block_resources` | `string` | Block specific resources (images, fonts, etc.) from loading to speed up scraping. |
| `response_type` | `string` | Convert HTML to other formats. Options: "markdown", "plaintext", "pdf". |
| `css_extractor` | `string` | Extract specific elements using CSS selectors (JSON format). |
| `autoparse` | `boolean` | Automatically extract structured data from HTML (default: False). |
| `screenshot` | `string` | Capture an above-the-fold screenshot of the page (default: "false"). |
| `screenshot_fullpage` | `string` | Capture a full-page screenshot (default: "false"). |
| `screenshot_selector` | `string` | Capture a screenshot of a specific element using CSS Selector. |
| `screenshot_format` | `string` | Choose between "png" (default) and "jpeg" formats for screenshots. |
| `screenshot_quality` | `integer` | For JPEG format, set the quality from 1 to 100. Lower values reduce file size but decrease quality. |
| `original_status` | `boolean` | Return the original HTTP status code from the target page (default: False). |
| `allowed_status_codes` | `string` | Returns the content even if the target page fails with the specified status codes. Useful for debugging or when you need content from error pages. |
| `json_response` | `boolean` | Capture network requests in JSON format, including XHR or Fetch data. Ideal for intercepting API calls made by the web page (default: False). |
| `outputs` | `string` | Specify which data types to extract from the scraped HTML. Accepted values: emails, phone numbers, headings, images, audios, videos, links, menus, hashtags, metadata, tables, favicon. |

<Note>
  For complete parameter documentation and details, see the official [Zenrows API Reference](/fetch/api-reference#parameter-overview).
</Note>

## Troubleshooting

### Token limit exceeded

* **Solution 1**: If you hit the LLM token limit, it means the output size has exceeded what the model can process in a single request. You can parse specific data and then feed it to the LLM model.
* **Solution 2**: If the issue is related to usage-based token quotas or the model version's capabilities, consider upgrading your plan or switching to a higher model with higher bandwidth. For instance, moving from gpt-3.5 to gpt-4o-mini increases the token limit significantly.

### API key error

* **Solution 1**: Ensure you've added your Zenrows and the LLM's API keys to your environment variables.
* **Solution 2**: Cross-check the API keys and ensure you've entered the correct keys.

### Empty or incomplete data/tool response

* **Solution 1**: Activate JS rendering to handle dynamic content and increase the success rate.
* **Solution 2**: Increase the wait time using the Zenrows `wait` or `wait_for` parameter. The `wait` parameter introduces a general delay to allow the entire page to load, whereas `wait_for` targets a specific element, pausing execution until that element appears before scraping continues.
* **Solution 3**: If you've used the `css_extractor` parameter to target specific elements, ensure you've entered the correct selectors.

## Helpful Resources

* <a href="https://pypi.org/project/langchain-zenrows/">LangChain-Zenrows PyPI package</a>
* <a href="https://github.com/ZenRows-Hub/langchain-zenrows">LangChain-Zenrows GitHub repository</a>
* <a href="https://github.com/ZenRows-Hub/langchain-zenrows/tree/main/examples">Check our examples for more use cases</a>

## Frequently Asked Questions (FAQ)

<Accordion title="Which LLMs does langchain-zenrows support?">
  `langchain-zenrows` is compatible with all LLMs supported by LangChain. Check <a href="https://python.langchain.com/docs/integrations/chat/">LangChain's official chat models documentation</a> for more information.
</Accordion>

<Accordion title="Can I use selectors with the LLM agent option?">
  Yes, you can extract data from specific elements by explicitly specifying their selectors in your prompt.
</Accordion>

<Accordion title="Does langchain-zenrows support custom JavaScript execution?">
  Yes, you can include custom JavaScript via Zenrows' `js_instructions` parameter. Check our [JavaScript instructions guide](/fetch/features/js-instructions) for more.
</Accordion>

<Accordion title="Is antibot bypass automatic with the LLM agent option?">
  Yes, Zenrows' antibot bypass features are activated automatically when using Zenrows as the agent's tool.
</Accordion>

<Accordion title="Does the LLM agent integration handle JS rendering?">
  Yes. The JS rendering parameter is activated on demand while scraping a JavaScript-rendered site. This enables you to scrape dynamic pages with ease.
</Accordion>

<Accordion title="How do I extract specific data with CSS selectors?">
  To extract data from specific elements, use Zenrows' `css_extractor` parameter to specify the selectors of the elements containing the data you want to scrape.
</Accordion>

<Accordion title="Can I take screenshots with the LLM agent integration?">
  Yes, you can prompt the LLM to take a half, full, or a specific element screenshot, and it will return your desired result using Zenrows' screenshot parameter.
</Accordion>

<Accordion title="What's the difference between this and other web scraping tools in LangChain?">
  Zenrows offers enterprise-grade reliability, featuring built-in antibot bypass, premium proxies, JavaScript rendering, and more. Unlike basic scrapers, it can handle protected sites, geo-restricted content, and modern SPAs without getting blocked.
</Accordion>


# How to Integrate Lindy with Zenrows
Source: https://docs.zenrows.com/integrations/lindy

Connect Zenrows with Lindy AI to automate web scraping workflows. Set up intelligent data extraction using Lindy assistants without coding.

Automate web scraping workflows by connecting Zenrows with Lindy's AI assistant platform. This integration extracts and processes web data through intelligent automation, eliminating manual data collection tasks.

## What Is Lindy?

Lindy creates AI assistants that automate repetitive business tasks. When you integrate Lindy with Zenrows, you can scrape websites automatically and use the extracted data to trigger intelligent workflows like generating reports, updating spreadsheets, or managing sales pipelines.

## Use Cases

Here are some ways you can use Zenrows and Lindy together:

* **Competitive Analysis:** Extract competitor data and generate automated reports.
* **Lead Generation:** Scrape business directories and qualify prospects for your sales team.
* **Brand Monitoring:** Track online mentions and respond automatically.
* **Content Curation:** Collect and categorize articles for your team.
* **Market Research:** Gather and analyze property listings or market data.

## Real-World End-to-End Integration Example

This example shows how to build an automated system that monitors e-commerce products and stores the data in Google Sheets.

### Step 1: Create a new Lindy assistant

1. Log in to <a href="https://lindy.ai/">Lindy</a> and open your workspace dashboard.
2. Click `+ New Lindy`, select **Start from scratch**, and name your assistant (e.g., Product Data Extractor).
   <img alt="create-new-lindy" />

### Step 2: Configure Zenrows integration with Lindy

<Steps>
  <Step title="Generate the HTTP request URL from Zenrows">
    1. Go to [Zenrows' Fetch Request Playground](https://app.zenrows.com/fetch).
    2. Input `https://www.scrapingcourse.com/ecommerce` as your target URL for scraping.
    3. Enable **JS Rendering** to handle dynamic page content.
    4. Configure **Output type** to **Specific Data** and navigate to the **Parsers** section.
    5. Set up the CSS selector configuration: `{ "name": ".product-name" }`.
    6. Navigate to the cURL tab and copy the generated HTTP request URL. It'll look like this:
       ```bash theme={"dark"}
       https://api.zenrows.com/v1/?apikey=<YOUR_ZENROWS_API_KEY>&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&js_render=true&css_extractor=%257B%2522name%2522%253A%2522.product-name%2522%257D
       ```
       <Note>The CSS selectors provided in this example (`.product-name`) are specific to the page used in this guide. Selectors may vary across websites. For guidance on customizing selectors, refer to the [CSS Extractor documentation](/fetch/features/css-extractor). If you're having trouble, the [Advanced CSS Selectors Troubleshooting Guide](/fetch/troubleshooting/advanced-css-selectors) can help resolve common issues.</Note>
  </Step>

  <Step title="Set up the HTTP Request in Lindy">
    1. Within your Lindy assistant, select **Perform an action** from the available options.
       <img alt="lindy-perform-an-action" />
    2. Choose **HTTP Request** action from the action menu.
       <img alt="lindy-http-request-action" />
    3. In the **Url** field, select the **Set Manually** option.
    4. Enter the URL generated from your Zenrows configuration.
       <img alt="lindy-zenrows-configuration" />
    5. Change the request type for the remaining fields to **Auto** to allow Lindy to handle the configuration automatically.
    6. Delete the **Select Trigger** node and proceed directly to testing since we'll not use a trigger for this example.
    7. Click `Turn on` to activate your Lindy assistant.
    8. Click `Test` to verify the integration works correctly and returns the expected product name data.
  </Step>
</Steps>

### Step 3: Process and transform the scraped data

1. Click `Enter AI Agent` to add an intelligent data processing step to your workflow.
   <img alt="lindy-enter-ai-agent" />
2. Enter the following prompt to help you format the extracted product names into a format suitable for spreadsheet storage:
   ```plaintext theme={"dark"}
       Convert the output object values into individual strings
   ```
3. Click `+ Add exit condition` to define when the AI agent should complete its task.
4. In the **Condition** field, enter the following prompt:
   ```plaintext theme={"dark"}
   You have individual values extracted from the output object
   ```
5. Follow the same **Turn on** and **Test** steps to verify that the AI agent properly transforms your data.

### Step 4: Save results to Google Sheets

1. Add Google Sheets' **Append rows** action to your Lindy workflow.
   <img alt="lindy-google-sheets-append-rows-action" />
2. Connect your Google account to enable Lindy access to your spreadsheets.
3. Choose your target spreadsheet from the available options.
4. Select the appropriate **Sheet Title** where you want to store the extracted product data.
5. For the **Rows** field, click `Prompt AI` and enter the prompt:
   ```plaintext theme={"dark"}
       Fill the rows with individual product names received from the previous step.
   ```
   <img alt="lindy-configure-google-sheets-action" />
6. Follow the same **Turn on** and **Test** steps. The product names will be appended to your specified spreadsheet location.

## Troubleshooting

### AI Agent Processing Issues

* If your AI agent isn't properly converting data formats, review the prompt specificity and ensure it clearly describes the desired output structure.
* Verify that the exit condition is correctly defined. The agent should know exactly when its task is complete.
* Check the input data from the previous HTTP request step to ensure it contains the expected product information.

### Connection and Authentication Errors

* Ensure your Zenrows account has sufficient usage and isn't hitting concurrency limits during testing.
* Verify Google Sheets permissions are correctly granted when connecting your account to Lindy.

### Data Format and Output Problems

* If Google Sheets isn't receiving data in the expected format, test each step individually to isolate where the data transformation fails.
* Review the AI prompts in both the data processing and Google Sheets steps to ensure they're requesting compatible formats.
* Check that your target spreadsheet exists and the specified sheet name matches exactly.

### Request Issues

Different websites require specific configurations to retrieve content successfully. Some sites need additional parameters to handle dynamic content effectively.

**Common parameters you might need:**

* `wait` or `wait_for`: Ensures the page fully loads JavaScript-rendered elements before extraction
* `premium_proxy`: Provides high-quality proxies for reliable access
* `proxy_country`: Specifies the geographical location for the request to bypass geo-restrictions

For comprehensive troubleshooting guidance, explore [Zenrows' troubleshooting documentation](/fetch/troubleshooting/troubleshooting-guide).

## Tips and best practices

Follow these practices when integrating Lindy with Zenrows:

1. Define Clear Objectives
   * Identify your specific goals before starting. This guides your configuration and workflow design.
   * Example: For e-commerce scraping, decide whether you need product names, prices, descriptions, or all three.

2. Write Precise AI Prompts
   * Craft specific, goal-oriented prompts for AI agents.
   * Good: "Extract product names and format them as a list."
   * Avoid: "Process this data."

3. Optimize Data Transformation
   * Ensure the AI agent's output format matches subsequent steps (like spreadsheet rows).
   * Test different prompts to refine the transformation process.

4. Test Each Step
   * Verify each workflow component individually before connecting them.
   * Use Lindy's testing tools to debug issues step by step.

## Conclusion

Integrating Zenrows with Lindy automates web data extraction and streamlines your workflows. This combination eliminates manual data collection tasks and enables intelligent processing of scraped information.

## Frequently Asked Questions (FAQ)

<Accordion title="What is the purpose of integrating Lindy with Zenrows?">
  Integrating Lindy with Zenrows lets you automate web scraping and data processing workflows. For example, you can extract product data from websites and automatically store it in Google Sheets without manual work.
</Accordion>

<Accordion title="What kind of data can I scrape using Zenrows?">
  Zenrows can scrape a wide range of data, such as product details, business listings, articles, and more. The exact data depends on the website and the CSS selectors you set up.
</Accordion>

<Accordion title="How do I troubleshoot CSS selector issues?">
  If your CSS selectors aren't extracting the right data, use Zenrows' Parsers section to test and refine selectors. For more help, see the [Advanced CSS Selectors Troubleshooting Guide](/fetch/troubleshooting/advanced-css-selectors).
</Accordion>

<Accordion title="What should I do if my AI agent isn't processing data correctly?">
  Review your prompt to make sure it clearly describes the desired output. Check that your input data matches what the AI agent expects, and use Lindy's testing tools to debug the step.
</Accordion>

<Accordion title="Can I integrate other tools with Lindy?">
  Yes, Lindy supports integrations with many tools, including Google Sheets and Slack. You can combine these to build more complex workflows.
</Accordion>

<Accordion title="What happens if the scraped data changes format?">
  If the target website's structure changes, your CSS selectors may stop working. Regularly review and update your scraping setup to keep it working.
</Accordion>

<Accordion title="Can I schedule workflows to run automatically?">
  Yes, Lindy lets you set triggers or schedules for workflows. This is helpful for tasks like daily scraping or weekly reports.
</Accordion>

<Accordion title="What if I need help with advanced Lindy configurations?">
  Check Lindy's official documentation for advanced setups or contact their support teams for help with specific cases.
</Accordion>


# How to Integrate LlamaIndex with Zenrows
Source: https://docs.zenrows.com/integrations/llamaindex

Integrate Zenrows with LlamaIndex to build RAG applications that scrape, index, and query live web content from anti-bot protected websites.

Integrate <a href="https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/zenrows_web">Zenrows with LlamaIndex</a> to enable your RAG applications to access, index, and synthesize up-to-date web content from any website, including those with anti-bot protection and dynamic content.

## What Is LlamaIndex?

LlamaIndex is an open-source framework that connects LLMs to external data sources, databases, documents, and APIs. It provides tools for data ingestion, indexing, and query-based retrieval, commonly used to build retrieval-augmented generation (RAG) applications, which can be used to feed AI agents with up-to-date information.

## Key Integration Benefits

* **Uninterrupted access to data**: Build a reliable data layer that can access information from any website without getting blocked by anti-bot measures.
* **Real-time information retrieval**: Extract real-time data faster and more efficiently before it becomes stale.
* **Direct extraction of LLM-friendly data**: Get pre-formatted LLM-friendly data, such as the Markdown or JSON version of any website. Zenrows also enables the extraction of specific data directly.
* **Less code, more data**: Scrape data continuously with an auto-managed and auto-scaled solution with a simple API call.
* **Business-oriented development**: No extra engineering time and resources will be wasted on debugging or fixes.
* **Handle dynamic content easily**: Access heavily dynamic websites without performing complex waits and user simulations.
* **Borderless data retrieval**: Expose AI applications to data from any specific location without IP limitations using residential proxies with geo-targeted IPs.

## Use Cases of LlamaIndex-Zenrows Integration

* **Real-time price monitoring**: Use Zenrows to scrape prices from several product sites in real-time and synthesize a comprehensive comparison with an LLM.
* **Competitive research**: Scrape several competitors' offerings, product launches, strategies, and more with Zenrows and draw a correlation between the data using an LLM.
* **News and trends summarization**: Use Zenrows to aggregate news, trends, hashtags, and more, across similar platforms. Summarize the aggregated data with an LLM and extract specific insights.
* **Dynamic chatbots**: Build a chatbot that can access the web or specific web pages in real time to provide updated information.

## Getting Started: Basic Usage

This example demonstrates how to extract content from a protected website using the `ZenRowsWebReader`.

The `ZenRowsWebReader` enables you to use the official [Zenrows Fetch](https://www.zenrows.com/products/universal-scraper) as a data loader for web scraping in LlamaIndex.

<Steps>
  <Step title="Install the package">
    ```bash theme={"dark"}
    pip3 install llama-index-readers-web
    ```
  </Step>

  <Step title="Basic implementation">
    Import `ZenRowsWebReader` from `llama-index-readers-web`. Initialize `ZenRowsWebReader` as a reader instance. Then, set your Zenrows parameters through this instance.

    Load the target site as a document and return its content in the specified format (Markdown response):

    ```python Python theme={"dark"}
    # pip3 install llama-index-readers-web
    from llama_index.readers.web import ZenRowsWebReader

    api_key = "YOUR_ZENROWS_API_KEY"

    # initialize the reader
    reader = ZenRowsWebReader(
        api_key=api_key,
        js_render=True,
        premium_proxy=True,
        response_type="markdown",
    )

    # scrape a single URL
    documents = reader.load_data(["https://www.scrapingcourse.com/antibot-challenge/"])
    print(documents[0].text)
    ```

    The code returns a Markdown format of the target site, as shown:

    ```markdown Markdown theme={"dark"}
    [![](https://www.scrapingcourse.com/assets/images/logo.svg) Scraping Course](http://www.scrapingcourse.com/)

    # Antibot Challenge

    ![](https://www.scrapingcourse.com/assets/images/challenge.svg)

    ## You bypassed the Antibot challenge! :D
    ```
  </Step>
</Steps>

## Advanced Usage: Building a Simple RAG System

This example creates a simple RAG system that indexes multiple websites and responds to queries using the collected data.

You'll need an OpenAI API key to use the LLM and embedding features. So, prepare your OpenAI API key.

<Steps>
  <Step title="Install the packages">
    ```bash theme={"dark"}
    pip3 install llama-index-readers-web llama-index-llms-openai llama-index-embeddings-openai
    ```
  </Step>

  <Step title="Set up ZenRowsWebReader">
    Import the required packages and specify your Zenrows and OpenAI API keys. Initialize `ZenRowsWebReader` using the desired Zenrows parameters. Include `js_render` and `premium_proxy` to effectively bypass anti-bot measures.

    ```python Python theme={"dark"}
    # pip3 install llama-index-readers-web llama-index-llms-openai llama-index-embeddings-openai
    from llama_index.core import VectorStoreIndex
    from llama_index.readers.web import ZenRowsWebReader
    import os

    os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"

    api_key = "YOUR_ZENROWS_API_KEY"

    # set up ZenRowsWebReader
    reader = ZenRowsWebReader(
        api_key=api_key,
        js_render=True,
        premium_proxy=True,
        response_type="markdown",
        wait=2000,
    )
    ```
  </Step>

  <Step title="Set up a vector index">
    Specify the target URLs in a list, load their web pages as documents, and create a vectorized index of the documents:

    ```python Python theme={"dark"}
    # ...
    urls = [
        "https://www.scrapingcourse.com/ecommerce",
        "https://www.scrapingcourse.com/button-click",
        "https://www.scrapingcourse.com/infinite-scrolling",
    ]

    # load each URL as a document
    documents = reader.load_data(urls)

    # create index
    index = VectorStoreIndex.from_documents(documents)
    ```
  </Step>

  <Step title="Query the index">
    Initialize a query engine from the index, pass a prompt to query it, and return the query response:

    ```python Python theme={"dark"}
    # ...
    # query the content
    query_engine = index.as_query_engine()
    response = query_engine.query("What are the key features?")
    print(response)
    ```
  </Step>

  <Step title="Complete code">
    ```python Python theme={"dark"}
    # pip3 install llama-index-readers-web llama-index-llms-openai llama-index-embeddings-openai
    from llama_index.core import VectorStoreIndex
    from llama_index.readers.web import ZenRowsWebReader
    import os

    os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"

    api_key = "YOUR_ZENROWS_API_KEY"

    # set up ZenRowsWebReader
    reader = ZenRowsWebReader(
        api_key=api_key,
        js_render=True,
        premium_proxy=True,
        response_type="markdown",
        wait=2000,
    )

    urls = [
        "https://www.scrapingcourse.com/ecommerce",
        "https://www.scrapingcourse.com/button-click",
        "https://www.scrapingcourse.com/infinite-scrolling",
    ]

    # load each URL as a document
    documents = reader.load_data(urls)

    # create index
    index = VectorStoreIndex.from_documents(documents)

    # query the content
    query_engine = index.as_query_engine()
    response = query_engine.query("What are the key features?")
    print(response)
    ```

    LlamaIndex uses Zenrows to retrieve each website's information in Markdown format, vectorizes it, and synthesizes a response based on the query.

    Here's a sample response from the above code:

    ```markdown Markdown theme={"dark"}
    The key features include a menu with options like Shop, Home, Cart, Checkout, and My account. Additionally, there is a list of products with images, names, prices, and options to select or add to cart for each item.
    ```
  </Step>
</Steps>

Congratulations! 🎉You've integrated Zenrows with LlamaIndex.

## API Reference

| Parameter | Type | Description |
| - | - | - |
| `url` | str | Required. The URL to scrape |
| `js_render` | bool | Enable JavaScript rendering with a headless browser. Essential for modern web apps, SPAs, and sites with dynamic content (default: False) |
| `js_instructions` | str | Execute custom JavaScript on the page to interact with elements, scroll, click buttons, or manipulate content |
| `premium_proxy` | bool | Use residential IPs to bypass anti-bot protection. Essential for accessing protected sites (default: False) |
| `proxy_country` | str | Set the country of the IP used for the request. Use for accessing geo-restricted content. Two-letter country code |
| `session_id` | int | Maintain the same IP for multiple requests for up to 10 minutes. Essential for multi-step processes |
| `custom_headers` | dict | Include custom headers in your request to mimic browser behavior |
| `wait_for` | str | Wait for a specific CSS Selector to appear in the DOM before returning content |
| `wait` | int | Wait a fixed amount of milliseconds after page load |
| `block_resources` | str | Block specific resources (images, fonts, etc.) from loading to speed up scraping |
| `response_type` | str | Convert HTML to other formats. Options: "markdown", "plaintext", "pdf" |
| `css_extractor` | str | Extract specific elements using CSS selectors (JSON format) |
| `autoparse` | bool | Automatically extract structured data from HTML (default: False) |
| `screenshot` | str | Capture an above-the-fold screenshot of the page (default: "false") |
| `screenshot_fullpage` | str | Capture a full-page screenshot (default: "false") |
| `screenshot_selector` | str | Capture a screenshot of a specific element using CSS Selector |
| `screenshot_format` | str | Choose between "png" (default) and "jpeg" formats for screenshots |
| `screenshot_quality` | int | For JPEG format, set the quality from 1 to 100. Lower values reduce file size but decrease quality |
| `original_status` | bool | Return the original HTTP status code from the target page (default: False) |
| `allowed_status_codes` | str | Returns the content even if the target page fails with the specified status codes. Useful for debugging or when you need content from error pages |
| `json_response` | bool | Capture network requests in JSON format, including XHR or Fetch data. Ideal for intercepting API calls made by the web page (default: False) |
| `outputs` | str | Specify which data types to extract from the scraped HTML. Accepted values: emails, phone numbers, headings, images, audios, videos, links, menus, hashtags, metadata, tables, favicon |

<Note>For complete parameter documentation and details, see the official [Zenrows' Fetch Reference](/fetch/api-reference).</Note>

## Troubleshooting

### The returned response is incomplete:

* **Solution 1**: Ensure you activate `js_render` and `premium_proxy` to bypass anti-bot measures and scrape reliably.
* **Solution 2**: Apply enough `wait` time to allow dynamic content to load completely before scraping. If a specific element holding the required data loads slowly, you can also wait for it using the **wait\_for** parameter.
* **Solution 3**: If only partial responses are returned, the LLM may be missing relevant information in the chunk. Adjust the engine query retrieval by increasing the number of chunks the LLM receives from the documents. Increase the chunk by adding a **similarity\_top\_k** parameter to the query engine as shown:
  ```python Python theme={"dark"}
  # ...
  # query the content
  query_engine = index.as_query_engine(similarity_top_k=10)
  # …
  ```
* **Solution 4**: If you've used the `css_extractor` parameter to target specific elements, ensure you've entered the correct selectors.

### API key or authentication error

* **Solution**: Ensure you've supplied your LLM (e.g., OpenAI) and Zenrows API keys correctly.

### Module not found

* **Solution**: Install all the required modules:
  * `llama-index-readers-web`
  * `llama-index-llms-openai`
  * `llama-index-embeddings-openai`

## Resources

* <a href="https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/zenrows_web">ZenRowsWebReader on GitHub</a>

## Frequently Asked Questions (FAQ)

<Accordion title="What is the main use case of LlamaIndex-Zenrows integration?">
  The use cases of LlamaIndex-Zenrows integration are diverse. However, the primary application is to enable AI applications to access and reason over live, real-world web data, even from sites with anti-bot protections or dynamic content.
</Accordion>

<Accordion title="Does LlamaIndex-Zenrows integration support extraction via CSS selectors?">
  Yes, you can scrape data from specific elements using their CSS selectors via the `css_extractor` parameter.
</Accordion>

<Accordion title="Can I use all of Zenrows' parameters with ZenRowsWebReader?">
  Yes. The `ZenRowsWebReader` inherits all the features and capabilities of the Zenrows Fetch.
</Accordion>

<Accordion title="Which LLM integrations does LlamaIndex support?">
  LlamaIndex supports many popular LLMs, such as Groq, OpenAI, Anthropic, and more. Check LlamaIndex's official documentation for the supported LLMs.
</Accordion>

<Accordion title="Can I use Zenrows with LlamaIndex for Web Scraping?">
  LlamaIndex isn't explicitly designed for web scraping information from websites. However, you can add a scraping layer to LlamaIndex by pairing it with a web scraping tool like Zenrows, which provides it with anti-bot bypass capabilities.
</Accordion>


# How to Integrate Make with Zenrows
Source: https://docs.zenrows.com/integrations/make

Step-by-step guide to integrating Zenrows with Make for no-code web scraping automation. Build data extraction scenarios without programming.

Zenrows' integration with Make lets you extract web data and build automated pipelines without writing any code. This guide takes you through the steps of integrating Zenrows with Make using a real-life scraping example.

## What Is Make?

<a href="https://www.make.com/en?utm_source=zenrows-app&utm_medium=partner&utm_campaign=zenrows-app-partner-program">Make</a> is a visual platform that lets you design, build, and automate anything, from simple tasks to complex workflows, in minutes. With Make, you can send information between Zenrows and thousands of apps. It's fast and easy to use, visually intuitive, and requires zero coding expertise.

## Why Connect Zenrows with Make?

* **Automate web data collection end-to-end:** Connect Zenrows with 1,000+ apps in just a few clicks, no coding required. Pull web data from any site and route it directly to Google Sheets, CRMs, databases, or AI pipelines.
* **Visualize your data workflows:** Make lets you visually design and automate scenarios using a drag-and-drop interface. Connect Zenrows to any downstream app, set your scenario to run on a schedule or on demand, and watch it execute in real time whenever you need.
* **Bypass anti-bot protection without code:** Zenrows handles Cloudflare, DataDome, PerimeterX, and other bot protection systems automatically. Combined with Make's scenario builder, you can scrape JS-rendered, bot-protected pages and deliver structured data to any destination, without managing infrastructure or writing a single line of code.
* **Build faster and scale without limits:** Design workflows with as many steps as you need, add as many URLs as your pipeline requires, and schedule scraping jobs to refresh your data at any interval. Debug your workflow in real time rather than searching through logs or code.

## Available Zenrows Integrations

* **Monitor your API usage**: Returns Fetch usage information
* **Make an API Call**: Sends a GET, POST, or PUT request to a target URL via Fetch.
* **Scraping a URL with Autoparse**: Automates Fetch to auto-parse a target website's HTML elements. It returns structured JSON data.
  <Note>The autoparse option only works for some websites. Learn more about how it works in the [autoparse FAQ](/fetch/faq#what-is-autoparse).</Note>
* **Scraping a URL with CSS Selectors**: Automates data extraction from an array of CSS selectors.

## Watch the Video Tutorial

Learn how to set up the Make ↔ Zenrows integration step-by-step by watching this video tutorial:

<iframe title="Zenrows ↔ Make" />

## Real-World End-to-End Integration Example

We'll build an end-to-end Amazon scraping Scenario using Make and Zenrows with the following steps:

1. Pull target URLs from Google Sheets into Make's Iterator
2. Utilize Zenrows' **Scraping a URL with Autoparse** integration option to extract data from URLs automatically
3. Save the extracted data to Google Sheets

### Step 1: Create a new Scenario on Make

1. Log in to your account at <a href="https://www.make.com/en/register?utm_source=zenrows-app&utm_medium=partner&utm_campaign=zenrows-app-partner-program">[https://make.com/](https://make.com/)</a>
2. Go to **Scenarios** on the left sidebar
3. Click **+ Create a new scenario** at the top right

<img alt="Start Make scenario" />

### Step 2: Connect the Google Sheets containing the target URLs

1. Enter your Scenario's name at the top-left (e.g., Amazon Scraper)
2. Click the `+` icon in the middle of the screen
   <img alt="Name scenario" />
3. Search and select **Google Sheets** from the modal
   <img alt="Flow control module selection" />
4. Select **Get Range Values** from the options
   <img alt="Make Google Sheets range setup" />
5. If prompted, click **Create a connection** to link Make with your Google account
   <img alt="Google connection Make" />
6. Click the **Search Method** dropdown and select **Enter manually**
7. Paste the Google Sheets ID in the **Spreadsheet ID** field
8. Enter the sheet's name in the Sheet Name field
9. Provide the range of cells containing the target URLs in the **Range** field (e.g., A2:A11)
10. Click **Save**

<img alt="Make Google Sheets setup box" />

### Step 3: Load the Target URLs into an Iterator

1. Click `+` next to the Google Sheets module to create a new connection
2. Search and select **Flow Control**, then click **Iterator**
   <img alt="Make Iterator setup" />
3. Place your cursor in the Array field and select the column containing the target URLs (i.e., **A**) from the integrated Google Sheets
4. Click **Save**
   <img alt="Make Iterator cell" />

### Step 4: Connect Zenrows with the Make workflow

1. Click the `+` icon
2. Search and select **Zenrows**
3. Click **Scrape a URL with Autoparse**
   <img alt="Make Zenrows integration" />
4. Click **Create a Connection**
   <img alt="Make Zenrows connection setup" />
5. Under **Connection type**, select **Zenrows**
6. Enter a connection name in the **Connection name** field
7. Provide your Zenrows API key in the **API Key** field
   <img alt="Make Zenrows connection setup" />
8. Click **Save**
9. Place your cursor in the URL field, then select **Value** from the Flow Control option in the modal box
   <img alt="Make Zenrows URL field setup" />
10. Click **Show advanced settings**
11. Select **Yes** for **Premium Proxy** and **JavaScript Rendering**
12. Click **Save**

<img alt="Make Zenrows advanced setup" />

### Step 5: Store the scraped data in a Google Sheets

1. Click **Run once** in the lower left corner to scrape data from the URLs
   <img alt="Make Zenrows run" />
2. Add another named sheet (e.g., **Products**) to the connected Google Sheets. Include the following columns in the new sheet:
   * Name
   * Rating Value
   * Review Count
   * Price (USD)
   * Out of Stock
3. Click the `+` icon to add another module. Then, select **Google Sheets**.
4. Select **Add a row**
   <img alt="Make Google Sheets add row setup" />
5. Select **Enter manually** from the **Search Method** dropdown ⇒ enter the **Spreadsheet ID** and **Sheet Name**.
6. Under **Column Range**, select **A-Z**. Then, map the columns using the data extracted from Zenrows.
   <img alt="Make Google Sheets column mapping" />
7. Place your cursor in each column field and map it with the extracted data field as follows:
   * **A** = `Value` (from the Iterator)
   * **B** = `title`
   * **C** = `avg_rating`
   * **D** = `review_count`
   * **E** = `price`
   * **F** = `out_of_stock`
   <img alt="Make Zenrows scraped data mapping" />
8. Click **Save**

### Step 6: Run the Make Scenario and Validate the Extraction

Click **Run once** in the lower-left corner of the screen. The workflow runs iteratively for each URL in the connected Google Sheets.

The workflow scrapes the data into the Products sheet as shown:

<img alt="Make Google Sheets add row setup" />

Congratulations! You've successfully integrated Zenrows with Make and automated your web scraping workflow.

## Zenrows Configuration Options

Zenrows supports the following configuration options on Make:

| Configuration | Function |
| - | - |
| **URL** | The URL of the target website |
| **Headers** | <Tooltip>Adds the custom headers to the request</Tooltip> |
| **Premium Proxy** | When activated, it routes requests through the Zenrows Residential Proxies, instead of the default Datacenter proxies |
| **Proxy Country** | <Tooltip>The country geolocation to use in a request</Tooltip> |
| **JavaScript Rendering** | Ensures that dynamic content loads before scraping |
| **Wait for Selector** | <Tooltip>Pauses scraping execution until a particular selector is visible in the DOM</Tooltip> |
| **Wait Milliseconds** | <Tooltip>Waits for a fixed amount of time before executing the scraper</Tooltip> |
| **Window Width** | Sets the browser's window width |
| **Window Height** | Sets the browser's window height |
| **JavaScript Instructions** | <Tooltip>Passes JavaScript code to execute actions like clicking, scrolling, and more</Tooltip> |
| **Session ID** | Uses a session ID to maintain the same IP for multiple API requests for up to 10 minutes |
| **Original Status** | Returns the original status code returned by the target website |

## Troubleshooting

### Issue: The workflow returns incomplete data

**Solution**: Ensure that JavaScript Rendering and Premium Proxy are enabled in the Zenrows configuration.

### Issue: \[400] Error with internal code 'REQS004'

**Solution 1**: Double-check the target URL and ensure it's not malformed or missing essential query strings.

**Solution 2**: If using the CSS selector integration, ensure the extractor parameter is a valid JSON array of objects (e.g., `[{"title":"#productTitle", "price": ".a-price-whole"}]`).

### Issue: Failed request or 401 unauthorized response

**Solution 1**: Ensure you supply your Zenrows API key.

**Solution 2**: Double-check to ensure you've provided the correct Zenrows API key.

### Issue: The Google Sheets module cannot find the spreadsheet

**Solution**: Double-check the Spreadsheet ID and ensure the Google Sheets API is enabled for your account.

## Frequently Asked Questions (FAQ)

<Accordion title="Can I use My Existing Zenrows API Key with Make?">
  Yes. You don't need a separate Zenrows API key for Make. You need to provide your existing API key while connecting Make with Zenrows.
</Accordion>

<Accordion title="Can I Scrape with CSS Selectors on Make?">
  Make supports Zenrows CSS selector integration. It allows you to manually pass the target website's CSS selectors as an array.
</Accordion>

<Accordion title="Can I use the autoparse integration for all websites?">
  The autoparse integration isn't available for all websites. If you use the **Scraping a URL with Autoparse** integration for an unsupported website, the scraper will return empty or incomplete data or an error. Visit the [Zenrows web scraper](https://www.zenrows.com/scraper) page to view the supported websites.
</Accordion>

## How do I get started with Make?

1. <a href="https://www.make.com/en/register?utm_source=zenrows-app&utm_medium=partner&utm_campaign=zenrows-app-partner-program">Sign up for Make:</a> a free account is available forever.
2. Create a new scenario by connecting app modules and selecting triggers and actions.
3. Run a quick test to verify the scenario works correctly.
4. Set the scenario to run on a schedule and let it run automatically.

<Tip>For more information and templates, visit <a href="https://www.make.com/en/integrations/zenrows?utm_source=zenrows-app&utm_medium=partner&utm_campaign=zenrows-app-partner-program">Make's Zenrows integration page</a>, <a href="https://apps.make.com/zenrows?utm_source=zenrows-app&utm_medium=partner&utm_campaign=zenrows-app-partner-program">documentation</a>, or the <a href="https://www.make.com/en/templates?utm_source=zenrows-app&utm_medium=partner&utm_campaign=zenrows-app-partner-program">Make template library</a>.</Tip>


# How to Integrate Zenrows with Mastra
Source: https://docs.zenrows.com/integrations/mastra

Add the Zenrows MCP to Mastra and give your agents reliable access to any website without getting blocked. Turn live web data into structured output and ship AI features your users actually want right from your TypeScript application.

## What Is Mastra?

<a href="https://mastra.ai">Mastra</a> is an open-source TypeScript framework for building AI agents and workflows. It gives developers everything needed to ship production-grade AI features: a model router with 3,000+ models, agents with memory and tool use, multi-step workflows with branching logic, RAG pipelines, evals, and observability, all in one framework. It integrates with Next.js, React, Express, SvelteKit, Hono, and any other modern TypeScript stack.

Unlike no-code or desktop agent tools, Mastra agents are code. They live in your codebase, deploy with your app, and connect to your existing infrastructure.

## Why Connect Zenrows to Mastra?

* **Ship agents that access any site:** Mastra agents call tools to do their work. Without Zenrows, any page protected by Cloudflare, DataDome, or similar systems returns a blocked or empty response. Add Zenrows and your agents bypass the blocks to fetch the actual content every time.
* **Structured output, ready for your app:** Zenrows returns clean Markdown or JSON instead of raw HTML. Mastra agents receive structured, LLM-friendly data, resulting in less post-processing, fewer tokens, and cleaner output to pass downstream.
* **Give any model on Mastra's router real-time web access:** Mastra supports 3,000+ models from OpenAI, Anthropic, Google, Groq, Mistral, and more. Zenrows MCP enables real-time web access regardless of which model your agent uses.
* **Multi-agent workflows with live web data:** Mastra supports supervisor agents that coordinate subagents. Each subagent can call Zenrows independently, and a coordinator synthesizes the results.
* **Ship web data features into your product without building a scraping layer:** With Zenrows MCP, your agent calls a tool and gets structured content back. No scraping infrastructure to maintain and no browser pool to spin up alongside your app.
* **Keep your RAG pipelines current:** Connect Zenrows with Mastra and your RAG ingestion workflow can pull live content from any source on demand, including pages behind bot protection and heavy JavaScript rendering that a plain HTTP fetch would never reach.

## What You Can Build with Mastra and Zenrows

* **A competitive pricing monitor shipped inside your product:** Build an agent your users can query that scrapes live pricing across sites via Zenrows and returns a structured answer. Deploy it as part of your Next.js or Express app using Mastra's server adapters.
* **An e-commerce research workflow:** Chain a multi-step Mastra workflow to scrape a product category page, extract product names and prices, enrich with reviews from a second page, run a structured-output step to rank by value, and return a clean JSON response to your frontend.
* **A sales intelligence agent:** Build an internal copilot that researches a prospect before a call, scraping their company site, recent news, and job postings via Zenrows, and returns a structured brief via your Slack channel or internal tool.
* **Content automation for a CMS:** Set up a Mastra workflow that scrapes a set of sources on a schedule, extracts key content, runs it through a summarization or transformation step, and writes structured output directly to your CMS or knowledge base.
* **A data analysis agent with live market data:** Extend Mastra's text-to-SQL and data analysis agent template with Zenrows to pull live market data, competitor pricing, or industry benchmarks from the web, then feed it into the analysis pipeline alongside your internal database.

## Prerequisites

* Node.js 18+ installed.
* A Mastra project scaffolded via `npm create mastra@latest`, which creates the full project structure (`src/mastra/agents`, `tools`, `workflows`, `index.ts`) that you'll build on top of.
* An Anthropic API key (or your preferred provider).
* A Zenrows API key from the [Zenrows Playground](https://app.zenrows.com/fetch).

If you haven't created a project yet, run:

```bash theme={"dark"}
npm create mastra@latest
```

The setup wizard will ask for your project name and LLM provider. Choose **Anthropic** to follow this guide. It will prompt you to enter your **Anthropic API key** during setup and automatically write it to your `.env` file as `ANTHROPIC_API_KEY`.

## Setup

<Steps>
  <Step title="Install the required packages">
    Install `@mastra/mcp` and the Anthropic AI SDK provider:

    ```bash theme={"dark"}
    npm install @mastra/mcp@latest @ai-sdk/anthropic
    ```

    `@mastra/mcp` enables MCP client support in your Mastra project. `@ai-sdk/anthropic` is the Anthropic provider for the AI SDK, enabling your agent to connect directly to Claude without going through Mastra's model router.
  </Step>

  <Step title="Configure Zenrows as an MCP server">
    Create an `MCPClient` that points to the Zenrows MCP server.

    Create `src/mastra/mcp/zenrows-client.ts`:

    ```typescript TypeScript theme={"dark"}
    // npm install @mastra/mcp@latest
    import { MCPClient } from '@mastra/mcp'

    // Connect to the MCP server using the MCPClient class
    export const zenrowsClient = new MCPClient({
      id: 'zenrows-mcp-client',
      servers: {
        zenrows: {
          command: 'npx',
          args: ['-y', '@zenrows/mcp'],
          env: {
            ZENROWS_API_KEY: process.env.ZENROWS_API_KEY!,
          },
        },
      },
    })
    ```

    Ensure both API keys are in your `.env` file. `ANTHROPIC_API_KEY` should already be there from the project setup wizard. Add `ZENROWS_API_KEY` alongside it:

    ```bash theme={"dark"}
    ANTHROPIC_API_KEY=YOUR_ANTHROPIC_API_KEY
    ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY
    ```

    Alternatively, connect via the remote HTTP server instead:

    ```typescript TypeScript theme={"dark"}
    export const zenrowsClient = new MCPClient({
      id: 'zenrows-mcp-client',
      servers: {
        zenrows: {
          url: new URL('https://mcp.zenrows.com/mcp'),
          requestInit: {
            headers: {
              Authorization: `Bearer ${process.env.ZENROWS_API_KEY}`,
            },
          },
        },
      },
    })
    ```
  </Step>

  <Step title="Add Zenrows tools to your agent">
    Create a product research agent powered by Claude Sonnet that scrapes live product listings via Zenrows and returns structured results. Import the client and pass its tools to your agent using `listTools()`.

    Create `src/mastra/agents/research-agent.ts`:

    ```typescript TypeScript theme={"dark"}
    // npm install @mastra/mcp@latest @ai-sdk/anthropic
    import { Agent } from '@mastra/core/agent'
    import { anthropic } from '@ai-sdk/anthropic'
    import { zenrowsClient } from '../mcp/zenrows-client.js'

    export const researchAgent = new Agent({
      id: 'research-agent',
      name: 'Research Agent',
      instructions: `You are a web research assistant. Use Zenrows tools to
        scrape pages and extract structured data. Always return clean,
        structured results.`,
      model: anthropic('claude-sonnet-4-6'),
      tools: await zenrowsClient.listTools(),
    })
    ```

    `listTools()` loads all Zenrows tools into the agent. If you only need a subset of tools, load them first and filter to keep the prompt lighter:

    ```typescript TypeScript theme={"dark"}
    // npm install @mastra/mcp@latest @ai-sdk/anthropic
    import { Agent } from '@mastra/core/agent'
    import { anthropic } from '@ai-sdk/anthropic'
    import { zenrowsClient } from '../mcp/zenrows-client.js'

    // Fetch the list of tools from the Zenrows MCP server
    const allTools = await zenrowsClient.listTools()

    export const researchAgent = new Agent({
      id: 'research-agent',
      name: 'Research Agent',
      instructions: `You are a web research assistant. Use Zenrows tools to
        scrape pages and extract structured data. Always return clean,
        structured results.`,
      model: anthropic('claude-sonnet-4-6'),
      // Only include zenrows_scrape if it's available
      tools: allTools['zenrows_scrape'] ? { zenrows_scrape: allTools['zenrows_scrape'] } : {},
    })
    ```

    Register the agent in your Mastra instance. `src/mastra/index.ts` already exists from the project scaffold. Add `researchAgent` to the imports and include it in the `agents` object:

    ```typescript TypeScript theme={"dark"}
    import { researchAgent } from './agents/research-agent.js'

    // Add researchAgent to your existing Mastra instance
    export const mastra = new Mastra({
      agents: { researchAgent },
    })
    ```
  </Step>

  <Step title="Run your agent">
    Start the Mastra dev server:

    ```bash theme={"dark"}
    npm run dev
    ```

    **Test via Swagger UI**

    Open `http://localhost:4111/swagger-ui` in your browser to see all your registered agents as callable API endpoints.

    Go to `http://localhost:4111/agents`, select `research-agent`, and you'll see all available Zenrows MCP tools connected to your agent in the right sidebar.

    Fill in a scraping prompt and send it:

    ```text wrap theme={"dark"}
    Scrape https://www.scrapingcourse.com/antibot-challenge and summarize the page.
    ```

    Sample response:

    ```text wrap theme={"dark"}
    I'll scrape that page for you and provide a summary.

    [zenrows_scrape]

    Summary

    The page at https://www.scrapingcourse.com/antibot-challenge is a test page designed to
    challenge web scrapers with anti-bot protection.

    Key Content:
    - Site: Scraping Course (scrapingcourse.com)
    - Page Title: "Antibot Challenge"
    - Main Message: "You bypassed the Antibot challenge! :D"

    This appears to be a training/testing page that implements anti-bot measures.
    [Truncated for brevity]
    ```

    <Note>
      Zenrows is connected as an `MCPClient`, not an `MCPServer`. `MCPClient` connects your Mastra agent *to* an external MCP server (in this case, Zenrows) to consume its tools. `MCPServer` does the opposite: it exposes your Mastra agents and tools *to* other MCP-compatible clients. Use `MCPClient` here because you want Mastra to consume Zenrows tools, not the other way around.
    </Note>

    **Test in code**

    You can also call the agent directly from a script. Create `src/scripts/test-agent.ts`:

    ```typescript TypeScript theme={"dark"}
    // Import .env config and the mastra instance from the index file
    import 'dotenv/config'
    import { mastra } from '../mastra/index.js'

    // Get the research agent from the mastra instance
    const agent = mastra.getAgentById('research-agent')

    // Generate a response using a research query
    const response = await agent.generate(
      'Get the top 5 deals on https://www.amazon.com/s?k=headset based on product price and rating.'
    )

    console.log(response.text)
    ```

    Run it from your terminal:

    ```bash theme={"dark"}
    npx tsx src/scripts/test-agent.ts
    ```

    **Expose as a custom API endpoint**

    Once you're happy with the agent, expose it as a proper HTTP endpoint so any frontend, Slack bot, or external service can call it. `registerApiRoute` is a helper from `@mastra/core/server` that lets you define custom HTTP routes on Mastra's built-in server.

    Update `src/mastra/index.ts`:

    ```typescript TypeScript theme={"dark"}
    import { researchAgent } from './agents/research-agent.js'
    import { registerApiRoute } from '@mastra/core/server'

    export const mastra = new Mastra({
      agents: { researchAgent },
      server: {
        apiRoutes: [
          registerApiRoute('/research', {
            method: 'POST',
            handler: async (c: any) => {
              const mastra = c.get('mastra')
              const agent = mastra.getAgentById('research-agent')
              const { prompt } = await c.req.json()
              const response = await agent.generate(prompt)
              return c.json({ result: response.text })
            },
          }),
        ],
      },
    })
    ```

    With the dev server running, your agent is now callable at:

    ```bash theme={"dark"}
    curl -X POST http://localhost:4111/research \
      -H "Content-Type: application/json" \
      -d "{\"prompt\": \"Scrape the top headlines from news.ycombinator.com and summarize the top 3 stories in one paragraph.\"}"
    ```
  </Step>
</Steps>

## Dynamic Tools for Multi-Tenant Apps

If you're building a SaaS app where each user provides their own Zenrows API key, use `listToolsets()` instead of `listTools()`. This lets you pass per-request credentials without reinitializing the agent.

In this pattern, `zenrows-client.ts` is not used. Instead, a new `MCPClient` is created per request using the user's API key and discarded after the response is received. This logic lives in your request handler inside a `registerApiRoute` handler in `index.ts`.

Add the import at the top of `index.ts`:

```typescript TypeScript theme={"dark"}
import { MCPClient } from '@mastra/mcp'
```

Then add a new route inside `server.apiRoutes`:

```typescript TypeScript theme={"dark"}
registerApiRoute('/research-user', {
  method: 'POST',
  handler: async (c: any) => {
    const { prompt, apiKey } = await c.req.json()
    const userMcp = new MCPClient({
      servers: {
        zenrows: {
          url: new URL('https://mcp.zenrows.com/mcp'),
          requestInit: {
            headers: {
              Authorization: `Bearer ${apiKey}`,
            },
          },
        },
      },
    })
    const mastra = c.get('mastra')
    const agent = mastra.getAgentById('research-agent')
    const response = await agent.generate(prompt, {
      toolsets: await userMcp.listToolsets(),
    })
    await userMcp.disconnect()
    return c.json({ result: response.text })
  },
}),
```

With the dev server running, test the endpoint by passing the prompt and the user's Zenrows API key in the request body:

```bash theme={"dark"}
curl -X POST http://localhost:4111/research-user \
  -H "Content-Type: application/json" \
  -d "{\"prompt\": \"Scrape https://www.scrapingcourse.com/antibot-challenge and summarize the page.\", \"apiKey\": \"YOUR_ZENROWS_API_KEY\"}"
```

<Note>
  In this pattern, `ZENROWS_API_KEY` doesn't need to be set in your `.env`. Each request supplies its own key at runtime, so there is no shared key on the server.
</Note>

## Troubleshooting

### This Site Can't Be Reached or localhost:4111 Unreachable

The Mastra dev server isn't running. Start it with:

```bash theme={"dark"}
npm run dev
```

If the server starts but the port is still unreachable, check that no other process is using port 4111. You can also confirm the server is up by checking the Mastra startup log in your terminal before making any requests.

### 500 Internal Server Error on Custom Routes

Check that you're using `mastra.getAgentById('research-agent')` and not `mastra.getAgent('research-agent')` in your route handlers. The correct method is `getAgentById`. Using `getAgent` will cause a 500 with no clear error message in the response.

### Could Not Find Config for Provider Anthropic Error

This occurs when using Mastra's model router string format (`anthropic/claude-sonnet-4-6`) without Mastra's gateway configured. You may also see a follow-up error such as `"model: anthropic/claude-sonnet-4-6 not found"` if the router tries to pass the full string directly to the Anthropic API.

The fix is to use Anthropic's AI SDK provider directly instead of the router string:

```bash theme={"dark"}
npm install @ai-sdk/anthropic
```

For other providers, the pattern is the same with a different package. For example, `@ai-sdk/openai` for OpenAI and `@ai-sdk/google` for Google. See the <a href="https://ai-sdk.dev/providers/ai-sdk-providers">AI SDK providers docs</a> for the full list.

When using the AI SDK directly, pass only the model ID without the `anthropic/` prefix:

```typescript TypeScript theme={"dark"}
import { anthropic } from '@ai-sdk/anthropic'

// Do not use 'anthropic/claude-sonnet-4-6'
// Use this instead:
model: anthropic('claude-sonnet-4-6'),
```

### Prompt Is Too Long Error

Mastra injects all registered tool schemas into the prompt on every request. When combined with page content returned by Zenrows, this can push the total token count past the model's context limit.

**Option 1:** Load tools first and select only the ones you need, such as `zenrows_scrape`:

```typescript TypeScript theme={"dark"}
const allTools = await zenrowsClient.listTools()

export const researchAgent = new Agent({
  id: 'research-agent',
  name: 'Research Agent',
  instructions: `You are a web research assistant. Use Zenrows tools to
    scrape pages and extract structured data. Always return clean,
    structured results.`,
  model: anthropic('claude-sonnet-4-6'),
  tools: allTools['zenrows_scrape'] ? { zenrows_scrape: allTools['zenrows_scrape'] } : {},
})
```

**Option 2:** Switch to a model with a larger context window. Claude Opus 4.6 supports up to 1M tokens. Update the `model` field in `research-agent.ts`:

```typescript theme={"dark"}
model: anthropic('claude-opus-4-6'),
```

### Cannot Find Name 'process' or Relative Import Path Errors

Add `"types": ["node"]` to your `tsconfig.json` compiler options:

```json theme={"dark"}
{
  "compilerOptions": {
    "types": ["node"]
  }
}
```

### Zenrows Tools Not Appearing on the Agent

* Confirm `@mastra/mcp` is installed: `npm list @mastra/mcp`.
* Check that `ZENROWS_API_KEY` is set in your `.env` and that dotenv is loading it.
* When running scripts directly with `npx tsx`, add `import 'dotenv/config'` at the top of the file. The dev server loads `.env` automatically, but standalone scripts don't.
* Make sure `await zenrowsClient.listTools()` is called at the module level in `research-agent.ts`, outside the `Agent` constructor. The `await` is required and must be at the top level of the file.

### Authentication Error from Zenrows

Verify your API key on the <a href="https://app.zenrows.com">Zenrows dashboard</a>. If using the HTTP connection option, ensure the `Authorization` header is set to `Bearer YOUR_ZENROWS_API_KEY` with no extra whitespace.

### Workflow Step Not Receiving Zenrows Output

Check the `outputSchema` of your scrape step matches what the next step expects. Use Mastra Studio at `http://localhost:4111` to inspect step inputs and outputs visually.

## Further Reading

* <a href="https://mastra.ai/docs/mcp/overview">Mastra MCP Overview</a>
* <a href="https://mastra.ai/docs/agents/overview">Mastra Agents</a>
* <a href="https://mastra.ai/docs/workflows/overview">Mastra Workflows</a>
* <a href="https://mastra.ai/reference/tools/mcp-client">MCPClient Reference</a>
* <a href="https://mastra.ai/docs/mcp/overview#dynamic-tools">Dynamic Tools for Multi-Tenant Apps</a>
* [Zenrows MCP Documentation](https://docs.zenrows.com/mcp/overview)

## Frequently Asked Questions

<Accordion title="Does Zenrows work with all models on Mastra's model router?">
  Yes. Zenrows MCP works regardless of which model you configure on your agent. That said, Claude Sonnet and Opus handle MCP tool call sequences most reliably. Mastra's model router gives you access to thousands of models across 100+ providers. See the full list at the <a href="https://mastra.ai/models">Mastra Providers page</a>. You can switch models without changing anything else in your agent code.
</Accordion>

<Accordion title="Can each user in my app use their own Zenrows API key?">
  Yes. Use `listToolsets()` instead of `listTools()` and create a new `MCPClient` per request with the user's key. In this case, you don't need to set `ZENROWS_API_KEY` in your `.env`. Each request is self-contained. See the [Dynamic Tools for Multi-Tenant Apps](#dynamic-tools-for-multi-tenant-apps) section for the full implementation.
</Accordion>

<Accordion title="Should I use listTools() or listToolsets()?">
  Use `listTools()` for single-user or internal tools where the API key is shared. Use `listToolsets()` for multi-tenant SaaS apps where each user provides their own credentials, avoiding the need to reinitialize the agent per request.
</Accordion>

<Accordion title="Can I require human approval before Zenrows runs?">
  Yes. Set `requireToolApproval: true` on the Zenrows server block in your `MCPClient` config:

  ```typescript TypeScript theme={"dark"}
  export const zenrowsClient = new MCPClient({
      servers: {
          zenrows: {
              command: 'npx',
              args: ['-y', '@zenrows/mcp'],
              env: {
              ZENROWS_API_KEY: process.env.ZENROWS_API_KEY!,
              },
              requireToolApproval: true,
          },
      },
  })
  ```

  This integrates with Mastra's human-in-the-loop approval flow.
</Accordion>

<Accordion title="Can I use Zenrows alongside other MCP servers in the same agent?">
  Yes. Add multiple servers to the same `MCPClient`:

  ```typescript TypeScript theme={"dark"}
  export const mcpClient = new MCPClient({
      id: 'multi-mcp-client',
      servers: {
          zenrows: {
              command: 'npx',
              args: ['-y', '@zenrows/mcp'],
              env: {
              ZENROWS_API_KEY: process.env.ZENROWS_API_KEY!,
              },
          },
          github: {
              command: 'npx',
              args: ['-y', '@modelcontextprotocol/server-github'],
              env: {
              GITHUB_PERSONAL_ACCESS_TOKEN: process.env.GITHUB_TOKEN!,
              },
          },
      },
  })
  ```

  All tools from all servers are available to the agent via a single `listTools()` call.
</Accordion>

<Accordion title="Can I test my Zenrows agent in Mastra Studio?">
  Yes. Mastra Studio (`http://localhost:4111`) lets you run your agent interactively, inspect every tool call including Zenrows requests and responses, view token usage, and debug step by step without writing test scripts. See the <a href="https://mastra.ai/docs/studio/overview">Mastra Agent Studio overview</a> for more information.
</Accordion>


# MCP Integrations
Source: https://docs.zenrows.com/integrations/mcp/mcp-integrations

Every AI assistant, code editor, SDK, framework, and agent platform that connects to the live web through the Zenrows MCP server.

The [Zenrows MCP server](/mcp/overview) is the standard way AI systems use Zenrows. Any MCP-compatible client can connect, here's every tool with its own setup guide.

## AI Assistants

<CardGroup>
  <Card title="Claude Desktop" icon="message-chatbot" href="/integrations/claude-desktop">
    Connect Claude Desktop to Zenrows via MCP for real-time web access in your conversations.
  </Card>

  <Card title="Claude Code" icon="terminal" href="/integrations/claude-code">
    Give Claude Code the ability to scrape and extract data while it works in your codebase.
  </Card>
</CardGroup>

## AI Code Editors

<CardGroup>
  <Card title="Cursor" icon="device-laptop" href="/integrations/cursor">
    Scrape and extract data from inside Cursor's AI coding assistant.
  </Card>

  <Card title="Windsurf" icon="device-laptop" href="/integrations/windsurf">
    Connect Windsurf's AI agent to real-time web data.
  </Card>

  <Card title="VS Code" icon="device-laptop" href="/integrations/vscode">
    Add Zenrows to VS Code's AI features via MCP.
  </Card>

  <Card title="Zed" icon="device-laptop" href="/integrations/zed">
    Connect Zed's AI assistant to Zenrows.
  </Card>

  <Card title="JetBrains IDEs" icon="device-laptop" href="/integrations/jetbrains">
    Add Zenrows to JetBrains' AI assistant across IntelliJ, PyCharm, WebStorm, and more.
  </Card>

  <Card title="OpenCode" icon="device-laptop" href="/integrations/opencode">
    Connect OpenCode's AI agent to real-time web data.
  </Card>

  <Card title="Google Antigravity 2.0" icon="device-laptop" href="/integrations/antigravity-20">
    Add Zenrows to Google's agentic development platform.
  </Card>

  <Card title="Google Antigravity IDE" icon="device-laptop" href="/integrations/antigravity-ide">
    Connect Zenrows to the Antigravity IDE.
  </Card>
</CardGroup>

## Agent SDKs & Frameworks

<CardGroup>
  <Card title="Claude Agent SDK" icon="code" href="/integrations/claude-agent-sdk">
    Build autonomous agents that scrape and automate any site with Anthropic's official SDK.
  </Card>

  <Card title="OpenAI Agents SDK" icon="code" href="/integrations/openai-agents-sdk">
    Build multi-step agents with OpenAI's official Agents SDK.
  </Card>

  <Card title="Google ADK" icon="code" href="/integrations/google-adk">
    Build multi-agent workflows with Google's Agent Development Kit.
  </Card>

  <Card title="Mastra" icon="code" href="/integrations/mastra">
    Give agents built with the Mastra TypeScript framework reliable access to any website.
  </Card>
</CardGroup>

## AI Tools

<CardGroup>
  <Card title="Anthropic Claude API" icon="wand" href="/integrations/anthropic-claude-api">
    Ground Claude's answers in live, protected, or JavaScript-heavy web data.
  </Card>
</CardGroup>

## Agent Platforms

<CardGroup>
  <Card title="Claude Managed Agents" icon="cloud" href="/integrations/claude-managed-agents">
    Let Anthropic manage the agent loop and sandbox while Zenrows handles the scraping layer.
  </Card>

  <Card title="Hermes" icon="cloud" href="/integrations/hermes">
    Give Hermes Agent reliable, scheduled access to any website.
  </Card>

  <Card title="OpenClaw" icon="cloud" href="/integrations/openclaw">
    Connect OpenClaw's autonomous agent to Zenrows MCP for real-time web access.
  </Card>
</CardGroup>

<Note>
  Looking for a framework that doesn't use MCP directly, like LangChain, LlamaIndex, or Flowise? See the full [Integrations overview](/integrations/overview) for every way to connect Zenrows, MCP or otherwise.
</Note>


# How to Integrate MuleSoft with Zenrows
Source: https://docs.zenrows.com/integrations/mulesoft

Connect Zenrows web scraping capabilities to MuleSoft Anypoint Platform workflows for enterprise-grade automated data extraction pipelines.

Connect Zenrows' web scraping capabilities to your MuleSoft workflows through the Anypoint platform. This integration enables automated data extraction from websites directly within your existing automation processes.

## What Is MuleSoft?

MuleSoft is a no-code/low-code automation platform that connects and exchanges data between various applications and systems. One of its core components is the MuleSoft Anypoint platform, which enables users to integrate systems via an API-focused approach.

## Use Cases

Here are some use cases of MuleSoft-Zenrows integration.

* **Competitor monitoring**: Schedule automated scraping of competitor websites to track pricing, product changes, or content updates.
* **Sentiment analysis**: Gather product reviews data from various sources with Zenrows and analyze their sentiment using LLM integration.
* **Demand forecasting**: Use cron jobs to scrape demand signals from retail and ecommerce sites with Zenrows. Calculate trends and historical moves with JavaScript via Anypoint's Scripting module. Store time-series data in a database and forecast anticipated demand using LLM.
* **Best deal recommendation**: Collect product data from various platforms with Zenrows and use an LLM to analyze key points, such as price, demand history, reviews, and ratings, to recommend the best deal for your customers.

## Initial Integration Steps via Anypoint Platform

The best way to integrate MuleSoft with Zenrows is via Anypoint. This guide assumes that you already have an Anypoint account with the <a href="https://www.mulesoft.com/platform/studio">Anypoint Studio</a> downloaded and installed.

We'll create a scraping workflow that extracts data from an <a href="https://www.amazon.com/Gamrombo-Controller-Compatible-Vibration-Playstation-5/dp/B0DZCSBCLF/">Amazon product page</a> and stores it in a JSON file.

### Step 1: Create and deploy a Zenrows API Spec

1. Log in to MuleSoft via the Anypoint platform.

2. Click the icon at the top left and go to **Design Center**.

   <img alt="anypoint-menu.png" />

3. Click `Create +` at the top-left and select **New API Specification**.

   <img alt="anypoint-design-center-create" />

4. Give your project a name (e.g., Zenrows Universal Scraper). Then click `Create API`.

   <img alt="anypoint-project-setup" />

5. The Design Center creates a new `zenrows-universal-scraper.raml` file. Remove the existing content in the **RAML** code box and paste the following Zenrows RAML configuration inside the code box. This configuration defines the required [specs and parameters](/fetch/api-reference#parameter-overview) to use the Zenrows Fetch:

   ```yml RAML theme={"dark"}
   #%RAML 1.0
   title: Zenrows Fetch
   version: v1
   baseUri: https://api.zenrows.com/v1

   /:
     get:
       description: A versatile tool designed to simplify and enhance the process of extracting data from websites.
       queryParameters:
         apikey:
           description: Your unique API key for authentication.
           type: string
           required: true
           example: ""
         url:
           description: The URL of the page you want to scrape.
           type: string
           required: true
           example: ""
         js_render:
           description: Enable JavaScript rendering with a headless browser.
           type: boolean
           required: false
           default: false
           example: true
         premium_proxy:
           description: Use residential IPs to bypass anti-bot protection.
           type: boolean
           required: false
           default: false
           example: true
         proxy_country:
           description: Set the country of the IP used for the request (requires Premium Proxies).
           type: string
           required: false
           example: ""
         autoparse:
           description: Automatically parse the content of supported websites and return the data as a JSON object.
           type: boolean
           required: false
           default: false
           example: false
         response_type:
           description: Convert HTML to other formats (Markdown, Plaintext, PDF).
           type: string
           required: false
           example: markdown
         wait:
           description: Wait a fixed number of milliseconds after page load.
           type: integer
           required: false
           example: 0
         wait_for:
           description: Wait for a specific CSS Selector to appear in the DOM before returning content.
           type: string
           required: false
           example: ""
         css_extractor:
           description: Extract specific elements using CSS selectors. 
           type: string
           required: false
           example: ""
         json_response:
           description: Capture network requests in JSON format, including XHR or Fetch data.
           type: boolean
           required: false
           default: false
           example: true
         js_instructions:
           description: Execute custom JavaScript on the page to interact with elements, scroll, click buttons, or manipulate content.
           type: string
           required: false
           example: ""
         original_status:
           description: Return the original HTTP status code from the target page.
           type: boolean
           required: false
           default: false
           example: true
         outputs:
           description: Specify which data types to extract from the scraped HTML.
           type: string
           required: false
           example: "tables,hashtags,emails"
         block_resources:
           description: Block specific resources (images, fonts, etc.)
           type: string
           required: false
           example: "image,media,font"
         screenshot:
           description: Capture an above-the-fold screenshot of the page.
           type: boolean
           required: false
           default: false
           example: false
         screenshot_selector:
           description: Capture a screenshot of a specific element using CSS Selector. 
           type: string
           required: false
           example: ""
         screenshot_fullpage:
           description: Capture a full-page screenshot. 
           type: boolean
           required: false
           default: false
           example: true
         screenshot_format:
           description: Choose the screenshot format. 
           type: string
           required: false
           example: "png"
         screenshot_quality:
           description: For JPEG format, set quality from 1 to 100. 
           type: integer
           required: false
           example: 70
       responses:
         200:
           body:
             application/json:
               example: |
                 {
                   "success": true,
                   "data": ""
                 }
   ```

6. Click `Get`. Then, click `Publish` at the top right.

   <img alt="anypoint-api-spec-creation" />

7. Keep the **Asset Version** and **API Version** as `1.0.0` and `v1`, respectively. Under **LifeCycle State**, select **Stable**.

8. Click `Publish to Exchange`.

9. Close the confirmation modal.

   <img alt="anypoint-exchange-deployment-confirmation" />

10. Click the menu icon at the top left and go to **Exchange** to view your published API spec in the Anypoint Exchange marketplace.

    <img alt="anypoint-exchange-zenrows-scraper-displayed" />

### Step 2: Import the Zenrows API Spec into Anypoint Studio

1. Launch the Anypoint Studio on your machine.

2. Click `File` at the top left and go to **New**. Then, select **Mule Project**.

   <img alt="anypoint-studio-file-menu-options" />

3. Give your project a name (e.g., Scraper) and click `Finish`.

   <img alt="anypoint-studio-mule-project" />

4. Double-click the `<project.xml>` (`scraperflow.xml`) file on the left sidebar to load the flow canvas.

5. From the Mule Palette on the right side of the canvas, click `Search in Exchange`.

   <img alt="anypoint-exchange-option" />

6. Click `Add Account` and authenticate Anypoint Studio with your Anypoint account if you haven't done so already.

7. Search for Zenrows in the search bar and select the **Zenrows Fetch** spec from the result table.

8. Click `Add` to load the API spec into the Anypoint Studio Mule Palette.

9. Click `Finish`.

   <img alt="anypoint-studio-api-spec-connection" />

10. To test the import, search for Zenrows via the Mule Palette search bar. The Zenrows Universal Scraper spec now appears in the Mule Palette.

    <img alt="anypoint-studio-mule-pallet-api-spec" />

### Step 3: Create the scraping workflow in Anypoint Studio

1. From the Mule Palette, search for **Scheduler** and drag it into the canvas. Rename your scheduler as you desire.

2. From the **Scheduling Strategy** dropdown, choose between **Frequency** or **Cron**. We've chosen **Frequency** in this case and scheduled the flow to run the request within 10 seconds using a 10-second delay.

   <img alt="schedule-flow-settings" />

3. Search for **Logger** and drag it into the Process tab inside the Scraperflow. We'll use this to log the beginning of automation.

4. In the **Message** box, type a trigger alert message (e.g., Schedule triggered!).

   <img alt="anypoint-studio-logger-flow" />

5. Search for **Zenrows** in the Mule Palette. Then, drag the Zenrows Fetch spec into the workflow.

6. Click the **Connector Configuration** dropdown and select **Create a new configuration**.

7. Rename the Zenrows flow as you desire (e.g., Product Scraper Flow).

8. Click the `+` icon next to **Connector Configuration** and click okay to set it as Zenrows.

   <img alt="anypoint-studio-connector-settings" />

9. Set the necessary parameters. We've used the following in this guide:
   * Zenrows API key.
   * Target URL
   * Js\_render = true
   * Premium\_proxy = true
   * Proxy\_country = us
   * css\_extractor:
     ```json JSON theme={"dark"}
     {
         "name": "span#productTitle",
         "price": "span.aok-offscreen",
         "ratings": "span.reviewCountTextLinkedHistogram span.a-size-base.a-color-base",
         "reviewSummary": "#product-summary p span",
         "reviewCount": "#acrCustomerReviewText"
     } 
     ```

10. Press `Ctrl+S` on your keyboard to save the current flow.

    <img alt="anypoint-studio-zenrows-scraper-api-spec-params" />

11. Again, search and drag a **Logger** flow into the workflow to output the scraping result.

12. Rename the new logger (e.g., Data Logger)

13. In the Logger's **Message**, click `fx` and type `#[payload`.

    <img alt="anypoint-data-logger" />

### Step 4: Save the data

1. Search for **File Write** in the Mule Palette and drag it into your workflow.
2. Rename the Writer (e.g., JSON Writer).
3. Paste the destination folder path in the **Path** field (e.g., `D:/Anypoint-result/result.json`). Keep the **Content** field as **payload** and the **Write Mode** as **OVERWRITE**.
4. Press Ctrl+S to save the changes.

   <img alt="anypoint-studio-final-flow" />

### Step 5: Run the flow

To run the flow, right-click `scraperflow.xml`, click `Run As` > `Mule Application`.

<img alt="anypoint-mule-application-execution-options" />

Check your JSON storage directory and open the created JSON file to see the scraped data.

Here's a sample JSON result:

```json result.json theme={"dark"}
{
    "name": "Gamrombo LED Wireless Controller for PS5, Compatible with PS5 Pro/Slim/PC, Dual Vibration, Marco/Turbo Function, 3.5mm Audio Jack, 6-Axis Motion Contro Gamepad with Speaker",
    "price": [
        "$49.99 with 17 percent savings",
        "List Price: $59.99"
    ],
    "ratings": [
        "4.5",
        "4.5"
    ],
    "reviewCount": [
        "423 ratings",
        "423 ratings"
    ],
    "reviewSummary": [
        "Customers find the controller's ergonomics responsive and comfortable during long gaming sessions, with buttons that provide crisp feedback and moderate rebound strength. Moreover, the device functions well on both PC and PS5 systems, and customers appreciate its build quality with wear-resistant materials. Additionally, they like its style, particularly the colors and lights, and consider it surprisingly good for its price. The response time receives positive feedback, with customers noting very low wireless delay and fast charging capabilities.",
        "",
        "AI Generated from the text of customer reviews",
        ""
    ]
}
```

Congratulations! 🎉You just integrated Zenrows into your MuleSoft Anypoint automation workflow.

## Troubleshooting

### Error 429 (Too Many Requests)

* **Solution**: Increase the Scheduler's frequency and delay to space out the requests and prevent multiple requests from running within a short time frame.

### Build failed during Anypoint Studio compilation

* **Solution 1**: Check for errors in each flow and fix them individually.
* **Solution 2**: Ensure you enter the correct Zenrows parameters. The `js_render` and `premium_proxy` parameters should be set to `true` to increase the success rate.
* **Solution 3**: Verify that your API key is entered correctly.

### Incomplete or empty data

* **Solution 1**: Ensure that you enter the correct CSS selectors.
* **Solution 2**: Validate that the CSS extractor array is formatted correctly.
* **Solution 3**: If using autoparse, ensure that Zenrows supports the target page. Check the [Zenrows Data Collector Marketplace](https://www.zenrows.com/scraper) to view the supported websites.

### File write access denied:

* **Solution 1**: If storing data locally, give the current Anypoint Studio workplace access to the storage location.
* **Solution 2**: Ensure that you append the file name to the file path specified in the Anypoint Studio Writer flow. For example, `D:/Anypoint-result` is wrong. The correct file path format should be `D:/Anypoint-result/result.json`.

## Frequently Asked Questions (FAQ)

<Accordion title="Does MuleSoft and Zenrows integration incur additional costs?">
  Zenrows doesn't charge you extra for making scraping requests via Anypoint. However, while Anypoint comes with a 30-day free trial, it requires an upfront financial commitment, which limits your ability to use the scraping workflow.
</Accordion>

<Accordion title="Does MuleSoft support writing scraped data to a CSV file or a remote database?">
  Yes, you can save data locally to a CSV file using Anypoint's File Write module. MuleSoft's Anypoint also supports remote database connection using the JDBC (Java Database Connectivity) driver.
</Accordion>

<Accordion title="Is Zenrows integration built into MuleSoft's Anypoint platform?">
  No, but integrating Zenrows with MuleSoft is straightforward. You only need to paste the RAML file containing the required Zenrows Fetch specifications into the Anypoint Platform and publish it to the Exchange. You can then interact with Zenrows visually in your workflow via the published API specification.
</Accordion>

<Accordion title="Does MuleSoft's Anypoint support AI integration?">
  Although Anypoint doesn't have built-in support for LLM tools, you can load your desired model using Anypoint's HTTP request flow. This involves authorizing the LLM via the Bearer Authorization header and returning its response using a Logger flow or directly writing it into a file.
</Accordion>


# How to Integrate n8n with Zenrows
Source: https://docs.zenrows.com/integrations/n8n

Integrate Zenrows with n8n to add automated web scraping nodes to your workflows. Collect structured data from websites with anti-bot bypass.

Zenrows provides powerful web scraping capabilities that integrate seamlessly with n8n. This integration enables automated collection of structured data from any website directly into your n8n workflow.

## What Is n8n?

<a href="https://n8n.io/">n8n</a> is a workflow automation platform that connects different services and builds custom workflows with a visual interface. Unlike many other automation tools, n8n can be self-hosted, giving you complete control over your data.

With its node-based workflow editor, n8n makes it easy to connect apps, services, and APIs without writing code. You can process, transform, and route data between different services, and trigger actions based on schedules or events.

## Use Cases

Integrating Zenrows with n8n opens up numerous automation possibilities:

* **E-commerce Monitoring**: Scrape product prices, availability, and ratings from competitor websites and store them in Google Sheets or Airtable for analysis.
* **Lead Generation**: Extract contact information from business directories and automatically add it to your CRM system, such as HubSpot or Salesforce.
* **Job Market Analysis**: Monitor job listings across multiple platforms and receive daily notifications of new opportunities via Slack or email.
* **Content Aggregation**: Scrape blog posts, news articles, or research papers and compile summaries in Notion or other knowledge bases.
* **Real Estate Data Collection**: Gather property listings, prices, and features from real estate websites for market analysis.

## Watch the Video Tutorial

Learn how to set up the n8n ↔ Zenrows integration step-by-step by watching this video tutorial:

<iframe title="Zenrows ↔ n8n" />

## How to Set Up Zenrows with n8n

Follow these steps to integrate Zenrows with n8n:

### Step 1: Set up Zenrows

1. [Sign up](https://app.zenrows.com/register?prod=universal_scraper) for a Zenrows account if you don't already have one.
2. Navigate to Zenrows' Fetch Playground.
3. Configure the settings:

* Enter the URL to Scrape.
* Activate JS Rendering and/or Premium Proxies (as per your requirements).
* Customize other settings as needed for your specific scraping task.
  <Note>Check out the [Fetch documentation](/fetch/api-reference) for all available features</Note>

4. Click on the cURL tab on the right and copy the generated code.

<img alt="Zenrows Request Playground" />

### Step 2: Set up your n8n workflow

1. <a href="https://app.n8n.cloud/register">Sign up or log in</a> to open your n8n instance and create a new workflow.
2. Add a **Manual Trigger** node as the first node. The Manual Trigger node allows you to manually start the workflow for testing or debugging purposes.
   <img alt="Manual Trigger Node Setup" />
3. Next, add an **HTTP Request** node:
   * Click on the **+** button next to the Manual Trigger node.
   * Select the **Core** tab on the right.
   * Then, click the **HTTP Request** tab to open the configuration window.
   <img alt="HTTP Request Node Setup" />
   * Click the **Import cURL** button and paste the Zenrows cURL code you copied earlier. The HTTP Request node will automatically configure all the necessary parameters, including headers, authentication, and query parameters.
   * Finally, click **Test step** to confirm your setup is configured correctly. You should get the full HTML of the target page as the output.
   <img alt="Zenrows n8n Setup Test" />

## Example Workflow: Scraping Product Titles from a Website

Let's build a complete workflow that scrapes product titles from an e-commerce website and stores them in a Google Sheet.

### Step 1: Create a new workflow

1. In your n8n dashboard, click **Create workflow** and give it a name (e.g., "Product Title Scraper").
2. Add a **Manual Trigger** node as your starting point.

### Step 2: Configure Zenrows

1. Open Zenrows' [Fetch Request Playground](https://app.zenrows.com/fetch).
2. Enter `https://www.scrapingcourse.com/ecommerce/` as the URL to Scrape.
3. Activate JS Rendering.
4. Select **Specific Data** as the Output type and configure the CSS selector under the Parsers tab: `{ "product-names": ".product-name" }`.
5. Copy the generated cURL code.
6. In n8n, add an **HTTP Request** node after your trigger:
   * Click on **Import cURL** and paste the generated cURL code.
   * This configures a GET request to Zenrows' Fetch, sending the target URL and parser configuration.
   * The request will extract all elements matching the CSS selector `.product-name` (which contains product titles).

### Step 3: Transform the response with AI Transform

After getting the Zenrows response, we need to extract the content values:

1. Click the **+** button next to the **HTTP Request** node and select the **Data transformation** tab on the right.
2. Then, select the <a href="https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.aitransform/">AI Transform</a> node. The AI Transform node allows you to process and manipulate data using AI-generated JavaScript code.
   <img alt="AI Transform Node Setup" />
3. In the instructions field, enter: "Extract the value of 'product-names' from 'data' as an array of objects" and click the **Generate code** button.

   The AI will generate JavaScript code that gets all items from the input, maps through each item, extracts the value from the JSON data, and returns an array of product names.

   <img alt="AI Transform Node Code Generation" />

<Warning>If you encounter an error, refer to <a href="https://docs.n8n.io/code/ai-code/#writing-good-prompts">Writing Good Prompts</a> for the correct code. Also, note that in n8n, the data passed between nodes should be an <a href="https://docs.n8n.io/data/data-structure/">array of objects</a>. </Warning>

### Step 4: Split the results

Now we need to split the array of product names into individual items:

1. Add a <a href="https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.splitout/">Split Out</a> node after the **AI Transform**, as it takes an array of data and creates a separate output item for each entry.
2. Configure it to split out the `product-name` field.
   <img alt="Split Out Node Setup" />
   This node will take the array of items and create a separate output item for each content value.

### Step 5: Send data to Google Sheets

Finally, we'll send our scraped product titles to a Google Sheet:

1. Add a **Google Sheets** node after the **Split Out** node.
2. Under **Sheet Within Document Actions**, select the **Append or Update Row in sheet action**.
   <img alt="Google Sheets Node Setup" />
3. Next, select your Google Sheets account or create a new connection.
4. Specify the Google Sheets document URL and select the sheet where you want to store the data.
5. Select the Map Each Column Manually mode and choose the column to match.
6. Drag the `['product-name']` variable under Split Out on the left to the Values to Send field.
   <img alt="Google Sheets Node Configuration" />
   This configuration will append each product title as a new row in your Google Sheet.

### Step 6: Run the workflow

Click the **Test workflow** to run your automation. If everything is configured correctly, you should see product titles from the website appearing in your Google Sheet.

## Tips and Best Practices

When working with Zenrows and n8n together, keep these best practices in mind:

* **Enable JS Rendering**: For modern websites with dynamic content, always enable the JS Rendering option in Zenrows.
* **Set Appropriate Wait Selectors**: Use the `wait_for` parameter when scraping sites that load content asynchronously.
  <Tip>Make sure to test the selectors by testing them in your browser's developer tools. If the page CSS selector is dynamic, you can also use the `wait` parameter to make the request wait for a specific number of seconds. For more information, check the [Wait Milliseconds](/fetch/features/wait) and [Wait for Selector](/fetch/features/wait-for) documentation!</Tip>
* **Implement Retry Logic**: Add Error Handling with retry logic for failed requests using n8n's Error Trigger nodes.
* **Paginate Properly**: For multi-page scraping, create loops with the <a href="https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/">Loop Over Items</a> node and increment page numbers.
* **Monitor Your Workflows**: Set up notification triggers to receive alerts for failures or successful runs.

## Troubleshooting

Common issues and solutions:

* **API Key Not Working**
  * Verify your Zenrows API key is correctly entered in the HTTP Request node.
  * Check that your subscription is active in your Zenrows dashboard.
* **Empty Results**
  * Confirm your CSS selectors are correct by testing them in your browser's developer tools.
  * Try enabling JavaScript rendering if the content is loaded dynamically.
  * Increase wait time if the content takes longer to load.
* **Workflow Execution Errors**
  * Check the JSON response format in the HTTP Request node output.
  * Ensure all nodes are correctly configured with the correct field mappings.
  * View execution logs for detailed error messages.

## Conclusion

Congratulations on successfully integrating Zenrows with n8n! This integration allows you to automate your web scraping workflows, collect structured data, and store it effortlessly in tools like Google Sheets.


# How to Integrate Node-RED with Zenrows
Source: https://docs.zenrows.com/integrations/node-red

Set up Zenrows web scraping in Node-RED to build automated data extraction flows. Connect HTTP nodes with Zenrows API for reliable scraping.

Automate data extraction in your Node-RED workflows using Zenrows' web scraping capabilities. This guide shows you how to integrate Zenrows with Node-RED and build a continuous data retrieval process.

## What Is Node-RED?

Node-RED is a low-code programming tool that connects different services and APIs using a visual, drag-and-drop interface. It includes built-in workflow nodes for automating repetitive tasks, including action triggers, scheduling, data storage, and more.

## Use Cases

• **Price comparison**: Use Node-RED's schedule node to run a periodic price scraping operation on competitors' sites using Zenrows. Connect your flow to an LLM node to analyze price disparities.

• **Property analysis**: Use Zenrows to scrape property listing pages across popular real-estate sites and use Node-RED workflow to automate continuous data extraction, cleaning, storage, and analysis.

• **Web page monitoring**: Periodically monitor a web page by combining Node-RED's scheduler with Zenrows' scraping capabilities.

• **Sentiment analysis**: Use a Zenrows scraping node to collect data from various platforms, including social media, review sites, Google review pages, and more. Pass the scraped data to an LLM node for sentiment extraction.

• **Lead generation**: Use Node-RED's automation features to automate quality lead scraping using Zenrows' unique scraping capabilities.

## Integration Steps

In this guide, we'll create a scheduled Node-RED workflow that extracts product data from an Amazon product page using Zenrows' [`css_extractor`](/fetch/features/css-extractor).

### Step 1: Install and Launch Node-RED

1. Install the **node-red** package globally:
   ```bash theme={"dark"}
   npm install -g --unsafe-perm node-red
   ```

2. Launch the Node-RED server:
   ```bash theme={"dark"}
   node-red
   ```
   This command starts Node-RED on `http://localhost:1880`.

3. Visit the localhost URL via your browser to load Node-RED's platform as shown
   <img alt="node-red-interface" />

### Step 2: Get the request URL from Zenrows

1. Open the **Zenrows Universal Scraper [API Playground](https://app.zenrows.com/fetch)**.
2. Paste the target URL in the **link box**, activate **JS Rendering** and **Premium Proxies**.
3. Under **Output**, click **Specific data** and select **Parsers**.
4. Enter the CSS selector array in the **Parsers** field. We'll use the following CSS extractors in this guide:
   ```json JSON theme={"dark"}
   {
       "name":"span#productTitle",
       "price":"span.a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay",
       "ratings":"#acrPopover",
       "description":"ul.a-unordered-list.a-vertical.a-spacing-mini",
       "review summary":"#product-summary p span",
       "review count":"#acrCustomerReviewLink"
   } 
   ```
5. Select **cURL** as your language. Then, copy the URL generated after the cURL command.

### Step 3: Create your scraping workflow

1. Drag the **Inject Node** into the canvas.
   <img alt="node-red-timestamp-module-view" />

2. Double-click the `timestamp` node and enter a name in the **Name** field to rename it (e.g., "Scheduler").

3. Click the **Repeat** dropdown and select a schedule. We'll use a 10-second **Interval** in this guide.

4. Click `Done`.
   <img alt="node-red-timestamp-node-edit" />

5. From the search bar at the top left, search for **http request**. Then, drag the **http request** node into the canvas.

6. Link both nodes by dragging a line to connect them at either end.
   <img alt="node-red-http-request-node" />

7. Double-click the **http request** node.

8. Paste the Zenrows connection URL you copied previously in the **URL** field.

9. Change the **Return** field to a **Parsed JSON object** to parse the returned string as a JSON object.

10. Enter a name for your HTTP node in the **Name** field (e.g., Scraper).

11. Click `Done`.
    <img alt="node-red-http-request-node-setup" />

### Step 4: Add storage and output nodes

1. Search for **write file** in the search bar and drag the **write file** node into the canvas.

2. Link the **write file** node with the **Scraper** node.
   <img alt="node-red-write-file-node" />

3. Double-click the **write file** node to configure it.

4. In the **Filename** field, enter the JSON file path.

5. Under **Action**, select a suitable action. We'll choose the **overwrite file** option in this guide.

6. Enter a name in the **Name** field to rename the node (e.g., JSON Writer).

7. Click `Done`.
   <img alt="node-red-file-writer-setup" />

8. Drag the **debug** node into the canvas and connect it to the file writer.

9. Double-click the **debug** node and rename it as desired (e.g., Output Tracker).

10. Leave the **Output** as **msg** and **payload**.

11. Click `Done`.
    <img alt="node-red-output-tracker-setup" />

### Step 5: Deploy and run the flow

Click `Deploy` at the top right to deploy the flow. Once deployed, the schedule is now triggered. Click the bug icon at the top right. This displays the JSON outputs in the right sidebar for each run.

<img alt="node-red-complete-flow-deploy" />

The workflow also creates a JSON file in the specified directory and updates it at the chosen interval. Here's a sample JSON response:

```json JSON theme={"dark"}
{
    "name": "Gamrombo LED Wireless Controller for PS5, Compatible with PS5 Pro/Slim/PC, Dual Vibration, Marco/Turbo Function, 3.5mm Audio Jack, 6-Axis Motion Contro Gamepad with Speaker",
    "price": "$59.99",
    "ratings": [
        "4.4",
        "4.4"
    ],
    "reviewCount": [
        "455 ratings",
        "455 ratings"
    ],
    "reviewSummary": [
        "Customers find the PS5 controller smooth and responsive, with buttons that provide crisp feedback and moderate rebound strength. Moreover, the controller is comfortable during long gaming sessions and features wear-resistant materials. Additionally, they appreciate its compatibility with both PS5 and PC, its vibrant lighting design, and consider it good value for money.",
        "",
        "AI Generated from the text of customer reviews",
        ""
    ]
}
```

Congratulations! 🎉 You've now integrated Zenrows with Node-RED.

## Troubleshooting

### Invalid URL/flow fails to run

* **Solution 1**: Ensure you explicitly copied the URL generated by cURL from the Playground.
* **Solution 2**: Check the URL format for extra character strings, such as `cURL` appended to the URL (e.g., the URL must be in the form of `http://…` and not `cURL "http"`).

### File write access unavailable or denied

* **Solution 1**: Ensure you set the proper write permission inside the storage location.
* **Solution 2**: While specifying the file path in the **write file** node, you should include the file's absolute path, including the expected file name. For instance, `D:/Node-RED-result` is wrong. The correct file path format should be `D:/Node-RED-result/result.json`.

### Node-RED server jammed or localhost address unavailable

* **Solution**: Stop the running server and restart it. Then, re-open the server address on your browser.

### `node-red` is not recognized as an internal or external command

* **Solution**: Ensure you've installed the Node-RED module globally, not in a localized working directory with `package.json`.

## Frequently Asked Questions (FAQ)

<Accordion title="Can I store the scraped data from Node-RED in a CSV file or a database?">
  Yes, Node-RED has a built-in file writer node that allows you to access local and remote directories to write files in various formats, including JSON, CSV, Markdown, images, and more. Although there are no built-in database nodes, you can install them individually by clicking the menu at the top right > **Manage palette** > **Install** section. Then, search and install the relevant database module. Once installed, you can then drag the database module into the canvas and configure it for data storage.

  <Tip>Check the <a href="https://flows.nodered.org/">Node-RED library page</a> for more information.</Tip>
</Accordion>

<Accordion title="Does Zenrows offer built-in Node-RED integration?">
  No, Zenrows doesn't offer a built-in node on Node-RED. However, you can easily connect to Zenrows via Node-RED's built-in **http request** node and your target website.
</Accordion>

<Accordion title="Does Node-RED support AI integration?">
  While Node-RED doesn't have a built-in Node for interacting with LLMs like `gpt-4o-mini`, you can interact with them via a custom function node or use community-made nodes like <a href="https://flows.nodered.org/node/node-red-openai-chatgpt">`node-red-openai-chatgpt`</a>, <a href="https://flows.nodered.org/node/node-red-contrib-anthropic">`node-red-contrib-anthropic`</a>, or other available LLM nodes.
</Accordion>


# How to Integrate OpenAI with Zenrows
Source: https://docs.zenrows.com/integrations/openai

Give OpenAI's GPT models real-time access to any website using Zenrows' Fetch and Zenrows' MCP.

Combine OpenAI's GPT models with Zenrows' [Fetch](https://docs.zenrows.com/fetch/api-reference) to build AI workflows that can read any website in real time. This guide shows how to use Zenrows with OpenAI's Responses API, function calling, structured outputs, and the hosted MCP tool, so your AI applications can ground their answers in live web content, even from anti-bot-protected and JavaScript-heavy sites.

This guide covers the standard OpenAI API. If you are building autonomous, multi-step agents with the OpenAI Agents SDK, see the [OpenAI Agents SDK integration guide](https://docs.zenrows.com/integrations/openai-agents-sdk).

<Note>
  **Just want to plug GPT into the live web? Use Zenrows MCP with OpenAI.**

  The fastest path to a web-aware GPT is the [Zenrows hosted MCP server](https://docs.zenrows.com/mcp/overview).

  The Responses API natively supports remote MCP servers, so you can register `https://mcp.zenrows.com/mcp` as a tool and let GPT invoke Zenrows' scraping capabilities directly. No function-calling boilerplate, no orchestration code. Jump to [Using Zenrows MCP with the Responses API](#using-zenrows-mcp-with-the-responses-api) for a complete code example.
</Note>

## What is OpenAI?

OpenAI is the company behind GPT, the family of large language models powering ChatGPT and a developer platform used by hundreds of thousands of teams. The OpenAI API provides programmatic access to these models through the <a href="https://developers.openai.com/api/docs/guides/text">Responses API</a>, the recommended interface for all new projects, with built-in support for function calling, structured outputs, reasoning models, and remote MCP servers.

GPT models excel at reasoning, summarization, and structured generation, but **they get blocked while accessing the web** on their own, especially at scale. **Pairing OpenAI with Zenrows closes that gap.**

### Key benefits of integrating OpenAI with Zenrows

* **Real-time web grounding for any GPT model**: GPT has a fixed knowledge cutoff. Zenrows feeds live, up-to-date web content into any response, eliminating stale or hallucinated answers about recent events, prices, or product details.
* **Anti-bot bypass out of the box**: [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode) (`mode=auto`) automatically handles JavaScript rendering, premium proxies, fingerprinting, and bot detection, so your AI can read pages that would block a regular scraper.
* **Token-efficient Markdown output**: Zenrows returns clean Markdown in addition to raw HTML, which reduces token usage and improves model accuracy on the same context window.
* **Native function calling**: Expose Zenrows as a function tool and let GPT decide when to scrape based on the user's question, with no orchestration code required.
* **Works with every OpenAI API surface**: Use Zenrows with the Responses API, structured outputs (`responses.parse`), reasoning models, and the hosted MCP tool. The integration is a plain HTTP call, so it fits into any pattern.
* **MCP-ready out of the box**: Plug the [Zenrows hosted MCP server](https://docs.zenrows.com/mcp/overview) directly into the Responses API as a tool for zero-code access to all of Zenrows' scraping capabilities.

## Use cases

The OpenAI and Zenrows combination unlocks a wide range of AI workflows:

* **Web-aware chatbots**: Build assistants that can answer questions about any URL the user provides, including protected sites like e-commerce stores, real estate portals, and news outlets.
* **Real-time competitive intelligence**: Have GPT analyze competitor pricing pages, product launches, and changelogs as they happen.
* **Lead enrichment**: Scrape company websites and let GPT extract industry, headcount signals, tech stack hints, and product summaries into your CRM.
* **Automated research and reporting**: Pull and summarize industry articles, financial filings, or technical documentation into ready-to-share briefs.
* **Structured data extraction**: Pull strongly-typed JSON (products, jobs, listings, reviews) from any page using GPT's structured outputs grounded in Zenrows-scraped content.

## Getting started: Basic Usage

Let's start with a simple example: scrape a JavaScript-heavy, anti-bot-protected demo page using Zenrows and summarize it with `gpt-5-mini` through the Responses API.

<Steps>
  <Step title="Install the OpenAI Python library and requests">
    ```bash theme={"dark"}
    pip install openai requests
    ```
  </Step>

  <Step title="Create a .env file and set your API keys as environment variables">
    ```bash theme={"dark"}
    ZENROWS_API_KEY=your_zenrows_key
    OPENAI_API_KEY=your_openai_key
    ```
  </Step>

  <Step title="Run the following script">
    ```python Python theme={"dark"}
    import os
    import requests
    from openai import OpenAI

    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

    def scrape(url: str) -> str:
        """Scrape any URL with Zenrows and return clean Markdown."""
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "apikey": ZENROWS_API_KEY,
                "url": url,
                "mode": "auto",
                "response_type": "markdown",
            },
        )
        response.raise_for_status()
        return response.text

    # Scrape a JS-rendered, anti-bot-protected page
    markdown = scrape("https://www.scrapingcourse.com/antibot-challenge")

    # Summarize with GPT through the Responses API
    response = client.responses.create(
        model="gpt-5-mini",
        instructions="You are a concise technical writer.",
        input=f"Summarize this page in 2 sentences:\n\n{markdown}",
    )

    print(response.output_text)

    ```
  </Step>
</Steps>

Two things are happening here:

1. Zenrows handles the scrape. `mode=auto` tells the API to start with the cheapest viable configuration and automatically escalate to JavaScript rendering or premium proxies if the target site requires it. `response_type=markdown` returns clean Markdown instead of raw HTML, which is ideal for LLM context.
2. OpenAI handles the reasoning. The Markdown is fed into `client.responses.create()`, with system-level guidance passed through the `instructions` parameter and the user prompt passed through `input`.

You've scraped a protected page and grounded a GPT response in live web content. Let's look at more advanced patterns.

## Advanced Usage: Building a web-aware AI assistant with function calling

Function calling is OpenAI's mechanism for letting a model decide when to call an external tool. Instead of always scraping before each prompt, you expose Zenrows as a function and let GPT call it only when needed.

This pattern is the foundation of any production-grade AI application that needs web access.

<Steps>
  <Step title="Set up the environment">
    Run the following command in your terminal to install the OpenAI Python library and requests:

    ```bash theme={"dark"}
    pip install openai requests
    ```

    Create a `.env` file:

    ```bash theme={"dark"}
    ZENROWS_API_KEY=your_zenrows_key
    OPENAI_API_KEY=your_openai_key
    ```
  </Step>

  <Step title="Define the scraping tool">
    Wrap the Zenrows REST call in a Python function and describe it to the Responses API using a JSON schema:

    ```python Python theme={"dark"}
    import json
    import os
    import requests
    from openai import OpenAI

    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

    def scrape_website(url: str) -> str:
        """Scrape any website with Zenrows and return Markdown content."""
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "apikey": ZENROWS_API_KEY,
                "url": url,
                "mode": "auto",
                "response_type": "markdown",
            },
        )
        response.raise_for_status()
        return response.text

    # Tell GPT what scrape_website does and how to call it
    tools = [
        {
            "type": "function",
            "name": "scrape_website",
            "description": (
                "Scrape any public website and return its content as Markdown. "
                "Use this whenever you need up-to-date information from a specific URL."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The fully qualified URL of the website to scrape",
                    }
                },
                "required": ["url"],
                "additionalProperties": False,
            },
            "strict": True,
        }
    ]
    ```

    The Responses API uses a flat tool definition (`name`, `description`, and `parameters` at the top level of the tool object).
  </Step>

  <Step title="Let GPT decide when to scrape">
    Send the user's question along with the tool definition. GPT returns either a direct answer or a `function_call` item in the response output:

    ```python Python theme={"dark"}
    input_list = [
        {
            "role": "user",
            "content": "What is Hacker News? Visit https://news.ycombinator.com/ and tell me what the product does.",
        }
    ]

    response = client.responses.create(
        model="gpt-5-mini",
        tools=tools,
        input=input_list,
    )

    # Capture the entire response output (function calls, reasoning items, etc.)
    input_list += response.output

    # Run the tool for any function_call item the model produced
    for item in response.output:
        if item.type == "function_call" and item.name == "scrape_website":
            args = json.loads(item.arguments)
            result = scrape_website(args["url"])
            input_list.append({
                "type": "function_call_output",
                "call_id": item.call_id,
                "output": result,
            })

    # Send the tool result back to the model for the final answer
    final = client.responses.create(
        model="gpt-5-mini",
        tools=tools,
        input=input_list,
    )

    print(final.output_text)
    ```

    <Note>
      When using reasoning models like ChatGPT-5, the response output may include reasoning items alongside function calls. The line `input_list += response.output` captures all output items, including reasoning, which must be passed back with tool call outputs for the model to continue correctly.
    </Note>
  </Step>

  <Step title="Complete Code Example and Output">
    ```python Python theme={"dark"}
    import json
    import os
    import requests
    from openai import OpenAI

    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

    def scrape_website(url: str) -> str:
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "apikey": ZENROWS_API_KEY,
                "url": url,
                "mode": "auto",
                "response_type": "markdown",
            },
        )
        response.raise_for_status()
        return response.text

    tools = [
        {
            "type": "function",
            "name": "scrape_website",
            "description": (
                "Scrape any public website and return its content as Markdown. "
                "Use this whenever you need up-to-date information from a specific URL."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The fully qualified URL of the website to scrape",
                    }
                },
                "required": ["url"],
                "additionalProperties": False,
            },
            "strict": True,
        }
    ]

    def ask(question: str) -> str:
        input_list = [{"role": "user", "content": question}]

        response = client.responses.create(
            model="gpt-5-mini",
            tools=tools,
            input=input_list,
        )

        input_list += response.output
        has_tool_call = False

        for item in response.output:
            if item.type == "function_call" and item.name == "scrape_website":
                has_tool_call = True
                args = json.loads(item.arguments)
                result = scrape_website(args["url"])
                input_list.append({
                    "type": "function_call_output",
                    "call_id": item.call_id,
                    "output": result,
                })

        if not has_tool_call:
            return response.output_text

        final = client.responses.create(
            model="gpt-5-mini",
            tools=tools,
            input=input_list,
        )

        return final.output_text

    print(ask("What is Hacker News? Visit https://news.ycombinator.com/ and tell me what the product does."))
    ```

    **Example output**

    ```wrap Output theme={"dark"}
    Hacker News is a minimalist social-news and discussion site run by Y Combinator that surfaces tech-, startup-, and hacker-oriented links and posts. It’s essentially a community-curated news aggregator + forum where users submit items, vote them up or down, and discuss them in threaded comments.

    What the product does (quick summary)
    - Aggregates links and text posts about programming, startups, science, and related topics.
    - Lets registered users submit stories, vote (upvote) items, and comment; votes and recency determine ranking on the front page.
    - Provides named formats like "Show HN" (projects/demos) and "Ask HN" (questions) that signal post type to the community.
    - Offers sections/filters: new, past/front, comments, ask, show, jobs, and submit.
    - ...
    ```
  </Step>
</Steps>

GPT decides on its own that the question requires fresh information from `news.ycombinator.com`, calls `scrape_website` with the right URL, then synthesizes a grounded answer from the scraped Markdown.

## Structured data extraction

OpenAI's <a href="https://developers.openai.com/api/docs/guides/structured-outputs">structured outputs</a> let you guarantee a model returns JSON that matches a specific schema. Combined with Zenrows, this is the cleanest way to extract typed data from any web page.

The Responses API exposes structured outputs through `client.responses.parse()`, which takes a Pydantic model directly and returns a parsed Python object.

The example below scrapes a company homepage and extracts a strongly-typed `CompanyInfo` object:

```python Python theme={"dark"}
import os
import requests
from openai import OpenAI
from pydantic import BaseModel

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

class CompanyInfo(BaseModel):
    name: str
    industry: str
    description: str

# Scrape the company website
response = requests.get(
    "https://api.zenrows.com/v1/",
    params={
        "apikey": ZENROWS_API_KEY,
        "url": "https://www.zillow.com/",
        "mode": "auto",
        "response_type": "markdown",
    },
)
response.raise_for_status()
markdown = response.text

# Extract structured data through the Responses API
parsed = client.responses.parse(
    model="gpt-5-mini",
    instructions="Extract company information from the provided website content.",
    input=markdown,
    text_format=CompanyInfo,
)

company = parsed.output_parsed
print(company.model_dump_json(indent=2))
```

### Example output

```json theme={"dark"}
{
  "name": "Zillow (Zillow Group)",
  "industry": "Real estate / Online real estate marketplace",
  "description": "Zillow is an online real estate marketplace and services company offering home listings for sale and rent, rental-management and agent-finding tools, home value estimates (Zestimates), mortgage products through Zillow Home Loans, market research, news and mobile apps."
}
```

The same pattern works for any structured extraction job: product listings, job postings, real estate, reviews, articles, and so on. Define the Pydantic schema, scrape the page with `mode=auto`, and let GPT return validated JSON.

<Note>
  **Skip the schema with Zenrows Autoparse**

  For common site types, you can skip LLM-based parsing entirely. Adding `autoparse=true` to your Zenrows request automatically identifies and extracts product details, article content, job listings, property data, and similar information into clean JSON, with no Pydantic schema or model call required.

  [Zenrows Autoparse](https://docs.zenrows.com/fetch/features/autoparse) is included at no additional cost. Return the JSON directly to your application for known-structure pages, or pass it to GPT for downstream enrichment, normalization, or categorization. Pre-structured input also uses far fewer tokens than raw Markdown, which keeps your context window lean and your model calls cheaper.

  Note that Autoparse cannot be combined with `response_type=markdown` in the same request.
</Note>

## Using Zenrows MCP with the Responses API

The Responses API natively supports remote [Model Context Protocol (MCP)](https://docs.zenrows.com/mcp/overview) servers as tools. Zenrows publishes a hosted MCP server that exposes web scraping capabilities, so you can give GPT real-time web access without writing any function-calling boilerplate.

```python theme={"dark"}
import os
from openai import OpenAI

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

response = client.responses.create(
    model="gpt-5",
    tools=[
        {
            "type": "mcp",
            "server_label": "zenrows",
            "server_description": "Web scraping MCP server for accessing live web content.",
            "server_url": "https://mcp.zenrows.com/mcp",
            "authorization": ZENROWS_API_KEY,
            "require_approval": "never",
        }
    ],
    input="Visit https://news.ycombinator.com/ and summarize the three most recent posts.",
)

print(response.output_text)
```

A few notes on this configuration:

* `server_url` points at Zenrows' hosted MCP endpoint.
* `authorization` carries your Zenrows API key. OpenAI does not retain the value between requests, so it must be present on every API call.
* `require_approval="never"` skips the per-tool-call approval step, which is appropriate for trusted servers like Zenrows. To require explicit approval, remove this field or set it to `"always"`.

For details on the Zenrows MCP server and its capabilities, see the [MCP overview](https://docs.zenrows.com/mcp/overview).

## API reference

The most useful Fetch parameters when working with OpenAI:

| Parameter | Type | Description |
| - | - | - |
| `apikey` | string | **Required.** Your Zenrows API key, passed as a query parameter. |
| `url` | string | **Required.** The URL to scrape. |
| `mode` | string | Set to `auto` to enable [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode), which automatically picks the cheapest working configuration for each request. Recommended default for AI workflows. |
| `response_type` | string | Convert HTML to other formats. Use `markdown` for LLM context (recommended), or `plaintext`, `pdf`. |
| `js_render` | boolean | Enable JavaScript rendering with a headless browser. Set automatically by `mode=auto` when needed. |
| `js_instructions` | string | Execute custom JavaScript on the page (click, scroll, fill forms) before returning content. |
| `premium_proxy` | boolean | Use residential IPs to bypass anti-bot protection. Set automatically by `mode=auto` when needed. |
| `proxy_country` | string | Two-letter country code for geo-restricted content (e.g. `us`, `gb`, `de`). Requires premium proxies, which `mode=auto` enables automatically when needed. |
| `session_id` | integer | Maintain the same IP across multiple requests for up to 10 minutes. |
| `wait_for` | string | Wait for a specific CSS selector to appear in the DOM before returning content. |
| `wait` | integer | Wait a fixed number of milliseconds after page load. |
| `css_extractor` | string | Extract specific elements using CSS selectors (JSON format). |
| `outputs` | string | Extract specific data types as structured JSON using a comma-separated list of filters: `emails`, `phone_numbers`, `headings`, `images`, `audios`, `videos`, `links`, `menus`, `hashtags`, `metadata`, `tables`, `favicon`. Use `outputs=*` to retrieve all available data types. See the [Output Filters documentation](/fetch/features/output-filters) for details. |
| `screenshot` / `screenshot_fullpage` | boolean | Capture a screenshot of the page. Useful for multimodal GPT inputs. |

For the full parameter list, see the [Fetch reference](https://docs.zenrows.com/fetch/api-reference#parameter-overview).

## Troubleshooting

### Token limit exceeded

* **Option 1**: Use `response_type=markdown` (already shown in every example above). Markdown reduces token usage significantly compared to raw HTML.
* **Option 2**: Use the `css_extractor` or `outputs` parameter to scrape only the parts of the page you need (a product card, a pricing table, an article body) instead of the entire DOM.
* **Option 3**: Chunk the scraped content into smaller pieces, summarize each chunk separately, and combine the results, or switch to a model with a larger context window.

### API key errors

* **Option 1**: Confirm both `ZENROWS_API_KEY` and `OPENAI_API_KEY` are set in your environment.
* **Option 2**: Verify your Zenrows API key in the [dashboard](https://app.zenrows.com/settings/api-keys) and your OpenAI key in the <a href="https://platform.openai.com/api-keys">OpenAI platform</a>.
* **Option 3**: Check that your Zenrows subscription is active and has remaining quota on the [Analytics page](https://app.zenrows.com/overview#performance).

### Empty or incomplete tool responses

* **Option 1**: Confirm `mode=auto` is set. Without it, JavaScript rendering and premium proxies are off by default and protected sites return blocked or empty pages.
* **Option 2**: For sites that load content asynchronously, add `wait_for=<css-selector>` (waits for a specific element) or `wait=5000` (waits a fixed duration in milliseconds).
* **Option 3**: If the model is calling the tool with a malformed URL, tighten the function description in the JSON schema or add validation in your `scrape_website` wrapper before sending the request to Zenrows.

### Reasoning items missing after tool calls

When using reasoning models like `gpt-5`, the response output may include reasoning items alongside function calls. Always append the entire `response.output` back to your input list (not just function calls) before sending the tool result. The pattern `input_list += response.output` shown in the function calling example handles this automatically.

## Helpful resources

* [Zenrows Fetch reference](https://docs.zenrows.com/fetch/api-reference)
* [Zenrows MCP server documentation](https://docs.zenrows.com/mcp/overview)
* [Adaptive Stealth Mode documentation](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode)
* [Zenrows + OpenAI Agents SDK integration](https://docs.zenrows.com/integrations/openai-agents-sdk)
* [OpenAI Responses API guide](https://developers.openai.com/api/docs/guides/text)
* [OpenAI function calling guide](https://developers.openai.com/api/docs/guides/function-calling)
* [OpenAI structured outputs guide](https://developers.openai.com/api/docs/guides/structured-outputs)

## Frequently asked questions

<Accordion title="Which OpenAI models work with this integration?">
  All current models. Function calling and structured outputs work best with `gpt-4o`, `gpt-4o-mini`, and other recent models. The hosted MCP tool requires a model that supports remote tools on the Responses API.
</Accordion>

<Accordion title="How do I handle pages that exceed the model's token limit?">
  Three options, in order of preference: use `response_type=markdown` to keep token usage low; use `css_extractor` or `outputs` to scrape only the relevant section of the page; chunk the scraped content and summarize each chunk before combining, or switch to a model with a larger context window.
</Accordion>

<Accordion title="Does Zenrows handle JavaScript-heavy and anti-bot-protected sites automatically?">
  Yes. Setting `mode=auto` enables Adaptive Stealth Mode, which automatically activates JavaScript rendering, premium residential proxies, and stealth fingerprinting only when the target site requires them. You only pay for what succeeds.
</Accordion>

<Accordion title="Can I use this with the OpenAI Agents SDK?">
  Yes. The Agents SDK is built on top of the Responses API, so the same patterns apply. For agent-specific workflows including multi-step research, handoffs, and the Agents SDK's built-in MCP support, see the [OpenAI Agents SDK integration guide](https://docs.zenrows.com/integrations/openai-agents-sdk).
</Accordion>

<Accordion title="Can I let GPT browse multiple pages autonomously?">
  Yes. Function calling supports multi-turn tool use, so GPT can call `scrape_website` repeatedly with different URLs, read what it gets back, and decide what to scrape next. For more complex agent behavior, pair this with the [hosted MCP tool](https://docs.zenrows.com/mcp/overview), which exposes a richer scraping surface without requiring you to maintain the function-calling loop yourself. For autonomous, multi-step research workflows, see the [OpenAI Agents SDK integration guide](https://docs.zenrows.com/integrations/openai-agents-sdk).
</Accordion>

<Accordion title="How does this compare to OpenAI's built-in web search tool?">
  OpenAI's built-in web search is a black box: you don't choose the source, control the depth, or get raw page content. Zenrows gives you full control over which URL is fetched, how the page is rendered, what format the response comes back in, and which countries the request is routed through. It also unlocks pages that public search engines don't surface or that block scrapers, such as logged-in views, region-restricted content, and deep e-commerce listings.
</Accordion>

<Accordion title="Is there a rate limit I should be aware of?">
  Zenrows has plan-based concurrency limits documented on the [Concurrency page](https://docs.zenrows.com/fetch/features/concurrency). When using function calling or agents that may issue many parallel scrapes, monitor your usage on the Analytics page and consider adding retry logic with exponential backoff in your `scrape_website` wrapper.
</Accordion>


# How to Integrate OpenAI Agents SDK with Zenrows
Source: https://docs.zenrows.com/integrations/openai-agents-sdk

Build autonomous, multi-step AI agents that scrape and automate any site via Zenrows' MCP and OpenAI Agents SDK. No orchestration code.

Combine the <a href="https://openai.github.io/openai-agents-python/">OpenAI Agents SDK</a> with the [Zenrows hosted MCP server](https://docs.zenrows.com/mcp/overview) to build autonomous, multi-step agents that can read any website in real time. The Agents SDK has native support for MCP servers, so plugging in Zenrows takes a single block of code, with no function-calling boilerplate, no scraping infrastructure, and no anti-bot tuning.

This guide covers the OpenAI Agents SDK. If you want to use Zenrows with the standard OpenAI API, see [How to Integrate OpenAI with Zenrows](https://docs.zenrows.com/integrations/openai).

## What is OpenAI Agents SDK?

The <a href="https://openai.github.io/openai-agents-python/">OpenAI Agents SDK</a> is a lightweight Python framework for building multi-agent workflows. It provides a small set of primitives, including agents, tools, handoffs, guardrails, sessions, and tracing, that compose into autonomous AI applications. Agents are LLMs equipped with instructions and tools. The SDK handles the orchestration loop, tool calls, handoffs between agents, and conversation state.

Agents are well-suited to tasks that need multiple steps, tool selection, or coordination across specialized roles. Adding Zenrows MCP gives every agent in your workflow real-time access to any website, including JavaScript-heavy and bot-protected ones.

### Key benefits of integrating OpenAI Agents SDK with Zenrows

* **Native MCP support, zero glue code**: The Agents SDK's `HostedMCPTool` wrapper takes the Zenrows server URL and your API key. The agent then sees Zenrows' tools (`scrape` and 30+ browser automation tools) automatically, with no function-calling code to maintain.
* **Autonomous, multi-step web tasks**: Agents can scrape one page, decide what to scrape next based on what they read, and chain together browser interactions like form fills and clicks across multiple pages, without you writing the orchestration loop.
* **Anti-bot bypass on every tool call**: Premium residential proxies, JavaScript rendering, and stealth fingerprinting come standard. Your agents reach JavaScript-heavy SPAs, Cloudflare-protected sites, and geo-restricted pages without configuration.
* **Composes with handoffs, tracing, and sessions**: Zenrows tools work alongside every Agents SDK feature. Pass them to multiple agents, hand off mid-workflow, persist sessions across runs, and trace every tool call in the OpenAI dashboard.
* **Hosted execution, no local install**: The hosted MCP server runs on Zenrows infrastructure. Your agent code stays clean, with no proxy management, scraping retries, or headless browser orchestration in your codebase.

## Use cases

The Agents SDK and Zenrows combination unlocks a wide range of agent workflows:

* **Autonomous research agents**: Agents that visit multiple URLs, follow citations, and synthesize findings from across the web into a single brief.
* **Lead enrichment pipelines**: Agents that take a company name, scrape the company website, find pricing pages and product launches, and write enriched profiles into your CRM.
* **Competitive monitoring**: Agents that periodically check competitor pricing pages, blog posts, and changelogs, then surface what changed since the last run.
* **Multi-agent web automation**: A research agent that gathers information and hands off to a writing agent that produces the final report, both pulling from live web data.
* **Form-filling and extraction agents**: Agents that navigate multi-step checkout flows using Zenrows' browser tools.

## Getting started: Basic Usage

Let's start with a simple example: a single agent equipped with Zenrows MCP that can answer questions by reading any website in real time.

<Steps>
  <Step title="Install the OpenAI Agents SDK (Python 3.10+ required)">
    ```bash theme={"dark"}
    pip install openai-agents
    ```
  </Step>

  <Step title="Create a .env file with your API keys">
    ```bash theme={"dark"}
    ZENROWS_API_KEY=your_zenrows_key
    OPENAI_API_KEY=your_openai_key
    ```
  </Step>

  <Step title="Run the following script">
    ```python Python theme={"dark"}
    import asyncio
    import os
    from agents import Agent, HostedMCPTool, Runner

    ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

    async def main() -> None:
        agent = Agent(
            name="Web Research Assistant",
            instructions=(
                "You are a research assistant with live web access. "
                "Scrape pages when you need real-time information."
            ),
            tools=[
                HostedMCPTool(
                    tool_config={
                        "type": "mcp",
                        "server_label": "zenrows",
                        "server_description": "Web scraping MCP server for accessing live web content.",
                        "server_url": "https://mcp.zenrows.com/mcp",
                        "authorization": ZENROWS_API_KEY,
                        "require_approval": "never",
                    }
                )
            ],
        )

        result = await Runner.run(
            agent,
            "Visit https://news.ycombinator.com/ and summarize the three most recent posts.",
        )

        print(result.final_output)

    asyncio.run(main())
    ```
  </Step>
</Steps>

`HostedMCPTool` registers the Zenrows hosted MCP server as a tool. The `tool_config` mirrors the Responses API MCP tool config: `server_url` points at Zenrows, `authorization` carries your API key as the OAuth Bearer token, and `require_approval="never"` skips the per-call approval step.

`Agent` equips the agent with that tool. The agent now has access to all Zenrows tools (`scrape`, `browser_navigate`, `browser_click`, and so on) automatically.

`Runner.run()` executes the agent. The agent reads the prompt, decides to call Zenrows' `scrape` tool with the Hacker News URL, gets the content back as Markdown, and writes a summary.

### Example output

```wrap Output theme={"dark"}
Here are the three most recent posts on the front page of Hacker News (https://news.ycombinator.com/) as of now:

### 1. [Accelerating Gemma 4: faster inference with multi-token prediction drafters](https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/)
- **Source:** blog.google
- **Summary:** Google shares an update on improving inference speed for its Gemma 4 AI model...
---
### 2. [Three Inverse Laws of AI](https://susam.net/inverse-laws-of-robotics.html)
- **Source:** susam.net
- **Summary:** A blog discusses three proposed "inverse laws" for artificial intelligence, likely as a counterpoint to Isaac Asimov's classic...
---
### 3. [Computer Use is 45x more expensive than structured APIs](https://reflex.dev/blog/computer-use-is-45x-more-expensive-than-structured-apis/)
- **Source:** reflex.dev
- **Summary:** This post analyzes the cost disparity between general computer use (running code, scripts, unstructured processing) and consuming structured APIs...
```

**You've built a working web-aware agent in under 30 lines of code.** Let's look at more advanced patterns.

## Advanced Usage: Building an autonomous research agent

The real power of the Agents SDK is letting an agent decide for itself which tools to call across multiple steps. With Zenrows MCP, that means the agent can autonomously read several pages, follow links it finds interesting, and synthesize findings, all from a single prompt.

The example below builds a research agent that visits a topic landing page, identifies the most relevant linked articles, scrapes each one, and produces a synthesized report:

```python Python theme={"dark"}
import asyncio
import os
from agents import Agent, HostedMCPTool, Runner

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

async def main() -> None:
    research_agent = Agent(
        name="Research Agent",
        instructions=(
            "You are a thorough web researcher. When given a topic and a starting URL, "
            "visit the URL, identify the 3 most relevant linked articles, scrape each one, "
            "and produce a synthesized report covering key findings, common themes, and "
            "any contradictions. Cite each source with its URL."
        ),
        tools=[
            HostedMCPTool(
                tool_config={
                    "type": "mcp",
                    "server_label": "zenrows",
                    "server_description": "Web scraping MCP server for accessing live web content.",
                    "server_url": "https://mcp.zenrows.com/mcp",
                    "authorization": ZENROWS_API_KEY,
                    "require_approval": "never",
                }
            )
        ],
        model="gpt-4o",
    )

    result = await Runner.run(
        research_agent,
        "Research recent developments in AI. Start at https://huggingface.co/blog and follow the most relevant links.",
    )

    print(result.final_output)

asyncio.run(main())
```

A few things worth noting:

* No orchestration loop in your code. The agent decides on its own when to scrape, which URLs to follow, and when it has enough information to write the report. The Agents SDK runs the multi-step loop automatically.
* Use a capable model for multi-step tasks. Models like `gpt-4o` work better for autonomous workflows because they can plan their next action based on what they've learned so far.
* Same MCP config as the basic example. The only things that changed are the agent's instructions and the prompt. Zenrows handles the scraping; the SDK handles the loop.

## Structured outputs

The Agents SDK supports strongly-typed outputs through the `output_type` parameter on the `Agent` constructor. Combined with Zenrows, this is the cleanest way to extract typed data from any web page in an agent workflow.

The example below scrapes a company homepage and extracts a strongly-typed `CompanyInfo` object:

```python Python theme={"dark"}
import asyncio
import os
from agents import Agent, HostedMCPTool, Runner
from pydantic import BaseModel

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

class CompanyInfo(BaseModel):
    name: str
    industry: str
    description: str

async def main() -> None:
    agent = Agent(
        name="Company Researcher",
        instructions="Visit the URL the user provides and extract structured company information from the page.",
        tools=[
            HostedMCPTool(
                tool_config={
                    "type": "mcp",
                    "server_label": "zenrows",
                    "server_description": "Web scraping MCP server for accessing live web content.",
                    "server_url": "https://mcp.zenrows.com/mcp",
                    "authorization": ZENROWS_API_KEY,
                    "require_approval": "never",
                }
            )
        ],
        output_type=CompanyInfo,
        model="gpt-4o-mini",
    )

    result = await Runner.run(
        agent,
        "Extract company information from https://www.zillow.com/",
    )

    company: CompanyInfo = result.final_output
    print(company.model_dump_json(indent=2))

asyncio.run(main())
```

### Example output

```json JSON theme={"dark"}
{
  "name": "Zillow (Zillow Group)",
  "industry": "Real estate / Online real estate marketplace",
  "description": "Zillow is an online real estate marketplace and services company offering home listings for sale and rent, rental-management and agent-finding tools, home value estimates (Zestimates), mortgage products through Zillow Home Loans, and affiliated brands such as Trulia, StreetEasy, and HotPads."
}
```

The `output_type` parameter makes the SDK enforce the Pydantic schema on the final output. The agent autonomously calls Zenrows' `scrape` tool to fetch the page, then parses the Markdown content into the `CompanyInfo` model. Type-safe access to the result is available through `result.final_output`.

The same pattern works for any structured extraction job: product listings, job postings, real estate, reviews, articles, and so on. Define the Pydantic schema, point the agent at the URL, and let it return validated data.

<Note>
  **Skip the schema with Zenrows Autoparse**

  For common site types like e-commerce products, articles, jobs, and real estate listings, you can skip LLM-based parsing entirely. Tell your agent to scrape with autoparse enabled, and Zenrows automatically identifies and extracts the page's structured data into clean JSON, with no Pydantic schema or extra model call required.

  [Zenrows Autoparse](https://docs.zenrows.com/fetch/features/autoparse) is included at no additional cost. The agent can use the structured JSON directly, or pass it to GPT for downstream enrichment, normalization, or categorization. Pre-structured input also uses far fewer tokens than raw Markdown, which keeps your context window lean and your agent runs cheaper.

  Note that Autoparse cannot be combined with Markdown output in the same scrape call.
</Note>

## Handoffs and multi-agent workflows

Because Zenrows MCP is a standard tool on the agent, you can mix it with the SDK's other primitives like handoffs, sessions, and guardrails. A common pattern is to use one agent for research (with Zenrows access) and another for writing (without web tools), passing control between them via <a href="https://openai.github.io/openai-agents-python/handoffs/">handoffs</a>:

```python Python theme={"dark"}
import os
from agents import Agent, HostedMCPTool

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

zenrows_tool = HostedMCPTool(
    tool_config={
        "type": "mcp",
        "server_label": "zenrows",
        "server_description": "Web scraping MCP server for accessing live web content.",
        "server_url": "https://mcp.zenrows.com/mcp",
        "authorization": ZENROWS_API_KEY,
        "require_approval": "never",
    }
)

writer_agent = Agent(
    name="Writer Agent",
    instructions="Take research findings and produce a polished article.",
    model="gpt-4o-mini",
)

researcher_agent = Agent(
    name="Researcher Agent",
    instructions=(
        "Gather research from the web on the user's topic. "
        "Once you have enough material, hand off to the Writer Agent."
    ),
    tools=[zenrows_tool],
    handoffs=[writer_agent],
    model="gpt-4o",
)
```

The researcher agent calls Zenrows tools to gather material, then hands off to the writer agent (which has no web access) for the final draft. See the <a href="https://openai.github.io/openai-agents-python/handoffs/">Agents SDK handoffs documentation</a> for the full pattern.

## Tool filtering and approval gates

The Zenrows MCP server exposes 30+ tools by default. For most agents this is fine, but two SDK features are worth knowing about for production use.

### Restrict which tools the agent sees

If you only want the agent to use specific Zenrows tools (for example, only `scrape` and not the browser automation tools), you can filter at the SDK level. See <a href="https://openai.github.io/openai-agents-python/mcp/#tool-filtering">Tool filtering in the Agents SDK MCP docs</a> for the exact API.

### Require approval for sensitive tool calls

By default, the examples above set `require_approval="never"`, which lets the agent call tools without confirmation. For agents that interact with logged-in pages, submit forms, or affect external state, set `require_approval="always"` and provide an `on_approval_request` callback to gate execution behind your own logic. See <a href="https://openai.github.io/openai-agents-python/mcp/#optional-approval-flows">Optional approval flows</a> for details.

## Troubleshooting

### 424 failed dependency or 401 unauthorized from the MCP tool

* **Option 1**: Confirm the `authorization` field is set on the `tool_config` and that the value is your active Zenrows API key. OpenAI does not retain header values between requests, so the field must be present on every call.
* **Option 2**: Verify your key in the [Zenrows dashboard](https://app.zenrows.com/settings/api-keys) and confirm your subscription has remaining quota.
* **Option 3**: Confirm `server_url` is exactly `https://mcp.zenrows.com/mcp`.

### Empty tool list or empty output

If the agent's response indicates it didn't find any tools, the connection to the MCP server failed silently. Check your API key and `server_url`.

### Agent repeatedly scrapes the same page

Add explicit guidance in the agent's instructions about what to do once it has the data, for example "After scraping, summarize and stop." For complex flows, set a tool-call limit in `model_settings`.

### Reasoning items missing after multi-step runs

The Agents SDK handles reasoning-item passing automatically. If you're seeing empty model responses on multi-step runs, ensure you're using a capable model like `gpt-4o` or `gpt-4o-mini` and that your prompt asks the model to take multiple steps.

### Content-level issues (page blocked, missing data)

If the agent connects and invokes tools correctly but the scraped content comes back blocked or incomplete, the issue is on the target page rather than the MCP connection. Add explicit guidance to your prompt, for example "Scrape this page using JavaScript rendering" or "Use Premium Proxies." See the [Fetch troubleshooting guide](https://docs.zenrows.com/fetch/troubleshooting/troubleshooting-guide) for parameter-level fixes.

## Helpful resources

* [Zenrows MCP server documentation](https://docs.zenrows.com/mcp/overview)
* [Adaptive Stealth Mode documentation](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode)
* [How to Integrate OpenAI with Zenrows (Responses API)](https://docs.zenrows.com/integrations/openai)
* [OpenAI Agents SDK documentation](https://openai.github.io/openai-agents-python/)
* [Agents SDK MCP guide](https://openai.github.io/openai-agents-python/mcp/)
* [Agents SDK handoffs guide](https://openai.github.io/openai-agents-python/handoffs/)
* [Agents SDK tracing guide](https://openai.github.io/openai-agents-python/tracing/)

## Frequently asked questions

<Accordion title="What's the difference between this and the standard OpenAI integration?">
  The [standard OpenAI integration](https://docs.zenrows.com/integrations/openai) covers the Responses API, function calling, and structured outputs. These are patterns where you make one or two model calls and process the result.

  The Agents SDK is for autonomous, multi-step agents: the SDK runs the orchestration loop, decides which tools to call, and chains tool calls without you writing the glue. Use the standard integration when you need direct control. Use the Agents SDK when you want autonomy and multi-step behavior.
</Accordion>

<Accordion title="Do I need to add Zenrows to mcp_servers as well as tools?">
  No. `HostedMCPTool` runs through OpenAI's infrastructure, so it goes only in the agent's `tools` list. `mcp_servers` is for SDK-managed connections (`MCPServerStreamableHttp`, `MCPServerStdio`, `MCPServerSse`).
</Accordion>

<Accordion title="Can I use the local stdio version of Zenrows MCP with the Agents SDK?">
  Yes. Use `MCPServerStdio` from `agents.mcp` with the `@zenrows/mcp` npm package. This is less common in agent applications because the hosted remote URL is simpler to set up and runs on OpenAI's infrastructure rather than in your process. See the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview) for the local stdio configuration.
</Accordion>

<Accordion title="Can multiple agents share the same Zenrows MCP tool?">
  Yes. Build the `HostedMCPTool` once and pass it to as many agents as you need:

  ```python theme={"dark"}
  zenrows_tool = HostedMCPTool(tool_config={...})
  agent_a = Agent(name="A", tools=[zenrows_tool])
  agent_b = Agent(name="B", tools=[zenrows_tool])
  ```

  Each agent gets independent tool calls. Zenrows handles them all on the same hosted server.
</Accordion>

<Accordion title="Is streaming supported?">
  Yes. Use `Runner.run_streamed()` instead of `Runner.run()` to receive incremental events as the agent works. See <a href="https://openai.github.io/openai-agents-python/mcp/#streaming-hosted-mcp-results">Streaming hosted MCP results</a> in the Agents SDK docs.
</Accordion>

<Accordion title="Are MCP tool calls traced automatically?">
  Yes. The Agents SDK <a href="https://openai.github.io/openai-agents-python/tracing/">traces</a> every MCP `list_tools` and tool-call invocation. You can view them in the OpenAI dashboard alongside your agent's other steps.
</Accordion>

<Accordion title="How does this compare to LangChain or LangGraph?">
  The Agents SDK is OpenAI's first-party agent framework, optimized for the Responses API and reasoning models. LangChain and LangGraph are framework-agnostic and have a larger ecosystem of integrations. Zenrows MCP works with both. Pick the framework that fits your stack.
</Accordion>


# Zenrows MCP Integration for OpenClaw
Source: https://docs.zenrows.com/integrations/openclaw

Give OpenClaw's AI agent real-time access to any website through Zenrows MCP. Scrape protected pages, extract live data, and run scheduled monitoring tasks delivered directly to your messaging app.

## What is OpenClaw?

<a href="https://docs.openclaw.ai/">OpenClaw</a> is an open-source, self-hosted AI agent gateway that connects your messaging apps to a persistent AI agent. It runs a single Gateway process on your own machine or server, which acts as the bridge between any messaging app (Telegram, WhatsApp, Slack, Discord, iMessage, Microsoft Teams, Signal, and more) and an always-available AI assistant with access to tools, including file read/write, shell execution, web search, external APIs, and persistent memory across sessions.

## Key benefits of using Zenrows as OpenClaw's search backend

* **No more broken schedules:** When OpenClaw's built-in tools hit a blocked page mid-run, your entire automation fails silently. Zenrows handles those requests transparently, so your daily competitor monitoring, price-tracking digests, and scheduled reports run every time without manual intervention.

* **Read pages the way a browser does:** Many competitor sites and dashboards load content dynamically via JavaScript. Zenrows renders the full page before returning it, so your agent sees the complete content rather than an empty HTML shell. You can also instruct the agent to click, scroll, fill forms, and navigate across pages, all in plain English, no code required.

* **Full scraping infrastructure, zero code:** Proxy rotation, anti-bot evasion, browser orchestration, and session management all run on Zenrows' infrastructure. Just describe the task to your agent in Telegram, WhatsApp, or Slack, and Zenrows handles the rest in the background.

* **Automate multi-step browser workflows:** For sites that require interaction before data surfaces, such as clicking through pagination, filling in search fields, or navigating across multiple pages, instruct your agent in plain English. Zenrows handles the browser session while OpenClaw delivers the result to your messaging app.

* **LLM-friendly output that keeps token usage in check:** Raw HTML is full of boilerplate (navigation menus, scripts, ads, and markup) that adds tokens without adding value. Zenrows strips all of that and returns clean, structured content by default. Your agent gets straight to the useful data, and since LLM APIs charge per token processed, less noise means lower costs per automation run.

## What can you do with this integration?

* **Track competitor pricing and get notified of changes:** Ask your agent to check a competitor's pricing page every morning and send you a Telegram message when something changes, so you always know without visiting the site yourself.

* **Build a daily industry briefing delivered to your phone:** Point your agent at a list of news sites, blogs, or forums and have it pull the top stories each morning and summarize them into a single message in your WhatsApp or Slack.

* **Get the best deals delivered to your phone every day:** Ask your agent to scan Amazon, eBay, and other e-commerce sites for deals matching your criteria and send you a curated list each morning before you start your day.

* **Monitor job boards for sales prospects:** Set up your agent to check LinkedIn, Indeed, or niche job boards daily for companies hiring for roles that signal buying intent, such as Head of Sales or VP of Marketing, and deliver the leads directly to your chat.

* **Dedicate separate agents to different tasks from a single Gateway:** Set up one OpenClaw agent dedicated to competitor monitoring via Telegram and another handling deal alerts via WhatsApp, both running from a single Gateway. Each agent has its own context, memory, and instructions, so they stay focused without mixing tasks or sessions.

## Zenrows as OpenClaw's search backend

OpenClaw natively supports MCP (Model Context Protocol) server connections. You can register any MCP-compatible server directly in your `openclaw.json` config file, and your agent gains access to its tools immediately. This guide walks through adding the [Zenrows MCP](/mcp/overview) server in that config file, so your agent can scrape any website and deliver results to your messaging app of choice.

### Prerequisites

Before you begin, make sure you have:

* Your Zenrows API key from the [Zenrows dashboard](https://app.zenrows.com/register).
* A messaging app to connect to OpenClaw (Telegram is the fastest to set up).
* An Anthropic API key or that of any LLM provider you choose.

<Steps>
  <Step title="Install OpenClaw">
    OpenClaw requires Node.js 24 (recommended) or Node.js 22.19+. If you don't have Node.js installed, download it from <a href="https://nodejs.org">nodejs.org</a> before continuing.

    **Install via npm:**

    ```bash theme={"dark"}
    npm install -g openclaw@latest
    ```

    If you don't have npm available, use the installer script for your operating system instead.

    **macOS / Linux / WSL2:**

    ```bash theme={"dark"}
    curl -fsSL https://openclaw.ai/install.sh | bash -s -- --no-onboard
    ```

    **Windows (PowerShell):**

    ```powershell theme={"dark"}
    & ([scriptblock]::Create((iwr -useb https://openclaw.ai/install.ps1))) -NoOnboard
    ```

    Check the <a href="https://docs.openclaw.ai/install">OpenClaw installation page</a> for more installation options.
  </Step>

  <Step title="Configure OpenClaw's AI agent and messaging channel">
    This guide uses Telegram as the chat channel. Set it up before continuing:

    * Open the app on your PC or phone and start a chat with `@BotFather`.
    * Run `/newbot`. Follow the prompt to obtain a bot token, then copy it.

    You can use any other supported channel. Check <a href="https://docs.openclaw.ai/channels#supported-channels">OpenClaw's supported channels page</a> for setup instructions per platform.

    To start the interactive onboarding process, run:

    ```bash theme={"dark"}
    openclaw onboard --install-daemon
    ```

    During onboarding, follow the prompts for these key steps:

    * **Model/auth provider:** Select `Anthropic` or any other AI model provider.
    * **Anthropic auth method:** Pick the API key option and enter your API key.
    * **Default model:** Choose your preferred model (e.g., `anthropic/claude-sonnet-4-6`).
    * **Select channel:** Choose Telegram, then enter the bot token you copied from `@BotFather`. Check <a href="https://docs.openclaw.ai/channels/telegram">OpenClaw's Telegram setup</a> for more information.

    Once onboarding is complete, click or tap the bot URL provided in the `@BotFather` response to initiate a chat with your new bot.
  </Step>

  <Step title="Add Zenrows MCP to OpenClaw's configuration">
    OpenClaw config lives at `~/.openclaw/openclaw.json`. Open this file and add the following Zenrows MCP connection configuration to the `mcp` block:

    ```json theme={"dark"}
    "mcp": {
      "servers": {
        "zenrows": {
          "type": "url",
          "url": "https://mcp.zenrows.com/mcp",
          "transport": "streamable-http",
          "headers": {
            "Authorization": "Bearer <YOUR_ZENROWS_API_KEY>"
          },
          "timeout": 30000
        }
      }
    }
    ```

    In the same file, find the `mcporter` configuration and ensure it is set to `true`:

    ```json theme={"dark"}
    "mcporter": {
      "enabled": true
    }
    ```

    Confirm the MCP integration by running:

    ```bash theme={"dark"}
    openclaw mcp list
    ```

    The above command returns `- zenrows`, confirming you've successfully integrated OpenClaw with the Zenrows MCP.
  </Step>

  <Step title="Start the OpenClaw server">
    Launch OpenClaw's local server:

    ```bash theme={"dark"}
    openclaw gateway start
    ```
  </Step>
</Steps>

## Getting started: Basic usage

Open the bot you created in Telegram and send a scrape message to your AI agent:

```text theme={"dark"}
Go to https://www.scrapingcourse.com/antibot-challenge and summarize the page.
```

OpenClaw's Gateway receives the Telegram message, passes it to the AI agent, which then uses the Zenrows MCP to fetch the page and bypass its anti-bot protection. The scraped content is summarized by your configured model and sent back to you in Telegram.

```text Output wrap theme={"dark"}
Here's the summary:

Page: scrapingcourse.com/antibot-challenge
Title: Antibot Challenge

It's a demo page from ScrapingCourse designed to test whether a scraper/bot can bypass anti-bot protections (like Cloudflare's JS challenge). When accessed, the page initially shows a "Please wait..." screen while it runs bot detection.

Once bypassed successfully, it displays a single success message:

You bypassed the Antibot challenge! :D

That's essentially the whole content: it's a minimal test page whose sole purpose is to verify that a scraping tool can get past bot detection and reach the actual page content. Zenrows handled it cleanly.
```

## Advanced usage: Scheduling scraping jobs with OpenClaw and Zenrows

OpenClaw natively supports cron-style scheduling, so you can instruct your agent to run any scraping task on a recurring schedule directly from your chat.

Open your Telegram bot and send:

```text wrap theme={"dark"}
Every day at 9AM, scrape https://www.amazon.com/s?k=headset, get the top 3 best deals based on price and ratings, and send me a summary.
```

OpenClaw's agent registers the schedule and runs it automatically at the specified time each day. At 9:00 AM, it fetches the Amazon search page via Zenrows, bypassing Amazon's anti-bot protection and rendering the JavaScript-loaded results, then extracts the top deals and delivers a formatted summary to your Telegram chat without any manual input.

```text Output wrap theme={"dark"}
Done! Cron job set up:

• Schedule: Every day at 9:00 AM
• Task: Scrapes Amazon for headset, picks the top 3 best deals by price + ratings
• Delivery: Summary sent directly to you here on Telegram
• Next run: Tomorrow morning at 9 AM

Each morning you'll get a breakdown with product name, price, rating, review count, and a link.
```

When the schedule runs, here is what a sample response looks like:

```text Output wrap theme={"dark"}
Here's today's summary of the top 3 headset deals from Amazon

Top 3 Headset Deals on Amazon - June 1, 2026

#1 - Logitech H390 Wired USB Headset
$15.99 $24.99 (36% off)
4.4/5 - 68,300+ reviews
Wired USB-A, noise-cancelling mic, in-line controls, Chromebook compatible
amazon.com/dp/B000UXZQ42

Why it's #1: Lowest price, highest rating, and massively battle-tested with 68k reviews. Best bang for buck, full stop.

# ...further responses truncated for brevity
```

To view your active schedules, send the following message via Telegram:

```text theme={"dark"}
List my active scheduled tasks.
```

The agent returns a list of all scheduled jobs currently running on your Gateway:

```text Output theme={"dark"}
You've got 1 active scheduled task:

Daily Amazon Headset Deals
• Every day at 9:00 AM (<local time>)
• Scrapes Amazon for top 3 headset deals by price + ratings
• Delivered to you on Telegram
• Next run: Tomorrow, Jun 2 at 9 AM
```

## Local vs. remote OpenClaw

By default, OpenClaw runs the Gateway on your local machine, which means the agent is only active while that machine is on and connected. This works fine for manual tasks but means scheduled jobs like your daily Zenrows scraping tasks will stop running if your machine sleeps or goes offline.

For always-on scraping schedules, OpenClaw supports running the Gateway on a persistent remote host such as a VPS or home server, with your devices connecting to it over Tailscale or an SSH tunnel. This keeps the agent running continuously so Zenrows tasks execute at the scheduled time regardless of whether your local machine is active.

For remote setup instructions, see the <a href="https://docs.openclaw.ai/gateway/remote">OpenClaw remote access guide</a>.

## Troubleshooting

### WSL issues on Windows

If you installed OpenClaw inside WSL but are running Node.js from Windows (or vice versa), `npx` may not be found because the two environments have separate PATH configurations. Run `which npx` inside WSL to confirm it resolves correctly. If it doesn't, install Node.js directly inside WSL via `nvm` rather than relying on a Windows installation.

### McpError: MCP error -32001: Request timed out

This happens when OpenClaw cannot complete the MCP handshake with `mcp.zenrows.com` within the default timeout window. Common causes:

* **Invalid or missing API key:** Confirm your key is active in the [Zenrows dashboard](https://app.zenrows.com/register) and that `YOUR_ZENROWS_API_KEY` has been replaced with the actual value in your config.
* **Slow network conditions:** If the connection succeeds sometimes and fails at other times, the handshake is timing out intermittently. Increase the `timeout` value in your MCP server config to give it more time. Intermittent timeouts on a remote MCP server are often due to transient network issues rather than a config problem.
* **Check Zenrows MCP connection status:** Run `openclaw doctor` to check the Zenrows MCP connection status for any errors or warnings.
* **Check `mcporter` configuration:** Ensure that `mcporter` is set to `enabled: true` in your `openclaw.json`.
* **Gateway restart timing:** If the Gateway hasn't fully started before attempting the MCP connection, the handshake times out. Wait a few seconds after `openclaw gateway start` before testing.

### SSE error: undefined

OpenClaw is attempting to connect using SSE transport instead of Streamable HTTP. Make sure both `"type": "url"` and `"transport": "streamable-http"` are set together in your config. Using only one of them can cause OpenClaw to fall back to an incompatible transport.

### web\_fetch failed: Web fetch failed (403)

The agent is using OpenClaw's built-in web fetcher instead of Zenrows, and the target page is blocking the request. This usually means the Zenrows MCP server has not connected successfully. Run `openclaw mcp list` to confirm the Zenrows MCP server is listed as active. If the connection is working but the 403 error persists, explicitly instruct the agent:

```text theme={"dark"}
Use Zenrows to scrape this page.
```

### No response is returned in Telegram

* Check that your bot token is correct and the bot was created via `@BotFather`.
* Make sure `dmPolicy` is set to `"pairing"` or `"open"` in your Telegram channel config in `openclaw.json`, as a restrictive policy can silently drop messages from unpaired users.
* Confirm you are messaging the correct Telegram bot. You can verify the bot token value inside the `channels` configuration of your `openclaw.json` file.

## FAQ (Frequently asked questions)

<Accordion title="Should I run OpenClaw locally or remotely?">
  It depends on your use case. Running locally is simpler to set up and works well for on-demand tasks that you trigger manually. If you rely on scheduled scraping jobs, such as daily competitor monitoring or deal alerts, a remote setup on a VPS or home server is the better choice, since the Gateway runs continuously regardless of whether your local machine is on.
</Accordion>

<Accordion title="How can I access OpenClaw remotely?">
  OpenClaw supports remote Gateway access via Tailscale or an SSH tunnel. You run the Gateway on a persistent host and connect your devices to it over your tailnet or a forwarded port. See the <a href="https://docs.openclaw.ai/gateway/remote">OpenClaw remote access guide</a> for setup instructions.
</Accordion>

<Accordion title="Does OpenClaw have built-in search services?">
  Yes. OpenClaw includes built-in search providers such as Brave, Tavily, Exa, and Perplexity, as well as a web fetch tool powered by Firecrawl. These work well for general search but can fail when the agent tries to read pages protected by anti-bot systems. Zenrows fills that gap by handling pages that block standard requests.
</Accordion>

<Accordion title="Is OpenClaw an AI agent?">
  OpenClaw is a Gateway that connects messaging apps to an AI agent. The agent runs persistently, maintains memory across sessions, and has access to tools including file operations, shell execution, web search, and MCP servers. It can take actions autonomously based on your instructions.
</Accordion>

<Accordion title="Which AI models can I use with OpenClaw?">
  OpenClaw supports models from Anthropic, OpenAI, Google Gemini, and others via a `provider/model` format in your config. You can set a different model for each agent, enabling you to use a faster model for everyday tasks and a more capable one for complex research workflows.
</Accordion>

<Accordion title="Can I use the local stdio version of Zenrows MCP via npx?">
  Yes, but it requires Node.js to be installed and `npx` to be accessible from the environment OpenClaw's Gateway runs in. If you installed OpenClaw as a background daemon, it may not inherit your shell's PATH, which can cause a `spawn npx ENOENT` error. To fix this, either provide the full path to `npx` in your config or use the hosted remote MCP server with Streamable HTTP instead (as done in this guide), which avoids the local subprocess entirely.
</Accordion>

<Accordion title="Can I use other OpenClaw tools and skills alongside Zenrows?">
  Yes. Zenrows MCP registers as one tool provider among many. Your agent can combine Zenrows scraping with OpenClaw's built-in tools such as file write, shell execution, and search providers in the same task. For example, the agent can scrape a page via Zenrows, process the data, and write the output to a file in a single uninterrupted workflow.
</Accordion>

<Accordion title="Which messaging channels does OpenClaw support?">
  OpenClaw supports Telegram, WhatsApp, Slack, Discord, Signal, iMessage, Microsoft Teams, and more. Each channel is configured under its own key in `openclaw.json`, and you can run multiple channels simultaneously, routing different conversations to different agents if needed.
</Accordion>


# How to Integrate Zenrows with OpenCode
Source: https://docs.zenrows.com/integrations/opencode

Add the Zenrows MCP to OpenCode and let your AI coding agent browse the web and fetch web content reliably without anti-bot or JavaScript rendering limitations. Fetch live documentation, pull real API specs, and scrape technical references directly inside your coding session.

## What is OpenCode?

OpenCode is an open-source AI coding agent available as a terminal UI, desktop app, or IDE extension. It connects to your codebase, understands your project structure, and helps you build features, refactor code, fix bugs, and navigate unfamiliar code, all from your editor or terminal.

Like other AI coding tools, OpenCode can call external tools via MCP. Connect Zenrows and it gains reliable access to any web resource: protected documentation sites, JavaScript-rendered API references, paywalled technical content, and any page that would otherwise return a blocked or empty response.

## Why connect Zenrows to OpenCode?

* **Fetch live docs that block automated requests.** Many documentation sites (Cloudflare, AWS, Stripe, and others) return challenges or empty pages to automated clients. Zenrows bypasses that so OpenCode gets the actual content.
* **Pull real API responses, not cached snapshots.** When OpenCode needs to understand an external API, Zenrows fetches the live spec or reference page rather than relying on potentially stale training data.
* **Research technical content during a coding session.** Ask OpenCode to scrape Stack Overflow threads, GitHub issues, technical blogs, or framework changelogs and incorporate the findings directly into the task it's working on.
* **Access JavaScript-rendered pages.** Developer tools, dashboards, and interactive docs often require JavaScript to render their content. Zenrows handles that automatically.

## What you can build

* **Live documentation lookup during development.** Ask OpenCode to fetch the latest docs for a library you're integrating and use them to generate correct, up-to-date code, not code based on training data that may be months old.
* **Automated competitor or dependency research.** Have OpenCode scrape changelogs, release notes, or breaking change announcements across your key dependencies and summarize what needs updating in your project.
* **Technical reference extraction.** Describe a problem and let OpenCode scrape relevant Stack Overflow threads, GitHub issues, or RFC documents to inform its solution.
* **Live API testing context.** Before generating integration code, have OpenCode scrape the target API's reference page to confirm endpoint signatures, auth patterns, and response shapes.

## Prerequisites

* A Zenrows API key from the <a href="https://app.zenrows.com/fetch">Zenrows dashboard</a>.

## Getting started

<Steps>
  <Step title="Install OpenCode">
    **Windows / Linux / macOS:**

    ```bash theme={"dark"}
    npm install -g opencode-ai
    ```

    Or via the install script (Windows PowerShell / Linux):

    ```bash theme={"dark"}
    curl -fsSL https://opencode.ai/install | bash
    ```

    Other options: Homebrew (`brew install anomalyco/tap/opencode`), Chocolatey (`choco install opencode`), Scoop (`scoop install opencode`). See the <a href="https://opencode.ai/docs/#install">full install guide</a> for all options.

    OpenCode runs natively on Windows via PowerShell or Command Prompt. WSL is optional.

    Verify the install:

    ```bash theme={"dark"}
    opencode --version
    ```
  </Step>

  <Step title="Connect an LLM provider">
    OpenCode needs an LLM provider to process your prompts and call tools like Zenrows. Run the `/connect` command inside OpenCode and select your provider:

    ```text theme={"dark"}
    /connect
    ```

    For Anthropic, select Anthropic, enter your API key, then run `/models` to pick a model like `claude-sonnet-4-6`. For other providers, see the <a href="https://opencode.ai/docs/providers/">OpenCode full provider guide</a>.

    Once a model is selected, you're ready to add Zenrows.
  </Step>

  <Step title="Add Zenrows MCP to OpenCode">
    Zenrows connects as a remote MCP server. You'll need to create an `opencode.json` config file and add it under the `mcp` key.

    Create the file at one of these locations depending on your preferred scope:

    * **Project-level** (`opencode.json` in your project root): Zenrows available only in this project.
    * **Global** (`~/.config/opencode/opencode.json`): Zenrows available across all projects.

    If neither file exists yet, create it. Then paste the following:

    ```json JSON theme={"dark"}
    {
      "$schema": "https://opencode.ai/config.json",
      "mcp": {
        "zenrows": {
          "type": "remote",
          "url": "https://mcp.zenrows.com/mcp",
          "oauth": false,
          "enabled": true,
          "headers": {
            "Authorization": "Bearer YOUR_ZENROWS_API_KEY"
          }
        }
      }
    }
    ```

    <Note>`oauth: false` is required. Without it, OpenCode's automatic OAuth detection will intercept the request before the Bearer token header is sent.</Note>
  </Step>

  <Step title="Try it out">
    Start OpenCode in your project:

    ```bash theme={"dark"}
    opencode
    ```

    Then prompt it to use Zenrows:

    ```text wrap theme={"dark"}
    Fetch https://www.scrapingcourse.com/antibot-challenge and summarize the page.
    ```

    Or for a development-focused example:

    ```text wrap theme={"dark"}
    Use zenrows to scrape https://github.com/vercel/next.js/issues and find open issues related to hydration errors. Summarize the most common causes and suggested fixes.
    ```

    OpenCode will call the right Zenrows tool and incorporate the response directly into its reply.
  </Step>
</Steps>

## Scoping Zenrows to a specific agent (optional)

OpenCode supports custom agents: specialized AI assistants with their own prompts, models, and tool access. You can create one using:

```bash theme={"dark"}
opencode agent create
```

Then follow the interactive prompts to create one.

If you want Zenrows available only to a specific agent and not the default build agent, turn off the Zenrows tools globally using the `permission` key and re-enable them per agent:

```json JSON theme={"dark"}
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "zenrows": {
      "type": "remote",
      "url": "https://mcp.zenrows.com/mcp",
      "oauth": false,
      "enabled": true,
      "headers": {
        "Authorization": "Bearer YOUR_ZENROWS_API_KEY"
      }
    }
  },
  "permission": {
    "zenrows_*": "deny"
  },
  "agent": {
    "researcher": {
      "description": "Web research agent with Zenrows access",
      "mode": "subagent",
      "permission": {
        "zenrows_*": "allow"
      }
    }
  }
}
```

The `researcher` agent is now the only one that can call Zenrows tools. Invoke it by @ mentioning it in your session:

```text wrap theme={"dark"}
@researcher scrape https://github.com/vercel/next.js/issues and find open issues related to hydration errors.
```

For more on creating and configuring agents, see the <a href="https://opencode.ai/docs/agents/">OpenCode agents docs</a>.

## Add a Zenrows instruction to your agent file (optional)

When you run `opencode agent create` and follow the interactive prompts, OpenCode generates a markdown agent file at `.opencode/agents/<agent-name>.md`. The file has a frontmatter block for config and a body for the system prompt.

Add the Zenrows instruction to the **Guidelines** section of the body if one exists, or anywhere in the prompt body if not. For example:

```markdown Markdown theme={"dark"}
## Guidelines

- Use `zenrows_scrape` for fast, single-page fetches
- Use `zenrows_browser_*` tools when pages require complex navigation or user interaction
```

This tells the agent to use Zenrows when automatically fetching external content, so you don't need to specify it in every prompt.

## Troubleshooting

### Command 'opencode' not found after install

The `npm` global bin directory isn't in your PATH. Fix it by adding it.

**Linux / macOS:**

```bash theme={"dark"}
export PATH="$(npm config get prefix)/bin:$PATH"
```

Add that line to your `~/.bashrc` or `~/.zshrc` to make it permanent, then run `source ~/.bashrc`.

**Windows CMD:**

```batch theme={"dark"}
for /f "delims=" %i in ('npm config get prefix') do set PATH=%i;%PATH%
```

**Windows PowerShell:**

```powershell theme={"dark"}
$env:PATH = "$(npm config get prefix)" + ";$env:PATH"
```

Alternatively, install via snap:

```bash theme={"dark"}
sudo snap install opencode
```

### ERROR: CreateProcessCommon: execvpe(/bin/bash) failed: No such file or directory on Windows

This means WSL is installed, but no proper Linux distro is set up. The `docker-desktop` distro that ships with Docker Desktop is headless and doesn't provide a usable shell. You have two options:

Install a proper WSL distro:

```powershell theme={"dark"}
wsl --install -d Ubuntu
```

Or skip WSL entirely. OpenCode runs natively on Windows via PowerShell without needing WSL.

### Zenrows tools not appearing in OpenCode

* Confirm `enabled: true` is set in the Zenrows MCP block.
* Check that `ZENROWS_API_KEY` is correct.
* Restart OpenCode after saving the config.
* Run `opencode mcp list` to see all registered MCP servers and their status.

### Authentication error when calling Zenrows tools

Verify your API key is active in the <a href="https://app.zenrows.com">Zenrows dashboard</a>. Also confirm `oauth: false` is in your config. Without it, OpenCode's OAuth auto-detection will intercept the connection before the Bearer token is applied.

### OpenCode not using Zenrows even when available

Be explicit in your prompt: `use zenrows to fetch https://www.scrapingcourse.com/antibot-challenge`. Alternatively, add an instruction in your `<agent-file.md>` to set Zenrows as the default tool for external web fetches.

### Context limit warnings after adding Zenrows

Zenrows adds its full tool list to the context on every request. If you're hitting context limits, restrict OpenCode to only the tools you need using glob patterns in the `tools` config:

```json JSON theme={"dark"}
"tools": {
  "zenrows_browser_*": false
}
```

This turns off all browser automation tools and keeps only `zenrows_scrape` active. Or switch to a model with a larger context window.

## FAQ (Frequently asked questions)

<Accordion title="Should I add Zenrows to the project config or the global config?">
  Use project config to make Zenrows available only when working in a specific codebase. Use global config if you want it available across all your OpenCode sessions. The project config takes precedence over the global config when both are present.
</Accordion>

<Accordion title="Can I use the local stdio version of Zenrows MCP instead?">
  Yes. Replace the remote config with the local stdio format:

  ```json JSON theme={"dark"}
  {
      "$schema": "https://opencode.ai/config.json",
      "mcp": {
          "zenrows": {
              "type": "local",
              "command": ["npx", "-y", "@zenrows/mcp"],
              "enabled": true,
              "environment": {
                  "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
              }
          }
      }
  }
  ```

  Both options expose the same tools. The remote approach requires no local installation.
</Accordion>

<Accordion title="Can I connect multiple MCP servers alongside Zenrows?">
  Yes. Add each server as a named block under `mcp`:

  ```json JSON theme={"dark"}
  {
      "$schema": "https://opencode.ai/config.json",
      "mcp": {
          "zenrows": {
              "type": "remote",
              "url": "https://mcp.zenrows.com/mcp",
              "oauth": false,
              "enabled": true,
              "headers": {
                  "Authorization": "Bearer YOUR_ZENROWS_API_KEY"
              }
          },
          "sentry": {
              "type": "remote",
              "url": "https://mcp.sentry.dev/mcp",
              "oauth": {}
          }
      }
  }
  ```
</Accordion>

<Accordion title="Does Zenrows work with all LLM providers in OpenCode?">
  Yes. Zenrows MCP works regardless of which provider you connect via `/connect`. The MCP layer is independent of the model.
</Accordion>

<Accordion title="Can I temporarily turn off Zenrows without removing it from the config?">
  Yes. Set `enabled: false` in the Zenrows block:

  ```json JSON theme={"dark"}
  {
      "mcp": {
          "zenrows": {
              "type": "remote",
              "url": "https://mcp.zenrows.com/mcp",
              "oauth": false,
              "enabled": false,
              "headers": {
                  "Authorization": "Bearer YOUR_ZENROWS_API_KEY"
              }
          }
      }
  }
  ```

  Restart OpenCode, and the server won't load. Flip it back to `true` when you need it again.
</Accordion>

<Accordion title="Does Zenrows work in OpenCode's Plan mode?">
  Plan mode restricts file edits and bash commands but doesn't restrict MCP tool calls by default. Zenrows scraping tools remain available in Plan mode unless you explicitly deny them via the `permission` config.
</Accordion>

<Accordion title="Can I use Zenrows with OpenCode's built-in webfetch tool simultaneously?">
  Yes, they coexist. `webfetch` is OpenCode's native HTTP fetch tool, useful for simple, unprotected pages. Zenrows is better for pages behind bot protection, JavaScript rendering, or geo-restrictions. OpenCode will reach for whichever tool you specify, or make its own judgment if you don't specify one.
</Accordion>


# Zenrows' Integration Options
Source: https://docs.zenrows.com/integrations/overview

Explore every Zenrows integration: AI assistants, code editors, agent SDKs and frameworks, no-code platforms, and browser libraries.

Zenrows is a platform built around a small set of primitives, Fetch, Extract, Batch, and Browser Sessions, available however you already build. Integrations connect those primitives to the tools, editors, agents, and platforms you already use, so you don't have to change how you work to get reliable access to the live web.

## AI Assistants

Chat-based AI assistants that can scrape and extract data on your behalf when connected to Zenrows.

<Columns>
  <Card href="/integrations/claude-desktop" title="Claude Desktop" icon="message-chatbot">
    Connect Claude Desktop to Zenrows via MCP for real-time web access in your conversations.
  </Card>

  <Card href="/integrations/claude-code" title="Claude Code" icon="terminal">
    Give Claude Code the ability to scrape and extract data while it works in your codebase.
  </Card>
</Columns>

## AI Code Editors

AI-native editors and IDEs that can call Zenrows directly while you write code.

<Columns>
  <Card href="/integrations/cursor" title="Cursor" icon="device-laptop">
    Scrape and extract data from inside Cursor's AI coding assistant.
  </Card>

  <Card href="/integrations/windsurf" title="Windsurf" icon="device-laptop">
    Connect Windsurf's AI agent to real-time web data.
  </Card>

  <Card href="/integrations/vscode" title="VS Code" icon="device-laptop">
    Add Zenrows to VS Code's AI features via MCP.
  </Card>

  <Card href="/integrations/zed" title="Zed" icon="device-laptop">
    Connect Zed's AI assistant to Zenrows.
  </Card>

  <Card href="/integrations/jetbrains" title="JetBrains IDEs" icon="device-laptop">
    Add Zenrows to JetBrains' AI assistant across IntelliJ, PyCharm, WebStorm, and more.
  </Card>

  <Card href="/integrations/opencode" title="OpenCode" icon="device-laptop">
    Connect OpenCode's AI agent to real-time web data.
  </Card>

  <Card href="/integrations/antigravity-20" title="Google Antigravity 2.0" icon="device-laptop">
    Add Zenrows to Google's agentic development platform.
  </Card>

  <Card href="/integrations/antigravity-ide" title="Google Antigravity IDE" icon="device-laptop">
    Connect Zenrows to the Antigravity IDE.
  </Card>
</Columns>

## Agent SDKs & Frameworks

Official vendor SDKs and third-party frameworks for building autonomous, multi-step agents.

<Columns>
  <Card href="/integrations/claude-agent-sdk" title="Claude Agent SDK" icon="code">
    Build autonomous agents that scrape and automate any site with Anthropic's official SDK.
  </Card>

  <Card href="/integrations/openai-agents-sdk" title="OpenAI Agents SDK" icon="code">
    Build multi-step agents with OpenAI's official Agents SDK.
  </Card>

  <Card href="/integrations/google-adk" title="Google ADK" icon="code">
    Build multi-agent workflows with Google's Agent Development Kit.
  </Card>

  <Card href="/integrations/langchain" title="LangChain" icon="code-branch">
    Orchestrate intelligent, multi-step data extraction workflows using language models.
  </Card>

  <Card href="/integrations/llamaindex" title="LlamaIndex" icon="book">
    Build scalable Retrieval-Augmented Generation (RAG) applications by indexing web data.
  </Card>

  <Card href="/integrations/flowise" title="Flowise" icon="diagram-project">
    Design AI-powered workflows and data pipelines with a visual, low-code builder.
  </Card>

  <Card href="/integrations/dify" title="Dify" icon="diagram-project">
    Add Zenrows Fetch, Extract, and Batch tools to Dify workflows and agents with the official plugin.
  </Card>

  <Card href="/integrations/mastra" title="Mastra" icon="code">
    Give agents built with the Mastra TypeScript framework reliable access to any website.
  </Card>
</Columns>

## AI Tools

Direct integrations with foundation model APIs and AI-powered platforms.

<Columns>
  <Card href="/integrations/openai" title="OpenAI" icon="wand">
    Give GPT models real-time access to any website using Fetch and MCP.
  </Card>

  <Card href="/integrations/anthropic-claude-api" title="Anthropic Claude API" icon="wand">
    Ground Claude's answers in live, protected, or JavaScript-heavy web data.
  </Card>

  <Card href="/integrations/clay" title="Clay" icon="database">
    Enrich leads and automate outbound workflows using AI.
  </Card>

  <Card href="/integrations/lindy" title="Lindy" icon="robot">
    Deploy AI agents for complex scraping and data extraction, no code required.
  </Card>
</Columns>

## Agent Platforms

Hosted, autonomous agent products that run the agent loop and infrastructure for you.

<Columns>
  <Card href="/integrations/claude-managed-agents" title="Claude Managed Agents" icon="cloud">
    Let Anthropic manage the agent loop and sandbox while Zenrows handles the scraping layer.
  </Card>

  <Card href="/integrations/hermes" title="Hermes" icon="cloud">
    Give Hermes Agent reliable, scheduled access to any website.
  </Card>

  <Card href="/integrations/openclaw" title="OpenClaw" icon="cloud">
    Connect OpenClaw's autonomous agent to Zenrows MCP for real-time web access.
  </Card>
</Columns>

## No-Code & Automation

Visual workflow builders for automating scraping without writing code.

<Columns>
  <Card href="/integrations/zapier" title="Zapier" icon="bolt">
    Automate workflows between Zenrows and 5,000+ SaaS apps.
  </Card>

  <Card href="/integrations/make" title="Make" icon="gears">
    Visual workflow builder for custom automations.
  </Card>

  <Card href="/integrations/n8n" title="n8n" icon="sitemap">
    Open-source workflow automation for self-hosted integrations.
  </Card>

  <Card href="/integrations/pipedream" title="Pipedream" icon="chart-line">
    Event-driven automation platform for APIs and data pipelines.
  </Card>

  <Card href="/integrations/mulesoft" title="MuleSoft" icon="cubes">
    Enterprise integration platform for connecting apps, data, and APIs.
  </Card>

  <Card href="/integrations/node-red" title="Node-RED" icon="route">
    Visual flow-based programming for IoT and web automation workflows.
  </Card>
</Columns>

<Tip>Integration not on the list? Chances are you can still integrate Zenrows. [Contact us](mailto:support@zenrows.com) for assistance.</Tip>

## Browser & Scraping Frameworks

Code libraries for developers building custom, code-driven scraping workflows.

<Columns>
  <Card href="/integrations/playwright" title="Playwright" icon="code">
    Modern browser automation for dynamic website scraping.
  </Card>

  <Card href="/integrations/puppeteer" title="Puppeteer" icon="chrome">
    Headless Chrome automation for advanced scraping and testing.
  </Card>

  <Card href="/integrations/scrapy" title="Scrapy" icon="spider">
    Python framework for scalable, modular scraping projects.
  </Card>

  <Card href="/integrations/selenium" title="Selenium" icon="globe">
    Browser-based automation and testing across languages.
  </Card>
</Columns>

<Card href="/integrations/undici" title="Undici" icon="plug">
  High-performance HTTP/1.1 client built for Node.js
</Card>

## Why Integrate Zenrows?

Connecting Zenrows with your existing stack unlocks its full potential. Integrations help you:

* Automate end-to-end workflows by triggering scraping jobs, transforming results, and sending data wherever you need it.
* Boost productivity by reducing manual work and connecting Zenrows to your favorite automation and workflow tools.
* Scale operations by orchestrating hundreds or thousands of scraping tasks in parallel, all managed through platforms you already know.
* Improve reliability by combining Zenrows' anti-bot capabilities with third-party automation and AI tools.

## How Integrations Work

Zenrows integrations are designed to be plug-and-play. Each integration provides:

* **Step-by-step setup guides** to get you started quickly, whether you are connecting code libraries, workflow builders, or AI platforms.
* **Code samples and templates** you can copy, customize, and deploy for your use case.
* **Best practices** for securing your API keys, handling errors, and optimizing performance.
* **Real-world examples** showing how fast-growing startups and enterprises use Zenrows integrations to drive results.

## Example Use Cases

* Monitor competitor pricing using [Playwright](/integrations/playwright) or [Puppeteer](/integrations/puppeteer) with Zenrows to extract prices daily and feed insights into Google Sheets through [Zapier](/integrations/zapier).
* Automate lead enrichment by scraping company data with Zenrows, enriching it using [Clay](/integrations/clay), and pushing it to your CRM in a single automated workflow.
* Power AI-driven research by combining Zenrows with [Lindy](/integrations/lindy) to collect, structure, and analyze data for market research or content generation.
* Scale scraping operations by orchestrating thousands of parallel scraping jobs in [n8n](/integrations/n8n) or [Make](/integrations/make).
* Create an AI-powered knowledge base by scraping industry news and documentation with Zenrows, indexing the data using [LlamaIndex](/integrations/llamaindex), and enabling instant search and automated question-answering for your team.
* Give an autonomous agent standing web access by connecting [Claude Managed Agents](/integrations/claude-managed-agents) or [OpenClaw](/integrations/openclaw) to Zenrows MCP, so it can scrape and monitor sites without you writing orchestration code.

## Getting Started

1. Select the integration that best aligns with your workflow and technical expertise.
2. Follow the integration guide for setup instructions, best practices, and troubleshooting tips.
3. Secure your credentials by using environment variables or secure vaults for API keys, such as `YOUR_ZENROWS_API_KEY`.
4. Test your workflow with a simple example, then scale up and customize as your needs grow.

## Best Practices

* Always secure your API keys and never hard-code them in public code repositories.
* Monitor your Zenrows quota at the [Analytics Page](https://app.zenrows.com/overview#performance) and integration limits to avoid interruptions.
* Use stable CSS selectors when scraping to minimize maintenance. See our [Selector Best Practices](/fetch/troubleshooting/advanced-css-selectors) for more guidance.

Zenrows integrations are built to help you move faster, automate more, and unlock the full power of web data, regardless of your stack or skill set. If you have questions, feedback, or would like to suggest a new integration, please get in touch with our team via <a href="mailto:success@zenrows.com">email</a> anytime.

## Frequently Asked Questions (FAQ)

<Accordion title="Which integration category is right for my team?">
  Choose based on your technical expertise and goals:

  * **AI Assistants / AI Code Editors**: For anyone who wants their AI assistant or editor to scrape and extract data directly, via MCP.
  * **Agent SDKs & Frameworks**: For engineers building autonomous, multi-step agents with an official vendor SDK or a framework like LangChain, LlamaIndex, Flowise, Dify, or Mastra.
  * **Agent Platforms**: For teams that want a hosted, autonomous agent product to run the loop for them.
  * **No-Code & Automation**: For operations, marketing, and growth teams automating processes without engineering resources.
  * **Browser & Scraping Frameworks**: For engineers building custom, code-driven scraping workflows.
</Accordion>

<Accordion title="Do I need coding experience to use Zenrows integrations?">
  Not necessarily. No-code integrations like Zapier, Make, n8n, and Pipedream let you build automated workflows using visual interfaces without writing code. AI assistants and agent platforms like Claude Desktop, Lindy, and OpenClaw also require no code. Developer tools like Playwright, Puppeteer, Scrapy, and Selenium, along with agent SDKs and frameworks, require programming knowledge in languages like Python, JavaScript, or others.
</Accordion>

<Accordion title="How do I secure my Zenrows API key when using integrations?">
  Always use environment variables or secure vaults to store your API keys. Never hard-code credentials like `YOUR_ZENROWS_API_KEY` directly in your code, especially in public repositories. Most integration platforms provide secure credential storage features you should use.
</Accordion>

<Accordion title="Can I use multiple integrations together?">
  Yes, you can combine multiple integrations to create powerful workflows. For example, you might use Playwright to scrape data, Zapier to process and route it, and Clay to enrich it with AI - all in a single automated pipeline.
</Accordion>

<Accordion title="What's the difference between an Agent SDK, an Agent Framework, and an Agent Platform?">
  * **Agent SDK** (Claude Agent SDK, OpenAI Agents SDK, Google ADK): an official vendor code library you use to build an agent yourself.
  * **Agent Framework** (LangChain, LlamaIndex, Flowise, Dify, Mastra): a third-party library or visual builder for the same job, not tied to one model vendor.
  * **Agent Platform** (Claude Managed Agents, Hermes, OpenClaw): a hosted product that runs the agent loop and infrastructure for you, you connect Zenrows and point it at a task.
</Accordion>

<Accordion title="What should I do if my integration stops working?">
  Follow these troubleshooting steps:

  1. Check your Zenrows quota and integration limits at the Analytics Page
  2. Verify your API keys are correctly configured and haven't expired
  3. Review the integration's error logs for specific error messages
  4. Ensure your CSS selectors are still valid (websites may have changed)
  5. Check our [Selector Best Practices](/fetch/troubleshooting/advanced-css-selectors) for guidance
  6. Contact our support team
</Accordion>

<Accordion title="Can I scale my scraping operations using these integrations?">
  Yes, integrations enable massive scaling through parallel processing. Platforms like n8n, Make, and Pipedream can orchestrate hundreds or thousands of scraping jobs simultaneously. Combine this with Zenrows' anti-bot capabilities to build enterprise-scale scraping operations.
</Accordion>


# How to Integrate Pipedream with Zenrows
Source: https://docs.zenrows.com/integrations/pipedream

Build automated web scraping workflows using Zenrows and Pipedream with no coding required. Collect product data, leads, and content updates.

Zenrows simplifies web scraping, allowing you to extract data from websites. By connecting Zenrows with Pipedream, a platform for automating workflows, you can automatically collect and process data like product prices, business leads, or content updates with no coding required!

This guide will walk you through setting up a workflow to scrape a demo e-commerce product data and save it to Google Sheets.

## What Is Pipedream?

Pipedream is a serverless platform that connects APIs and automates workflows. With its event-driven architecture, you can build workflows that respond to triggers like webhooks or schedules.

When combined with Zenrows, Pipedream enables you to automate web scraping and integrate structured data into your processes.

## Use Cases

Combining Zenrows with Pipedream creates powerful automation opportunities:

* **Market Intelligence**: Extract competitor pricing and analyze trends automatically.
* **Sales Prospecting**: Scrape business directories and enrich contact data for your CRM.
* **Content Monitoring**: Track brand mentions and trigger notifications for specific keywords.
* **Research Automation**: Collect and organize articles, reports, or blog posts.
* **Property Intelligence**: Gather real estate listings and market data for analysis.

## Real-World End-to-End Integration Example

We'll build a basic end-to-end e-commerce product scraping workflow using Pipedream and Zenrows with the following steps:

* Set up a scheduled trigger to automatically track product prices at regular intervals.
* Use `https://www.scrapingcourse.com/pagination` as the target URL.
* Utilize Zenrows' Fetch to automatically extract product names and prices from the URL.
* Process and transform the scraped data.
* Save the extracted product information to Google Sheets.

### 1. Create a new workflow on Pipedream

1. Log in to your account at [https://pipedream.com/](https://pipedream.com/).
2. Access your `Workspace` and go to `Projects` in the left sidebar.
3. Click `+ New project` at the top right and enter a name for your project (e.g., "E-commerce Product Scraper").
   <img alt="pipedream-name-project" />
4. Click `+ New Workflow` and provide a name for your workflow (e.g., "Product name and price scraper").
   <img alt="pipedream-create-workflow" />
5. Click the `Create Workflow` button.

### 2. Set up a Schedule trigger

1. Click `Add Trigger` in your new workflow.
   <img alt="pipedream-add-trigger" />
2. Search for and select `Schedule`.
   <img alt="pipedream-schedule" />
3. Choose `Custom Interval` from the trigger options.
4. Set your desired frequency (e.g., "Daily at 1:00 AM UTC").
5. Click `Save and continue`.

### 3. Configure Zenrows integration with Pipedream

To extract specific elements from a dynamic page using Zenrows with Pipedream, follow these steps:

#### Generate the cURL request from Zenrows

1. Open [Zenrows' Fetch Request Playground](https://app.zenrows.com/fetch).
2. Enter `https://www.scrapingcourse.com/pagination` as the URL to scrape.
3. Activate **JS Rendering** to handle dynamic content.
4. Set **Output type** to **Specific Data** and configure the CSS selectors under the **Parsers** tab:
   ```json theme={"dark"}
   { "name": ".product-name", "price": ".product-price" }
   ```
   <Note>The CSS selectors provided in this example (`.product-name`, `.product-price`) are specific to the page used in this guide. Selectors may vary across websites. For guidance on customizing selectors, refer to the [CSS Extractor documentation](/fetch/features/css-extractor). If you're having trouble, the [Advanced CSS Selectors Troubleshooting Guide](/fetch/troubleshooting/advanced-css-selectors) can help resolve common issues.</Note>
5. Click on the `cURL` tab on the right and copy the generated code.
   Example code:
   ```bash theme={"dark"}
   curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fpagination&js_render=true&premium_proxy=true&proxy_country=us&css_extractor=%257B%2522name%2522%253A%2522.product-name%2522%252C%2522price%2522%253A%2522.product-price%2522%257D"
   ```

#### Import the cURL into Pipedream

1. In Pipedream, add an `HTTP request` step:
   * Click the `+` button below your trigger to add a new step.
   * Search for and select `HTTP / Webhook`.
   * Choose `Build an HTTP request` from the actions.
   <img alt="pipedream-http-request" />
2. Click `Import cURL` and configure the HTTP request using the copied cURL code.
   <img alt="pipedream-configure-zenrows-request" />
3. Test the step to confirm your setup works correctly. You should receive the extracted product names and prices.

### 4. Process and transform the scraped data

1. Click the `+` button to add another step.
2. Select `Node` action and choose `Run Node code`.
3. Add the following data transformation code to format the data as an array of arrays, which will be used in the next step:
   ```javascript JavaScript theme={"dark"}
   export default defineComponent({
       async run({ steps, $ }) {
           // get the scraped data from the previous HTTP request step
           const data = steps.custom_request.$return_value;
           
           // ensure we're working with an array of products
           // if data is already an array, use it; otherwise, wrap single item in array
           const products = Array.isArray(data) ? data : [data];
           
           // transform each product into a row format for Google Sheets
           const rowValues = products.map(product => [product.name, product.price]);
           
           // return the formatted data that Google Sheets "Add Multiple Rows" expects
           return {
               rowValues
           }
       },
   })
   ```
4. Test the step to confirm the code works.
   <img alt="pipedream-code-transformation" />

<Tip>Use the **Edit with AI** feature of Pipedream to write the transformation code.</Tip>

### 5. Save results to Google Sheets

1. Click the `+` button to add a final step.
2. Search for and select `Google Sheets`.
3. Choose `Add Multiple Rows` from the Pre-built Actions.
   <img alt="pipedream-save-to-google-sheets" />
4. Click `Connect new account` to link your Google account with Pipedream.
5. Select your target Spreadsheet from the dropdown (or provide the Spreadsheet ID).
6. Choose the Worksheet where you want to store the data.
7. Configure Row Values. This field expects an array of arrays. Map the data from your processing step: `{{steps.code.$return_value.rowValues[0]}}`
8. Click `Test step` to verify that the data is correctly added to your Google Sheets.
9. Click `Deploy` in the top right corner of your workflow.

The workflow will now automatically scrape product names and prices from the e-commerce pagination page at your specified intervals and save the results to your Google Sheets.

Congratulations! You've successfully integrated Zenrows with Pipedream and automated your web scraping workflow.

## Troubleshooting

### Authentication Issues

* **Check your Zenrows API key** <br />Go to your Zenrows dashboard and copy your API key. Make sure it is pasted exactly as it appears into your workflow or tool. Even a small typo can prevent it from working.

* **Verify your subscription status** <br />Make sure your Zenrows subscription is active and that your account has enough usage. If your usage runs out, the scraping requests will stop working.
  <Tip>You can increase your usage by purchasing Top-Ups or upgrading. For details, see our [Pricing Documentation](/first-steps/pricing).</Tip>

### No Data Retrieved

* **Check if the page loads content right away** <br />Visit the page in your browser. If you don't see the data immediately, it might load a few seconds later. This means the content is dynamic.

* **Enable JavaScript rendering** <br />If the website loads content using JavaScript, you need to enable JS Rendering in Zenrows. This allows Zenrows to wait and capture the full content, just like a real browser.

* **Check your CSS selectors** <br />Right-click on the item you want to scrape (like a product name or price) and select "Inspect" to find the correct CSS selector. Make sure the selectors you are using match the content on the website.

* **Allow more time for the page to load**<br />If the content takes a few seconds to appear, try increasing the wait time in Zenrows using the wait or `wait_for` options. This gives the page more time to fully load before Zenrows tries to scrape it.

### Workflow Execution Failures

* **Look at the error logs in your tool** <br />If you are using a tool like Pipedream, check the logs to see if there is a specific error message. This can help you understand what went wrong.

* **Review each step in your workflow** <br />Make sure each step has the correct data and settings. If a step depends on information from a previous one, double-check that everything is connected properly.

* **Confirm the format of the API response** <br />Some tools expect data in a specific format. Make sure your setup is handling the response correctly, especially if you are extracting specific fields like text or prices.

## Conclusion

You've successfully learned how to integrate Zenrows with Pipedream! This powerful combination enables you to build sophisticated web scraping automations that can collect, process, and distribute data across your entire tech stack efficiently.

## Frequently Asked Questions (FAQs)

<Accordion title="How do I handle dynamic content in Zenrows?">
  Some websites load their content a few seconds after the page opens. To make sure Zenrows captures everything, turn on JavaScript rendering in your API request. This lets Zenrows wait and load the page like a real browser before scraping.
</Accordion>

<Accordion title="What CSS selectors should I use for scraping specific data?">
  CSS selectors are used to tell Zenrows what data to extract from the page. For example, if you want to collect product names, you might use `.product-name`. To find the right selector, open the page in your browser, right-click the item you want, and select "Inspect" to view its code.
</Accordion>

<Accordion title="Can I integrate Zenrows with other tools besides Google Sheets?">
  Yes. You can connect Zenrows to other platforms using tools like Pipedream or Zapier. This allows you to send scraped data to CRMs, databases, Slack, or any tool that accepts API connections.
</Accordion>

<Accordion title="How do I schedule automatic scraping at regular intervals?">
  In Pipedream, use the “Schedule” trigger to run your scraping workflow automatically. You can choose how often it runs, such as daily at a specific time (for example, every day at 1:00 AM UTC).
</Accordion>

<Accordion title="What are the limits of the Zenrows and Pipedream integration?">
  Limits depend on your Zenrows plan, such as how much usage limit you have. Pipedream also has limits on how many times workflows can run. You can check both platforms to see your current usage and available limits.
</Accordion>


# How to Integrate Playwright with Zenrows
Source: https://docs.zenrows.com/integrations/playwright

Set up Playwright with Zenrows residential proxies and Browser Sessions to bypass anti-bot detection and scrape JavaScript-rendered web pages.

Playwright is a powerful headless browser that can be integrated with Zenrows to avoid blocks and ensure smooth and reliable data extraction.

<Note>In Python, Playwright supports two variations: synchronous (great for small-scale scraping where concurrency isn't an issue) and asynchronous (recommended for projects where concurrency, scalability, and performance are essential factors)</Note>
<Warning>This tutorial focuses on the Playwright asynchronous API. So, we need to import the `async_playwright` and `asyncio` modules.</Warning>

## Use Zenrows' Proxies with Playwright to Avoid Blocks

Zenrows offers residential proxies in 190+ countries that auto-rotate the IP address for you and offer Geolocation and http/https protocols. Integrate them into Playwright to appear as a different user every time so that your chances of getting blocked are reduced exponentially.

You have three ways to get a proxy with Zenrows, one is via [Residential Proxies](/legacy/residential-proxies/introduction), where you get our proxy, and it's charged by the bandwidth; the other way is via Fetch's Premium Proxy, which is our residential proxy for the API, and you are charged by the request, depending on the params you choose; and the third is by using [Browser Sessions](/browser-sessions/get-started/playwright) where you can integrate into your code with just one line of code.
<Warning>For this tutorial, we'll focus on the Residential Proxies, the recommended Zenrows proxy for Playwright. In case you already have a running Playwright code, consider testing our [Browser Sessions](/browser-sessions/get-started/playwright).</Warning>

After logging in, you'll get redirected to the Request Playground page, then go to the [Proxies Generator](https://app.zenrows.com/configure/residential-proxies) page and create your proxy:

<Frame>
  <img alt="Zenrows Residential Proxies Generator Page" />
</Frame>

Select your Proxy Username, Proxy Country, Protocol, and Sticky TTL. Finally, copy your Proxy URL or use the cURL example at the bottom of the page.

```bash theme={"dark"}
http://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1337
```

The target site of this tutorial section will be <a href="https://httpbin.io/ip">httpbin.io/ip</a>, an endpoint that returns the origin IP of the incoming request. You'll use it to verify that Zenrows is working.

Let's assume you have set the Playwright environment with the initial script below in *Python*.

```python scraper.py theme={"dark"}
import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto("https://httpbin.io/ip")

        # Get the page content
        content = await page.content()
        print(content)

        await browser.close()

# Run the asynchronous main function
asyncio.run(main())
```

## Configure your Residential Proxy in Playwright

Make sure you have Playwright installed and in your `scraper.py` file and add the following code:

```python scraper.py theme={"dark"}
import asyncio
from playwright.async_api import async_playwright

async def main():
    proxy = {
        "server": "http://superproxy.zenrows.com:1337",
        "username": "<YOUR_ZENROWS_PROXY_USERNAME>",  # Replace with your Zenrows proxy username
        "password": "<YOUR_ZENROWS_PROXY_PASSWORD>"   # Replace with your Zenrows proxy password
    }

    async with async_playwright() as p:
        # Configure the browser to use the proxy
        browser = await p.chromium.launch(proxy=proxy, headless=True)
        page = await browser.new_page()
        await page.goto("https://httpbin.io/ip")

        # Get the page content
        content = await page.content()
        print(content)

        await browser.close()

asyncio.run(main())
```

Awesome! You just integrated Zenrows' Residential Proxies into Playwright. 🚀

## Pricing

Zenrows operates on a pay-per-success model on Fetch (that means you only pay for requests that produce the desired result); on the Residential Proxies, it's based on bandwidth use.

To optimize your scraper's success rate, fully replace Playwright with Zenrows. Different pages on the same site may have various levels of protection, but using the parameters recommended above will ensure that you are covered.

Zenrows offers a free plan (5,000 credits/month, no card required) and paid plans starting at \$19/month. For more detailed information, please refer to our [pricing page](https://www.zenrows.com/pricing).

## Frequently Asked Questions (FAQs)

<Accordion title="Why do I need a proxy for Playwright?">
  Playwright is widely recognized by websites' anti-bot systems, which can block your requests. Using residential proxies from Zenrows allows you to rotate IP addresses and appear as a legitimate user, helping to bypass these restrictions and reduce the chances of being blocked.
</Accordion>

<Accordion title="How do I know if my proxy is working?">
  You can test the proxy connection by running the script provided in the tutorial and checking the output from `httpbin.io/ip`. If the proxy is working, the response will display a different IP address than your local machine's.
</Accordion>

<Accordion title="What should I do if my requests are blocked?">
  Many websites employ advanced anti-bot measures, such as CAPTCHAs and Web Application Firewalls (WAFs), to prevent automated scraping. Simply using proxies may not be enough to bypass these protections.

  Instead of relying solely on proxies, consider using [Zenrows' Fetch](https://app.zenrows.com/fetch), which provides:

  * **JavaScript Rendering and Interaction Simulation:** Optimized with anti-bot bypass capabilities.
  * **Comprehensive Anti-Bot Toolkit:** Zenrows offers advanced tools to overcome complex anti-scraping solutions.
</Accordion>


# How to Integrate Puppeteer with Zenrows
Source: https://docs.zenrows.com/integrations/puppeteer

Configure Puppeteer with Zenrows residential proxies and Browser Sessions to bypass anti-bot systems and scrape dynamic JavaScript websites.

Puppeteer is the most popular headless browser library in JavaScript, but anti-bot solutions can easily detect and prevent it from accessing web pages. Avoid that limitation with a Zenrows Puppeteer integration!

## Use Zenrows' Proxies in Puppeteer to Avoid Blocks

Zenrows offers residential proxies in 190+ countries that auto-rotate the IP address for you and offer Geolocation and http/https protocols. Integrate them into Puppeteer to appear as a different user every time so that your chances of getting blocked are reduced exponentially.

You have three ways to get a proxy with Zenrows, one is via [Residential Proxies](/legacy/residential-proxies/introduction), where you get our proxy, and it's charged by the bandwidth; the other way is via Fetch's Premium Proxy, which is our residential proxy for the API, and you are charged by the request, depending on the params you choose; and the third is by using [Browser Sessions](/browser-sessions/get-started/puppeteer) where you can integrate into your code with just one line of code.
<Warning>For this tutorial, we'll focus on the Residential Proxies, the recommended Zenrows proxy for Puppeteer. In case you already have a running Puppeteer code, consider testing our [Browser Sessions](/browser-sessions/get-started/puppeteer).</Warning>

After logging in, you'll get redirected to the Request Playground page, then go to the [Proxies Generator](https://app.zenrows.com/configure/residential-proxies) page and create your proxy:

<Frame>
  <img alt="Zenrows Residential Proxies Generator Page" />
</Frame>

Select your Proxy Username, Proxy Country, Protocol, and Sticky TTL. Finally, copy your Proxy URL or use the cURL example at the bottom of the page.

```bash theme={"dark"}
http://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1337
```

The target site of this tutorial section will be <a href="https://httpbin.io/ip">httpbin.io/ip</a>, an endpoint that returns the origin IP of the incoming request. You'll use it to verify that Zenrows is working.

Consider the following basic Puppeteer script. It launches a headless browser, navigates to a target URL, and extracts the page's content (in this case, the IP address returned by the API endpoint).

```javascript scraper.js theme={"dark"}
import puppeteer from "puppeteer";

(async () => {
    // Launch Chrome or Chromium in headless mode
    // and open a new blank page
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    // Visit the target URL
    await page.goto("https://httpbin.io/ip");

    // Retrieve the page content and log it
    const body = await page.waitForSelector("body");
    const response = await body.getProperty("textContent");
    const jsonResponse = await response.jsonValue();
    console.log(jsonResponse);

    await browser.close();
})();
```

## Configure your Residential Proxy in Puppeteer

To use authenticated residential proxies with Puppeteer, you'll need to use Puppeteer's `--proxy-server` flag and then `page.authenticate` with username and password.

<Steps>
  <Step title="Configure the Proxy">
    To use Zenrows' premium proxy with Puppeteer, you need to anonymize the proxy URL provided by Zenrows. Replace `<YOUR_USERNAME>` and `<YOUR_PASSWORD>` with your actual Zenrows proxy credentials.

    ```javascript scraper.js theme={"dark"}
    const browser = await puppeteer.launch({
      args: [`--proxy-server=superproxy.zenrows.com:1337`],
    });
    const page = await browser.newPage();
    await page.authenticate({username: '<YOUR_USERNAME>', password: '<YOUR_PASSWORD>'});
    ```
  </Step>

  <Step title="Launch Puppeteer with the Configured Proxy">
    Use the anonymized proxy URL to launch Puppeteer:

    ```javascript scraper.js theme={"dark"}
    import puppeteer from "puppeteer";

    (async () => {
        // Launch a headless Chromium instance with Zenrows Residential Proxies configured
        const browser = await puppeteer.launch({
            args: [`--proxy-server=superproxy.zenrows.com:1337`],
        });
        const page = await browser.newPage();
        await page.authenticate({username: '<YOUR_USERNAME>', password: '<YOUR_PASSWORD>'});

        // Visit the target URL
        await page.goto("https://httpbin.io/ip");

        // Retrieve the page content and log it
        const body = await page.waitForSelector("body");
        const response = await body.getProperty("textContent");
        const jsonResponse = await response.jsonValue();
        console.log(jsonResponse);

        // Release resources
        await browser.close();
    })();
    ```
  </Step>

  <Step title="Run Your Script">
    Execute your script using Node.js:

    ```bash theme={"dark"}
    node scraper.js
    ```
  </Step>
</Steps>

## Troubleshooting

### Chrome Not Found Error

If you encounter the error `Error: Could not find Chrome`, it indicates that Puppeteer is unable to locate the Chrome or Chromium browser on your system. This issue often arises when the browser is installed in a non-standard location or if you're using a custom installation of Chrome.

To resolve this issue, explicitly specify the path to the Chrome executable in your Puppeteer configuration. You can do this by setting the `executablePath` option to the correct path where Chrome is installed on your system.

```javascript scraper.js theme={"dark"}
const browser = await puppeteer.launch({
  executablePath: '/your/path/to/chrome-or-chromium',
  args: [`--proxy-server=superproxy.zenrows.com:1337`],
});
```

Replace `/your/path/to/chrome-or-chromium` with the actual path to the Chrome executable on your machine.

### Invalid SSL Certificate

You might encounter a `net::ERR_CERT_AUTHORITY_INVALID` error due to SSL certificate issues. Configure Puppeteer to ignore SSL certificate errors by adding the following flag to the `launch` options to resolve this.

```javascript scraper.js theme={"dark"}
const browser = await puppeteer.launch({
  args: [`--proxy-server=superproxy.zenrows.com:1337`],
  ignoreHTTPSErrors: true
});
```

This allows Puppeteer to access any HTTPS page without SSL certificate issues.

### Stopped by Bot Detection with Puppeteer: CAPTCHAs, WAFs, and Beyond

Many websites employ advanced anti-bot measures like CAPTCHAs and Web Application Firewalls (WAFs) to prevent automated scraping. Simply using proxies may not be enough to bypass these protections.

Instead of relying solely on Puppeteer, consider using [Zenrows' Fetch](https://app.zenrows.com/fetch), which provides:

* **JavaScript Rendering and Interaction Simulation:** Similar to Puppeteer but optimized with anti-bot bypass capabilities.
* **Comprehensive Anti-Bot Toolkit:** Zenrows offers advanced tools to overcome complex anti-scraping solutions.

## Pricing

Zenrows operates on a bandwidth usage model on the Residential Proxies; it is pay-per-success on Fetch (that means you only pay for requests that produce the desired result).

To optimize your scraper's success rate, fully replace Puppeteer with Zenrows. Different pages on the same site may have various levels of protection, but using the parameters recommended above will ensure that you are covered.

Zenrows offers a free plan (5,000 credits/month, no card required) and paid plans starting at \$19/month. For more detailed information, please refer to our [pricing page](https://www.zenrows.com/pricing).

## Frequently Asked Questions (FAQs)

<Accordion title="Why do I need a proxy for Puppeteer?">
  Puppeteer is widely recognized by websites' anti-bot systems, which can block your requests. Using residential proxies from Zenrows allows you to rotate IP addresses and appear as a legitimate user, helping to bypass these restrictions and reduce the chances of being blocked.
</Accordion>

<Accordion title="How do I know if my proxy is working?">
  You can test the proxy connection by running the script provided in the tutorial and checking the output from `httpbin.io/ip`. If the proxy is working, the response will display a different IP address than your local machine's.
</Accordion>

<Accordion title="What should I do if my requests are blocked?">
  Many websites employ advanced anti-bot measures, such as CAPTCHAs and Web Application Firewalls (WAFs), to prevent automated scraping. Simply using proxies may not be enough to bypass these protections.

  Instead of relying solely on proxies, consider using [Zenrows' Fetch](https://app.zenrows.com/fetch), which provides:

  * **JavaScript Rendering and Interaction Simulation:** Optimized with anti-bot bypass capabilities.
  * **Comprehensive Anti-Bot Toolkit:** Zenrows offers advanced tools to overcome complex anti-scraping solutions.
</Accordion>


# How to Integrate Scrapy with Zenrows
Source: https://docs.zenrows.com/integrations/scrapy

Integrate Zenrows proxy rotation and anti-bot bypass into Scrapy spiders using middleware configuration or meta parameters for web scraping.

Scrapy is a powerful web scraping library, but anti-scraping measures can make it challenging. A Zenrows Scrapy integration can overcome these obstacles.

In this tutorial, you'll learn how to get your Zenrows proxy and integrate it with Scrapy using two methods: via Meta Parameter and Custom Middleware.

## Use Zenrows' Proxies with Scrapy to Avoid Blocks

Zenrows offers premium proxies in 190+ countries that auto-rotate the IP address for you, as well as the `User-Agent` header with [Fetch](/fetch/api-reference). Integrate them into Scrapy to appear as a different user every time so that your chances of getting blocked are reduced exponentially.

Zenrows provides two options for integrating proxies with Scrapy:

1. **Residential Proxies:** With Residential Proxies, you can directly access our dedicated proxy network, billed by bandwidth usage. This option is ideal if you need flexible, on-demand proxy access.

2. **Fetch with Zenrows Middleware:** Our Fetch is optimized for high-demand scraping scenarios and billed per request based on chosen parameters. Using <a href="https://pypi.org/project/scrapy-zenrows/">Zenrows Middleware for Scrapy</a> allows you to seamlessly connect your Scrapy project to Fetch, automatically routing requests through the Premium Proxy and handling API-specific configurations.

<Note>In this tutorial, we'll focus on using Fetch with the Zenrows Middleware, the recommended setup for seamless Scrapy integration.</Note>

Let's assume you have set the Scrapy environment with the initial script below.

```python scraper.py theme={"dark"}
import scrapy

class ScraperSpider(scrapy.Spider):
    name = "scraper"
    allowed_domains = ["httpbin.io"]
    start_urls = ["https://httpbin.io/ip"]

    def parse(self, response):
        pass
```

Follow the steps below to integrate Zenrows proxies into this scraper!

## Integrate the Zenrows Middleware into Scrapy!

The Zenrows Middleware for Scrapy allows seamless integration of the Zenrows Fetch into Scrapy projects. This middleware helps you manage proxy settings, enable advanced features like JavaScript rendering, and apply custom headers and cookies.

### Installation

First, install the `scrapy-zenrows` package, which provides the necessary middleware for integrating Zenrows with Scrapy.

```bash theme={"dark"}
pip install scrapy-zenrows
```

### Usage

To use the Zenrows Fetch with Scrapy, [sign in on Zenrows](https://www.zenrows.com/) to obtain your API key. The API key allows you to access the Premium Proxy, JavaScript rendering, and other advanced scraping features.

### Setting Up Global Middleware

To enable Zenrows as the default proxy across all Scrapy requests, add Zenrows Middleware to your project's `settings.py` file. This setup configures your Scrapy spiders to use the Zenrows API for every request automatically.

```python settings.py theme={"dark"}
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsMiddleware": 543,  # Add Zenrows Middleware
}

# Zenrows API Key
ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"
```

### Enabling Premium Proxy and JavaScript Rendering

Zenrows offers [Premium Proxy](/fetch/features/premium-proxy) and [JavaScript rendering](/fetch/features/js-rendering) features, which are essential for handling websites that require complex interactions or are protected by anti-bot systems. To enable these features for all requests, configure them in `settings.py`:

```python settings.py theme={"dark"}
# ...

USE_ZENROWS_PREMIUM_PROXY = True  # Enable Premium Proxy for all requests (default is False)
USE_ZENROWS_JS_RENDER = True      # Enable JavaScript rendering for all requests (default is False)
```

<Note>By default, both features are disabled to keep requests lean and cost-effective.</Note>

### Customizing Zenrows Middleware for Specific Requests

In scenarios where you don't need Premium Proxy or JavaScript rendering for every request (e.g., for only certain pages or spiders), you can override global settings and apply these features only to specific requests. This is done using the `ZenRowsRequest` class, which provides a flexible way to configure Zenrows on a per-request basis.

```python scraper.py theme={"dark"}
from scrapy_zenrows import ZenRowsRequest

class YourSpider(scrapy.Spider):
    name = "your_spider"
    start_urls = ["https://httpbin.io/ip"]

    def start_requests(self):
        # Use ZenRowsRequest to customize settings per request
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                params={
                    "js_render": "true",       # Enable JavaScript rendering for this request
                    "premium_proxy": "true",   # Enable Premium Proxy for this request
                },
            )
```

In this example, ZenRowsRequest is configured with js\_render and premium\_proxy set to true, ensuring that only this specific request uses JavaScript rendering and Premium Proxy.

### Using Additional Request Parameters

The `ZenRowsRequest` function supports several other parameters, allowing you to customize each request to meet specific requirements. Here are some useful parameters:

* `proxy_country`: Specifies the country for the proxy, useful for geo-targeting.
* `js_instructions`: Allows custom JavaScript actions on the page, such as waiting for elements to load.
* `autoparse`: Automatically extracts data from supported websites.
* `outputs`: Extracts specific content types like tables, images, or links.
* `css_extractor`: Allows CSS-based content extraction.

Here's an example of using these advanced parameters:

```python scraper.py theme={"dark"}
class YourSpider(scrapy.Spider):
    name = "your_spider"
    start_urls = ["https://httpbin.io/ip"]

    def start_requests(self):
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                params={
                    "js_render": "true",       # Enable JavaScript rendering for this request
                    "premium_proxy": "true",   # Enable Premium Proxy for this request
                    "proxy_country": "ca",     # Use a proxy from Canada
                    "js_instructions": '[{"wait": 500}]',    # Wait 500ms after page load
                    "autoparse": "true",                     # Enable automatic parsing
                    "outputs": "tables",                     # Extract tables from the page
                    "css_extractor": '{"links":"a @href","images":"img @src"}'  # Extract links and images
                },
            )
```

<Tip>Refer to the [Zenrows Fetch documentation](/fetch/api-reference) for a complete list of supported parameters.</Tip>

### Customizing Headers with Zenrows

Certain websites require specific headers (such as `Referer` or `Origin`) for successful scraping. Zenrows Middleware allows you to set custom headers on a per-request basis. When using custom headers, set the `custom_headers` parameter to `"true"` so that Zenrows includes your headers while managing essential browser headers on its end.

Here's an example of setting a custom Referer header:

```python scraper.py theme={"dark"}
class YourSpider(scrapy.Spider):
    name = "your_spider"
    start_urls = ["https://httpbin.io/anything"]

    def start_requests(self):
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                params={
                    "custom_headers": "true",  # Enable custom headers for this request
                },
                headers={
                    "Referer": "https://www.google.com/",  # Set a custom Referer header
                },
            )
```

For `cookies` add them to the cookies dictionary in the request's meta parameter. Just as with custom headers, `custom_headers` must be set to `"true"` for Zenrows to allow custom cookies. This is particularly useful for handling sessions or accessing region-specific content.

```python scraper.py theme={"dark"}
class YourSpider(scrapy.Spider):
    name = "your_spider"
    start_urls = ["https://httpbin.io/anything"]

    def start_requests(self):
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                params={
                    "custom_headers": "true",  # Allow custom cookies
                },
                cookies={
                    "currency": "USD",
                    "country": "UY",
                },
            )
```

<Tip>Cookies are often required to maintain user sessions or comply with location-based content restrictions. For more information on cookies and headers, see [Zenrows headers feature documentation](/fetch/features/headers).</Tip>

## Pricing

Zenrows operates on a pay-per-success model on Fetch (that means you only pay for requests that produce the desired result); on the Residential Proxies, it's based on bandwidth use.

To optimize your scraper's success rate, fully replace Scrapy with Zenrows. Different pages on the same site may have various levels of protection, but using the parameters recommended above will get you covered.

Zenrows offers a free plan (5,000 credits/month, no card required) and paid plans starting at \$19/month. For more detailed information, please refer to our [pricing page](https://www.zenrows.com/pricing).

## Troubleshooting Guide

Even with Zenrows handling most scraping challenges, you might encounter issues. Here's how to diagnose and resolve common problems:

### Anti-Bot Detection Issues

#### Problem: Content doesn't match what you see in browser

**Solutions:**

1. **Enable JavaScript rendering**: Some sites load content dynamically

```python theme={"dark"}
# Enable JavaScript rendering for a specific request
yield ZenRowsRequest(
    url=url,
    params={
        "js_render": "true",
    },
)
```

2. **Check if Premium Proxies are needed**: Some sites may block datacenter IPs

```python theme={"dark"}
# Enable Premium Proxies for a specific request
yield ZenRowsRequest(
    url=url,
    params={
        "premium_proxy": "true",
    },
)
```

3. **Use custom headers to appear more like a real browser**: add a valid referer like Google or Bing

```python theme={"dark"}
yield ZenRowsRequest(
    url=url,
    params={
        "custom_headers": "true",
    },
    headers={
        "Referer": "https://www.google.com/",
    },
)
```

#### Problem: Getting redirected to CAPTCHA or security pages

**Solution:**

1. **Use full browser emulation with JS rendering and Premium Proxies**:

```python theme={"dark"}
yield ZenRowsRequest(
    url=url,
    params={
        "js_render": "true",
        "premium_proxy": "true",
        "wait": "3000",  # Wait 3 seconds for page to load fully
    },
)
```

2. **Try different geographic locations**:

```python theme={"dark"}
# Try accessing from a different country
yield ZenRowsRequest(
    url=url,
    params={
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "ca",  # Canada
    },
)
```

## Frequently Asked Questions (FAQs)

<Accordion title="Why do I need a proxy for Scrapy?">
  Scrapy is widely recognized by websites' anti-bot systems, which can block your requests. Using residential proxies from Zenrows allows you to rotate IP addresses and appear as a legitimate user, helping to bypass these restrictions and reduce the chances of being blocked.
</Accordion>

<Accordion title="Do you have any code examples?">
  Yes! You can find code examples demonstrating how to use the `scrapy_zenrows` middleware <a href="https://github.com/ZenRows/scrapy-zenrows-middleware/tree/main/examples/examples/examples/spiders">here</a>!
</Accordion>

<Accordion title="How do I know if my proxy is working?">
  You can test the proxy connection by running the script provided in the tutorial and checking the output from `httpbin.io/ip`. If the proxy is working, the response will display a different IP address than your local machine's.
</Accordion>

<Accordion title="What should I do if my requests are blocked?">
  Many websites employ advanced anti-bot measures, such as CAPTCHAs and Web Application Firewalls (WAFs), to prevent automated scraping. Simply using proxies may not be enough to bypass these protections.

  Instead of relying solely on proxies, consider using [Zenrows' Fetch](https://app.zenrows.com/fetch), which provides:

  * **JavaScript Rendering and Interaction Simulation:** Optimized with anti-bot bypass capabilities.
  * **Comprehensive Anti-Bot Toolkit:** Zenrows offers advanced tools to overcome complex anti-scraping solutions.
</Accordion>


# Selenium Integration with Zenrows
Source: https://docs.zenrows.com/integrations/selenium

Configure Zenrows residential proxies with Selenium WebDriver to bypass anti-bot detection and IP blocks in automated browser scraping tasks.

Selenium is one of the most popular headless browser automation libraries. However, its widespread use has made it a prime target for anti-scraping technologies, which can identify and block its requests.

To avoid this, you can integrate your Selenium setup with premium proxies, which will help you bypass these blocks and maintain successful web scraping.

<Note>The code snippets below are in Python, but the instructions apply to any programming language. Additionally, this tutorial is relevant to all tools that utilize Selenium as a base, such as Selenium Wire, Selenium Stealth, and Undetected ChromeDriver.</Note>

## Integrate Selenium with Zenrows' Proxies to Avoid Getting Blocked<span />

Zenrows offers residential proxies in 190+ countries that auto-rotate the IP address for you and offer Geolocation and http/https protocols. Integrate them into Selenium to appear as a different user every time so that your chances of getting blocked are reduced exponentially.

You have two ways to get a proxy with Zenrows, one is via [Residential Proxies](/legacy/residential-proxies/introduction), where you get our proxy, and it's charged by the bandwidth; the other way is via Fetch's Premium Proxy, which is our residential proxy for the API, and you are charged by the request, depending on the params you choose.
<Warning>For this tutorial, we'll focus on the Residential Proxies, the recommended Zenrows proxy for Selenium</Warning>

After logging in, you'll get redirected to the Request Playground page, then go to the [Proxies Generator](https://app.zenrows.com/configure/residential-proxies) page and create your proxy:

<Frame>
  <img alt="Zenrows Residential Proxies Generator Page" />
</Frame>

Select your Proxy Username, Proxy Country, Protocol, and Sticky TTL. Finally, copy your Proxy URL or use the cURL example at the bottom of the page.

```bash theme={"dark"}
http://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1337
```

The target site of this tutorial section will be <a href="https://httpbin.io/ip">httpbin.io/ip</a>, an endpoint that returns the origin IP of the incoming request. You'll use it to verify that Zenrows is working.

Consider you have the following Python script using Selenium:

```python scraper.py theme={"dark"}
# installed selenium 4.11+
from selenium import webdriver
from selenium.webdriver.chrome.service import Service

from selenium.webdriver.common.by import By

# initialize a controllable Chrome instance
# in headless mode
service = Service()
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(service=service, options=options)

# visit the target site in the browser
driver.get("https://httpbin.io/ip")

# get the page content and print it
response = driver.find_element(By.TAG_NAME, "body").text
print(response)

driver.quit()
```

## Configure your Residential Proxy in Selenium

To use an authenticated proxy in Selenium, you can utilize the <a href="https://pypi.org/project/selenium-wire/">selenium-wire</a> library, as Chrome does not natively support authenticated proxies.

Selenium Wire extends Selenium's functionality, allowing you to easily configure and use proxies in your scraping projects.

<Steps>
  <Step title="Install Selenium Wire">
    First, install Selenium Wire:

    ```bash theme={"dark"}
    pip install selenium-wire
    ```
  </Step>

  <Step title="Configure the Residential Proxy in Selenium">
    With Selenium Wire installed, you can configure the proxy at the browser level in Selenium. Below is an example configuration where you should replace `<YOUR_USERNAME>` and `<YOUR_PASSWORD>` with the actual credentials.

    ```python scraper.py theme={"dark"}
    from seleniumwire import webdriver
    from selenium.webdriver.chrome.service import Service
    from selenium.webdriver.common.by import By

    # Initialize a controllable Chrome instance in headless mode
    service = Service()
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")

    # Configure the residential proxy
    proxy_url = "http://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1337"
    seleniumwire_options = {
        "proxy": {
            "http": f"{proxy_url}",
            "https": f"{proxy_url}",
        },
    }

    # Initialize the Chrome driver with the proxy settings
    driver = webdriver.Chrome(
        service=service,
        seleniumwire_options=seleniumwire_options,
        options=options
    )

    # Visit the target site through the proxy and retrieve the IP address
    driver.get("https://httpbin.io/ip")

    # Get the page content and print it to verify the IP rotation
    response = driver.find_element(By.TAG_NAME, "body").text
    print(response)

    driver.quit()
    ```
  </Step>

  <Step title="Run Your Script">
    Execute your script using Python:

    ```bash theme={"dark"}
    python scraper.py
    ```
  </Step>
</Steps>

Congrats! You just saw how Zenrows enables you to overcome the limitations of Selenium. 🥳

## Troubleshooting

### Chromedriver Version Incompatibility or Chrome Not Found Error

If you encounter an error like `chromedriver version detected in PATH might not be compatible with the detected Chrome version` or `session not created from unknown error: no chrome binary at /usr/bin/google-chrome`, this indicates that the version of Chromedriver in your system's PATH is incompatible with the installed version of Chrome. Additionally, this error may also occur if the Chrome binary is not found at the expected location.

To resolve this issue, explicitly specify the path to the Chrome executable in your Selenium configuration. You can do this by setting the `executable_path` option to the correct path where Chrome is installed on your system.

```python scraper.py theme={"dark"}
service = Service(executable_path="/your/path/to/chrome-or-chromium")
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
```

### Invalid SSL Certificate

You might encounter an `ERR_CERT_AUTHORITY_INVALID` error due to SSL certificate issues. These errors can prevent your scraper from accessing the content you need.

Instruct Chrome to ignore SSL/certificate errors by setting the `accept_insecure_certs` option to `True`.

```python scraper.py theme={"dark"}
options = webdriver.ChromeOptions()
options.accept_insecure_certs = True
```

This setting allows Selenium to access the page's content without being blocked by SSL certificate errors.

### Stopped by Bot Detection with Selenium: CAPTCHAs, WAFs, and Beyond

Many websites employ advanced anti-bot measures like CAPTCHAs and Web Application Firewalls (WAFs) to prevent automated scraping. Simply using proxies may not be enough to bypass these protections.

Instead of relying solely on Selenium, consider using [Zenrows' Fetch](https://app.zenrows.com/fetch), which provides:

* **JavaScript Rendering and Interaction Simulation:** Similar to Selenium but optimized with anti-bot bypass capabilities.
* **Comprehensive Anti-Bot Toolkit:** Zenrows offers advanced tools to overcome complex anti-scraping solutions.

## Pricing

Zenrows operates on a bandwidth usage model on the Residential Proxies; it is pay-per-success on Fetch (that means you only pay for requests that produce the desired result).

To optimize your scraper's success rate, fully replace Selenium with Zenrows. Different pages on the same site may have various levels of protection, but using the parameters recommended above will ensure that you are covered.

Zenrows offers a free plan (5,000 credits/month, no card required) and paid plans starting at \$19/month. For more detailed information, please refer to our [pricing page](https://www.zenrows.com/pricing).

## Frequently Asked Questions (FAQs)

<Accordion title="Why do I need a proxy for Selenium?">
  Selenium is widely recognized by websites' anti-bot systems, which can block your requests. Using residential proxies from Zenrows allows you to rotate IP addresses and appear as a legitimate user, helping to bypass these restrictions and reduce the chances of being blocked.
</Accordion>

<Accordion title="How do I know if my proxy is working?">
  You can test the proxy connection by running the script provided in the tutorial and checking the output from `httpbin.io/ip`. If the proxy is working, the response will display a different IP address than your local machine's.
</Accordion>

<Accordion title="What should I do if my requests are blocked?">
  Many websites employ advanced anti-bot measures, such as CAPTCHAs and Web Application Firewalls (WAFs), to prevent automated scraping. Simply using proxies may not be enough to bypass these protections.

  Instead of relying solely on proxies, consider using [Zenrows' Fetch](https://app.zenrows.com/fetch), which provides:

  * **JavaScript Rendering and Interaction Simulation:** Optimized with anti-bot bypass capabilities.
  * **Comprehensive Anti-Bot Toolkit:** Zenrows offers advanced tools to overcome complex anti-scraping solutions.
</Accordion>


# How to Integrate Undici with Zenrows
Source: https://docs.zenrows.com/integrations/undici

Use Undici HTTP client with Zenrows Fetch to build high-performance Node.js web scrapers with anti-bot bypass and proxy support.

Extract web data with enterprise-grade performance using Undici's HTTP/1.1 client and Zenrows' Fetch. This guide demonstrates how to leverage Undici's superior performance with Zenrows' robust scraping infrastructure to build high-speed web scraping applications.

## What Is Undici?

<a href="https://github.com/nodejs/undici">Undici</a> is a fast, reliable HTTP/1.1 client written from scratch for Node.js. It provides significant performance improvements over traditional HTTP clients, such as Axios and node-fetch, making it ideal for high-throughput scraping applications. Undici supports advanced features like HTTP/1.1 pipelining, connection pooling, and multiple request methods.

### Key Benefits of Integrating Undici with Zenrows

The undici-zenrows integration brings the following advantages:

* **Superior Performance**: Undici delivers up to 3x better performance compared to traditional HTTP clients, making your scraping operations faster and more efficient.
* **Advanced Connection Management**: Built-in connection pooling and HTTP/1.1 pipelining capabilities optimize resource usage for high-volume scraping.
* **Enterprise-Grade Scraping Reliability**: Combine Undici's robust HTTP handling with Zenrows' scraping infrastructure.
* **Multiple Request Methods**: Choose from `request`, `fetch`, `stream`, or `pipeline` <a href="https://github.com/nodejs/undici?tab=readme-ov-file#common-api-methods">API methods</a> based on your specific <a href="https://github.com/nodejs/undici?tab=readme-ov-file#performance-comparison">use case requirements</a>.
* **Memory Efficient**: Stream-based processing reduces memory overhead for large-scale scraping operations.
* **TypeScript Support**: Full TypeScript support for a better development experience and type safety.

## Getting Started: Basic Usage

Let's start with a simple example that uses Undici to scrape the <a href="https://www.scrapingcourse.com/antibot-challenge">Antibot Challenge</a> page through Zenrows' Fetch.

### Step 1: Install Undici

Install the `undici` package using npm:

```bash theme={"dark"}
npm install undici
```

### Step 2: Set Up Your Project

You'll need your Zenrows API key, which you can get from the <a href="https://app.zenrows.com/fetch">Request Playground dashboard</a>.

<img alt="Zenrows Request Playground" />

<Note>
  You can explore different parameter configurations using the [Request Playground dashboard](https://app.zenrows.com/fetch), then apply those settings in your Undici implementation. For a complete list of available parameters and their descriptions, refer to the [API Reference documentation](/fetch/api-reference).
</Note>

Create a new file and import the necessary modules.

```javascript Node.js theme={"dark"}
// npm install undici
import { request } from "undici";

// set your Zenrows API key
const ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY";
const targetUrl = "https://www.scrapingcourse.com/antibot-challenge";
```

### Step 3: Make Your First Request

Use Undici's `request` method to call Zenrows' Fetch:

```javascript Node.js theme={"dark"}
// npm install undici
import { request } from "undici";

// set your Zenrows API key
const ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY";
const targetUrl = "https://www.scrapingcourse.com/antibot-challenge";

async function scrapeWithUndici() {
  try {
    const { body } = await request("https://api.zenrows.com/v1/", {
      method: "GET",
      query: {
        url: targetUrl,
        apikey: ZENROWS_API_KEY,
        js_render: "true",
        premium_proxy: "true",
      },
    });

    // get the HTML content
    const htmlContent = await body.text();
    console.log(htmlContent);
  } catch (error) {
    console.error("Scraping failed:", error.message);
  }
}

// execute the scraper
scrapeWithUndici();
```

<Tip>
  Understanding the Parameters

  In the code above, we're using two key parameters to handle the antibot:

  `js_render: "true"`: Activates JavaScript execution to handle dynamic content and render the page completely

  `premium_proxy: "true"`: Routes requests through high-quality residential IP addresses for protected access

  These parameters work together for reliable access to sites with sophisticated anti-bot protection. For a comprehensive list of all available parameters and their usage, check the parameter overview table.
</Tip>

Replace `YOUR_ZENROWS_API_KEY` with your actual API key and run the script:

```bash theme={"dark"}
node scraper.js
```

The script will successfully bypass the antibot and return the HTML content:

```html Output theme={"dark"}
<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>
```

Perfect! You've successfully integrated Undici with Zenrows to bypass antibot protection.

## Complete Example: E-commerce Product Scraper

Now let's build a complete scraper for an e-commerce site to demonstrate practical use cases with Undici and Zenrows.

### Step 1: Scrape the E-commerce Site

Let's start by scraping an <a href="https://www.scrapingcourse.com/ecommerce/">e-commerce demo</a> site to see how Undici handles real-world scenarios. We'll extract the complete HTML content of the page:

```javascript Node.js theme={"dark"}
// npm install undici
import { request } from "undici";

// set your Zenrows API key
const ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY";
const targetUrl = "https://www.scrapingcourse.com/ecommerce/";

async function scrapeEcommerceSite() {
  try {
    const { body } = await request("https://api.zenrows.com/v1/", {
      method: "GET",
      query: {
        url: targetUrl,
        apikey: ZENROWS_API_KEY,
      },
    });

    // get the HTML content
    const htmlContent = await body.text();
    console.log(htmlContent);
  } catch (error) {
    console.error("Scraping failed:", error.message);
  }
}

// execute the scraper
scrapeEcommerceSite();
```

This will return the complete HTML content of the e-commerce page.

```html Output theme={"dark"}
<!DOCTYPE html>
<html lang="en-US">
<head>
    <!--- ... --->
  
    <title>Ecommerce Test Site to Learn Web Scraping - ScrapingCourse.com</title>
    
  <!--- ... --->
</head>
<body class="home archive ...">
    <p class="woocommerce-result-count">Showing 1-16 of 188 results</p>
    <ul class="products columns-4">

        <!--- ... --->

    </ul>
</body>
</html>
```

### Step 2: Parse the Scraped Data

Now that you can successfully scrape the site, let's extract specific product information using CSS selectors. We'll use Zenrows' [`css_extractor`](/fetch/features/css-extractor) parameter to get structured data:

```javascript Node.js theme={"dark"}
// npm install undici
import { request } from "undici";

// set your Zenrows API key
const ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY";
const targetUrl = "https://www.scrapingcourse.com/ecommerce/";

async function scrapeEcommerceSite() {
  try {
    const { body } = await request("https://api.zenrows.com/v1/", {
      method: "GET",
      query: {
        url: targetUrl,
        apikey: ZENROWS_API_KEY,
        css_extractor: JSON.stringify({
          "product-names": ".product-name",
        }),
      },
    });

    // get the product data
    const productData = await body.json();
    console.log(productData);
  } catch (error) {
    console.error("Scraping failed:", error.message);
  }
}

// execute the scraper
scrapeEcommerceSite();
```

The response will be a JSON object containing the extracted product names:

```html Output theme={"dark"}
{
  "product-names": [
    "Abominable Hoodie",
    "Adrienne Trek Jacket",
    // ...
    "Ariel Roll Sleeve Sweatshirt",
    "Artemis Running Short"
  ]
}
```

### Step 3: Export Data to CSV

Finally, save the scraped product data to a CSV file for further analysis:

```javascript Node.js theme={"dark"}
// npm install undici
import { request } from "undici";
import { writeFileSync } from "fs";

// set your Zenrows API key
const ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY";
const targetUrl = "https://www.scrapingcourse.com/ecommerce/";

async function scrapeEcommerceSite() {
  try {
    const { body } = await request("https://api.zenrows.com/v1/", {
      method: "GET",
      query: {
        url: targetUrl,
        apikey: ZENROWS_API_KEY,
        css_extractor: JSON.stringify({
          "product-names": ".product-name",
        }),
      },
    });

    // get the product data
    const productData = await body.json();
    const productNames = productData["product-names"];

    // create CSV content
    let csvContent = "Product Name\n";
    productNames.forEach((name) => {
      csvContent += `"${name}"\n`;
    });

    // write to CSV file
    writeFileSync("products.csv", csvContent, "utf8");
    console.log(
      `Successfully exported ${productNames.length} products to products.csv`
    );
  } catch (error) {
    console.error("Scraping failed:", error.message);
  }
}

// execute the scraper
scrapeEcommerceSite();
```

Run the script and you'll get a CSV file containing all the product names:

<img alt="e_commerce demo csv" />

Congratulations! You now have a complete working scraper that can extract product data from an e-commerce site and export it to a CSV file.

## Next Steps

You now have a complete foundation for high-performance web scraping with Undici and Zenrows. Here are some recommended next steps to optimize your scraping operations:

* <a href="https://github.com/nodejs/undici">Undici Documentation</a>: Explore Undici's advanced features like HTTP/1.1 pipelining, custom dispatchers, and performance optimizations.
* [Complete API Reference](/fetch/api-reference): Explore all available Zenrows parameters and advanced configuration options to customize your Undici requests for specific use cases.
* [JavaScript Instructions Guide](/fetch/features/js-instructions): Learn how to perform complex page interactions like form submissions, infinite scrolling, and multi-step workflows using Zenrows' browser automation.
* [Output Formats and Data Extraction](/fetch/features/markdown): Master advanced data extraction with CSS selectors, convert responses to Markdown or PDF, and capture screenshots using Undici's streaming capabilities.

## Getting Help

Request failures can happen for various reasons when using Undici with Zenrows. For detailed [troubleshooting guidance](/fetch/troubleshooting/troubleshooting-guide), visit our comprehensive troubleshooting guide and check <a href="https://github.com/nodejs/undici">Undici's documentation</a> for HTTP client-specific issues.

If you're still facing issues despite following the troubleshooting tips, our support team is available to help you. Use the chatbox in the [Request Playground](https://app.zenrows.com/fetch) dashboard or <a href="mailto:success@zenrows.com">contact us via email</a> to get personalized help from Zenrows experts.

When contacting support, always include the `X-Request-Id` from your [response headers](/fetch/api-reference#response-headers) to help us diagnose issues quickly.

## Frequently Asked Questions (FAQ)

<Accordion title="How do I handle websites that block my Undici requests?">
  Enable both `js_render` and `premium_proxy` parameters in your Zenrows API calls. This combination offers the highest success rate against sophisticated antibot protection by simulating real browser behavior and utilizing high-quality residential IP addresses.
</Accordion>

<Accordion title="How do I handle JavaScript-heavy websites with Undici and Zenrows?">
  Enable the `js_render` parameter in your Zenrows API calls. This uses a real browser to execute JavaScript and capture the fully rendered page. Combine with Undici's efficient request handling for optimal performance on modern web applications.
</Accordion>

<Accordion title="Can I use Undici's streaming capabilities with Zenrows?">
  Absolutely! Undici's `stream()` and `pipeline()` methods work perfectly with Zenrows. This is especially useful for processing large responses efficiently without loading everything into memory.
</Accordion>

<Accordion title="What's the difference between undici.request() and undici.fetch() with Zenrows?">
  `undici.request()` provides better performance and more control over the request/response lifecycle, while `undici.fetch()` offers a more familiar API similar to browser fetch. For maximum performance with Zenrows, use `undici.request()`.
</Accordion>

<Accordion title="How do I extract specific data from pages using Undici and Zenrows?">
  Use the `css_extractor` parameter in your Zenrows API calls to extract content using CSS selectors directly. The response will be JSON instead of HTML, making it easier to process with Undici.
</Accordion>

<Accordion title="Can I use Undici with Zenrows for real-time monitoring?">
  Yes! Undici's performance characteristics make it excellent for real-time monitoring. Use session management with Zenrows' `session_id` parameter and implement efficient polling with Undici's connection pooling.
</Accordion>

<Accordion title="Does Undici work with Zenrows' advanced features like screenshots and PDF outputs?">
  Yes! All Zenrows features work with Undici.
</Accordion>


# Zenrows MCP Integration for VS Code
Source: https://docs.zenrows.com/integrations/vscode

Learn how to configure the Zenrows MCP server in VS Code with GitHub Copilot to scrape web pages and extract data from your editor.

## Before You Start

You need:

<Columns>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">A **Zenrows account**</Card>
  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">Your **Zenrows API key**</Card>
</Columns>

<Card icon="code" href="https://nodejs.org/en/download/" title="Node.js">Latest version of **Node.js** installed on your machine</Card>
<Card icon="robot" href="https://code.visualstudio.com" title="VS Code">**VS Code** installed</Card>

## Setup

<Steps>
  <Step title="Step 1: Open the configuration file">
    Open or create the MCP configuration file in your project:

    ```
    .vscode/mcp.json
    ```
  </Step>

  <Step title="Step 2: Add the Zenrows MCP server">
    Paste the following into the configuration file, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

    ```json theme={"dark"}
    {
    "servers": {
        "zenrows": {
            "type": "stdio",
            "command": "npx",
            "args": ["-y", "@zenrows/mcp"],
            "env": {
                "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
                }
            }
        }
    }
    ```
  </Step>

  <Step title="Step 3: Reload the window">
    Save the file and reload the VS Code window (**Cmd+Shift+P** / **Ctrl+Shift+P**, then **Developer: Reload Window**) to load the new configuration.
  </Step>
</Steps>

## Start scraping

Ask GitHub Copilot to scrape a webpage or automate a browser workflow in plain English:

```
Get the product names and prices from https://www.scrapingcourse.com/ecommerce/.
```

```
Scrape https://www.scrapingcourse.com/antibot-bypass using Premium Proxies.
```

```
Fetch https://www.scrapingcourse.com/javascript-rendering/ with JavaScript rendering enabled and return the page as Markdown.
```

```
Open https://scrapingcourse.com/login in a browser, fill in the credentials, click submit, then extract the account details.
```

## Troubleshooting

**The `scrape` tool is not available**

Confirm `.vscode/mcp.json` is present in your project root and contains valid JSON. You can validate the JSON syntax at <a href="https://jsonlint.com">jsonlint.com</a>. Then reload the VS Code window.

**Page content is missing or incomplete**

The page likely loads content dynamically. Ask the assistant to enable JavaScript rendering:

```
Scrape https://www.scrapingcourse.com/javascript-rendering/ with `js_render: true`.
```

**Requests return a 403 or bot detection page**

Ask the assistant to use Premium Proxies:

```
Scrape https://www.scrapingcourse.com/antibot-bypass using `premium_proxy: true`.
```

For more troubleshooting options, see the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview#troubleshooting).


# Zenrows MCP Integration for Windsurf
Source: https://docs.zenrows.com/integrations/windsurf

Integrate the Zenrows MCP server with Windsurf to enable AI-powered web scraping and data extraction from your coding environment.

## Before You Start

You need:

<Columns>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">A **Zenrows account**</Card>
  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">Your **Zenrows API key**</Card>
</Columns>

<Card icon="code" href="https://nodejs.org/en/download/" title="Node.js">Latest version of **Node.js** installed on your machine</Card>
<Card icon="robot" href="https://windsurf.com" title="Windsurf">**Windsurf** installed</Card>

## Setup

<Steps>
  <Step title="Step 1: Open the configuration file">
    Open the Windsurf MCP configuration file:

    ```
    ~/.codeium/windsurf/mcp_config.json
    ```

    If the file does not exist yet, create it.
  </Step>

  <Step title="Step 2: Add the Zenrows MCP server">
    Paste the following into the configuration file, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

    ```json theme={"dark"}
    {
    "mcpServers": {
        "zenrows": {
            "command": "npx",
            "args": ["-y", "@zenrows/mcp"],
            "env": {
                "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
                }
            }
        }
    }
    ```
  </Step>

  <Step title="Step 3: Restart Windsurf">
    Save the file and restart Windsurf to load the new configuration.
  </Step>
</Steps>

## Start scraping

Ask Windsurf's AI assistant to scrape a webpage or automate a browser workflow in plain English:

```
Get the product names and prices from https://www.scrapingcourse.com/ecommerce/.
```

```
Scrape https://www.scrapingcourse.com/antibot-bypass using Premium Proxies.
```

```
Fetch https://www.scrapingcourse.com/javascript-rendering/ with JavaScript rendering enabled and return the page as Markdown.
```

```
Open https://scrapingcourse.com/login in a browser, fill in the credentials, click submit, then extract the account details.
```

## Troubleshooting

**The `scrape` tool is not available**

Confirm the configuration file is saved at `~/.codeium/windsurf/mcp_config.json` and contains valid JSON. You can validate the JSON syntax at <a href="https://jsonlint.com">jsonlint.com</a>. Then restart Windsurf.

**Page content is missing or incomplete**

The page likely loads content dynamically. Ask the assistant to enable JavaScript rendering:

```
Scrape https://www.scrapingcourse.com/javascript-rendering/ with `js_render: true`.
```

**Requests return a 403 or bot detection page**

Ask the assistant to use Premium Proxies:

```
Scrape https://www.scrapingcourse.com/antibot-bypass using `premium_proxy: true`.
```

For more troubleshooting options, see the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview#troubleshooting).


# How to Integrate Zapier With Zenrows
Source: https://docs.zenrows.com/integrations/zapier

Set up Zenrows and Zapier integration to automate web scraping tasks across your apps. Build no-code workflows for data extraction and syncing.

Zenrows integrates seamlessly with Zapier to automate scraping tasks across several applications. This guide will walk you through a practical example, showing you how to integrate Zapier with Zenrows to efficiently automate your web scraping tasks.

## What Is Zapier & Why Integrate It with Zenrows?

Zapier is a no-code platform that automates repetitive tasks by linking applications into workflows. Each workflow consists of a trigger and one or more actions. Triggers can be scheduled, event-based, or manually initiated using webhooks.

When combined with Zenrows, Zapier enables full automation of your scraping process. Here are some key benefits:

* **Scheduled scraping**: Run scraping tasks on a regular schedule, such as hourly or daily, without manual input.
* **No-code setup**: Create scraping workflows without needing to write or maintain code.
* **Simplified scraping**: Zenrows handles JavaScript rendering, rate limits, dynamic content, anti-bot protection, and geo-restrictions automatically.
* **Seamless data integration**: Store results directly in tools like Google Sheets, Excel, SQL databases, or visualization platforms like Tableau.
* **Automated monitoring**: Track price changes, stock updates, and website modifications with minimal effort.

This integration is ideal for businesses that want to scale scraping workflows quickly and reliably.

## Watch the Video Tutorial

Learn how to set up the Zapier ↔ Zenrows integration step-by-step by watching this video tutorial:

<iframe title="Zenrows ↔ Zapier" />

## Zenrows Integration Options

With Zenrows, you can perform various web scraping tasks through Zapier:

* **Scraping a URL**: Returns a full-page HTML from a given URL.
* **Scraping a URL With CSS Selectors**: Extracts specific data from a URL based on given selectors.
* **Scraping a URL With Autoparse**: Parses a web page automatically and returns relevant data in JSON format.
  <Note>The autoparse option only works for some websites. Learn more about how it works in the [Autoparse FAQ](/fetch/faq#what-is-autoparse).</Note>

## Real-World Integration

In this guide, we'll use Zapier's schedule to automate web scraping with Zenrows' autoparse integration option. This setup will enable you to collect data at regular intervals and store it in a Google Sheet.

### Step 1: Create a new trigger on Zapier

1. Log in to your Zapier account at <a href="https://zapier.com/">Zapier</a>.
2. Click `Create` at the top left and select `Zap`.
   <img alt="zapier-create-zap" />
3. Click `Untitled Zap` at the top and select `Rename` to give your Zap a specific name.
4. Click `Trigger`.
   <img alt="zapier-workflow-trigger" />
5. Select `Schedule` to create a scheduled trigger.
   <img alt="zapier-workflow-schedule" />
6. Click the `Choose an event` dropdown and choose a frequency. You can customize the frequency or choose from existing ones (e.g., "Every Day").
   <img alt="zapier-trigger-setup" />
7. Click `Continue`.
   <img alt="zapier-frequency-schedule-configuration" />
8. Click the `Time of the day` dropdown and choose a scheduled time for the trigger, or click the option icon to customize.
   <img alt="zapier-trigger-time-configuration" />
9. Click `Continue`.
   <img alt="zapier-trigger-final-setup" />
10. Click `Test trigger` and then `Continue with selected record`.
    <img alt="zapier-schedule-final-step" />

### Step 2: Add Zenrows Scraping Action

1. Type `ZenRows` in the search bar and click it when it appears.
   <img alt="zenrows-search-on-zapier" />
2. Click the `Action event` dropdown and select `Scraping a URL With Autoparse`. Then click `Continue`.
   <img alt="zapier-zenrows-integration-options" />
3. Click the `Connect Zenrows` box and paste your Zenrows API key in the pop-up box. Click `Yes, Continue to Zenrows`, then `Continue`.
   <img alt="zapier-zenrows-connection" />
   <img alt="zapier-zenrows-setup-continue" />
4. Paste the following URL in the URL box: `https://www.amazon.com/dp/B0DKJMYXD2/`.
5. Select `True` for **Adaptive Stealth Mode** and click `Continue`.
   <img alt="zapier-zenrows-configuration-step" />
   <Tip>For more information about Adaptive Stealth Mode, [click here](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode).</Tip>
6. Click `Test step` to confirm the integration and pull initial data from the page.
   <img alt="zapier-zenrows-test-step" />

### Step 3: Save the extracted data

1. Click the `+` icon below the Zenrows step. Then, search and select `Google Sheets`.
2. Click the `Action event` dropdown and select `Create Spreadsheet Row`.
3. Click the `Account` box to connect your Google account, and click `Continue`.
   <img alt="zapier-google-sheets-connection" />
4. Add the following column names to the spreadsheet you want to connect:
   * Name
   * Price
   * Discount
   * SKU
   * Average rating
   * Review count
   * Timestamp
5. Click the `Drive` box and select your Google Sheets location. Choose the target `Spreadsheet` and `Worksheet`.
   <img alt="zapier-google-sheets-drive selection" />
6. Map the columns with the scraped data by clicking the `+` icon next to each column name. Select the corresponding data for each column from the **Scraping a URL With Autoparse** step.
   <img alt="zapier_scraper_mapping" />
7. Map the `Timestamp` column with the ID data from the schedule trigger and click `Continue`.
   <img alt="zapier-timestamp-selection" />
8. Click `Test step` to confirm the workflow.
   <img alt="zapier-google-sheets-test-step" />
9. Click `Publish` to activate your scraping schedule.
   <img alt="zapier-workflow-publish" />

### Step 4: Validate the Workflow

The workflow runs automatically on schedule every day and adds a new row of data to the connected spreadsheet.

<img alt="scraped-data-sample" />

Congratulations! 🎉 You just integrated Zenrows with Zapier and are now automating your scraping workflow.

## Zenrows Configuration Options

Zenrows accepts the following configuration options during Zapier integration:

| **Configuration** | **Function** |
| - | - |
| **URL** | The URL of the target website. |
| **Premium Proxy** | When activated, it routes requests through the Zenrows Residential Proxies, instead of the default Datacenter proxies. |
| **Proxy Country** | <Tooltip>The country geolocation to use in a request.</Tooltip> |
| **JavaScript Rendering** | Ensures that dynamic content loads before scraping. |
| **Wait for Selector** | <Tooltip>Pauses scraping execution until a particular selector is visible in the DOM.</Tooltip> |
| **Wait Milliseconds** | <Tooltip>Waits for a fixed amount of time before executing the scraper.</Tooltip> |
| **JavaScript Instructions** | <Tooltip>Passes JavaScript code to execute actions like clicking, scrolling, and more.</Tooltip> |
| **Headers** | <Tooltip>Adds the custom headers to the request.</Tooltip> |
| **Session ID** | Uses a session ID to maintain the same IP for multiple API requests for up to 10 minutes. |
| **Original Status** | Returns the original status code returned by the target website. |

## Troubleshooting

### Common Issues and Solutions

* **Issue**: Failed to create a URL in Zenrows ('REQS004').
  * **Solution**: Double-check the target URL and ensure it's not malformed or missing essential query strings.
  * **Solution 2**: If using the CSS selector integration, ensure you pass the selectors as an array (e.g., `[{"title":"#productTitle", "price": ".a-price-whole"}]`).

* **Issue**: Authentication failed (`AUTH002`).
  * **Solution**: Double-check your Zenrows API key and ensure you enter a valid one.

* **Issue**: Empty data.
  * **Solution**: Ensure Zenrows supports autoparsing for the target website. Check the Zenrows Data Collector Marketplace to view the supported websites.
  * **Solution 2**: If using the CSS selector integration, supply the correct CSS selectors that match the data you want to scrape.

***

## Conclusion

You've successfully integrated Zenrows with Zapier and are now automating your scraping workflow, from scheduling to data storage. You can extend your workflow with more applications, including databases like SQL and analytic tools like Tableau.

## Frequently Asked Questions (FAQs)

<Accordion title="How do I authenticate my requests?">
  To authenticate requests, use your Zenrows API key. Replace `ZENROWS API KEY` in the authentication modal that Zapier shows.
</Accordion>

<Accordion title="How do I use the wait or wait_for parameters?">
  Use `wait` to pause the rendering for a specific number of milliseconds. Use `wait_for` with a CSS selector to wait until that element appears before capturing the response. Both require JS Rendering.
</Accordion>

<Accordion title="How do I rotate IPs per request?">
  Zenrows rotates IPs automatically per request.
</Accordion>


# Zenrows MCP Integration for Zed
Source: https://docs.zenrows.com/integrations/zed

Learn how to set up the Zenrows MCP server in the Zed code editor so your AI assistant can scrape web pages and extract data easily.

## Before You Start

You need:

<Columns>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">A **Zenrows account**</Card>
  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">Your **Zenrows API key**</Card>
</Columns>

<Card icon="code" href="https://nodejs.org/en/download/" title="Node.js">Latest version of **Node.js** installed on your machine</Card>
<Card icon="robot" href="https://zed.dev" title="Zed">**Zed** installed</Card>

## Setup

<Steps>
  <Step title="Step 1: Open the settings file">
    Open the Zed global settings file:

    ```
    ~/.config/zed/settings.json
    ```
  </Step>

  <Step title="Step 2: Add the Zenrows MCP server">
    Add the following block inside your existing settings, replacing `YOUR_ZENROWS_API_KEY` with your actual API key:

    ```json theme={"dark"}
    {
        "context_servers": {
            "zenrows": {
                "command": {
                    "path": "npx",
                    "args": ["-y", "@zenrows/mcp"],
                    "env": {
                        "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
                    }
                }
            }
        }
    }
    ```
  </Step>

  <Step title="Step 3: Restart Zed">
    Save the file and restart Zed to load the new configuration.
  </Step>
</Steps>

## Start scraping

Ask Zed's AI assistant to scrape a webpage or automate a browser workflow in plain English:

```
Get the product names and prices from https://www.scrapingcourse.com/ecommerce/.
```

```
Scrape https://www.scrapingcourse.com/antibot-bypass using Premium Proxies.
```

```
Fetch https://www.scrapingcourse.com/javascript-rendering/ with JavaScript rendering enabled and return the page as Markdown.
```

```
Open https://scrapingcourse.com/login in a browser, fill in the credentials, click submit, then extract the account details.
```

## Troubleshooting

**The `scrape` tool is not available**

Confirm the `context_servers` block is correctly nested inside `~/.config/zed/settings.json` and the file contains valid JSON. You can validate the JSON syntax at <a href="https://jsonlint.com">jsonlint.com</a>. Then restart Zed.

**Page content is missing or incomplete**

The page likely loads content dynamically. Ask the assistant to enable JavaScript rendering:

```
Scrape https://www.scrapingcourse.com/javascript-rendering/ with `js_render: true`.
```

**Requests return a 403 or bot detection page**

Ask the assistant to use Premium Proxies:

```
Scrape https://www.scrapingcourse.com/antibot-bypass using `premium_proxy: true`.
```

For more troubleshooting options, see the [Zenrows MCP overview](https://docs.zenrows.com/mcp/overview#troubleshooting).


# Residential Proxies (Legacy) FAQ: Protocols and Geo-Targeting
Source: https://docs.zenrows.com/legacy/residential-proxies/faq

Find answers to frequently asked questions about Zenrows Residential Proxies, including protocols, geo-targeting, sticky sessions, and billing.

<Accordion title="What's the difference between Residential Proxies and Premium Proxy?">
  Both draw from the same pool of residential IPs, but they're different products for different use cases.

  **Premium Proxy** is a feature of [Fetch](/fetch/features/premium-proxy) (formerly, Universal Scraper API). Every API request already routes through a proxy, a datacenter IP by default. Setting `premium_proxy=true` switches that proxy to a residential IP for the request, alongside the API's other built-in features, such as JavaScript rendering, anti-bot bypass, and CAPTCHA avoidance.

  **Residential Proxies** is a standalone product. It gives you direct access to the residential IP pool itself, with no anti-bot toolkit, JavaScript rendering, or CAPTCHA handling included. It exists purely for anonymity and IP rotation, so you plug it into your own scraping setup and handle everything else yourself.

  **In short:**

  * Use Fetch with `premium_proxy=true` if you want residential IPs combined with anti-bot handling in a single request.
  * Use Residential Proxies if you only need the residential IP itself and prefer to build your own scraping logic around it.

  <Tip>
    See the [Premium Proxy documentation](/fetch/features/premium-proxy) and the [Residential Proxies introduction](/legacy/residential-proxies/introduction) to compare both in detail.
  </Tip>
</Accordion>

<Accordion title="Can I use HTTPS with Zenrows Residential Proxies?">
  Yes, Zenrows Residential Proxies support both HTTP and HTTPS.
</Accordion>

<Accordion title="How do I rotate the IPs?">
  Zenrows automatically rotates IPs for each request unless you configure a Sticky TTL to keep the same IP for a certain time period. This helps in scenarios where maintaining the same IP for a session is required.
</Accordion>

<Accordion title="What should I do if my requests are blocked?">
  Many websites employ advanced anti-bot measures, such as CAPTCHAs and Web Application Firewalls (WAFs), to prevent automated scraping. Simply using proxies may not be enough to bypass these protections.

  Instead of relying solely on proxies, consider using [Zenrows' Fetch](https://app.zenrows.com/fetch), which provides:

  * **JavaScript Rendering and Interaction Simulation:** Optimized with anti-bot bypass capabilities.
  * **Comprehensive Anti-Bot Toolkit:** Zenrows offers advanced tools to overcome complex anti-scraping solutions.
</Accordion>

<Accordion title="Can I use both the World Region Targeting and Country Targeting together?">
  No, you cannot use both World Region Targeting and Country Targeting at the same time in the same proxy request. Zenrows only accepts one geo-targeting parameter per request, either a region or a country. Attempting to use both will result in an error, and your request will not be processed.

  <Info>If you need to scrape data from multiple countries within a region, run separate requests for each country using the appropriate country code.</Info>
</Accordion>

<Accordion title="Can I use both the Sticky TTL and Geolocation together?">
  Yes, you can combine Sticky TTL (for session persistence) and geo-targeting (region or country) in your proxy credentials. This allows you to maintain the same IP address for a specified duration while also targeting a specific region or country.

  **How does it work?**

  * Add the TTL and session parameters after your password, followed by your region or country targeting code.
  * Zenrows will assign an IP from your chosen location and keep it consistent for the duration of the TTL and session.

  **Example:**

  ```bash theme={"dark"}
  # Sticky session for 30 seconds, targeting Spain
  http://<YOUR_USERNAME>:<YOUR_PASSWORD>_ttl-30s_session-abc123_country-es@superproxy.zenrows.com:1337
  ```
</Accordion>


# Country Targeting with Zenrows Residential Proxies (Legacy)
Source: https://docs.zenrows.com/legacy/residential-proxies/features/country

Use country-level geo-targeting with Zenrows Residential Proxies to access localized content and scrape region-restricted websites.

Zenrows Residential Proxies allow you to target specific countries for your web scraping operations, providing even more precise geo-targeting than region-level settings. This is invaluable for accessing country-restricted content, conducting localized research, or verifying ads and offers as they appear to users in a particular country.

## Why Use Country Targeting?

Country-level targeting is ideal for:

* Accessing country-restricted content and services
* Monitoring localized pricing, ads, or search results
* Testing website localization and legal compliance
* Ad verification and fraud prevention

## How to Set a Country

To target a country, append the `country` code to your proxy password in the proxy URL. You cannot set both a region and a country at the same time.

### Example: Targeting Spain

```bash theme={"dark"}
http://<YOUR_USERNAME>:<YOUR_PASSWORD>_country-es@superproxy.zenrows.com:1337
```

<Frame>
  <img alt="Zenrows Residential Proxies World Region and Country selectors" />
</Frame>

<Tip>
  For a complete list of supported countries and their codes, see our [Premium Proxy Countries List](/first-steps/faq#what-is-geolocation-and-what-are-all-the-premium-proxy-countries).
</Tip>

**Note:** If you specify both a region and a country, only one will be used. Always double-check your proxy password for the correct format.

## Best Practices

* Use country targeting for highly localized research, price comparison, or compliance checks.
* Make sure to use the correct country code to avoid errors.
* Rotating proxies within a country can help reduce detection and blocking.

## Troubleshooting

* If you get an error, ensure the country code is supported and correctly formatted.
* Some countries may have limited proxy pools; if you encounter access issues, try a broader region.


# Protocol and sticky sessions for Residential Proxies (Legacy)
Source: https://docs.zenrows.com/legacy/residential-proxies/features/protocol-sticky-ttl

Learn how to choose between HTTP and HTTPS ports and configure sticky sessions with custom TTL values for Zenrows Residential Proxies.

Zenrows Residential Proxies support both HTTP and HTTPS protocols, each available on its own dedicated port, so you can connect with any HTTP client. You can also use sticky sessions (TTL) to control how long a particular IP address stays assigned to your requests, which is essential for maintaining session continuity in tasks like login persistence, checkout processes, or paginated data extraction.

## Choosing a Protocol: HTTP or HTTPS

Zenrows Residential Proxies expose two ports, one per protocol:

| Protocol | Port | Proxy URL format |
| - | - | - |
| HTTP | `1337` | `http://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1337` |
| HTTPS | `1338` | `https://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1338` |

<CodeGroup>
  ```bash HTTP theme={"dark"}
  http://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1337
  ```

  ```bash HTTPS theme={"dark"}
  https://<YOUR_USERNAME>:<YOUR_PASSWORD>@superproxy.zenrows.com:1338
  ```
</CodeGroup>

Replace `<YOUR_USERNAME>` and `<YOUR_PASSWORD>` with your credentials from the [Residential Proxies dashboard](https://app.zenrows.com/residential-proxies).

<Tip>
  Most HTTP clients, such as Python's `requests` or Node.js's `axios`, only need the HTTP port (`1337`) since they tunnel HTTPS traffic through it automatically. Use port `1338` only if your client or setup requires you to connect through a dedicated HTTPS proxy endpoint.
</Tip>

### Example: Using Both Ports

```python Python theme={"dark"}
proxy_username = "YOUR_USERNAME"
proxy_password = "YOUR_PASSWORD"
proxy_domain = "superproxy.zenrows.com"

proxies = {
    "http": f"http://{proxy_username}:{proxy_password}@{proxy_domain}:1337",
    "https": f"https://{proxy_username}:{proxy_password}@{proxy_domain}:1338",
}

import requests
response = requests.get("https://www.scrapingcourse.com/ecommerce/", proxies=proxies)
print(response.status_code)
```

## Why Use Sticky TTL?

Sticky sessions allow you to:

* Maintain the same IP for a set period, from 30 seconds up to 1 day.
* Ensure session consistency for workflows that require login or multi-step navigation.
* Reduce the risk of bans or captchas by mimicking real user behavior.

## How to Enable Sticky Sessions

To enable sticky sessions, append the `ttl` parameter and a session ID to your proxy password in the proxy URL. This works the same way regardless of which port or protocol you use.

### Example: 30-Second Sticky Session

```bash theme={"dark"}
http://<YOUR_USERNAME>:<YOUR_PASSWORD>_ttl-30s_session-1cLiFzDgsq36@superproxy.zenrows.com:1337
```

<Frame>
  <img alt="Zenrows Residential Proxies Protocol and Sticky TTL options" />
</Frame>

* `ttl-30s` ensures the IP will be retained for 30 seconds.
* `session-1cLiFzDgsq36` is a random session identifier that you can generate yourself. You can use any combination of letters and numbers, similar in length and style to the example. This allows Zenrows to allocate the same IP for multiple requests as long as you use the same session ID.

### Available Sticky TTL Values

* **30 seconds:** `ttl-30s`
* **1 minute:** `ttl-1m`
* **30 minutes:** `ttl-30m`
* **1 hour:** `ttl-1h`
* **1 day:** `ttl-1d`

<Info>
  The session ID is a random string that uniquely identifies your session. Generate it manually, or copy one from the Residential Proxy Dashboard.
</Info>

## Best Practices

* Use sticky sessions for login-required flows, shopping carts, or any scenario where the same IP is needed across requests.
* Choose the shortest TTL that meets your needs to maximize IP pool rotation and avoid detection.
* Always generate a unique session ID for each session to prevent conflicts.
* Match the port to the protocol you're connecting with (`1337` for HTTP, `1338` for HTTPS), unless your client tunnels both through the HTTP port.

## Troubleshooting

* If you observe frequent IP changes, double-check your TTL and session ID format.
* Some websites may require longer TTLs for certain workflows; experiment with different values to find the best fit.
* If your connection fails on port `1338`, confirm your client supports connecting directly to an HTTPS proxy endpoint, then fall back to port `1337`.

<Note>
  Learn more about session management and advanced proxy features in the [Zenrows Proxy documentation](/fetch/features/premium-proxy).
</Note>


# World Region Targeting with Zenrows Residential Proxies (Legacy)
Source: https://docs.zenrows.com/legacy/residential-proxies/features/world-region

Filter residential proxy traffic by world region to collect geo-specific data and bypass regional restrictions with Zenrows proxies.

Zenrows Residential Proxies empower you to access and collect data from websites as if you were located in different parts of the world. This is crucial for bypassing regional restrictions, monitoring localized content, and simulating user behavior from specific continents or large areas.

## Why Use World Region Targeting?

Many websites tailor their content, pricing, or availability based on the visitor's region. By targeting a world region, you can:

* Bypass geo-blocks and access region-restricted content
* Gather region-specific data for market research or compliance
* Test how websites behave for users in different parts of the world
* Rotate IPs within a continent for broader coverage and reduced detection risk

## How to Set a World Region

Zenrows makes it easy to select a region by appending a region code to your proxy password in the proxy URL. This tells Zenrows to assign your requests to residential IPs from the specified region.

### Example: Targeting Europe

```bash theme={"dark"}
http://<YOUR_USERNAME>:<YOUR_PASSWORD>_region-eu@superproxy.zenrows.com:1337
```

<Frame>
  <img alt="Zenrows Residential Proxies World Region and Country selectors" />
</Frame>

### Available Region Codes

* **Europe:** `region-eu`
* **North America:** `region-na`
* **Asia Pacific:** `region-ap`
* **South America:** `region-sa`
* **Africa:** `region-af`
* **Middle East:** `region-me`

<Tip>
  Use region targeting for broad coverage or when you want to rotate IPs within a large area. For more precise targeting, see our [Country Targeting guide](/fetch/features/premium-proxy).
</Tip>

## Best Practices

* Only one region or country can be set per request. If both are specified, only one will be used.
* Always double-check your proxy password for the correct format.
* Use region targeting for tasks like price comparison, ad verification, or compliance testing across continents.

## Troubleshooting

* If you receive an error, verify that the region code is correct and supported.
* Some websites may still apply additional restrictions; consider using sticky sessions or country targeting for more advanced scenarios.


# Make Your First Request with Zenrows Residential Proxies (Legacy)
Source: https://docs.zenrows.com/legacy/residential-proxies/get-started/first-request

Send your first request through Zenrows Residential Proxies, verify the proxy connection works, and explore code examples in any language.

Learn how to make your first web scraping request using Zenrows' Residential Proxies. This guide walks you through the basics, shows you how to verify your proxy setup, and introduces advanced proxy features, such as geo-targeting and sticky sessions.

Zenrows Residential Proxies enable you to access region-restricted content by routing requests through a pool of millions of real residential IP addresses. You can use them in any programming language that supports HTTP proxies.

## 1. Make a Request Without a Proxy

An initial request without a proxy uses your default IP address.

To confirm the IP address being used, we'll request `https://httpbin.io/ip`, a web page that returns your IP address.

<CodeGroup>
  ```python Python theme={"dark"}
  # pip install requests
  import requests
  response = requests.get("https://httpbin.io/ip")
  if response.status_code != 200:
      print(f"An error occurred with {response.status_code}")
  else:
      print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');
  axios.get('https://httpbin.io/ip')
    .then((response) => {
      if (response.status !== 200) {
        console.log(`An error occurred with ${response.status}`);
      } else {
        console.log(response.data);
      }
    })
    .catch((error) => {
      console.log('An error occurred:', error.message);
    });
  ```
</CodeGroup>

You'll see your device's public IP address in the output. Next, let's use Zenrows Residential Proxies.

## 2. Integrate Zenrows Residential Proxies (Legacy)

Improve the previous request with the Residential Proxies by following these steps.

### Step 1: Get Your Proxy Credentials

1. Go to your [Zenrows Residential Proxies dashboard](https://app.zenrows.com/residential-proxies).
2. Copy your proxy username, password, proxy domain, and port.

<img alt="Zenrows Residential Proxies Dashboard" />

### Step 2: Configure Your Proxy Settings

Prepare your proxy credentials and build the proxy URL for your HTTP client.

<CodeGroup>
  ```python Python theme={"dark"}
  proxy_username = "PROXY_USERNAME"
  proxy_password = "PROXY_PASSWORD"
  proxy_domain = "superproxy.zenrows.com"
  proxy_port = "1337"
  proxy_url = f"http://{proxy_username}:{proxy_password}@{proxy_domain}:{proxy_port}"
  proxies = {
      "http": proxy_url,
      "https": proxy_url,
  }
  ```

  ```javascript Node.js theme={"dark"}
  const proxy = {
    protocol: 'http',
    host: 'superproxy.zenrows.com',
    port: '1337',
    auth: {
      username: 'PROXY_USERNAME',
      password: 'PROXY_PASSWORD',
    },
  };
  ```
</CodeGroup>

### Step 3: Use Residential Proxies (Legacy) for Scraping Requests

To use the Residential Proxies in your request, include the proxy dictionary as a request parameter:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip3 install requests
  import requests

  # ...

  response = requests.get("https://httpbin.io/ip", proxies=proxies)
  if response.status_code != 200:
      print(f"An error occurred with {response.status_code}")
  else:
      print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  // ...

  // send a request with the proxies
  axios
      .get('https://httpbin.org/ip', {
          proxy: proxy,
      })
      .then((res) => {
          console.log(res.data);
      })
      .catch((err) => console.error(err));
  ```
</CodeGroup>

**Put it all together**

Combine all the snippets, and you'll have the following complete code:

<CodeGroup>
  ```python Python theme={"dark"}
  # pip3 install requests
  import requests

  # define your proxy credentials
  proxy_username = "PROXY_USERNAME"
  proxy_password = "PROXY_PASSWORD"
  proxy_domain = "superproxy.zenrows.com"
  proxy_port = "1337"

  # build the proxy URL
  proxy_url = f"http://{proxy_username}:{proxy_password}@{proxy_domain}:{proxy_port}"

  # configure proxy protocols
  proxies = {
      "http": proxy_url,
      "https": proxy_url,
  }

  response = requests.get("https://httpbin.io/ip", proxies=proxies)
  if response.status_code != 200:
      print(f"An error occurred with {response.status_code}")
  else:
      print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  // npm install axios
  const axios = require('axios');

  // define your proxy credentials
  const proxy = {
      protocol: 'http',
      host: 'superproxy.zenrows.com',
      port: '1337',
      auth: {
          username: 'PROXY_USERNAME',
          password: 'PROXY_PASSWORD',
      },
  };

  // send a request with the proxies
  axios
      .get('https://httpbin.org/ip', {
          proxy: proxy,
      })
      .then((res) => {
          console.log(res.data);
      })
      .catch((err) => console.error(err));
  ```
</CodeGroup>

Run the code and you'll see a new IP address for each request, confirming your requests are routed through Zenrows Residential Proxies.

### Example Output

```json theme={"dark"}
{
    "origin": "71.172.140.38:44752"
}
```

## 3. Advanced Proxy Options

Zenrows Residential Proxies offer advanced features for more control:

### Geo-targeting

Route requests through IPs from a specific country or region. This is useful for:

* Comparing product prices across regions
* Job availability and salary comparison
* Housing and rent price variation
* Logistics and shipping cost analysis
* Testing website localization and compliance
* Demand analysis
* Ad verification

**How to set the proxy country:**

<CodeGroup>
  ```python Python theme={"dark"}
  proxy_username = "PROXY_USERNAME"
  proxy_password = "PROXY_PASSWORD_country-ca"  # Canada
  proxy_domain = "superproxy.zenrows.com"
  proxy_port = "1337"
  ```

  ```javascript Node.js theme={"dark"}
  const proxy = {
    protocol: 'http',
    host: 'superproxy.zenrows.com',
    port: '1337',
    auth: {
      username: 'PROXY_USERNAME',
      password: 'PROXY_PASSWORD_country-ca',
    },
  };
  ```
</CodeGroup>

Replace `country-ca` with the desired country code. See the [geo-location FAQ](/fetch/features/proxy-country) for more details.

### Regional Rotation

Rotate proxies within a specific region (e.g., Europe):

<CodeGroup>
  ```python Python theme={"dark"}
  proxy_password = "PROXY_PASSWORD_region-eu"  # Europe
  ```

  ```javascript Node.js theme={"dark"}
  password: 'PROXY_PASSWORD_region-eu',
  ```
</CodeGroup>

<Warning>You can only set either a country or a region, not both.</Warning>

### Sticky Sessions (TTL)

The sticky TTL (Time-to-Live) feature allows you to maintain a single proxy for a specified duration.

To add a stick TTL, go to your Residential Proxies dashboard and select a TTL option. The TTL option is added to the generated proxy URL:

<img alt="Res. Proxies session ttl" />

Include the generated TTL session in the password string:

<CodeGroup>
  ```python Python theme={"dark"}
  proxy_password = "PROXY_PASSWORD_ttl-30s_session-qiGzEUZSGCh9"
  ```

  ```javascript Node.js theme={"dark"}
  password: 'PROXY_PASSWORD_ttl-30s_session-qiGzEUZSGCh9',
  ```
</CodeGroup>

## Troubleshooting

* **Tunnel failed with response 407 (authentication error):**
  * Ensure you enter the correct authentication credentials.
  * Enter the correct proxy country or region code.
* **Access denied | 403 forbidden error:**
  * Try another country or region.
  * Reduce the session TTL to avoid using the same IP address for too long.
  * Switch to Fetch to increase the scraping success rate.
* **Could not resolve proxy:**
  * Check and ensure you've used the correct proxy domain.

## Tips for Accessing Highly Protected Websites

* **Scrape during off-peak hours:** Scraping when the website is less busy can increase your success rate, as anti-bots may be less active. Off-peak hours vary per site but usually fall at night or in the mornings.
* **Use exponential backoffs and retries:** Adequate use of backoffs and retries helps simulate real user behavior and can reduce the chances of IP bans, especially for long TTL sessions.
* **Persist session with proxies to solve CAPTCHAs:** CAPTCHAs are tied to sessions. When solving CAPTCHAs manually while using proxies, maintain the same proxy for the duration of the challenge. Set an appropriate session TTL to keep your IP consistent and prevent repeated challenges.
* **Combine proxies with custom request headers:** Use Residential Proxies with custom headers (like a real browser's User Agent or a trusted Referer) to increase your scraping success rate.
* **Use Fetch:** For increased success without manual configuration, switch to the Zenrows Fetch. It pre-configures all the necessary tools for successful scraping, including bypassing anti-bots and scraping dynamic content.

Check out our article on pro tips to scrape without getting blocked to learn more.

## Frequently Asked Questions (FAQ)

<Accordion title="Can Residential Proxies bypass anti-bots?">
  While Residential Proxies significantly reduce the chances of anti-bot detection and IP bans, they don't guarantee you'll escape blocks. Anti-bots analyze more than IPs. They examine headers, fingerprints, behavioral patterns, and more. For a guaranteed anti-bot bypass, switch to Fetch.
</Accordion>

<Accordion title="Is the TTL session flexible or fixed?">
  The Residential Proxies Sticky TTL is flexible, allowing you to persist a single proxy from as little as 1 second up to 24 hours.
</Accordion>

<Accordion title="Are Residential Proxies available on the Free plan?">
  No, Zenrows' Residential Proxies are only available on paid subscriptions. However, Zenrows offers a unified pricing model: subscribing to one plan gives you access to all four products (Fetch, Extract, Batch, Browser Sessions).
</Accordion>

<Accordion title="How do the Residential Proxies prevent IP bans and rate limiting?">
  Residential proxies auto-rotate proxies from a pool of 55+ million IPs across 190+ countries. They support geo-targeting of specific countries and regions, enhancing anonymity and access to region-specific content. This prevents anti-bot measures from tracking you to a single IP or determining your exact location.
</Accordion>


# Introduction to Residential Proxies (Legacy)
Source: https://docs.zenrows.com/legacy/residential-proxies/introduction

Overview of Zenrows Residential Proxies featuring 55M+ rotating IPs across 190+ countries, geo-targeting, and 99.9% uptime for scraping.

Zenrows provides an extensive residential proxy service that includes a pool of over 55 million residential IPs from more than 190 countries. Our premium residential proxies deliver 99.9% uptime and an average response time of 0.35 seconds, ensuring reliable and efficient connections.

**With features like IP auto-rotation and flexible geo-targeting, Zenrows Residential Proxies help you avoid rate limits and bypass geo-restrictions on a large scale.**

## Key Applications

Proxies are critical for web scraping and data extraction, but their applications extend beyond just avoiding IP blocks. Below are the primary benefits of using residential proxies for data extraction:

* **Bypassing Anti-Bot Systems:** Many websites deploy anti-bot mechanisms to block suspicious IP addresses. Proxies allow you to rotate IP addresses with every request, making your scraping operations undetectable. However, it's important to note that only premium rotating proxies can effectively handle sophisticated anti-bot systems. Free proxies or manual switching will provide a different level of protection.

* **Geolocation Targeting:** Some websites restrict access based on your geographic location. With residential proxies, you gain access to IP addresses from precise regions worldwide, allowing you to retrieve localized data and avoid geo-restrictions.

* **Boosted Performance:** Residential proxies enhance your scraping performance by enabling you to send more requests while minimizing errors, blocks, and timeouts. The result is a higher success rate, leading to more efficient data extraction.

* **Maintaining Anonymity:** Proxies conceal information about your device and identity, ensuring that your web scraping activities remain anonymous and secure.

## Residential Proxies vs. Premium Proxy

It's easy to confuse this product with the Premium Proxy feature in Zenrows' [Fetch](/fetch/features/premium-proxy) (formerly, Universal Scraper API), since both use the same residential IPs. The difference is in what you get around them:

* **Residential Proxies** (this product) gives you direct, standalone access to the residential IP pool. It's purely the proxy, meant for anonymity and IP rotation. There's no anti-bot toolkit, JavaScript rendering, or CAPTCHA handling bundled in, so you connect it to your own scraping stack and control the rest.
* **Premium Proxy** is a parameter (`premium_proxy=true`) inside Fetch. Every API request already goes through a proxy, a datacenter IP by default, and this parameter swaps it for a residential IP for that request, combined with the API's other anti-bot features.

<Tip>
  If you want the residential IP on its own, use Residential Proxies. If you want residential IPs plus built-in anti-bot handling in one API call, use Fetch with `premium_proxy=true`. See the [FAQ](/legacy/residential-proxies/faq) for a more detailed comparison.
</Tip>

## Parameter Overview

When using Zenrows Residential Proxies, you can customize your scraping requests with the following parameters to enhance control over proxy usage, geolocation targeting, and response behavior:

| PARAMETER | TYPE | DEFAULT | DESCRIPTION |
| - | - | - | - |
| **username** `required` | `string` | | Your unique username for authentication |
| **password** `required` | `string` | | Your password |
| [**region**](https://docs.zenrows.com/legacy/residential-proxies/features/world-region) | `string` | `global` | Focus your scraping on a specific geographic region (e.g., `region-eu` for Europe). **When set to global, no parameter is needed** |
| [**country**](https://docs.zenrows.com/legacy/residential-proxies/features/country) | `string` | | Target a specific country for geo-restricted data (e.g., `country-es` for Spain). Available only when not using `region` |
| [**ttl**](https://docs.zenrows.com/legacy/residential-proxies/features/protocol-sticky-ttl) | `number` | | Control how long the same IP address is used for your requests Minimum: 30, Maximum: 1 day. |

## Pricing

Residential Proxies, used standalone, bill in credits: a flat **25,000 credits per GB** on every plan. The service charges per bandwidth used. For the full plan table and credit costs, see our [pricing page](/first-steps/pricing).


# Setting Up Residential Proxies (Legacy)
Source: https://docs.zenrows.com/legacy/residential-proxies/setup

Create your Zenrows Residential Proxies credentials and configure proxy settings like world region, country, protocol, and sticky TTL.

In this guide, we'll walk you through setting up your Zenrows Residential Proxies. You'll learn how to create and access your credentials, including your username, password, and domain, which are essential for integrating the proxies into your applications.

Additionally, we'll cover the available configuration options, such as World Region, Country, Protocol, and Sticky TTL, to help you optimize the use of the proxies based on your specific requirements.

## Initial Setup

To get started with the Residential Proxies, follow these steps:

<Steps>
  <Step title="Sign Up for Zenrows">
    To begin, you'll need to create a Zenrows account. Visit the [Registration Page](https://app.zenrows.com/register) to sign up. If you already have an account, simply log in to access your dashboard.
  </Step>

  <Step title="(Optional) Create Credentials for Residential Proxies">
    <Note>A credentials pair (user and password) will be created for you automatically, so this step is not mandatory.</Note>

    Once you're signed in, navigate to the [Credentials Page](https://app.zenrows.com/configure/residential-proxies) under the Residential Proxies section. Here, you can create credentials by specifying your proxy quota and generating your unique username and password. You can create as many credentials as needed, allowing you to manage multiple projects easily!

    <Frame>
      <img alt="Zenrows Residential Proxies Credentials tab with proxy users" />
    </Frame>
  </Step>

  <Step title="Verify Credentials and Configure Your Proxies">
    After generating your credentials, you'll be taken to the [Generator Page](https://app.zenrows.com/configure/residential-proxies), where your credentials (username, password, and various configuration options) will be displayed.

    You can copy the Proxy URL and start integrating it into your applications.

    <Frame>
      <img alt="Zenrows Residential Proxies Generator Page" />
    </Frame>
  </Step>
</Steps>

## Proxy Configuration Options

Zenrows Residential Proxies are designed to be functional right out of the box. With auto-rotation and residential IPs pre-configured, you can start using them immediately by copying the Proxy URL. However, we also provide additional customization options to help tailor the proxies to your use case.

<Note>Auto-rotate and Residential IPs are enabled by default for all users.</Note>

You can adjust the following configuration options:

### Username and Password

To make proxy management easier, you can switch between usernames directly in the dashboard. Simply select the desired username, and the proxy URL will automatically update with the corresponding username and password. Note that the domain and port will remain the same.

<Frame>
  <img alt="Zenrows Residential Proxies username, password, and proxy URL fields" />
</Frame>


# Zenrows MCP Server
Source: https://docs.zenrows.com/mcp/overview

Give any MCP-compatible AI tool, agent, or application real-time access to the live web. One integration combines fast scraping and full browser automation, with anti-bot bypass on every tool.

The [Zenrows MCP (Model Context Protocol) server](https://github.com/ZenRows/zenrows-mcp) is the standard way AI systems use Zenrows. A single connection gives your AI assistant, agent, or application real-time access to any website.

<Card title="View on GitHub" icon="github" href="https://github.com/ZenRows/zenrows-mcp">
  Zenrows MCP is open source. Star the repository, file issues, or contribute.
</Card>

## Why Zenrows MCP

* **Reach sites that normally block bots:** Get access to any website at scale without getting blocked by anti-bot systems.
* **Managed scraping infrastructure:** Proxy rotation, headless browser orchestration, anti-bot evasion, and session management run on Zenrows infrastructure.
* **Plug into any AI you already use:** Works with any MCP client, including AI assistants, agent frameworks, AI SDKs, IDE plugins, and custom applications.
* **Plain English, no scraping code:** Describe the task naturally and the AI picks the right tool. No selectors, no proxy management, no anti-bot tuning.

## What you can do

Once connected, your AI assistant can:

* Scrape any webpage by describing the task in plain English
* Extract structured data from e-commerce sites, news pages, job boards, and more
* Render pages as Markdown, plain text, HTML, or PDF
* Take above-the-fold or full-page screenshots
* Access geo-restricted, JavaScript-heavy, or bot-protected pages
* Automate multi-step browser workflows: navigate, click, fill forms, scroll, drag, and extract across pages
* Queue a large set of URLs as one asynchronous job, then come back for the results
* Run JavaScript in a live browser context
* Manage cookies, local storage, multiple tabs, and persistent sessions
* Check the plan's remaining credits before starting a large job

## Before you start

<CardGroup>
  <Card icon="user" href="https://app.zenrows.com/register" title="Zenrows Account">
    A **Zenrows account**
  </Card>

  <Card icon="key" href="https://app.zenrows.com/dashboard" title="Zenrows API Key">
    Your **Zenrows API key**
  </Card>
</CardGroup>

## Connection options

The Zenrows MCP server supports two transport options. Pick the one that fits your client.

| Option | Transport | Authentication |
| - | - | - |
| [**Remote MCP server**](#remote-mcp-server) | Streamable HTTP | OAuth Bearer token |
| [**Local MCP server**](#local-mcp-server) | STDIO (local subprocess) | Environment variable |

Both options expose the same set of tools and capabilities. The choice is purely about how your client connects to the server.

### Remote MCP server

The hosted Zenrows MCP server is the recommended path for any AI application that calls an LLM API directly. The server runs on Zenrows' infrastructure, so there is nothing to install, configure, or update.

* **Server URL:** <br /> `https://mcp.zenrows.com/mcp`
* **Transport:** <br /> Streamable HTTP
* **Authentication:** <br /> OAuth-based. Pass your Zenrows API key as a Bearer token. See [Authentication](#authentication) for details.

**Example: OpenAI Responses API**

Most modern AI SDKs accept the connection details through a single MCP tool definition. Here is the complete configuration for the OpenAI Responses API:

```python Python theme={"dark"}
import os
from openai import OpenAI

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

response = client.responses.create(
    model="gpt-5",
    tools=[
        {
            "type": "mcp",
            "server_label": "zenrows",
            "server_description": "Web scraping MCP server for accessing live web content.",
            "server_url": "https://mcp.zenrows.com/mcp",
            "authorization": ZENROWS_API_KEY,
            "require_approval": "never",
        }
    ],
    input="Visit https://news.ycombinator.com/ and summarize the three most recent posts.",
)

print(response.output_text)
```

**Generic configuration**

| Field | Value |
| - | - |
| Endpoint | `https://mcp.zenrows.com/mcp` |
| Transport | Streamable HTTP |
| Auth header | `Authorization: Bearer YOUR_ZENROWS_API_KEY` |

Most MCP clients accept either an `authorization` shorthand field that automatically wraps the value as a Bearer token, or a free-form `headers` field where you set the `Authorization` header yourself. Either approach works with Zenrows MCP.

### Local MCP server

If your AI client runs the MCP server as a local subprocess instead of calling a remote URL, use the [`@zenrows/mcp`](https://www.npmjs.com/package/@zenrows/mcp) npm package. This is the standard configuration for desktop AI clients.

<Warning>
  You need [Node.js](https://nodejs.org/) installed for `npx` to work.
</Warning>

The exact location of the configuration file varies by client, but every supported client uses a similar MCP server config shape. Replace `YOUR_ZENROWS_API_KEY` with your actual key:

```json JSON theme={"dark"}
{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": {
        "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
      }
    }
  }
}
```

#### Per-client setup guides

For step-by-step instructions including the exact config file path and restart steps for each client:

| Client | Type |
| - | - |
| [Claude Desktop](/integrations/claude-desktop) | AI assistant |
| [Claude Code](/integrations/claude-code) | AI coding assistant |
| [Cursor](/integrations/cursor) | AI code editor |
| [Windsurf](/integrations/windsurf) | AI code editor |
| [VS Code](/integrations/vscode) | AI code editor |
| [Zed](/integrations/zed) | AI code editor |
| [JetBrains IDEs](/integrations/jetbrains) | AI code editor |

## Authentication

The Zenrows MCP server uses OAuth-based Bearer token authentication. Your Zenrows API key acts as the access token.

<Tabs>
  <Tab title="Remote MCP server">
    Send the key in the HTTP `Authorization` header on every request:

    ```text theme={"dark"}
    Authorization: Bearer YOUR_ZENROWS_API_KEY
    ```

    OpenAI's MCP tool, Anthropic's MCP tool, and most other MCP clients expose this through a single `authorization` field on the tool config and forward it as the Bearer token automatically. Some clients use a free-form `headers` field instead. Both approaches are valid.
  </Tab>

  <Tab title="Local STDIO MCP server">
    Pass the API key through the `ZENROWS_API_KEY` environment variable in your client's MCP configuration:

    ```json JSON theme={"dark"}
    {
      "mcpServers": {
        "zenrows": {
          "env": {
            "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
          }
        }
      }
    }
    ```
  </Tab>
</Tabs>

You can find or rotate your API key in your [Zenrows dashboard](https://app.zenrows.com/settings/api-keys).

<Warning>
  Treat your API key like a password. Do not commit it to source control or share it in client-side code.
</Warning>

## Tools

A tool in MCP terms is a named capability the server exposes. For example, `scrape` or `browser_navigate`.

When you ask your AI to fetch a webpage or fill a form, the AI sees the available tools, picks the one that fits the task, and invokes it. **You don't call tools yourself in code.** You set up the MCP connection once, and the AI handles tool selection and invocation from there.

The Zenrows MCP exposes four groups of tools, plus one standalone tool for checking credits.

### The `scrape` tool

The `scrape` tool fetches a webpage in a single request and returns its content in the format you specify: clean Markdown (default), plain text, raw HTML, a screenshot, a PDF, or structured JSON. It uses [Fetch](/fetch/api-reference) under the hood.

Best for quick data extraction where the AI doesn't need to interact with the page.

### The `extract` tool

The `extract` tool returns a page as structured JSON rather than as a body of text. Prefer it over `scrape` when the answer is a set of fields, such as a product's name and price, an article's author and date, or the rows of a listing: it returns a fraction of the tokens a full page costs.

It runs in one of three modes:

| Mode | What it does |
| - | - |
| `auto` (default) | Site-tailored extraction (`extract=auto`). In open beta, and currently free. Falls back to `autoparse` when the domain is not yet enabled. |
| `autoparse` | General-purpose structured JSON, on any domain. |
| `css` | Your own selector map, passed as `css_extractor`. |

The same stealth options as `scrape` apply: `js_render`, `premium_proxy`, `proxy_country`, and `mode_auto` for Adaptive Stealth Mode.

For a full page as Markdown, HTML, or a screenshot, use `scrape` instead.

### Batch tools

The `batch_*` tools submit and manage [Batch](/batch/introduction) jobs: one job that fans out many URLs asynchronously, so the AI doesn't hold a connection open per URL. Best when the URL set is large enough that waiting on each request in turn is impractical, or when the assistant should submit work now and collect it later.

`batch_create` returns as soon as the job is accepted, so a job that runs for hours doesn't block a synchronous tool call. Poll with `batch_status`, or hand the waiting to `batch_wait`.

| Tool | Description |
| - | - |
| `batch_create` | Submit a job that fans out many URLs asynchronously. Returns a `job_id` plus the run's status and stats. Returns immediately by default; set `wait` to block until the run is terminal |
| `batch_status` | Get the current status and stats for a job |
| `batch_results` | List the per-task result rows for a job, filtered to successful, failed, or all |
| `batch_wait` | Poll until the job reaches a terminal state — completed, stopped, or deleted |
| `batch_cancel` | Stop an in-flight run |

<Note>
  **`browser_batch` is a different tool.** It sequences several browser actions into one call against a browser session and has nothing to do with the Batch API. See [Available browser tools](#available-browser-tools).
</Note>

### Browser tools

The `browser_*` tools give your AI assistant control of a cloud-hosted browser session powered by the [Zenrows Browser Sessions](/browser-sessions/introduction). The browser comes with built-in residential proxies and anti-bot bypass.

Browser tools cover tasks that involve:

* Navigation across multiple pages in a single session
* Form fills, button clicks, and other element interactions
* Waiting for dynamically loaded content
* JavaScript execution in the browser context
* Cookies, local storage, and multi-tab management

Every browser workflow starts with `browser_navigate`, which opens a session and returns a `session_id`. All subsequent browser calls use this `session_id`. Call `browser_close` when done.

#### Available browser tools

<AccordionGroup>
  <Accordion title="Session">
    | Tool | Description |
    | - | - |
    | `browser_navigate` | Open a browser session and navigate to a URL. Returns a `session_id` for subsequent calls. |
    | `browser_close` | Close a browser session and free all associated resources. |
  </Accordion>

  <Accordion title="Navigation">
    | Tool | Description |
    | - | - |
    | `browser_go_back` | Navigate back in browser history. |
    | `browser_go_forward` | Navigate forward in browser history. |
    | `browser_reload` | Reload the current page. |
  </Accordion>

  <Accordion title="Interaction">
    | Tool | Description |
    | - | - |
    | `browser_click` | Click an element using a CSS selector. |
    | `browser_hover` | Hover over an element to trigger hover effects. |
    | `browser_type` | Type text into an input field. Appends to existing content unless `clear_first` is set. |
    | `browser_fill` | Clear an input field and set its value. |
    | `browser_select_option` | Select an option from a `<select>` dropdown. |
    | `browser_check` | Check a checkbox or radio button. |
    | `browser_uncheck` | Uncheck a checkbox. |
    | `browser_focus` | Move focus to an element. |
    | `browser_press_key` | Press a keyboard key (e.g. `Enter`, `Tab`, `Escape`, `Control+a`). |
    | `browser_scroll` | Scroll the page in a given direction. |
    | `browser_drag` | Drag an element to a target element. |
  </Accordion>

  <Accordion title="Extraction">
    | Tool | Description |
    | - | - |
    | `browser_get_accessibility_tree` | Get the page's accessibility tree as readable text. Call this after `browser_navigate` to understand page structure. |
    | `browser_get_url` | Get the current URL. |
    | `browser_get_title` | Get the page title. |
    | `browser_get_text` | Get the visible text of an element or the entire page. |
    | `browser_get_attribute` | Get the value of a specific attribute on an element. |
    | `browser_get_html` | Get the HTML source of an element or the full page. |
    | `browser_query_selector_all` | Find all elements matching a CSS selector. Returns text, HTML, and attributes. |
  </Accordion>

  <Accordion title="Visual">
    | Tool | Description |
    | - | - |
    | `browser_screenshot` | Take a screenshot of the page or a specific element. |
    | `browser_generate_pdf` | Render the current page as a PDF. |
  </Accordion>

  <Accordion title="Wait">
    | Tool | Description |
    | - | - |
    | `browser_wait_for_selector` | Wait until an element is stable in the DOM. Set `visible=true` to require visibility. |
    | `browser_wait_for_navigation` | Wait for a page navigation. Call this before the action that triggers navigation. |
    | `browser_wait` | Wait for a fixed duration in milliseconds. |
  </Accordion>

  <Accordion title="JavaScript">
    | Tool | Description |
    | - | - |
    | `browser_evaluate` | Execute JavaScript in the browser context and return the result. |
  </Accordion>

  <Accordion title="Cookies">
    | Tool | Description |
    | - | - |
    | `browser_get_cookies` | Get all cookies for the current page. |
    | `browser_set_cookies` | Set one or more cookies. |
    | `browser_clear_cookies` | Clear all cookies for the session. |
  </Accordion>

  <Accordion title="Storage">
    | Tool | Description |
    | - | - |
    | `browser_local_storage` | Read, write, or clear localStorage. |
  </Accordion>

  <Accordion title="Tabs">
    | Tool | Description |
    | - | - |
    | `browser_new_tab` | Open a new tab. Returns a `tab_id`. |
    | `browser_switch_tab` | Switch focus to a different tab. |
  </Accordion>

  <Accordion title="Batch">
    | Tool | Description |
    | - | - |
    | `browser_batch` | Execute a sequence of browser actions in a single call for better performance. |
  </Accordion>
</AccordionGroup>

### The `account_usage` tool

The `account_usage` tool reports the current plan's credit allowance and how much of it is spent. It is free and does not consume a concurrency slot, so an assistant can call it safely before fanning out a large job, or after a request fails for credits.

Credit cost per request: 1 basic, 5 with `js_render`, 10 with `premium_proxy`, 25 with both.

`AUTH004` means the allowance is spent. It renews at the end of the billing period, so it is not a permanent block and should never be retried in a loop. `AUTH006` is the concurrency limit, which is a different thing.

`AUTH014` means the API key reached one of its [credit caps](/first-steps/api-key-credit-caps), while your other keys keep working. Raise or remove the cap, or wait for its window to reset, instead of retrying.

## Pricing

The Zenrows MCP server itself is free. There is no separate fee for connecting, listing tools, or using the integration. Each tool call is billed against your Zenrows subscription using the standard pricing of the underlying product:

* Calls to the `scrape` tool follow [Fetch pricing](/fetch/api-reference#pricing).
* Calls to the `extract` tool use [Extract](/extract/sdk-overview), which is in open beta and currently free. Billing may apply later.
* Calls to `batch_*` tools follow [Batch](/batch/introduction) pricing.
* Calls to `browser_*` tools follow [Browser Sessions pricing](/browser-sessions/introduction).
* `account_usage` is free and is not billed.

You can monitor real-time usage and remaining quota on your [Analytics page](https://app.zenrows.com/overview#performance).

## Troubleshooting

### Connection issues

| Issue | Cause | Solution |
| - | - | - |
| `424 Failed Dependency` or `401 Unauthorized` | Missing or invalid Bearer token | Confirm `Authorization: Bearer YOUR_ZENROWS_API_KEY` is set on every request. For OpenAI/Anthropic SDKs, use the `authorization` field on the MCP tool config. Verify your key is active in your [dashboard](https://app.zenrows.com/settings/api-keys) |
| Tool list is empty when connecting to the remote URL | Authentication failed silently or wrong endpoint | Confirm the URL is exactly `https://mcp.zenrows.com/mcp` and the Bearer token is present and valid |
| The `scrape` tool is unavailable in your AI assistant (local STDIO) | Configuration not loaded yet | Restart your AI tool after saving the configuration file. For Claude Desktop, fully quit and reopen the application |
| API key is not recognized (local STDIO) | Key not replaced in configuration | Confirm `YOUR_ZENROWS_API_KEY` has been replaced with your actual key from the [Zenrows dashboard](https://app.zenrows.com). The value must have no extra spaces or surrounding quotation marks |

### Scraped content issues

If your AI is connecting and invoking tools correctly but the scraped content comes back missing, blocked, or incomplete, the issue is with the target page rather than the MCP connection.

Common causes include JavaScript-loaded content, anti-bot protection, geo-restrictions, and dynamic loading. See the [Fetch troubleshooting guide](/fetch/troubleshooting/troubleshooting-guide) for fixes you can ask your AI to apply. For example, "scrape with JavaScript rendering enabled" or "use Premium Proxies."

## Example prompts

Once Zenrows MCP is connected, describe the task to your AI assistant in plain English and it handles the scraping automatically.

<Tabs>
  <Tab title="Scrape tool">
    ```text wrap theme={"dark"}
    Fetch https://www.scrapingcourse.com/javascript-rendering/. It uses React, so enable JavaScript rendering.
    ```

    ```text wrap theme={"dark"}
    Get the top 5 products from https://www.scrapingcourse.com/ecommerce/ and extract just the names and prices.
    ```

    ```text wrap theme={"dark"}
    Take a full-page screenshot of https://news.ycombinator.com.
    ```

    ```text wrap theme={"dark"}
    Scrape https://www.scrapingcourse.com/antibot-challenge. It's blocking requests. Use Adaptive Stealth Mode.
    ```
  </Tab>

  <Tab title="Browser tools">
    ```text wrap theme={"dark"}
    Open https://www.scrapingcourse.com/ecommerce/ in a browser, type "hoodie" into the search field, press Enter, wait for the results to load, then extract all matching product names and prices.
    ```

    ```text wrap theme={"dark"}
    Navigate to https://scrapingcourse.com/ecommerce/, click on the first product, extract its full description and price, then go back and do the same for the second product.
    ```

    ```text wrap theme={"dark"}
    Open https://news.ycombinator.com in a browser, scroll down to load more stories, then extract all the headlines and links.
    ```
  </Tab>
</Tabs>

## Related resources

<CardGroup>
  <Card title="Fetch Reference" icon={<svg xmlns="http://www.w3.org/2000/svg"  width="24"  height="24"  viewBox="0 0 24 24"  fill="none"  stroke="#7C4FE8"  strokeWidth="2"  strokeLinecap="round"  strokeLinejoin="round"  className="icon icon-tabler icons-tabler-outline icon-tabler-chisel"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M14 14l1.5 1.5" /><path d="M18.347 15.575l2.08 2.079a1.96 1.96 0 0 1 -2.773 2.772l-2.08 -2.079a1.96 1.96 0 0 1 2.773 -2.772z" /><path d="M3 6l3 -3l7.414 7.414a2 2 0 0 1 .586 1.414v2.172h-2.172a2 2 0 0 1 -1.414 -.586l-7.414 -7.414z" /></svg>} href="/fetch/api-reference">
    Full parameter reference for the `scrape` tool's underlying API.
  </Card>

  <Card title="Browser Sessions Introduction" icon={<svg xmlns="http://www.w3.org/2000/svg"  width="24"  height="24"  viewBox="0 0 24 24"  fill="none"  stroke="#7C4FE8"  strokeWidth="2"  strokeLinecap="round"  strokeLinejoin="round"  className="icon icon-tabler icons-tabler-outline icon-tabler-click"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M3 12l3 0" /><path d="M12 3l0 3" /><path d="M7.8 7.8l-2.2 -2.2" /><path d="M16.2 7.8l2.2 -2.2" /><path d="M7.8 16.2l-2.2 2.2" /><path d="M12 12l9 3l-4 2l-2 4l-3 -9" /></svg>} href="/browser-sessions/introduction">
    Learn more about the browser powering the `browser_*` tools.
  </Card>

  <Card title="Zenrows Dashboard" icon="chart-bar" href="https://app.zenrows.com">
    Monitor usage, manage your API key, and track quota.
  </Card>

  <Card title="NPM Package" icon="npm" href="https://www.npmjs.com/package/@zenrows/mcp">
    `@zenrows/mcp` on npm for local STDIO deployments.
  </Card>
</CardGroup>

## Contributing

The Zenrows MCP server is open source. If you want to modify the server, fix a bug, or suggest an improvement, clone the [GitHub repository](https://github.com/ZenRows/zenrows-mcp) and run it locally:

```bash Bash theme={"dark"}
git clone https://github.com/ZenRows/zenrows-mcp
cd zenrows-mcp
npm install
cp .env.example .env   # Add your API key
npm run dev            # Run with .env loaded (requires Node.js 20.6+)
npm run build          # Compile to dist/
npm run inspect        # Open the MCP inspector UI
```

Pull requests and issues are welcome on the [GitHub repository](https://github.com/ZenRows/zenrows-mcp).


# Content guidelines for Zenrows affiliates
Source: https://docs.zenrows.com/partners/affiliates/content-guidelines

A step-by-step guide for Zenrows affiliates: how to prepare, which topics to cover, how to shape a piece, where to publish it, and how to get it amplified.

This is the working guide for the Zenrows Affiliates program. Follow it in order for each piece you produce.

For which topics to cover, see [Topics that convert](/partners/affiliates/content-topics). This page covers everything else.

<Note>
  New to the program? Do [Step 0](#step-0-prepare-once) once, then repeat Steps 1 to 5 for every piece.
</Note>

## Step 0: Prepare once

Spending a little time with Zenrows before your first piece makes everything after it easier to write, and it is the difference between content that reads like a press release and content that reads like experience.

<Steps>
  <Step title="Create a Zenrows account">
    The Free plan gives you 5,000 credits per month, enough to explore every product. See [Pricing](/first-steps/pricing) for what a credit buys.
  </Step>

  <Step title="Try the thing you plan to write about">
    Send [your first request](/fetch/first-request), or connect the [Zenrows MCP server](/mcp/overview) to your AI assistant, or run a command with the [CLI](/cli/introduction). Whichever surface your audience uses.
  </Step>

  <Step title="Note the moments that surprised you">
    Whatever made you think "oh, that's neat" will do the same for your audience. Those moments are your content, not the feature list.
  </Step>

  <Step title="Bookmark your sources">
    Keep [the docs](/first-steps/welcome), [the topic list](/partners/affiliates/content-topics), and the [Zenrows media kit](https://www.zenrows.com/media) open while you work.
  </Step>
</Steps>

## Step 1: Pick your topics and intent

Match each piece to a real search intent and stay with it. A reader searching "how to give Claude web access" wants a setup walkthrough, not a product overview.

* **Choose topics** from [Topics that convert](/partners/affiliates/content-topics). Every cluster there is backed by real search demand.
* **Choose the funnel stage** and shape the piece around it. Awareness pieces teach, evaluation pieces solve a specific problem, decision pieces help someone choose.
* **Play to your own audience.** A Python-heavy audience wants code. A founder or ops audience wants the outcome and the setup steps. The same topic works for both if you change the framing.

<Tip>
  **AI and integrations convert best right now.** Pieces about connecting Zenrows to the tools people already use, especially through the [Zenrows MCP server](/mcp/overview) and the [CLI](/cli/introduction), reach an audience actively looking for a solution and facing very little competing content. "How to give Claude web data access at scale with Zenrows MCP" is the shape to aim for.
</Tip>

## What to emphasize about Zenrows

Whatever the topic, these are the things worth showing. They are where Zenrows is strongest, and they are what readers are actually evaluating.

* **Fetch and Batch on protected sites, at scale.** [Fetch](/fetch/api-reference) handles the pages that block ordinary tools, and [Batch](/batch/introduction) runs the same job across large lists of URLs without you building the orchestration. The pairing is the real story: one request shape that works on a hard page, and a managed way to run it a hundred thousand times.
* **High-volume recurring extraction.** Most readers do not need one page once, they need the same pages every day. Show a job that submits a URL list, retries transient failures on its own, and returns results you can collect on a schedule.
* **Reliability where ordinary tools give up.** This is the clearest differentiator you can demonstrate. Show a request that fails without Zenrows and succeeds with it, and point to the public track record: 100% Fetch uptime over the last 90 days, across more than 10 billion requests a month.
* **The CLI and MCP server as an agent toolkit.** The [CLI](/cli/introduction) and the [MCP server](/mcp/overview) turn Zenrows into something an AI agent installs and uses directly rather than something a developer wires up by hand. The [Agent Toolkit](/cli/agent-toolkit-overview) is the part built specifically for that. This is the most under-covered angle available to you right now.
* **Security and compliance, for business readers.** Enterprise plans include SOC 2 and ISO 27001 compliance, an SLA on uptime and success rate, and SSO/SAML. Worth a line in any piece aimed at teams who need procurement approval. Link to [Pricing](/first-steps/pricing) rather than restating the terms.

## Step 2: Show Zenrows in action

You do not need to build anything elaborate. A clear demonstration, even a short one, does more work than paragraphs of description. Pick whichever of these fits your format:

| What to show | Why it lands |
| - | - |
| **A page that would not load, now loading** | The most immediate before and after you can show, in any format |
| **An AI assistant answering from live web data** | Connect Zenrows MCP to Claude, Cursor, or another client and let it pull real pages |
| **One call replacing a lot of setup** | Everything a developer would otherwise assemble themselves, handled in a single request |
| **Structured data instead of raw HTML** | A clean, ready-to-use response |
| **The same job running across many pages** | Shows the production use case rather than the toy example |

<Tip>
  If you are writing for a non-developer audience, the MCP and integration angles need no code at all. A short screen recording of an assistant fetching live data is often the strongest demonstration you can make.
</Tip>

## Step 3: Shape the piece

These are examples of shapes that hold attention, not rules. Adapt or ignore them to suit your audience and your format.

<Tabs>
  <Tab title="Written tutorial">
    Around 1,800 to 3,000 words works well.

    1. **Hook.** The problem and why it matters.
    2. **What you will build.** Describe the result and show it.
    3. **Prerequisites.** Tools, accounts, and a Zenrows API key.
    4. **The basic setup.** The simplest thing that works.
    5. **Handling the harder part.** Add capability step by step.
    6. **Using the output.** Parsing, storing, or feeding it to an assistant.
    7. **Conclusion.** Recap, link to the docs, link to your other pieces.
  </Tab>

  <Tab title="Integration or MCP guide">
    Often shorter than a tutorial, and frequently higher converting.

    1. **The outcome.** What the reader's tool can do once this is set up.
    2. **Prerequisites.** The tool, a Zenrows account, an API key.
    3. **Connect it.** The configuration, exactly as it should be pasted.
    4. **Prove it works.** One real prompt, command, or run with its output.
    5. **Where to take it next.** Two or three things the reader can now do.
  </Tab>

  <Tab title="Video">
    Around 7 to 12 minutes for a tutorial.

    1. **Introduction** (30 to 45 sec). State the problem.
    2. **Outcome preview** (30 to 45 sec). Show the finished result. Viewers who see the result first drop off far less.
    3. **Setup** (1 to 2 min). Install, connect, get the API key.
    4. **Build it on screen** (5 to 8 min).
    5. **Pro tips** (30 to 60 sec). What you learned using it for real.
    6. **Wrap-up** (30 to 45 sec). Recap and one clear next step.
  </Tab>

  <Tab title="Short-form video">
    15 to 60 seconds, shot in portrait.

    1. **Hook in the first two seconds.** Lead with the result, not with your face.
    2. **One idea only.**
    3. **Captions always.** Most viewers watch on mute.
    4. **One next step.** Point to the full piece, not to a signup page.

    The same cut works on YouTube Shorts, TikTok, and Instagram Reels. Post it to all three.
  </Tab>

  <Tab title="Newsletter">
    400 to 900 words. Newsletter audiences in this niche are small, technical, and unusually high intent.

    * A short case study.
    * A recurring "tool of the week" or "target of the week" breakdown.
    * A walkthrough of a setup you actually use.

    One tracked link per issue, clearly placed.
  </Tab>
</Tabs>

## Step 4: Publish it where it will be found

Publish to your own site first so you own the traffic, then syndicate.

| Channel | What works there | Notes |
| - | - | - |
| **Your own blog** | Any format | Your canonical home. Everything else should point back here. |
| **dev.to** | Tutorials and integration guides, 1,500 to 2,000 words | Write like you are helping a colleague. Up to 4 tags. Set a canonical URL when cross-posting. |
| **Hashnode** | Long technical guides | Enable the table of contents and use the TL;DR field. |
| **Medium** | Pieces already published elsewhere | Strong distribution through publications like Better Programming and Towards Data Science. Member-only stories get lift inside Medium but are gated for Google, so set a canonical tag back to your original. |
| **LinkedIn Articles** | Use-case and integration pieces | Lead with the business outcome. Publish as an Article for indexing, then share a teaser post that links to it. |
| **X Articles** | Long-form posts for an audience you already have on X | Keeps readers on-platform, which X rewards in reach. Pair with a short thread pointing to it. |
| **Substack or beehiiv** | Newsletter issues and essays | Both index in Google and build a list you own. |
| **GitHub** | The demo repo or config behind your piece | A README with working code or an MCP config, and a link back. Frequently outranks the article itself. |
| **YouTube** | Tutorials and short-form | Put the full written version in the description. |
| **Product Hunt** | A tool or template you built on Zenrows | Launch Tuesday to Thursday morning with a demo video. |

## Step 5: Promote it

| Surface | What works there |
| - | - |
| **X (Twitter)** | Short technical tips, code snippets, before and after demos. Screen recordings outperform text. Tag [@ZenRowsHQ](https://x.com/ZenRowsHQ). |
| **LinkedIn** | Frame web data as a business capability. Native video outperforms links. Best shape is a 3 to 5 paragraph narrative plus the link. Post Tuesday to Thursday, 8 to 10 AM. |
| **Reddit** | r/webscraping, r/learnpython, r/webdev, r/datascience, r/ClaudeAI, r/mcp. "I built X to solve Y" posts with real detail. Read each subreddit's rules on self-promotion first. |
| **Hacker News** | Original benchmarks, teardowns, and research. Submit weekday mornings, US Eastern. |
| **Discord and Slack** | Contribute for a few days before you share anything. Answer the question fully, then offer your piece as a resource. |
| **Stack Overflow** | Answer completely and technically under `[web-scraping]`, `[python-requests]`, `[beautifulsoup]`, `[selenium]`. Mention Zenrows where it genuinely solves the asker's problem. |
| **Quora** | 100 to 300 word answers that address the question directly, linking to your piece rather than a product page. |

<Tip>
  Answer platforms and forums rank well in Google and are quoted heavily by AI assistants. A good Stack Overflow or Reddit answer keeps sending readers to your content years after you write it.
</Tip>

## Accuracy rules

These apply to every piece, in every format.

* **Describe capabilities as they are documented.** "Handles common anti-bot systems" is accurate and is what the docs support.
* **Use the real names.** Parameters like `js_render`, `premium_proxy`, and `extract=auto` are exact strings. Products are Fetch (formerly, Universal Scraper API), Extract, Batch, and Browser Sessions (formerly, Scraping Browser).
* **Check prices against the live page.** Plans and credit costs change. Link to [Pricing](/first-steps/pricing) rather than restating numbers that will go stale.
* **Target public data.** Publicly accessible pages such as product listings, search results, and public profiles make the strongest pieces.
* **Avoid personal data at scale.** Pieces that harvest names, emails, or phone numbers create GDPR and CCPA problems for your readers.
* **Do not target the forbidden list.** See [Forbidden sites](/forbidden-sites) for sites Zenrows does not serve.

## Brand assets

Download logos, symbols, app icons, screenshots, and approved descriptions from the [Zenrows media kit](https://www.zenrows.com/media). It is the single source of truth for Zenrows brand assets.

The rules that matter most:

* **The name is always "Zenrows"**, capital Z and the rest lowercase. Not a capital R mid-word, not all lowercase.
* **Keep clear space** around the wordmark, at least the height of the "o".
* **Pick the variant with the most contrast** against your background. The wordmark ships in white, dark, teal, and purple.
* **Use the symbol** where the full wordmark will not fit. The glow variant needs a dark background.
* **Do not** add effects, rotate, stretch, recolor, rebuild the wordmark in another typeface, or place it on a busy background.

Brand colors: purple `#7C4FE8`, teal `#7ADAE6`, warm dark `#1D1612`, white `#FFFFFF`.

## Get your content amplified

Zenrows regularly reshares affiliate content from its own channels. To be considered:

1. Tag [@ZenRowsHQ](https://x.com/ZenRowsHQ) on X or mention Zenrows on LinkedIn when you publish.
2. Send the link to [marketing@zenrows.com](mailto:marketing@zenrows.com).

What gets reshared has a clear angle, working examples, and describes Zenrows accurately.

## Pre-publish checklist

Run through this before anything goes live.

<AccordionGroup>
  <Accordion title="Topic and intent">
    * [ ] The topic comes from [Topics that convert](/partners/affiliates/content-topics) or matches a real search query you verified
    * [ ] The piece serves a clear funnel stage
    * [ ] The framing fits the audience you are publishing to
  </Accordion>

  <Accordion title="Accuracy">
    * [ ] Any code or configuration in the piece works as written
    * [ ] Parameter and product names are spelled exactly as documented
    * [ ] Pricing links to the live page rather than restating figures
  </Accordion>

  <Accordion title="Structure">
    * [ ] The piece demonstrates Zenrows rather than only describing it
    * [ ] Screenshots have descriptive alt text
    * [ ] The primary keyword appears in the title, the first H2, and the first 100 words
  </Accordion>

  <Accordion title="Ethics and compliance">
    * [ ] The target is public data
    * [ ] The target is not on the [forbidden sites](/forbidden-sites) list
    * [ ] No personal data collected at scale
  </Accordion>

  <Accordion title="Brand and distribution">
    * [ ] "Zenrows" is spelled correctly everywhere
    * [ ] Logo assets came from the [media kit](https://www.zenrows.com/media) and are unmodified
    * [ ] Canonical URL is set if you are cross-posting
    * [ ] Internal links point to your other Zenrows pieces
    * [ ] The link is sent to [marketing@zenrows.com](mailto:marketing@zenrows.com)
  </Accordion>
</AccordionGroup>


# Topics that convert for Zenrows affiliates
Source: https://docs.zenrows.com/partners/affiliates/content-topics

Five proven topic clusters for Zenrows affiliate content, with example angles and the funnel stage each one serves.

Five clusters, drawn from Zenrows' own search and demand research. Each one has verified search demand and a clear path from reader to customer. Use this page to choose a topic, then follow the [content guidelines](/partners/affiliates/content-guidelines) to produce it.

**Funnel stages** used below: TOFU (awareness, reader is learning), MOFU (evaluation, reader has a problem), BOFU (decision, reader is choosing a tool).

| Cluster | Funnel | Converts because |
| - | - | - |
| [AI agents and MCP](#ai-agents-and-mcp) | TOFU to BOFU | Fast-growing demand with very little existing coverage |
| [Integrations and workflows](#integrations-and-workflows) | MOFU to BOFU | Reaches people already committed to a tool they want to extend |
| [Command line and automation](#command-line-and-automation) | MOFU | Short pieces, high intent, almost no competing content |
| [Site-specific guides](#site-specific-guides) | MOFU | Highest volume and most durable traffic of any cluster |
| [Working use cases](#working-use-cases) | MOFU | Reaches decision-makers who do not yet know web data is the answer |

## AI agents and MCP

The strongest-performing cluster right now. Start from [the MCP server docs](/mcp/overview).

* How to give Claude web data access at scale with Zenrows MCP
* Setting up Zenrows MCP in Cursor, VS Code, or Windsurf
* What "agent-ready" web data actually requires
* Building a vertical AI agent with LangChain and Zenrows

## Integrations and workflows

Connecting Zenrows to a tool the reader already uses. See the [integrations section](/integrations/overview) for the shapes these take.

* Adding live web data to an n8n or Zapier workflow
* Enriching CRM records with public web data
* Feeding a RAG pipeline from live pages instead of a stale index
* Streaming collected data into a warehouse or spreadsheet

## Command line and automation

Underserved and quick to produce. Start from [the CLI docs](/cli/introduction).

* Scraping a list of URLs from your terminal with the Zenrows CLI
* Scheduling a recurring data pull with the CLI and cron
* Piping Zenrows output straight into `jq`, a CSV, or a database
* Using the CLI inside a CI job

## Site-specific guides

"How to scrape X" pieces. Pick one site, cover its specific defenses and page structure, and ship something that works.

* How to scrape Google Shopping
* How to scrape Etsy
* How to scrape Indeed
* How to scrape Idealista

## Working use cases

Business problems rather than techniques. Lower competition and reaches readers with purchasing authority.

* Build an Amazon price tracker
* Automated competitor price tracking
* Real estate data aggregation at scale
* Company data enrichment from public web sources

<Note>
  These are starting angles, not assigned titles. Adapt the wording to your own audience and keyword research.
</Note>


# Zenrows SDKs
Source: https://docs.zenrows.com/sdk/overview

Official, typed SDKs for every primitive, Fetch, Extract, Batch, and Browser Sessions.

Zenrows gives every primitive a clear path in code: official, typed clients for Fetch and Batch, Extract through that same Fetch client, and Browser Sessions through your existing browser automation setup.

<CardGroup>
  <Card title="Fetch" icon="chisel" href="/fetch/sdk/overview">
    Official clients for Python, Node.js, and Go, wrapping the same endpoint with authentication, retries, and concurrency control.
  </Card>

  <Card title="Extract" icon="wand" href="/extract/sdk-overview">
    <Badge>New</Badge> No separate client. Use the same Fetch SDK with `extract=auto` as a parameter.
  </Card>

  <Card title="Batch" icon="layers-subtract" href="/batch/sdk/python">
    <Badge>New</Badge> Official Python and Node.js clients, separate from the Fetch SDKs, for submitting and tracking batch jobs.
  </Card>

  <Card title="Browser Sessions" icon="click" href="/browser-sessions/sdk-overview">
    No SDK needed. Point your existing browser automation setup at Zenrows with one line.
  </Card>
</CardGroup>


# Build a Price Monitoring System
Source: https://docs.zenrows.com/zenrows-academy/build-price-monitoring

Build a price monitoring system with Python and Zenrows that scrapes, stores, and tracks e-commerce product prices automatically over time.

Create a complete price monitoring system from scratch to track price changes on e-commerce sites like Amazon. This step-by-step guide walks you through building an automated system that scrapes, stores, and monitors price data.

## Who Is this for?

This guide is designed for developers who want to build a price monitoring system from the ground up using web scraping. No existing price monitoring infrastructure required.

## What you'll learn

* Extract product price data from an Amazon product page using Zenrows.
* Clean the extracted raw data and structure it for storage and price monitoring.
* Store price history with timestamps.
* Schedule automated scraping at regular intervals.
* Set up price change notifications.
* Optimize performance and manage costs.

## Step 1: Set Up Data Extraction

Extract specific product data related to pricing, availability, and social cues using Zenrows `css_extractor`. Stringify the `css_extractor` for compatibility:

```python Python theme={"dark"}
import json

# specify the CSS selectors for extracting data
css_extractor = json.dumps(
    {
        "name": "span#productTitle",
        "price": "span.aok-offscreen",
        "ratings": "#acrPopover",
        "reviewCount": "#acrCustomerReviewText",
        "demandHistory": "#social-proofing-faceout-title-tk_bought",
        "availability": "#availability",
    }
)
```

Define a scraper function to extract data from a product URL based on the `css_extractor`.

<Warning>CSS selectors can change when websites update their code. To maintain a reliable scraper, monitor your selectors regularly and update them as needed. Learn more about CSS selectors [here](/fetch/features/css-extractor).</Warning>

The `js_render` parameter handles the site's dynamic rendering, while `premium_proxy` routes the request through premium proxies to avoid blocking. `proxy_country` is set to `us` to send requests from US-based IP addresses:

```python Python theme={"dark"}
pip3 install requests
#...
import requests
# define the scraper function
def scraper(url, css_extractor):
    apikey = "YOUR_ZENROWS_API_KEY"
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "css_extractor": css_extractor,
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.json()
```

<Tip>
  The **proxy\_country** parameter is optional. If not specified, Zenrows will use a random IP address worldwide. See more about geolocation [here](/fetch/features/proxy-country).
</Tip>

Execute the scraper function and print the extracted raw data. Here's the updated code:

```python Python theme={"dark"}
# pip3 install requests
import requests
import json

# specify the CSS selectors for extracting data
css_extractor = json.dumps(
    {
        "name": "span#productTitle",
        "price": "span.aok-offscreen",
        "ratings": "#acrPopover",
        "reviewCount": "#acrCustomerReviewText",
        "demandHistory": "#social-proofing-faceout-title-tk_bought",
        "availability": "#availability",
    }
)


# define the scraper function
def scraper(url, css_extractor):
    apikey = "YOUR_ZENROWS_API_KEY"
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "css_extractor": css_extractor,
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.json()


product_url = "https://www.amazon.com/dp/B001CXYMFS"

raw_data = scraper(product_url, css_extractor)

print(raw_data)
```

The code returns the required data from the product page.

Here's what it looks like:

```json JSON response theme={"dark"}
{
    "availability": [
        'In Stock                 {"isInternal":false,"isRobot":false,"showFaceout":true,"merchantId":"ATVPDKIKX0DER","availableBadges":"","loggedIn":false,"asin":"B001CXYMFS","showBadge":false,"availableFaceouts":"TK_BOUGHT"}',
        "Only 1 left in stock - order soon.",
    ],
    "demandHistory": "500+ bought in past month",
    "name": "Thrustmaster T-Flight Hotas X (Compatible with PC)",
    "price": ["$69.99", "$65.79 with 6 percent savings", "New Condition Price: $69.99"],
    "ratings": ["4.4", "4.4"],
    "reviewCount": ["8,067 ratings", "8,067 ratings"],
}
```

## Step 2: Clean and Structure the Data

The returned data is unsuitable for price monitoring in its current state since it contains undesired strings.

Since some fields are returned as a list, define a `consistency_handler` to handle them as a list before cleaning:

```python Python theme={"dark"}
# treat specific fields as a list for consistency
def consistency_handler(field):
    if not field == "":
        if isinstance(field, str):
            return [field]
        elif isinstance(field, list):
            return field
    else:
        return []
```

Define a function to obtain the current price (discounted), the original price, and the discount percentage from the price field:

```python Python theme={"dark"}
# ...
import re

# parse price information from the price list
def parse_price_info(price_list):
    # apply the consistency_handler function to treat as a list
    price_list = consistency_handler(price_list)
    original_price = None
    current_price = None
    discount_percent = None
    currency = "USD"

    # obtain price and discount information
    for item in price_list:
        price_match = re.search(r"\$(\d+[\.,]?\d*)", item)
        percent_match = re.search(r"(\d+)\s*percent", item)
        if "with" in item and price_match and percent_match:
            current_price = float(price_match.group(1).replace(",", ""))
            discount_percent = int(percent_match.group(1))
        elif "New Condition Price" in item and price_match:
            original_price = float(price_match.group(1).replace(",", ""))
        elif price_match and not current_price:
            original_price = float(price_match.group(1).replace(",", ""))
    if current_price is None:
        current_price = original_price
    return original_price, current_price, discount_percent, currency
```

Clean the `reviewCount` field by extracting the count integer from the list of strings:

```python Python theme={"dark"}
# parse review count
def parse_review_count(review_list):
    # apply the consistency_handler function to treat as a list
    review_list = consistency_handler(review_list)
    if review_list:
        count_str = review_list[0].split()[0].replace(",", "")
        return int(count_str)
    return None
```

Obtain the `average_rating` as a float from the raw data:

```python Python theme={"dark"}
# parse average rating
def parse_rating(ratings_list):
    # apply the consistency_handler function to treat as a list
    ratings_list = consistency_handler(ratings_list)
    if ratings_list:
        try:
            return float(ratings_list[0])
        except Exception:
            return None
    return None
```

Similarly, parse the estimated number of units sold from the `demandHistory` field:

```python Python theme={"dark"}
# parse units sold from demand history
def parse_units_sold(demand_history):
    if not demand_history:
        return None
    # apply the consistency_handler function to treat as a list
    demand_history = consistency_handler(demand_history)
    match = re.search(r"(\d+[+]?)(?= bought)", demand_history[0])
    if match:
        return match.group(1)
    return None
```

Clean the `availability` field by extracting the `In Stock` string from the strings:

```python Python theme={"dark"}
# parse availability
def parse_availability(availability):
    # apply the consistency_handler function to treat as a list
    availability = consistency_handler(availability)
    for item in availability:
        if "In Stock" in item:
            return "In Stock"
    return None
```

Finally, parse the cleaned data as a separate dictionary. Use `datetime` and `timestamp` to get the timestamp of the scraping operation to track historical data:

```python Python theme={"dark"}
# ...
from datetime import datetime, timezone

# parse the cleaned data
def parse_cleaned_data(raw_data):
    original_price, current_price, discount_percent, currency = parse_price_info(
        raw_data.get("price", [])
    )
    average_rating = parse_rating(raw_data.get("ratings", []))
    review_count = parse_review_count(raw_data.get("reviewCount", []))
    units_sold = parse_units_sold(raw_data.get("demandHistory", ""))
    availability = parse_availability(raw_data.get("availability", []))
    # get the scraping timestamp
    scrape_timestamp = datetime.now(timezone.utc).isoformat()

    cleaned_data = {
        "name": raw_data.get("name", ""),
        "original_price": original_price,
        "current_price": current_price,
        "discount_percent": discount_percent,
        "currency": currency,
        "average_rating": average_rating,
        "review_count": review_count,
        "units_sold": units_sold,
        "availability": availability,
        "scrape_timestamp": scrape_timestamp,
    }
    return cleaned_data
```

Apply the `parse_cleaned_data` to the `raw_data` to get a cleaned version with a recorded timestamp:

```python Python theme={"dark"}
# ...
product_url = "https://www.amazon.com/dp/B001CXYMFS"
raw_data = scraper(product_url, css_extractor)
cleaned_data = parse_cleaned_data(raw_data)
print(cleaned_data)
```

Here's a sample of the cleaned data:

```json JSON theme={"dark"}
{
    "name": "Thrustmaster T-Flight Hotas X (Compatible with PC)",
    "original_price": 69.99,
    "current_price": 65.79,
    "discount_percent": 6,
    "currency": "USD",
    "average_rating": 4.4,
    "review_count": 8067,
    "units_sold": "500+",
    "availability": "In Stock",
    "scrape_timestamp": "2025-08-08T06:11:35.662685+00:00",
}
```

## Step 3: Store the Price History Data

Since the scraping will be scheduled, start the storage procedure by setting up `logging` to track the extraction process in real-time:

```python Python theme={"dark"}
# ...
import logging

# set up logging
logging.basicConfig(filename="scraper.log", level=logging.INFO)
```

Define a function to create a new price record for each scraping request and store it in a `product_history.json` file. Index the stored data by product name:

```python Python theme={"dark"}
#...
import os

# update the price history for each product in the JSON file
def update_and_save_price_history(new_product, filename="product_history.json"):
    # load existing history
    if os.path.exists(filename):
        with open(filename, "r", encoding="utf-8") as json_file:
            product_history = json.load(json_file)
    else:
        product_history = []

    # index products by name for easy lookup
    history_dict = {product["name"]: product for product in product_history}

    # create a new price record
    price_record = {
        "timestamp": new_product["scrape_timestamp"],
        "current_price": new_product["current_price"],
        "original_price": new_product["original_price"],
        "discount_percent": new_product["discount_percent"],
        "currency": new_product["currency"],
    }
```

Set up an alert trigger (`alert_triggered=False`) and check for price changes by comparing the previously scraped price to the current one. Set `alert_triggered` to True if a price change is detected. Then, log the price difference. Append new price records with existing ones in the JSON file. If a product doesn't exist in the JSON file, create a new record for it:

```python Python theme={"dark"}
# ...
def update_and_save_price_history(new_product, filename="product_history.json"):
    # ...

    # create a flag to indicate if an alert was triggered
    alert_triggered = False

    # check for price changes
    if new_product["name"] in history_dict:
        price_history = history_dict[new_product["name"]]["price_history"]
        if price_history:
            last_price = price_history[-1]["current_price"]
            if new_product["current_price"] != last_price:
                alert_triggered = True
                logging.info(
                    f"ALERT: Price changed for {new_product['name']} from {last_price} to {new_product['current_price']}"
                )

        # append to existing price history
        price_history.append(price_record)
        # update ratings, review count, and units sold
        history_dict[new_product["name"]]["average_rating"] = new_product.get(
            "average_rating"
        )
        history_dict[new_product["name"]]["review_count"] = new_product.get(
            "review_count"
        )
        history_dict[new_product["name"]]["units_sold"] = new_product.get("units_sold")
        history_dict[new_product["name"]]["availability"] = new_product.get(
            "availability"
        )
    else:
        # create a new product entry with price history
        history_dict[new_product["name"]] = {
            "name": new_product["name"],
            "price_history": [price_record],
            "average_rating": new_product.get("average_rating"),
            "review_count": new_product.get("review_count"),
            "units_sold": new_product.get("units_sold"),
        }
```

Finally, update the existing records with the new pricing records and return the alert status:

```python Python theme={"dark"}
def update_and_save_price_history(new_product, filename="product_history.json"):
    # ...

    # write updated history back to the JSON file
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(list(history_dict.values()), f, ensure_ascii=False, indent=2)

    return alert_triggered
```

Here's the complete `update_and_save_price_history` function:

```python Python expandable theme={"dark"}
# update the price history for each product in the JSON file
def update_and_save_price_history(new_product, filename="product_history.json"):
    # load existing history
    if os.path.exists(filename):
        with open(filename, "r", encoding="utf-8") as json_file:
            product_history = json.load(json_file)
    else:
        product_history = []

    # index products by name for easy lookup
    history_dict = {product["name"]: product for product in product_history}

    # create a new price record
    price_record = {
        "timestamp": new_product["scrape_timestamp"],
        "current_price": new_product["current_price"],
        "original_price": new_product["original_price"],
        "discount_percent": new_product["discount_percent"],
        "currency": new_product["currency"],
    }

    # create a flag to indicate if an alert was triggered
    alert_triggered = False

    # check for price changes
    if new_product["name"] in history_dict:
        price_history = history_dict[new_product["name"]]["price_history"]
        if price_history:
            last_price = price_history[-1]["current_price"]
            if new_product["current_price"] != last_price:
                alert_triggered = True
                logging.info(
                    f"ALERT: Price changed for {new_product['name']} from {last_price} to {new_product['current_price']}"
                )

        # append to existing price history
        price_history.append(price_record)
        # update ratings, review count, and units sold
        history_dict[new_product["name"]]["average_rating"] = new_product.get(
            "average_rating"
        )
        history_dict[new_product["name"]]["review_count"] = new_product.get(
            "review_count"
        )
        history_dict[new_product["name"]]["units_sold"] = new_product.get("units_sold")
        history_dict[new_product["name"]]["availability"] = new_product.get(
            "availability"
        )
    else:
        # create a new product entry with price history
        history_dict[new_product["name"]] = {
            "name": new_product["name"],
            "price_history": [price_record],
            "average_rating": new_product.get("average_rating"),
            "review_count": new_product.get("review_count"),
            "units_sold": new_product.get("units_sold"),
        }

    # write updated history back to the JSON file
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(list(history_dict.values()), f, ensure_ascii=False, indent=2)

    return alert_triggered
```

## Step 4: Create a Scraping Job

Create a `job` function to execute the extraction, data cleaning, and storage logics. This function logs the scraping timestamps and price change notifications:

```python Python theme={"dark"}
## define a scraping job
def job():
    logging.info(f"scraping job started at {datetime.now(timezone.utc).isoformat()}")

    product_url = "https://www.amazon.com/dp/B001CXYMFS"

    raw_data = scraper(product_url, css_extractor)
    result = parse_cleaned_data(raw_data)
    alert_triggered = update_and_save_price_history(result)

    logging.info(f"scraping job stopped at {datetime.now(timezone.utc).isoformat()}")
    if alert_triggered:
        logging.info("ALERT: Price change detected!")
    else:
        logging.info("No price change detected.")
```

## Step 5: Schedule the Scraping Job

Schedule the extraction process using Python's `schedule` module. The code below runs the scraping job every 30 seconds and checks for jobs every second:

```python Python theme={"dark"}
# ...
# pip3 install schedule 
import schedule
import time

# schedule the job to run every 2 minutes
schedule.every(2).minutes.do(job)

# run the scheduler
while True:
    schedule.run_pending()
    # check for pending jobs every second
    time.sleep(60)
```

## Step 6: Put Everything Together

Here's the complete code:

```python Python expandable theme={"dark"}
# pip3 install requests, schedule
import requests
import json
import re
from datetime import datetime, timezone
import logging
import os
import schedule
import time

# specify the CSS selectors for extracting data
css_extractor = json.dumps(
    {
        "name": "span#productTitle",
        "price": "span.aok-offscreen",
        "ratings": "#acrPopover",
        "reviewCount": "#acrCustomerReviewText",
        "demandHistory": "#social-proofing-faceout-title-tk_bought",
        "availability": "#availability",
    }
)

# define the scraper function
def scraper(url, css_extractor):
    apikey = "YOUR_ZENROWS_API_KEY"
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "css_extractor": css_extractor,
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.json()

# treat specific fields as a list for consistency
def consistency_handler(field):
    if not field == "":
        if isinstance(field, str):
            return [field]
        elif isinstance(field, list):
            return field
    else:
        return []

# parse price information from the price list
def parse_price_info(price_list):
    # apply the consistency_handler function to treat as a list
    price_list = consistency_handler(price_list)
    original_price = None
    current_price = None
    discount_percent = None
    currency = "USD"
    # obtain price and discount information
    for item in price_list:
        price_match = re.search(r"\$(\d+[\.,]?\d*)", item)
        percent_match = re.search(r"(\d+)\s*percent", item)
        if "with" in item and price_match and percent_match:
            current_price = float(price_match.group(1).replace(",", ""))
            discount_percent = int(percent_match.group(1))
        elif "New Condition Price" in item and price_match:
            original_price = float(price_match.group(1).replace(",", ""))
        elif price_match and not current_price:
            original_price = float(price_match.group(1).replace(",", ""))
    if current_price is None:
        current_price = original_price
    return original_price, current_price, discount_percent, currency

# parse review count
def parse_review_count(review_list):
    # apply the consistency_handler function to treat as a list
    review_list = consistency_handler(review_list)
    if review_list:
        # Remove any non-digit characters (parentheses, commas, etc.)
        import re
        count_str = re.sub(r"[^\d]", "", review_list[0])
        if count_str:
            return int(count_str)
    return None

# parse average rating
def parse_rating(ratings_list):
    # apply the consistency_handler function to treat as a list
    ratings_list = consistency_handler(ratings_list)
    if ratings_list:
        try:
            return float(ratings_list[0])
        except Exception:
            return None
    return None

# parse units sold from demand history
def parse_units_sold(demand_history):
    if not demand_history:
        return None
    # apply the consistency_handler function to treat as a list
    demand_history = consistency_handler(demand_history)
    match = re.search(r"(\d+[+]?)(?= bought)", demand_history[0])
    if match:
        return match.group(1)
    return None

# parse availability
def parse_availability(availability):
    # apply the consistency_handler function to treat as a list
    availability = consistency_handler(availability)
    for item in availability:
        if "In Stock" in item:
            return "In Stock"
    return None

# parse the cleaned data
def parse_cleaned_data(raw_data):
    original_price, current_price, discount_percent, currency = parse_price_info(
        raw_data.get("price", [])
    )
    average_rating = parse_rating(raw_data.get("ratings", []))
    review_count = parse_review_count(raw_data.get("reviewCount", []))
    units_sold = parse_units_sold(raw_data.get("demandHistory", ""))
    availability = parse_availability(raw_data.get("availability", []))
    # get the scraping timestamp
    scrape_timestamp = datetime.now(timezone.utc).isoformat()

    cleaned_data = {
        "name": raw_data.get("name", ""),
        "original_price": original_price,
        "current_price": current_price,
        "discount_percent": discount_percent,
        "currency": currency,
        "average_rating": average_rating,
        "review_count": review_count,
        "units_sold": units_sold,
        "availability": availability,
        "scrape_timestamp": scrape_timestamp,
    }
    return cleaned_data

# set up logging
logging.basicConfig(filename="scraper.log", level=logging.INFO)

# update the price history for each product in the JSON file
def update_and_save_price_history(new_product, filename="product_history.json"):
    # load existing history
    if os.path.exists(filename):
        with open(filename, "r", encoding="utf-8") as json_file:
            product_history = json.load(json_file)
    else:
        product_history = []

    # index products by name for easy lookup
    history_dict = {product["name"]: product for product in product_history}

    # create a new price record
    price_record = {
        "timestamp": new_product["scrape_timestamp"],
        "current_price": new_product["current_price"],
        "original_price": new_product["original_price"],
        "discount_percent": new_product["discount_percent"],
        "currency": new_product["currency"],
    }

    # create a flag to indicate if an alert was triggered
    alert_triggered = False

    # check for price changes
    if new_product["name"] in history_dict:
        price_history = history_dict[new_product["name"]]["price_history"]
        if price_history:
            last_price = price_history[-1]["current_price"]
            if new_product["current_price"] != last_price:
                alert_triggered = True
                logging.info(
                    f"ALERT: Price changed for {new_product['name']} from {last_price} to {new_product['current_price']}"
                )

        # append to existing price history
        price_history.append(price_record)
        # update ratings, review count, and units sold
        history_dict[new_product["name"]]["average_rating"] = new_product.get(
            "average_rating"
        )
        history_dict[new_product["name"]]["review_count"] = new_product.get(
            "review_count"
        )
        history_dict[new_product["name"]]["units_sold"] = new_product.get("units_sold")
        history_dict[new_product["name"]]["availability"] = new_product.get(
            "availability"
        )
    else:
        # create a new product entry with price history
        history_dict[new_product["name"]] = {
            "name": new_product["name"],
            "price_history": [price_record],
            "average_rating": new_product.get("average_rating"),
            "review_count": new_product.get("review_count"),
            "units_sold": new_product.get("units_sold"),
        }

    # write updated history back to the JSON file
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(list(history_dict.values()), f, ensure_ascii=False, indent=2)

    return alert_triggered

# schedule the scraping job
def job():
    logging.info(f"scraping job started at {datetime.now(timezone.utc).isoformat()}")
    product_url = "https://www.amazon.com/dp/B001CXYMFS"

    raw_data = scraper(product_url, css_extractor)
    result = parse_cleaned_data(raw_data)
    alert_triggered = update_and_save_price_history(result)
    logging.info(f"scraping job stopped at {datetime.now(timezone.utc).isoformat()}")
    if alert_triggered:
        logging.info("ALERT: Price change detected!")
    else:
        logging.info("No price change detected.")

# schedule the job to run every 2 minutes
schedule.every(2).minutes.do(job)

# run the scheduler
while True:
    schedule.run_pending()
    # check for pending jobs every second
    time.sleep(60)
```

The code initiates a scheduled scraping process, extracts price data every 30 seconds, and mocks an alert for price changes. Below is a sample of data for two schedules from the `price_history` JSON file:

```json JSON theme={"dark"}
[
    {
        "name": "Thrustmaster T-Flight Hotas X (Compatible with PC)",
        "price_history": [
            {
                "timestamp": "2025-08-08T08:37:59.510891+00:00",
                "current_price": 65.79,
                "original_price": 69.99,
                "discount_percent": 6,
                "currency": "USD",
            },
            {
                "timestamp": "2025-08-08T09:07:59.510891+00:00",
                "current_price": 65.79,
                "original_price": 69.99,
                "discount_percent": 6,
                "currency": "USD",
            },
        ],
        "average_rating": 4.4,
        "review_count": 8067,
        "units_sold": "500+",
        "availability": "In Stock",
    }
]
```

Here's a sample log to track price changes and scraping intervals for the two records:

```bash theme={"dark"}
INFO:root:scraping job started at 2025-08-08T08:37:37.287507+00:00
INFO:root:scraping job stopped at 2025-08-08T08:37:59.511894+00:00
INFO:root:No price change detected.

INFO:root:scraping job started at 2025-08-08T09:07:37.287507+00:00
INFO:root:scraping job stopped at 2025-08-08T09:07:59.511894+00:00
INFO:root:No price change detected.
```

Congratulations! 🎉 You've built a price monitoring system using Zenrows as the scraping solution.

## Next Steps: Enhance Your System

### Add Email Notifications

Set up email alerts when prices change:

```python Python theme={"dark"}
import smtplib
from email.mime.text import MIMEText

def send_price_alert(product_name, old_price, new_price):
    """Send email notification for price changes"""
    msg = MIMEText(f"Price changed for {product_name}: ${old_price} → ${new_price}")
    msg['Subject'] = f'Price Alert: {product_name}'
    msg['From'] = 'your-email@gmail.com'
    msg['To'] = 'recipient@gmail.com'
    
    # Configure your email server
    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.starttls()
    server.login('your-email@gmail.com', 'your-password')
    server.send_message(msg)
    server.quit()
```

### Monitor Multiple Products

Track multiple products by modifying the monitoring function:

```python Python theme={"dark"}
def monitor_multiple_products():
    """Monitor multiple products"""
    products = [
        "https://www.amazon.com/dp/B001CXYMFS",
        "https://www.amazon.com/dp/B08N5WRWNW",
        # Add more product URLs
    ]
    
    for url in products:
        try:
            raw_data = scrape_product_data(url, css_extractor)
            cleaned_data = clean_product_data(raw_data)
            price_changed = save_price_history(cleaned_data)
            
            if price_changed:
                logging.info(f"Price alert for {cleaned_data['name']}")
        except Exception as e:
            logging.error(f"Failed to monitor {url}: {e}")
```

### Database Integration

Replace JSON storage with a database for better performance:

```python Python theme={"dark"}
import sqlite3

def init_database():
    """Initialize SQLite database"""
    conn = sqlite3.connect('price_monitor.db')
    cursor = conn.cursor()
    
    cursor.execute('''
        CREATE TABLE IF NOT EXISTS products (
            id INTEGER PRIMARY KEY,
            name TEXT UNIQUE,
            current_price REAL,
            average_rating REAL,
            review_count INTEGER,
            availability TEXT,
            last_updated TIMESTAMP
        )
    ''')
    
    cursor.execute('''
        CREATE TABLE IF NOT EXISTS price_history (
            id INTEGER PRIMARY KEY,
            product_id INTEGER,
            price REAL,
            timestamp TIMESTAMP,
            FOREIGN KEY (product_id) REFERENCES products (id)
        )
    ''')
    
    conn.commit()
    conn.close()
```

## Troubleshooting

### Missing Data

* Enable `js_render` for dynamic content
* Increase `wait` time for slow-loading pages or consider switching to `wait_for`
* Verify if the CSS selectors are correct

### Rate Limiting

* Use `premium_proxy` for residential IPs
* Add delays between requests
* Monitor your API usage

### Selector Changes

* Test selectors regularly
* Use more stable selectors when possible
* Implement fallback selectors

## Best Practices

* **Monitor selector stability**: Check if your CSS selectors still work monthly
* **Handle errors gracefully**: Always include try-catch blocks for network requests
* **Log everything**: Comprehensive logging helps debug issues
* **Start small**: Begin with one product before scaling to multiple products
* **Respect rate limits**: Don't overwhelm target websites with too many requests


# Capture Network Requests (XHR/Fetch)
Source: https://docs.zenrows.com/zenrows-academy/capturing-network-requests

Intercept XHR and Fetch API network requests to extract structured JSON data from JavaScript-heavy dynamic websites using Zenrows.

Many modern websites load data dynamically through background API calls rather than including everything in the initial HTML. These XHR (XMLHttpRequest) and Fetch requests often contain the exact data you need in a clean JSON format, making them more reliable than scraping HTML elements.

This guide shows you how to identify and capture these network requests using two Zenrows approaches: Fetch for simple integration, and Browser Sessions for advanced control.

<Warning>
  While Zenrows can capture XHR/Fetch requests during page rendering, this feature is designed as a supplementary capability rather than our core offering. Zenrows specializes in HTML content extraction and JavaScript rendering for traditional web scraping scenarios.

  This guide demonstrates the capability for educational purposes and specific use cases where understanding background API calls enhances your web scraping strategy.
</Warning>

## Who is this for?

This guide is for developers who need to extract data from websites that load content dynamically through JavaScript and API calls, such as infinite scroll pages, search results, or real-time data feeds.

## What you'll learn

* Identify XHR/Fetch requests using browser developer tools
* Capture network requests with Fetch
* Use Browser Sessions to intercept and analyze API calls

## Understanding Network Requests

The first step is understanding which network requests load the data you need. We'll use Chrome's Developer Tools to monitor network activity and identify the relevant API calls.

Let's use a practical example with a "Load More" page:

<Steps>
  <Step title="Open Developer Tools">
    1. Navigate to [https://www.scrapingcourse.com/button-click](https://www.scrapingcourse.com/button-click)
    2. Right-click anywhere on the page and select **Inspect**
    3. Go to **Network** → **Fetch/XHR**

    <Frame>
      <img alt="Chrome Network Inspection" />
    </Frame>
  </Step>

  <Step title="Trigger Network Activity">
    1. Click the **Load More** button on the site to trigger a data-loading event
    2. Observe the changes in Fetch/XHR calls
    3. Each click calls a `products?offset` API as shown in the image below
    4. Click one of the offset requests to view the API endpoint that was called

    <Frame>
      <img alt="Load More API Network" />
    </Frame>
  </Step>

  <Step title="Analyze the API Endpoint">
    The network inspection shows that the requested API endpoint is: `https://www.scrapingcourse.com/ajax/products?offset=10`

    Opening this URL directly in your browser returns the raw data without visual design:

    <Frame>
      <img alt="API Call No Visual Design" />
    </Frame>

    The `offset` query parameter determines which set of data is loaded. For example:

    * `offset=0` loads the initial data visible in the viewport
    * `offset=10` loads the next set of products
    * `offset=50` loads data for the sixth scroll position
  </Step>
</Steps>

<Note>Network request behavior can vary significantly between websites. Some endpoints may return HTML pages, while others provide JSON or different data formats. Always inspect the specific network responses of your target site to determine the appropriate handling method.</Note>

## Method 1: Fetch

Fetch provides the simplest way to capture network requests. Use the `json_response` parameter to automatically capture all XHR/Fetch requests made during page rendering, combined with `js_instructions` to trigger the necessary interactions.

```python Python theme={"dark"}
# pip install requests
import requests

url = 'https://www.scrapingcourse.com/button-click'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
	'js_render': 'true',
	'json_response': 'true', # Capture network requests made during page render
	'js_instructions': """[
        {"wait_for":"#load-more-btn"},
        {"click":"#load-more-btn"},
        {"wait":1000}
    ]""",
	'premium_proxy': 'true',
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
```

<Note>You can find more information about the JavaScript Instructions parameter in the [JavaScript Instructions](/fetch/features/js-instructions) and JSON Response parameters in the [JSON Response](/fetch/features/json-response) documentation pages.</Note>

Response example:

```json JSON Response theme={"dark"}
{
  "html": "<html><body><h1>..... HTML content .....</h1></body></html>",
  "xhr": [
    ... other XHR requests ...
    {
      "url": "https://www.scrapingcourse.com/ajax/products?offset=10",
      "body": "<div class=\"product-item flex flex-col items-center rounded-lg\">...</div>...</body></html>",
      "status_code": 200,
      "method": "GET",
      "headers": {
        headers ...
      },
      "request_headers": {
        "Referer": "https://www.scrapingcourse.com/button-click",
        request_headers ...
      }
    }
  ],
  "js_instructions_report": { ... other JavaScript Instructions report ... }
}
```

## Method 2: Browser Sessions

Browser Sessions provides more control over network request capture and allows you to process requests in real-time as they occur. It is recommended to use Browser Sessions for more complex scraping tasks or when Fetch is not enough.

<CodeGroup>
  ```javascript Puppeteer expandable theme={"dark"}
  import { connect } from 'puppeteer-core';

  async function captureNetworkRequests(url) {
      const browser = await connect({
          browserWSEndpoint: 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY'
      });

      const page = await browser.newPage();

      // Array to store captured requests
      const capturedRequests = [];

      // Intercept network requests
      await page.setRequestInterception(true);

      page.on('request', (request) => {
          // Log outgoing requests
          console.log(`Request: ${request.method()} ${request.url()}`);
          request.continue();
      });

      page.on('response', async (response) => {
          const url = response.url();
          const method = response.request().method();
          const status = response.status();

          // Filter for XHR/Fetch requests (adjust as needed)
          if (
              url.includes('api') || url.includes('ajax') ||
              response.request().resourceType() === 'xhr' ||
              response.request().resourceType() === 'fetch'
          ) {
              try {
                  const responseBody = await response.text();
                  const headers = response.headers();
                  const requestHeaders = response.request().headers();

                  capturedRequests.push({
                      url: url,
                      method: method,
                      status_code: status,
                      body: responseBody,
                      headers: headers,
                      request_headers: requestHeaders,
                      resource_type: response.request().resourceType()
                  });

                  console.log(`Captured: ${method} ${url} (${status})`);
              } catch (error) {
                  console.error(`Error capturing response from ${url}:`, error);
              }
          }
      });

      try {
          // Navigate to the page with increased timeout
          await page.goto(url, { waitUntil: 'networkidle2', timeout: 60000 });

          // Wait for the "load more" button to be visible
          await page.waitForSelector('#load-more-btn', { visible: true, timeout: 30000 });

          // Wait for the AJAX request after clicking the button
          const [ajaxResponse] = await Promise.all([
              page.waitForResponse(
                  response =>
                      response.url().includes('/ajax/products?offset=10') &&
                      response.status() === 200,
                  { timeout: 20000 }
              ),
              page.click('#load-more-btn')
          ]);

          // Optionally: Log the AJAX response body directly
          const ajaxBody = await ajaxResponse.text();
          console.log('AJAX Response Body:', ajaxBody);

          // Wait a bit for any additional network activity
          await page.waitForTimeout(3000);

          // Get the final HTML
          const html = await page.content();

          return {
              html: html,
              xhr_requests: capturedRequests
          };
      } catch (error) {
          console.error('Error during capture:', error);
          throw error;
      } finally {
          await browser.close();
      }
  }

  // Usage
  captureNetworkRequests('https://www.scrapingcourse.com/button-click')
      .then(result => {
          console.log(`Captured ${result.xhr_requests.length} network requests`);

          result.xhr_requests.forEach((request, index) => {
              console.log(`\nRequest ${index + 1}:`);
              console.log(`URL: ${request.url}`);
              console.log(`Method: ${request.method}`);
              console.log(`Status: ${request.status_code}`);

              // Try to parse JSON response
              try {
                  const jsonData = JSON.parse(request.body);
                  console.log('JSON Response:', JSON.stringify(jsonData, null, 2));
              } catch (e) {
                  console.log('Raw Response:', request.body.substring(0, 200) + '...');
              }

              // Special print for the request you care about
              if (request.url.includes('/ajax/products?offset=10')) {
                  console.log('\n*** This is the AJAX call you wanted! ***');
                  console.log('Full internals:');
                  console.log(JSON.stringify(request, null, 2));
              }
          });
      })
      .catch(error => {
          console.error('Error:', error);
      });
  ```

  ```javascript Playwright expandable theme={"dark"}
  const { chromium } = require('playwright'); // or 'playwright-core'

  async function captureNetworkRequests(url) {
      const browser = await chromium.connectOverCDP('wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY');

      const context = await browser.newContext();
      const page = await context.newPage();

      const capturedRequests = [];

      // Listen for all responses
      page.on('response', async (response) => {
          try {
              const req = response.request();
              const url = response.url();
              const method = req.method();
              const status = response.status();
              const resourceType = req.resourceType();

              // Filter for XHR/Fetch requests (adjust as needed)
              if (
                  url.includes('api') || url.includes('ajax') ||
                  resourceType === 'xhr' || resourceType === 'fetch'
              ) {
                  let responseBody;
                  try {
                      responseBody = await response.text();
                  } catch (e) {
                      responseBody = '[Could not retrieve body]';
                  }

                  capturedRequests.push({
                      url: url,
                      method: method,
                      status_code: status,
                      body: responseBody,
                      headers: response.headers(),
                      request_headers: req.headers(),
                      resource_type: resourceType
                  });

                  console.log(`Captured: ${method} ${url} (${status})`);
              }
          } catch (err) {
              console.error('Error in response handler:', err);
          }
      });

      try {
          // Go to the page
          await page.goto(url, { waitUntil: 'networkidle', timeout: 60000 });

          // Wait for the button to be visible
          await page.waitForSelector('#load-more-btn', { state: 'visible', timeout: 30000 });

          // Wait for the AJAX request after clicking the button
          const [ajaxResponse] = await Promise.all([
              page.waitForResponse(
                  response =>
                      response.url().includes('/ajax/products?offset=10') &&
                      response.status() === 200,
                  { timeout: 20000 }
              ),
              page.click('#load-more-btn')
          ]);

          // Optionally: Log the AJAX response body directly
          const ajaxBody = await ajaxResponse.text();
          console.log('AJAX Response Body:', ajaxBody);

          // Wait a bit for any additional network activity
          await page.waitForTimeout(3000);

          // Get the final HTML
          const html = await page.content();

          await browser.close();

          return {
              html: html,
              xhr_requests: capturedRequests
          };
      } catch (error) {
          await browser.close();
          console.error('Error during capture:', error);
          throw error;
      }
  }

  // Usage
  captureNetworkRequests('https://www.scrapingcourse.com/button-click')
      .then(result => {
          console.log(`Captured ${result.xhr_requests.length} network requests`);

          result.xhr_requests.forEach((request, index) => {
              console.log(`\nRequest ${index + 1}:`);
              console.log(`URL: ${request.url}`);
              console.log(`Method: ${request.method}`);
              console.log(`Status: ${request.status_code}`);

              // Try to parse JSON response
              try {
                  const jsonData = JSON.parse(request.body);
                  console.log('JSON Response:', JSON.stringify(jsonData, null, 2));
              } catch (e) {
                  console.log('Raw Response:', request.body.substring(0, 200) + '...');
              }

              // Special print for the request you care about
              if (request.url.includes('/ajax/products?offset=10')) {
                  console.log('\n*** This is the AJAX call you wanted! ***');
                  console.log('Full internals:');
                  console.log(JSON.stringify(request, null, 2));
              }
          });
      })
      .catch(error => {
          console.error('Error:', error);
      });
  ```

  ```python Pyppeteer expandable theme={"dark"}
  import asyncio
  from pyppeteer import connect

  async def capture_network_requests(url):
      browser = await connect(
          browserWSEndpoint='wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY'
      )
      page = await browser.newPage()
      captured_requests = []

      # Intercept network requests
      await page.setRequestInterception(True)

      async def on_request(request):
          print(f"Request: {request.method} {request.url}")
          await request.continue_()

      async def on_response(response):
          req = response.request
          url_ = response.url
          method = req.method
          status = response.status
          resource_type = req.resourceType

          # Filter for XHR/Fetch requests
          if (
              'api' in url_ or 'ajax' in url_ or
              resource_type in ['xhr', 'fetch']
          ):
              try:
                  responseBody = await response.text()
              except Exception:
                  responseBody = '[Could not retrieve body]'
              headers = response.headers
              request_headers = req.headers

              captured_requests.append({
                  'url': url_,
                  'method': method,
                  'status_code': status,
                  'body': responseBody,
                  'headers': headers,
                  'request_headers': request_headers,
                  'resource_type': resource_type
              })
              print(f"Captured: {method} {url_} ({status})")

      page.on('request', on_request)
      page.on('response', on_response)

      try:
          # Navigate to the page with increased timeout
          await page.goto(url, waitUntil='networkidle2', timeout=60000)

          # Wait for the "load more" button to be visible
          await page.waitForSelector('#load-more-btn', visible=True, timeout=30000)

          # Wait for the AJAX request after clicking the button
          async def ajax_predicate(response):
              return (
                  '/ajax/products?offset=10' in response.url and
                  response.status == 200
              )

          wait_for_ajax = page.waitForResponse(ajax_predicate, timeout=20000)
          await page.click('#load-more-btn')
          ajax_response = await wait_for_ajax

          ajax_body = await ajax_response.text()
          print('AJAX Response Body:', ajax_body)

          # Wait a bit for any additional network activity
          await asyncio.sleep(3)

          # Get the final HTML
          html = await page.content()

          await browser.disconnect()

          return {
              'html': html,
              'xhr_requests': captured_requests
          }
      except Exception as e:
          await browser.disconnect()
          print('Error during capture:', e)
          raise

  # Usage
  async def main():
      result = await capture_network_requests('https://www.scrapingcourse.com/button-click')
      print(f"Captured {len(result['xhr_requests'])} network requests")

      for idx, request in enumerate(result['xhr_requests'], 1):
          print(f"\nRequest {idx}:")
          print(f"URL: {request['url']}")
          print(f"Method: {request['method']}")
          print(f"Status: {request['status_code']}")

          # Try to parse JSON response
          try:
              import json
              jsonData = json.loads(request['body'])
              print('JSON Response:', json.dumps(jsonData, indent=2))
          except Exception:
              print('Raw Response:', request['body'][:200] + '...')

          # Special print for the request you care about
          if '/ajax/products?offset=10' in request['url']:
              print('\n*** This is the AJAX call you wanted! ***')
              print('Full internals:')
              import pprint
              pprint.pprint(request)

  if __name__ == '__main__':
      asyncio.get_event_loop().run_until_complete(main())

  ```
</CodeGroup>

## Best Practices

**Request Filtering**:

* Filter by URL patterns to capture only relevant API calls
* Use resource type filtering to focus on XHR/Fetch requests
* Monitor specific endpoints that contain the data you need

**Performance Optimization**:

* Set appropriate `wait` times to ensure all requests complete
* Use targeted selectors in `wait_for` instructions
* Limit capture duration to avoid unnecessary data collection

**Data Processing**

* Parse JSON responses when possible for structured data
* Handle different response formats (JSON, XML, HTML)
* Store request metadata (headers, status codes) for debugging

**Error Handling**:

* Implement retry logic for failed network captures
* Validate captured data before processing
* Log network errors for troubleshooting

## Troubleshooting

**No XHR requests captured**:

* Ensure `js_render` is enabled
* Increase `wait` times to allow requests to complete
* Check if the target element exists and is clickable
* Verify that JavaScript is required to load the content

**Missing authentication in captured requests**:

* Use `custom_headers` parameter with Fetch
* Set headers before navigation with Browser Sessions
* Check if the site requires cookies or session tokens

**Incomplete request data**:

* Increase the `wait` parameter to allow all requests to finish
* Use `wait_for` with specific selectors instead of fixed timeouts
* Monitor the network tab to understand the request timing

**Request filtering not working**:

* Adjust URL pattern matching for your specific target site
* Check the `resource_type` of requests you want to capture
* Use broader filters initially, then narrow down based on results


# Extract Property Data
Source: https://docs.zenrows.com/zenrows-academy/extract-property-data

Scrape detailed property data from individual Zillow listing pages using Zenrows CSS selectors and scale to process multiple properties.

Extract detailed property information from individual Zillow property pages using Zenrows' Fetch. This tutorial shows you how to scrape specific data fields from single properties and scale up to process multiple listings efficiently.

## What you'll learn

* Extract specific property data using CSS selectors
* Set up scraping requests with anti-bot bypass and JavaScript rendering
* Clean and structure scraped data for analysis
* Handle complex data fields like price history
* Scale from single property to multiple property extraction
* Store property data in structured formats

## Why Extract Property Data

Real estate professionals need detailed property information for market analysis, investment decisions, and lead generation. Extracting structured data from property listings enables:

### Market analysis

* Compare property specifications across multiple listings
* Track pricing trends and market conditions
* Identify investment opportunities based on specific criteria

### Investment research

* Access comprehensive property details for due diligence
* Analyze property features, pricing history, and market positioning
* Build databases for portfolio management

### Automated workflows

* Integrate property data into existing business systems
* Create alerts for properties matching specific criteria
* Generate reports and analytics from structured data

## Step 1: Set Up Your Basic Scraping Function

Start by creating a scraping function that handles Zillow's anti-bot measures. You'll need [JavaScript Rendering](/fetch/features/js-rendering) to process dynamic content and [Premium Proxies](/fetch/features/premium-proxy) to avoid IP blocking.

```python theme={"dark"}
# pip install requests
import requests

def scraper(url):
    apikey = "YOUR_ZENROWS_API_KEY"

    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",  # Processes JavaScript-generated content
        "premium_proxy": "true",  # Uses residential IP addresses
        "proxy_country": "us",  # Routes requests through US-based IPs
        "wait": "2000",  # Wait 2 seconds for elements to load
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.text
```

This function returns the website's HTML content. The js\_render parameter enables JavaScript processing (essential for dynamic websites like Zillow), while premium\_proxy provides residential IP addresses that appear as regular user traffic.

<Tip>
  The `proxy_country` parameter is optional. If you don't specify it, Zenrows will use a random IP address from anywhere in the world. Learn more about geolocation [here](/fetch/features/proxy-country).
</Tip>

## Step 2: Test With a Single Property URL

Before scaling to multiple properties, test your scraper with a single Zillow property page. Use Zenrows' `css_extractor` feature to automatically extract specific data fields.

Create CSS selectors to target the property data you need:

```python Python theme={"dark"}
import json

# Property page CSS extractor
property_css_extractor = json.dumps(
    {
        "price": "span[data-testid='price']",
        "location": "ul.footer-breadcrumbs a, ul.footer-breadcrumbs strong",
        "dimension": "div[data-testid='bed-bath-sqft-facts']",
        "description": "div[data-testid='description']",
        "listed_by": "div[data-testid='seller-attribution']",
        "price_change_dates": "span[data-testid='date-info']",
        "price_changes": "td[data-testid='price-money-cell']",
    }
)
```

<Warning>CSS selectors can change when websites update their code. To maintain a reliable scraper, monitor your selectors regularly and update them as needed. Learn more about CSS selectors [here](/fetch/features/css-extractor).</Warning>

Update your scraper function to handle CSS extraction and return JSON data:

```python Python theme={"dark"}
# pip install requests
import requests
import json

def scraper(url, css_extractor=None):
    apikey = "YOUR_ZENROWS_API_KEY"
    
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "wait": "2000",
        "css_extractor": css_extractor,
    }
    
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.json()  # Return JSON when using CSS extractor

# Property page CSS extractor
property_css_extractor = json.dumps(
    {
        "price": "span[data-testid='price']",
        "location": "ul.footer-breadcrumbs a, ul.footer-breadcrumbs strong",
        "dimension": "div[data-testid='bed-bath-sqft-facts']",
        "description": "div[data-testid='description']",
        "listed_by": "div[data-testid='seller-attribution']",
        "price_change_dates": "span[data-testid='date-info']",
        "price_changes": "td[data-testid='price-money-cell']",
    }
)

# Test with a single property
property_url = "https://www.zillow.com/homedetails/130-3rd-St-California-PA-15419/49745481_zpid/"

property_data = scraper(
    property_url,
    css_extractor=property_css_extractor,
)

print(json.dumps(property_data, indent=2))
```

This returns structured data for the property. Here's what a typical response looks like:

```json JSON Response theme={"dark"}
{
    "price": "$142,500",
    "location": "130 3rd St, California, PA 15419",
    "dimension": "4 beds • 2 baths • 2,764 sqft",
    "description": "Sprawling single family home...only missing you!",
    "listed_by": "Listed by: Thomas Althoff 724-933-6300, RE/MAX SELECT REALTY",
    "price_change_dates": ["7/29/2025", "Listed for sale"],
    "price_changes": ["$142,500+9.7%$52/sqft"]
}
```

## Step 3: Scale to Multiple Properties

Once your single property extraction works reliably, scale up to process multiple property URLs. Here are some example Zillow property URLs you can use for testing:

```python Python theme={"dark"}
# Example property URLs from Zillow
property_urls = [
    "https://www.zillow.com/homedetails/130-3rd-St-California-PA-15419/49745481_zpid/",
    "https://www.zillow.com/homedetails/882-Highpoint-Dr-Coal-Center-PA-15423/2087529083_zpid/",
    "https://www.zillow.com/homedetails/508-5th-St-California-PA-15419/49745256_zpid/",
    "https://www.zillow.com/homedetails/721-Spring-St-Roscoe-PA-15477/49794732_zpid/"
]

all_property_data = []

# Extract data from each property
for property_url in property_urls:
    try:
        properties = scraper(
            property_url,
            css_extractor=property_css_extractor,
        )
        
        # Clean and structure the data
        cleaned_properties = clean_property_data(properties)
        
        # Add the URL for reference
        cleaned_properties["source_url"] = property_url
        
        all_property_data.append(cleaned_properties)
        
        print(f"Successfully extracted data from: {property_url}")
        
    except Exception as e:
        print(f"Error extracting data from {property_url}: {str(e)}")
        continue

print(f"Successfully extracted data for {len(all_property_data)} properties")
```

Save the property data in JSON format to preserve the nested structure:

```python Python theme={"dark"}
# Store the data as JSON
with open("properties.json", "w", encoding="utf-8") as f:
    json.dump(all_property_data, f, ensure_ascii=False, indent=2)

print(f"Data saved to properties.json")
```

For flat data structures, you can also export to CSV:

```python Python theme={"dark"}
import pandas as pd

# Convert to DataFrame and save as CSV (flattens nested data)
df = pd.json_normalize(all_property_data)
df.to_csv("properties.csv", index=False)

print("Data saved to properties.csv")
```

## Step 4: Complete Implementation

Here's the complete code that extracts property data from multiple URLs:

```python Python expandable theme={"dark"}
# pip install requests pandas
import requests
import json
import pandas as pd

def scraper(url, css_extractor=None):
    apikey = "YOUR_ZENROWS_API_KEY"
    
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "wait": "2000",
        "css_extractor": css_extractor,
    }
    
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.json()

def clean_property_data(properties):
    """Clean and structure property data"""
    
    # Clean the "listed_by" field
    listed_by = properties.get("listed_by")
    if listed_by:
        properties["listed_by"] = listed_by.replace("Listed by:", "").strip()
    
    # Process and merge price history data
    price_history_dates = properties.get("price_change_dates")
    price_changes = properties.get("price_changes")

    if price_history_dates and price_changes:
        combined_price_history = []
        
        for i in range(0, len(price_history_dates), 2):
            date = price_history_dates[i]
            event = (
                price_history_dates[i + 1] if i + 1 < len(price_history_dates) else ""
            )
            price = price_changes[i // 2] if i // 2 < len(price_changes) else ""
            
            combined_price_history.append({
                "Date": date,
                "Event": event,
                "Price": price
            })
        
        properties["price_history"] = combined_price_history

        # Remove the original fields after combining
        for key in ["price_change_dates", "price_changes"]:
            if key in properties:
                del properties[key]
    
    return properties

# CSS extractor for property data
property_css_extractor = json.dumps(
    {
        "price": "span[data-testid='price']",
        "location": "ul.footer-breadcrumbs a, ul.footer-breadcrumbs strong",
        "dimension": "div[data-testid='bed-bath-sqft-facts']",
        "description": "div[data-testid='description']",
        "listed_by": "div[data-testid='seller-attribution']",
        "price_change_dates": "span[data-testid='date-info']",
        "price_changes": "td[data-testid='price-money-cell']",
    }
)

# Example property URLs
property_urls = [
    "https://www.zillow.com/homedetails/130-3rd-St-California-PA-15419/49745481_zpid/",
    "https://www.zillow.com/homedetails/882-Highpoint-Dr-Coal-Center-PA-15423/2087529083_zpid/",
    "https://www.zillow.com/homedetails/508-5th-St-California-PA-15419/49745256_zpid/",
    "https://www.zillow.com/homedetails/721-Spring-St-Roscoe-PA-15477/49794732_zpid/"
]

all_property_data = []

# Extract data from each property
for property_url in property_urls:
    try:
        properties = scraper(
            property_url,
            css_extractor=property_css_extractor,
        )
        
        # Clean and structure the data
        cleaned_properties = clean_property_data(properties)
        
        # Add the URL for reference
        cleaned_properties["source_url"] = property_url
        
        all_property_data.append(cleaned_properties)
        
        print(f"✓ Successfully extracted data from: {property_url}")
        
    except Exception as e:
        print(f"✗ Error extracting data from {property_url}: {str(e)}")
        continue

# Save the data
with open("properties.json", "w", encoding="utf-8") as f:
    json.dump(all_property_data, f, ensure_ascii=False, indent=2)

# Also save as CSV for easy analysis
df = pd.json_normalize(all_property_data)
df.to_csv("properties.csv", index=False)

print(f"\n🎉 Successfully extracted data for {len(all_property_data)} properties")
print("Data saved to properties.json and properties.csv")
```

Congratulations! 🎉 You've successfully extracted detailed property data from Zillow using Zenrows' web scraping capabilities.

## Data Management Best Practices

### Structure for your use case

Design your data structure to match your specific business needs rather than using a generic approach. For example, if you need to analyze price trends, structure price history as date/price pairs, not raw HTML. Map scraped fields directly to your business entities (property, agent, transaction) and use consistent, clear field names that work for your team.

### Validate before storage

Check for missing or malformed fields, unexpected data types, duplicate entries, and values outside the expected range. Use validation scripts or schema checks (such as Python's pydantic or JSON Schema). Automate this process to maintain consistency throughout your data extraction workflow.

### Preserve raw data

Store both cleaned and raw data separately. Raw data serves as a backup for debugging and reprocessing when requirements change.

### Choose the right storage format

Select the storage format that best fits your data and use case:

* **JSON**: Best for nested, hierarchical, or complex data structures. Human-readable and widely supported across platforms.
* **CSV**: Ideal for flat, tabular data. Easy to use in spreadsheets and analytics tools.
* **Databases (MongoDB, PostgreSQL)**: Suitable for large datasets that require frequent updates and querying.
* **Vector databases**: Designed for storing vectorized data, such as embeddings for Large Language Model consumption.

## Troubleshooting

### Missing data

**Solution 1**: Use adequate delay strategies to allow dynamic content to load. Increase the wait parameter value or add specific waits for elements that load asynchronously.

**Solution 2**: Verify you're using the correct CSS selectors. Test each selector using the Zenrows Request Playground before integrating it into your code. Create a monitoring strategy to spot site structural changes and isolate selectors from your codebase for easy debugging.

### CAPTCHA and anti-bot challenges

**Solution 1**: Ensure you're using anti-bot bypass parameters like js\_render and premium\_proxy.

**Solution 2**: If you continue encountering CAPTCHAs, integrate a CAPTCHA-solving service like 2Captcha through our solver integration options. Check our 2Captcha integration guide for implementation details.

**Solution 3**: Use fallback mechanisms or alternative pathways to prevent scraping failures.

### Rate limiting and geo-blocking

**Solution 1**: Use the premium\_proxy parameter to automatically rotate IP addresses.

**Solution 2**: Implement request retry mechanisms with exponential backoff delays between failed requests.

**Solution 3**: Add delays between requests to avoid overwhelming the target website.


# Scrape Property Listings
Source: https://docs.zenrows.com/zenrows-academy/extract-property-listings

Scrape Zillow property listings with Zenrows to extract URLs from search results while bypassing anti-bot protection and rendering JavaScript.

Extract property URLs and data from Zillow listing pages using Zenrows' Fetch. This tutorial covers setting up your scraper to handle Zillow's anti-bot protection, extracting property links from search results, and processing dynamic content.

## What you'll learn

* Set up scraping requests with anti-bot bypass and JavaScript rendering
* Extract property URLs from Zillow listing pages using CSS selectors
* Handle dynamic content loading and page scrolling
* Configure custom JavaScript instructions for reliable data extraction

## Why Scrape Real Estate Data

Real estate professionals need timely, comprehensive market data to make informed decisions. Web scraping enables automated data collection that provides several advantages:

### Market intelligence

* Monitor new listings as they appear on property websites
* Track pricing trends across different neighborhoods and property types
* Identify properties that match specific investment criteria

### Investment analysis

* Access comprehensive property data for market research
* Compare property specifications across multiple listings
* Analyze market conditions in target areas

### Lead generation

* Identify potential investment opportunities from listing data
* Build databases of properties that meet client requirements
* Monitor competitor listings and market activity

## Step 1: Test With a Basic Scraping Setup

Start by creating a scraping function that handles Zillow's anti-bot measures. The site requires [JavaScript Rendering](/fetch/features/js-rendering) and [Premium Proxies](/fetch/features/premium-proxy) for reliable access.

```python Python theme={"dark"}
# pip install requests
import requests

def scraper(url):
    apikey = "YOUR_ZENROWS_API_KEY"

    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    return response.text
```

This function returns the website's HTML content. The `js_render` parameter enables JavaScript processing, while `premium_proxy` provides residential IP addresses to avoid blocking. Setting `proxy_country` to "us" positions requests from IP addresses based in the US.

<Tip>
  The `proxy_country` parameter is optional. If not specified, Zenrows will use a random IP address worldwide. See more about geolocation [here](/fetch/features/proxy-country).
</Tip>

## Step 2: Handle Dynamic Content

Parameters, such as `wait`, `wait_for`, and `js_instructions`, allow customizing requests to handle dynamic rendering.

Modify the request parameter with a 2-second generic delay (`wait`) to allow elements to load.

```python Python theme={"dark"}
#...
params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",
    "premium_proxy": "true",
    "proxy_country": "us",
    "wait": "2000",
}
#...
```

Add a scrolling logic using the `js_instructions` parameter. This helps capture property listings that extend beyond the viewport.

Also define the `js_instructions` separately as a stringified parameter. The instructions include an extra `wait` command. This adds a specific delay for elements to load after the scrolling action has completed.

```python Python theme={"dark"}
# ...
import json


#... scraper function

# custom JS instructions
listing_js_instructions = json.dumps(
    [
        {"evaluate": "window.scrollTo(0, document.body.scrollHeight);"},
        {"wait": 2000},
    ]
)
```

Use the same scraper function to scrape the listings and individual property pages. To make the function more customizable for each scenario, update it to accept an optional `js_instructions` parameter.

Here's the updated code:

```python Python theme={"dark"}
# pip install requests
import requests
import json

def scraper(
    url,
    js_instructions=None,
):
    apikey = "YOUR_ZENROWS_API_KEY"
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "js_instructions": js_instructions,
        "wait": "2000",
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)

    return response.text

# custom JS instructions
listing_js_instructions = json.dumps(
    [
        {"evaluate": "window.scrollTo(0, document.body.scrollHeight);"},
        {"wait": 2000},
    ]
)
```

## Step 3: Extract Property URLs from Listing Pages

Now you'll extract individual property page links from the Zillow listing. Use Zenrows' `css_extractor` feature to automatically pull these URLs from the page.

Here's a stringified format of the `css_extractor`:

```python Python theme={"dark"}
# listing page CSS extractor
listing_css_extractor = json.dumps(
    { "Links": "a[data-testid*=carousel][href]@href" }
)
```

<Warning>CSS selectors can change when websites update their code. To maintain a reliable scraper, monitor your selectors regularly and update them as needed. Learn more about CSS selectors [here](/fetch/features/css-extractor).</Warning>

Add an optional `css_extractor` parameter to the scraper function and specify it in the Zenrows `params`. Since we're extracting specific content, update the scraper function to return the data as JSON rather than plain text. Execute the function with the `listing_url`, `listing_css_extractor`, and `listing_js_instructions` as parameters.

Here's the updated code:

```python Python expandable theme={"dark"}
# pip install requests
import requests
import json

def scraper(
    url,
    js_instructions=None,
    css_extractor=None,
):
    apikey = "YOUR_ZENROWS_API_KEY"
    params = {
        "url": url,
        "apikey": apikey,
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "js_instructions": js_instructions,
        "wait": "2000",
        "css_extractor": css_extractor,
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)

    return response.json()

# custom JS instructions
listing_js_instructions = json.dumps(
    [
        {"evaluate": "window.scrollTo(0, document.body.scrollHeight);"},
        {"wait": 2000},
    ]
)

# listing page CSS extractor
listing_css_extractor = json.dumps(
    { "Links": "a[data-testid*=carousel][href]@href" }
)

listing_url = "https://www.zillow.com/districts/8494/california-area-school-district/"

property_urls = scraper(
    listing_url,
    js_instructions=listing_js_instructions,
    css_extractor=listing_css_extractor,
)["Links"]

# use a set to avoid duplicate URLs
property_urls = set(property_urls)
print(property_urls)
```

The above returns the URLs of each property on the listing page. See a sample response below:

```json JSON Response theme={"dark"}
{
    "https://www.zillow.com/homedetails/130-3rd-St-California-PA-15419/49745481_zpid/",
    "https://www.zillow.com/homedetails/882-Highpoint-Dr-Coal-Center-PA-15423/2087529083_zpid/",
    # ...,
    "https://www.zillow.com/homedetails/508-5th-St-California-PA-15419/49745256_zpid/",
    "https://www.zillow.com/homedetails/721-Spring-St-Roscoe-PA-15477/49794732_zpid/",
}
```

You now have a complete foundation for scraping Zillow property listings. Your scraper handles anti-bot protection, processes dynamic content, and extracts property URLs efficiently from any Zillow listing page.

## Next Steps

Once you have the property URLs, you can use them to extract detailed property data from individual listing pages.

Learn how to do this in our [Extract Property Data](/zenrows-academy/extract-property-data) tutorial.

## Data management best practices

<Steps>
  <Step title="Structure for your use case">
    Design your data structure to match your specific business needs, rather than using a generic approach. For example, if you need to analyze price trends, structure price history as date/price pairs, not raw HTML. Map scraped fields directly to your business entities (e.g., property, agent, transaction) and use consistent, clear field names that work for your team and future use.
  </Step>

  <Step title="Validate before storage">
    Check for missing or malformed fields, unexpected data types, duplicate entries, or values outside the expected range. Use validation scripts or schema checks (e.g., using Python’s pydantic or JSON Schema). Automate this process to maintain consistency throughout your data extraction workflow.
  </Step>

  <Step title="Preserve raw data">
    Store both cleaned and raw data separately. Raw data serves as a backup for debugging and reprocessing when requirements change.
  </Step>

  <Step title="Determine the storage format">
    Choose the storage format that best fits your data and use case. Common options include:

    * **JSON**: Best for nested, hierarchical, or complex data structures. Human-readable and widely supported across platforms and protocols.
    * **CSV**: Ideal for flat, tabular data. Easy to use in spreadsheets and many analytics tools.
    * **Databases (e.g., MongoDB, PostgreSQL)**: Suitable for large datasets that require frequent updates and querying.
    * **Vector databases**: Designed for storing vectorized data, such as embeddings for LLM (Large Language Model) consumption.

    Select the format that aligns with your workflow and future data needs.
  </Step>
</Steps>

## Troubleshooting

### Missing data

**Solution 1**: Employ adequate delay strategies to allow dynamic content to load. These include generic waits, waiting for specific elements, or pausing after scrolling or navigation.

**Solution 2**: If using `css_extractor`, check and ensure you've used the correct CSS selectors. Test each selector using the Zenrows Request Playground before integrating it into your codebase. Create a DOM monitoring strategy to spot site structural changes. Isolate selectors from your codebase for easy debugging and troubleshooting.

### CAPTCHA/anti-bot challenges

**Solution 1**: Ensure you've applied anti-bot bypass parameters like `js_render` and `premium_proxy`.

**Solution 2**: If you continue to be blocked by in-page CAPTCHAs or those attached to form fields, easily integrate a CAPTCHA-solving service like 2Captcha from our solver integration options. Check our 2Captcha integration guide for more information.

**Solution 3**: Use fallbacks or alternative pathways to avoid abrupt scraping failures.

### Rate limiting/geo-blocking

**Solution 1**: Ensure you use the `premium_proxy` parameter to automatically switch IPs.

**Solution 2**: Use request retry mechanisms, such as exponential backoff delay and retries between failed requests.


# CSS Selectors and HTML Parsing
Source: https://docs.zenrows.com/zenrows-academy/how-to-extract-data

Extract web data with Zenrows using three methods: CSS selectors for JSON output, output filters for transformation, and raw HTML parsing.

Zenrows provides multiple ways to extract and format data from web pages. You can use CSS Selectors for direct JSON extraction, apply output filters for data transformation, or retrieve raw HTML for custom processing.

This guide covers three main approaches to data extraction with Zenrows.

<Tabs>
  <Tab title="Using CSS Selectors">
    CSS Selectors are a query language for selecting HTML elements. When you enable the `css_extractor` parameter, Zenrows returns structured JSON data instead of raw HTML.

    Let's say you want to scrape the title from the <a href="https://www.scrapingcourse.com/ecommerce/">ScrapingCourse eCommerce page</a>. The title is contained in an `h1` tag.

    To extract it, send the `css_extractor` parameter with the value `{"title": "h1"}`. Make sure the parameter is properly encoded!

    <CodeGroup>
      ```python Python theme={"dark"}
      import requests

      api_key = "YOUR_ZENROWS_API_KEY"
      url = "https://www.scrapingcourse.com/ecommerce/"
      css_extractor = {"title": "h1"}

      response = requests.get(
          "https://api.zenrows.com/v1/",
          params={
              "apikey": api_key,
              "url": url,
              "css_extractor": css_extractor
          }
      )

      print(response.json())
      ```

      ```javascript JavaScript theme={"dark"}
      const axios = require("axios");

      const apiKey = "YOUR_ZENROWS_API_KEY";
      const url = "https://www.scrapingcourse.com/ecommerce/";
      const cssExtractor = {"title": "h1"};

      axios.get("https://api.zenrows.com/v1/", {
          params: {
              apikey: apiKey,
              url: url,
              css_extractor: cssExtractor
          }
      })
      .then(response => console.log(response.data))
      .catch(error => console.error(error));
      ```

      ```bash cURL theme={"dark"}
      curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%257B%2522title%2522%253A%2520%2522h1%2522%257D"
      ```
    </CodeGroup>

    This code sends a request to Zenrows with the CSS selector `h1` mapped to the key "title". Zenrows extracts the content from the first `h1` element and returns it as structured JSON data.

    ### Extracting Multiple Elements

    Now let's extract multiple elements. Add the product names using the selector `.product-name`:

    <CodeGroup>
      ```python Python theme={"dark"}
      import requests

      api_key = "YOUR_ZENROWS_API_KEY"
      url = "https://www.scrapingcourse.com/ecommerce/"
      css_extractor = {
          "title": "h1",
          "products": ".product-name"
      }

      response = requests.get(
          "https://api.zenrows.com/v1/",
          params={
              "apikey": api_key,
              "url": url,
              "css_extractor": css_extractor
          }
      )

      print(response.json())
      ```

      ```javascript JavaScript theme={"dark"}
      const axios = require("axios");

      const apiKey = "YOUR_ZENROWS_API_KEY";
      const url = "https://www.scrapingcourse.com/ecommerce/";
      const cssExtractor = {
          "title": "h1",
          "products": ".product-name"
      };

      axios.get("https://api.zenrows.com/v1/", {
          params: {
              apikey: apiKey,
              url: url,
              css_extractor: cssExtractor
          }
      })
      .then(response => console.log(response.data))
      .catch(error => console.error(error));
      ```

      ```bash cURL theme={"dark"}
      curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%257B%2522title%2522%253A%2522h1%2522%252C%2522product-list%2522%253A%2522.product-name%2522%257D"
      ```
    </CodeGroup>

    This request extracts both the page title and all product names. When a CSS selector matches multiple elements, Zenrows automatically returns them as an array.

    The response looks like this:

    ```json theme={"dark"}
    {
        "title": "E-commerce Products",
        "products": [
            "Product 1",
            "Product 2",
            "Product 3"
            // ...
        ]
    }
    ```

    ### Extracting attributes

    You might need product links to continue scraping individual product details. To extract the `href` attribute instead of text content, add `@href` to your selector.

    Let's filter links to only include those starting with `/product/`:

    <CodeGroup>
      ```python Python theme={"dark"}
      import requests

      api_key = "YOUR_ZENROWS_API_KEY"
      url = "https://www.scrapingcourse.com/ecommerce/"
      css_extractor = {
          "title": "h1",
          "products": ".product-name",
          "links": "a[href*='/product/'] @href"
      }

      response = requests.get(
          "https://api.zenrows.com/v1/",
          params={
              "apikey": api_key,
              "url": url,
              "css_extractor": css_extractor
          }
      )

      print(response.json())
      ```

      ```javascript JavaScript theme={"dark"}
      const axios = require("axios");

      const apiKey = "YOUR_ZENROWS_API_KEY";
      const url = "https://www.scrapingcourse.com/ecommerce/";
      const cssExtractor = {
          "title": "h1",
          "products": ".product-name",
          "links": "a[href*='/product/'] @href"
      };

      axios.get("https://api.zenrows.com/v1/", {
          params: {
              apikey: apiKey,
              url: url,
              css_extractor: cssExtractor
          }
      })
      .then(response => console.log(response.data))
      .catch(error => console.error(error));
      ```

      ```bash cURL theme={"dark"}
      curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&css_extractor=%257B%2522title%2522%253A%2522h1%2522%252C%2522products%2522%253A%2522.product-name%2522%252C%2522links%2522%253A%2522a%255Bhref*%253D%27%252Fproduct%252F%27%255D%2540href%2522%257D"
      ```
    </CodeGroup>

    The `@href` syntax tells Zenrows to extract the `href` attribute value instead of the element's text content. The `[href*='/product/']` part filters links to only include those containing `/product/` in their href attribute.

    This returns:

    ```json theme={"dark"}
    {
        "title": "Shop",
        "products": [
            "Product 1",
            "Product 2",
            "Product 3"
            // ...
        ],
        "links": [
            "/product/1",
            "/product/2",
            "/product/3"
            // ...
        ]
    }
    ```
  </Tab>

  <Tab title="Using Output Filters">
    The `outputs` parameter extracts predefined data types from scraped HTML. This allows you to efficiently retrieve only the data types you need, reducing processing time and focusing on relevant information.

    The parameter accepts a comma-separated list of filter names and returns results in structured JSON format.

    <Tip>Use `outputs=*` to retrieve all available data types.</Tip>

    ### Extracting page structure

    Get headings, links, and menu items to understand page structure:

    <CodeGroup>
      ```python Python theme={"dark"}
      import requests
      api_key = "YOUR_ZENROWS_API_KEY"
      url = "https://www.scrapingcourse.com/ecommerce/"

      response = requests.get(
          "https://api.zenrows.com/v1/",
          params={
              "apikey": api_key,
              "url": url,
              "outputs": "headings,links,menus"
          }
      )

      print(response.json())
      ```

      ```javascript Node.js theme={"dark"}
      const axios = require("axios");

      const apiKey = "YOUR_ZENROWS_API_KEY";
      const url = "https://www.scrapingcourse.com/ecommerce/";

      axios.get("https://api.zenrows.com/v1/", {
          params: {
              apikey: apiKey,
              url: url,
              outputs: "headings,links,menus"
          }
      })
      .then(response => console.log(response.data))
      .catch(error => console.error(error));
      ```

      ```bash cURL theme={"dark"}
      curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&outputs=headings,links,menus"
      ```
    </CodeGroup>

    This extracts heading text from `h1` through `h6` elements, URLs from `a` tags, and menu items from `li` elements inside `menu` tags.

    ### Extracting media content

    Get all images, videos, and audio files from a page:

    <CodeGroup>
      ```python Python theme={"dark"}
      import requests
      api_key = "YOUR_ZENROWS_API_KEY"
      url = "https://www.scrapingcourse.com/ecommerce/"

      response = requests.get(
          "https://api.zenrows.com/v1/",
          params={
              "apikey": api_key,
              "url": url,
              "outputs": "images,videos,audios"
          }
      )

      print(response.json())
      ```

      ```javascript Node.js theme={"dark"}
      const axios = require("axios");

      const apiKey = "YOUR_ZENROWS_API_KEY";
      const url = "https://www.scrapingcourse.com/ecommerce/";

      axios.get("https://api.zenrows.com/v1/", {
          params: {
              apikey: apiKey,
              url: url,
              outputs: "images,videos,audios"
          }
      })
      .then(response => console.log(response.data))
      .catch(error => console.error(error));
      ```

      ```bash cURL theme={"dark"}
      curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fecommerce%2F&outputs=images,videos,audios"
      ```
    </CodeGroup>

    This extracts image sources from `img` tags, video sources from `source` elements inside video tags, and audio sources from `source` elements inside audio tags.

    <Tip>
      For complete output filter options, see our [Output Filters Documentation](/fetch/features/output-filters) page.
    </Tip>
  </Tab>

  <Tab title="Using External Libraries">
    If you prefer using your favorite HTML parsing library, you can retrieve raw HTML from Zenrows and process it with tools like `BeautifulSoup` or `Cheerio`.

    ### Python with BeautifulSoup

    ```python scraper.py theme={"dark"}
    # pip install requests beautifulsoup4
    import requests
    from bs4 import BeautifulSoup

    zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY"
    url = "https://www.scrapingcourse.com/ecommerce/"

    response = requests.get(zenrows_api_base, params={'url': url})
    soup = BeautifulSoup(response.text, "html.parser")

    title = soup.find("h1").text
    products = [product.text for product in soup.select(".product-title")]
    links = [link.get("href") for link in soup.select("a[href^='/product/']")]

    result = {
        "title": title,
        "products": products,
        "links": links,
    }
    print(result)
    ```

    This approach gives you full control over HTML parsing. You first retrieve the raw HTML from Zenrows, then use `BeautifulSoup` to parse and extract the data you need.

    ### JavaScript with Cheerio

    ```javascript scraper.js theme={"dark"}
    // npm i axios cheerio
    const axios = require("axios");
    const cheerio = require("cheerio");

    const zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY";
    const url = "https://www.scrapingcourse.com/ecommerce/";

    axios
        .get(zenrows_api_base, { params: { url } })
        .then((response) => {
            const $ = cheerio.load(response.data);

            const title = $("h1").text();
            const products = $(".product-title")
                .map((_, a) => $(a).text())
                .toArray();
            const links = $("a[href^='/product/']")
                .map((_, a) => $(a).attr("href"))
                .toArray();

            console.log({ title, products, links });
        })
        .catch((error) => console.log(error));
    ```

    Cheerio provides a jQuery-like interface for server-side HTML manipulation. This example shows how to extract the same data using Cheerio's familiar syntax.

    Both approaches (CSS extractors and external libraries) offer flexibility for different use cases. CSS extractors provide immediate JSON output, while external libraries give you more control over complex parsing logic.
  </Tab>
</Tabs>

## Testing Your Selectors

Before implementing your scraper at scale, test your CSS selectors using our [Playground](https://app.zenrows.com/fetch). The Playground shows you the extracted data in real-time and generates code in multiple programming languages for easy integration.

<Frame>
  <img alt="Extract Data using CSS Selectors" />
</Frame>

<Tip>
  For more details, [check CSS Selectors documentation](/fetch/features/css-extractor).
</Tip>

## When to use each method

Choose your data extraction method based on your specific needs:

* **CSS Selectors** - Best for custom data extraction when you know exactly what elements you need. Returns clean JSON data with your own key names and structure.
* **Output Filters** - Ideal for extracting common data types like emails, phone numbers, images, and links. Perfect when you need standard web data without custom parsing.
* **External Libraries** - Perfect when you need complex parsing logic, custom data transformations, or when integrating with existing parsing workflows.

## Further Reading

For more advanced CSS selector patterns and examples for complex web layouts, check out our [Advanced CSS Selector Examples](/fetch/troubleshooting/advanced-css-selector-examples) guide.


# Integrate Zenrows with an Existing Price Monitoring System
Source: https://docs.zenrows.com/zenrows-academy/integrate-price-monitoring

Integrate Zenrows into your existing price monitoring system to bypass anti-bot protection, improve scraping reliability, and reduce maintenance.

Replace your current web scraping solution with Zenrows to improve reliability, bypass anti-bot protection, and reduce maintenance overhead. This guide shows you how to integrate Zenrows into existing price monitoring systems.

## Who Is this for?

This guide is for developers who already have a price monitoring system and want to integrate Zenrows to improve scraping reliability and reduce blocking issues.

## What you'll learn

* Replace existing HTTP clients with Zenrows requests
* Migrate from HTML parsing to CSS extraction
* Optimize performance with concurrency controls
* Monitor and control scraping costs
* Scale across multiple regions

## Prerequisites

* An existing price monitoring system
* A Zenrows API key ([SIGN UP HERE](https://app.zenrows.com/register))
* Basic understanding of web scraping concepts

## Integration Approaches

Choose the integration approach that best fits your current system:

### Approach 1: Minimal Integration (HTTP Client Replacement)

Replace your current HTTP client with Zenrows while keeping existing parsing logic.

**Before (typical implementation):**

```python Python theme={"dark"}
import requests
from bs4 import BeautifulSoup

def get_page_content(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }
    response = requests.get(url, headers=headers)
    
    if response.status_code != 200:
        raise Exception(f"Request failed: {response.status_code}")
    
    return response.content

def scrape_price_data(url):
    html_content = get_page_content(url)
    soup = BeautifulSoup(html_content, "html.parser")
    
    # Extract data using BeautifulSoup
    name_elem = soup.select_one("#productTitle")
    price_elem = soup.select_one("span.aok-offscreen")
    
    return {
        "name": name_elem.get_text(strip=True) if name_elem else None,
        "price": price_elem.get_text(strip=True) if price_elem else None,
    }
```

**After (Zenrows integration):**

```python Python theme={"dark"}
import requests
from bs4 import BeautifulSoup

def get_page_content(url):
    params = {
        "url": url,
        "apikey": "YOUR_ZENROWS_API_KEY",
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "wait": 2000,
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    
    if response.status_code != 200:
        raise Exception(f"Request failed: {response.status_code}")
    
    return response.text

# Keep existing parsing logic unchanged
def scrape_price_data(url):
    html_content = get_page_content(url)
    soup = BeautifulSoup(html_content, "html.parser")
    
    # Same parsing logic as before
    name_elem = soup.select_one("#productTitle")
    price_elem = soup.select_one("span.aok-offscreen")
    
    return {
        "name": name_elem.get_text(strip=True) if name_elem else None,
        "price": price_elem.get_text(strip=True) if price_elem else None,
    }
```

**Benefits:**

* Minimal code changes required
* Immediate anti-bot protection
* Keep existing data processing logic
* Easy rollback if needed

### Approach 2: Full Integration (CSS Extraction)

Replace both HTTP client and HTML parsing with Zenrows CSS extraction for cleaner, more maintainable code.

**Before:**

```python Python theme={"dark"}
import requests
from bs4 import BeautifulSoup

def scrape_price_data(url):
    # HTTP request + HTML parsing
    headers = {"User-Agent": "Mozilla/5.0..."}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, "html.parser")
    
    # Manual element extraction
    name_elem = soup.select_one("#productTitle")
    price_elem = soup.select_one("span.aok-offscreen")
    rating_elem = soup.select_one("span.a-size-base.a-color-base")
    
    return {
        "name": name_elem.get_text(strip=True) if name_elem else None,
        "price": price_elem.get_text(strip=True) if price_elem else None,
        "rating": rating_elem.get_text(strip=True) if rating_elem else None,
    }
```

**After:**

```python Python theme={"dark"}
import requests
import json

def scrape_price_data(url):
    # Define CSS selectors
    css_extractor = json.dumps({
        "name": "#productTitle",
        "price": "span.aok-offscreen",
        "rating": "span.a-size-base.a-color-base",
    })
    
    # Single Zenrows request with extraction
    params = {
        "url": url,
        "apikey": "YOUR_ZENROWS_API_KEY",
        "js_render": "true",
        "premium_proxy": "true",
        "proxy_country": "us",
        "css_extractor": css_extractor,
    }
    
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    
    if response.status_code != 200:
        raise Exception(f"Request failed: {response.status_code}")
    
    return response.json()
```

**Benefits:**

* Eliminates HTML parsing dependencies
* Cleaner, more maintainable code
* Built-in data extraction
* Reduced code complexity

## Step-by-Step Integration

<Steps>
  <Step title="Assess Your Current Implementation">
    Before integrating Zenrows, analyze your current scraping setup to choose the best integration approach.
    **Identify your current components:**

    * HTTP client (requests, urllib, etc.)
    * HTML parser (BeautifulSoup, lxml, etc.)
    * Data extraction logic
    * Error handling mechanisms
    * Proxy management (if any)

    **Common patterns to look for:**

    ```python Python expandable theme={"dark"}
    # Pattern 1: Simple requests + BeautifulSoup
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, "html.parser")

    # Pattern 2: Session-based scraping
    session = requests.Session()
    session.headers.update(headers)
    response = session.get(url)

    # Pattern 3: Selenium-based scraping
    driver = webdriver.Chrome()
    driver.get(url)
    element = driver.find_element(By.CSS_SELECTOR, selector)

    # Pattern 4: Custom proxy rotation
    proxies = {"http": proxy_url, "https": proxy_url}
    response = requests.get(url, proxies=proxies)
    ```

    **Integration complexity assessment:**

    * **Low complexity**: Simple requests + BeautifulSoup → Use Approach 1
    * **Medium complexity**: Custom headers/sessions → Use Approach 1 or 2
    * **High complexity**: Selenium/complex proxy logic → Use Approach 2
  </Step>

  <Step title="Replace HTTP Client">
    Start with the minimal integration approach by replacing your HTTP client with Zenrows.

    **For requests-based systems:**

    ```python Python theme={"dark"}
    # Original function
    def fetch_page(url, headers=None, proxies=None):
        response = requests.get(url, headers=headers, proxies=proxies)
        return response.text

    # Zenrows replacement
    def fetch_page(url, apikey, country="us"):
        params = {
            "url": url,
            "apikey": apikey,
            "js_render": "true",
            "premium_proxy": "true",
            "proxy_country": country,
        }
        response = requests.get("https://api.zenrows.com/v1/", params=params)
        return response.text
    ```

    **For Puppeteer/Playwright-based systems:**

    ```javascript JavaScript expandable theme={"dark"}
    // Original Puppeteer approach
    const puppeteer = require('puppeteer');

    async function scrapeWithPuppeteer(url) {
        const browser = await puppeteer.launch();
        const page = await browser.newPage();
        await page.goto(url, { waitUntil: 'networkidle2' });
        
        const html = await page.content();
        await browser.close();
        return html;
    }

    // Zenrows Browser Sessions replacement
    const puppeteer = require('puppeteer-core');

    async function scrapeWithZenRows(url) {
        const browser = await puppeteer.connect({
            browserWSEndpoint: 'wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY'
        });
        
        const page = await browser.newPage();
        await page.goto(url, { waitUntil: 'networkidle2' });
        
        const html = await page.content();
        await browser.close();
        return html;
    }
    ```

    ```python Python expandable theme={"dark"}
    # Original Playwright approach
    from playwright.sync_api import sync_playwright

    def scrape_with_playwright(url):
        with sync_playwright() as p:
            browser = p.chromium.launch()
            page = browser.new_page()
            page.goto(url)
            html = page.content()
            browser.close()
            return html

    # Zenrows Browser Sessions replacement
    from playwright.sync_api import sync_playwright

    def scrape_with_zenrows_browser(url):
        with sync_playwright() as p:
            browser = p.chromium.connect_over_cdp(
                "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"
            )
            page = browser.new_page()
            page.goto(url)
            html = page.content()
            browser.close()
            return html
    ```
  </Step>

  <Step title="Migrate to CSS Extraction (Optional)">
    For cleaner code and better maintainability, replace HTML parsing with Zenrows CSS extraction.

    **Identify your current selectors:**

    ```python Python theme={"dark"}
    # Current BeautifulSoup selectors
    soup = BeautifulSoup(html, "html.parser")
    name = soup.select_one("#productTitle").get_text(strip=True)
    price = soup.select_one("span.aok-offscreen").get_text(strip=True)
    rating = soup.select_one("span.a-size-base").get_text(strip=True)
    reviews = soup.select_one("#acrCustomerReviewText").get_text(strip=True)
    ```

    **Convert to CSS extractor:**

    ```python Python theme={"dark"}
    import json

    # Define CSS extractor
    css_extractor = json.dumps({
        "name": "#productTitle",
        "price": "span.aok-offscreen", 
        "rating": "span.a-size-base",
        "reviews": "#acrCustomerReviewText",
    })

    def scrape_with_css_extractor(url, apikey):
        params = {
            "url": url,
            "apikey": apikey,
            "js_render": "true",
            "premium_proxy": "true",
            "css_extractor": css_extractor,
        }
        response = requests.get("https://api.zenrows.com/v1/", params=params)
        return response.json()  # Returns structured data directly
    ```
  </Step>
</Steps>

## Migration Checklist

**Pre-Migration**

* Document current scraping logic and selectors
* Identify proxy and header requirements
* Test Zenrows with sample requests
* Plan rollback strategy

**During Migration**

* Replace HTTP client with Zenrows API calls
* Update error handling for Zenrows responses
* Test with production URLs
* Monitor request costs and concurrency

**Post-Migration**

* Remove old proxy management code
* Clean up unused HTML parsing dependencies
* Update monitoring and alerting
* Document new Zenrows integration

## Best Practices

**Cost Management**

* Set daily cost limits to prevent unexpected charges
* Monitor request costs with the `X-Request-Cost` header
* Use `Concurrency-Limit` and `Concurrency-Remaining` response headers to optimize throughput and avoid [IP block errors](/fetch/troubleshooting/ip-address-blocked)
* Cache results when appropriate to reduce requests

**Reliability**

* Implement retry logic with exponential backoff
* Monitor selector stability and update as needed
* Use fallback selectors for critical data points
* Log all requests responses and errors for debugging and monitoring

**Performance**

* Use appropriate concurrency based on your plan limits
* Batch requests when monitoring multiple products
* Leverage geographic targeting for region-specific data

## Troubleshooting

**Selector Compatibility**

* Test selectors in Zenrows Request Playground before migration
* Some selectors may behave differently with JavaScript rendering
* Use more specific selectors if extraction returns unexpected data

**Cost Optimization**

* Disable `js_render` for static content to reduce costs
* Use `wait_for` instead of `wait` when possible
* Monitor `Concurrency-Limit` and `Concurrency-Remaining` response headers to maximize throughput

**Error Handling**

* Zenrows returns different error codes. See the [API Error Codes](/api-error-codes) documentation for more details.
* Implement specific handling for rate limits (`429`) and quota exceeded (`402`). A `402` can mean the account has no credits left ([AUTH004](/api-error-codes#AUTH004)) or the API key reached one of its [credit caps](/first-steps/api-key-credit-caps) ([AUTH014](/api-error-codes#AUTH014))
* Log Zenrows-specific headers for debugging


# Retry Failed Requests
Source: https://docs.zenrows.com/zenrows-academy/retry-failed-requests

Set up auto-retry policies for failed web scraping requests in Python and JavaScript using Zenrows with exponential backoff and error handling.

Implementing auto-retry policies in web scraping is crucial for maintaining data accuracy and reliability, especially since achieving a 100% success rate is difficult. Factors such as site downtimes or interrupted connections can cause occasional failures.

However, with Zenrows, you are not charged for these failed attempts. For instance, if you make 130 requests and 15 need to be retried, you will only be charged for the 115 successful ones, providing a cost-effective solution.

Below, we offer detailed solutions in Python and JavaScript to help you implement these retry policies. If you require support for other languages or frameworks, please feel free to <a href="mailto:success@zenrows.com">contact us</a>.

<Tip>Zenrows <a href="https://pypi.org/project/zenrows/">Python SDK</a> and <a href="https://www.npmjs.com/package/zenrows">JavaScript SDK</a> offer built-in support for retries. Simply pass a `retries` parameter in the constructor to enable this feature and enhance your scraping efficiency.</Tip>

## Python with Requests

To begin, ensure you have <a href="https://www.python.org/downloads/">Python 3 installed</a> on your system. Some systems come with it pre-installed. Once Python is ready, install the necessary libraries by running the following command:

```bash theme={"dark"}
pip install requests
```

We will use the <a href="https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util">`Retry`</a> class from `urllib3` and the <a href="https://docs.python-requests.org/en/master/_modules/requests/adapters/">`HTTPAdapter`</a> from requests. These are part of the `requests` library, so you don't need to install them separately.

Instead of making direct `get` calls, we create a requests session. This approach allows us to set up and reuse the session configuration, including retry settings, for all subsequent requests. This setup is efficient and ensures consistent retry behavior across all requests.

You can adjust the `Retry` parameters to suit your needs. Here are some key parameters:

* `total` sets the maximum number of retries allowed.
* `backoff_factor` defines the delay between retries using an exponential backoff strategy. The formula is: **$\text{{backoff\_factor}} \times (2^{(\text{{number of total retries}} - 1)})$**. For example, a backoff factor of 1 results in delays of 1, 2, and 4 seconds for three retries.
* `status_forcelist` is a list of HTTP status codes that will force a retry.

```python scraper.py theme={"dark"}
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

apikey = "YOUR_ZENROWS_API_KEY"
urls = [
	""  # ... your URLs here
]
zenrows_api_base = "https://api.zenrows.com/v1/"

requests_session = requests.Session()
retries = Retry(
	total=3,
	backoff_factor=1,
	status_forcelist=[429, 500, 502, 503, 504]
)
requests_session.mount("https://", HTTPAdapter(max_retries=retries))

for url in urls:
	try:
		response = requests_session.get(zenrows_api_base, params={
			"apikey": apikey,
			"url": url,
		})

		print(response.text)  # process response
	except Exception as e:
		print(e)  # will print "Max retries exceeded"
```

In this example, we use a list of URLs and process them sequentially. For improved performance, you can process them [concurrently with a maximum](/fetch/features/concurrency#using-concurrency). This approach is particularly useful when dealing with large numbers of URLs or when response times are slow.

## JavaScript with axios-retry

To get started, ensure you have <a href="https://nodejs.org/en/">Node.js</a> (or <a href="https://github.com/nvm-sh/nvm">nvm</a>) and npm installed on your system. Many systems come with these pre-installed. Once set up, install the necessary libraries by running:

```bash theme={"dark"}
npm install axios axios-retry
```

Instead of making direct calls to the Zenrows API using `axios`, we'll use the <a href="https://github.com/softonic/axios-retry">axiosRetry</a> library to manage retries. This configuration will automatically retry requests for all `axios` calls, and you can also create a specific axios `client` with retry capabilities.

You can adjust the `axiosRetry` parameters to fit your specific requirements. The configuration provided below should work well for most scenarios:

* `retries` sets the number of allowed retries.
* `retryDelay` this applies an exponential delay between attempts, with an additional 0-20% random delay margin to prevent repeated requests from hitting the server at regular intervals.
* `retryCondition` this function determines whether an error is eligible for a retry. By default, the function retries only 5xx errors. However, in our example, we've added a check for 429 (Too Many Requests) errors to ensure these are also retried.

```javascript theme={"dark"}
const axios = require("axios");
const axiosRetry = require("axios-retry");

const apikey = "YOUR_ZENROWS_API_KEY";
const zenrowsApiBase = "https://api.zenrows.com/v1/";
const urls = [
	// ... your URLs here
];

axiosRetry(axios, {
	retries: 3,
	retryDelay: axiosRetry.exponentialDelay,
	retryCondition: (error) => {
		if (error.response && error.response.status === 429) {
			return true; // example for custom retry condition
		}

		// fallback to default condition
		return axiosRetry.isNetworkOrIdempotentRequestError(error);
	},
});

(async () => {
	for (const url of urls) {
		try {
			const response = await axios.get(zenrowsApiBase, {
				params: { apikey, url },
			});
			console.log(response.data); // process response
		} catch (error) {
			console.log(error);
		}
	}
})();
```

The example processes a list of URLs sequentially for simplicity. However, you can also execute them concurrently, allowing multiple requests to be handled simultaneously.

This approach can be particularly useful when dealing with a large number of URLs, improving efficiency and reducing the overall time required. For more information on running requests concurrently, refer to our guide on using [concurrency with Zenrows SDK for JavaScript](/fetch/features/concurrency#zenrows-sdk-for-javascript).


# Scrape and Crawl from a Seed URL
Source: https://docs.zenrows.com/zenrows-academy/scrape-and-crawl-from-a-seed-url

Build a web crawler in Python that starts from a seed URL, discovers internal links, and scrapes pages at scale using Zenrows API.

To perform web scraping at scale, it's essential to extract data and continually discover new URLs. This process ensures you cover all relevant website pages, especially when dealing with dynamically generated content or paginated sections.

We'll start with a single URL (the “seed URL”) and extract internal links. These internal links are URLs that keep the scraping process within the same website, avoiding external links that could lead to different sites. This method is commonly used to gather all website pages, such as scraping all product pages from an e-commerce site or all articles from a blog.

## Prerequisites

Before starting, ensure you have <a href="https://www.python.org/downloads/">Python 3</a> installed. Some systems come with Python pre-installed. Once you have Python set up, install the necessary libraries by running the following command:

```bash theme={"dark"}
pip install requests beautifulsoup4
```

The `requests` library will handle HTTP requests, while `BeautifulSoup` will help parse the HTML content and extract links.

## Extracting Links from the Seed URL

We'll use BeautifulSoup to extract the links from the page's HTML content. Although BeautifulSoup isn't required for the Zenrows API to function, it simplifies parsing and filtering the extracted data. We'll also define separate functions for better code organization and maintainability.

```python scraper.py theme={"dark"}
import requests
from bs4 import BeautifulSoup

zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY"
seed_url = "" # ... your seed URL here

def extract_links(soup):
	return [a.get("href")
			for a in soup.find_all("a")
			if a.get("href") and a.get("href").startswith("/")]

def call_url(url):
	response = requests.get(zenrows_api_base, params={"url": url})
	soup = BeautifulSoup(response.text, "html.parser")
	links = extract_links(url, soup)
	print(links)

call_url(seed_url)
```

## Managing Crawl Limits and Visited URLs

Setting a maximum number of requests and keeping track of visited URLs is crucial to preventing infinite loops and excessive requests. This prevents the script from endlessly looping over the same pages or making thousands of unnecessary calls.

```python theme={"dark"}
max_visits = 10
visited = set()
```

## Setting Up a Queue and Worker Threads

The next step involves setting up a queue and worker threads to manage the URLs that must be crawled. This allows for concurrent processing of multiple URLs, improving the efficiency of the scraping process.

```python scraper.py theme={"dark"}
import queue
from threading import Thread

num_workers = 5

def queue_worker(i, q):
	while True:
		url = q.get() # Get an item from the queue, blocks until one is available
		# Some processing, to be defined
		q.task_done() # Notifies the queue that the item has been processed

q = queue.Queue()
for i in range(num_workers):
	Thread(target=queue_worker, args=(i, q), daemon=True).start()

q.put(seed_url)
q.join() # Blocks until all items in the queue are processed and marked as done
```

## Full Implementation: Crawling and Data Extraction

Finally, we combine these elements to create a fully functional crawler. The script manages the queue, processes URLs, and extracts the desired content.

<Note>For simplicity, error handling and data storage are not included but can be added as needed.</Note>

```python scraper.py theme={"dark"}
import requests
from bs4 import BeautifulSoup
import queue
from threading import Thread

zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY"
seed_url = ""  # ... your seed URL here
max_visits = 10
num_workers = 5

visited = set()
data = []

def extract_links(soup):
	# get the links you want to follow here
	return [a.get("href")
			for a in soup.find_all("a")
			if a.get("href") and a.get("href").startswith("/")]

def extract_content(url, soup):
	# extract the content you want here
	data.append({
		"url": url,
		"title": soup.title.string,
		"h1": soup.find("h1").text,
	})

def crawl(url):
	visited.add(url)
	print("Crawl: ", url)
	response = requests.get(zenrows_api_base, params={"url": url})
	soup = BeautifulSoup(response.text, "html.parser")
	extract_content(url, soup)
	links = extract_links(soup)
	for link in links:
		if link not in visited:
			q.put(link)

def queue_worker(i, q):
	while True:
		url = q.get()
		if (len(visited) < max_visits and url not in visited):
			crawl(url)
		q.task_done()

q = queue.Queue()
for i in range(num_workers):
	Thread(target=queue_worker, args=(i, q), daemon=True).start()

q.put(seed_url)
q.join()

print("Visited:", visited)
print("Data:", data)
```

This code sets up a basic web crawler using Zenrows, `requests`, and `BeautifulSoup`. It starts from a seed URL, extracts links, and follows them up to a defined limit. Be cautious when using this method on large websites, as it can quickly generate a massive number of pages to crawl. Proper error handling, rate limiting, and data storage should be added for production use.

For a more detailed guide and additional techniques, check out our [scraping with python series](https://www.zenrows.com/blog/web-crawler-python). If you have any questions, feel free to <a href="mailto:success@zenrows.com">contact us</a>.


# Scrape from a List of URLs
Source: https://docs.zenrows.com/zenrows-academy/scrape-from-a-list-of-urls

Scrape data from multiple URLs in parallel or sequentially with Zenrows using Python and JavaScript for efficient bulk web scraping.

When you need to extract data from multiple web pages, you can process a list of URLs using either sequential or parallel approaches. Sequential processing handles URLs one at a time, while parallel processing (handling multiple requests simultaneously) significantly reduces scraping time for large URL lists.

<Warning>This guide assumes you already have a list of URLs ready for scraping</Warning>

If you need to discover URLs from a starting page, check our [Scrape and Crawl from a Seed URL](/zenrows-academy/scrape-and-crawl-from-a-seed-url).

<Info>The techniques shown in this guide work with any programming language that supports HTTPS requests (PHP, Java, Ruby, Go, C#, etc.). We're using Python and Node.js examples for simplicity, but you can adapt these patterns to your preferred language.</Info>

## Prerequisites

### Python Setup

You'll need <a href="https://www.python.org/downloads/">Python 3</a> installed. Install the required libraries:

```bash theme={"dark"}
pip install requests beautifulsoup4
```

The `requests` library handles HTTP requests to Zenrows, while `beautifulsoup4` parses HTML content for data extraction.

### Node.js Setup

You'll need <a href="https://nodejs.org/">Node.js</a> installed. Install the required packages:

```bash theme={"dark"}
npm install axios cheerio
```

The `axios` library handles HTTP requests, while `cheerio` provides server-side HTML parsing similar to jQuery.

## Sequential Processing

Sequential processing handles URLs one after another in a simple loop. This approach works well for small URL lists or when you want to avoid overwhelming target servers with concurrent requests.

### Basic sequential scraping

<CodeGroup>
  ```python Python theme={"dark"}
  import requests

  zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY"
  urls = [
  	# ... your URLs here
  ]

  for url in urls:
  	response = requests.get(zenrows_api_base, params={"url": url})
  	print(response.text)
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');

  const zenrowsApiBase = "https://api.zenrows.com/v1/";
  const apiKey = "YOUR_ZENROWS_API_KEY";
  const urls = [
  	// ... your URLs here
  ];

  async function scrapeSequentially() {
  	for (const url of urls) {
  		try {
  			const response = await axios.get(zenrowsApiBase, {
  				params: {
  					apikey: apiKey,
  					url: url
  				}
  			});
  			console.log(response.data);
  		} catch (error) {
  			console.error(`Error scraping ${url}:`, error.message);
  		}
  	}
  }

  scrapeSequentially();
  ```
</CodeGroup>

This code sends each URL to Zenrows sequentially and prints the returned HTML. The loop processes URLs in order, waiting for each request to complete before moving to the next.

### Extracting specific data

To make the scraper more useful, extract specific data like page titles and headings:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests
  from bs4 import BeautifulSoup

  zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY"
  urls = [
  	# ... your URLs here
  ]

  def extract_content(url, soup):
  	return {
  		"url": url,
  		"title": soup.title.string,
  		"h1": soup.find("h1").text,
  	}

  results = []
  for url in urls:
  	response = requests.get(zenrows_api_base, params={"url": url})
  	soup = BeautifulSoup(response.text, "html.parser")
  	results.append(extract_content(url, soup))

  print(results)
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');
  const cheerio = require('cheerio');

  const zenrowsApiBase = "https://api.zenrows.com/v1/";
  const apiKey = "YOUR_ZENROWS_API_KEY";
  const urls = [
  	// ... your URLs here
  ];

  function extractContent(url, $) {
  	return {
  		url: url,
  		title: $('title').text() || "No title",
  		h1: $('h1').first().text() || "No H1"
  	};
  }

  async function scrapeWithExtraction() {
  	const results = [];
  	
  	for (const url of urls) {
  		try {
  			const response = await axios.get(zenrowsApiBase, {
  				params: {
  					apikey: apiKey,
  					url: url
  				}
  			});
  			
  			const $ = cheerio.load(response.data);
  			results.push(extractContent(url, $));
  		} catch (error) {
  			console.error(`Error scraping ${url}:`, error.message);
  			results.push({ url, error: error.message });
  		}
  	}
  	
  	console.log(results);
  }

  scrapeWithExtraction();
  ```
</CodeGroup>

The `extract_content` function parses each page and extracts the title and main heading. Results are stored in a list for further processing. The function includes safety checks to handle pages missing title or H1 elements.

### Adding error handling

For production use, add error handling to manage failed requests:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests
  from bs4 import BeautifulSoup
  import time

  zenrows_api_base = "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY"
  urls = [
  	# ... your URLs here
  ]

  def extract_content(url, soup):
  	return {
  		"url": url,
  		"title": soup.title.string if soup.title else "No title",
  		"h1": soup.find("h1").text if soup.find("h1") else "No H1"
  	}

  results = []
  for url in urls:
  	try:
  		response = requests.get(zenrows_api_base, params={"url": url})
  		response.raise_for_status()
  		
  		soup = BeautifulSoup(response.text, "html.parser")
  		results.append(extract_content(url, soup))
  		
  		# Add delay between requests
  		time.sleep(1)
  		
  	except requests.exceptions.RequestException as e:
  		print(f"Error scraping {url}: {e}")
  		results.append({"url": url, "error": str(e)})

  print(results)
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');
  const cheerio = require('cheerio');

  const zenrowsApiBase = "https://api.zenrows.com/v1/";
  const apiKey = "YOUR_ZENROWS_API_KEY";
  const urls = [
  	// ... your URLs here
  ];

  function extractContent(url, $) {
  	return {
  		url: url,
  		title: $('title').text() || "No title",
  		h1: $('h1').first().text() || "No H1"
  	};
  }

  function delay(ms) {
  	return new Promise(resolve => setTimeout(resolve, ms));
  }

  async function scrapeWithErrorHandling() {
  	const results = [];
  	
  	for (const url of urls) {
  		try {
  			const response = await axios.get(zenrowsApiBase, {
  				params: {
  					apikey: apiKey,
  					url: url
  				}
  			});
  			
  			const $ = cheerio.load(response.data);
  			results.push(extractContent(url, $));
  			
  			// Add delay between requests
  			await delay(1000);
  			
  		} catch (error) {
  			console.error(`Error scraping ${url}:`, error.message);
  			results.push({ url, error: error.message });
  		}
  	}
  	
  	console.log(results);
  }

  scrapeWithErrorHandling();
  ```
</CodeGroup>

This version includes try-catch blocks to handle network errors, validates HTTP responses, and adds a delay between requests to be respectful to servers.

## Parallel Processing

Parallel processing handles multiple URLs simultaneously, dramatically reducing total scraping time. However, you must manage concurrency carefully to avoid overwhelming servers or exceeding rate limits.

<Info>While these examples use Python and Node.js, the same parallel processing concepts apply to other languages. Most modern programming languages provide similar concurrency features (async/await, thread pools, promises, etc.).</Info>

### Using standard libraries

<CodeGroup>
  ```python Python theme={"dark"}
  import requests
  from bs4 import BeautifulSoup
  from concurrent.futures import ThreadPoolExecutor
  import time

  zenrows_api_base = "https://api.zenrows.com/v1/"
  api_key = "YOUR_ZENROWS_API_KEY"

  urls = [
  	# ... your URLs here
  ]

  def scrape_single_url(url):
  	try:
  		response = requests.get(zenrows_api_base, params={"apikey": api_key, "url": url})
  		response.raise_for_status()
  		soup = BeautifulSoup(response.text, "html.parser")
  		return {
  			"url": url,
  			"title": soup.title.string if soup.title else "No title",
  			"h1": soup.find("h1").text if soup.find("h1") else "No H1"
  		}
  	except Exception as e:
  		return {"url": url, "error": str(e)}

  def scrape_parallel_threads():
  	start_time = time.time()
  	with ThreadPoolExecutor(max_workers=5) as executor:
  		results = list(executor.map(scrape_single_url, urls))
  	end_time = time.time()
  	print(f"Completed in {end_time - start_time:.2f} seconds")
  	print(results)

  if __name__ == "__main__":
  	scrape_parallel_threads()
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');
  const cheerio = require('cheerio');

  const zenrowsApiBase = "https://api.zenrows.com/v1/";
  const apiKey = "YOUR_ZENROWS_API_KEY";
  const urls = [
  	// ... your URLs here
  ];

  async function scrapeSingleUrl(url) {
  	try {
  		const response = await axios.get(zenrowsApiBase, {
  			params: {
  				apikey: apiKey,
  				url: url
  			}
  		});
  		
  		const $ = cheerio.load(response.data);
  		return {
  			url: url,
  			title: $('title').text() || "No title",
  			h1: $('h1').first().text() || "No H1"
  		};
  	} catch (error) {
  		return { url, error: error.message };
  	}
  }

  async function scrapeParallel() {
  	const startTime = Date.now();
  	
  	// Process all URLs concurrently
  	const results = await Promise.all(urls.map(scrapeSingleUrl));
  	
  	const endTime = Date.now();
  	console.log(`Completed in ${(endTime - startTime) / 1000} seconds`);
  	console.log(results);
  }

  scrapeParallel();
  ```
</CodeGroup>

This approach uses `ThreadPoolExecutor` in Python and `Promise.all` in Node.js to handle multiple requests simultaneously. The concurrency is limited by the `max_workers` parameter in Python and naturally managed by Node.js's event loop.

### Controlling concurrency with batching

For better control over concurrency, process URLs in batches:

<CodeGroup>
  ```python Python theme={"dark"}
  import requests
  from bs4 import BeautifulSoup
  from concurrent.futures import ThreadPoolExecutor
  import time

  zenrows_api_base = "https://api.zenrows.com/v1/"
  api_key = "YOUR_ZENROWS_API_KEY"

  def scrape_single_url(url):
  	try:
  		response = requests.get(zenrows_api_base, params={"apikey": api_key, "url": url})
  		response.raise_for_status()
  		soup = BeautifulSoup(response.text, "html.parser")
  		return {
  			"url": url,
  			"title": soup.title.string if soup.title else "No title",
  			"h1": soup.find("h1").text if soup.find("h1") else "No H1"
  		}
  	except Exception as e:
  		return {"url": url, "error": str(e)}

  def process_batch(urls_batch, max_workers=5):
  	with ThreadPoolExecutor(max_workers=max_workers) as executor:
  		return list(executor.map(scrape_single_url, urls_batch))

  def scrape_with_batching(all_urls, batch_size=10, max_workers=5):
  	all_results = []
  	for i in range(0, len(all_urls), batch_size):
  		batch = all_urls[i:i + batch_size]
  		print(f"Processing batch {i//batch_size + 1} ({len(batch)} URLs)")
  		batch_results = process_batch(batch, max_workers)
  		all_results.extend(batch_results)
  		# Optional delay between batches
  		time.sleep(1)
  	return all_results

  if __name__ == "__main__":
  	urls = [
  		# ... your URLs here
  	]
  	results = scrape_with_batching(urls, batch_size=5, max_workers=3)
  	print(f"Total results: {len(results)}")
  ```

  ```javascript Node.js theme={"dark"}
  const axios = require('axios');
  const cheerio = require('cheerio');

  const zenrowsApiBase = "https://api.zenrows.com/v1/";
  const apiKey = "YOUR_ZENROWS_API_KEY";

  async function scrapeSingleUrl(url) {
  	try {
  		const response = await axios.get(zenrowsApiBase, {
  			params: {
  				apikey: apiKey,
  				url: url
  			}
  		});
  		
  		const $ = cheerio.load(response.data);
  		return {
  			url: url,
  			title: $('title').text() || "No title",
  			h1: $('h1').first().text() || "No H1"
  		};
  	} catch (error) {
  		return { url, error: error.message };
  	}
  }

  async function processBatch(urlsBatch, concurrency = 5) {
  	const semaphore = new Array(concurrency).fill(null);
  	let index = 0;
  	
  	async function processNext() {
  		if (index >= urlsBatch.length) return null;
  		const currentIndex = index++;
  		return await scrapeSingleUrl(urlsBatch[currentIndex]);
  	}
  	
  	const results = await Promise.all(
  		semaphore.map(async () => {
  			const batchResults = [];
  			let result;
  			while ((result = await processNext()) !== null) {
  				batchResults.push(result);
  			}
  			return batchResults;
  		})
  	);
  	
  	return results.flat();
  }

  async function scrapeWithBatching(allUrls, batchSize = 10, concurrency = 5) {
  	const allResults = [];
  	
  	for (let i = 0; i < allUrls.length; i += batchSize) {
  		const batch = allUrls.slice(i, i + batchSize);
  		console.log(`Processing batch ${Math.floor(i/batchSize) + 1} (${batch.length} URLs)`);
  		
  		const batchResults = await processBatch(batch, concurrency);
  		allResults.push(...batchResults);
  		
  		// Optional delay between batches
  		await new Promise(resolve => setTimeout(resolve, 1000));
  	}
  	
  	return allResults;
  }

  // Example usage
  const urls = [
  	// ... your URLs here
  ];

  scrapeWithBatching(urls, 5, 3).then(results => {
  	console.log(`Total results: ${results.length}`);
  	console.log(results);
  });
  ```
</CodeGroup>

This batching approach processes URLs in smaller groups, giving you better control over server load and memory usage. You can adjust batch size and concurrency based on your specific needs.

### Using Zenrows SDKs

Zenrows SDKs simplify parallel web scraping by providing built-in concurrency management, automatic error handling, and connection pooling. When you're scraping multiple URLs, the SDKs handle the complex orchestration automatically, so you can focus on extracting the data you need rather than managing technical details.

<Tabs>
  <Tab title="Python SDK">
    First, install the Zenrows Python SDK:

    ```bash theme={"dark"}
    pip install zenrows
    ```

    ```python Python SDK theme={"dark"}
    from zenrows import ZenRowsClient
    import asyncio
    from bs4 import BeautifulSoup

    client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1)

    urls = [
    	# ... your URLs here
    ]

    async def scrape_url(url):
    	try:
    		response = await client.get_async(url)
    		if response.ok:
    			soup = BeautifulSoup(response.text, "html.parser")
    			return {
    				"url": url,
    				"title": soup.title.string if soup.title else "No title",
    				"h1": soup.find("h1").text if soup.find("h1") else "No H1"
    			}
    		else:
    			return {"url": url, "error": f"HTTP {response.status_code}"}
    	except Exception as e:
    		return {"url": url, "error": str(e)}

    async def main():
    	results = await asyncio.gather(*[scrape_url(url) for url in urls])
    	valid_results = [r for r in results if r is not None]
    	print(valid_results)

    if __name__ == "__main__":
    	asyncio.run(main())
    ```
  </Tab>

  <Tab title="Node.js SDK">
    First, install the Zenrows Node.js SDK:

    ```bash theme={"dark"}
    npm install zenrows
    ```

    ```javascript Node.js SDK theme={"dark"}
    const { ZenRowsClient } = require('zenrows');
    const cheerio = require('cheerio');

    const client = new ZenRowsClient('YOUR_ZENROWS_API_KEY', { concurrency: 5, retries: 1 });

    const urls = [
    	// ... your URLs here
    ];

    async function scrapeUrl(url) {
    	try {
    		const response = await client.get(url);
    		if (response.ok) {
    			const $ = cheerio.load(response.data);
    			return {
    				url: url,
    				title: $('title').text() || "No title",
    				h1: $('h1').first().text() || "No H1"
    			};
    		} else {
    			return { url, error: `HTTP ${response.status}` };
    		}
    	} catch (error) {
    		return { url, error: error.message };
    	}
    }

    async function main() {
    	const results = await Promise.all(urls.map(scrapeUrl));
    	const validResults = results.filter(r => r !== null);
    	console.log(validResults);
    }

    main();
    ```
  </Tab>
</Tabs>

The SDKs handle connection pooling, automatic retries, and concurrency limits automatically. The `concurrency=5` parameter ensures no more than 5 requests run simultaneously.

### Processing large URL lists with SDKs

For very large URL lists, process them in batches to manage memory usage:

<CodeGroup>
  ```python Python SDK - Batching theme={"dark"}
  from zenrows import ZenRowsClient
  import asyncio
  from bs4 import BeautifulSoup

  client = ZenRowsClient("YOUR_ZENROWS_API_KEY", concurrency=5, retries=1)

  async def scrape_url(url):
  	try:
  		response = await client.get_async(url)
  		if response.ok:
  			soup = BeautifulSoup(response.text, "html.parser")
  			return {
  				"url": url,
  				"title": soup.title.string if soup.title else "No title",
  				"h1": soup.find("h1").text if soup.find("h1") else "No H1"
  			}
  	except Exception as e:
  		return {"url": url, "error": str(e)}

  async def process_batch(urls_batch):
  	results = await asyncio.gather(*[scrape_url(url) for url in urls_batch])
  	return [r for r in results if r is not None]

  async def main():
  	# Your large list of URLs
  	all_urls = [
  		# ... hundreds or thousands of URLs
  	]
  	
  	batch_size = 50
  	all_results = []
  	
  	for i in range(0, len(all_urls), batch_size):
  		batch = all_urls[i:i + batch_size]
  		batch_results = await process_batch(batch)
  		all_results.extend(batch_results)
  		print(f"Processed batch {i//batch_size + 1}, got {len(batch_results)} results")
  	
  	print(f"Total results: {len(all_results)}")

  asyncio.run(main())
  ```

  ```javascript Node.js theme={"dark"}
  const { ZenRowsClient } = require('zenrows');
  const cheerio = require('cheerio');

  const client = new ZenRowsClient('YOUR_ZENROWS_API_KEY', { concurrency: 5, retries: 1 });

  async function scrapeUrl(url) {
  	try {
  		const response = await client.get(url);
  		if (response.ok) {
  			const $ = cheerio.load(response.data);
  			return {
  				url: url,
  				title: $('title').text() || "No title",
  				h1: $('h1').first().text() || "No H1"
  			};
  		}
  	} catch (error) {
  		return { url, error: error.message };
  	}
  }

  async function processBatch(urlsBatch) {
  	const results = await Promise.all(urlsBatch.map(scrapeUrl));
  	return results.filter(r => r !== null && !r.error);
  }

  async function main() {
  	// Your large list of URLs
  	const allUrls = [
  		// ... hundreds or thousands of URLs
  	];
  	
  	const batchSize = 50;
  	const allResults = [];
  	
  	for (let i = 0; i < allUrls.length; i += batchSize) {
  		const batch = allUrls.slice(i, i + batchSize);
  		const batchResults = await processBatch(batch);
  		allResults.push(...batchResults);
  		console.log(`Processed batch ${Math.floor(i/batchSize) + 1}, got ${batchResults.length} results`);
  	}
  	
  	console.log(`Total results: ${allResults.length}`);
  }

  main();
  ```
</CodeGroup>

This approach processes URLs in batches of 50, reducing memory usage and providing progress updates for long-running scraping jobs.

## Best Practices

Follow these guidelines for efficient URL list scraping:

1. **Start small** - Test with a few URLs before scaling up
2. **Monitor performance** - Track success rates and response times
3. **Handle errors gracefully** - Always include error handling for production code
4. **Respect rate limits** - Don't exceed your plan's concurrency limits
5. **Save progress** - For large jobs, save results incrementally to avoid data loss
6. **Use appropriate concurrency** - Balance speed with server respect and resource usage

## Understanding Concurrency Limits

The concurrency level you choose affects both performance and resource usage:

* **Target server capacity** - Some servers handle more concurrent requests than others
* **Your Zenrows plan limits** - Higher plans support more concurrent requests
* **Data processing speed** - Don't set concurrency higher than your ability to process results
* **Memory usage** - Higher concurrency uses more memory for storing responses

For detailed concurrency management and rate limiting information, check our [Concurrency Guide](/fetch/features/concurrency).

## Further Reading

* [Scrape and Crawl from a Seed URL](/zenrows-academy/scrape-and-crawl-from-a-seed-url)
* [Concurrency Management](/fetch/features/concurrency)
* [Error Handling Best Practices](/fetch/troubleshooting/troubleshooting-guide)
* <a href="https://github.com/ZenRows/zenrows-python-sdk">Zenrows Python SDK Repository</a>
* <a href="https://github.com/ZenRows/zenrows-node-sdk">Zenrows Node.js SDK Repository</a>