> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.
# Scrape pages protected by anti-bot systems
> Get the real content of pages behind Cloudflare and other anti-bot systems with Zenrows Fetch, and see what each request cost.
Send a protected URL to Fetch with `mode=auto` and get the page a real visitor sees, not the challenge screen. Reach for this recipe whenever a plain request returns a block page, a CAPTCHA or a `REQS002` error.
**Uses:** [Fetch](/fetch/api-reference) with [Adaptive Stealth Mode](/fetch/features/adaptive-stealth-mode) (`mode=auto`), [`js_render`](/fetch/features/js-rendering) and [`premium_proxy`](/fetch/features/premium-proxy)
## Ask your AI agent
With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.
```text theme={"dark"}
Use Zenrows to read https://www.scrapingcourse.com/antibot-challenge
and tell me the exact message shown in the challenge box.
```
## Let Adaptive Stealth Mode pick the configuration
Set `mode=auto`. Zenrows starts with the cheapest configuration, adds JavaScript rendering and premium proxies only when the page needs them, and bills only the configuration that returns the page.
```python Python (Requests) theme={"dark"}
import requests
params = {
"apikey": "YOUR_ZENROWS_API_KEY",
"url": "https://www.scrapingcourse.com/antibot-challenge",
"mode": "auto",
}
response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()
print("Bypassed:", "You bypassed the Antibot challenge" in response.text)
print("Credits billed:", response.headers["X-Request-Credits"])
print("Cost (USD):", response.headers["X-Request-Cost"])
```
```javascript Node.js theme={"dark"}
const params = new URLSearchParams({
apikey: "YOUR_ZENROWS_API_KEY",
url: "https://www.scrapingcourse.com/antibot-challenge",
mode: "auto",
});
const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
const html = await response.text();
console.log("Bypassed:", html.includes("You bypassed the Antibot challenge"));
console.log("Credits billed:", response.headers.get("X-Request-Credits"));
console.log("Cost (USD):", response.headers.get("X-Request-Cost"));
```
```bash cURL theme={"dark"}
curl -s -D - -o page.html "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto"
```
```bash CLI theme={"dark"}
zenrows fetch https://www.scrapingcourse.com/antibot-challenge --json
```
The cURL command prints the response headers and saves the page to `page.html`. The CLI uses `mode=auto` by default and prints the billed credits after every request.
## Example output
```text theme={"dark"}
Bypassed: True
Credits billed: 25
Cost (USD): 0.025
```
The CLI's `--json` result carries the same numbers:
```json theme={"dark"}
{
"ok": true,
"mode": "auto",
"httpStatus": 200,
"costUsd": 0.025,
"costCredits": 25,
"finalUrl": "https://www.scrapingcourse.com/antibot-challenge"
}
```
## Read which configuration was billed
Every Fetch response carries the charge in its headers, so you can log it next to each URL:
| Header | Meaning | Example |
| - | - | - |
| `X-Request-Credits` | Credits this request consumed | `25` |
| `X-Request-Cost` | The same charge in USD (`0.001` = 1 credit) | `0.025` |
| `X-Request-Id` | ID to quote when you contact support | `6ad0054e...` |
With `mode=auto`, the billed credits tell you which configuration succeeded: `1` for a basic request, `5` with JavaScript rendering, `10` with premium proxies, `25` with both. A domain that always bills `25` is a good candidate for the fixed configuration below. Requests that don't succeed aren't charged.
## Fix the configuration with js\_render and premium\_proxy
When you already know a domain needs the full stack, set `js_render=true` and `premium_proxy=true` yourself. Every request then goes straight to that configuration. Send these parameters instead of `mode=auto`, never together with it.
```python Python (Requests) theme={"dark"}
import requests
params = {
"apikey": "YOUR_ZENROWS_API_KEY",
"url": "https://www.scrapingcourse.com/antibot-challenge",
"js_render": "true",
"premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()
print("Bypassed:", "You bypassed the Antibot challenge" in response.text)
print("Credits billed:", response.headers["X-Request-Credits"])
```
```javascript Node.js theme={"dark"}
const params = new URLSearchParams({
apikey: "YOUR_ZENROWS_API_KEY",
url: "https://www.scrapingcourse.com/antibot-challenge",
js_render: "true",
premium_proxy: "true",
});
const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
const html = await response.text();
console.log("Bypassed:", html.includes("You bypassed the Antibot challenge"));
console.log("Credits billed:", response.headers.get("X-Request-Credits"));
```
```bash cURL theme={"dark"}
curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true"
```
```bash CLI theme={"dark"}
zenrows fetch https://www.scrapingcourse.com/antibot-challenge --manual --js-render --premium-proxy --out page.html
```
The same code works on Cloudflare-protected pages: swap the URL for `https://www.scrapingcourse.com/cloudflare-challenge` and check for `You bypassed the Cloudflare challenge`.
## Customize it
| Change | How |
| - | - |
| Content from a specific country | Add `proxy_country=us` (any ISO code), see [Get content as seen from any country](/academy/fetch/geotargeting) |
| Wait for content that loads late | Use `js_render=true` with `wait_for=`, see [Scrape JavaScript-rendered pages](/academy/fetch/javascript-pages) |
| Markdown instead of HTML | Add `response_type=markdown`, see [Turn any web page into LLM-ready markdown](/academy/fetch/markdown-for-llms) |
| Get the target's own status code | Add `original_status=true`, see [Original HTTP code](/fetch/features/other#original-http-code) |
| Click or fill before reading | Add `js_instructions`, see [Click, fill, and scroll before you scrape](/academy/fetch/page-interactions) |
## Troubleshooting
* **`400` with code `REQS002`**: the domain needs JavaScript rendering, premium proxies or both. Send `mode=auto`, or add `js_render=true` and `premium_proxy=true`. See [REQS002](/api-error-codes#REQS002).
* **`400` with code `REQS004` mentioning `mode`**: `js_render` or `premium_proxy` was sent together with `mode=auto`. Keep one approach per request.
* **`422` with code `RESP001`**: the page did not come back on that attempt. Retry with backoff and, on a fixed configuration, switch to `mode=auto`. See [Retry failed requests](/academy/retry-failed-requests).
* **`499` or a client-side timeout**: your HTTP client gave up first. Set its timeout to 180 seconds or more.
* **The page loads but a section is empty**: the content arrives after the first render. Add `wait_for` with a selector of that section.
## Related recipes
Render the page and wait for the exact element you need.
Back off, retry and escalate to mode=auto on 422.
Route the request through a residential IP in any country.
Pick the cheapest configuration that works for each domain.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.