Scrape pages protected by anti-bot systems

Send a protected URL to Fetch with mode=auto and get the page a real visitor sees, not the challenge screen. Reach for this recipe whenever a plain request returns a block page, a CAPTCHA or a REQS002 error. Uses: Fetch with Adaptive Stealth Mode (mode=auto), js_render and premium_proxy

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use Zenrows to read https://www.scrapingcourse.com/antibot-challenge
and tell me the exact message shown in the challenge box.

Let Adaptive Stealth Mode pick the configuration

Set mode=auto. Zenrows starts with the cheapest configuration, adds JavaScript rendering and premium proxies only when the page needs them, and bills only the configuration that returns the page.

import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/antibot-challenge",
    "mode": "auto",
}

response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

print("Bypassed:", "You bypassed the Antibot challenge" in response.text)
print("Credits billed:", response.headers["X-Request-Credits"])
print("Cost (USD):", response.headers["X-Request-Cost"])

const params = new URLSearchParams({
  apikey: "YOUR_ZENROWS_API_KEY",
  url: "https://www.scrapingcourse.com/antibot-challenge",
  mode: "auto",
});

const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
  signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);

const html = await response.text();
console.log("Bypassed:", html.includes("You bypassed the Antibot challenge"));
console.log("Credits billed:", response.headers.get("X-Request-Credits"));
console.log("Cost (USD):", response.headers.get("X-Request-Cost"));

curl -s -D - -o page.html "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&mode=auto"

zenrows fetch https://www.scrapingcourse.com/antibot-challenge --json

The cURL command prints the response headers and saves the page to page.html. The CLI uses mode=auto by default and prints the billed credits after every request.

Example output

Bypassed: True
Credits billed: 25
Cost (USD): 0.025

The CLI’s --json result carries the same numbers:

{
  "ok": true,
  "mode": "auto",
  "httpStatus": 200,
  "costUsd": 0.025,
  "costCredits": 25,
  "finalUrl": "https://www.scrapingcourse.com/antibot-challenge"
}

Read which configuration was billed

Every Fetch response carries the charge in its headers, so you can log it next to each URL:

Header	Meaning	Example
`X-Request-Credits`	Credits this request consumed	`25`
`X-Request-Cost`	The same charge in USD (`0.001` = 1 credit)	`0.025`
`X-Request-Id`	ID to quote when you contact support	`6ad0054e...`

With mode=auto, the billed credits tell you which configuration succeeded: 1 for a basic request, 5 with JavaScript rendering, 10 with premium proxies, 25 with both. A domain that always bills 25 is a good candidate for the fixed configuration below. Requests that don’t succeed aren’t charged.

Fix the configuration with js_render and premium_proxy

When you already know a domain needs the full stack, set js_render=true and premium_proxy=true yourself. Every request then goes straight to that configuration. Send these parameters instead of mode=auto, never together with it.

import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://www.scrapingcourse.com/antibot-challenge",
    "js_render": "true",
    "premium_proxy": "true",
}

response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

print("Bypassed:", "You bypassed the Antibot challenge" in response.text)
print("Credits billed:", response.headers["X-Request-Credits"])

const params = new URLSearchParams({
  apikey: "YOUR_ZENROWS_API_KEY",
  url: "https://www.scrapingcourse.com/antibot-challenge",
  js_render: "true",
  premium_proxy: "true",
});

const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
  signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);

const html = await response.text();
console.log("Bypassed:", html.includes("You bypassed the Antibot challenge"));
console.log("Credits billed:", response.headers.get("X-Request-Credits"));

curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true"

zenrows fetch https://www.scrapingcourse.com/antibot-challenge --manual --js-render --premium-proxy --out page.html

The same code works on Cloudflare-protected pages: swap the URL for https://www.scrapingcourse.com/cloudflare-challenge and check for You bypassed the Cloudflare challenge.

Customize it

Change	How
Content from a specific country	Add `proxy_country=us` (any ISO code), see Get content as seen from any country
Wait for content that loads late	Use `js_render=true` with `wait_for=<selector>`, see Scrape JavaScript-rendered pages
Markdown instead of HTML	Add `response_type=markdown`, see Turn any web page into LLM-ready markdown
Get the target’s own status code	Add `original_status=true`, see Original HTTP code
Click or fill before reading	Add `js_instructions`, see Click, fill, and scroll before you scrape

Troubleshooting

400 with code REQS002: the domain needs JavaScript rendering, premium proxies or both. Send mode=auto, or add js_render=true and premium_proxy=true. See REQS002.
400 with code REQS004 mentioning mode: js_render or premium_proxy was sent together with mode=auto. Keep one approach per request.
422 with code RESP001: the page did not come back on that attempt. Retry with backoff and, on a fixed configuration, switch to mode=auto. See Retry failed requests.
499 or a client-side timeout: your HTTP client gave up first. Set its timeout to 180 seconds or more.
The page loads but a section is empty: the content arrives after the first render. Add wait_for with a selector of that section.

Scrape JavaScript-rendered pages

Render the page and wait for the exact element you need.

Retry failed requests

Back off, retry and escalate to mode=auto on 422.

Get content as seen from any country

Route the request through a residential IP in any country.

Track usage and control spend

Pick the cheapest configuration that works for each domain.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Scrape pages protected by anti-bot systems

Ask your AI agent

Let Adaptive Stealth Mode pick the configuration

Example output

Read which configuration was billed

Fix the configuration with js_render and premium_proxy

Customize it

Troubleshooting

Scrape JavaScript-rendered pages

Retry failed requests

Get content as seen from any country

Track usage and control spend

​Ask your AI agent

​Let Adaptive Stealth Mode pick the configuration

​Example output

​Read which configuration was billed

​Fix the configuration with js_render and premium_proxy

​Customize it

​Troubleshooting

​Related recipes

Scrape JavaScript-rendered pages

Retry failed requests

Get content as seen from any country

Track usage and control spend

Ask your AI agent

Let Adaptive Stealth Mode pick the configuration

Example output

Read which configuration was billed

Fix the configuration with js_render and premium_proxy

Customize it

Troubleshooting

Related recipes