> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows HTTP API serves Fetch and Extract (extract=auto) at https://api.zenrows.com/v1/, Crawl at https://api.zenrows.com/v1/crawls, and Batch at https://async.api.zenrows.com/v1. Fetch, Extract, Crawl and Batch are described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Every endpoint also has its own page under /api-reference with its schema. Use the spec for exact parameter names and types. Crawl's endpoints, parameters and errors are also at https://docs.zenrows.com/crawl/endpoints.md. To call Zenrows from an agent, connect the MCP server at https://mcp.zenrows.com/mcp. To search these docs from an agent, connect the docs MCP server at https://docs.zenrows.com/mcp. Agent skills for Zenrows are published at https://github.com/ZenRows/cli and install with `npx skills add ZenRows/cli`.

# Zenrows Fetch with Python: Getting Started

> Set up Zenrows Fetch with Python and Requests. Covers installation, API key configuration, and first scraping call.

Want to scrape the web effortlessly using Python? With Zenrows, you can handle complex scraping tasks, bypass anti-bot measures, and extract data from almost any website. This tutorial will guide you step-by-step in setting up a Zenrows scraper with Python.

## How to Use Zenrows with Python

Before we dive in, ensure you have Python 3 installed. If you're new to Python or web scraping, using an IDE such as [PyCharm](https://www.jetbrains.com/pycharm/) or [Visual Studio Code](https://code.visualstudio.com/docs/languages/python) with the Python extension is recommended for a smoother experience.

We'll create a Python script named `scraper.py` inside a `/scraper` directory. If you need help setting up your environment, check out our [Python web scraping guide](https://www.zenrows.com/blog/web-scraping-python#setup) for detailed instructions on preparing everything.

### Install Python's requests Library

To interact with the Zenrows API, you can use Python's `requests` library. It's a widely used Python library for making HTTP requests, and it simplifies sending requests and handling responses, making it a great tool for integrating with web services like Zenrows.

This approach allows you to manage your API requests directly, allowing greater control over the web scraping process.

To install the `requests` library, use the following command in your terminal:

```bash theme={"theme":"css-variables"}
pip install requests
```

This will install the library to send HTTP requests from your Python scripts.

### Make Your First Request

In this step, you will send your first request to Zenrows using the requests library to scrape content from a simple URL. We will use [HTTPBin.io/get](https://httpbin.io/get) endpoint to see how Zenrows processes the request and returns the data.

Here's an example:

```python scraper.py theme={"theme":"css-variables"}
# pip install requests
import requests

url = 'https://httpbin.io/get'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
```

Replace `YOUR_ZENROWS_API_KEY` with your actual API key and run the script:

```bash theme={"theme":"css-variables"}
python scraper.py
```

It'll print something similar to this:

```json theme={"theme":"css-variables"}
{
    "args": {},
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
        // additional headers omitted for brevity...
    },
    "origin": "38.154.5.224:6693",
    "url": "http://httpbin.io/get"
}
```

The response includes useful information, like the `origin`, which shows the IP address from which the request was made. Zenrows automatically rotates your IP address and changes the `User-Agent` for each request, helping to maintain anonymity and avoid blocks.

Perfect, you just learned how to make scraping requests with Python!

### Scrape More Complex Web Pages

While scraping simple sites like HTTPBin is straightforward, many websites, especially those with dynamic content or strict anti-scraping measures, require additional features. Zenrows allows you to bypass these defenses by enabling [JavaScript Rendering](https://docs.zenrows.com/fetch/features/js-rendering.md) and using [Premium Proxies](https://docs.zenrows.com/fetch/features/premium-proxy.md).

For example, if you try to scrape a page like [G2's Asana reviews](https://www.g2.com/products/asana/reviews) without any extra configurations, you'll encounter an error:

```json theme={"theme":"css-variables"}
{
    "code":"REQS002",
    "detail":"The requested URL domain needs JavaScript rendering and/or Premium Proxies due to its high-level security defenses. Please retry by adding 'js_render' and/or 'premium_proxy' parameters to your request.",
    "instance":"/v1",
    "status":400,
    "title":"Www.g2.com requires javascript rendering and premium proxies enabled (REQS002)",
    "type":"https://docs.zenrows.com/api-error-codes#REQS002"
}
```

This error happens because G2 employs advanced security measures that block basic scraping attempts.

Here's how you can modify the request to enable both:

```python scraper.py theme={"theme":"css-variables"}
# pip install requests
import requests

url = 'https://www.g2.com/products/asana/reviews'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
	'js_render': 'true',
	'premium_proxy': 'true',
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
```

Run the script, and this time, Zenrows will handle the heavy lifting by rendering the page's JavaScript and routing your request through premium residential proxies. The response will contain the entire HTML content of the page:

```html theme={"theme":"css-variables"}
<!DOCTYPE html>
<head>
    <meta charset="utf-8">
    <title>Asana Reviews 2024: Details, Pricing, & Features | G2</title>
    <!-- omitted for brevity -->
</head>
<body>
    <!-- page content -->
</body>
</html>
```

This demonstrates how you can scrape more advanced websites that rely on JavaScript or have stringent anti-bot mechanisms in place.

## Troubleshooting

Request failures can happen for various reasons. While some issues can be resolved by adjusting Zenrows parameters, others are beyond your control, such as the target server being temporarily down.

Below are some quick troubleshooting steps you can take:

**1. Retry the Request**

Network issues or temporary failures can cause your request to fail. Implementing retry logic can solve this by automatically repeating the request. Learn how to add retries in our [Python requests retry guide](https://www.zenrows.com/blog/python-requests-retry).

Example of retry logic using requests:

```python scraper.py theme={"theme":"css-variables"}
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Define the retry strategy
retry_strategy = Retry(
    total=4,  # Maximum number of retries
    status_forcelist=[429, 500, 502, 503, 504],  # HTTP status codes to retry on
)
# Create an HTTP adapter with the retry strategy and mount it to session
adapter = HTTPAdapter(max_retries=retry_strategy)

# Create a new session object
session = requests.Session()
session.mount('http://', adapter)
session.mount('https://', adapter)

# Make a request using the session object
response = session.get('https://scrapingcourse.com/ecommerce/')

if response.status_code == 200:
    print(f'SUCCESS: {response.text}')
else:
    print("FAILED")
```

**2. Verify the Site is Accessible in Your Country**

Sometimes, the target site might be region-restricted and only available to some proxies. Zenrows automatically selects the best proxy, but if the site is available only in specific regions, specify a geolocation using `proxy_country`.

Here's how to choose a proxy in the US:

```python theme={"theme":"css-variables"}
params = {
    'premium_proxy': 'true',
    'proxy_country': 'us' # <- choose a premium proxy in the US
    # other configs...
}
response = client.get(url, params=params)
```

If the target site requires access from a specific region, adding the `proxy_country` parameter will help.

> [!TIP]
>
> Check out more about it on our [Geolocation Documentation Page](https://docs.zenrows.com/fetch/features/proxy-country.md).

**3. Check if the Site is Publicly Accessible**

Some websites may require a session, so verifying if the site can be accessed without logging in is a good idea. Open the target page in an incognito browser to check this.

You must handle session management in your requests if authentication credentials are required. You can learn how to scrape a website that requires authentication in our guide: [Web scraping with Python](https://www.zenrows.com/blog/web-scraping-login-python).

**4. Get Help From Zenrows Experts**

If the issue persists despite following these tips, our support team is available to assist you. Use the Playground page or [contact us via email](mailto:success@zenrows.com) to get personalized help from Zenrows experts.

[Open the Playground](https://app.zenrows.com/fetch)

![Zenrows Request Playground Page](https://mintcdn.com/zenrows/bpr0c_CycI-XglbC/images/request_playground.png?fit=max\&auto=format\&n=bpr0c_CycI-XglbC\&q=85\&s=cbfeabccbfeaafb4c4a84c35664b2a3f)

## Frequently Asked Questions (FAQ)

> [accordion] How can I bypass CloudFlare and other protections?
>
> To reliably access sites behind CloudFlare or similar security mechanisms, enable both `js_render` and `premium_proxy` in your requests. These features simulate a full browser environment and route through high-quality residential proxies for protected access.
>
> You can also enhance your request by adding options like `wait` or `wait_for` to ensure the page fully loads before extracting data, improving accuracy.
>
> > [!TIP]
> >
> > Check out our documentation about the [wait](https://docs.zenrows.com/fetch/features/wait.md) and [wait\_for](https://docs.zenrows.com/fetch/features/wait-for.md) params

> [accordion] How can I ensure my requests don't fail?
>
> You can configure retry logic to handle failed HTTP requests. Learn more in our [guide on retrying requests](https://docs.zenrows.com/academy/retry-failed-requests.md).

> [accordion] How do I extract specific content from a page?
>
> You can use the `css_extractor` parameter to directly extract content from a page using CSS selectors. Find out more in our [tutorial on data parsing](https://docs.zenrows.com/academy/how-to-extract-data.md).

> [accordion] Can I integrate Zenrows with Python's Requests and BeautifulSoup?
>
> Yes! You can use Zenrows alongside Python Requests and BeautifulSoup for HTML parsing. Learn how in our guide on [Python Requests and BeautifulSoup integration](https://docs.zenrows.com/academy/how-to-extract-data.md#python-with-beautifulsoup).

> [accordion] How can I simulate user interactions on the target page?
>
> Use the `js_render` and `js_instructions` features to simulate actions such as clicking buttons or filling out forms. Discover more about interacting with web pages in our [JavaScript instructions guide](https://docs.zenrows.com/fetch/features/js-instructions.md).

> [accordion] How can I scrape faster using Zenrows?
>
> You can scrape multiple URLs simultaneously by making concurrent API calls. Check out our guide on [using concurrency](https://docs.zenrows.com/fetch/features/concurrency.md#using-concurrency) to boost your scraping speed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.