Make Your First Request with ZenRows' Universal Scraper API

Learn how to extract data from any website using ZenRows’ Universal Scraper API. This guide walks you through creating your first scraping request that can handle sites at any scale.

ZenRows’ Universal Scraper API is designed to simplify web scraping. Whether you’re dealing with static content or dynamic JavaScript-heavy sites, you can get started in minutes with any programming language that supports HTTP requests.

​

1. Set Up Your Project

​

Set Up Your Development Environment

Before diving in, ensure you have the proper development environment and required HTTP client libraries for your preferred programming language. ZenRows works with any language that can make HTTP requests.

# Install Python (if not already installed)
# Visit https://www.python.org/downloads/ or use package managers:

# macOS (using Homebrew)
brew install python

# Ubuntu/Debian
sudo apt update && sudo apt install python3 python3-pip

# Windows (using Chocolatey)
choco install python

# Install the requests library
pip install requests

# Install Python (if not already installed)
# Visit https://www.python.org/downloads/ or use package managers:

# macOS (using Homebrew)
brew install python

# Ubuntu/Debian
sudo apt update && sudo apt install python3 python3-pip

# Windows (using Chocolatey)
choco install python

# Install the requests library
pip install requests

# Install Node.js (if not already installed)
# Visit https://nodejs.org/ or use package managers:

# macOS (using Homebrew)
brew install node

# Ubuntu/Debian (using NodeSource)
curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y nodejs

# Windows (using Chocolatey)
choco install nodejs

# Install the axios library
npm install axios

# Install Java (if not already installed)
# Visit https://adoptium.net/ or use package managers:

# macOS (using Homebrew)
brew install openjdk

# Ubuntu/Debian
sudo apt update && sudo apt install openjdk-17-jdk

# Windows (using Chocolatey)
choco install openjdk

<dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5-fluent</artifactId>
    <version>5.2.1</version>
</dependency>

# Install PHP (if not already installed)
# Visit https://www.php.net/downloads or use package managers:

# macOS (using Homebrew)
brew install php

# Ubuntu/Debian
sudo apt update && sudo apt install php php-curl

# Windows (using Chocolatey)
choco install php

# Install Go (if not already installed)
# Visit https://golang.org/dl/ or use package managers:

# macOS (using Homebrew)
brew install go

# Ubuntu/Debian
sudo apt update && sudo apt install golang-go

# Windows (using Chocolatey)
choco install golang

# Install Ruby (if not already installed)
# Visit https://www.ruby-lang.org/en/downloads/ or use package managers:

# macOS (using Homebrew)
brew install ruby

# Ubuntu/Debian
sudo apt update && sudo apt install ruby-full

# Windows (using Chocolatey)
choco install ruby

# Install the faraday gem
gem install faraday

# cURL is typically pre-installed on most systems. If not, install it:

# macOS (using Homebrew)
brew install curl

# Ubuntu/Debian
sudo apt update && sudo apt install curl

# Windows (using Chocolatey)
choco install curl

​

Get Your API Key

Sign up for a free ZenRows account and get your API key from the Builder dashboard. You’ll need this key to authenticate your requests.

​

2. Make Your First Request

Start with a simple request to understand how ZenRows works. We’ll use the HTTPBin.io/get endpoint to demonstrate how ZenRows processes requests and returns data.

# pip install requests
import requests

url = 'https://httpbin.io/get'
apikey = 'YOUR_ZENROWS_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
}

response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)

Replace YOUR_ZENROWS_API_KEY with your actual API key and run the script:

python your_script.py

​

Expected Output

The script will print the contents of the website, for HTTPBin.io/get it’s something similar to this:

{
    "args": {},
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
        // additional headers omitted for brevity...
    },
    "origin": "38.154.5.224:6693",
    "url": "http://httpbin.io/get"
}

Perfect! You’ve just made your first web scraping request with ZenRows.

​

3. Scrape More Complex Websites

Modern websites often use JavaScript to load content dynamically and employ sophisticated anti-bot protection. ZenRows provides powerful features to handle these challenges automatically.

Use the Request Builder in your ZenRows dashboard to easily configure and test different parameters. Enter the target URL (for this demonstration, the Anti-bot Challenge page) in the URL to Scrape field to get started.

​

Use Premium Proxies

Premium Proxies provide access to over 55 million residential IP addresses from 190+ countries with 99.9% uptime, ensuring the ability to bypass sophisticated anti-bot protection.

​

Enable JavaScript Rendering

JavaScript Rendering uses a real browser to execute JavaScript and capture the fully rendered page. This is essential for modern web applications, single-page applications (SPAs), and sites that load content dynamically.

​

Combine Features for Maximum Success

For the most protected sites, enable both JavaScript Rendering and Premium Proxies. This provides the highest success rate for challenging targets.

import requests

url = 'https://www.scrapingcourse.com/antibot-challenge'
apikey = 'YOUR_ZENROWS_API_KEY'

params = {
    'url': url,
    'apikey': apikey,
    'js_render': 'true',
    'premium_proxy': 'true',
}

response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)

This code sends a GET request to the ZenRows API endpoint with your target URL and authentication. The js_render parameter enables JavaScript processing, while premium_proxy routes your request through residential IP addresses.

​

Run Your Application

Execute your script to test the scraping functionality and verify that your setup works correctly.

python your_script.py

​

Example Output

Run the script, and ZenRows will handle the heavy lifting by rendering the page’s JavaScript and routing your request through premium residential proxies. The response will contain the entire HTML content of the page:

<html lang="en"><head>
    <title>Antibot Challenge - ScrapingCourse.com</title>
</head><body>
    <h2>You bypassed the Antibot challenge! :D</h2>
</body></html>

Congratulations! You now have a ZenRows integration that can scrape websites at any scale while bypassing anti-bot protection. You’re ready to tackle more advanced scenarios and customize the API to fit your scraping needs.

​

Troubleshooting

Request failures can happen for various reasons. While some issues can be resolved by adjusting ZenRows parameters, others are beyond your control, such as the target server being temporarily down.

Below are some quick troubleshooting steps you can take:

params = {
    'premium_proxy': 'true',
    'proxy_country': 'us' # <- choose a premium proxy in the US
    # other configs...
}
response = requests.get('https://api.zenrows.com/v1/', params=params)

# Use a wait parameter or a wait_for parameter
params = {
    'wait': '3000', # <- Adds a delay of 3 seconds
    # other configs...
}
response = requests.get('https://api.zenrows.com/v1/', params=params)

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Define the retry strategy
retry_strategy = Retry(
    total=4,  # Maximum number of retries
    status_forcelist=[429, 500, 502, 503, 504],  # HTTP status codes to retry on
)
# Create an HTTP adapter with the retry strategy and mount it to session
adapter = HTTPAdapter(max_retries=retry_strategy)

# Create a new session object
session = requests.Session()
session.mount('http://', adapter)
session.mount('https://', adapter)

# Make a request using the session object
response = session.get('https://scrapingcourse.com/ecommerce/')

if response.status_code == 200:
    print(f'SUCCESS: {response.text}')
else:
    print("FAILED")

​

Next Steps

You now have a solid foundation for web scraping with ZenRows. Here are some recommended next steps to take your scraping to the next level:

• Complete API Reference:
  Explore all available parameters and advanced configuration options to customize ZenRows for your specific use cases.
• JavaScript Instructions Guide:
  Learn how to perform complex page interactions like form submissions, infinite scrolling, and multi-step workflows.
• Output Formats and Data Extraction:
  Learn advanced data extraction with CSS selectors, output formats including Markdown and PDF conversion, and screenshot configurations.
• Pricing and Plans:
  Understand how request costs are calculated and choose the plan that best fits your scraping volume and requirements.

​

Frequently Asked Questions (FAQ)