> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Retry Failed Requests
> Set up auto-retry policies for failed web scraping requests in Python and JavaScript using ZenRows with exponential backoff and error handling.
Implementing auto-retry policies in web scraping is crucial for maintaining data accuracy and reliability, especially since achieving a 100% success rate is difficult. Factors such as site downtimes or interrupted connections can cause occasional failures.
However, with ZenRows®, you are not charged for these failed attempts. For instance, if you make 130 requests and 15 need to be retried, you will only be charged for the 115 successful ones, providing a cost-effective solution.
Below, we offer detailed solutions in Python and JavaScript to help you implement these retry policies. If you require support for other languages or frameworks, please feel free to contact us.
ZenRows® Python SDK and JavaScript SDK offer built-in support for retries. Simply pass a `retries` parameter in the constructor to enable this feature and enhance your scraping efficiency.
## Python with Requests
To begin, ensure you have Python 3 installed on your system. Some systems come with it pre-installed. Once Python is ready, install the necessary libraries by running the following command:
```bash theme={null}
pip install requests
```
We will use the `Retry` class from `urllib3` and the `HTTPAdapter` from requests. These are part of the `requests` library, so you don't need to install them separately.
Instead of making direct `get` calls, we create a requests session. This approach allows us to set up and reuse the session configuration, including retry settings, for all subsequent requests. This setup is efficient and ensures consistent retry behavior across all requests.
You can adjust the `Retry` parameters to suit your needs. Here are some key parameters:
* `total` sets the maximum number of retries allowed.
* `backoff_factor` defines the delay between retries using an exponential backoff strategy. The formula is: **$\text{{backoff\_factor}} \times (2^{(\text{{number of total retries}} - 1)})$**. For example, a backoff factor of 1 results in delays of 1, 2, and 4 seconds for three retries.
* `status_forcelist` is a list of HTTP status codes that will force a retry.
```python scraper.py theme={null}
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
apikey = "YOUR_ZENROWS_API_KEY"
urls = [
"" # ... your URLs here
]
zenrows_api_base = "https://api.zenrows.com/v1/"
requests_session = requests.Session()
retries = Retry(
total=3,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504]
)
requests_session.mount("https://", HTTPAdapter(max_retries=retries))
for url in urls:
try:
response = requests_session.get(zenrows_api_base, params={
"apikey": apikey,
"url": url,
})
print(response.text) # process response
except Exception as e:
print(e) # will print "Max retries exceeded"
```
In this example, we use a list of URLs and process them sequentially. For improved performance, you can process them [concurrently with a maximum](/universal-scraper-api/features/concurrency#using-concurrency). This approach is particularly useful when dealing with large numbers of URLs or when response times are slow.
## JavaScript with axios-retry
To get started, ensure you have Node.js (or nvm) and npm installed on your system. Many systems come with these pre-installed. Once set up, install the necessary libraries by running:
```bash theme={null}
npm install axios axios-retry
```
Instead of making direct calls to the ZenRows API using `axios`, we'll use the axiosRetry library to manage retries. This configuration will automatically retry requests for all `axios` calls, and you can also create a specific axios `client` with retry capabilities.
You can adjust the `axiosRetry` parameters to fit your specific requirements. The configuration provided below should work well for most scenarios:
* `retries` sets the number of allowed retries.
* `retryDelay` this applies an exponential delay between attempts, with an additional 0-20% random delay margin to prevent repeated requests from hitting the server at regular intervals.
* `retryCondition` this function determines whether an error is eligible for a retry. By default, the function retries only 5xx errors. However, in our example, we've added a check for 429 (Too Many Requests) errors to ensure these are also retried.
```javascript theme={null}
const axios = require("axios");
const axiosRetry = require("axios-retry");
const apikey = "YOUR_ZENROWS_API_KEY";
const zenrowsApiBase = "https://api.zenrows.com/v1/";
const urls = [
// ... your URLs here
];
axiosRetry(axios, {
retries: 3,
retryDelay: axiosRetry.exponentialDelay,
retryCondition: (error) => {
if (error.response && error.response.status === 429) {
return true; // example for custom retry condition
}
// fallback to default condition
return axiosRetry.isNetworkOrIdempotentRequestError(error);
},
});
(async () => {
for (const url of urls) {
try {
const response = await axios.get(zenrowsApiBase, {
params: { apikey, url },
});
console.log(response.data); // process response
} catch (error) {
console.log(error);
}
}
})();
```
The example processes a list of URLs sequentially for simplicity. However, you can also execute them concurrently, allowing multiple requests to be handled simultaneously.
This approach can be particularly useful when dealing with a large number of URLs, improving efficiency and reducing the overall time required. For more information on running requests concurrently, refer to our guide on using [concurrency with ZenRows SDK for JavaScript](/universal-scraper-api/features/concurrency#zenrows-sdk-for-javascript).