FAQ
Frequently Asked Questions
Getting Started
ZenRows® makes scraping a breeze and integrates easily in any development flow you might have. We offer both API and Proxy options for connection.
We offer a complete set of features. For more advanced options, such as JavaScript Rendering or Premium Proxies, check out the documentation.
You’ll need an API Key to be able to connect, register here or go and copy yours.
You will only need this API Key and a target URL for simple requests, which must be encoded.
Encoding URLs
It would be best to encode target URLs to avoid a problem with ZenRows API and your URLs’ possible search parameters. We will decode it on our side, and there should be no problem.
Let’s take a Google search example: https://www.google.com/search?q=scraping&channel=fs. If you were to send that string without encoding but with Premium Proxy param active, the call would be like this.
As you can see, there might be confusion there with the channel and premium_proxy parameters. You can use an online tool to encode your URL, but that does not scale. Some clients (like axios or requests) do that for you, but in case they don’t, most programming languages have functions to do it.
Once encoded, this is how the result would look like.
What is JavaScript Rendering
Some websites rely heavily on JavaScript to load content. That makes the initial HTML unusable from a scraping point of view. All the actual content loads dynamically in subsequent calls.
Long story short, we need to load the initial HTML (as always) but then follow all the instructions in JavaScript to call backend endpoints to populate the data. And that’s the reason we need JavaScript Rendering active for those sites.
For performance reasons, scrapers tend to avoid JS Rendering and try to load the minimum possible. But when needed, a headless browser will do the work, which is precisely what JS Rendering parameter activates. ZenRows will call the target URL with a real browser and perform all subsequent calls when sent to true, thus having access to the data that was not present initially.
Here is a sample call in cURL with JS Rendering active. Remember to encode the URL.
The good thing about loading headless browsers is that it allows some excellent features as waiting for an element to be present.
Wait for Element or Certain Time
When loading a page with JavaScript Rendering, we get the feature to wait for an element to be present.
By default, ZenRows will wait until there are no network requests. But that sometimes is not enough for whatever reason (maybe the page loads certain content a couple of seconds after the initial load). In those cases, the usual thing to do is wait for an element to be present. To do that, you’ll need to inspect the target page by hand (i.e., DevTools) and locate said element. Let’s say it is an element with a .my-test class. You’ll need to send that in the wait_for parameter.
Another option to wait is to set a concrete number. In cases when there isn’t a selector, or you’re not sure, you can wait for, say, 3 seconds. Do that with the wait param and send the number in milliseconds! Careful with this; wait=3 will not wait 3 seconds.
The API will modify its default behavior when called with either of these parameters and will wait until that class is present or the specified time.
If it still does not work after trying this, contact us and we’ll help you set up your calls.
What is Geotargeting or Regional IP
There are two primary cases when you want regional IPs: localized content and geoblocking. Amazon, for example, will show different products on its .uk and .fr sites. And others like CNN will block visits from outside the US.
To avoid problems with retailers showing different products every visit, you can send a country code in the request. That will effectively localize the request to the country you want, achieving results you can reliably replicate.
You will also need to send the Premium Proxy parameter to true to geotarget the request.
List of some available countries and their code, you can check out the whole list on the builder.
- United States: us
- Canada: ca
- United Kingdom: gb
- Germany: de
- France: fr
- Spain: es
- Brazil: br
- Mexico: mx
- India: in
- Japan: jp
- China: cn
What are Residential IPs
In scraping proxies, there are two main IP types: data center and residential. To summarize, data center IPs work reliably but might be listed and thus easier to block. Residential, however, are harder to stop since they belong to a provider (ISP) that might assign it to an actual user.
If not stated in the requests, ZenRows will use data center connections. But for a better success rate or accessing heavy banners like Google, you can send premium_proxy to true. Note that residential IPs come at an extra cost.
We also offer localization or geotargeting when using Residential.
If the blocks continue, contact us and we’ll help you.
CSS Selectors do not work or “parser is not valid”
The main error with CSS Selector is not encoding the content properly. You can use our Builder or an online tool to encode it. Then send it as css_extractor, in the example we’ll get the contents of .my-class selector in a test property: {"test": ".my-class"}.
If you are still getting empty responses, try to get Plain HTML. The content might be different from the one you are getting on your browsers. Maybe they are blocking the requests or serving localized content based on another country.
To check that the selector is correct, please review the docs and test each selector in a browser (i.e., using Chrome Dev Tools’ Console: document.querySelectorAll(".my-class")).
If the HTML looks good, the selector works on the browser, but the parser does not work, contact us and we’ll help you.
What is Autoparse
ZenRows has custom parsers written for some domains that will work out-of-the-box. When calling the API, the default return type will be Plain HTML. You need to activate it with the autoparse parameter. The output in those cases will be JSON instead of HTML.
The results will be missing or empty for sites not on the list. Try removing the autoparse parameter a try again. If you want custom scrapers built, contact us.
Builder
The Builder will automatically activate the autoparse option for the sites we offer. If you paste a URL, say https://www.amazon.com/dp/B01LD5GO7I/, you will see that Data Extraction changes from Plain HTML to Autoparse. You can disable it by clicking on Plain HTML.
Click on “Perform request in the browser” to see the result. ZenRows will convert the HTML into a structured result. The example case will output price, rating, category, description, and several other fields.
How to Send Custom Headers
You can add Custom Headers to requests, but not as parameters as usual. There are several standard fields such as Accept, Cookie, Referer, or User-Agent. The format is Header: Value without quotes; see examples below.
ZenRows sends domain-tailored headers to achieve the highest possible success rate. Custom Headers will overwrite the default ones, which might cause a drop in the success rate. That might happen if you send, for example, a User-Agent that usually goes with other headers that are missing, like Sec-Ch-Ua for Chrome.
It is essential to set custom_headers to true.
As stated above, the success rate might decrease due to header changes. If you find yourself there and need to send a custom header, contact us and we’ll help you.
SSL Certificate Error
If you are getting an SSL Certificate Error (i.e., SSLCertVerificationError in python), try sending the request with verification disabled.
If the problem continues, contact us and we’ll help you.
Concurrency
Concurrency refers to the number of ongoing requests that happen at any given time. By different means, computers and languages can call the API in parallel and wait for results while others are still running. You can use concurrency with any ZenRows plan; check out pricing for more details.
For more details, check our how-to guide on concurrency to see details about implementation in Python and JavaScript.
How many concurrent requests are included in my plan?
The concurrency limit starts at 5 concurrent requests for free trial plans and the maximum concurrency increases the bigger your plan is:
- Trial plan: 5 concurrent requests (see what ZenRows trial includes)
- Developer plan: 20 concurrent requests
- Startup plan: 50 concurrent requests
- Business plan: starting at 100 concurrent requests
Enterprise plans can include custom concurrency limits to fit your needs. Contact us for tailor-made Enterprise plans.
If get a “429 Too Many Requests” error, do I lose that request or is it queued?
You’ll receive an error, and that request won’t be queued or retried automatically. You’ll need to manage retries on your end, ensuring you don’t exceed your concurrency limit.
Can I increase my concurrency limit?
Absolutely! We offer various plans with different concurrency limits to suit your needs. If you find yourself frequently hitting the concurrency limit, consider upgrading.
How can I speed up requests?
Using js_render=true parameter will make requests slower (as JS rendering is a slower process).
Also, waiting for specific conditions (using wait or wait_for parameters) will slow down the process as conditions must pass for the request to end and the concurrency slot to be released.
We recommend you be extremely careful when using these parameters.
The use of the premium_proxy parameter will speed up your requests, as the better the proxy to use to retrieve the contents, the faster it will be and the less time a request will take.
I’ve been blocked by repeatedly exceeding my concurrency limit. Why?
Whenever you exceed your plan concurrency limit, you’ll start receiving “429 Too Many Requests” errors. If you keep sending more and more requests exceeding your plan concurrency limit in a short period of time, the service may temporarily block your IP address to prevent API misuse.
The IP address ban will last only a few minutes, but repeatedly being blocked might end in a long-lasting block. Check the concurrency optimization section for more information on how to limit concurrent requests to prevent being blocked.
Was this page helpful?