Using Premium Proxies + JS Render and still blocked
If you are scraping a site with high-security measures and encounter blocks even with Premium Proxies and JS Render enabled, try these steps to unblock your requests:
Add Geotargeting
Use geotargeting by selecting a country for the proxy, e.g., proxy_country=us. Many sites respond better to proxies close to their operation centers.
Use Wait For Selector
Implement Wait For Selector to have the scraper look for specific content before returning. This feature can change how the system interacts with the site and might help unblock the request.
Change Default Block Resources
Adjust the Block Resources settings. ZenRows blocks certain resources by default, such as CSS or images, to speed up scraping. Use your browser’s DevTools to identify other resources to block, such as media or xhr (block_resources=stylesheet,image,media,xhr). Alternatively, disable blocking by setting it to false (block_resources=none).
Add Custom Headers
Many websites inspect the headers of a request to determine if it is coming from a legitimate browser. Adding custom headers to mimic normal browser behavior can help bypass these checks.
Refer to our documentation on Custom Headers for more details.
Combining these methods may yield the expected results.
For high-security endpoints or inner pages, you may need to simulate a typical user session to avoid detection. First, obtain session cookies from a less protected page on the same site. This step mimics the initial user interaction with the site. Then, use these session cookies to access the more secure target page.
Additionally, you can use the Session ID feature to maintain the same IP address for up to 10 minutes, ensuring consistency in your requests and reducing the likelihood of being blocked.