Many modern websites use JavaScript to dynamically load content, meaning that the data you need might not be available in the initial HTML response. To handle such cases, you can use our JavaScript rendering feature, which simulates a real browser environment to fully load and render the page before extracting the data.
To activate JavaScript rendering, append js_render=true to the request. This tells our system to process the page using a headless browser, allowing you to scrape content that is loaded dynamically by JavaScript.
Enabling JavaScript rendering incurs a higher cost than standard requests. Five times the cost of a standard request
Several features rely on js_render being set to true. These include:
Wait: Introduces a delay before proceeding with the request. Useful for scenarios where you need to allow time for JavaScript to load content.
Wait For: Waits for a specific element to appear on the page before proceeding. When used with js_render, this parameter will cause the request to fail if the selector is not found.
JSON Response: Retrieves the rendered page content in JSON format, including data loaded dynamically via JavaScript.
Block Resources: Block specific types of resources from being loaded.
JavaScript Instructions: Allows you to execute custom JavaScript code on the page. This includes additional parameters.
Screenshot: Capture an above-the-fold screenshot of the target page by adding screenshot=true to the request.
For websites that take longer to load, you might need to introduce a fixed delay to ensure that all content is fully loaded before retrieving the HTML. You can specify this delay in milliseconds using the wait=10000 parameter.In this example, wait=10000 will cause ZenRows to wait for 10,000 milliseconds (or 10 seconds) before returning the HTML content. You can adjust this value based on your needs, with a maximum total allowable wait time of 30 seconds.
In some cases, you may need to wait for a specific CSS selector to be present in the DOM before ZenRows returns the content. This can be particularly useful for ensuring that dynamic elements or data have been fully loaded.To implement this, add the wait_for=.price parameter to your request URL. Replace .price with the CSS selector of the element you are targeting.
To capture and analyze the response of XHR, Fetch, or AJAX requests, you can use the json_response=true parameter in your API call. This will return a JSON object with detailed information about the page and its requests.
Why download and process data that you won’t be using? Blocking resources means preventing your headless browser from downloading specific types of content that you don’t need for your scraping task. This can include images, stylesheets, fonts, and other elements that might not be essential for your data extraction.To improve scraping efficiency, reduce loading times, optimize performance, and reduce bandwidth usage, you can block specific types of resources from being loaded using the block_resources parameter.
ZenRows automatically blocks certain resource types by default, such as stylesheets and images, to optimize scraping speed and reduce unnecessary data load. So we recommend not using this feature unless it’s really necessary.If you prefer to disable resource blocking entirely, set the parameter to “none”: block_resources=none.
Sometimes, blocking particular resources, especially Javascript files, results in an error or missing content. That might happen, for example, when the target website expects XHR calls after the initial render.Follow these steps to troubleshoot:
1
Compare HTML Outputs
Compare the Plain HTML obtained with ZenRows and a sample obtained manually. The HTML should be similar.
2
Adjust Blocked Resources
If essential elements are missing, test again by removing the blocked resources (likely JavaScript or XHR).
If the issue persists, please contact us, and we’ll assist you.
ZenRows provides an extensive set of JavaScript Instructions, allowing you to interact with web pages dynamically.These instructions enable you to click on elements, fill out forms, submit them, or wait for specific elements to appear, providing flexibility for tasks such as clicking the read more buttons or submitting forms.
To use JavaScript Instructions, you must include two parameters: js_render and js_instructions. The js_instructions parameter must be encoded.You can use our Builder or an online tool to encode the instructions.Here is an example of how to encode and use the instructions:
Copy
Ask AI
[ {"click": ".button-selector"}]
This set of instructions will load the page, locate the first element matching the .button-selector CSS selector, and click on it. The instructions parameter accepts an array of commands that ZenRows will execute sequentially.
Here are some common actions you can perform with JavaScript Instructions:
Copy
Ask AI
{"click": ".button-selector"} // Click on the first element that matches the CSS Selector{"wait_for": ".late-selector"} // Wait for a given CSS Selector to load in the DOM{"wait": 2000} // Wait an exact amount of time in ms{"fill": [".input-selector", "value"]} // Fill in an input{"check": ".checkbox-selector"} // Check a checkbox input{"uncheck": ".checkbox-selector"} // Uncheck a checkbox input{"select_option": [".select-selector", "option_value"]} // Select an option by its value{"scroll_y": 1500} // Vertical scroll in pixels{"scroll_x": 1500} // Horizontal scroll in pixels{"evaluate": "document.body.style.backgroundColor = '#c4b5fd';"} // Execute JavaScript code
The click action lets you programmatically interact with webpage elements like buttons or links. It’s essential for navigating sites or accessing additional content, such as expanding sections or moving through pagination.This action is often paired with wait_for to handle elements that load dynamically. For example, on some pages, you might click a read more button to reveal the full content of an article.
The wait_for instruction pauses the script until a specific element appears on the page, making it ideal for handling delayed content loading in Single Page Applications (SPAs) or dynamic websites.This ensures that all necessary elements, like data fields or navigation buttons, are fully loaded before further actions are taken. For instance, after clicking a button, you might use wait_for to ensure the next page’s key elements are present before proceeding with data extraction. This step is crucial for accurate and complete data retrieval in web scraping.
If the selector takes too long to load or is not present, the instruction will fail and move on to the next one
The wait instruction pauses execution for a specified duration, defined in milliseconds. For example, {"wait": 1000} pauses the script for one second.This can be useful for ensuring specific actions, such as animations or data loading processes, are given enough time to complete before proceeding with further steps. It’s a straightforward way to handle timing issues and ensure all elements are ready for interaction or extraction.
The fill instruction populates form fields with specified values, using a CSS selector to target the input element. This is particularly useful for automating form submissions, such as logging into a website.The syntax below specifies the CSS selector for the input field and the value to enter. This method allows you to automate interactions with web forms, making it easier to perform tasks like login automation or data entry.
The check instruction is used to select the checkbox or radio input elements on a webpage specified by a CSS selector. It helps ensure that options like’ Remember me’ are selected during form submissions.
Calling check on an already checked input will not uncheck it
The uncheck instruction is used to deselect checkbox or radio input elements on a webpage, specified by a CSS selector. This is useful for clearing default selections or ensuring specific options are not selected in forms.
To select an option from a dropdown menu, use the select_option instruction. This requires an array with two strings: the first is the CSS selector for the dropdown, and the second is the value of the option you want to select.
To scroll the page horizontally, use the scroll_y instruction with the number of pixels you want to scroll. Below is the example for scrolling 1500 px.
Use evaluate instructions to execute custom JavaScript on the page. If none of the previous ones fits your needs, you can write JavaScript code, and ZenRows will run it. Let’s say you want to scroll to a given element to trigger a “load more” event. Then, you can add another instruction to wait for the new part to load.
ZenRows bypasses most CAPTCHAs, but for in-page CAPTCHAs (such as those that appear when submitting forms) you can integrate a paid solver (2Captcha). To use it, add your API Key in the integrations section.You can solve various CAPTCHA types including reCAPTCHA and Cloudflare Turnstile. For invisible CAPTCHAs, send solve_inactive set as true inside options.
To ensure the CAPTCHA is solved before proceeding, add wait instructions before and after the CAPTCHA-solving step, allowing time for the CAPTCHA to load and be resolved.
Copy
Ask AI
[ {"wait": 3000}, // Wait for 3 seconds to allow the page to load {"solve_captcha": {"type": "recaptcha"}}, {"wait": 2000} // Wait 2 seconds to confirm CAPTCHA resolution]
Specific actions require waiting for the browser to finish an action or navigation. The service can wait for the browser to trigger an event like load or networkidle.
The instructions mentioned above won’t work inside iframes
Instructions for interacting with iframes are prefixed with frame_ and follow a similar syntax but require specifying the iframe.
Copy
Ask AI
[ {"frame_click": ["#iframe", ".read-more-selector"]} {"frame_click": ["#iframe", ".button-selector"]} {"frame_wait_for": ["#iframe", ".late-selector"]} {"frame_fill": ["#iframe", ".input-selector", "value"]} {"frame_check": ["#iframe", ".checkbox-selector"]} {"frame_uncheck": ["#iframe", ".checkbox-selector"]} {"frame_select_option": ["#iframe", ".select-selector", "option_value"]} {"frame_evaluate": ["iframe-name", "document.body.style.backgroundColor = '#c4b5fd';"]} // won't work with selectors, will match iframe's name or URL {"frame_reveal": "#iframe"} // will create a node with the class "iframe-content-element"]
For security, iframe’s content isn’t returned on the response. To get that content, use frame_reveal. It will append a node with the content encoded in base64 to avoid problems with JS or HTML inyection. The new node will have an attribute data-id with the given param and a iframe-content-element class.
In addition to CSS selectors, you can use XPath to locate elements on a web page.XPath is particularly useful when dealing with dynamic selectors or when more precise element selection is needed. However, if the website owner makes changes, the XPath might fail.
Copy
Ask AI
[ {"click": "//h2[text()='Example']"}]
In this example, ZenRows will find the first <h2> element that contains the text “Example” and click on it.The most common use cases for the XPath are:
Dynamic Content: When dealing with dynamic web pages with unreliable CSS selectors due to frequent changes.
Nested Elements: When selecting elements deeply nested within other elements.
Text-based Selection: Selecting elements based on their text content might not be easily achievable with CSS selectors.
To see a detailed report of the JS Instructions’ execution, set json_response to true. That will return the result in JSON format, one of the fields being js_instructions_report. Useful for testing and debugging. For more details, check the JSON Response documentation.Here is an example report:
It shows only part of the potential that this functionality adds. You could calculate different shipping prices by changing the shipping address. Or execute custom JavaScript logic with evaluate to click an element from a list. The possibilities are endless.Although we show examples with login forms, we discourage this usage. It would require you to log in to every request. If you need to scrape content as a logged-in user, don’t hesitate to contact us.
If your JavaScript Instructions are not interacting with the expected elements, the issue often lies with the selector used. Here are some quick steps to help you troubleshoot:
See if your instruction is failing: Add the parameter json_response=true to your request and check if the instruction is failing.
Double-check your selector: Make sure the CSS or XPath selector you’re using matches an element on the page. Open the page in your browser’s DevTools and test the selector directly in the console.
Wait for dynamic content: If the element is loaded dynamically, use the wait_for instruction to pause until it appears. If the selector is still not found, verify that the element is present in the final rendered DOM.
Check for frames/iframes: Elements inside iframes require specific instructions (e.g., frame_click). Standard selectors will not work on elements within iframes.
Selector specificity: Ensure your selector is specific enough to avoid matching multiple elements, but not so specific that it breaks if the page layout changes.
Use alternate strategies: If CSS selectors are unreliable, try using XPath, or leverage text-based or attribute-based selection.
If you continue to have issues, review the page structure in your browser’s DevTools, and consider if the element is loaded asynchronously or inside a shadow DOM or iframe.
Does enabling JavaScript rendering increase costs?
Yes, enabling JavaScript rendering incurs a higher cost — specifically, it is five times the cost of a standard request. This is due to the additional resources required to render the page fully.
What should I do if I encounter missing content after blocking resources?
If you notice that essential elements are missing after blocking certain resources (especially JavaScript), try removing the blocked resources to see if that resolves the issue. Essential elements may depend on JavaScript execution or XHR requests.
Can I use XPath for selecting elements in JavaScript instructions?
Yes, you can use XPath in addition to CSS selectors when using JavaScript instructions. XPath is especially useful for selecting dynamic or nested elements. Just ensure that the XPath expressions are correctly formulated.
How can I debug my JavaScript instructions?
To debug your JavaScript instructions, set json_response=true in your request. This will return a detailed execution report, including the success or failure of each instruction and the time taken for execution.
What should I do if my requests keep failing after implementing JavaScript instructions?
If your requests fail even after implementing JavaScript instructions, compare the HTML outputs obtained through ZenRows with a manually obtained sample. Ensure that the HTML structures are similar. Adjust the JavaScript instructions as needed based on the findings.