Headless browser / JavaScript rendering

Some websites rely heavily on JavaScript to load content. Enable this feature if you need to extract data that are loaded dynamically.

You can enable JavaScript by adding &js_render=true to the request. This request costs 5 credits.

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"
params = {
    "js_render": "true"
}

response = client.get(url, params=params)

print(response.text)

Step-by-step tutorial: How to overcome Cloudflare with ZenRows.

​

JavaScript Instructions

Interact with the page once the content is loaded. You can perform actions as a user would (i.e., click on an element), and ZenRows will execute them. Once the Instructions finish, it will return the current HTML.

Following the click example, below are the instructions to click on a ".button-selector" element.

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"
params = {
    "js_render": "true",
    "js_instructions": "[{\"click\":\".button-selector\"}]"
}

response = client.get(url, params=params)

print(response.text)

The original instructions are a JSON array containing the commands to run.

[{"click": ".button-selector"}]

They then need to be stringified and encoded. You can use our Builder or an online tool to encode it. Tools like requests will do it for you if passed as parameters.

`[{"click":".button-selector"}]` // stringified
`%5B%7B%22click%22%3A%22.button-selector%22%7D%5D` // encoded

&js_instructions=[{...}] accepts an array of commands, and you can add as many as needed. ZenRows will execute them in order. Here is a summary of the actions you can run.

{"click": ".button-selector"} // Click on the first element that matches the CSS Selector
{"wait_for": ".late-selector"} // Wait for a given CSS Selector to load in the DOM
{"wait": 2000} // Wait an exact amount of time in ms
{"fill": [".input-selector", "value"]} // Fill in an input
{"check": ".checkbox-selector"} // Check a checkbox input
{"uncheck": ".checkbox-selector"} // Uncheck a checkbox input
{"select_option": [".select-selector", "option_value"]} // Select an option by its value
{"scroll_y": 1500} // Vertical scroll in pixels
{"scroll_x": 1500} // Horizontal scroll in pixels
{"evaluate": "document.body.style.backgroundColor = '#c4b5fd';"} // Execute JavaScript code
{"solve_captcha": {"type": "recaptcha"}} // Solve a CAPTCHA given the type
{"wait_event": "networkidle"} // Wait for a browser event or status (networkidle, networkalmostidle, load, domcontentloaded)

These instructions won’t work inside iframes, we need another set for that. The syntax is similar but with an extra parameter to choose the iframe.

For security, iframe’s content isn’t returned on the response. To get that content, use frame_reveal. It will append a node with the content encoded in base64 to avoid problems with JS or HTML injection.

{"frame_click": ["#iframe", ".button-selector"]}
{"frame_wait_for": ["#iframe", ".late-selector"]}
{"frame_fill": ["#iframe", ".input-selector", "value"]}
{"frame_check": ["#iframe", ".checkbox-selector"]}
{"frame_uncheck": ["#iframe", ".checkbox-selector"]}
{"frame_select_option": ["#iframe", ".select-selector", "option_value"]}
{"frame_evaluate": ["iframe-name", "document.body.style.backgroundColor = '#c4b5fd';"]} // won't work with selectors, will match iframe's name or URL
{"frame_reveal": "#iframe"} // will create a node with the class "iframe-content-element"

Requires javascript rendering (&js_render=true).

Visit our JavaScript Instructions guide for a detailed explanation for each action and usage examples.

Note: to see a detailed report of the instructions’ execution, set json_response to true. That will return the result in JSON format, one of the fields being js_instructions_report. Useful for testing and debugging. For more details, check the JSON Response section.

​

Wait For Selector

Sometimes you may want to wait for a given CSS Selector to load in the DOM before ZenRows returns the content. You can get this behaviour by adding &wait_for=.background-load parameter into the request.

Requires javascript rendering (&js_render=true).

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"
params = {
    "js_render": "true",
    "wait_for": ".content"
}

response = client.get(url, params=params)

print(response.text)

​

Wait Milliseconds

Some websites take a lot time to load. If you need to wait a fixed amount of time until everything is loaded, you can define the time in milliseconds with &wait=10000 parameter, which will wait 10000 milliseconds (10 seconds) before returning the HTML. The maximum wait time is 30 seconds.

Requires javascript rendering (&js_render=true).

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"
params = {
    "js_render": "true",
    "wait": 10000
}

response = client.get(url, params=params)

print(response.text)

​

Block Resources

Many websites load dozens of resources delaying the HTML response. You can block specific resources from loading using the &block_resources=image parameter.

ZenRows API allows to block the following resources: stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other. Separate by commas to block multiple resources.

ZenRows will block certain resources by default, such as stylesheets or images, to speed up your scraping. You can disable blocking by setting it to “none”: block_resources=none.

Requires javascript rendering (&js_render=true).

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"
params = {
    "js_render": "true",
    "block_resources": "image,media,font"
}

response = client.get(url, params=params)

print(response.text)

​

JSON Response

To intercept the response of XHR / Fetch / Ajax requests, add &json_response=true to the API call. The response will be a JSON with three fields and an optional fourth: html, xhr, js_instructions_report and screenshot.

• HTML will contain the content of the page. You’ll have to decode it since it will be encoded in JSON.
• XHR will be an array with one object per performed request. Those will contain URL, body, status code and many more. See the example below.
• js_instructions_report will be an object containing a report for the executed JS Instructions.
• screenshot will be an object with 4 fields (data, type, width and height) containing the image taken on the target site.

Requires javascript rendering (&js_render=true).

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"
params = {
    "js_render": "true",
    "json_response": "true"
}

response = client.get(url, params=params)

print(response.text)

And the response will look like this:

{
	"html": "<!DOCTYPE html><html>...</html>",
	"xhr": [{
		"url": "https://www.example.com/fetch",
		"body": "{\"success\": true}\n",
		"status_code": 200,
		"method": "GET",
		"headers": {
			"content-encoding": "gzip",
			// ...
		},
		"request_headers": {
			"accept": "*/*",
			// ...
		}
	}]
}

If the request includes JS Instructions, the output would have an extra field like this one:

{
	"html": "...",
	"xhr": [],
	"js_instructions_report": {
		"instructions_duration": 1041,
		"instructions_executed": 2,
		"instructions_succeeded": 2,
		"instructions_failed": 0,
		"instructions": [{
			"instruction": "wait_for_selector",
			"params": {
				"selector": "div"
			},
			"success": true,
			"duration": 40
		}, {
			"instruction": "wait",
			"params": {
				"timeout": 1000
			},
			"success": true,
			"duration": 1001
		}]
	}
}

If the request includes Screenshot, the output would have an extra field like this one:

{
	"html": "...",
	"xhr": [],
	"screenshot": {
		"data": "... image data in base64 ...",
		"type": "image/png",
		"width": 1920,
		"height": 1080
	},
	"js_instructions_report": null
}

​

Window Width/Height

If you need to change the browser’s window width and height, you can use the &window_width=1920 and &window_height=1080 parameters. Since ZenRows will always try to use common browser and header settings, the service will apply the values in a best-effort approach.

Requires javascript rendering (&js_render=true).

# pip install zenrows
from zenrows import ZenRowsClient

client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
url = "https://httpbin.io/anything"
params = {
    "js_render": "true",
    "window_width": 1920,
    "window_height": 1080
}

response = client.get(url, params=params)

print(response.text)