> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.
# How to Integrate Flowise with ZenRows
> Integrate ZenRows with Flowise to build AI-powered web scraping workflows using a visual drag-and-drop builder for data collection pipelines.
Integrate ZenRows with Flowise to add reliable web scraping capabilities to your AI workflows. This guide shows you how to build data collection pipelines that combine ZenRows' scraping technology with Flowise's visual workflow builder.
## What is Flowise?
Flowise is an open-source, low-code platform for visually building, managing, and interacting with AI agents and large language model (LLM) workflows in a chat-based environment. Its drag-and-drop interface allows users to compose AI agents into interconnected, LLM-powered workflows **with little to no programming knowledge**.
## Use Cases
You can leverage Flowise-ZenRows integration to drive several business opportunities. Here are some of them:
* **Product research**: Discover the best marketing position for new products with insights from scraped data.
* **Demand forecasting**: Scrape products' data to predict the demand ahead of competitors. Make data-driven decisions on re-stocks and pricing.
* **Real-estate intelligence**: Collect real-estate data and analyze it to gain data-driven investment insights.
* **Competitor's research**: Gather competitor's data for information gain and predict their next moves.
* **Product optimization**: Collect customers' reviews, ratings, and other metrics from social platforms, review sites, and brand pages and fine-tune products to meet consumers' expectations.
* **Lead generation**: Collect quality lead contacts from various sources and automate captures, outreaches, and follow-ups.
* **Product recommendation**: Scrape and analyze relevant product data points to recommend the best deals to potential buyers.
## Basic Integration: Amazon Product Scraper
Let's start by building a simple Amazon product scraper workflow that fetches data using ZenRows and returns it as a response via the Flowise chat box. We'll use the Flowise Agentflows for this tutorial.
The Flowise platform is accessible locally or remotely via the official Flowise website. If you use Flowise remotely via the website, no installation steps are required. However, the local setup requires installing Flowise globally to run a local server.
In this guide, **we'll use the local setup**, starting with the installation and server startup steps.
### Step 1: Install Flowise and set up a server
This tutorial assumes you've set up Node.js on your machine. Otherwise, download and install it from the official website.
1. Install the `flowise` package globally:
```bash theme={null}
npm install -g flowise
```
2. Run the `flowise` server. The command below starts a server that defaults to `http://localhost:3000`:
```bash theme={null}
npx flowise start
```
3. Visit this URL to launch the Flowise interface.
4. Complete the initial registration steps, and you'll get to the platform's dashboard, as shown below:
### Step 2: Add ZenRows API Key to Flowise environment variables
Adding your ZenRows API key to the Flowise environment variable makes it accessible across the platforms.
1. From the dashboard, go to **Variables**.
2. Click `+ Add Variable` at the top-right.
3. Fill in the **Variable Name** field with the name of your API key. Then, paste your ZenRows API key in the **Value** field.
4. Click `Add`.
### Step 3: Initiate a new Agentflow
1. Go to **Agentflows** and click `+ Add New` at the top right to open the Flowise canvas.
2. Once in the Flowise Canvas, click the save icon at the top right.
3. Enter a name for your new workflow and click `Save`.
### Step 4: Create a scraper flow with ZenRows
We'll integrate the ZenRows API using an HTTP agent node, which enables you to send an API request via Flowise.
1. Click the `+` icon at the top left and search for **"HTTP"** using the search bar.
2. Drag and drop the **HTTP** node into your Canvas, placing it directly in front of the **Start** agent.
3. Link the agents by dragging a line from the **Start** node to connect with the **HTTP** agent node.
4. Double-click the **HTTP** agent node.
5. Click the edit icon at the top of the modal to change the agent's name. Then, press **Enter** on your keyboard.
6. Click `+ Add Query Params` to set the target URL parameter. We'll retrieve this URL dynamically from the chat box:
* Type `url` in the **Key** field.
* Type double braces (`{{`) in the **Value** field. This loads dynamic input options.
* Select **question** from the options to load the URL from the chat box.
7. Similarly, for the ZenRows API key, click `+ Add Query Params` and type `apikey` in the **Key** field. Type `{{` inside the **Value** and select `$vars.ZENROWS_API_KEY`.
8. Click `+ Add Query Params` repeatedly to add each of the following [ZenRows parameters](/universal-scraper-api/api-reference#parameter-overview) in order:
* `js_render` = `true`
* `premium_proxy` = `true`
* `wait` = `2500`
* `css_extractor` = {}
* `original_status` = `true`
9. Add your [CSS selectors](/universal-scraper-api/troubleshooting/advanced-css-selectors) as an array in the `css_extractor` value: We'll use the following `css_extractor` for this tutorial:
```json theme={null}
{
"name":"span#productTitle",
"price":"span.a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay",
"ratings":"#acrPopover",
"description":"ul.a-unordered-list.a-vertical.a-spacing-mini",
"review summary":"#product-summary p span",
"review count":"#acrCustomerReviewLink"
}
```
10. Under **Response Type**, select **Text**.
11. Click on the `Flowise Canvas`. Then, click the `Validate Nodes` icon at the top right and select `Validate Nodes` to confirm that your setup is correct.
12. Finally, click the `save` icon at the top right to save your changes.
### Step 5: Run the flow
1. Click the chat icon at the top right.
2. Type an Amazon product page URL in the box and press Enter to send your request.
3. Click the `Process Flow banner` in the chat box.
4. Click the expansion icon next to Scraper Agent.
5. You'll see the following response page, showing the scraped data:
However, we want the LLM to return the data as the sole response in the chat. Let's set this up using OpenAI.
### Step 6: Integrate an LLM flow
1. Click the `+` icon at the top left. Search **LLM**. Then, drag and drop it in the canvas next to the Scraper Agent.
2. Link the **Scraper Agent** to the **LLM** node.
3. Double-click the **LLM** agent and rename it.
4. Under **Model**, search and select `ChatOpenAI`.
5. Click `ChatOpenAI` Parameters:
* Click on `Connect Credentials`, then on `Create New` and set up your OpenAI API key and click `Add`.
* Select your desired model, temperature (preferably, 0.1 or 0.2 for increased accuracy), and other parameters.
6. Click `+ Add Messages`:
* Under **Role**, select `Assistant`.
* Type `{{` inside the Content field and select `httpAgentflow_0`.
7. Toggle on `Enable Memory`.
8. Type the following prompt in the **Input Message** field. We've instructed the LLM to only show `{{ httpAgentflow_0 }}` as the output and not attempt to visit any URL provided in the chat:
```text theme={null}
You are a web scraping assistant. Only display the response provided by the Scraper Agent in the chat box. Do not attempt to access or retrieve information from external URLs yourself; rely solely on the Scraper Agent for all web data. Your output must be {{ httpAgentflow_0 }}.
```
9. Set the **Return Response As** field to **Assistant Message**. This allows the chat model to return the Scraper Agent's output in its response.
10. Click anywhere in the canvas. Then click the save icon at the top right.
11. To run the flow, click the chat icon and send the URL of an Amazon product page in the chat. The model now streams the scraped data as a response in the chat box:
Nice! You just created your first scraping flow by integrating ZenRows with Flowise.
## Advanced Integration: Demand Analysis System
Besides the HTTP option, you can also integrate ZenRows with Flowise using custom functions. We'll see a real-life use case using the following agents:
* Create an agent that scrapes an Amazon listing page based on a search term using ZenRows.
* Build an agent node to handle data cleaning.
* Create an agent to analyze the data using LLM.
### Step 1: Create a custom web scraper function with ZenRows
1. Go to **Agentflows** and click `+ Add New` to start a new flow.
2. Click the save icon and name your flow.
3. Click `+` at the top left of the Flowise canvas.
4. Search for `Custom Function` and drag its card to the canvas.
5. Link the **Start** node with the **Custom Function** agent.
6. Double-click the Custom Function agent and rename it.
7. Initiate the ZenRows scraping logic inside the **JavaScript Function** field. Import `axios`, configure the target URL to accept a search term, and load your ZenRows API key from the Flowise environment variables.
```javascript Node.js theme={null}
const axios = require("axios");
// declare the search term variable
const searchTerm = $searchTerm;
// load your API key from the environment variables
const apikey = $vars.ZENROWS_API_KEY;
const url = `https://www.amazon.com/s?k=${encodeURIComponent(searchTerm)}`;
```
8. To create the `searchTerm` variable, click `+ Add Input Variables`.
9. Type `searchTerm` inside the **Variable Name** field.
10. We want the scraping logic to accept this search term dynamically from the user's question in the chat box. Type `{{` in the **Variable Value** and select `question`.
11. Extend the scraping logic by adding the required ZenRows parameters. Scrape the data using the `css_extractor` and return the response data. See the complete code below:
```javascript Node.js theme={null}
const axios = require("axios");
// declare the search term variable
const searchTerm = $searchTerm;
// load your API key from the environment variables
const apikey = $vars.ZENROWS_API_KEY;
const url = `https://www.amazon.com/s?k=${encodeURIComponent(searchTerm)}`;
try {
const response = await axios({
url: "https://api.zenrows.com/v1/",
method: "GET",
params: {
url: url,
apikey: apikey,
js_render: "true",
wait: "2500",
premium_proxy: "true",
css_extractor: JSON.stringify({
name: "div.s-title-instructions-style",
url: "div.s-title-instructions-style a @href",
listingPrice: "span.a-price",
demandHistory:
"div.a-section.a-section.a-spacing-none.a-spacing-top-micro span.a-size-base.a-color-secondary",
reviewCount: "span.a-size-base.s-underline-text",
}),
},
});
return response.data;
} catch (error) {
console.error("Error fetching data:", error.message);
throw error;
}
```
12. Click the save icon at the top right to save your changes.
### Step 2: Create a data cleaning agent
1. Click the `+` icon.
2. Search for **Custom Function** and drag and drop it onto the canvas.
3. Link this new custom function agent with the **Scraper** agent.
4. Double-click the `Custom Function 1` agent node and rename it (e.g., Data Cleaner).
5. We need to load the Scraper agent's output into the Data Cleaner agent node. Create this variable by clicking `+ Add Input Variables`. Then, type **scrapedData** in the **Variable Name** field. Then, enter `{{` in the **Variable Value** field and select `customFunctionAgentflow_0`.
6. Enter the following data cleaning code in the JavaScript Function field. This code first converts the scraped data into an object, merges it, and fixes anomalies, such as data inconsistencies.
```javascript Node.js theme={null}
// convert the scraped data to an object
let data;
try {
data = JSON.parse($scrapedData);
} catch (e) {
return [];
}
// ensure all arrays exist
const demand = Array.isArray(data.demandHistory) ? data.demandHistory : [];
const name = Array.isArray(data.name) ? data.name : [];
const listingPrice = Array.isArray(data.listingPrice) ? data.listingPrice : [];
const reviewCount = Array.isArray(data.reviewCount) ? data.reviewCount : [];
const url = Array.isArray(data.url) ? data.url : [];
const baseUrl = "https://www.amazon.com";
// check if the demand entry is a real monthly demand
const isValidDemand = (d) =>
typeof d === "string" && d.includes("bought in past month");
// merge and clean the data
const products = [];
let demandIdx = 0;
for (let i = 0; i < name.length; i++) {
// filter out sponsored/ad names
if (
typeof name[i] !== "string" ||
name[i].toLowerCase().startsWith("sponsored") ||
name[i].toLowerCase().includes("ad feedback")
) {
continue;
}
// find the next valid demand entry
let demandValue = "Unavailable";
while (demandIdx < demand.length) {
if (isValidDemand(demand[demandIdx])) {
demandValue = demand[demandIdx];
demandIdx++;
break;
}
demandIdx++;
}
// clean doubled price issues
let price = listingPrice[i] || "Unavailable";
if (typeof price === "string") {
const match = price.match(/\$\d+(\.\d{2})?/);
if (match) {
price = match[0];
}
}
// ensure URL is absolute
let productUrl = url[i] || "Unavailable";
if (
typeof productUrl === "string" &&
productUrl !== "Unavailable" &&
!productUrl.startsWith("http")
) {
productUrl = baseUrl + productUrl;
}
products.push({
name: name[i] || "Unavailable",
demand: demandValue,
price: price,
reviews: reviewCount[i] || "Unavailable",
url: productUrl,
});
}
return products;
```
7. Click anywhere on the canvas and click the save icon.
### Step 3: Create a Data Analyst Agent LLM
1. Click the `+` icon.
2. Search for **LLM** and drag and drop it into the canvas.
3. Link the LLM agent with the Data Cleaner agent. The LLM will act on the cleaned data returned from the Data Cleaner agent.
4. Double-click the LLM agent and give it a descriptive name.
5. From the **Model** dropdown, search and select `ChatOpenAI` or your preferred model.
6. Set up your credentials, select your model, and choose a temperature.
7. Click `+ Add Messages` under the **Messages** section.
8. Create a prompt that instructs the Analyst agent to get the most in-demand products from the cleaned data and establish a correlation between price and demand. We'll use the following prompt, where `customFunctionAgentflow_1` is the cleaned data from the Data Cleaner agent node.
```text theme={null}
Analyze the products in {{ customFunctionAgentflow_1 }}. Select the 4 most in-demand products based on the demand history (highest demand first). Also, tell if there is a correlation between price and demand across these products. For each product, format the product's URL as an HTML anchor tag with the text: Check product out.
```
9. Under Role, select `Assistant`. Then, enter the prompt into the **Content** field.
10. Under **Return Response As**, select `Assistant Message`.
11. Finally, validate and save your workflow by clicking the validation and save icons at the top right, respectively.
### Step 4: Run the demand analysis workflow
Click the message icon to initiate a chat. Then, prompt the Analyst agent with a product search term (e.g., headphones, mouse, or teddy bears).
Here's a sample response for "mouse" product listings:
Congratulations! 🎉 You just built a product research workflow with ZenRows and Flowise integration.
## Troubleshooting
### Error 401 or ZenRows authentication error
* **Solution 1**: Ensure you've entered the correct ZenRows API key in the Flowise environment variables.
* **Solution 2**: Ensure you load the ZenRows API key into your scraping workflow.
### Error 400
* **Solution 1**: Make sure the supplied ZenRows parameters are correct and don't contain unsupported strings.
* **Solution 2**: Use the correct CSS selectors and format the `css_extractor` array correctly.
### LLM agent fails
* **Solution 1**: Check your network connectivity and try your prompt again.
* **Solution 2**: Ensure you still have enough API credit for your chosen LLM.
* **Solution 3**: Make sure you've supplied the correct API key of your LLM agent.
### Custom function fails
* **Solution 1**: Execute the code in an external text editor, such as VS Code, and simulate the expected output for debugging.
* **Solution 2**: Make sure all variables in your code are listed as **Input Variables**.
### `localhost unavailable` or the Flowise interface doesn't load
* **Solution**: Stop and restart the running server. Then, reload the Flowise interface.
## Conclusion
You've learned to connect ZenRows with Flowise for a streamlined data flow across several agents, including data collection, cleaning, and analysis. While Flowise enables you to build agentic workflows for data automation and other purposes, ZenRows ensures the integrity of your data pipeline with a consistent data supply.
## Frequently Asked Questions (FAQ)
ZenRows doesn't charge you extra for integrating with Flowise. Flowise is also an open-source tool with no initial cost implications. Although Flowise charges for advanced features like cloud hosting, the free plan provides most of what you need, including building Agentflows, Chatflows, Assistants, and more.
Yes, ZenRows integrates with other AI-powered low-code and no-code workflow tools, including Pipedream, Make, Clay, Zapier, and more. Check our [integration options](/integrations/overview) for more details.
Although Flowise supports Google Sheets as a tool, connecting to it involves some technical steps. Currently, Flowise has limitations in terms of Google Sheets write operations compared to tools like Zapier, which offer easier and more seamless integration.
Yes, integrating ZenRows with Flowise enables your scraper agent to bypass anti-bots easily.