Scroll and paginate to collect every item

Collect a full listing with Zenrows Browser, the browser for agents: scroll an infinite feed until it stops loading, or follow “Next” links to the last page, then save every item to a CSV file. Uses: Browser with Playwright’s mouse.wheel, wait_for_function, evaluate_all and route

Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

Use the Zenrows browser to open https://www.scrapingcourse.com/infinite-scrolling.
Keep scrolling to the bottom until no new products load, then give me every product
name and price as a CSV table. Close the browser session when you finish.

Scroll an infinite feed to the end

Scroll, then wait for the page to add items. When no new items arrive within five seconds, you reached the end. Blocking images keeps the session light, since the data is in the text.

import asyncio, csv
from playwright.async_api import async_playwright, TimeoutError as PlaywrightTimeoutError

CONNECTION_URL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(CONNECTION_URL)
        try:
            page = await browser.contexts[0].new_page()
            # Skip images to save bandwidth: the data is in the text
            await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", lambda route: route.abort())
            await page.goto("https://www.scrapingcourse.com/infinite-scrolling", wait_until="domcontentloaded")

            while True:
                count = await page.locator(".product-item").count()
                await page.mouse.wheel(0, 20000)
                try:
                    # Wait until the page appends more items
                    await page.wait_for_function(
                        "n => document.querySelectorAll('.product-item').length > n",
                        arg=count,
                        timeout=5000,
                    )
                except PlaywrightTimeoutError:
                    break  # nothing new loaded: you reached the end

            products = await page.locator(".product-item").evaluate_all(
                """items => items.map(item => ({
                    name: item.querySelector('.product-name').innerText.trim(),
                    price: item.querySelector('.product-price').innerText.trim(),
                    url: item.querySelector('a').href,
                }))"""
            )
        finally:
            await browser.close()

    # The page can repeat items while loading: keep one row per URL
    products = list({p["url"]: p for p in products}.values())

    with open("infinite-scroll.csv", "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=["name", "price", "url"])
        writer.writeheader()
        writer.writerows(products)
    print(f"Saved {len(products)} products to infinite-scroll.csv")

asyncio.run(main())

import { chromium } from "playwright-core";
import { writeFileSync } from "node:fs";

const connectionURL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY";

const browser = await chromium.connectOverCDP(connectionURL);
let products;
try {
  const page = await browser.contexts()[0].newPage();
  // Skip images to save bandwidth: the data is in the text
  await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", (route) => route.abort());
  await page.goto("https://www.scrapingcourse.com/infinite-scrolling", { waitUntil: "domcontentloaded" });

  while (true) {
    const count = await page.locator(".product-item").count();
    await page.mouse.wheel(0, 20000);
    try {
      // Wait until the page appends more items
      await page.waitForFunction(
        (n) => document.querySelectorAll(".product-item").length > n,
        count,
        { timeout: 5000 },
      );
    } catch {
      break; // nothing new loaded: you reached the end
    }
  }

  products = await page.locator(".product-item").evaluateAll((items) =>
    items.map((item) => ({
      name: item.querySelector(".product-name").innerText.trim(),
      price: item.querySelector(".product-price").innerText.trim(),
      url: item.querySelector("a").href,
    })),
  );
} finally {
  await browser.close();
}

// The page can repeat items while loading: keep one row per URL
const unique = [...new Map(products.map((p) => [p.url, p])).values()];
const csv = ["name,price,url", ...unique.map((p) => `"${p.name}","${p.price}","${p.url}"`)];
writeFileSync("infinite-scroll.csv", csv.join("\n"));
console.log(`Saved ${unique.length} products to infinite-scroll.csv`);

Follow “Next” links to the last page

Read the items on the page, click the “Next” link, and repeat until the link is gone.

import asyncio, csv
from playwright.async_api import async_playwright

CONNECTION_URL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"

async def main():
    products = []
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(CONNECTION_URL)
        try:
            page = await browser.contexts[0].new_page()
            # Skip images to save bandwidth: the data is in the text
            await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", lambda route: route.abort())
            await page.goto("https://www.scrapingcourse.com/ecommerce/")

            page_number = 1
            while True:
                rows = await page.locator("li.product").evaluate_all(
                    """items => items.map(item => ({
                        name: item.querySelector('.product-name').innerText.trim(),
                        price: item.querySelector('.price').innerText.trim(),
                        url: item.querySelector('a').href,
                    }))"""
                )
                products.extend(rows)
                print(f"Page {page_number}: {len(rows)} products")

                next_link = page.locator("a.next.page-numbers").first
                if await next_link.count() == 0:
                    break  # last page reached
                await next_link.click()
                await page.wait_for_load_state()
                page_number += 1
        finally:
            await browser.close()

    with open("products.csv", "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=["name", "price", "url"])
        writer.writeheader()
        writer.writerows(products)
    print(f"Saved {len(products)} products to products.csv")

asyncio.run(main())

import { chromium } from "playwright-core";
import { writeFileSync } from "node:fs";

const connectionURL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY";

const browser = await chromium.connectOverCDP(connectionURL);
const products = [];
try {
  const page = await browser.contexts()[0].newPage();
  // Skip images to save bandwidth: the data is in the text
  await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", (route) => route.abort());
  await page.goto("https://www.scrapingcourse.com/ecommerce/");

  for (let pageNumber = 1; ; pageNumber++) {
    const rows = await page.locator("li.product").evaluateAll((items) =>
      items.map((item) => ({
        name: item.querySelector(".product-name").innerText.trim(),
        price: item.querySelector(".price").innerText.trim(),
        url: item.querySelector("a").href,
      })),
    );
    products.push(...rows);
    console.log(`Page ${pageNumber}: ${rows.length} products`);

    const nextLink = page.locator("a.next.page-numbers").first();
    if ((await nextLink.count()) === 0) break; // last page reached
    await nextLink.click();
    await page.waitForLoadState();
  }
} finally {
  await browser.close();
}

const csv = ["name,price,url", ...products.map((p) => `"${p.name}","${p.price}","${p.url}"`)];
writeFileSync("products.csv", csv.join("\n"));
console.log(`Saved ${products.length} products to products.csv`);

Example output

Infinite scroll:

Saved 147 products to infinite-scroll.csv

name,price,url
Chaz Kangeroo Hoodie,$52,https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie
Teton Pullover Hoodie,$70,https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie
Bruno Compete Hoodie,$63,https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie

Pagination, trimmed:

Page 1: 16 products
Page 2: 16 products
...
Page 11: 16 products
Page 12: 12 products
Saved 188 products to products.csv

name,price,url
Abominable Hoodie,$69.00,https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/
Adrienne Trek Jacket,$57.00,https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/

Customize it

Change	How
Feeds with a “Load more” button	Replace `mouse.wheel` with `await page.click("#load-more-btn")` and keep the same wait-for-more-items check
Stop after a fixed number of items	Break out of the loop when `count` reaches your target
Paginate by URL	`await page.goto(next_href)` with the link’s `href`, the reliable choice when the “Next” control is a plain link
Listings in another country	Add `&proxy_country=de` to the connection URL, see Country targeting
Long listings	Add `&session_ttl=10m` so the session outlasts the scroll, see Session TTL
Many listings at once	One session per listing, see Run browser sessions in parallel
Skip the browser for static pages	Fetch each page URL with Fetch or crawl them, see Crawl a site from a seed URL

Troubleshooting

The loop stops early: the next set of items took longer than the wait. Raise timeout in wait_for_function to 10000.
The CSV has duplicate rows: the feed re-sends items while loading. Keep one row per URL, as the scroll example does.
page.goto times out waiting for load: a script or asset on the page keeps loading. Pass wait_until="domcontentloaded".
The click on “Next” waits until it times out: the site navigates in a way Playwright does not track as a page load. Read the link’s href and call page.goto(href) instead of clicking.

Run browser sessions in parallel

Scrape many listings at once with a concurrency limit.

Fill and submit multi-step forms

Search, filter and submit before you collect the results.

Turn a whole catalog into a structured dataset

Turn a whole catalog into a clean dataset.

Click, fill, and scroll before you scrape

Scroll and click with js_instructions in one request.

Start Here

Foundations Course

Fetch Recipes

Extract Recipes

Crawl Recipes

Batch Recipes

Browser Recipes

Pipelines

Use Cases

By Surface

Production

Scroll and paginate to collect every item

Ask your AI agent

Scroll an infinite feed to the end

Follow “Next” links to the last page

Example output

Customize it

Troubleshooting

Run browser sessions in parallel

Fill and submit multi-step forms

Turn a whole catalog into a structured dataset

Click, fill, and scroll before you scrape

​Ask your AI agent

​Scroll an infinite feed to the end

​Follow “Next” links to the last page

​Example output

​Customize it

​Troubleshooting

​Related recipes

Run browser sessions in parallel

Fill and submit multi-step forms

Turn a whole catalog into a structured dataset

Click, fill, and scroll before you scrape

Ask your AI agent

Scroll an infinite feed to the end

Follow “Next” links to the last page

Example output

Customize it

Troubleshooting

Related recipes