> ## Documentation Index > Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types. # Scroll and paginate to collect every item > Collect every item from infinite-scroll feeds and multi-page listings with Zenrows Browser, and save them to CSV. Collect a full listing with Zenrows Browser, the browser for agents: scroll an infinite feed until it stops loading, or follow "Next" links to the last page, then save every item to a CSV file. **Uses:** [Browser](/browser-sessions/introduction) with Playwright's `mouse.wheel`, `wait_for_function`, `evaluate_all` and `route` ## Ask your AI agent With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own. ```text theme={"dark"} Use the Zenrows browser to open https://www.scrapingcourse.com/infinite-scrolling. Keep scrolling to the bottom until no new products load, then give me every product name and price as a CSV table. Close the browser session when you finish. ``` ## Scroll an infinite feed to the end Scroll, then wait for the page to add items. When no new items arrive within five seconds, you reached the end. Blocking images keeps the session light, since the data is in the text. ```python Python (Playwright) theme={"dark"} import asyncio, csv from playwright.async_api import async_playwright, TimeoutError as PlaywrightTimeoutError CONNECTION_URL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY" async def main(): async with async_playwright() as p: browser = await p.chromium.connect_over_cdp(CONNECTION_URL) try: page = await browser.contexts[0].new_page() # Skip images to save bandwidth: the data is in the text await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", lambda route: route.abort()) await page.goto("https://www.scrapingcourse.com/infinite-scrolling", wait_until="domcontentloaded") while True: count = await page.locator(".product-item").count() await page.mouse.wheel(0, 20000) try: # Wait until the page appends more items await page.wait_for_function( "n => document.querySelectorAll('.product-item').length > n", arg=count, timeout=5000, ) except PlaywrightTimeoutError: break # nothing new loaded: you reached the end products = await page.locator(".product-item").evaluate_all( """items => items.map(item => ({ name: item.querySelector('.product-name').innerText.trim(), price: item.querySelector('.product-price').innerText.trim(), url: item.querySelector('a').href, }))""" ) finally: await browser.close() # The page can repeat items while loading: keep one row per URL products = list({p["url"]: p for p in products}.values()) with open("infinite-scroll.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=["name", "price", "url"]) writer.writeheader() writer.writerows(products) print(f"Saved {len(products)} products to infinite-scroll.csv") asyncio.run(main()) ``` ```javascript Node.js (Playwright) theme={"dark"} import { chromium } from "playwright-core"; import { writeFileSync } from "node:fs"; const connectionURL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"; const browser = await chromium.connectOverCDP(connectionURL); let products; try { const page = await browser.contexts()[0].newPage(); // Skip images to save bandwidth: the data is in the text await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", (route) => route.abort()); await page.goto("https://www.scrapingcourse.com/infinite-scrolling", { waitUntil: "domcontentloaded" }); while (true) { const count = await page.locator(".product-item").count(); await page.mouse.wheel(0, 20000); try { // Wait until the page appends more items await page.waitForFunction( (n) => document.querySelectorAll(".product-item").length > n, count, { timeout: 5000 }, ); } catch { break; // nothing new loaded: you reached the end } } products = await page.locator(".product-item").evaluateAll((items) => items.map((item) => ({ name: item.querySelector(".product-name").innerText.trim(), price: item.querySelector(".product-price").innerText.trim(), url: item.querySelector("a").href, })), ); } finally { await browser.close(); } // The page can repeat items while loading: keep one row per URL const unique = [...new Map(products.map((p) => [p.url, p])).values()]; const csv = ["name,price,url", ...unique.map((p) => `"${p.name}","${p.price}","${p.url}"`)]; writeFileSync("infinite-scroll.csv", csv.join("\n")); console.log(`Saved ${unique.length} products to infinite-scroll.csv`); ``` ## Follow "Next" links to the last page Read the items on the page, click the "Next" link, and repeat until the link is gone. ```python Python (Playwright) theme={"dark"} import asyncio, csv from playwright.async_api import async_playwright CONNECTION_URL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY" async def main(): products = [] async with async_playwright() as p: browser = await p.chromium.connect_over_cdp(CONNECTION_URL) try: page = await browser.contexts[0].new_page() # Skip images to save bandwidth: the data is in the text await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", lambda route: route.abort()) await page.goto("https://www.scrapingcourse.com/ecommerce/") page_number = 1 while True: rows = await page.locator("li.product").evaluate_all( """items => items.map(item => ({ name: item.querySelector('.product-name').innerText.trim(), price: item.querySelector('.price').innerText.trim(), url: item.querySelector('a').href, }))""" ) products.extend(rows) print(f"Page {page_number}: {len(rows)} products") next_link = page.locator("a.next.page-numbers").first if await next_link.count() == 0: break # last page reached await next_link.click() await page.wait_for_load_state() page_number += 1 finally: await browser.close() with open("products.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=["name", "price", "url"]) writer.writeheader() writer.writerows(products) print(f"Saved {len(products)} products to products.csv") asyncio.run(main()) ``` ```javascript Node.js (Playwright) theme={"dark"} import { chromium } from "playwright-core"; import { writeFileSync } from "node:fs"; const connectionURL = "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"; const browser = await chromium.connectOverCDP(connectionURL); const products = []; try { const page = await browser.contexts()[0].newPage(); // Skip images to save bandwidth: the data is in the text await page.route("**/*.{png,jpg,jpeg,webp,gif,svg}", (route) => route.abort()); await page.goto("https://www.scrapingcourse.com/ecommerce/"); for (let pageNumber = 1; ; pageNumber++) { const rows = await page.locator("li.product").evaluateAll((items) => items.map((item) => ({ name: item.querySelector(".product-name").innerText.trim(), price: item.querySelector(".price").innerText.trim(), url: item.querySelector("a").href, })), ); products.push(...rows); console.log(`Page ${pageNumber}: ${rows.length} products`); const nextLink = page.locator("a.next.page-numbers").first(); if ((await nextLink.count()) === 0) break; // last page reached await nextLink.click(); await page.waitForLoadState(); } } finally { await browser.close(); } const csv = ["name,price,url", ...products.map((p) => `"${p.name}","${p.price}","${p.url}"`)]; writeFileSync("products.csv", csv.join("\n")); console.log(`Saved ${products.length} products to products.csv`); ``` ## Example output Infinite scroll: ```text theme={"dark"} Saved 147 products to infinite-scroll.csv ``` ```text theme={"dark"} name,price,url Chaz Kangeroo Hoodie,$52,https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie Teton Pullover Hoodie,$70,https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie Bruno Compete Hoodie,$63,https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie ``` Pagination, trimmed: ```text theme={"dark"} Page 1: 16 products Page 2: 16 products ... Page 11: 16 products Page 12: 12 products Saved 188 products to products.csv ``` ```text theme={"dark"} name,price,url Abominable Hoodie,$69.00,https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/ Adrienne Trek Jacket,$57.00,https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/ ``` ## Customize it | Change | How | | - | - | | Feeds with a "Load more" button | Replace `mouse.wheel` with `await page.click("#load-more-btn")` and keep the same wait-for-more-items check | | Stop after a fixed number of items | Break out of the loop when `count` reaches your target | | Paginate by URL | `await page.goto(next_href)` with the link's `href`, the reliable choice when the "Next" control is a plain link | | Listings in another country | Add `&proxy_country=de` to the connection URL, see [Country targeting](/browser-sessions/features/country) | | Long listings | Add `&session_ttl=10m` so the session outlasts the scroll, see [Session TTL](/browser-sessions/features/session-ttl) | | Many listings at once | One session per listing, see [Run browser sessions in parallel](/academy/browser/parallel-sessions) | | Skip the browser for static pages | Fetch each page URL with [Fetch](/academy/fetch/javascript-pages) or crawl them, see [Crawl a site from a seed URL](/academy/scrape-and-crawl-from-a-seed-url) | ## Troubleshooting * **The loop stops early**: the next set of items took longer than the wait. Raise `timeout` in `wait_for_function` to 10000. * **The CSV has duplicate rows**: the feed re-sends items while loading. Keep one row per URL, as the scroll example does. * **`page.goto` times out waiting for `load`**: a script or asset on the page keeps loading. Pass `wait_until="domcontentloaded"`. * **The click on "Next" waits until it times out**: the site navigates in a way Playwright does not track as a page load. Read the link's `href` and call `page.goto(href)` instead of clicking. ## Related recipes Scrape many listings at once with a concurrency limit. Search, filter and submit before you collect the results. Turn a whole catalog into a clean dataset. Scroll and click with js\_instructions in one request. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.