Skip to main content
In Lesson 2 you read the products on one catalog page. The store has 188 products, and the tracker needs the URL of each one. Instead of paging through the catalog yourself, you hand the start URL to Crawl and let it follow the links. What you’ll build: discover.py, a script that crawls the store from its catalog page and writes every product URL it finds to product_urls.txt. Lesson 4 scrapes that list. What you’ll learn:
  • How a crawl starts from one URL and follows links in the background.
  • How include_patterns steers the crawl to the pages you want.
  • How max_pages sets the size, and the cost, of a crawl.
  • How to wait for a crawl and read its results with ZenRowsCrawlClient.
Time: about 10 minutes.

​
How the crawl finds products

Crawl opens the start URL, collects the links on it, and keeps opening the links that match your patterns. Each page it opens is one Fetch request with Adaptive Stealth Mode, so protected sites work the same way they did in Lesson 1. For the tracker you give it two settings:
  • include_patterns=["/product/"] keeps product URLs and opens them. Every product page links to related products, so each page opened leads to more products.
  • max_pages=20 stops the crawl after it fetches 20 pages. That number is your cost ceiling: on this store, each page costs 1 credit.

​
Write the discovery script

The zenrows package you installed in Lesson 2 includes ZenRowsCrawlClient. Create discover.py:
Python (SDK)
Here is what each call does:
1

create starts the crawl

client.create() returns at once with the crawl’s ID and the status running. The crawl runs on Zenrows’ side, so your script is free while it works. max_items=200 lets the crawl keep up to 200 URLs, room for the whole catalog.
2

wait polls until it ends

client.wait() checks the crawl until its status is completed, stopped or failed, and returns it with its coverage counters: pages fetched and URLs kept.
3

results reads every URL

client.results() pages through the kept URLs for you. Each result has a url, already normalized and deduplicated.

​
Run it

The crawl takes a few seconds on this store:
Twenty pages turned up 42 product URLs: the 16 on the catalog page plus the related products linked from each product page opened. Open product_urls.txt to see the full list.

​
Make it your own

The crawl stopped because it reached max_pages. To cover more of the store, raise the ceiling, for example max_pages=200 for the full 188-product catalog. Each extra page is one more Fetch request, so max_pages is the dial between coverage and cost. For the rest of the course, the 42 URLs you have are plenty: every later lesson works the same on 42 URLs or 4,200.
Do the same from the CLI. The Zenrows CLI starts the crawl, waits with --follow, and saves the URLs as JSON lines:

​
Ask your AI agent

With the Zenrows MCP server connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

​
What you built

discover.py turns one start URL into a list of product URLs in product_urls.txt. You can:
  • Start a crawl with ZenRowsCrawlClient.create() and wait for it with wait().
  • Steer it with include_patterns and size it with max_pages.
  • Read every kept URL with results().

​
Next lesson

Lesson 4: Scrape every product with Batch

Send all the product URLs as one Batch job and turn the results into JSON and CSV files.