> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Lesson 3: Discover every product with Crawl

> Foundations lesson 3: find every product page in an online store with Zenrows Crawl and save the URLs.

In [Lesson 2](/academy/foundations/extract-product-data) you read the products on one catalog page. The store has 188 products, and the tracker needs the URL of each one. Instead of paging through the catalog yourself, you hand the start URL to Crawl and let it follow the links.

**What you'll build:** `discover.py`, a script that crawls the store from its catalog page and writes every product URL it finds to `product_urls.txt`. Lesson 4 scrapes that list.

**What you'll learn:**

* How a crawl starts from one URL and follows links in the background.
* How `include_patterns` steers the crawl to the pages you want.
* How `max_pages` sets the size, and the cost, of a crawl.
* How to wait for a crawl and read its results with `ZenRowsCrawlClient`.

**Time:** about 10 minutes.

## How the crawl finds products

[Crawl](/crawl/endpoints) opens the start URL, collects the links on it, and keeps opening the links that match your patterns. Each page it opens is one Fetch request with Adaptive Stealth Mode, so protected sites work the same way they did in Lesson 1.

For the tracker you give it two settings:

* `include_patterns=["/product/"]` keeps product URLs and opens them. Every product page links to related products, so each page opened leads to more products.
* `max_pages=20` stops the crawl after it fetches 20 pages. That number is your cost ceiling: on this store, each page costs 1 credit.

## Write the discovery script

The `zenrows` package you installed in Lesson 2 includes `ZenRowsCrawlClient`. Create `discover.py`:

```python Python (SDK) theme={"dark"}
from zenrows import ZenRowsCrawlClient

ZENROWS_API_KEY = "YOUR_ZENROWS_API_KEY"
CATALOG_URL = "https://www.scrapingcourse.com/ecommerce/"

client = ZenRowsCrawlClient(api_key=ZENROWS_API_KEY)

crawl = client.create(
    CATALOG_URL,
    include_patterns=["/product/"],
    max_pages=20,
    max_items=200,
)
print("Started crawl", crawl.crawl_id, crawl.status.value)

crawl = client.wait(crawl.crawl_id)
print("Finished:", crawl.status.value)
print("Pages fetched:", crawl.coverage.pages_fetched, "| Product URLs found:", crawl.coverage.items_found)

product_urls = [result.url for result in client.results(crawl.crawl_id)]

with open("product_urls.txt", "w") as f:
    f.write("\n".join(product_urls) + "\n")

print(f"Saved {len(product_urls)} product URLs to product_urls.txt")
for url in product_urls[:5]:
    print(" ", url)
```

Here is what each call does:

<Steps>
  <Step title="create starts the crawl">
    `client.create()` returns at once with the crawl's ID and the status `running`. The crawl runs on Zenrows' side, so your script is free while it works. `max_items=200` lets the crawl keep up to 200 URLs, room for the whole catalog.
  </Step>

  <Step title="wait polls until it ends">
    `client.wait()` checks the crawl until its status is `completed`, `stopped` or `failed`, and returns it with its `coverage` counters: pages fetched and URLs kept.
  </Step>

  <Step title="results reads every URL">
    `client.results()` pages through the kept URLs for you. Each result has a `url`, already normalized and deduplicated.
  </Step>
</Steps>

## Run it

```bash theme={"dark"}
python discover.py
```

The crawl takes a few seconds on this store:

```text theme={"dark"}
Started crawl c_01M4Y99D5SE1AHNYEZW66AVEHA running
Finished: completed
Pages fetched: 20 | Product URLs found: 42
Saved 42 product URLs to product_urls.txt
  https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/
  https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/
  https://www.scrapingcourse.com/ecommerce/product/aeon-capri/
  https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee/
  https://www.scrapingcourse.com/ecommerce/product/aether-gym-pant/
```

Twenty pages turned up 42 product URLs: the 16 on the catalog page plus the related products linked from each product page opened. Open `product_urls.txt` to see the full list.

## Make it your own

The crawl stopped because it reached `max_pages`. To cover more of the store, raise the ceiling, for example `max_pages=200` for the full 188-product catalog. Each extra page is one more Fetch request, so `max_pages` is the dial between coverage and cost. For the rest of the course, the 42 URLs you have are plenty: every later lesson works the same on 42 URLs or 4,200.

| Change | How |
| - | - |
| Crawl the whole store | Raise `max_pages`, see [Crawl endpoints](/crawl/endpoints#start-a-crawl) |
| Also open the numbered catalog pages | `include_patterns=["/product/", "/page/"]`, then keep the URLs that contain `/product/` |
| Skip a section | `exclude_patterns=["/product-category/"]` |
| Get each page's HTML too | `output_format="html"`, then `client.content(crawl_id, result)`, see the [Crawl Python SDK](/crawl/sdk/python) |

<Tip>
  **Do the same from the CLI.** The [Zenrows CLI](/cli/crawl/introduction) starts the crawl, waits with `--follow`, and saves the URLs as JSON lines:

  ```bash theme={"dark"}
  zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --include-patterns /product/ --max-pages 20 --follow
  zenrows crawl results CRAWL_ID --out product_urls.jsonl
  ```
</Tip>

## Ask your AI agent

With the [Zenrows MCP server](/mcp/overview) connected to Claude, Cursor or any MCP client, paste this prompt. Your agent picks the right Zenrows tools and settings on its own.

```text theme={"dark"}
Use Zenrows to find every product page on https://www.scrapingcourse.com/ecommerce/,
spending up to 20 page loads. When it finishes, list the product URLs and tell me how many there are.
```

## What you built

`discover.py` turns one start URL into a list of product URLs in `product_urls.txt`. You can:

* Start a crawl with `ZenRowsCrawlClient.create()` and wait for it with `wait()`.
* Steer it with `include_patterns` and size it with `max_pages`.
* Read every kept URL with `results()`.

## Next lesson

<Card title="Lesson 4: Scrape every product with Batch" icon="arrow-right" href="/academy/foundations/scrape-at-scale">
  Send all the product URLs as one Batch job and turn the results into JSON and CSV files.
</Card>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.