> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zenrows.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> The Zenrows API (Fetch, Extract via extract=auto, and Batch) is described in OpenAPI 3.1 at https://www.zenrows.com/openapi.json and listed in the RFC 9727 API catalog at https://www.zenrows.com/.well-known/api-catalog. Use the spec for exact parameter names and types.

# Go SDK

> Start a crawl, wait for it, and read its URLs or HTML with the official Zenrows Go SDK. Crawl is the service/crawl module at v1.0.0.

<Note>
  Crawl is **New**. Any active Zenrows API key can use it. The client is the module `github.com/zenrows/zenrows-go-sdk/service/crawl` at v1.0.0, separate from the Fetch client in `service/api`. Go 1.25 or later is required.
</Note>

Use this guide to start a crawl, wait for it, and read the URLs it keeps, or each page's HTML, with the official Zenrows Go SDK.

<Tip>
  Prefer raw HTTP? [Crawl setup](/crawl/setup) covers the same calls against the REST API. Parameters and errors are in the [endpoint reference](/crawl/endpoints). For the complete method reference, see the [SDK repository on GitHub](https://github.com/ZenRows/zenrows-go-sdk).
</Tip>

## Install and authenticate

```bash theme={"dark"}
go get github.com/zenrows/zenrows-go-sdk/service/crawl@v1.0.0
```

```go theme={null} theme={"dark"}
package main

import (
    "github.com/zenrows/zenrows-go-sdk/service/crawl"
)

func main() {
    client := crawl.NewClient(crawl.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    _ = client // the examples below call client
}
```

`NewClient` also reads `ZENROWS_API_KEY` when you omit `WithAPIKey`. `WithBaseURL` defaults to `https://api.zenrows.com/v1`. `WithRetries` defaults to 3, and `WithTimeout` defaults to 30 seconds per request.

## Quickstart

`Create` returns as soon as the crawl is accepted, with `StatusRunning`. `Wait` polls until the crawl ends, or until `WaitOptions.Timeout` runs out (default 600 seconds). On timeout it returns the crawl still `StatusRunning`, without an error. A crawl that ends `StatusFailed` is returned too, with `Error` set. `Results` follows the cursor and does not poll, so call `Wait` first to read every URL.

`Depth` is optional. Leave it at 0 and the crawl follows links until `MaxItems` or `MaxPages` stops it. Both default to 100 and cap at 100,000. A response crawl's `Depth` is nil when the create left it out.

```go theme={null} theme={"dark"}
package main

import (
    "context"
    "fmt"
    "log"

    "github.com/zenrows/zenrows-go-sdk/service/crawl"
)

func main() {
    client := crawl.NewClient(crawl.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    ctx := context.Background()

    created, err := client.Create(ctx, crawl.CreateParams{
        URL:             "https://www.scrapingcourse.com/ecommerce/",
        Depth:           1, // leave 0 for no depth limit
        MaxPages:        20,
        IncludePatterns: []string{"/product/"},
    })
    if err != nil {
        log.Fatal(err)
    }

    done, err := client.Wait(ctx, created.CrawlID, crawl.WaitOptions{})
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(done.Status, done.Coverage.ItemsFound)

    for result, err := range client.Results(ctx, created.CrawlID, crawl.ResultsOptions{}) {
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(result.URL)
    }
}
```

<Note>
  `IncludePatterns` chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only `/product/` skips listing paths that lack that text, such as `/page/`. See [Crawl setup](/crawl/setup#start-a-crawl).
</Note>

Set `CreateParams.IdempotencyKey` so a retry with the same key and body returns the crawl the first request created. `Create` retries transient failures only when that key is set, and it never retries `429`.

`Get` returns the crawl and one page of results. `GetOptions.Limit` is 1 to 10,000 (default 1,000). `NextCursor` is nil only once the crawl has ended and you have read its last results.

## Read the pages

Set `OutputFormat` to `crawl.OutputFormatHTML` to also fetch the page of every kept URL. `Content` takes a content id or a result's `ContentURL`. `Download` streams every result as NDJSON. `DownloadLine.HTML` decodes the page when the line carries HTML.

```go theme={null} theme={"dark"}
created, err := client.Create(ctx, crawl.CreateParams{
    URL:             "https://www.scrapingcourse.com/ecommerce/",
    MaxPages:        20,
    IncludePatterns: []string{"/product/"},
    OutputFormat:    crawl.OutputFormatHTML,
})
if err != nil {
    log.Fatal(err)
}
if _, err = client.Wait(ctx, created.CrawlID, crawl.WaitOptions{}); err != nil {
    log.Fatal(err)
}

for result, err := range client.Results(ctx, created.CrawlID, crawl.ResultsOptions{}) {
    if err != nil {
        log.Fatal(err)
    }
    if result.ContentStatus == crawl.ContentStatusFetched {
        html, err := client.Content(ctx, created.CrawlID, result.ContentURL)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(result.URL, len(html))
    }
}

dl, err := client.Download(ctx, created.CrawlID)
if err != nil {
    log.Fatal(err)
}
defer dl.Close()
fmt.Println(dl.Status) // StatusRunning means a later download may hold more lines
for line, err := range dl.Lines() {
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(line.URL)
}
```

On a running crawl, `Download` holds only the results kept so far. Call `Wait` first for the full file.

With `OutputFormatHTML`, each kept page is one more fetch, and `MaxPages` counts it. Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See [Crawl pricing](/crawl/endpoints#pricing).

## List and stop

`List` returns one page of the account's crawls, newest first, without their results. `Stop` is safe to repeat: a crawl that already ended answers as it ended. Read the final counts with `Get`.

```go theme={null} theme={"dark"}
page, err := client.List(ctx, crawl.ListOptions{Limit: 20})
if err != nil {
    log.Fatal(err)
}
for _, item := range page.Crawls {
    fmt.Println(item.CrawlID, item.Status)
}

if _, err = client.Stop(ctx, created.CrawlID); err != nil {
    log.Fatal(err)
}
```

Pass `page.NextCursor` as `ListOptions.Cursor` for the next page. It is empty on the last page.

## Errors

A non-2xx response is an `APIError`. Match it with `errors.As`. `Code` is the problem code, `StatusCode` is the HTTP status, and `RetryAfter` is the `Retry-After` header. A crawl that ended `StatusFailed` is not an `APIError`: read `Error.Code` and `Error.Detail` on it. A missing API key is a `NotConfiguredError`.

```go theme={null} theme={"dark"}
_, err := client.Create(ctx, crawl.CreateParams{
    URL:      "https://www.scrapingcourse.com/ecommerce/",
    MaxPages: 20,
})
var apiErr crawl.APIError
if errors.As(err, &apiErr) && apiErr.Code() == "too_many_crawls" {
    fmt.Println(apiErr.StatusCode, apiErr.RetryAfter)
}
```

An account already running 3 active jobs, shared with [Batch](/batch/introduction), gets `429` `too_many_crawls`. `Create` does not retry that response. The other codes are in the [endpoint reference](/crawl/endpoints#errors).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.