Go SDK

Crawl is New. Any active Zenrows API key can use it. The client is the module github.com/zenrows/zenrows-go-sdk/service/crawl at v1.0.0, separate from the Fetch client in service/api. Go 1.25 or later is required.

Use this guide to start a crawl, wait for it, and read the URLs it keeps, or each page’s HTML, with the official Zenrows Go SDK.

Prefer raw HTTP? Crawl setup covers the same calls against the REST API. Parameters and errors are in the endpoint reference. For the complete method reference, see the SDK repository on GitHub.

Install and authenticate

go get github.com/zenrows/zenrows-go-sdk/service/crawl@v1.0.0

package main

import (
    "github.com/zenrows/zenrows-go-sdk/service/crawl"
)

func main() {
    client := crawl.NewClient(crawl.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    _ = client // the examples below call client
}

NewClient also reads ZENROWS_API_KEY when you omit WithAPIKey. WithBaseURL defaults to https://api.zenrows.com/v1. WithRetries defaults to 3, and WithTimeout defaults to 30 seconds per request.

Quickstart

Create returns as soon as the crawl is accepted, with StatusRunning. Wait polls until the crawl ends, or until WaitOptions.Timeout runs out (default 600 seconds). On timeout it returns the crawl still StatusRunning, without an error. A crawl that ends StatusFailed is returned too, with Error set. Results follows the cursor and does not poll, so call Wait first to read every URL. Depth is optional. Leave it at 0 and the crawl follows links until MaxItems or MaxPages stops it. Both default to 100 and cap at 100,000. A response crawl’s Depth is nil when the create left it out.

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/zenrows/zenrows-go-sdk/service/crawl"
)

func main() {
    client := crawl.NewClient(crawl.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    ctx := context.Background()

    created, err := client.Create(ctx, crawl.CreateParams{
        URL:             "https://www.scrapingcourse.com/ecommerce/",
        Depth:           1, // leave 0 for no depth limit
        MaxPages:        20,
        IncludePatterns: []string{"/product/"},
    })
    if err != nil {
        log.Fatal(err)
    }

    done, err := client.Wait(ctx, created.CrawlID, crawl.WaitOptions{})
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(done.Status, done.Coverage.ItemsFound)

    for result, err := range client.Results(ctx, created.CrawlID, crawl.ResultsOptions{}) {
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(result.URL)
    }
}

IncludePatterns chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only /product/ skips listing paths that lack that text, such as /page/. See Crawl setup.

Set CreateParams.IdempotencyKey so a retry with the same key and body returns the crawl the first request created. Create retries transient failures only when that key is set, and it never retries 429. Get returns the crawl and one page of results. GetOptions.Limit is 1 to 10,000 (default 1,000). NextCursor is nil only once the crawl has ended and you have read its last results.

Read the pages

Set OutputFormat to crawl.OutputFormatHTML to also fetch the page of every kept URL. Content takes a content id or a result’s ContentURL. Download streams every result as NDJSON. DownloadLine.HTML decodes the page when the line carries HTML.

created, err := client.Create(ctx, crawl.CreateParams{
    URL:             "https://www.scrapingcourse.com/ecommerce/",
    MaxPages:        20,
    IncludePatterns: []string{"/product/"},
    OutputFormat:    crawl.OutputFormatHTML,
})
if err != nil {
    log.Fatal(err)
}
if _, err = client.Wait(ctx, created.CrawlID, crawl.WaitOptions{}); err != nil {
    log.Fatal(err)
}

for result, err := range client.Results(ctx, created.CrawlID, crawl.ResultsOptions{}) {
    if err != nil {
        log.Fatal(err)
    }
    if result.ContentStatus == crawl.ContentStatusFetched {
        html, err := client.Content(ctx, created.CrawlID, result.ContentURL)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(result.URL, len(html))
    }
}

dl, err := client.Download(ctx, created.CrawlID)
if err != nil {
    log.Fatal(err)
}
defer dl.Close()
fmt.Println(dl.Status) // StatusRunning means a later download may hold more lines
for line, err := range dl.Lines() {
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(line.URL)
}

On a running crawl, Download holds only the results kept so far. Call Wait first for the full file. With OutputFormatHTML, each kept page is one more fetch, and MaxPages counts it. Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See Crawl pricing.

List and stop

List returns one page of the account’s crawls, newest first, without their results. Stop is safe to repeat: a crawl that already ended answers as it ended. Read the final counts with Get.

page, err := client.List(ctx, crawl.ListOptions{Limit: 20})
if err != nil {
    log.Fatal(err)
}
for _, item := range page.Crawls {
    fmt.Println(item.CrawlID, item.Status)
}

if _, err = client.Stop(ctx, created.CrawlID); err != nil {
    log.Fatal(err)
}

Pass page.NextCursor as ListOptions.Cursor for the next page. It is empty on the last page.

Errors

A non-2xx response is an APIError. Match it with errors.As. Code is the problem code, StatusCode is the HTTP status, and RetryAfter is the Retry-After header. A crawl that ended StatusFailed is not an APIError: read Error.Code and Error.Detail on it. A missing API key is a NotConfiguredError.

_, err := client.Create(ctx, crawl.CreateParams{
    URL:      "https://www.scrapingcourse.com/ecommerce/",
    MaxPages: 20,
})
var apiErr crawl.APIError
if errors.As(err, &apiErr) && apiErr.Code() == "too_many_crawls" {
    fmt.Println(apiErr.StatusCode, apiErr.RetryAfter)
}

An account already running 3 active jobs, shared with Batch, gets 429 too_many_crawls. Create does not retry that response. The other codes are in the endpoint reference.

Get Started

Fetch

Extract

Crawl

Batch

Browser Sessions

Install and authenticate

Quickstart

Read the pages

List and stop

Errors

​Install and authenticate

​Quickstart

​Read the pages

​List and stop

​Errors

Install and authenticate

Quickstart

Read the pages

List and stop

Errors