Go SDK

Batch is available to every Zenrows account. You need an active Zenrows API key. The client is the module github.com/zenrows/zenrows-go-sdk/service/batch, separate from the Fetch client in service/api. Go 1.25 or later is required.

Use this guide to submit batch scraping jobs, track them, and collect results with the official Zenrows Go SDK.

Prefer raw HTTP? The REST API guide covers the same use cases against the plain REST API. Limits are in the introduction and error codes in API error codes. For the complete method reference, see the SDK repository on GitHub.

Install and authenticate

go get github.com/zenrows/zenrows-go-sdk/service/batch@latest

This page describes service/batch v1.1.0 or later. VerifyWebhookSignature, the Code* constants, DownloadToDirOptions.Skipped, AddTasksWithOptions and the paused-run waiter behavior arrived in v1.1.0.

package main

import (
    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    client := batch.NewClient(batch.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    _ = client // the examples below call client
}

NewClient also reads ZENROWS_API_KEY when you omit WithAPIKey. WithBaseURL defaults to https://async.api.zenrows.com/v1, WithRetries defaults to 3, and WithTimeout sets how long one request may take (no limit unless set).

Quickstart: submit, wait, get results

SubmitRegular returns a JobRef as soon as the job is accepted. job.Run().Wait polls the job’s current run until it reaches a terminal state (completed, stopped, or failed), or until WaitForRunOptions.Timeout runs out (default 300 seconds). Results follows the cursor for you.

package main

import (
    "context"
    "fmt"
    "log"
    "time"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    client := batch.NewClient(batch.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    ctx := context.Background()

    job, err := client.SubmitRegular(ctx, batch.SubmitRegularOptions{
        Tasks: []batch.Task{
            {URL: "https://www.scrapingcourse.com/ecommerce/"},
            {URL: "https://www.scrapingcourse.com/ecommerce/page/2/"},
        },
    })
    if err != nil {
        log.Fatal(err)
    }

    run, err := job.Run().Wait(ctx, batch.WaitForRunOptions{Timeout: 30 * time.Minute})
    if err != nil {
        log.Fatal(err)
    }
    if run.Status() == batch.RunStatusFailed {
        // An account reason stopped the run; FailureDetail explains it.
        log.Fatalf("run failed: %s", run.Data.FailureReason)
    }
    fmt.Println("run", run.Status(), "successful:", run.Stats().Successful)

    for row, err := range run.Results(ctx, "successful") {
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(row.TaskID, "->", row.ResultURL)
    }
}

A run that ends failed is returned without an error. It stopped for an account reason: Data.FailureReason is batch.FailureReasonInsufficientCredits, batch.FailureReasonSubscriptionInactive or batch.FailureReasonAPIKeyCapReached, and Data.FailureDetail (a *string) explains it, for example which cap and when it resets. A paused run makes no progress until someone resumes it, so Wait returns it with a batch.RunPausedError instead of polling until the timeout. Set WaitForRunOptions.WaitThroughPause to keep polling, for example when another process resumes it.

Configuring the scrape and correlating results

ZenRowsParams on the job applies to every task. A task’s own ZenRowsParams are merged over the job’s, and the task wins on a collision. Set ExternalID to match results back to your own records: letters, digits, ., _ and -, up to 128 characters, and it doesn’t have to be unique. A task can also fetch its target with POST: set Method: "POST" and a Body (a JSON value or a string, up to 16 KiB). A POST task can’t use js_render, js_instructions or json_response.

package main

import (
    "context"
    "log"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    client := batch.NewClient(batch.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    ctx := context.Background()

    job, err := client.SubmitRegular(ctx, batch.SubmitRegularOptions{
        ZenRowsParams: map[string]any{"js_render": true},
        Tasks: []batch.Task{
            {URL: "https://www.scrapingcourse.com/ecommerce/", ExternalID: "home"},
            {
                URL:           "https://www.scrapingcourse.com/javascript-rendering",
                ExternalID:    "js-page",
                ZenRowsParams: map[string]any{"wait_for": ".product-item"},
            },
        },
    })
    if err != nil {
        log.Fatal(err)
    }

    if _, err := job.Run().Wait(ctx, batch.WaitForRunOptions{}); err != nil {
        log.Fatal(err)
    }

    // One file per successful task, named by external_id. Rows with no body to fetch
    // (a failed task, or a result past its retention window) are collected in skipped.
    var skipped []batch.TaskResult
    n, err := job.Run().DownloadToDir(ctx, "./out", batch.DownloadToDirOptions{
        UseExternalID: true,
        Skipped:       &skipped,
    })
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("downloaded %d files, skipped %d", n, len(skipped))
}

Batch accepts the Fetch parameters as ZenRowsParams, including mode, extract and extract_fields. It doesn’t take the screenshot parameters, and the API rejects an unknown key with 400. Each ResultURL is a presigned link valid for 2 hours, and the SDK downloads straight from it. A successful task’s body stays available for 14 days by default, counted from the task’s UpdatedAt: when it finished, or when a partial rerun carried it over. After that its row reads batch.TaskStatusExpired with an empty ResultURL, and a successful filter still returns it. DownloadToDir and DownloadToMemory skip those rows and failed ones, and Skipped collects them. GetTaskContent on an expired task returns an APIError with Code() equal to batch.CodeResultExpired (410).

Open jobs: streaming URLs in batches

An open job keeps accepting tasks, up to 10,000 per call and 100,000 in total. Use AddTasksWithOptions with an IdempotencyKey so a retried call adds its tasks once; the SDK then retries it on a transient failure. Set LastBatch on the final call, or call Close. An open job counts toward your account’s 3 active jobs until it’s deleted, or closed with its run ended.

package main

import (
    "context"
    "log"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    client := batch.NewClient(batch.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    ctx := context.Background()

    job, err := client.SubmitOpen(ctx, batch.SubmitOpenOptions{})
    if err != nil {
        log.Fatal(err)
    }
    _, err = job.AddTasksWithOptions(ctx,
        []batch.Task{{URL: "https://www.scrapingcourse.com/ecommerce/"}},
        batch.AddTasksOptions{IdempotencyKey: "pages-batch-1"})
    if err != nil {
        log.Fatal(err)
    }
    _, err = job.AddTasksWithOptions(ctx,
        []batch.Task{{URL: "https://www.scrapingcourse.com/ecommerce/page/2/"}},
        batch.AddTasksOptions{IdempotencyKey: "pages-batch-2", LastBatch: true})
    if err != nil {
        log.Fatal(err)
    }
    log.Println("job", job.JobID(), "closed")
}

Retrying failures and scheduled jobs

RetryFailed starts a new run that re-scrapes only the failed tasks and carries the successful results over. Set IncludePending to also run the tasks a stopped run never started. Start it only once the previous run is terminal, or the API answers 409 run_not_terminal. A partial rerun must start within 10 days of when the run you retry started (409 partial_rerun_window_expired after that; a full Rerun still works). The API accepts every rerun with 202 and writes its tasks in the background, so set WaitForIngest to return only once the new run’s results pages are complete. A scheduled job fires on its own. Build the schedule with NewAt, NewRate, or NewCalendar; the shortest interval is 60 minutes, times of day must be full hours, and an account holds up to 5 scheduled jobs.

package main

import (
    "context"
    "log"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    client := batch.NewClient(batch.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    ctx := context.Background()

    retry, err := client.Job("JOB_ID").RetryFailed(ctx, batch.RetryFailedOptions{
        IncludePending: true,
        WaitForIngest:  true,
    })
    if err != nil {
        log.Fatal(err)
    }
    log.Println("retry run", retry.RunID())

    every6h, err := batch.NewRate(6, "hour")
    if err != nil {
        log.Fatal(err)
    }
    scheduled, err := client.SubmitScheduled(ctx, batch.SubmitScheduledOptions{
        Schedule: every6h,
        Tasks:    []batch.Task{{URL: "https://www.scrapingcourse.com/ecommerce/"}},
    })
    if err != nil {
        log.Fatal(err)
    }
    if _, err := scheduled.Schedule().Pause(ctx); err != nil {
        log.Fatal(err)
    }
}

Webhooks and HMAC keys

A webhook gets one event when a run finishes on its own: run.completed, or run.failed when an account-level error ends it, with the run’s stats. Stopping a run yourself sends no event. A delivery that fails or takes longer than 10 seconds is retried with backoff for up to 3 days and then dropped, so the same event can arrive more than once: deduplicate on the X-ZenRows-Event-Id header (WebhookEvent.EventID). Signed deliveries need an active HMAC key on your account, so create one before you set Signature: true. RotateHMACKey returns the base64-encoded secret once; store it. The first call creates your active key. Later calls stage a candidate key that FinalizeHMACKey promotes, so receivers that verify with the old key keep working until you finalize.

package main

import (
    "context"
    "log"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    client := batch.NewClient(batch.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    ctx := context.Background()

    key, err := client.RotateHMACKey(ctx)
    if err != nil {
        log.Fatal(err)
    }
    log.Println("store this secret now:", key.Kid, key.Secret)

    job, err := client.SubmitRegular(ctx, batch.SubmitRegularOptions{
        Tasks:   []batch.Task{{URL: "https://www.scrapingcourse.com/ecommerce/"}},
        Webhook: &batch.WebhookConfig{URL: "https://your-app.com/webhooks/zenrows", Signature: true},
    })
    if err != nil {
        log.Fatal(err)
    }

    // Change the webhook later. The signing choice is a required argument.
    if _, err := job.SetWebhook(ctx, "https://your-app.com/webhooks/zenrows-v2", true); err != nil {
        log.Fatal(err)
    }
}

client.SetJobWebhookURL(ctx, jobID, url) changes only the URL and keeps the current signing choice. PutJobWebhook is deprecated: a WebhookConfig without Signature set turns signing off. VerifyWebhookSignature checks a signed delivery’s X-Signature header (t=<unix>,v1=<hex>,kid=<key id>). It picks the secret by kid, decodes it from base64, recomputes HMAC-SHA256(key, "<t>.<raw_body>"), compares in constant time, and rejects a timestamp more than 5 minutes from now (Tolerance changes that). Pass the body exactly as received, and keep both secrets in the map during a rotation.

package main

import (
    "encoding/json"
    "io"
    "log"
    "net/http"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

// The kid and base64 secret that RotateHMACKey returned, from your secret store.
var secrets = map[string]string{"YOUR_KEY_ID": "YOUR_BASE64_SECRET"}

func handleWebhook(w http.ResponseWriter, r *http.Request) {
    body, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "bad body", http.StatusBadRequest)
        return
    }
    err = batch.VerifyWebhookSignature(body, r.Header.Get(batch.SignatureHeader), secrets,
        batch.VerifyWebhookSignatureOptions{})
    if err != nil {
        http.Error(w, "invalid signature", http.StatusUnauthorized)
        return
    }
    var event batch.WebhookEvent
    if err := json.Unmarshal(body, &event); err != nil {
        http.Error(w, "bad json", http.StatusBadRequest)
        return
    }
    log.Println(event.EventID, event.EventType, event.JobID)
    w.WriteHeader(http.StatusNoContent) // answer fast, then process the event
}

func main() {
    http.HandleFunc("/webhooks/zenrows", handleWebhook)
    log.Fatal(http.ListenAndServe(":3000", nil))
}

Estimating cost

EstimateCost prices tasks locally, with no API call. It returns a CostEstimate with Min and Max credits; they’re equal unless a task uses Adaptive Stealth Mode (mode=auto), which costs 1 to 25 credits depending on the site. A flag counts as on only for true or "true", the values the API accepts.

package main

import (
    "fmt"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    tasks := []batch.Task{
        {URL: "https://www.scrapingcourse.com/ecommerce/"},
        {URL: "https://www.scrapingcourse.com/ecommerce/page/2/"},
    }
    est := batch.EstimateCost(tasks, map[string]any{"js_render": true})
    fmt.Println(est) // "10 credits (2 tasks)"
    fmt.Printf("%d to %d credits for %d tasks\n", est.Min, est.Max, est.TaskCount)
}

Batch bills like Fetch. Successful tasks are billed, and so are tasks that fail because the target answered 404 or 410. Other failed tasks aren’t billed. Read what a run actually used from run.Stats().Spend; result rows don’t carry a spend field.

Error handling

A non-2xx response returns a batch.APIError. StatusCode is the HTTP status, Code() the stable problem code, and Detail the parsed problem body, including InvalidTasks on a 400. Compare Code() with the batch.Code* constants, one for every code the API sends, and keep a default branch. See Batch error codes for every code.

package main

import (
    "context"
    "errors"
    "log"

    "github.com/zenrows/zenrows-go-sdk/service/batch"
)

func main() {
    client := batch.NewClient(batch.WithAPIKey("YOUR_ZENROWS_API_KEY"))
    _, err := client.SubmitRegular(context.Background(), batch.SubmitRegularOptions{
        Tasks: []batch.Task{{URL: "not-a-url"}},
    })
    var apiErr batch.APIError
    if !errors.As(err, &apiErr) {
        return
    }
    detail := ""
    if apiErr.Detail != nil {
        detail = apiErr.Detail.Detail
    }
    switch apiErr.Code() {
    case batch.CodeInvalidArgument:
        if apiErr.Detail != nil {
            for _, t := range apiErr.Detail.InvalidTasks {
                log.Println("task", t.Index, t.Reason, t.Value)
            }
        }
    case batch.CodePaymentRequired, batch.CodeSubscriptionInactive:
        log.Println("check Billing in your dashboard")
    case batch.CodeAPIKeyCapReached:
        log.Println("raise the key's credit cap:", detail)
    case batch.CodeQuotaExceeded:
        log.Println("an account limit, wait for a job to finish or delete one:", detail)
    default:
        log.Println(apiErr.StatusCode, apiErr.Code())
    }
}

quota_exceeded is one of three account limits: 3 active jobs (shared with Crawl), 5 scheduled jobs, or 100,000 tasks in one open job. It clears only when a job’s run ends (for an open job, once it’s also closed) or a job is deleted, so the client returns it at once instead of retrying. Task-level failures aren’t errors. They come back as failed rows whose Error holds the code and detail.

Timeouts, retries, and idempotency

Transient failures (network errors, 502, 503, 504, and any 429 other than quota_exceeded) are retried with jittered exponential backoff, honoring Retry-After up to 30 seconds. Retries apply to GET, PUT, and DELETE, and to submits, reruns and add-tasks calls that carry an IdempotencyKey. Set WithTimeout, or pass a context with a deadline, to bound a call. Set IdempotencyKey on a submit, a rerun or an add-tasks call so a retry can’t do the work twice. The API keeps each key for 24 hours:

The same key with the same request replays the original response.
The same key with a different request, or a repeat while the first call is still being processed, returns 409 idempotency_key_conflict.
A submit or rerun the API refused, for example with 429 quota_exceeded, doesn’t use up its key: fix the cause and retry with the same key. An add-tasks call keeps its key, refused or not: a retry with the same body is checked again, and a corrected body needs a new key.
A key can be up to 128 characters.

Get Started

Fetch

Extract

Crawl

Batch

Browser Sessions

Install and authenticate

Quickstart: submit, wait, get results

Configuring the scrape and correlating results

Open jobs: streaming URLs in batches

Retrying failures and scheduled jobs

Webhooks and HMAC keys

Estimating cost

Error handling

Timeouts, retries, and idempotency

​Install and authenticate

​Quickstart: submit, wait, get results

​Configuring the scrape and correlating results

​Open jobs: streaming URLs in batches

​Retrying failures and scheduled jobs

​Webhooks and HMAC keys

​Estimating cost

​Error handling

​Timeouts, retries, and idempotency

Install and authenticate

Quickstart: submit, wait, get results

Configuring the scrape and correlating results

Open jobs: streaming URLs in batches

Retrying failures and scheduled jobs

Webhooks and HMAC keys

Estimating cost

Error handling

Timeouts, retries, and idempotency