Node.js SDK

Crawl is New. Any active Zenrows API key can use it. Crawl is on the main zenrows package from version 2.2.0, as client.crawl on a ZenRows instance or as its own ZenRowsCrawlClient. Node.js 20 or later is required.

Use this guide to start a crawl, wait for it, and read the URLs it keeps, or each page’s HTML, with the official Zenrows Node.js SDK. Response fields stay in the API’s form (crawl.crawl_id, result.content_url). The arguments you pass to create are camelCase (maxPages, includePatterns).

Prefer raw HTTP? Crawl setup covers the same calls against the REST API. Parameters and errors are in the endpoint reference. For the complete method reference, see the SDK repository on GitHub.

Install and authenticate

npm install "zenrows@^2.2.0"

import { ZenRows } from "zenrows";

const client = new ZenRows("YOUR_ZENROWS_API_KEY");

The same client also works on its own:

import { ZenRowsCrawlClient } from "zenrows";

const crawl = new ZenRowsCrawlClient("YOUR_ZENROWS_API_KEY");

Both forms take { baseURL, retries, timeout }. baseURL defaults to https://api.zenrows.com/v1, retries to 3, and timeout to 30 seconds per request. Pass them as the second argument of ZenRowsCrawlClient, or as crawl on the ZenRows options (new ZenRows(apiKey, { crawl: { retries: 3 } })).

Quickstart

create returns as soon as the crawl is accepted, with status "running". wait polls until the crawl ends, or until timeout seconds pass (default 600). It does not throw on timeout: the crawl comes back still "running". A crawl that ends "failed" resolves too, with error set. results is an async stream that follows next_cursor and does not poll, so call wait first to read every URL. depth is optional. Leave it out and the crawl follows links until maxItems or maxPages stops it. Both default to 100 and cap at 100,000.

const crawl = await client.crawl.create({
  url: "https://www.scrapingcourse.com/ecommerce/",
  depth: 1, // omit for no depth limit
  maxPages: 20, // stop after fetching 20 pages (default 100)
  includePatterns: ["/product/"], // keep only URLs containing one of these
});

const ended = await client.crawl.wait(crawl.crawl_id);
console.log(ended.status, ended.coverage.items_found);

for await (const result of client.crawl.results(crawl.crawl_id)) {
  console.log(result.url);
}

includePatterns chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only /product/ skips listing paths that lack that text, such as /page/. See Crawl setup.

Pass { idempotencyKey } as the second argument of create so a retry with the same key and body returns the crawl the first request created. get(crawlId, { cursor, limit }) returns one page of results. limit is 1 to 10,000 (default 1,000). next_cursor is null only once the crawl has ended and you have read its last results.

Read the pages

Set outputFormat: "html" to also fetch the page of every kept URL. content takes a result, its content_url, or its content id. download returns the crawl’s status (from the X-Crawl-Status header) and an async stream of parsed lines.

const crawl = await client.crawl.create({
  url: "https://www.scrapingcourse.com/ecommerce/",
  maxPages: 20,
  includePatterns: ["/product/"],
  outputFormat: "html",
});
await client.crawl.wait(crawl.crawl_id);

for await (const result of client.crawl.results(crawl.crawl_id)) {
  if (result.content_status === "fetched") {
    const html = await client.crawl.content(crawl.crawl_id, result);
    console.log(result.url, html.length);
  }
}

const download = await client.crawl.download(crawl.crawl_id);
console.log(download.status); // "running" means a later download may hold more lines
for await (const line of download.lines) {
  console.log(line.url);
}

On a running crawl, download holds only the results kept so far. Call wait first for the full file. With outputFormat, each kept page is one more fetch, and maxPages counts it. Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See Crawl pricing.

List and stop

list returns one page of the account’s crawls, newest first, without their results. stop is safe to repeat: a crawl that already ended answers as it ended. Read the final counts with get.

const page = await client.crawl.list({ limit: 20 });
for (const item of page.crawls) {
  console.log(item.crawl_id, item.status);
}

await client.crawl.stop(crawl.crawl_id); // the id from create

Pass page.next_cursor as cursor for the next page. It is absent on the last page.

Errors

Every non-2xx response throws ZenRowsCrawlError, with status, code, detail, and retryAfter. A crawl that ended failed is not an exception: read error.code and error.detail on it.

import { ZenRowsCrawlError } from "zenrows";

try {
  await client.crawl.create({
    url: "https://www.scrapingcourse.com/ecommerce/",
    maxPages: 20,
  });
} catch (err) {
  if (err instanceof ZenRowsCrawlError && err.code === "too_many_crawls") {
    console.log(err.status, err.detail, err.retryAfter);
  } else {
    throw err;
  }
}

An account already running 3 active jobs, shared with Batch, gets 429 too_many_crawls. create does not retry that response. The other codes are in the endpoint reference.

Get Started

Fetch

Extract

Crawl

Batch

Browser Sessions

Install and authenticate

Quickstart

Read the pages

List and stop

Errors

​Install and authenticate

​Quickstart

​Read the pages

​List and stop

​Errors

Install and authenticate

Quickstart

Read the pages

List and stop

Errors