Crawlee

Overview

Crawlee is a web scraping and crawling library that handles the hard parts — request queuing, retries, proxy rotation, browser fingerprinting, and rate limiting. Use Cheerio for fast HTML-only scraping or Playwright/Puppeteer for JavaScript-rendered pages. Built-in storage for datasets, request queues, and key-value stores. Scales from single pages to millions of URLs.

When to Use

• Scraping data from websites (product prices, job listings, articles)
• Crawling entire sites for content or link analysis
• JavaScript-rendered pages (SPAs, React/Vue sites)
• Scraping at scale with proxy rotation and anti-blocking
• Structured data extraction with automatic retries

Instructions

Setup

npx crawlee create my-crawler        # official template, or add to an existing project:
npm install crawlee                  # CheerioCrawler / HttpCrawler only
npm install crawlee playwright       # browser crawling (Playwright or Puppeteer are separate installs)
npx playwright install chromium

Use ES modules ("type": "module" in package.json) so top-level await crawler.run(...) works. Checked against crawlee 3.18.

HTTP Crawling with Routing (Fast, No Browser)

Give start requests a label and route them to handlers with a router; enqueueLinks stays on the same hostname by default and deduplicates URLs.

// scraper.ts
import { CheerioCrawler, Dataset, createCheerioRouter } from "crawlee";

const router = createCheerioRouter();

router.addHandler("LISTING", async ({ enqueueLinks }) => {
  await enqueueLinks({ selector: "article.product_pod h3 a", label: "DETAIL" });
  await enqueueLinks({ selector: "li.next a", label: "LISTING" }); // pagination
});

router.addHandler("DETAIL", async ({ request, $, pushData }) => {
  await pushData({
    url: request.loadedUrl,
    title: $("h1").text().trim(),
    price: $(".price_color").first().text().trim(),
    scrapedAt: new Date().toISOString(),
  });
});

const crawler = new CheerioCrawler({
  requestHandler: router,
  maxConcurrency: 10,
  maxRequestRetries: 3,
  maxRequestsPerMinute: 300,       // politeness cap
  maxRequestsPerCrawl: 500,        // safety limit while developing
  respectRobotsTxtFile: true,      // skip URLs disallowed by robots.txt
  failedRequestHandler({ request, log }) {
    log.error(`Failed: ${request.url} after ${request.retryCount} retries`);
  },
});

await crawler.run([{ url: "https://books.toscrape.com/", label: "LISTING" }]);

const dataset = await Dataset.open();
await dataset.exportToCSV("books");   // saved in the key-value store as books.csv

Run it with npx tsx scraper.ts. Results land in ./storage/datasets/default/*.json and the CSV in ./storage/key_value_stores/default/books.csv. Set CRAWLEE_STORAGE_DIR to move the folder.

Browser Crawling (JavaScript-Rendered Pages)

// browser-scraper.ts
import { PlaywrightCrawler } from "crawlee";

const crawler = new PlaywrightCrawler({
  maxConcurrency: 5,           // browsers are heavy
  headless: true,              // set false to watch while developing

  async requestHandler({ page, request, pushData, infiniteScroll, log }) {
    await page.waitForSelector(".product-card");   // wait for the rendered content
    await infiniteScroll({ timeoutSecs: 20 });     // built-in helper for lazy lists

    const items = await page.$$eval(".product-card", (cards) =>
      cards.map((card) => ({
        name: card.querySelector("h3")?.textContent?.trim(),
        price: card.querySelector(".price")?.textContent?.trim(),
      }))
    );
    log.info(`${items.length} products on ${request.url}`);
    await pushData(items.map((item) => ({ ...item, sourceUrl: request.url })));
  },
});

await crawler.run(["https://shop.acme-outdoors.com/products"]);

Crawlee's browser crawlers apply browser fingerprints by default, so extra launch flags to hide automation are rarely needed. Prefer waitForSelector or locator waits over fixed waitForTimeout sleeps.

Proxy Rotation

import { CheerioCrawler, ProxyConfiguration } from "crawlee";

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: [
    `http://${process.env.PROXY_USER}:${process.env.PROXY_PASS}@proxy-eu.acme-proxies.net:8080`,
    `http://${process.env.PROXY_USER}:${process.env.PROXY_PASS}@proxy-us.acme-proxies.net:8080`,
  ],
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  useSessionPool: true,   // ties cookies and proxies to sessions; blocked sessions are retired
  async requestHandler({ request, proxyInfo, log }) {
    log.info(`${request.url} via ${proxyInfo?.hostname}`);
  },
});

Examples

Example 1: Scrape product data from an e-commerce site

User prompt: "Scrape all product names, prices, and ratings from books.toscrape.com and export to CSV."

The agent installs crawlee, builds a CheerioCrawler with a LISTING/DETAIL router following the "next" links, runs it, and exports books.csv from the default key-value store. The run log ends with Finished! Total N requests: N succeeded, 0 failed.

Example 2: Monitor competitor prices

User prompt: "Build a daily scraper that checks competitor prices and alerts when they change."

The agent builds a PlaywrightCrawler for the JS-rendered shop, saves prices into a named dataset (so the default purge does not erase history), compares each run with the previous one, and prints or sends the price changes.

Guidelines

• Cheerio for static HTML - much faster and lighter than a browser; use Playwright only when content needs JavaScript.
• Routers and labels - one handler per page type keeps listing and detail logic apart.
• pushData for output - writes to the default dataset; export with exportToCSV / exportToJSON or read with getData().
• Default storage is purged at the start of each run - copy results out, or use a named storage (Dataset.open("books")) to keep data across runs. Do not rely on it for incremental "compare with last run" without a named dataset.
• Be polite - keep respectRobotsTxtFile: true, set maxRequestsPerMinute, and check the site's terms before scraping.
• Limit while developing - maxRequestsPerCrawl stops a runaway crawl.
• Handle failures - failedRequestHandler runs after retries are exhausted; check request.retryCount.
• Proxy credentials from the environment - never hardcode them in source.