Geekflare
PermissiveApache-2.0Tested by Geekflare

Crawlee review

Node.js library for building crawlers that use plain HTTP or a headless browser

Website GitHub Docs
Data refreshedMethodologyHow we tested
Founder of Geekflare

Chandan Kumar is a founder of Geekflare. He is a technology enthusiast and entrepreneur who loves to help businesses and people around the world. Chandan has worked at BNP Paribas, Citibank, Deutsche Bank, Motorola, and HP. He’s got an in-depth understanding of business software and IT infrastructure.

Verdict

Crawlee gives JavaScript teams one API for HTTP and browser crawling, with a persistent queue, proxy rotation and automatic scaling built in. It is a library, so scheduling, monitoring and a user interface are up to you or the Apify platform, and the Playwright template needed two fixes before it built and ran on my server.

Best for

JavaScript and TypeScript teams building production crawlers

Not for

Teams that want a ready-made service with an API and dashboard

Pros

  • Switch between HTTP, Playwright and Puppeteer crawlers with the same code
  • Persistent request queue and storage for results
  • Proxy rotation, sessions and automatic scaling to the server's resources
  • A Python version with a similar API

Cons

  • No built-in scheduler or web interface
  • No built-in Markdown or LLM extraction
  • Version 4 is in release candidates, so expect breaking changes
Ask Sparkian

Is Crawlee right for your project?

Describe what you need and AI will check the fit against this review.

Or try:

Crawlee features and capabilities

One API for HTTP and browsers

Start with a fast HTTP crawler and switch to Playwright or Puppeteer for pages that need JavaScript, keeping the same routing code.

Persistent queue and storage

URLs to visit and scraped results are kept on disk, so a crawl can stop and resume and the results are ready to export.

Proxy and session rotation

Crawlee rotates proxies, keeps sessions tied to them and retries failed requests on its own.

Automatic scaling

It runs more pages in parallel when the server has spare CPU and memory, and fewer when it does not.

Ready Docker setup

Project templates include a Dockerfile built on Apify's browser images, so the crawler runs the same way on any server.

JavaScript renderingRenders pages that build their content with JavaScript
YesPlaywright or Puppeteer crawlers
LLM-ready MarkdownReturns clean Markdown for AI and RAG pipelines
No
Structured extractionPulls fields into JSON with selectors or an LLM
YesCheerio, JSDOM or browser selectors in your code
Site crawlingFollows links across a site, not just single pages
Yes
Proxy supportRoutes requests through your own proxies
YesBuilt-in rotation and sessions
REST APIAccepts jobs over HTTP so other services can call it
No
MCP serverLets AI agents call it as a tool
No

At a glance

Type
Crawling framework
Stack
TypeScript / Node.js
SSO
No
Roles and permissions
No
Audit logs
No

Free vs paid editions

What you get when you self-host, and what needs a paid plan.

Crawlee is free and has no paid edition. Apify, which builds it, runs Crawlee projects on its cloud platform. Apify has a free plan with $5 of usage a month, and paid plans start at $19 a month, or $17 a month billed yearly.

Self-hosting Crawlee

Crawlee's README asks for Node.js 22.13 or later, and Playwright or Puppeteer has to be installed separately for browser crawling. It gives no CPU or RAM figures.

New cloud server to ready

4 min 10 s

50-page crawl

16 s, 499 MB peak

Version tested

3.18.2 · Oct 18, 2026

  • The playwright-ts template failed to compile, because it installs TypeScript 6, which wants an explicit rootDir. Adding one line to tsconfig.json fixed it.
  • The built image then couldn't launch a browser: the template asks for Playwright 1.64.1, while its Docker base image ships the browsers for 1.58.2. Pinning Playwright to 1.58.2 fixed it.
  • With both fixes, an empty server got to a finished crawl in about four minutes, most of it building a 3.3 GB image.
  • The crawler fetched 50 pages in 16 seconds with a real browser, peaking at 499 MB, and rendered all 10 quotes on the JavaScript test page. It runs as a job and exits, so there is no idle memory to report.
Commands we ran, with the two fixes the template needed
npx crawlee create books -t playwright-ts
cd books
# TypeScript 6 needs an explicit rootDir: add "rootDir": "src" to tsconfig.json
# Match Playwright to the browsers in the template's Docker image
npm pkg set dependencies.playwright=1.58.2
docker build -t books-crawler .
docker run --rm books-crawler

What you can do with Crawlee

Crawling JavaScript-heavy sites

Use real browsers for single-page apps and switch to fast HTTP crawling for pages that don't need them.

Scrapers in a Node.js codebase

Keep crawlers in the same TypeScript repo and CI as the rest of your product.

Long crawls that resume

The persistent request queue lets a large crawl continue after a crash or a redeploy.

Crawlee alternatives

Crawl4AIOpen source crawler that turns web pages into Markdown and structured data for LLMsFirecrawlWeb scraping API that returns Markdown and structured data for AI apps, with a self-hosted editionScrapyPython framework for writing fast, large-scale web crawlers and scrapersMaxunNo-code web scraping platform that turns websites into APIs and spreadsheetsScraplingPython scraping library with fast HTTP requests, browser fetching, Scrapy-style spiders and Markdown outputPlaywrightBrowser automation library from Microsoft for Chromium, Firefox and WebKit