Geekflare
PermissiveApache-2.0Tested by Geekflare

Crawl4AI review

Open source crawler that turns web pages into Markdown and structured data for LLMs

Website GitHub Docs
Data refreshedMethodologyHow we tested
Founder of Geekflare

Chandan Kumar is a founder of Geekflare. He is a technology enthusiast and entrepreneur who loves to help businesses and people around the world. Chandan has worked at BNP Paribas, Citibank, Deutsche Bank, Motorola, and HP. He’s got an in-depth understanding of business software and IT infrastructure.

Verdict

Crawl4AI is the most complete free option for turning websites into Markdown or JSON for an LLM, and its self-hosted server comes with a playground, a dashboard and an MCP endpoint. On my test server it was the fastest service here and used 538 MB at idle. You bring your own proxies and LLM keys, and deep crawls over the API have to be switched on first.

Best for

Teams feeding web pages into RAG or LLM pipelines from their own servers

Not for

Teams that want proxies and bot handling managed for them

Pros

  • Clean Markdown, with filters that drop menus, footers and boilerplate
  • Extracts JSON with CSS or XPath schemas, or with any LLM including local models
  • Docker server with a REST API, playground, dashboard and MCP endpoint
  • Deep crawls that can resume after a crash

Cons

  • Deep crawls and proxies over the API need settings on the server first
  • The LICENSE file adds an attribution requirement to Apache 2.0
  • Web search and SERP endpoints are only in the paid cloud
What people say
Ask Sparkian

Is Crawl4AI right for your project?

Describe what you need and AI will check the fit against this review.

Or try:

Crawl4AI features and capabilities

Markdown built for LLMs

Pages become Markdown with headings, tables and code kept, and filters can drop menus and footers, so you send fewer tokens to the model.

Extraction with or without an LLM

Fields come out as JSON through CSS or XPath schemas, or through any LLM including local models, so you only pay for a model where selectors are not enough.

Self-hosted API server

The Docker image serves Markdown, crawl, screenshot and PDF endpoints behind a token, with a playground and a monitoring dashboard, so other services can call it.

MCP endpoint

AI agents such as Claude Code can call your own Crawl4AI server as a tool, without sending pages through an outside service.

Deep crawls that resume

Crawls follow links breadth-first, depth-first or by relevance, and a long crawl can pick up where it stopped after a crash.

JavaScript renderingRenders pages that build their content with JavaScript
Yes
LLM-ready MarkdownReturns clean Markdown for AI and RAG pipelines
Yes
Structured extractionPulls fields into JSON with selectors or an LLM
YesCSS or XPath schemas, or any LLM including local models
Site crawlingFollows links across a site, not just single pages
Yes
Proxy supportRoutes requests through your own proxies
YesWith authentication and rotation
REST APIAccepts jobs over HTTP so other services can call it
YesDocker server, token required since 0.9.0
MCP serverLets AI agents call it as a tool
YesServed by the Docker server

At a glance

Type
AI crawler
Stack
Python 3.10+ / Playwright
Licence terms
The LICENSE file adds an attribution requirement to the Apache 2.0 text, asking public uses and derivative works to credit Crawl4AI. The README describes attribution as recommended.
SSO
No
Roles and permissions
No
Audit logs
No

Free vs paid editions

What you get when you self-host, and what needs a paid plan.

The library and the Docker server are free and include crawling, Markdown output, LLM extraction, proxy support and an MCP endpoint. Crawl4AI Cloud adds a web search API, a SERP API, extraction without your own LLM key and managed handling of bot walls. The cloud has 1,000 free credits a month, and paid plans start at $10 a month.

Self-hosting Crawl4AI

Crawl4AI's self-hosting guide asks for at least 4 GB of RAM for the container and a 1 GB shared memory size, and the Python package needs Python 3.10 or later.

New cloud server to ready

1 min 14 s

Memory in use at idle

538 MB of 4 GB

50-page crawl

13 s, 644 MB peak

Version tested

0.9.5 · Oct 18, 2026

  • One docker run and it answered with Markdown 74 seconds later, even though the image is 8.2 GB.
  • At idle it used 538 MB, well under the 4 GB its docs ask for. The 50-page crawl took 13 seconds and peaked at 644 MB.
  • The first deep crawl request was rejected: since 0.9 the server refuses deep crawls from API callers unless the operator sets CRAWL4AI_ALLOW_DEEP_CRAWL=true. Proxies are locked the same way and are set by the operator, not per request.
  • The JavaScript test page rendered, and product pages came back as clean Markdown with only one breadcrumb line of clutter.
Commands we ran
TOKEN=$(openssl rand -hex 32)
printf "CRAWL4AI_API_TOKEN=%s\nCRAWL4AI_ALLOW_DEEP_CRAWL=true\n" "$TOKEN" > c4a.env
docker run -d --name crawl4ai -p 127.0.0.1:11235:11235 --shm-size=1g \
  --env-file c4a.env unclecode/crawl4ai:0.9.5
# Playground: http://127.0.0.1:11235/playground  Monitor: /dashboard
curl http://127.0.0.1:11235/md -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"url": "https://books.toscrape.com/"}'
Turning a product page into Markdown in the Crawl4AI 0.9.5 playground

Turning a product page into Markdown in the Crawl4AI 0.9.5 playground

Turning a product page into Markdown in the Crawl4AI 0.9.5 playground
The Crawl4AI monitor after our 50-page crawl

The Crawl4AI monitor after our 50-page crawl

The Crawl4AI monitor after our 50-page crawl

What you can do with Crawl4AI

Knowledge base for a chatbot

Crawl your docs and help centre into clean Markdown and load it into a vector database for retrieval.

Web reading for AI agents

Connect the MCP endpoint so tools such as Claude Code can fetch and read pages through your own server.

Structured data from listings

Pull product, job or directory listings into JSON with a CSS schema, and only call an LLM for pages that need it.

Crawl4AI alternatives

FirecrawlWeb scraping API that returns Markdown and structured data for AI apps, with a self-hosted editionScrapyPython framework for writing fast, large-scale web crawlers and scrapersCrawleeNode.js library for building crawlers that use plain HTTP or a headless browserMaxunNo-code web scraping platform that turns websites into APIs and spreadsheetsScraplingPython scraping library with fast HTTP requests, browser fetching, Scrapy-style spiders and Markdown outputPlaywrightBrowser automation library from Microsoft for Chromium, Firefox and WebKit