Geekflare
PermissiveBSD-3-ClauseTested by Geekflare

Scrapy review

Python framework for writing fast, large-scale web crawlers and scrapers

Website GitHub Docs
Data refreshedMethodologyHow we tested
Founder of Geekflare

Chandan Kumar is a founder of Geekflare. He is a technology enthusiast and entrepreneur who loves to help businesses and people around the world. Chandan has worked at BNP Paribas, Citibank, Deutsche Bank, Motorola, and HP. He’s got an in-depth understanding of business software and IT infrastructure.

Verdict

Scrapy is the standard choice for Python teams that crawl many pages on a regular basis, with mature middleware, throttling and export pipelines. It does not run JavaScript on its own, so dynamic sites need the scrapy-playwright plugin.

Best for

Python teams running large or recurring crawls

Not for

Teams that want Markdown for LLMs or a point-and-click interface

Pros

  • Mature and widely used, maintained by Zyte with over 500 contributors
  • Built-in throttling, retries, caching, proxy support and exports to JSON, CSV and XML
  • Large plugin ecosystem, including Playwright rendering
  • Scrapyd runs spiders on a server through an HTTP API

Cons

  • No JavaScript rendering without a plugin such as scrapy-playwright
  • No built-in Markdown or LLM output
  • Scrapyd starts runs on request, so recurring schedules need cron or another tool
Ask Sparkian

Is Scrapy right for your project?

Describe what you need and AI will check the fit against this review.

Or try:

Scrapy features and capabilities

Spiders and selectors

You write a spider that follows links and pulls fields with CSS or XPath, and Scrapy handles requests, retries and concurrency for you.

Polite crawling

AutoThrottle adjusts the request rate to how fast each site responds, and new projects obey robots.txt, which keeps large crawls from overloading sites.

Exports and pipelines

Results go to JSON, CSV or XML files, or through item pipelines into a database, without custom code for the common cases.

Scrapyd for servers

Scrapyd runs your spiders on a server and lets other systems start and check jobs over HTTP.

Plugins and middleware

Middleware and plugins add browser rendering through Playwright, proxy rotation and other behaviour without changing your spiders.

JavaScript renderingRenders pages that build their content with JavaScript
PartlyVia the scrapy-playwright plugin
LLM-ready MarkdownReturns clean Markdown for AI and RAG pipelines
No
Structured extractionPulls fields into JSON with selectors or an LLM
YesCSS and XPath selectors
Site crawlingFollows links across a site, not just single pages
Yes
Proxy supportRoutes requests through your own proxies
YesBuilt-in proxy middleware; rotation needs a plugin
REST APIAccepts jobs over HTTP so other services can call it
PartlyScrapyd starts and checks spider jobs over HTTP
MCP serverLets AI agents call it as a tool
No

At a glance

Type
Crawling framework
Stack
Python 3.10+ / Twisted
SSO
No
Roles and permissions
No
Audit logs
No

Free vs paid editions

What you get when you self-host, and what needs a paid plan.

Scrapy is free and has no paid edition. Zyte, which maintains it, sells Scrapy Cloud for running spiders. Its free tier allows one hour of crawl time with one concurrent crawl, and paid capacity costs from $9 per unit a month, where a unit is 1 GB of RAM and one concurrent crawl.

Self-hosting Scrapy

Scrapy 2.19 needs Python 3.10 or later. The project gives no CPU or RAM figures.

New cloud server to ready

20 s

Memory in use at idle

63 MB of 4 GB

50-page crawl

8 s, 127 MB peak

Version tested

2.19.0 · Oct 18, 2026

  • Scrapy and Scrapyd installed with pip in a slim Python container and were ready in 20 seconds, using 63 MB at idle.
  • A 30-line spider pulled title, price and stock from each product page with CSS selectors. Capped at 50 pages it fetched 57 responses and 49 products in 1.9 seconds of crawling, 8 seconds including Scrapyd starting the job, and peaked at 127 MB.
  • On the JavaScript test page it found no rendered quotes, because the content is built in the browser. That needs the scrapy-playwright plugin.
  • Scrapyd's web page is a plain job list. Scheduling recurring runs needs cron or another scheduler.
Commands we ran, with Scrapyd in a container
# scrapyd.conf in the current folder sets bind_address = 0.0.0.0 for the container
docker run -d --name scrapyd -p 127.0.0.1:6800:6800 -v "$PWD":/work -w /work python:3.12-slim \
  sh -c "pip install scrapy scrapyd scrapyd-client && mkdir -p /etc/scrapyd && cp scrapyd.conf /etc/scrapyd/ && scrapyd"
# Deploy the project in ./books and schedule a run over Scrapyd's JSON API
docker exec scrapyd sh -c "cd /work/books && scrapyd-deploy"
curl http://127.0.0.1:6800/schedule.json -d project=books -d spider=catalogue
Our finished Scrapy job on the Scrapyd jobs page

Our finished Scrapy job on the Scrapyd jobs page

Our finished Scrapy job on the Scrapyd jobs page

What you can do with Scrapy

Large catalogue crawls

Crawl tens of thousands of product or listing pages on a schedule and export them to CSV, JSON or a database.

Price and stock monitoring

Run spiders nightly through Scrapyd and compare prices or availability with the previous run.

Public data collection

Gather research data from government or open data sites that serve plain HTML.

Scrapy alternatives

Crawl4AIOpen source crawler that turns web pages into Markdown and structured data for LLMsFirecrawlWeb scraping API that returns Markdown and structured data for AI apps, with a self-hosted editionCrawleeNode.js library for building crawlers that use plain HTTP or a headless browserMaxunNo-code web scraping platform that turns websites into APIs and spreadsheetsScraplingPython scraping library with fast HTTP requests, browser fetching, Scrapy-style spiders and Markdown outputPlaywrightBrowser automation library from Microsoft for Chromium, Firefox and WebKit