Geekflare

Crawl API. A whole website, as Markdown.

Send one URL and get every page on the site back as Markdown or HTML. Sitemaps, links, JavaScript and retries are handled for you, and you pay per page.

Get a free API key
4.9average across
G2 — 4.8 out of 5Google — 4.9 out of 5Product Hunt — 5.0 out of 5
  • 500 free credits every month
  • No card to start
  • Failed requests cost nothing

Crawl a site in a few seconds

The demo stops at 5 pages and shows up to 20,000 characters of each. Sign up to crawl whole sites and get every page in full.

Try:
Trusted by teams at
PfizerNBCUniversalTCSHostingerKissflowLookoutPlivoClearSaleSparkianCBSplitOmreon
PfizerNBCUniversalTCSHostingerKissflowLookoutPlivoClearSaleSparkianCBSplitOmreon

The crawler you don't have to build.

Finding pages, fetching them and keeping track of a large job are the hard parts of crawling.

Finds every page

Pages are found from the links on each page, and from any sitemap or text file the crawl reaches. Start from a sitemap URL to crawl every page in it. Each URL is fetched once.

maxDepth

Stays where you point it

Limit how deep and how many pages a crawl goes, and include or skip URLs by pattern, so a crawl of /docs/ doesn't wander into the blog.

allowPatterns

JavaScript rendering

Pages render in headless Chrome by default, so React and Vue sites return their content. Static sites can skip the browser for 0.25 credits a page.

renderJS

Blocked pages retried

Failed pages are retried, and with proxyMode set to auto, a page that blocks the crawler goes through a premium proxy for 4 credits more.

proxyMode

Progress you can watch

Large crawls run in the background. Check pages done, failed pages with the reason, and an estimated finish time, or cancel the job.

GET /crawl/{id}

Webhooks and a ZIP

Get a signed webhook when a crawl starts, finishes or fails, or one for every page. When it's done, download every page in one ZIP.

webhookUrl

Two ways to run a crawl.

Wait for the pages in one request, or start a job for a whole site and collect the pages as they finish.

POST /crawl/sync

Wait for the pages

One request that returns every page in the response. Suits a docs section, a small site, or an agent that needs the pages now.

  • Up to 500 pages
  • Pages and credits in one JSON response
  • If it runs out of time, you get the pages done so far, marked partial
POST /crawl

Run it in the background

Starts a job and returns its ID straight away. Suits whole sites, and pipelines that load pages as they come in.

  • Up to 10,000 pages
  • Progress, errors and a trace with GET /crawl/{id}
  • Webhooks as pages finish, and a ZIP at the end

What customers say.

Read all reviews
“Found Geekflare API to get markdown from URL for my AI agents. It is fast and cheaper and works on almost every website.”
Ram DasiArchitect, PA Consulting
“After trying many scraping services, I selected Geekflare to scrape public directories. Mainly for two reasons - it is cheaper and fast.”
Vishu SharmaCTO
“Documentation was easy to follow. We were scraping dynamic pages within hours. Very reliable service.”
Marco SilvaAnalytics Engineer

The parameters you'll use most.

What each one does and what it costs.

  • urlstring—

    Where the crawl starts. Any public URL.

  • maxPagesnumberIncluded

    The most pages to fetch. Defaults to 500, the most /crawl/sync allows. /crawl allows up to 10,000.

  • maxDepthnumberIncluded

    How many links away from the start URL to go. Defaults to 3.

  • allowPatternsstring[]Included

    Only crawl URLs that match one of these patterns, e.g. "/docs/*".

  • denyPatternsstring[]Included

    Skip URLs that match, e.g. "/tag/*" or "/login".

  • includeExternalbooleanIncluded

    Follow links to other domains. Defaults to false.

  • formatstringIncluded

    markdown or html. Defaults to markdown.

  • renderJSboolean0.25 to 1 a page

    Render pages in headless Chrome. Defaults to true. Set false for static sites.

  • proxyModestring+4 a page if used

    auto uses a premium proxy only for pages that block the crawler. true always uses one.

  • proxyCountrystring+4 a page

    Crawl through a premium proxy in this country, e.g. "de".

  • webhookUrlstringIncluded

    HTTPS URL that gets job events. Add webhookEvents with crawl.page for one call per page.

POST /crawl/sync returns up to 500 pages in the response. POST /crawl runs a background job of up to 10,000 pages. The endpoint reference has the full list.

Endpoint reference

What people use it for.

A knowledge base from a docs site

Point it at a product's documentation and load every page into a vector store. Markdown keeps the headings, so chunks split where the sections do.

  • Every page under /docs/, and nothing else
  • Headings kept for clean chunk boundaries
  • Re-crawl on a schedule to keep answers current
Typical request
POST /crawl
{
  "url": "https://docs.example.com/",
  "allowPatterns": ["/docs/*"],
  "maxPages": 2000,
  "format": "markdown"
}
What comes back
idthe crawl job
GET /crawl/{id}progress and pages
pages[].content in Markdown
pages[].depth0, 1, 2…
links away from the start URL

Your first crawl, in a few lines.

quickstart.pyplain REST
# pip install requests
import requests

response = requests.post(
    "https://api.geekflare.com/crawl/sync",
    headers={"x-api-key": "<api-key>"},
    json={"url":"https://example.com","maxPages":10},
)
print(response.json())
Crawl Reference

One API. Every stack.

One account works from an agent, a workflow builder, an SDK or your own service.

MCP server

Sign in once and Claude, ChatGPT, Cursor or Codex can use Geekflare's tools directly, with no glue code to write.

ClaudeChatGPTCursorCodexAny MCP client
Set up the MCP server

LLM & agent pipelines

LLM-ready Markdown straight into a retrieval pipeline, with the boilerplate stripped so you spend context on content, not navigation.

RAGAgentsClean MarkdownGrounded search
See the solutions

Workflow builders

Run any endpoint as a step in a visual scenario. Useful when the person who needs the data does not write code.

ZapierMaken8n
Browse integrations

SDKs & REST

First-party typed SDKs for Python and Node. Everything else talks plain REST, from any language.

PythonNode.jsPostman
Open the API reference

How many pages each plan crawls.

A crawl is charged per page, at the same rate as a scrape, with no charge per job. Failed pages cost nothing.

2 months free
  • Free
    $0/mo
    No card
    Credits a month
    500
    Pages without a browser0.25 credits
    2,000
    Pages with JavaScript1 credit
    500
    Pages with a premium proxy5 credits
    –
    Requests per second
    1
  • Starter
    $19/mo
    Credits a month
    20,000
    Pages without a browser0.25 credits
    80,000
    Pages with JavaScript1 credit
    20,000
    Pages with a premium proxy5 credits
    4,000
    Requests per second
    5
  • GrowthMost popular
    $69/mo
    Credits a month
    100,000
    Pages without a browser0.25 credits
    400,000
    Pages with JavaScript1 credit
    100,000
    Pages with a premium proxy5 credits
    20,000
    Requests per second
    10
  • Business 500K
    $299/mo
    Credits a month
    500,000
    Pages without a browser0.25 credits
    2,000,000
    Pages with JavaScript1 credit
    500,000
    Pages with a premium proxy5 credits
    100,000
    Requests per second
    25
  • Business 1M
    $499/mo
    Credits a month
    1,000,000
    Pages without a browser0.25 credits
    4,000,000
    Pages with JavaScript1 credit
    1,000,000
    Pages with a premium proxy5 credits
    200,000
    Requests per second
    25
Start freeCompare plans and credit packsEvery account starts free. Upgrade when you need more, or buy a credit pack from $10.

Before you write the integration.

Everything else is in the endpoint docs, and our support team replies to every email.

Open the endpoint docs

A crawl is charged per page, the same as a scrape: 1 credit for a page rendered with JavaScript, the default, 0.25 credits with renderJS: false, and 5 credits for a page that goes through a premium proxy. There's no charge for the crawl itself. Pages that fail cost nothing. Each page's cost is in pages[].credits, and the total in creditsConsumed.

$0.50 to $0.95 for pages rendered with JavaScript on monthly plans, depending on the plan, and a quarter of that without a browser. Yearly billing costs less. See pricing for each plan.

From the links on each page it fetches, and from any sitemap or text file it reaches. To crawl every page in a sitemap, start from the sitemap's URL, such as https://example.com/sitemap.xml. A URL is fetched once, however many pages link to it. GET /crawl/{id}/trace shows how any page was reached from the start URL.

Use allowPatterns to crawl only matching URLs, such as "/docs/*", and denyPatterns to skip others. maxDepth limits how many links away from the start URL it goes, and maxPages caps the total. Links to other domains are skipped unless you set includeExternal.

POST /crawl/sync waits for the crawl and returns up to 500 pages in the response. Use it for small sites, sections of a site and agent calls. POST /crawl starts a background job of up to 10,000 pages and returns its ID straight away. Check progress with GET /crawl/{id}, get a webhook when it finishes, and download the pages as a ZIP.

Yes. The Map API returns every URL a crawl would visit, with its status code, plus estimatedCredits for crawling them. It costs 1 credit per 1,000 URLs found, and nothing if it finds none. A crawl also stops when your credits run out, and marks the result as partial.

Yes. Pages render in headless Chrome by default, so React, Vue and Angular sites return their content. Failed pages are retried, and with proxyMode: "auto", a page that blocks the crawler goes through a premium proxy for 4 credits more. Some heavily protected sites may still block it.

No. Pages that return a 4xx or 5xx status are left out of the crawl and cost nothing. The Map API still lists them with their status code, which makes it a quick way to find broken pages.

Web Scraping fetches the one URL you send. Crawl starts from a URL, finds the other pages on the site and fetches each of them. A crawled page costs the same as a scrape.

Crawling publicly available pages is generally legal, but it depends on the site's terms of service and on privacy laws such as GDPR and CCPA. It is your responsibility to crawl public data, respect robots.txt and avoid collecting personal information.

Ask AI about Geekflare

Open the question in the assistant you already use.

Start with 500 free credits a month.

No card needed. That covers up to 2,000 pages without a browser, or 500 with JavaScript.

Get your API keyRead the docs