PermissiveApache-2.0Tested by Geekflare

agentgateway review

Rust proxy for LLM, MCP and agent-to-agent traffic, with per-key budgets, guardrails and a built-in UI

Website GitHub Docs

Data refreshedOctober 22, 2026MethodologyHow we tested

Founder of Geekflare

Chandan Kumar is a founder of Geekflare. He is a technology enthusiast and entrepreneur who loves to help businesses and people around the world. Chandan has worked at BNP Paribas, Citibank, Deutsche Bank, Motorola, and HP. He’s got an in-depth understanding of business software and IT infrastructure.

More guides by Chandan

Verdict

agentgateway puts LLM routing, MCP tool federation and agent-to-agent traffic behind one Rust proxy that runs from a single binary or on Kubernetes, with budgets, guardrails and a cost dashboard in the free version. The catch is that limits are counted per instance unless you buy Solo Enterprise, and the project moves fast, with a new minor version about every month.

Best for

Teams that want one gateway for LLM calls, MCP servers and agents

Not for

Teams that need response caching in the gateway

Pros

Handles LLM, MCP and A2A traffic in one proxy
Runs as a single binary with no Kubernetes needed
Per-key dollar budgets and a cost dashboard in the free version
Several guardrail types that can run in sequence

Cons

Rate limits are local to each instance unless you pay for Solo Enterprise
No response caching
Fast release pace means frequent upgrades
In the simple LLM config, failover helps the next request while the failing one still errors

Ask Sparkian

Is agentgateway right for your project?

Describe what you need and AI will check the fit against this review.

agentgateway features and capabilities

Virtual models with failover

One model name can point at several providers in priority groups, so traffic shifts to a backup group when the primary models are evicted as unhealthy.

Virtual keys with spend caps

Each client gets its own revocable key with a dollar or token budget that either blocks or just records overspend, and provider keys never leave the gateway.

Built-in cost dashboard

With a database and a model price list set, the UI charts spend by model, provider, user or group without a separate monitoring stack.

MCP tool federation

Several MCP servers, or REST APIs described by OpenAPI, can be offered to agents through one endpoint with per-tool authorisation.

Layered guardrails

Regex filters can mask card numbers and emails, and requests and responses can also be checked by OpenAI moderation, cloud guardrail services or your own webhook.

FallbacksSwitches to another model or provider when a call fails: YesPriority groups with health-based eviction
Load balancingSpreads traffic across keys, deployments or providers: Yes
Budgets and limitsSpending caps or rate limits per key, user or team: YesPer-key dollar or token budgets; limits across replicas are Enterprise
Cost trackingRecords spend per request and per key, user or team: YesNeeds SQLite or PostgreSQL
CachingExact or semantic response caching: No
GuardrailsPII redaction, content filters or prompt-injection checks: Yes
MCP gatewayExposes or governs MCP servers and tools: Yes

At a glance

Type: LLM, MCP and A2A gateway
Stack: Rust
Databases: SQLite, PostgreSQL
SSO: Free edition
Roles and permissions: Free edition
Audit logs: Free edition

Free vs paid editions

What you get when you self-host, and what needs a paid plan.

The open source project is free under Apache 2.0 and includes the LLM, MCP and A2A gateway, guardrails, virtual keys, per-key budgets, the cost dashboard and OIDC login for the UI. Solo Enterprise for agentgateway is a separate licensed distribution that adds global rate limits and budget resources across replicas, a cost management UI for Kubernetes, MCP tool search and code mode, on-behalf-of identity, FIPS builds and 24x7 support. Solo publishes no prices and quotes each deployment, with 30-day trial keys on request.

Self-hosting agentgateway

agentgateway gives no minimum CPU or RAM requirements; the binary runs on Linux, macOS and Windows, and the Kubernetes mode needs a cluster with the Gateway API.

New cloud server to ready

4 s

Memory in use at idle

8 MB of 4 GB

Median request

0.46 ms

Requests/s, 32 clients

3,336

Peak memory

76 MB

Version tested

1.6.0 · Oct 21, 2026

One container, ready 4 seconds after docker compose up, using 8 MB at idle. It is the lightest gateway we tested by a wide margin.
Against the mock it added about 0.5 ms per request and handled 3,336 requests a second with 32 clients, peaking at 76 MB.
The first request to the broken model returned OpenRouter's 400, as the docs warn, and evicted that target. The next requests went straight to the Haiku backup. Retries that would save the first request exist only in the routing-based config, not the simple llm one.
--validate-only accepted a config that the gateway then refused to start with, so start it once before you trust a change. There is no login on the API by default.

Config we ran

# docker run -p 127.0.0.1:4000:4000 -v ./config.yaml:/config.yaml -v ./data:/data \
#   cr.agentgateway.dev/agentgateway:v1.6.0 -f /config.yaml
config:
  database: { url: "sqlite:///data/data.db" }
llm:
  port: 4000
  models:
  - name: mock                              # the stand-in provider for the overhead test
    provider: { custom: { formats: [{ type: completions }] } }
    params: { model: mock-model, baseUrl: "http://mock/v1", apiKey: none }
  - name: haiku
    provider: openrouter
    params: { model: anthropic/claude-haiku-5.5, apiKey: "$OPENROUTER_API_KEY" }
  - name: broken-primary
    visibility: internal
    provider: openrouter
    params: { model: anthropic/claude-model-that-does-not-exist, apiKey: "$OPENROUTER_API_KEY" }
    health: { unhealthyExpression: "response.code >= 400", eviction: { consecutiveFailures: 1, duration: 60s } }
  - name: haiku-backup
    visibility: internal
    provider: openrouter
    params: { model: anthropic/claude-haiku-5.5, apiKey: "$OPENROUTER_API_KEY" }
  virtualModels:
  - name: broken
    routing:
      failover:
        targets: [{ model: broken-primary, priority: 0 }, { model: haiku-backup, priority: 1 }]

OutputPassed1.6.0 · served by the Haiku backup

Request: The second POST /v1/chat/completions with model "broken", after the first one failed and evicted the broken target; asked "In one sentence, what does an LLM gateway do?"

{
  "model": "anthropic/claude-haiku-5.5",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "An LLM gateway is a middleware layer that sits between applications and one or more large language model providers, routing requests and handling concerns like authentication, rate limiting, cost tracking, caching,"
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 60,
    "total_tokens": 85,
    "cost": 3.25e-05
  }
}

What you can do with agentgateway

One endpoint for every model

Give internal apps an OpenAI-compatible endpoint with virtual keys and dollar budgets per team, and fail over between providers.

Governed MCP access for coding agents

Put company MCP servers and internal REST APIs behind one gateway with OAuth login and per-tool rules.

Gateway for self-hosted models on Kubernetes

Route requests to self-hosted model servers by GPU load and KV cache with the Gateway API Inference Extension, next to hosted providers.

agentgateway alternatives

BifrostGo gateway that puts many LLM providers behind one OpenAI-compatible API, with failover, virtual keys, budgets, semantic caching and an MCP gateway

LiteLLMPython SDK and proxy server that puts 100+ LLM providers behind one OpenAI-format API, with virtual keys, budgets and spend tracking

Apache APISIXAPI gateway on NGINX and etcd with AI plugins for LLM proxying, fallbacks, token limits, caching and guardrails

New APISelf-hosted AI gateway with a web console for routing, quotas, billing and usage logs, built on One API

HigressAPI gateway built on Istio and Envoy, with Wasm plugins for LLM proxying, token quotas, semantic caching and MCP hosting

Agent RouterKubernetes control plane for LLM and MCP traffic built on Envoy Gateway, formerly Envoy AI Gateway