Skip to content

Add rate limiting middleware to service-runtime #41

@haasonsaas

Description

@haasonsaas

Problem

None of the 14 Go services implement rate limiting. Identity (OAuth issuer), llm-gateway (inference router), and gate (access proxy) are all exposed without request-rate protection. This is the most actionable security gap in the platform.

Proposal

Add a reusable rate limiting middleware to service-runtime/httpkit/ (or similar):

  1. Per-IP token bucket using golang.org/x/time/rate for single-instance limiting
  2. Distributed rate limiting via Redis for multi-replica deployments (service-runtime already depends on Redis)
  3. Configurable limits per route or scope (e.g., stricter on /oauth/token, looser on /healthz)
  4. Prometheus metrics for rate limit hits ({service}_rate_limit_hits_total with labels for route and action)

Suggested API

// In service-runtime/httpkit/ratelimit.go
func RateLimit(opts RateLimitOptions) func(http.Handler) http.Handler

type RateLimitOptions struct {
    RequestsPerSecond float64
    Burst             int
    KeyFunc           func(r *http.Request) string // default: client IP
    RedisClient       *redis.Client                // nil = in-memory only
    ExcludePaths      []string                     // e.g., /healthz, /metrics
}

Priority

High — this is a prerequisite for handling any significant traffic or defending against abuse.

Services that should adopt immediately

  1. identity (OAuth endpoints)
  2. llm-gateway (inference routing — cost amplification risk)
  3. gate (proxy — upstream protection)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions