Request pipeline

Every request to /v1/chat/completions or /v1/messages passes through nine stages in order. Each stage can independently reject, transform, or short-circuit the request.

Stage overview

Client request
      │
      ▼
┌─────────────────────────┐
│  01  Authentication     │  Identify user, resolve team
└─────────────┬───────────┘
              │
              ▼
┌─────────────────────────┐
│  02  Content Policy     │  Block patterns, check token count
└─────────────┬───────────┘
              │
              ▼
┌─────────────────────────┐
│  03  Token Count        │  Count prompt tokens, enforce model limit
└─────────────┬───────────┘
              │
              ▼
┌─────────────────────────┐
│  04  Rate Limiting      │  req/min, tokens/min, tokens/day
└─────────────┬───────────┘
              │
              ▼
┌─────────────────────────┐
│  05  PII Scrubbing      │  Detect & replace sensitive entities
└─────────────┬───────────┘
              │
              ▼
┌─────────────────────────┐
│  06  RAG Context        │  Semantic search, inject chunks
└─────────────┬───────────┘
              │
              ▼
┌─────────────────────────┐
│  07  Cache Lookup       │  Return cached response if hit
└─────────────┬───────────┘
              │ (miss)
              ▼
┌─────────────────────────┐
│  08  LLM Call           │  Route via LiteLLM, fallback models
└─────────────┬───────────┘
              │
              ▼
┌─────────────────────────┐
│  09  Metrics & Usage    │  Record to DB, restore PII, emit metrics
└─────────────────────────┘
              │
              ▼
        Client response

Stage details

01 — Authentication

• Extracts the API key from Authorization: Bearer
• Looks up key hash in the database (SHA-256 comparison)
• Resolves associated user and team
• Rejects expired or inactive keys and enforces the endpoint capability scope (chat, responses, or embeddings)
• Attaches user/team context to the request for downstream stages
• Rejects with 401 if key is missing, unknown, or revoked

02 — Content Policy

• Checks the concatenated prompt text against content_policy.blocked_patterns (case-insensitive literal match)
• Rejects with 400 (content_policy_violation) if any pattern matches
• Runs before token counting to fail fast on obvious attacks
• Disabled by setting content_policy.enabled: false

03 — Token Count

• Counts prompt tokens using tiktoken (model-appropriate encoding)
• Enforces content_policy.max_input_tokens (default 32 000)
• Stores the count for stage 04 (rate limiting deducts from buckets)
• Rejects with 400 if the prompt exceeds the token limit

04 — Rate Limiting

Five user/team limits are evaluated together; any can reject:

1. User req/min — rate_limiting.defaults.requests_per_minute
2. User tokens/min — rate_limiting.defaults.tokens_per_minute
3. User tokens/day — rate_limiting.defaults.tokens_per_day
4. Team tokens/min — team’s tpm_limit (if team has override)
5. Team tokens/day — team’s daily_token_limit

Rejects with 429 and Retry-After header on any overflow.

After the LLM responds, Relay reconciles actual prompt and completion tokens against the amount reserved before the call. This applies to regular and streaming responses.

See Rate limiting for bucket mechanics and Redis backend.

05 — PII Scrubbing

• Runs Presidio AnalyzerEngine across all message content
• Detected entities are replaced with request-local 128-bit placeholders: <<PII_EMAIL_ADDRESS_8e841b7a…>>
• The placeholder→original mapping is stored in request context for stage 09
• System instructions, diffs, typed text blocks, and tool/function arguments share the same boundary
• Disabled by setting pii.enabled: false

See PII scrubbing.

06 — RAG Context

• Embeds the last user message with all-MiniLM-L6-v2
• Derives repository filters from rag:repo:* or rag:* scopes on the authenticated key
• Queries ChromaDB only within those authorized repositories
• Irreversibly redacts PII found in retrieved chunks
• Appends source-labelled chunks after application instructions inside an untrusted-reference delimiter
• Recounts the enriched prompt before enforcing the input ceiling and reserving rate-limit tokens
• No-op if ChromaDB is empty or if rag.enabled: false

See RAG integration.

07 — Cache Lookup

• Hashes the (normalized messages + model) to a cache key
• Returns the cached response immediately on hit — stages 08–09 are skipped
• Disabled by setting cache.enabled: false (default)

08 — LLM Call

• Routes to the provider via LiteLLM based on the model name prefix
• On provider error (5xx, timeout): tries fallback_models in order
• Supports streaming (SSE) pass-through for both OpenAI and Anthropic formats

09 — Metrics & Usage

• Counts completion tokens from the response
• Commits a UsageRecord and corresponding AuditLog in one database transaction
• Restores PII placeholders in the response content (reverse of stage 05)
• Increments Prometheus counters
• Stores response in cache if cache.enabled: true (non-streaming only)

Skipping stages

Each non-authentication stage can be disabled in config.yaml:

rag:
  enabled: false
pii:
  enabled: false
content_policy:
  enabled: false
rate_limiting:
  enabled: false
cache:
  enabled: false

Authentication cannot be disabled for Relay-issued keys. Provider-key passthrough is a separate, explicitly enabled BYOK mode and does not create user-attributed usage/audit records.