Relay docs

Caching & idempotency

#Response cache

Relay can cache successful responses to cut cost and latency for repeated calls.

  • Enable with Gateway:Cache:TtlSeconds (0 = off). Size is capped by Gateway:Cache:MaxEntries (default 5000).
  • Scope — the chat cache key covers the model, the system prompt, the messages, temperature, top_p, max_tokens and stop. It does not cover tools, tool_choice, response_format, seed or other pass-through fields, so bypass the cache for requests that differ only in those. The embeddings key covers the model, the input, encoding_format and dimensions. Neither key is scoped to an API key or workspace: identical requests from two keys share an entry.
  • Per-request bypass — send X-Relay-Cache: no (or false; chat also accepts bypass) to force a fresh call.
  • Signal — the X-Relay-Cache response header reports HIT or MISS.

Non-streaming chat completions (including agent runs and /v1/responses) and embeddings honour the cache; a streamed response is never cached or served from it. Cached hits are still recorded to telemetry, flagged as cache hits and costed at $0, so usage stays accurate without double-billing.

#Idempotency

For safe retries of calls that must not run twice, send an Idempotency-Key header. Relay stores the result against that key — per API key, in process, for 24 hours, sharing the cache's MaxEntries cap — so a retry with the same key returns the original result instead of running the request again, with Idempotency-Replayed: true and X-Relay-Cost-Usd: 0. It works whether or not the response cache is on. Only successful responses are stored, so a retry after a failure runs again. Use a fresh unique key per logical operation (e.g. a UUID per user action).

Idempotency is honoured by the non-streaming chat pipeline — /v1/chat/completions with stream: false, non-streamed agent runs, /v1/responses, workflow LLM nodes — and by workflow run starts and webhook triggers. Streamed calls and embeddings ignore it.

#When to use which

  • Cache — read-heavy, repeatable prompts (classification of the same text, FAQ answers, embeddings of stable content).
  • Idempotency — write-like operations you might retry after a network blip, where running twice would be wrong.

Both stores are in-process. In a multi-instance deployment each instance has its own; treat them as a best-effort optimisation, not a shared store.