Model routing & fallback

When a request names a model, ModelRouter turns that into a concrete provider adapter and an ordered execution plan. This is where policy, auto selection, budgets and fallback live.

#Resolution order

1. Explicit model — the alias you asked for.
2. Default — if the request names none, the gateway-wide Gateway:DefaultModel. There is no per-workspace default. No model and no default → 400 model_not_found.
3. Alias remap — Gateway:ModelAliases:{requested} rewrites the name (see below).
4. auto / tier strategies — chosen from the catalog by price and latency (see below).
5. Lookup — the alias must exist and be enabled, with an enabled provider; otherwise 404 model_not_found.

Before executing, the router also enforces the per-key model allow-list (403 model_not_allowed), the workspace budget hard-stop (402 budget_exceeded) and, when the request names an end user, that user's spend limit (402 end_user_budget_exceeded). Model aliases are resolved globally by name: a workspace's own copy of an alias wins over a global one of the same name.

#Dynamic aliases

Gateway:ModelAliases:{requested} remaps a requested name to another at runtime — handy for redirecting gpt-4 → gpt-4o org-wide without touching apps or the catalog.

#auto strategies

Pass one of these as the model to let Relay choose among enabled models whose provider is enabled, narrowed to the key's allow-list:

Selection uses the catalog's pricing and the per-model EWMA latency (alpha 0.3) from ModelMetricsRegistry, updated from live outcomes. The registry also tracks a success ratio, shown at GET /v1/models/metrics, but selection does not use it. Models on a degraded provider are passed over while a healthy candidate exists. There is no capability filter: an auto request can land on any enabled chat model in the key's view, so keep realtime-only or embedding-only models off a key that uses auto by giving it an allow-list.

#Resilience

GatewayChatService wraps every call:

• Retry — on a non-streaming call, transient upstream failures (ProviderException.IsTransient: 5xx, 429, timeout, connection error) are retried with exponential backoff (Gateway:MaxRetries, default 2; Gateway:RetryBaseDelayMs, default 250, doubling per attempt). A streaming call is not retried: a transient failure before the first token moves straight to the next fallback target.
• Circuit breaking — ProviderHealthRegistry marks a provider degraded after 3 consecutive transient failures; it stays degraded for a 30-second cooldown, after which one request is let through. The router avoids degraded providers for auto and for fallback targets. A successful active probe clears the state.
• Fallback — try another model when this one fails transiently. See below.
• Active probing — HealthProbeService (opt-in, Gateway:HealthProbe:*) sends a token-free GET to each provider's base URL on an interval; any HTTP response counts as reachable. Results feed the circuit breaker and surface at GET /v1/providers/health, and a provider crossing the failure threshold — or recovering — sends an alert through the webhook / e-mail channel.

#Fallback

A model can name another model to be retried against when it fails. Set it per model in Models → Edit → Fallback model, which stores fallback_model_id on the row — no redeploy, no config change.

#Precedence

The router builds one fallback chain per request, first source that yields a target wins:

1. The model's own Fallback model, followed transitively — gpt-4o → gpt-4o-mini → claude-sonnet.
2. Gateway:FallbackModel — the global default, appended when the primary declares no chain of its own (and it is not the primary itself). This is the original behaviour and still works; a per-model link simply overrides it.
3. The auto runner-up — for auto / tier requests only, when neither of the above produced a target, second place in the ranking becomes the fallback.

A chain is capped at 3 targets including the primary. Each hop costs a full retry cycle before it is abandoned, so an unbounded chain would let one slow upstream stretch a single request across every model's retries.

#When it applies

#When it triggers

Fallback is not a general retry. It engages only when all of these hold:

• The failure is transient — upstream 5xx, 429, timeout, or connection error. A 400, 401 or 404 fails immediately, because a malformed or unauthorised request fails identically on any provider and retrying elsewhere only adds latency.
• The primary's own retries are exhausted first (non-streaming; a stream has none).
• For streaming, no token has reached the client yet. Once SSE output has begun it cannot be taken back, so a mid-stream failure ends the response with an error event.
• The next target's provider is not degraded. Degraded providers are skipped as fallbacks; the explicitly requested primary is always attempted regardless.

#Edge cases

• Loops are safe. A → B → A is a legitimate "each covers the other" setup and is accepted. The router tracks which models it has already tried and ends the chain rather than re-attempting one.
• A model cannot be its own fallback — rejected on save, since it would be a silent no-op.
• Dead links are skipped, not fatal. If the fallback is deleted, disabled, or its provider is disabled, the per-model chain ends there — a broken backup never stops a healthy primary from serving. If that leaves no fallback at all, Gateway:FallbackModel (when set) still applies. The panel shows such a link as (deleted model).
• Cost and telemetry are recorded against whichever model actually served the request. The generation is flagged fallback_used, and routed_provider records the provider it was routed away from, so fallback traffic can be told apart from direct traffic in the trace explorer and over OTLP (relay.fallback_used).

#Catalog cache

The model + provider catalog is cached in-process for Gateway:CatalogCacheSeconds (default 10) and shared across requests. The panel runs in a separate process and does not push edits to the gateway, so a fallback link or model change you just saved takes effect when the cache next expires — within that window.