Observability (traces, scores, evals)

Relay records every request as a trace: a tree of observations covering the model calls, the agent run around them, each tool call, each retrieval, workflow steps, and any spans your app sends itself. Traces carry who they were for (end user, session), how they were tagged, what they cost, and how they were scored. Everything lives in ClickHouse; the panel reads it back under Observability.

Because Relay sits in front of every provider, an app gets full traces without adding any tracing code — it only adds a few headers to say who the request is for.

#What is recorded automatically

Each generation records the requested and resolved model, provider, prompt / completion / cached / reasoning tokens, cost, latency, time to first token, retry attempts, whether a fallback served it (and from which provider), whether it was a cache hit or an idempotent replay (both cost $0), the generation parameters, and the prompt name and version that produced it.

Realtime voice sessions are traced too: one span per call and one generation per completed response, with the provider's text and audio token usage.

#Attributing traces: users, sessions, tags

Send these headers (any OpenAI SDK can, through its extra-headers option). Every observation in the request inherits them.

The OpenAI body fields work as well: user becomes the end user, and keys in metadata named trace_id, trace_name, session_id, user_id, tags, environment, release, parent_span_id (or parent_observation_id), prompt_name, prompt_version (optionally prefixed relay_ or langfuse_) are read; headers win over the body, and other keys become metadata (up to 50). Relay removes metadata before forwarding, because OpenAI rejects it on chat completions.

Every response returns X-Relay-Trace-Id (unless capture resolves to none for the request) and X-Relay-Observation-Id (plus the existing X-Relay-Request-Id), so your app can score the exact generation later. Traces open for chat, agent, responses, embeddings, workflow, realtime-mint and knowledge calls.

#Capture: metrics only, or full bodies

By default Relay records metadata only: cost, tokens, latency, models, users, sessions, tags and the trace structure, but no message text. Turning capture to full also stores inputs and outputs, which enables the transcript view, "add to dataset" and LLM-judge evaluators.

Captured text is redacted for PII (e-mail addresses, phone numbers and card-like numbers) when the workspace setting is on, truncated to the workspace's character cap, and removed after the body retention period while the metrics stay. Images in a captured body are replaced by a placeholder such as [image image/png 123KB]. Workspace settings reach the gateway within Telemetry:SettingsCacheSeconds (default 60); a workspace with no settings row gets the Telemetry:Capture:* and Telemetry:*RetentionDays config defaults. Captured text is other people's conversations: keep the default at metadata and enable full per key or per agent where you need it.

#Scores

A score is a quality signal attached to a trace, an observation or a session. Every score has a source:

Score configs (Scores → Score configs) define a score's type and allowed values — numeric with a range, boolean, or categorical with labels — so annotations and judges stay consistent. Scores appear on traces, sessions, users and prompt versions, and can be filtered on in the trace explorer.

#Annotation queues

Collect traces into a queue — from the trace list (select, Add to annotation queue) or a single trace — and pick which score configs reviewers fill in. Review walks through pending items one at a time, showing the input and output, with Complete & next and Skip.

#Evaluators (LLM-as-a-judge)

An evaluator scores live traffic continuously. Every Gateway:EvaluatorPollSeconds (default 60) it samples new generations — reading Gateway:EvaluatorLagSeconds (default 30) behind "now", so rows still in the writer buffer are not missed — that match its filter (agent, model, environment, tags, prompt, name) and have captured bodies, renders its judge prompt with {{input}}, {{output}} and {{metadata}}, asks the judge model for a verdict, and writes a judge score on the generation. Sampling is deterministic per generation, a per-run cap bounds cost, each generation is judged once, and judge and experiment traffic is never sampled. Judge calls are traced separately (evaluator: <name>) so production traces keep their real cost and latency.

#Datasets and experiments

A dataset is a curated set of test items — usually real traces worth keeping (Add to dataset on a trace or a selection), or imported JSONL/CSV. An experiment runs every active item through a model — optionally a variant model for A/B, and optionally a library prompt version — and scores each output by rule (contains, equals, regex) or with an LLM judge. Each case becomes its own trace with its score, so results can be opened, annotated and compared side by side on the dataset's Compare tab.

#Alerts

Workspace alert rules watch spend, request volume, error rate, p95 latency, p95 time to first token, a score's average, or error types not seen in the previous week — optionally narrowed to one key, model or environment. They are evaluated every Gateway:AlertRules:IntervalSeconds (default 60). A rule notifies once when it breaches and once when it recovers, through the operator channel and the rule's own Slack/Teams-compatible webhook (https only, sent through the gateway's SSRF-guarded client). The gateway-wide Gateway:Alerts:* thresholds described under Telemetry are a separate, operator-level channel.

#End-user spend limits

On a user's page, set a limit per day, week or month. With a hard stop, that user's requests through the chat pipeline are refused with 402 end_user_budget_exceeded once their spend (refreshed from telemetry every Gateway:BudgetRefreshMinutes, with the limits cached for Gateway:EndUserBudgetCacheSeconds) reaches it. These are keyed by your app's end-user id and are separate from panel member budgets — see Budgets.

#Retention, export and erasure

Retention is enforced by ClickHouse TTLs: rows can be deleted after N days, and captured text blanked after M days while metrics and scores stay (Observability settings). Export any window as JSON Lines or CSV from the trace list, or through GET /v1/telemetry/export. Erase data on a user's page deletes every observation and score for that end user (right to erasure); a trace can also be deleted from its page. Both are written to the audit log.

#Panel pages

The Dashboard adds traces, end users, sessions, cache-hit rate, p95 time to first token, top end users, score averages and spend by agent; the Prompts page has a per-version Metrics tab.

#Storage

The same spans are exported over OTLP when Gateway:Otlp:Enabled is on, with real trace, span and parent ids, so HyperDX shows the same tree. See Observability API for the endpoints.