Relay docs

Evaluations

Evals let you measure prompt/model quality with repeatable test cases, and A/B compare a candidate against a baseline before you ship a change.

Relay has two complementary evaluation tools. This page covers eval runs (the Evals & A-B page): a batch of cases scored against a model, with an optional variant. The observability side adds datasets and experiments (curated items, usually real traces, each run becoming its own scored trace) and LLM-as-a-judge evaluators that score live traffic continuously. Eval runs can draw their cases from a dataset, so the two meet in the middle.

#Model

An EvalRun holds a set of test cases and one scoring method. Each case has an input and an expected value; the run's method is applied to every case. You run the cases against a model — and optionally a variant model for A/B comparison — and Relay scores each output.

Cases come inline, or from a Dataset (its active items, same workspace). A run can also name a library prompt version (resolved once by label, so every case uses the same version) and its own system prompt. A case input that parses as a JSON messages array is sent as a conversation rather than a single user turn.

#Scoring methods

EvalProcessorService (a background service polling Gateway:EvalPollSeconds, default 10, with concurrency Gateway:EvalConcurrency, default 4, clamped 1–32) runs queued eval runs and scores each output with the run's method:

  • equals — trimmed, case-insensitive match against the expected value.
  • contains — case-insensitive substring.
  • regex — case-insensitive pattern (1-second timeout).
  • judge — an LLM judge scores the output 0–1; 0.5 or above passes.

A case with no expected value always fails the rule-based methods. It aggregates a pass rate per model, so an A/B run shows baseline vs. variant side by side.

Every case is its own trace (session eval:<runId>, environment eval), with bodies captured unless the workspace capture is none, and its score is written to relay_score with source eval — so a case can be opened in the trace explorer, annotated, or added to a dataset.

#Using evals

  • Panel — the Evals & A-B page: define cases (or pick a dataset), pick a model (+ optional variant, + optional prompt version), run, and compare pass rates.
  • Control-plane API — api/Evals: POST search, GET {id}, POST (create), POST {id}/cancel, DELETE {id}.

#A typical workflow

  1. Curate 20–50 representative cases for a task (with expected substrings / patterns), or collect them from real traces into a dataset.
  2. Run against your current production model → record the pass rate.
  3. Change a prompt version or try a cheaper model as the variant.
  4. Compare pass rates; promote the change (e.g. label a new prompt version production) only if it holds up.

Pair this with the prompt library's versioning and the cost simulator to trade off quality against spend deliberately.

#Evals never fall back

A model's fallback is deliberately not applied to the model under test. An eval measures a named model, and an A/B run compares two of them — if a transient failure quietly routed a case to the fallback, the score would be attributed to a model that never answered it, and the comparison would be against a mixture. A failed case is honest; a substituted one is not. The judge model of a judge-scored run goes through the normal chat pipeline and can fall back, since it is not what is being measured.

A run left running by a restart is requeued and re-run from the start.