Relay docs

The panel, page by page

The control panel is the Blazor admin app. Admins sign in through the company identity app (SSO), and everything they see is scoped to the active workspace chosen in the header. The sidebar has four sections; here is every page and what it is for.

#Overview

  • Dashboard (/) — usage for the active workspace over 24h / 7d / 30d, optionally compared with the previous period. Thirteen KPI tiles (requests, cost, tokens, average cost per request, p50/p95 latency, error rate and count, traces, end users, sessions, cache-hit rate, p95 time to first token), a time series switchable between requests, tokens, cost and errors, top models, spend by provider and by app, a prompt-vs-completion token split, top end users, quality (score averages), spend by agent, recent requests, and a cost attribution table. Filter by model, provider, app or status. Exports the series as PNG or CSV.
  • Realtime voice (/realtime) — a spoken conversation with a realtime model or an agent. Pick a voice, connect, and talk; shows the live transcript and the raw protocol frames. Only realtime-capable models, and public agents pointed at one, are offered. See Realtime.
  • Playground (/playground) — an interactive chat console. Pick a model or an agent and one of the workspace's API keys — the panel signs the call on the key's behalf, so its secret never reaches the browser (or paste a key to call the gateway directly). Streams responses, renders Markdown, takes file attachments via the paperclip, shows variable inputs when an agent's prompt needs them, and can save the chat to Conversations. Opens with ?agent=<id> from an agent's page.
  • Telemetry (/telemetry) — the detailed request log with filters and cost/usage attribution grouped by team, app, model, provider, end user, agent, prompt or environment.

#Observability

See Observability for the concepts behind these pages.

  • Traces (/traces, /traces/{id}) — every trace with full-text search and filters (user, session, tags, environment, model, status, operation, prompt, score, cost, latency), switchable to generations or all observations, with live mode, export, and bulk add to queue / dataset. A trace page shows the observation tree with a timing waterfall, each observation's input/output (chat-rendered), metadata, parameters, usage, cost and errors, its scores and annotation, open in Playground, export and delete.
  • Sessions (/sessions, /sessions/{id}) — conversations grouped by session id (or agent thread), with turn count, duration, cost and scores, and a transcript view.
  • End users (/users, /users/{id}) — per-user traces, sessions, spend, errors and scores; per-user charts, a spend limit (daily/weekly/monthly, hard stop or alert) and erasure.
  • Scores (/scores) — score analytics (averages, distributions, trends), every score, and score configs (numeric ranges, boolean, categorical labels).
  • Annotation queues (/annotations, /annotations/{id}) — collect traces for human review against chosen score configs, one item at a time.
  • Evaluators (/evaluators) — LLM-as-a-judge evaluators that sample live generations matching a filter and write judge scores.
  • Datasets (/datasets, /datasets/{id}) — curated test items (from traces or JSONL/CSV import) and the experiments run over them, with side-by-side comparison.
  • Alerts (/alerts) — workspace alert rules on spend, volume, error rate, latency, time to first token, score averages or new error types, with breach and recovery notifications.

#Gateway

  • API Keys (/api-keys) — issue keys with scopes, a model allow-list, a per-minute rate limit, an expiry date, a default environment and a trace-capture level; edit any of those later without changing the secret. The list leads with each key's prefix and status, filters by active / expiring / inactive, and a row opens a detail panel with the key's policy, its last 30 days of usage and a ready-to-run curl. The full key is shown once at creation, with a copy button and a snippet, and the dialog stays open until you confirm you have stored it; after that only its prefix, because only a hash is stored. Revoke or delete anytime, each behind a confirmation. See Authentication & keys.
  • Models (/models) — the catalog callers request. A public alias maps to a provider model, with pricing (including a cached-input rate), context window and max output, capability flags (tool calling, vision, realtime) and an optional fallback model retried against on transient upstream failure (routing & fallback). Each row shows the alias, the provider and upstream id it runs on, capabilities, price, fallback and the last 30 days of requests and spend; filter by provider or by status, where Needs attention collects unpriced models, dead fallback links and models on a missing or disabled provider. The editor is a sectioned panel that can list what the provider serves and look up Azure prices, previews the fallback chain, and reads the entry back as a sentence before saving. Clicking a model opens a detail view with everything the catalog holds: route and owning workspace, pricing with a worked example, limits, the resolved fallback chain and which models fall back to it, 30-day usage, a ready-to-run curl, and the audit trail. Enable, disable and delete (with confirmation) from the row menu. Copy to workspace… duplicates a model (and, if needed, its provider) into another workspace you belong to.
  • Providers (/providers) — upstream LLM endpoints: name, family, base URL, and the API key, which is entered here and stored encrypted (a badge shows whether a provider's credential is encrypted, from config, or missing); a config SecretRef is the optional fallback. Also an optional realtime endpoint override and the Azure deployment-discovery fields. Copy to workspace… copies a provider, with or without its stored key. See Providers.
  • Agents (/agents, /agents/{id}, /agents/new, /agents/{id}/edit) — compose an agent from a model, a prompt (written or from the library), skills, tools, MCP servers, workflows-as-tools, a connector, an optional knowledge base or dataset, generation defaults, PII redaction, trace capture and an audience (public or shared with named users). Each agent has its own page with an overview, a multi-turn Run console (through a picked API key, with variables and an acting user), versions (snapshot, publish, restore, diff), its traces and API snippets; it is created, edited and duplicated (?from=) on a full-page sectioned editor. Copy to workspace… moves the live draft to another workspace, relinking references by name. See Agents.
  • Workflows (/workflows, /workflows/{id}) — the visual, versioned workflow builder with a Test panel (dry run, pinned data, run node / run to here), templates, import/export and per-workflow access. Runs (/workflows/runs, /workflows/{id}/runs/{runId}) shows every run with a waterfall, step data, retry and debug in editor; Approvals (/workflows/approvals) is the human-in-the-loop inbox; Credentials & variables (/workflows/credentials, /workflows/variables) holds encrypted, host-bound credentials and {{ env.* }} variables. See Workflows.
  • Prompts (/prompts, /prompts/{name}, /prompts/new) — the versioned prompt library, grouped into folders by / in the name. Each prompt has its own page with a version timeline, labels, a live compile, a diff, per-version metrics and fetch snippets; new prompts and versions are written on a full-page editor. See Prompt library.
  • Skills (/skills) — reusable instruction blocks attached to agents.
  • Tools (/tools) — the function-tool registry (OpenAI shape), with schema validation on save.
  • MCP (/mcp) — register MCP servers (HTTP/SSE JSON-RPC), discover their tools, inspect what was discovered, and attach them to agents. See Skills, tools & MCP.
  • Connectors (/connectors) — register a Microsoft 365 OAuth app, connect or reconnect your own account, see who has connected, and delete. See Connectors.
  • Knowledge (/knowledge) — per-workspace knowledge bases (Qdrant collections); ingest documents and run searches. The Add document dialog takes multiple files at once (Markdown, text, CSV/TSV, JSON, logs, PDF, DOCX, XLSX, PPTX) or a URL — click the drop area or drag files onto it — and shows a row per file: queued, uploading with a percentage, indexing, then the chunk count or the failure reason, with an overall progress bar. Rows stay on screen after the run. See Knowledge bases (RAG).
  • Batch (/batch) — queue large workloads; the gateway processes each row asynchronously. Accepts multiple .csv/.jsonl files at once, queueing one job per file (named after the file), with per-file progress; shows status, results, dead-letter rows and lets you cancel. See Batch processing.
  • Evals & A-B (/evals) — run test cases (inline or from a dataset) against a model (+ optional variant and prompt version), score outputs by rule or judge, compare pass rates. See Evaluations.
  • Conversations (/conversations) — persisted agent threads; call an agent with ?thread=<id> for multi-turn memory. See Conversations.

#Workspace

  • Teams (/teams) — workspaces and their members: the unit that owns API keys, budgets, catalog entries and usage attribution.
  • Budgets (/budgets) — one workspace spend ceiling plus an optional allocation per member, where the member allocations can never sum past the ceiling. Hard-stop budgets block requests with a 402 once spend hits the cap; alert-only budgets just track it. See Budgets & spend limits.
  • Cost (/cost) — a cost explorer of actual spend from telemetry, plus a simulator that estimates a call's cost from token counts or sample text using catalog pricing. See Cost.
  • Observability settings (/observability-settings) — the workspace's trace body capture default, PII redaction, body size cap and retention. See Observability.
  • Audit (/audit) — every control-plane change: who created/updated/revoked/deleted/copied what, and when.
  • Developer (/develop) — generates copy-paste snippets (cURL, Python, JavaScript, .NET) for a chat, agent or prompt call against your gateway URL.
  • Settings (/settings) — gateway connection test, provider health and probe status, model performance, and a reference of runtime config keys. Configuration portability exports one JSON bundle (prompts, agents, skills, tools, MCP servers; secrets stripped) and imports one or more at a time, in the order shown, reporting each bundle's result separately.