Relay docs

Batch processing

Batches run large workloads asynchronously through the model router. Submit rows once; Relay works through them in the background.

See the Batches API for the request and response shapes.

#The processor

BatchProcessorService is a background service that, on each poll (Gateway:BatchPollSeconds, default 10):

  1. Spawns due recurring templates — any batch with a cron schedule that's due creates a fresh queued run.
  2. Picks the highest-priority queued job and processes its rows with bounded concurrency (Gateway:BatchConcurrency, default 4, clamped 1–32).

Rows use the same routing, retry and per-row fallback as live calls, and each row is recorded as a generation in telemetry. They do not go through the response cache, moderation, idempotency or a key's model allow-list: a job submitted through the API carries the key's workspace, not the key itself. The workspace budget is checked once, when the job's model is resolved at start.

#Reliability features

  • Priority — higher-priority jobs run first.
  • Crash recovery — a job left running by a restart is requeued and picked up again. Completed/failed counters are saved after every chunk; the per-row results are written when the job ends, so a requeued job re-runs its rows.
  • Dead-letter — a row that still fails after the retry cycle (whatever the cause) is recorded in the job's dead-letter list instead of failing the whole job. The job completes with the rest.
  • Webhooks — supply a webhookUrl and Relay POSTs a completion summary when the run ends (completed, failed or canceled).
  • Recurring (cron) — a template with a 5-field cron expression produces runs on schedule; the template's own status is scheduled.
  • Cancel — a running job can be cancelled from the panel; the processor stops at the next chunk boundary.

#Creating batches

  • API — POST /v1/batches with inline requests rows ({ custom_id?, messages }). Keep custom_id unique: it is how a result is matched to its row.
  • Panel — the Batch page accepts one or more .csv (one prompt per line) or .jsonl ({ custom_id?, messages } or { prompt } per line) files, queueing one job per file.

#Monitoring

GET /v1/batches/{id} returns status and counts (total / completed / failed); GET /v1/batches/{id}/results returns the per-row outputs once the job is in a terminal state. The panel shows the same, live, plus the dead-letter rows.

#Fallback

A row that fails transiently is retried against the model's fallback, if one is configured. This is evaluated per row, not per job — so a rate limit that clears part-way through lets later rows go back to the model you asked for, rather than committing the whole job to the fallback after the first failure. Each row's generation records which model served it and whether a fallback was used.