Batches API

Submit large workloads that Relay processes asynchronously through the model router. Useful for bulk classification, summarisation or generation over many rows.

See Batch processing for how the processor works (priority, cron, crash recovery, dead-letter, webhooks).

#Submit a batch

POST /v1/batches

{
  "name": "Review sentiment — October",
  "model": "gpt-4o-mini",
  "requests": [
    { "custom_id": "r-1", "messages": [{ "role": "user", "content": "Classify: great product!" }] },
    { "custom_id": "r-2", "messages": [{ "role": "user", "content": "Classify: terrible service" }] }
  ],
  "priority": 5,
  "webhookUrl": "https://my-app/relay-webhook",
  "cron": null
}

Field	Meaning
`model`	Public alias every row runs on. Required.
`requests`	The rows: a non-empty array of `{ custom_id?, messages: [{ role, content }] }`. `custom_id` defaults to `row-1`, `row-2`, … and is how a result is matched back to its row, so keep it unique.
`name`	Optional; shown in the panel. Defaults to a timestamp.
`priority`	Higher runs first among queued jobs (default 0).
`cron`	Optional 5-field cron expression. The job becomes a recurring template with status `scheduled`, and each due tick spawns a queued run. An expression with the wrong field count is a `400`.
`webhookUrl`	Optional; Relay POSTs a completion summary when a run ends.

Response:

{ "id": "9f2c1b7e4a8d4e51b0c6d3f2a1e5c7b9", "object": "batch", "status": "queued", "total": 2, "next_run_at": null }

Ids are 32-character hex strings. A cron template answers with status: "scheduled" and its next_run_at.

#Check status

GET /v1/batches/{id}

{ "id": "9f2c…", "object": "batch", "status": "running", "model": "gpt-4o-mini",
  "total": 2, "completed": 1, "failed": 0, "created_at": "2026-10-04T09:12:44Z", "completed_at": null, "error": null }

status is one of queued, scheduled (a cron template), running, completed, failed, canceled.

#Get results

GET /v1/batches/{id}/results

{ "id": "9f2c…", "status": "completed", "data": [
  { "custom_id": "r-1", "content": "positive" },
  { "custom_id": "r-2", "error": "upstream_error: …" }
] }

Each row yields { custom_id, content } or { custom_id, error }. data is empty until the job reaches a terminal state — results are written when the job completes, fails or is cancelled, not row by row.

#Completion webhook

When a run ends (completed, failed or canceled), Relay POSTs this to webhookUrl, with a 10-second timeout and no retry:

{ "id": "9f2c…", "name": "Review sentiment — October", "status": "completed",
  "total": 2, "completed": 2, "failed": 0, "dead_letter": 0, "completed_at": "2026-10-04T09:20:01Z" }

#Notes

The processor picks the highest-priority queued job and runs its rows with bounded concurrency (Gateway:BatchConcurrency, default 4, clamped 1–32). Rows use the model router, retries and per-row fallback; they do not go through the response cache, moderation or a key's model allow-list (a job submitted through the API carries the key's workspace, not the key).
A job left running by a restart is requeued and its rows re-run; completed/failed counters are saved per chunk, results at the end. Rows that still fail after retries land in the job's dead-letter list rather than failing the whole job.
You can also create batches from the panel under Batch (one job per uploaded .csv — one prompt per line — or .jsonl file), and cancel a running one there.