Chat completions

POST /v1/chat/completions — the primary endpoint. OpenAI-compatible, so most SDKs work by changing only the base URL and key.

#Request

{
  "model": "gpt-4o",
  "messages": [
    { "role": "system", "content": "You are concise." },
    { "role": "user", "content": "Summarize the benefits of an API gateway." }
  ],
  "temperature": 0.3,
  "max_tokens": 400,
  "stream": false
}

Attribution headers (X-Relay-User-Id, X-Relay-Session-Id, X-Relay-Tags, …) work on this endpoint; see Observability. The OpenAI user field is read as the end user, and a metadata object is read for attribution and then removed, because OpenAI rejects it.

#Response

{
  "id": "chatcmpl-…",
  "choices": [{ "index": 0, "message": { "role": "assistant", "content": "…" }, "finish_reason": "stop" }],
  "usage": { "prompt_tokens": 42, "completion_tokens": 88, "total_tokens": 130 }
}

Response headers include X-Relay-Request-Id, X-Relay-Trace-Id, X-Relay-Observation-Id and, on non-streaming calls only, X-Relay-Cost-Usd (the usage is not known when a stream's headers go out). The full header list is on the API overview.

#Streaming

curl -N http://localhost:5300/v1/chat/completions \
  -H "Authorization: Bearer $RELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "gpt-4o", "stream": true, "messages": [{ "role": "user", "content": "Write a haiku." }] }'

You receive data: {chunk} lines (each an OpenAI-shaped delta) ending with data: [DONE].

Every chunk is forwarded, including ones with no text: a tool-call fragment or the final finish_reason carries none. The trailing usage-only chunk ("choices": []) is sent only when you asked for it with stream_options — Relay always requests it from the provider so that streamed calls are costed exactly, and hides it unless you did.

If the provider fails after output has begun, the stream ends with a data: {"error": {"message", "type", "code"}} line and no [DONE]. Output already sent cannot be taken back, so a mid-stream failure is not retried or sent to a fallback model; a failure before the first token is.

#Images and content parts

content may be an array of parts instead of a string, exactly as in OpenAI's API. Relay keeps the shape: a string stays a string and parts stay parts on the way to an OpenAI-compatible provider.

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What is in this screenshot?" },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KGgo…", "detail": "auto" } }
  ]
}

Each image is limited to Gateway:MaxImageBytes decoded bytes (default 5 MB); a larger one is refused before anything is routed or billed, with 400, type invalid_request_error, code image_too_large. An image_url that is neither a data: URL nor an http(s) URL is refused with code invalid_image_url. Check capabilities.vision on GET /v1/models before sending images to a model.

A PDF can be sent the same way as a file part — { "type": "file", "file": { "file_data": "data:application/pdf;base64,…" } } — which Anthropic receives as a document block and Gemini as inlineData; OpenAI-compatible providers get it unchanged. Other part types (input_audio, …) are forwarded to OpenAI-compatible providers and dropped for Anthropic and Gemini. For documents a model should read rather than see, the file attachments path extracts the text once and works on every model.

Telemetry never stores image data: a captured body shows each image as a placeholder such as [image image/png 123KB]. Guardrails that rewrite text (PII redaction, prompt variables) rewrite only the text parts and keep the images.

#Tool calling

Send OpenAI tools, and the model answers with message.tool_calls and finish_reason: "tool_calls". Run the tool, then send the assistant message back followed by one role: "tool" message per call:

[
  { "role": "user", "content": "Weather in Paris?" },
  { "role": "assistant", "content": null, "tool_calls": [
    { "id": "call_1", "type": "function", "function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" } } ] },
  { "role": "tool", "tool_call_id": "call_1", "content": "{\"temp_c\":21}" }
]

This works the same on every provider family. For Anthropic models Relay translates tools to input_schema tools, tool_choice (auto / required → any / none / a named function → tool), parallel_tool_calls: false to disable_parallel_tool_use, assistant tool_calls to tool_use blocks, and consecutive tool messages to one user turn of tool_result blocks. For Gemini models it becomes functionDeclarations (JSON-schema keywords Gemini rejects, such as $schema and additionalProperties, are removed), functionCall and functionResponse parts, and Relay mints the tool-call ids Gemini does not have.

#Streamed tool calls

With stream: true, tool calls arrive as fragments in choices[0].delta.tool_calls. The first fragment for a call carries its index, id, type and function.name; later fragments carry the same index and a piece of function.arguments. Concatenate the arguments per index:

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"get_weather","arguments":""}}]}}]}
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\":"}}]}}]}
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"Paris\"}"}}]}}]}
data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}
data: [DONE]

Fields that are not set are omitted rather than sent as null. The same format comes back whatever the provider: Anthropic's tool_use / input_json_delta events and Gemini's whole functionCall parts are folded into it.

#Anthropic prompt caching and effort

A caller that knows it is talking to a Claude model can use two things the OpenAI shape has no field for. Both are opt-in, both are ignored by the OpenAI and Gemini adapters, and a fallback to a non-Anthropic model is unaffected.

Prompt caching. Put Anthropic's cache_control on a text part of the system message, or on a tool definition, and Relay forwards it as a cache breakpoint. Everything up to the breakpoint is written to the prompt cache on the first call and read back at the cached rate on the next ones, which is what makes a long system prompt and a large tool list cheap to send on every round of a tool loop.

{
  "model": "claude-sonnet",
  "messages": [
    { "role": "system", "content": [ { "type": "text", "text": "…a long system prompt…", "cache_control": { "type": "ephemeral" } } ] },
    { "role": "user", "content": "Hello" }
  ],
  "tools": [
    { "type": "function", "function": { "name": "first_tool", "parameters": { "type": "object" } } },
    { "type": "function", "function": { "name": "last_tool", "parameters": { "type": "object" } }, "cache_control": { "type": "ephemeral" } }
  ]
}

A system message sent as a plain string, or as parts without cache_control, is joined and sent exactly as before. Mark the last tool: a breakpoint caches everything before it, so one marker on the last tool covers the whole list. Usage reports what the cache did: prompt_tokens_details.cached_tokens were read from it, and prompt_tokens_details.cache_creation_tokens (a Relay addition) were written to it. Both are counted inside prompt_tokens, and cost prices the cached reads at the cached rate.

Effort. Send Claude's output_config as a top-level field and it is forwarded whole:

{ "model": "claude-opus", "output_config": { "effort": "medium" }, "messages": [ … ] }

Extended thinking is not carried through: a thinking block has to be replayed in later turns with its signature, and the OpenAI shape has nowhere to put it.

Web search. Claude's server tools cannot go in tools (only function tools translate), so send them in anthropic_tools. Types starting web_search_ or web_fetch_ are forwarded verbatim, after the function tools; anything else is skipped. Use web_search_20250305 on Azure Foundry (its Azure-hosted deployments have only the basic variant):

{ "model": "claude-sonnet", "anthropic_tools": [{ "type": "web_search_20250305", "name": "web_search", "max_uses": 3 }], "messages": [ … ] }

Anthropic runs the searches. On a non-streaming call the response gains relay_web_search: the queries, the pages found (results: url, title, page_age), the citations on the answer (url, title, cited_text), the answer (the text after the last search result, without the "let me search" lines before it), search errors, the billed search requests, and paused when the server loop stopped early (pause_turn, still reported as finish_reason: "stop"). message.content is unchanged: all text blocks, concatenated. A streamed call forwards only the text. Search requests are not yet included in X-Relay-Cost-Usd. A fallback to a non-Anthropic model drops anthropic_tools.

#What the gateway does for you

Behind this one call, Relay applies model routing, per-key model allow-list checks, budget hard-stops, transient retry, provider-health circuit breaking, fallback along the model's fallback chain, optional response caching, optional moderation of the request and (non-streamed) reply, and records a trace. See Model routing & fallback and Observability.

#Caching & idempotency

Enable the response cache with Gateway:Cache:TtlSeconds. Bypass per request with X-Relay-Cache: no (also false or bypass); the response says X-Relay-Cache: HIT or MISS. Provide an Idempotency-Key header to make a retried call safe: a repeat within 24 hours returns the stored response with Idempotency-Replayed: true and X-Relay-Cost-Usd: 0. Both apply to non-streaming calls only. See Caching & idempotency.