Streaming replies

How it works

With "stream": true, the model sends its reply in pieces over server-sent events (SSE). Guardrino reads the text as it accumulates and runs the output checks on it.

curl -N https://guardrino.example/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "your-model", "stream": true, "messages": [{"role": "user", "content": "Hello"}]}'

On a violation, the stream ends with an error event. Chunks the client has already received cannot be recalled.

data: {"choices":[{"delta":{"content":"The account number is "}}]}

data: {"error":{"message":"The request was blocked by Guardrino policy.","type":"guardrino_blocked","param":null,"code":"content_blocked","request_id":"req-42"}}

# the stream ends here: no "data: [DONE]" follows

Hold the reply until it is checked

When nothing may reach the user before it has been checked, set buffer_output = true. Guardrino then holds the reply, checks it whole, and sends it only if it passes. The cost is that the user sees nothing until the model has finished.

[proxy]
upstream = "https://llm.example/v1"
buffer_output = true       # nothing is sent until the whole reply has been checked
stream_check_chars = 256   # when false: check the reply every 256 new characters, and once more before [DONE]

Choose per deployment: the live feel of streaming, or the guarantee that a refused reply is never shown.