Guardrino in front of any model server

How it fits

Guardrino is designed to sit between your apps and any model server with an OpenAI-compatible API. Your app changes its base URL to Guardrino; Guardrino’s policy file points upstream at the model server. The model server, its models and its GPUs stay exactly as they are.

# Before: your app talks to the model server
base_url = "http://<model-server>/v1"

# After: your app talks to Guardrino, and Guardrino talks to the model server
base_url = "http://guardrino.internal:8080/v1"     # in your app
url      = "http://<model-server>/v1"              # [upstream] in guardrino.toml

Keep the model server reachable from Guardrino only, so every request goes through the policy.

Docker Compose

Two services on one GPU machine: vLLM serves the model, Guardrino is the only port your apps see.

services:
  vllm:
    image: vllm/vllm-openai:latest
    command: ["--model", "<your-model>"]
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    # No published port: only Guardrino reaches the model server.

  guardrino:
    image: guardrino:<version>          # loaded from the offline package (docker load -i ...)
    depends_on: [vllm]
    ports:
      - "8080:8080"                     # the only port your apps see
    volumes:
      - ./guardrino:/etc/guardrino      # guardrino.toml lives here
    env_file: ./guardrino/guardrino.env

# ./guardrino/guardrino.toml
mode = "observe"

[upstream]
url = "http://vllm:8000/v1"             # the vllm service, by its Compose name

docker compose up -d

vLLM on a server

Without containers: vLLM’s OpenAI-compatible server listens on port 8000 by default, under /v1. Bind it to an address Guardrino can reach and point upstream at it.

# On the model server
vllm serve <your-model> --host <vllm-host> --port 8000

# guardrino.toml on the Guardrino server
[upstream]
url = "http://<vllm-host>:8000/v1"

Installing Guardrino itself on Linux or Windows

Kubernetes

A Helm chart is planned. The offline package carries the chart and the images, so an air-gapped cluster installs from your internal registry. The policy goes in as a ConfigMap and the model-server key comes from a Secret you already manage.

# The offline package carries the chart and the images: push the images to your internal registry first.
helm install guardrino ./guardrino-<version>.tgz -n guardrino --create-namespace -f values.yaml

image:
  repository: <your-registry>/guardrino
  tag: "<version>"

replicaCount: 2

resources:
  requests:
    cpu: "4"
    memory: 8Gi
  limits:
    memory: 16Gi

# Rendered into a ConfigMap as guardrino.toml
policy: |
  mode = "observe"

  [upstream]
  url = "http://vllm.models.svc.cluster.local:8000/v1"

# The provider or model-server key, from a Secret you already have (leave out if the server needs none)
upstreamKeySecret:
  name: upstream-credentials
  key: api-key

service:
  port: 8080

The requests and limits match the planned requirements: 4 vCPU and 8 GB to start, up to 16 GB.

Red Hat OpenShift and OpenShift AI

OpenShift AI’s single-model serving platform is based on KServe and includes the vLLM ServingRuntime for KServe, whose models answer on OpenAI-compatible endpoints. Point upstream at the model’s internal endpoint, shown in the OpenShift AI dashboard.

# In values.yaml, policy: the model's internal endpoint from the OpenShift AI dashboard
[upstream]
url = "https://<model-name>-predictor.<model-project>.svc.cluster.local:8443/v1"

Install the same Helm chart in its own project and give your apps an OpenShift Route to it.

oc new-project guardrino
helm install guardrino ./guardrino-<version>.tgz -n guardrino -f values.yaml
oc expose service guardrino            # an OpenShift Route for your apps

Guardrino is designed to run as a non-root user under OpenShift’s default restricted-v2 security context constraint. That is the design intent, not a certification.

Other model servers and providers

Guardrino is designed to work with any OpenAI-compatible server, for example:

Ports are each project’s documented default; use the address your server actually listens on.

Hosted providers

Sidecar

Teams that want Guardrino right next to one app can run it in the same pod; the app talks to it on localhost.

containers:
  - name: app
    image: <your-app-image>
    env:
      - name: OPENAI_BASE_URL           # read by OpenAI's Python SDK
        value: http://localhost:8080/v1
  - name: guardrino
    image: <your-registry>/guardrino:<version>
    ports:
      - containerPort: 8080
    volumeMounts:
      - name: guardrino-policy
        mountPath: /etc/guardrino

Product names belong to their owners. They are named here only to show where Guardrino fits; no partnership, certification or endorsement is implied.