AI Gateway via Docker

This guide covers both standalone Docker (single host) and Docker Swarm (orchestrated cluster). Pick your deployment mode below; the selection persists across every step on this page.

Standalone Docker

Run the AI Gateway as a single container on one host. Best for evaluation, a single-node lab, or a small edge deployment.

Docker Swarm

Run the AI Gateway as a service in a Swarm-orchestrated cluster. Best for multi-node production deployments and for co-locating with other Swarm-managed services (for example, your LLM proxy) on a shared overlay network.

Setup

Prerequisites

Before installing the Levo AI Gateway, ensure you have:

• Met the system requirements listed in Install AI Gateway
• Docker Engine 20.10.0 or higher
• Admin privileges on the Docker host
• Your Environment ID and Authorization Key from the steps in Install AI Gateway

Standalone Docker

Nothing additional — a single Docker host is all you need.

Docker Swarm

Additional Swarm prerequisites:

• Docker Engine on every node that could host the service
• A Swarm cluster initialised, or admin privileges on a host that can become one

1. Pull the Images

With Guardrails

docker pull levoai/ai-gateway:latest
docker pull levoai/ai-guardrails-models:latest

Without Guardrails

docker pull levoai/ai-gateway:latest

2. Prepare the Environment

Standalone Docker

No additional preparation is required — skip to step 3.

Docker Swarm

Initialise Swarm (skip if the host is already a manager) and create an attachable overlay so other stacks — for example, an LLM proxy — can join it and be reached by the gateway using their service name:

docker swarm init
docker network create --driver overlay --attachable levoai-net

Export your credentials in the shell so the stack file can reference them at deploy time:

export LEVOAI_AUTH_KEY="<Authorization Key>"
export LEVO_ENVIRONMENT_ID="<Environment ID>"

3. Populate the Models Volume (Guardrails Only)

Skip this step if you are installing without guardrails.

Guardrails require ML models (~4 GB) stored on a local Docker volume. Create the volume and populate it — this runs once and takes a few minutes:

docker volume create levoai-models

docker run --rm \
  -v levoai-models:/opt/models \
  levoai/ai-guardrails-models:latest \
  cp -r /models/. /opt/models/

Standalone Docker

The volume lives on this host only.

Docker Swarm

note

The levoai-models volume is local to the node it is created on. For multi-node Swarms, either pre-populate the volume on every node that could schedule the service, or use a shared-storage volume driver (NFS, Azure Files, EFS). Alternatively, pin the service to a single node with a placement constraint if you prefer the simpler local-volume path.

4. Run the AI Gateway

Standalone Docker

With Guardrails

docker run -d \
  --name levoai-aigateway \
  --restart unless-stopped \
  -p 4141:8080 \
  -e LEVOAI_BASE_URL="https://api.levo.ai" \
  -e LEVO_ENVIRONMENT_ID="<Environment ID>" \
  -e LEVOAI_AUTH_KEY="<Authorization Key>" \
  -e LEVOAI_ORG_ID="<Organisation ID>" \
  -e LEVOAI_ENV_NAME="<Environment Name>" \
  -e LEVOAI_SATELLITE_URL="<Satellite URL>" \
  -e MODELS_PATH="/opt/models" \
  -v levoai-models:/opt/models:ro \
  levoai/ai-gateway:latest

Without Guardrails

docker run -d \
  --name levoai-aigateway \
  --restart unless-stopped \
  -p 4141:8080 \
  -e LEVOAI_BASE_URL="https://api.levo.ai" \
  -e LEVO_ENVIRONMENT_ID="<Environment ID>" \
  -e LEVOAI_AUTH_KEY="<Authorization Key>" \
  -e LEVOAI_ORG_ID="<Organisation ID>" \
  -e LEVOAI_ENV_NAME="<Environment Name>" \
  -e LEVOAI_SATELLITE_URL="<Satellite URL>" \
  levoai/ai-gateway:latest

Docker Swarm

Create levoai-aigateway-stack.yml:

With Guardrails

version: "3.8"

services:
  levoai-aigateway:
    image: levoai/ai-gateway:latest
    environment:
      LEVOAI_BASE_URL: "https://api.levo.ai"
      LEVO_ENVIRONMENT_ID: "${LEVO_ENVIRONMENT_ID}"
      LEVOAI_AUTH_KEY: "${LEVOAI_AUTH_KEY}"
      LEVOAI_ORG_ID: "<Organisation ID>"
      LEVOAI_ENV_NAME: "<Environment Name>"
      LEVOAI_SATELLITE_URL: "<Satellite URL>"
      MODELS_PATH: "/opt/models"
    volumes:
      - levoai-models:/opt/models:ro
    ports:
      - "4141:8080"
    networks:
      - levoai-net
    deploy:
      replicas: 1
      restart_policy: { condition: on-failure }

volumes:
  levoai-models:
    external: true

networks:
  levoai-net:
    external: true

Without Guardrails

version: "3.8"

services:
  levoai-aigateway:
    image: levoai/ai-gateway:latest
    environment:
      LEVOAI_BASE_URL: "https://api.levo.ai"
      LEVO_ENVIRONMENT_ID: "${LEVO_ENVIRONMENT_ID}"
      LEVOAI_AUTH_KEY: "${LEVOAI_AUTH_KEY}"
      LEVOAI_ORG_ID: "<Organisation ID>"
      LEVOAI_ENV_NAME: "<Environment Name>"
      LEVOAI_SATELLITE_URL: "<Satellite URL>"
    ports:
      - "4141:8080"
    networks:
      - levoai-net
    deploy:
      replicas: 1
      restart_policy: { condition: on-failure }

networks:
  levoai-net:
    external: true

Deploy the stack:

docker stack deploy -c levoai-aigateway-stack.yml levoai

For LEVOAI_SATELLITE_URL, use https://satellite.levo.ai (Levo-hosted) or your own on-premise satellite address. For accounts on the India domain, set LEVOAI_BASE_URL to https://api.india-1.levo.ai.

5. Verify the Installation

Standalone Docker

Check that the container is running:

docker ps -f name=levoai-aigateway

If the AI Gateway is healthy, you should see output similar to the following:

CONTAINER ID   IMAGE                       COMMAND                CREATED         STATUS         PORTS                    NAMES
a3f1c2b9d4e7   levoai/ai-gateway:latest   "/usr/local/bin/..."   1 minute ago    Up 1 minute    0.0.0.0:4141->8080/tcp   levoai-aigateway

Verify connectivity to Levo.ai by checking the logs:

docker logs levoai-aigateway | grep -iE "saas|policies"

Docker Swarm

Check the service is running:

docker stack services levoai

If the AI Gateway is healthy, you should see output similar to the following:

ID             NAME                       MODE         REPLICAS   IMAGE                       PORTS
a3f1c2b9d4e7   levoai_levoai-aigateway    replicated   1/1        levoai/ai-gateway:latest    *:4141->8080/tcp

Verify connectivity to Levo.ai by checking the service logs:

docker service logs levoai_levoai-aigateway | grep -iE "saas|policies"

If connectivity is healthy, you should see output similar to the following:

connecting to Levo SaaS platform; gateway will process all inbound requests for this environment
saas control plane started
policies control plane started

note

With guardrails enabled, the container may show 0/1 replicas (Swarm) or a non-ready health status (standalone) for the first 1–2 minutes while it warms up the ML model cache. This is expected — the gateway marks itself ready only after all scanner models are loaded.

Please contact support@levo.ai if you notice health or connectivity errors.

6. Configure Routes in the Levo Dashboard

The AI Gateway polls the Levo platform every 60 seconds and automatically applies the latest configuration. Routes, LLM backends, and guardrail policies are all managed from the dashboard.

Refer to step 5 in AI Gateway on Kubernetes for configuration instructions and an example YAML — the configuration format is identical for Docker and Kubernetes deployments.

7. Point Your Application at the Gateway

Update your application to send LLM requests to the gateway instead of directly to the LLM provider.

Standalone Docker

Replace the LLM provider's base URL with http://<Docker-host-IP>:4141/v1.

Docker Swarm

Replace the LLM provider's base URL with http://<swarm-ingress-IP>:4141/v1. Any Swarm node's IP works — the routing mesh forwards to the active replica.

For example, if you are using the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://<gateway-address>:4141/v1",
    api_key="<Your OpenAI API Key>",
)

---

AI Gateway Lifecycle Management

Upgrade AI Gateway

Standalone Docker

Pull the latest image:

docker pull levoai/ai-gateway:latest

If you are running with guardrails, also refresh the models:

docker pull levoai/ai-guardrails-models:latest

docker run --rm \
  -v levoai-models:/opt/models \
  levoai/ai-guardrails-models:latest \
  cp -r /models/. /opt/models/

Stop and remove the existing container, then re-run the docker run command from Step 4:

docker stop levoai-aigateway && docker rm levoai-aigateway

Docker Swarm

Pull the latest image on every Swarm node that could host the service, then update the service in place:

docker pull levoai/ai-gateway:latest

docker service update \
  --image levoai/ai-gateway:latest \
  --force \
  levoai_levoai-aigateway

If you are running with guardrails, refresh the models volume before updating the service:

docker pull levoai/ai-guardrails-models:latest

docker run --rm \
  -v levoai-models:/opt/models \
  levoai/ai-guardrails-models:latest \
  cp -r /models/. /opt/models/

Stop AI Gateway

Standalone Docker

docker stop levoai-aigateway

Docker Swarm

Scale the service to zero replicas — the stack, secret, and volume are preserved:

docker service scale levoai_levoai-aigateway=0

Bring it back with docker service scale levoai_levoai-aigateway=1.

Uninstall AI Gateway

Standalone Docker

docker stop levoai-aigateway && docker rm levoai-aigateway

If you created a models volume, remove it too:

docker volume rm levoai-models

Docker Swarm

docker stack rm levoai
docker network rm levoai-net

If you created a models volume, remove it too (per node):

docker volume rm levoai-models

---

Troubleshooting

Container or Service Fails to Start

Standalone Docker

Check the container logs for errors:

docker logs levoai-aigateway

Docker Swarm

Inspect the task's failure reason:

docker service ps levoai_levoai-aigateway --no-trunc
docker service logs levoai_levoai-aigateway

Common causes:

• Invalid credentials: Verify your LEVO_ENVIRONMENT_ID and LEVOAI_AUTH_KEY are correct and not expired.
• Network connectivity: Confirm the host can reach api.levo.ai (or api.india-1.levo.ai) on port 443.
• Missing models volume (guardrails only): the host or Swarm node running the container must have levoai-models populated. Repeat step 3 on that host.

Check Gateway Health

The readiness endpoint is on port 15021 inside the container and is not published externally:

Standalone Docker

docker exec levoai-aigateway curl -sf http://localhost:15021/healthz/ready

Docker Swarm

docker exec $(docker ps --filter name=levoai_levoai-aigateway --format '{{.Names}}' | head -1) \
  curl -sf http://localhost:15021/healthz/ready

A healthy gateway responds with HTTP 200.

Enable Debug Logging

Standalone Docker

Stop and re-run the container with RUST_LOG=debug:

docker stop levoai-aigateway && docker rm levoai-aigateway

docker run -d \
  --name levoai-aigateway \
  --restart unless-stopped \
  -p 4141:8080 \
  -e LEVOAI_BASE_URL="https://api.levo.ai" \
  -e LEVO_ENVIRONMENT_ID="<Environment ID>" \
  -e LEVOAI_AUTH_KEY="<Authorization Key>" \
  -e LEVOAI_ORG_ID="<Organisation ID>" \
  -e LEVOAI_ENV_NAME="<Environment Name>" \
  -e LEVOAI_SATELLITE_URL="<Satellite URL>" \
  -e RUST_LOG="debug" \
  levoai/ai-gateway:latest

Docker Swarm

Update the service to add RUST_LOG=debug:

docker service update \
  --env-add RUST_LOG=debug \
  levoai_levoai-aigateway

Revert with --env-rm RUST_LOG.

Need Help?

For further assistance, please reach out to support@levo.ai.