Skip to main content

AI Gateway via Docker

Prerequisites​

Before installing the Levo AI Gateway via Docker, ensure you have:

  • Met the system requirements listed in Install AI Gateway
  • Docker Engine 20.10.0 or higher
  • Admin privileges on the Docker host
  • Your Environment ID and Authorization Key from the steps in Install AI Gateway

1. Pull the Images​

docker pull levoai/ai-gateway:latest
docker pull levoai/ai-guardrails-models:latest

2. Run the AI Gateway​

Guardrails require ML models (~4 GB) stored on a local Docker volume. Create the volume and populate it with models — this runs once and takes a few minutes:

docker volume create levoai-models

docker run --rm \
  -v levoai-models:/opt/models \
  levoai/ai-guardrails-models:latest \
  cp -r /models/. /opt/models/

Then start the gateway with the models volume mounted:

docker run -d \
  --name levoai-aigateway \
  --restart unless-stopped \
  -p 4141:8080 \
  -e LEVOAI_BASE_URL="https://api.levo.ai" \
  -e LEVO_ENVIRONMENT_ID="<Environment ID>" \
  -e LEVOAI_AUTH_KEY="<Authorization Key>" \
  -e LEVOAI_ORG_ID="<Organisation ID>" \
  -e LEVOAI_ENV_NAME="<Environment Name>" \
  -e LEVOAI_SATELLITE_URL="<Satellite URL>" \
  -e MODELS_PATH="/opt/models" \
  -v levoai-models:/opt/models:ro \
  levoai/ai-gateway:latest

For LEVOAI_SATELLITE_URL, use https://satellite.levo.ai (Levo-hosted) or your own on-premise satellite address. For accounts on the India domain, set LEVOAI_BASE_URL to https://api.india-1.levo.ai.

3. Verify the Installation​

Check that the container is running:

docker ps -f name=levoai-aigateway

If the AI Gateway is healthy, you should see output similar to the following:

CONTAINER ID   IMAGE                       COMMAND                CREATED         STATUS         PORTS                    NAMES
a3f1c2b9d4e7   levoai/ai-gateway:latest   "/usr/local/bin/..."   1 minute ago    Up 1 minute    0.0.0.0:4141->8080/tcp   levoai-aigateway

Verify connectivity to Levo.ai by running:

docker logs levoai-aigateway | grep -Ei "connected|polling|saas"

If connectivity is healthy, you should see output similar to the following:

INFO aigateway: Connected to Levo SaaS. Polling for configuration every 60s.

Please contact support@levo.ai if you notice health or connectivity errors.

4. Configure Routes in the Levo Dashboard​

The AI Gateway polls the Levo platform every 60 seconds and automatically applies the latest configuration. Routes, LLM backends, and guardrail policies are all managed from the dashboard.

Refer to step 5 in AI Gateway on Kubernetes for configuration instructions and an example YAML — the configuration format is identical for Docker and Kubernetes deployments.

5. Point Your Application at the Gateway​

Update your application to send LLM requests to the gateway instead of directly to the LLM provider. Replace the LLM provider's base URL with http://<Docker-host-IP>:4141/v1.

For example, if you are using the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://<Docker-host-IP>:4141/v1",
    api_key="<Your OpenAI API Key>",
)

AI Gateway Lifecycle Management​

Upgrade AI Gateway​

Pull the latest images:

docker pull levoai/ai-gateway:latest

If you are running with guardrails, also refresh the models:

docker pull levoai/ai-guardrails-models:latest

docker run --rm \
  -v levoai-models:/opt/models \
  levoai/ai-guardrails-models:latest \
  cp -r /models/. /opt/models/

Stop and remove the existing container, then re-run the docker run command from Step 2:

docker stop levoai-aigateway && docker rm levoai-aigateway

Stop AI Gateway​

docker stop levoai-aigateway

Uninstall AI Gateway​

docker stop levoai-aigateway && docker rm levoai-aigateway

If you created a models volume, remove it too:

docker volume rm levoai-models

Troubleshooting​

Container Fails to Start​

Check the container logs for errors:

docker logs levoai-aigateway

Common causes:

  • Invalid credentials: Verify your LEVO_ENVIRONMENT_ID and LEVOAI_AUTH_KEY are correct and not expired.
  • Network connectivity: Confirm the host can reach api.levo.ai (or api.india-1.levo.ai) on port 443.

Enable Debug Logging​

Stop and re-run the container with RUST_LOG=debug:

docker stop levoai-aigateway && docker rm levoai-aigateway

docker run -d \
  --name levoai-aigateway \
  --restart unless-stopped \
  -p 4141:8080 \
  -e LEVOAI_BASE_URL="https://api.levo.ai" \
  -e LEVO_ENVIRONMENT_ID="<Environment ID>" \
  -e LEVOAI_AUTH_KEY="<Authorization Key>" \
  -e RUST_LOG="debug" \
  levoai/ai-gateway:latest

Need Help?​

For further assistance, please reach out to support@levo.ai.

Was this page helpful?