Skip to main content

AI Gateway on Kubernetes

Setup​

Prerequisites​

Before installing the Levo AI Gateway on Kubernetes, ensure you have your Environment ID and Authorization Key — refer to Install AI Gateway.

1. Add the Levo Helm Repository​

helm repo add levoai https://charts.levo.ai && helm repo update levoai

2. Install the AI Gateway​

helm upgrade --install -n levoai --create-namespace \
  --set config.saas.url="https://api.levo.ai" \
  --set config.saas.environmentId="<Environment ID>" \
  --set config.saas.refreshToken="<Authorization Key>" \
  --set config.levo.orgId="<Organisation ID>" \
  --set config.levo.envName="<Environment Name>" \
  --set config.levo.satelliteUrl="<Satellite URL>" \
  levoai-aigateway levoai/levoai-aigateway

For config.levo.satelliteUrl, use https://satellite.levo.ai (Levo-hosted) or your own on-premise satellite address. For accounts on the India domain, replace https://api.levo.ai with https://api.india-1.levo.ai.

note

The first installation downloads ML models (~4 GB) to a persistent volume. The pod will be in Init state for a few minutes until the download completes. To install without guardrails, add --set models.enabled=false.

3. Verify the Installation​

Wait a couple of minutes after installation, and check that the pod is running:

kubectl -n levoai get pods

If the AI Gateway is healthy, you should see output similar to the following:

NAME                                      READY   STATUS    RESTARTS   AGE
levoai-aigateway-6d9b8c7f4-xk2np         1/1     Running   0          2m

Verify connectivity to Levo.ai by running:

kubectl -n levoai logs deployment/levoai-aigateway | grep -Ei "connected|polling|saas"

If connectivity is healthy, you should see output similar to the following:

INFO aigateway: Connected to Levo SaaS. Polling for configuration every 60s.

Please contact support@levo.ai if you notice health or connectivity errors.

4. Note the Gateway Service Address​

The AI Gateway service is of type LoadBalancer on port 8080. Retrieve the external address:

kubectl -n levoai get svc levoai-aigateway

You should see output similar to the following:

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
levoai-aigateway    LoadBalancer   10.96.130.21    203.0.113.50    8080:31234/TCP   2m

Note the EXTERNAL-IP — your applications will send LLM traffic to this address instead of directly to the LLM provider.

5. Configure Routes in the Levo Dashboard​

The AI Gateway polls the Levo platform every 60 seconds and automatically applies the latest configuration. Routes, LLM backends, and guardrail policies are all managed from the dashboard.

  • Login to Levo.ai.
  • Navigate to AI Gateway → Configuration.
  • Paste your configuration YAML and save.

Below is an example configuration that routes traffic to OpenAI and blocks prompt injection attempts:

binds:
  - port: 8080
    listeners:
      - routes:
          - backends:
              - ai:
                  name: openai
                  provider:
                    openAI:
                      model: gpt-4o-mini
            policies:
              ai:
                promptGuard:
                  request:
                    - llmBastion:
                        failFast: true
                        scanners:
                          - scannerType: PromptInjection
                            enabled: true
                            params:
                              threshold: 0.92
                      rejection:
                        status: 400
                        body: |
                          {
                            "error": {
                              "message": "Request blocked by guardrails",
                              "type": "content_policy_violation"
                            }
                          }

The gateway picks up the new configuration within 60 seconds of saving.

6. Point Your Application at the Gateway​

Update your application to send LLM requests to the gateway instead of directly to the LLM provider. Replace the LLM provider's base URL with the gateway's external address.

For example, if you are using the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://<EXTERNAL-IP>:8080/v1",
    api_key="<Your OpenAI API Key>",
)

All traffic through the gateway is now observable in the Levo dashboard and subject to the guardrail policies you configured.

AI Gateway Lifecycle Management​

Upgrade AI Gateway​

helm repo update levoai

helm upgrade -n levoai \
  --set config.saas.url="https://api.levo.ai" \
  --set config.saas.environmentId="<Environment ID>" \
  --set config.saas.refreshToken="<Authorization Key>" \
  levoai-aigateway levoai/levoai-aigateway

Uninstall AI Gateway​

helm uninstall levoai-aigateway -n levoai

Troubleshooting​

Pod Fails to Start​

Check the pod events and logs for errors:

kubectl -n levoai describe pod -l app.kubernetes.io/name=levoai-aigateway

kubectl -n levoai logs deployment/levoai-aigateway

Common causes:

  • Invalid credentials: Verify your environmentId and refreshToken in the values file are correct and not expired.
  • Network connectivity: Confirm the cluster can reach api.levo.ai (or api.india-1.levo.ai) on port 443.

Enable Debug Logging​

helm upgrade -n levoai \
  --set config.saas.url="https://api.levo.ai" \
  --set config.saas.environmentId="<Environment ID>" \
  --set config.saas.refreshToken="<Authorization Key>" \
  --set extraEnv.RUST_LOG="debug" \
  levoai-aigateway levoai/levoai-aigateway

Need Help?​

For further assistance, please reach out to support@levo.ai.

Was this page helpful?