Skip to main content

AI Gateway as a Kubernetes Sidecar

Setup​

Prerequisites​

Before installing the Levo AI Gateway sidecar injector, ensure you have your Environment ID and Authorization Key — refer to Install AI Gateway. You will also need:

  • Kubernetes 1.25 or higher
  • A TLS certificate and key for the webhook server
  • A CA certificate and key for TLS interception — the sidecar uses this CA to sign the certificates it presents to your application

1. Add the Levo Helm Repository​

helm repo add levoai https://charts.levo.ai && helm repo update levoai

2. Create the Required Secrets​

Create the levo-auth secret with your Levo credentials:

kubectl create secret generic levo-auth \
  -n levoai --create-namespace \
  --from-literal=base-url="https://api.levo.ai" \
  --from-literal=auth-token="<Authorization Key>" \
  --from-literal=tenant-id="<Environment ID>"

For accounts on the India domain, replace https://api.levo.ai with https://api.india-1.levo.ai.

Create the webhook TLS secret from your certificate and key. The certificate must cover the webhook service DNS name levo-injector-levoai-sidecar-injector-webhook.levoai.svc:

kubectl create secret tls levo-injector-levoai-sidecar-injector-webhook-tls \
  --cert=<path/to/webhook-cert.crt> \
  --key=<path/to/webhook-key.key> \
  -n levoai

Create the TLS interception CA secret:

kubectl create secret tls levo-ca-cert \
  --cert=<path/to/ca.crt> \
  --key=<path/to/ca.key> \
  -n levoai

3. Install the Sidecar Injector​

WEBHOOK_CA_BUNDLE=$(base64 -w0 < <path/to/webhook-ca.crt>)

helm upgrade --install levo-injector \
  -n levoai --create-namespace \
  --set tls.generateCA=false \
  --set webhook.existingTlsSecret=levo-injector-levoai-sidecar-injector-webhook-tls \
  --set "webhook.caBundle=${WEBHOOK_CA_BUNDLE}" \
  levoai/levoai-sidecar-injector

webhook.caBundle must be the base64-encoded CA certificate that signed your webhook server certificate.

4. Verify the Installation​

Wait a moment after installation, then check that the injector pod is running:

kubectl -n levoai get pods -l app.kubernetes.io/component=sidecar-injector

If the sidecar injector is healthy, you should see output similar to the following:

NAME                                                         READY   STATUS    RESTARTS   AGE
levo-injector-levoai-sidecar-injector-6d9b8c7f4-xk2np       1/1     Running   0          1m

Please contact support@levo.ai if you notice health or connectivity errors.

5. Enable Injection on a Namespace​

The sidecar injector watches namespaces labeled levo.ai/inject: enabled and automatically injects the AI Gateway sidecar into every new pod created in those namespaces.

Label the namespace where you want injection enabled:

kubectl label namespace <your-namespace> levo.ai/inject=enabled

Create the TLS interception CA secret in the target namespace so that injected sidecars can read it:

kubectl create secret tls levo-ca-cert \
  --cert=<path/to/ca.crt> \
  --key=<path/to/ca.key> \
  -n <your-namespace>

Repeat for each namespace where you want injection enabled.

6. Configure Routes in the Levo Dashboard​

The AI Gateway polls the Levo platform every 60 seconds and automatically applies the latest configuration. Routes, LLM backends, and guardrail policies are all managed from the dashboard.

  • Login to Levo.ai.
  • Navigate to AI Gateway → Configuration.
  • Paste your configuration YAML and save.

In sidecar mode, the gateway listens on port 15001 — the port that iptables redirects intercepted HTTPS traffic to. Below is an example configuration that routes traffic to OpenAI and blocks prompt injection attempts:

binds:
  - port: 15001
    listeners:
      - routes:
          - backends:
              - ai:
                  name: openai
                  provider:
                    openAI:
                      model: gpt-4o-mini
            policies:
              ai:
                promptGuard:
                  request:
                    - llmBastion:
                        failFast: true
                        scanners:
                          - scannerType: PromptInjection
                            enabled: true
                            params:
                              threshold: 0.92
                      rejection:
                        status: 400
                        body: |
                          {
                            "error": {
                              "message": "Request blocked by guardrails",
                              "type": "content_policy_violation"
                            }
                          }

The gateway picks up the new configuration within 60 seconds of saving.

7. Trust the CA in Your Application​

The sidecar presents TLS certificates signed by your CA to your application. Your application must trust this CA for outbound HTTPS connections to succeed.

For Java applications, the CA is injected into the JVM trust store automatically — no changes to your application or image are required.

For all other runtimes, add the CA certificate to your container image's trust store and rebuild:

# Debian / Ubuntu
COPY <ca.crt> /usr/local/share/ca-certificates/levo-ca.crt
RUN update-ca-certificates

# Alpine
COPY <ca.crt> /usr/local/share/ca-certificates/levo-ca.crt
RUN apk add --no-cache ca-certificates && update-ca-certificates

All traffic from your application to LLM providers is now observable in the Levo dashboard and subject to the guardrail policies you configured.

AI Gateway Sidecar Lifecycle Management​

Upgrade Sidecar Injector​

helm repo update levoai

helm upgrade levo-injector \
  -n levoai \
  --set tls.generateCA=false \
  --set webhook.existingTlsSecret=levo-injector-levoai-sidecar-injector-webhook-tls \
  --set "webhook.caBundle=${WEBHOOK_CA_BUNDLE}" \
  levoai/levoai-sidecar-injector

Restart your application deployments to pick up the updated sidecar:

kubectl rollout restart deployment/<your-deployment> -n <your-namespace>

Disable Injection on a Namespace​

kubectl label namespace <your-namespace> levo.ai/inject-

Restart the deployment to remove the sidecar from running pods.

Uninstall Sidecar Injector​

helm uninstall levo-injector -n levoai

Troubleshooting​

Sidecar Not Injected​

Check that the namespace has the injection label:

kubectl get namespace <your-namespace> --show-labels

Look for levo.ai/inject=enabled. Check the injector logs for admission webhook errors:

kubectl -n levoai logs deploy/levo-injector-levoai-sidecar-injector

Pod Fails to Start After Injection​

Check the pod events and sidecar container logs:

kubectl describe pod <pod-name> -n <your-namespace>

kubectl -n <your-namespace> logs <pod-name> -c levo-gateway

Common causes:

  • Missing levo-ca-cert secret: Ensure the secret exists in the pod's namespace — refer to Step 5.
  • iptables permission denied: The levo-init init container requires NET_ADMIN. Verify your cluster's Pod Security Admission policy permits it.

Enable Debug Logging​

helm upgrade levo-injector \
  -n levoai \
  --set tls.generateCA=false \
  --set webhook.existingTlsSecret=levo-injector-levoai-sidecar-injector-webhook-tls \
  --set "webhook.caBundle=${WEBHOOK_CA_BUNDLE}" \
  --set extraEnv.RUST_LOG="debug" \
  levoai/levoai-sidecar-injector

Need Help?​

For further assistance, please reach out to support@levo.ai.

Was this page helpful?