Skip to main content

AI Firewall on Kubernetes

Setup​

Prerequisites​

Before installing the Levo AI Firewall on Kubernetes, ensure you have:

  • Kubernetes version >= v1.18.0
  • Helm v3 installed and configured
  • The Kubernetes cluster API endpoint is reachable from the machine running Helm
  • kubectl access to the cluster with cluster-admin permissions

1. Add the Levo Helm Repository​

helm repo add levoai https://charts.levo.ai && helm repo update levoai

2. Install the AI Firewall​

helm upgrade --install -n levoai --create-namespace \
  --set config.vigil.upstream.address="<Upstream LLM Host:Port>" \
  --set config.vigil.upstream.tls=true \
  --set config.vigil.upstream.sni="<Upstream LLM Host>" \
  --set levo.baseUrl="https://api.levo.ai" \
  --set-file levo.refreshToken=<path-to-token-file> \
  --set levo.orgId="<Organisation ID>" \
  --set levo.envName="<Environment Name>" \
  --set levo.satelliteUrl="<Satellite URL>" \
  levoai-firewall levoai/levoai-firewall

For example, to proxy traffic to OpenAI:

helm upgrade --install -n levoai --create-namespace \
  --set config.vigil.upstream.address="api.openai.com:443" \
  --set config.vigil.upstream.tls=true \
  --set config.vigil.upstream.sni="api.openai.com" \
  --set levo.baseUrl="https://api.levo.ai" \
  --set-file levo.refreshToken=<path-to-token-file> \
  --set levo.orgId="<Organisation ID>" \
  --set levo.envName="<Environment Name>" \
  --set levo.satelliteUrl="<Satellite URL>" \
  levoai-firewall levoai/levoai-firewall

For levo.satelliteUrl, use https://satellite.levo.ai (Levo-hosted) or your own on-premise satellite address.

3. Verify the Installation​

Wait a couple of minutes after installation, and check that the pod is running:

kubectl -n levoai get pods

If the AI Firewall is healthy, you should see output similar to the following:

NAME                                    READY   STATUS    RESTARTS   AGE
levoai-firewall-7c8b9f6d4-p9mnz      1/1     Running   0          2m

Verify the health endpoint:

kubectl -n levoai port-forward svc/levoai-firewall 8080:8080
curl http://localhost:8080/health

Please contact support@levo.ai if you notice health/connectivity related errors.

4. Configure the AI Firewall in the Levo Dashboard​

The AI Firewall polls the Levo platform every 60 seconds and automatically applies the latest configuration. Routing rules and guardrail policies are managed from the dashboard.

  • Login to Levo.ai.
  • Navigate to AI Firewalls → Configuration.
  • Paste your configuration YAML and click Save.

Below is an example configuration that routes traffic to OpenAI and blocks prompt injection:

routing:
  routes:
    - id: openai-chat
      match_criteria:
        paths:
          - pattern: /v1/chat/completions
            match_type: prefix
      upstream:
        address: api.openai.com:443
        tls: true
        sni: api.openai.com
      inspect: true
      priority: 10

promptGuard:
  request:
    - llmBastion:
        failFast: true
        scanners:
          - scannerType: PromptInjection
            enabled: true
            params:
              threshold: 0.92

The firewall picks up the new configuration within 60 seconds of saving.

5. Point Your Application at the Firewall​

Update your application to send LLM requests through the firewall instead of directly to the LLM provider. Replace the LLM provider's base URL with the firewall's service address.

kubectl -n levoai get svc levoai-firewall

For example, using the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://<FIREWALL-SERVICE-IP>:8080/v1",
    api_key="<Your OpenAI API Key>",
)

AI Firewall Lifecycle Management​

Upgrade AI Firewall​

helm repo update levoai

helm upgrade -n levoai \
  levoai-firewall levoai/levoai-firewall

Uninstall AI Firewall​

helm uninstall levoai-firewall -n levoai

Troubleshooting​

Check Pod Logs​

kubectl -n levoai logs -f deployment/levoai-firewall

Enable Debug Logging​

Add the following helm option to enable debug logging:

helm upgrade --install -n levoai --create-namespace \
  --set config.vigil.observability.log_level="debug" \
  levoai-firewall levoai/levoai-firewall

Need Help?​

For further assistance, please reach out to support@levo.ai.

Was this page helpful?