Operating guardrails

A guardrail has one failure mode that matters more than the rest: it can stop working without stopping. A missing model file does not crash the gateway, does not fail a health check, and does not change any response your users see. It just quietly passes traffic.

This page is about noticing that.

Fail-open and fail-closed

guardrailFailureMode sets the deployment's posture when a scanner cannot do its job — the model file is absent, or an inference call fails.

	fail_open (default)	fail_closed
A degraded scanner	Returns clean. Traffic passes unguarded	Returns a violation. Traffic is blocked
Model warmup at startup	Lazy — models load on first use	Forced, even if LEVOAI_MODEL_WARMUP_ON_STARTUP=false
Failure looks like	A working gateway	An outage
Right for	Availability-first deployments	Compliance-driven deployments where unguarded traffic is worse than no traffic

Set it in the config file or by environment variable:

config:
  guardrailFailureMode: fail_closed

LEVOAI_GUARDRAIL_FAILURE_MODE=fail_closed

Under fail_closed the gateway ignores a request to skip warmup and logs why:

ignoring LEVOAI_MODEL_WARMUP_ON_STARTUP=false because guardrailFailureMode is
fail_closed. Set guardrailFailureMode=fail_open if you want lazy model loading.

This is deliberate. Fail-closed promises that a degraded scanner blocks; lazy loading would defer discovering the degradation until the first production request, which is the worst possible moment to find out.

Verify fail-closed actually took effect on your guards

On gateway 0.13.10, in a static-YAML deployment, setting guardrailFailureMode: fail_closed at the top level was observed not to reach the scanners configured under config.policies.inspection. The startup log reports the posture correctly, GET /status/guardrails reports "failureMode": "fail_closed" — and a request with the model absent is still allowed through, with the gateway logging:

ML scanner degraded - model resources unavailable; applying failure policy
  scanner="PromptInjection" mode=FailOpen
LLM Bastion scanner degraded; failing open (traffic passed unguarded)

Setting the posture as a per-scanner parameter works, and is the reliable form today:

- type: llm_bastion
  action: block
  config:
    scanners:
      - scannerType: PromptInjection
        params:
          threshold: 0.92
          on_failure: fail_closed   # <-- this one is honoured

With that in place, the same request with the model absent returns HTTP 403 and the log reports mode=FailClosed.

Confirm your own deployment rather than trusting the setting: remove or rename a model directory in a staging gateway, send a benign request, and check that you get a 403.

Per-scanner failure policy

The on_failure parameter accepts fail_open or fail_closed, and can be split by cause:

Parameter	Covers
on_failure	Both cases below
on_model_unavailable	The model never loaded
on_inference_error	A single inference call failed

A per-cause parameter overrides on_failure. An unrecognized value is a hard configuration error rather than a silent default — a scanner that cannot parse how it should fail is not trusted to decide.

A blocking scanner whose configuration is invalid always fails closed, regardless of on_failure. The operator asked for enforcement with a configuration that cannot be honoured, so traffic is blocked rather than silently unguarded.

ML models

Which scanners need one

Everything in the rule-based tier needs no model. The rest map onto twelve ONNX models, several of them shared between scanners:

Model	Size	Backs
language	1.1 GB	Language, LanguageSame
prompt-injection	713 MB	PromptInjection
factual-consistency	713 MB	FactualConsistency
pii-ner	711 MB	Anonymize, Sensitive
toxicity	478 MB	Toxicity
ban-topics	478 MB	BanTopics
emotion	478 MB	EmotionDetection
malicious-urls	478 MB	MaliciousURLs
relevance	417 MB	Relevance
bias	316 MB	Bias
no-refusal	316 MB	NoRefusal
gibberish	257 MB	Gibberish

Roughly 6 GB on disk for the full set. Each model is pinned to a specific upstream revision and its weights are checksummed, so two gateways on the same image score identically.

Installing them

The models ship as their own container image. Populate a volume once:

docker volume create levoai-models
docker pull levoai/ai-guardrails-models:latest
docker run --rm -v levoai-models:/opt/models \
  levoai/ai-guardrails-models:latest \
  cp -r /models/. /opt/models/

Then mount it read-only and point the gateway at it:

docker run -d --name levoai-aigateway \
  -e LEVOAI_MODELS_BASE_PATH="/opt/models" \
  -v levoai-models:/opt/models:ro \
  ... levoai/ai-gateway:latest

Full platform instructions are in Install via Docker and Install on Kubernetes. On Kubernetes the volume is local to the node, so pre-populate it on every node that can schedule the pod, or use shared storage.

Warmup

With models mounted, the gateway loads them in the background after the listener binds and marks itself ready only once they are in memory. Expect 1–2 minutes before the first request is served. Warmup is non-blocking, so it never stalls a liveness probe.

Without models mounted, warmup completes in milliseconds and logs what is missing:

ML model cache warmup complete  scanner_count=13 loaded=0 missing=13
warmup: 13 model(s) missing — corresponding scanners will pass-through at scan
time. Check that the container has all model files under /opt/models/

will pass-through at scan time is the sentence to alert on. Under the default fail-open posture, that is the whole warning you get.

Sizing

The install prerequisites give the figures to provision against. The difference the ML tier makes:

	Rule-based only	With ML models
Provision	2 cores, 1 GB RAM	4 cores, 8 GB RAM
Disk	Negligible	~15 GB (6.3 GB of models, plus images and headroom)
Startup	Seconds	1–2 minutes of model warmup
Added latency	Sub-millisecond	Tens of milliseconds per ML scanner

The Helm chart ships requests: 500m / 4Gi and limits: 1000m / 8Gi. The 8 GB limit is headroom rather than a working-set target — it exists because smaller limits produced OOM kills on large prompts, and the working set scales with prompt size and pool size, not with idle load.

Rule-based-only is dramatically cheaper than the provisioning figure suggests: a gateway running the injection, secrets and PII guards with no models mounted measured 26 MB resident idle and 45 MB after 200 requests. Provision the 1 GB anyway for burst headroom, but do not size a node as though guardrails cost gigabytes when you are not running the ML tier.

Two knobs affect memory once ML is on:

• LEVOAI_ONNX_SESSION_POOL_SIZE — concurrent inferences per model. Each session holds its own copy of the weights, so memory is pool_size × model size, per model. It defaults to the core count clamped to [1, 4], which is 1 on a single-core pod. Raise it only together with the memory limit.
• maxScanBytes — caps the prefix of a body handed to the scanners. PII and credentials are nearly always in the first few KB. Masking scanners still re-run over the whole body so redaction stays complete.

Health and status

GET /status/guardrails

The operator view of whether guardrails are actually working. It lives on the admin listener — port 15000, bound to localhost by default. Set ADMIN_ADDR=0.0.0.0:15000 to reach it from outside the container, and treat it as an internal endpoint.

curl -s http://localhost:15000/status/guardrails

{
  "failureMode": "fail_open",
  "degraded": true,
  "ready": false,
  "warmupStatus": "complete",
  "unavailableModels": [
    "BanTopics", "Bias", "EmotionDetection", "FactualConsistency",
    "Gibberish", "Language", "MaliciousURLs", "NoRefusal",
    "PII-NER", "PII-NER/id2label", "PromptInjection", "Relevance", "Toxicity"
  ],
  "models": [
    {
      "model": "BanTopics",
      "state": "unavailable",
      "detail": "model file not resolved",
      "hf_repo_id": "protectai/MoritzLaurer-roberta-base-zeroshot-v2.0-c-onnx",
      "hf_revision": "5225ffccff19b15c6e95f80d1880c1fda986efd4"
    }
  ]
}

Field	Meaning
failureMode	The resolved deployment-wide posture
degraded	True when any model is unavailable
ready	True when every model this build knows about has loaded
warmupStatus	not_started, in_progress, complete, or disabled
unavailableModels	The ones that did not load
models	Per-model state, with the pinned upstream repository and revision

It returns 503 when degraded is true, and also when state is unknown under fail_closed. Otherwise 200.

This is not a readiness probe

/status/guardrails is a monitoring surface. A 503 here is informational and never affects request routing. Kubernetes probes belong on the readiness listener (port 15021), which reports the gateway's ability to serve traffic — and returns 200 even when every guardrail model is missing.

That gap is exactly the point of this endpoint: a pod can be perfectly ready and completely unguarded. Alert on /status/guardrails.

Metrics

Prometheus metrics are on port 15020 at /metrics.

Metric	Labels	Meaning
guardrail_degradations	phase, kind, mode, action	A scanner could not do its job
guardrail_checks	phase, action	Guardrail evaluations by outcome
guardrail_alerts	result	Alert delivery: success, error, dropped, dedup

guardrail_degradations with mode="fail_open" is the "silently stopped guarding" signal. Every increment is one request that passed without the check you configured. It should be zero in a healthy deployment; alert on any non-zero rate, not on a threshold.

# Any degradation at all, on either leg.
sum(rate(guardrail_degradations_total[5m])) > 0

# Specifically: checks that were skipped and traffic allowed anyway.
sum(rate(guardrail_degradations_total{mode="fail_open"}[5m])) > 0

Which series your deployment emits depends on how it is configured

Not every guard surface increments every counter, so confirm before you write an alert on one:

curl -s http://localhost:15020/metrics | grep guardrail

• guardrail_checks is incremented only from the backend-level AI prompt-guard path (policies.ai.promptGuard on a route). A gateway-level config.policies.inspection deployment never increments it.
• guardrail_degradations is incremented from gateway-level inspection guards — but only when the policy engine was built with a metrics handle, which is the case for dashboard-managed policy. On 0.13.10, a gateway running purely from static YAML with no Levo platform connection emits no guardrail_* series at all.

So for an air-gapped static-YAML deployment, the audit log and /status/guardrails are your only degradation signals. For a dashboard-managed deployment, guardrail_degradations is available and is the better alert.

An operational checklist

• /status/guardrails returns 200, or its unavailableModels list is empty of models you actually use.
• Alert on guardrail_degradations where mode="fail_open", and confirm the series is being emitted at all.
• Alert on the string will pass-through at scan time in gateway logs.
• Alert on skipping in gateway logs — a policy that could not be translated.
• A synthetic canary sends a known-bad prompt on a schedule and asserts 403. This is the only check that proves the whole path end to end.
• Your fail-closed posture has been tested by removing a model in staging, not just configured.