SLIs and SLOs

Service Level Indicators (SLIs)

Check Latency

Typical values (from the reproducible benchmark, measured 2026-08-12):

Note: LLM tier latency is dominated by the external LLM inference time. The pipeline itself adds only ~15 µs of deterministic processing even for escalated calls (mapping + deterministic check + escalation); the vast majority of wall time is the LLM HTTP request.

Availability

Error Rate

False-ALLOW Rate (Dangerous Commands Incorrectly Allowed)

Measurement: Not observable from production metrics alone. Requires periodic corpus evaluation (running the labeled corpus through the pipeline) and fuzzer results.

False-DENY Rate (Safe Commands Incorrectly Denied)

Measurement: The escalate_count from GET /metrics is a proxy — it includes both legitimate escalates (ambiguous calls) and potential false-escalations. The false-DENY rate is primarily measured through the labeled corpus.

Service Level Objectives (SLOs)

Recommended Targets

Rationale for Zero False-ALLOW SLO

A single false-ALLOW means a known-dangerous command like rm -rf / or DROP TABLE prod_db was allowed through. This is a direct security bypass. The guardrail is designed so that ANY false-ALLOW is a bug that must be fixed before the next release. There is no acceptable false-ALLOW budget.

Rationale for Non-Zero False-DENY SLO

The guardrail uses heuristic matching (prefix/contains rules, path normalization, wrapper detection). Some legitimate commands may be incorrectly blocked. A 1% false-DENY rate means that out of 10,000 safe operations, up to 100 may be blocked. This is considered acceptable for a security-first guardrail, with the understanding that blocked safe operations can be overridden or escalated.

Measurement Methodology

Latency Measurement

Latency is measured in-process for every check and returned in the response (latencies / service_wall_us). The GET /metrics snapshot also carries server-wide percentiles (latency_p50 / p95 / p99 / p99_9_us). For per-request export, use a sampling rate of at least 1% of requests and send them to a metrics backend (Prometheus, StatsD, etc.).

Availability Measurement

Availability is measured by load balancer health checks against GET /health. The service returns {"status":"ok"} with HTTP 200 when healthy, and {"status":"degraded"} with HTTP 503 when its database connection or pipeline lock check fails. The response also carries uptime, active connections, and memory RSS.

False-ALLOW and False-DENY Measurement

False-ALLOW and false-DENY are not directly observable from production traffic because we cannot label production tool calls as “safe” or “dangerous” a priori. Instead, they are measured through:

1. Corpus evaluation: The labeled corpus is run through the pipeline after every change. The guardrail_measure binary produces a report.
2. Fuzzer results: The mutation fuzzer generates variants and checks for unexpected verdicts. CI gates block merges that introduce false-ALLOW.
3. User reports: Manual reports from developers (P2/P3 tickets) that the guardrail blocked a legitimate operation.
4. Periodic red-teaming: Semi-annual manual bypass sweeps.

Dashboard Setup

Suggested Prometheus Queries

Scrape GET /metrics/prometheus directly (the server exposes Prometheus text format), or export via a sidecar if you prefer. The following queries compose a basic dashboard.

Request rate:

rate(guardrail_checks_total[5m])

Verdict distribution:

rate(guardrail_checks_allow_total[5m])
rate(guardrail_checks_deny_total[5m])
rate(guardrail_checks_escalate_total[5m])

Average latency:

rate(guardrail_latency_sum_us[5m]) / rate(guardrail_checks_total[5m])

Escalate rate (fraction of checks escalating):

rate(guardrail_checks_escalate_total[5m]) / rate(guardrail_checks_total[5m])

Error rate (parse errors, 4xx/5xx responses):

rate(guardrail_errors_total[5m]) / rate(guardrail_checks_total[5m])

Suggested Grafana Panels

Panel: Check Volume and Verdicts — Time series bar chart showing rate(guardrail_checks_total[5m]) broken down by verdict (allow/deny/escalate). Use stacked bars.

Panel: Latency Percentiles — Time series line chart showing p50, p95, p99 latency. Use a logarithmic Y-axis. Add threshold lines at 500 µs, 1 ms, 10 ms.

Panel: Escalate Rate — Single stat showing current escalate rate as a percentage. Alert when >5% (may indicate LLM tier issues or configuration gaps).

Panel: Error Rate — Single stat showing current error rate as a percentage. Threshold: >0.1%.

Panel: Top Escalated Actions — Table showing the most common escalated actions, useful for tuning.

Panel: Health Check Status — Single stat showing current health check status. Alert on non-200.

Alert Thresholds

Recommended Alert Rules

Alert Fatigue Prevention

• P1 alerts page the on-call engineer immediately.
• P2 alerts notify the security team promptly.
• P3 alerts are logged for daily review.
• HighEscalateRate should be tuned after 2 weeks of baseline data is available.
• HighLatencyP99 may trigger during heavy LLM use if the LLM tier is enabled — this is expected and not actionable unless the deterministic tier also exceeds thresholds.

Error Budget Policy

Since the guardrail has a zero false-ALLOW SLO, the error budget model is simple:

• Consumed: ANY false-ALLOW in a release (detected by CI fuzzer or user report) consumes 100% of the budget for that release.
• Consequence: The release must be rolled back or hotfixed before it can proceed.
• Budget reset: After the fixed release is deployed and verified, the budget resets.

For false-DENY SLO (1% of safe operations):

• Budget: 1% of safe operations per release.
• Consumed: Each false-DENY report counts against the budget.
• Tracking: Maintained in a spreadsheet or issue tracker. Reset per release.
• Consequence: If the budget is exceeded by >10% (1.1% false-DENY rate) in a release, a P3 review is triggered to examine the cause.

SLO Burn Rate

Given the zero-tolerance false-ALLOW policy, burn rate alerts are not applicable for security issues. For latency: