# How to Actually Set Meaningful SLOs (Most Teams Are Doing It Wrong) > Struggling with SLOs? Learn how to set meaningful Service Level Objectives that reflect real user impact. Avoid common mistakes, define better SLIs, and build effective SLO-based alerting. Source: https://openobserve.ai/blog/set-meaningful-slos/ Published: 2026-07-31 Authors: Simran Kumari Category: Engineering Tags: Alerts, Monitoring, Observability, SRE --- ## **TL;DR** An SLO is only meaningful if it's built from what users actually experience, not from infrastructure metrics like CPU or memory that can look perfectly healthy while every request fails. The three building blocks are the SLI (what you measure), the SLO (the target for that measurement), and the error budget (1 minus the SLO, the amount of unreliability you're allowed to spend). The single highest-leverage change most teams can make is switching from alerting on the final SLO breach to multi-window burn-rate alerting, which catches real outages fast without paging on noise. [OpenObserve](https://openobserve.ai) covers the practical side: SQL-based SLI queries, error budget dashboards, and scheduled burn-rate alerts, covered hands-on in [SLO-Based Alerting in OpenObserve](https://openobserve.ai/blog/slo-based-alerting/). ## **The Problem: SLOs on Paper vs. SLOs in Practice** Let's start with a confession most engineering teams won't make out loud: **the majority of SLOs in production today are theater.** Here's the scene. A manager reads a Google SRE blog post on a Sunday evening. Monday morning, they ask the team to "implement SLOs by end of sprint." By Friday, someone has wired up a Prometheus alert on CPU uptime and called it `availability: 99.9%`. The metric goes on the dashboard. The dashboard goes on the TV in the office. The TV never turns red. Meanwhile, users are seeing 10-second load times, checkout flows that silently fail, and API errors that return HTTP 200 with a JSON body that says `"status": "error"`. The infrastructure is healthy. The users are not. This is the central dysfunction: **teams confuse infrastructure health with user experience.** CPU uptime, memory usage, pod restarts, these are signals that *something might be wrong* with your systems. They are not signals that *something is definitely wrong for your users.* A server can have 15% CPU utilization and still be returning 500s to every request. Real SLOs are built from the user's perspective. They answer one question: *"Is this product doing what the user needs it to do, fast enough and reliably enough to keep their trust?"* Everything else is just a health check with a percentage attached. ## **SLA vs. SLO vs. SLI: What's the Difference?** These three terms get used interchangeably in casual conversation, which is exactly how teams end up tracking disk I/O and calling it a reliability target. They're related but distinct, and the difference matters: | Term | What it is | Who it's for | | --- | --- | --- | | **SLI** (Service Level Indicator) | The actual measurement: a ratio of good events to total events | Engineering, internal | | **SLO** (Service Level Objective) | The internal target for that SLI over a time window | Engineering, internal | | **SLA** (Service Level Agreement) | An external, often contractual promise to customers, usually with financial penalties or credits if missed | Customers, legal/business | The relationship is deliberate: your SLO should be stricter than your SLA. If your SLA promises customers 99.9% uptime, setting your internal SLO at 99.5% would mean you're already violating the customer contract by the time your own alerts fire. Setting the SLO tighter, say 99.95%, gives engineering a buffer to catch and fix problems before they ever become an SLA breach with financial consequences. For a shorter reference version of this same distinction, see the [SLO vs SLA vs SLI glossary entry](/glossary/slo-sla-sli/). ## **SLI, SLO and Error Budget: The Three Concepts** Before fixing the anti-patterns, the vocabulary needs to be tight. ### **What Is a Service Level Indicator (SLI)?** An SLI is **a quantitative measurement of a specific dimension of service quality**, expressed as a ratio of good events to total events. ``` SLI = (good events / total events) × 100 ``` "Good event" needs a crisp definition. For a REST API, a good event might be: *a request that returned a 2xx status code in under 500ms.* Every request that timed out, returned a 5xx, or took longer than 500ms is a bad event. The key word is **user-observable**. If the user cannot feel it, it probably should not be your primary SLI. ### **What Is a Service Level Objective (SLO)?** An SLO is **a target value for your SLI over a defined time window.** ``` SLO = SLI target over a rolling window ``` Example: *"99.5% of order placement requests return 2xx in under 500ms, measured over a rolling 30-day window."* The time window matters enormously. A 30-day rolling window is far more useful than a calendar-month window because it gives you a continuous, current picture of reliability, not one that resets on the 1st of the month and gives engineers a false sense of a clean slate. ### **What Is an Error Budget?** The error budget is **the allowable amount of unreliability derived from your SLO.** It's what you get to spend on deployments, experiments, and incidents before you've violated your reliability promise. ``` Error Budget = 1 - SLO target ``` If your SLO is 99.5%, your error budget is 0.5%. Over a 30-day window (43,200 minutes), that translates to **~216 minutes of allowable downtime**: roughly 3 hours and 36 minutes. The error budget is the forcing function that makes SLOs change behavior. When the budget is healthy, you ship fast and take risks. When it's draining, you slow down and focus on reliability. This is the tradeoff SLOs formalize: **reliability vs. velocity**, and without the error budget, you have no mechanism to enforce it. See the [error budget glossary entry](/glossary/what-is-error-budget/) for the short-form definition and a worked calculation example. ## **The 5 Anti-Patterns (And What to Do Instead)** ### **Anti-Pattern 1: Tracking Infrastructure Metrics Instead of User Signals** **What it looks like:** ``` # This is not an SLO - name: "CPU Availability" metric: avg(cpu_usage_idle) > 0.1 target: 99.9% ``` CPU idle percentage, memory utilization, disk I/O: these are *leading indicators* of potential problems, not measures of user experience. A system can have perfect infrastructure vitals and be silently failing every user request. **What to do instead:** Track HTTP success rate and latency at the application layer, from the user's perspective: * **Correctness SLI:** % of requests returning a 2xx status code * **Latency SLI:** % of requests completing under your defined threshold (e.g., 500ms) A useful starting filter is the [four golden signals](/glossary/four-golden-signals/): latency, traffic, errors, and saturation. Not all four make good SLIs for every journey, saturation especially is usually a cause, not a symptom, but they're a solid checklist for making sure you haven't skipped a user-facing dimension. If your SLI cannot be directly felt by a user making a request, it belongs in a runbook, not in an SLO. ### **Anti-Pattern 2: Setting SLOs Too Tight** **What it looks like:** A team sets 99.99% availability on their first SLO because it "sounds right" for a production system. 99.99% allows only **4.38 minutes of downtime per month.** Unless you have fully automated rollback, zero-downtime deployments, multi-region failover, and a battle-hardened incident process, you will blow this budget with a single botched deploy. And once the budget is gone, the SLO becomes a number everyone quietly ignores. Tight SLOs that are routinely missed train your team to dismiss SLOs entirely. **What to do instead:** Start by measuring your *current* reliability. Run your SLI query over the past 30 days and see what you're actually achieving. If you're at 98.7%, setting an SLO of 99.9% is aspirational fantasy. Set it at 99.2%, slightly tighter than current performance, to create accountability without setting the team up to fail. The SLO is not a marketing promise. It is an engineering contract. Set it at a level your team can defend, then tighten it as you improve. **Rule of thumb:** Start with `current_baseline - 0.3%` as your initial SLO, and revisit quarterly. ### **Anti-Pattern 3: No Burn Rate Alerting** **What it looks like:** Teams alert on the SLO breach itself: a PagerDuty notification only fires when reliability drops below the target. By the time that fires, the error budget may already be 80% consumed. **What to do instead:** Alert on **how fast you're consuming** the error budget, not on whether you've already breached the final target. A burn rate of 1.0 means you're on pace to exhaust the budget exactly at month-end. A burn rate of 14.0 means you'll exhaust it in under 2 days. This gives you early warning and proportional urgency: a page for fast burns (outages), a Slack alert for slow burns (quiet regressions). The full multi-window mechanics are covered later in this guide. ### **Anti-Pattern 4: SLOs Per Microservice Instead of User Journeys** **What it looks like:** ``` payment-service: availability 99.9% cart-service: availability 99.9% inventory-service: availability 99.9% user-service: availability 99.9% ``` This looks comprehensive. It is not. A user completing a checkout flow touches all four services. If each has 99.9% availability independently, the *joint* availability of the checkout flow is: ``` 0.999 × 0.999 × 0.999 × 0.999 ≈ 99.6% ``` You have four green dashboards and a user journey that's failing 1 in 250 times, and no single service owner sees it as their problem. **What to do instead:** Define SLOs for **user journeys**, not service instances. The checkout flow is one SLO. The authentication flow is one SLO. Search-and-discovery is one SLO. Map what users *do* in your system first. Then instrument it. The service topology is an implementation detail; the user journey is the contract. ### **Anti-Pattern 5: Never Reviewing Error Budgets** **What it looks like:** The error budget is calculated once, put on a dashboard, and never discussed again. Teams ship features. Incidents happen. The budget drains. Nobody changes behavior because nobody is looking. **What to do instead:** Run a **monthly SLO review**: a 30-minute ritual with a fixed agenda: 1. **Error budget status**: How much is left? What consumed the most? 2. **Incident debrief**: Which incidents hit SLOs? What's the remediation status? 3. **Velocity tradeoff**: Should we slow deploys this sprint to protect the budget? 4. **Accuracy check**: Is the SLO still measuring the right thing? Does the target need adjustment? Without this ritual, SLOs are numbers. With it, they become a decision-making tool. ## **How to Define Your First SLO** The methodology is the same regardless of your stack. Here's the thinking process using a concrete example: the `POST /orders` endpoint of a REST API. ### **Step 1: Identify the User Journey** Write this in plain English before touching any tooling. **Journey:** A registered user places an order. **Critical user interaction:** The user submits the order form and expects a confirmation promptly. The instrumentation follows the story, not the other way around. ### **Step 2: Define the SLI** A good SLI combines **correctness** (did it succeed?) and **speed** (was it fast enough?). **SLI:** *The percentage of `POST /orders` requests that return a 2xx HTTP status code within 500 milliseconds.* ``` SLI = (requests returning 2xx in < 500ms) / (total requests) × 100 ``` Both dimensions matter. A request that returns 200 after 8 seconds has failed the user even if it's technically "successful." ### **Step 3: Set the Target** Measure your baseline before committing to a number. Query your actual success rate over the past 30 days. Set the SLO slightly below that, creating accountability without demanding perfection. **Example target:** `99.5%` over a rolling 30-day window. ### **Step 4: Calculate the Error Budget** ``` Error Budget = 1 - 0.995 = 0.005 = 0.5% 30-day window = 43,200 minutes Allowable bad = 43,200 × 0.005 = 216 minutes (~3h 36m) ``` Write this number down and keep it visible. A 20-minute incident consumes ~9% of the monthly budget. A 2-hour incident consumes ~55%. When the budget is concrete, the cost of incidents becomes concrete too. ### **Step 5: Implement in OpenObserve** Once you have the SLI definition, target, and error budget worked out, the next step is getting it into your observability stack. This means writing the SQL query that evaluates the SLI, building the compliance dashboard, and setting up scheduled alerts that fire when the target is at risk. The OpenObserve blog has a complete walkthrough using a real-world login latency scenario that maps directly to this process: [**SLO-Based Alerting in OpenObserve →**](https://openobserve.ai/blog/slo-based-alerting/) ## **SLO and SLI Examples by Service Type** The `POST /orders` example above generalizes well, but the right SLI shape changes with what the service actually does. A few concrete starting points: | Service Type | Example SLI | Example SLO | | --- | --- | --- | | REST API endpoint | % of requests returning 2xx in under 500ms | 99.5% over 30 days | | Frontend / page load | % of page loads with Largest Contentful Paint under 2.5s | 95% over 30 days | | Async worker / queue consumer | % of queued jobs processed within 60 seconds of enqueue | 99% over 7 days | | Batch / ETL pipeline | % of scheduled runs that complete successfully before the downstream deadline | 99% over 30 days | | Search / query endpoint | % of queries returning results in under 1s with a non-error response | 99% over 30 days | None of these numbers are universal targets, they're starting shapes. The actual percentage still has to come from your own baseline (Step 3 above), and the time window should match how the service is actually consumed: a nightly batch job reviewed weekly doesn't need a 5-minute burn-rate alert the way a checkout API does. ## **What Is Burn Rate Alerting?** Burn rate is how fast you're consuming your error budget relative to your expected pace. ``` Burn Rate = (current error rate) / (1 - SLO target) ``` | Burn Rate | What it means | Budget exhausted in | | ----- | ----- | ----- | | 1× | On pace, normal | 30 days (end of window) | | 2× | Draining faster than expected | ~15 days | | 6× | Significant degradation | ~5 days | | 14× | Major incident / outage | ~2 days | ### **Why Single-Window Alerts Fail** If you alert on a 5-minute window alone, transient spikes will page your on-call for something that self-resolved 90 seconds later. If you alert only on a 6-hour window, a full outage won't fire for hours. The solution is **multi-window alerting**: alert only when burn rate is elevated in *both* a short and a long window simultaneously. The short window confirms it's real; the long window confirms it's sustained. | Window Pair | Catches | Urgency | | ----- | ----- | ----- | | 5m + 1h | Fast burn (outage) | Page immediately | | 30m + 6h | Medium burn (degradation) | Page within business hours | | 2h + 24h | Slow burn (subtle regression) | Slack / ticket | The two-window requirement is noise suppression by design. **A 2-minute spike should not wake someone at 3am.** A 2-hour sustained burn rate of 14× absolutely should. ### **Implementing Burn Rate Alerts in OpenObserve** OpenObserve's scheduled alerts, which evaluate SQL queries over configurable time windows, map directly to this multi-window model. The following guides cover the alert setup, query structure, and notification routing: [**SLO-Driven Monitoring: Build Better Alerts with OpenObserve →**](https://openobserve.ai/blog/slo-based-alerting/) [**Alerting 101: From Concept to Demo →**](https://openobserve.ai/blog/alerting-101-from-concept-to-demo/) ## **Visualizing SLOs in OpenObserve** An SLO that exists only in a config file is not an SLO: it's a query. For SLOs to actually change team behavior, they need to be visible constantly. ### **The Three Panels Every SLO Dashboard Needs** **1. Error Budget Remaining (Gauge)** Shows at a glance how much budget is left for the current 30-day window. Color-code it: green above 50%, yellow at 10–50%, red below 10%. This should be the first thing your team sees at standup, not the deploy count, not the ticket velocity. **2. Burn Rate Over Time (Time Series)** A line chart of current burn rate over the past 7 days. Overlay reference lines at 1× (expected pace), 2× (slow burn threshold), and 14× (fast burn / page threshold). At a glance, your team should be able to see whether the week was clean or problematic. **3. SLI Trend Over Time (Time Series)** Daily SLI values plotted over 30 days with a reference line at the SLO target. This tells you whether reliability is improving, degrading, or flat over time, which is the question the monthly review should be anchored to. ### **Building It in OpenObserve** For the dashboard build, panel creation, SQL queries, variable templating, and auto-refresh configuration, the OpenObserve documentation covers exactly this: [**Building Monitoring Dashboards with OpenObserve →**](https://openobserve.ai/blog/monitoring-dashboards-with-openobserve/) [**Observability Dashboards: How to Build Them and What to Show →**](https://openobserve.ai/blog/observability-dashboards/) ### **Making the Dashboard Matter** The one habit that turns an SLO dashboard from decoration into a decision-making tool: **make the error budget gauge the first screen shared at every standup.** When it's green, the conversation is five seconds: *"Budget's healthy, let's ship."* When it's yellow or red: *"What happened, who owns the fix, should we pause the release queue?"* That's the entire promise of SLOs: not a prettier dashboard, but a mechanism for making reliability a first-class engineering decision, visible and discussable, every single day. ## **SLO Best Practices Checklist** A quick recap of everything above, in the order you'll actually use it: * **Measure user-observable behavior**, not infrastructure metrics like CPU or memory * **Set targets from your real baseline**, not an arbitrary "nines" number that sounds impressive * **Define SLOs per user journey**, not per microservice, so compounding failures don't hide in green dashboards * **Alert on burn rate**, not just the final SLO breach, so you get proportional, early warning * **Use multi-window alerting** (a short window plus a long window) to suppress noise without missing real outages * **Put the error budget on a dashboard** that's visible constantly, not buried in a config file * **Review the error budget monthly** as a standing ritual, with a fixed agenda and a real velocity decision at the end ## **Conclusion** You'll know your SLOs are actually working when: * An engineer says *"we should hold this deploy, we're at 12% error budget for the month"* without being prompted. * A postmortem asks *"how much error budget did this incident consume?"* as a standard question. * The monthly SLO review leads to a concrete decision: tighten the target, or deprioritize a feature to pay down reliability debt. The tools are straightforward. The discipline is the hard part. *Further reading:* - *Google SRE Workbook* - [*OpenObserve SLO Alerting Guide*](https://openobserve.ai/blog/slo-based-alerting/) - [*Alert Fatigue: Why Teams Stop Trusting Alerts*](https://openobserve.ai/blog/alert-fatigue/) - [*How Many Nines Do You Actually Need?*](https://openobserve.ai/blog/how-many-nines-do-you-actually-need/) - [*SLO vs SLA vs SLI glossary entry*](https://openobserve.ai/glossary/slo-sla-sli/) - [*What Is an Error Budget? glossary entry*](https://openobserve.ai/glossary/what-is-error-budget/) - [*The Four Golden Signals glossary entry*](https://openobserve.ai/glossary/four-golden-signals/)