deveshThe problem
Incidents were being debugged by SSH-ing into boxes and grepping logs. There were dashboards — dozens of them — but no agreement on what "healthy" meant, no tracing across service boundaries, and alerts that paged for symptoms nobody could act on. Mean time to resolution was dominated not by fixing problems but by finding them.
The approach
SLOs before dashboards. For each critical user journey we defined a service level objective — availability and latency targets grounded in what users actually experience. Every dashboard and alert built afterwards existed to answer one question: are we meeting the SLO, and if not, why?
Metrics. Prometheus (via the kube-prometheus stack) for infrastructure and application metrics, with a small set of golden-signal dashboards in Grafana per service — latency, traffic, errors, saturation — replacing the graveyard of one-off panels.
Traces. OpenTelemetry SDKs instrumented the request path across services, so a slow checkout could be followed hop-by-hop instead of correlated by timestamp guesswork across six log files.
Alerting worth waking up for. Alerts were rebuilt on burn-rate rules against the SLOs: page only when the error budget is burning fast enough to threaten the target. Everything else became a ticket or a dashboard annotation.
The outcome
- MTTR down 65% — the time from page to root cause collapsed once traces
and golden-signal dashboards replaced log archaeology.
- Pager volume dropped alongside it, because alerts now mapped to user impact.
- Engineers self-serve: new services inherit instrumentation and dashboards
from templates instead of building from scratch.
- Leadership got an honest, numeric answer to "how reliable are we?" for the
first time.