Monitoring exists to implement the Feedback principle from The Three Ways of DevOps: catching problems as close to the moment they’re introduced as possible. Done well, it means an on-call engineer gets paged about a genuine issue before customers notice; done poorly, it means either critical problems slip through unnoticed or the team gets so many low-value alerts they start ignoring the channel entirely.

The Three Pillars of Observability

Modern monitoring is usually described as three complementary pillars. Metrics are numeric time-series data (CPU usage, request latency, error rate) that answer “what is happening right now, and how does it compare to normal.” Logs are detailed, timestamped event records that answer “what exactly happened, in what order, with what specific detail.” Traces follow a single request as it moves through multiple services, answering “where in this distributed system did the slowness or error actually occur.” Prometheus and Grafana commonly handle metrics, tools like the ELK stack or Loki handle logs, and Jaeger or Datadog APM handle tracing, though many modern platforms increasingly combine all three.

Fixing Alert Fatigue, the Single Biggest Monitoring Failure

Alert fatigue, being paged so often for low-value issues that real problems get ignored alongside the noise, is the most common monitoring failure we see. It’s usually caused by alerting on symptoms that don’t actually require immediate human action, setting thresholds too sensitively without tuning them against real baseline behavior, or failing to correlate related alerts into a single incident. Fixing it means auditing every existing alert against one question: if this fires at 3am, does a human genuinely need to wake up and act right now? Alerts that fail that test should become dashboards or daily digest emails, not pages.

A Real Incident Response Flow

Why Blameless Postmortems Are Non-Negotiable

Teams that assign blame during incident review create a strong incentive to hide mistakes or minimize reported severity, which directly undermines the honest information a postmortem needs to actually prevent recurrence. Blameless doesn’t mean consequence-free for genuinely reckless behavior, it means the default assumption is that engineers made reasonable decisions with the information available at the time, and the review focuses on what about the system allowed a reasonable decision to lead to an incident, rather than on punishing the individual involved.

Setting Meaningful SLOs Instead of Guessing at Thresholds

A more disciplined alternative to ad hoc alert thresholds is defining explicit Service Level Objectives (SLOs), for example “99.9% of requests complete in under 300ms,” backed by an error budget that quantifies how much unreliability is acceptable before it becomes a hard stop on new feature releases. This reframes monitoring from a scattered collection of individually-tuned alerts into a coherent system: you alert specifically when the error budget is burning down at a rate that threatens the SLO, not on every minor metric fluctuation in isolation.

This approach also gives teams a principled way to have the “should we ship this risky feature now or fix reliability first” conversation. When the error budget is healthy, teams have room to take on calculated risk; when it’s nearly exhausted, that’s an objective, pre-agreed signal to prioritize reliability work over new features, rather than a subjective argument decided fresh every time it comes up.

Frequently Asked Questions

Related reading: see The Three Ways of DevOps for the underlying Feedback principle, review our security best practices guide, or check AI in DevOps and AIOps.