Monitoring and observability are related but distinct: monitoring tells you a system is unhealthy, a dashboard turning red, an alert firing. Observability is the deeper capability to actually understand why, based on the data your systems already produce, without needing to add new instrumentation specifically to answer today’s question. A well-monitored system tells you something’s wrong. A well-observed system lets you find the root cause in minutes instead of hours, by querying data that was already being collected before the incident happened.

This guide covers the three pillars observability is built from, real metric and alerting configuration, how to avoid the alert fatigue that undermines monitoring at most companies, and where to actually start if you’re building this from scratch. It also covers the practices that separate ad hoc monitoring from a mature reliability program: formal SLOs and error budgets, structured logging that’s actually searchable at scale, runbooks that turn incident response into a repeatable process rather than improvised panic, and the cost-control habits that keep observability tooling from becoming a surprisingly large line item as a system grows.

The Three Pillars of Observability

No single pillar is sufficient alone. Metrics tell you something is wrong and roughly where; logs tell you the specific detail of what happened; traces tell you how a single request moved through a distributed system and where it slowed down or failed. A mature observability setup uses all three together, often letting you jump directly from a metric spike to the specific traces and logs from that exact time window.

The Four Golden Signals

Google’s SRE book popularized four metrics worth monitoring for nearly any service, widely referred to as the golden signals: latency (how long requests take), traffic (how much demand the system is under), errors (the rate of failed requests), and saturation (how close a resource, like CPU or memory, is to its limit). Starting monitoring for any new service around these four signals, before adding anything more specialized, covers the majority of what you need to know something’s wrong.

Instrumenting an Application

Modern applications typically expose metrics in a standard format that a monitoring system scrapes and stores. Prometheus, one of the most widely used open-source monitoring systems, pulls metrics from an HTTP endpoint your application exposes.

const client = require('prom-client');
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests',
  labelNames: ['method', 'route', 'status']
});

app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  res.on('finish', () => {
    end({ method: req.method, route: req.route?.path, status: res.statusCode });
  });
  next();
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

This middleware records request duration for every request, labeled by method, route, and status code, exposed at a /metrics endpoint that Prometheus scrapes on a regular interval. From this single metric, you can derive request rate, error rate, and latency percentiles, covering three of the four golden signals from one piece of instrumentation.

Alerting Without Drowning in Noise

Alert fatigue, getting paged so often that alerts stop being meaningful signals and start being background noise everyone learns to ignore, is one of the most common and damaging monitoring failures. The fix isn’t fewer metrics, it’s more deliberate alerting rules: alert on symptoms that actually affect users (elevated error rate, high latency) rather than every possible underlying cause, and set thresholds based on what genuinely requires human intervention, not just any deviation from baseline.

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 5 minutes"

This Prometheus alert only fires if the error rate exceeds 5% and stays elevated for a full 5 minutes (for: 5m), avoiding a page for a brief, self-resolving blip that doesn’t actually need human intervention. Requiring sustained deviation before alerting, rather than firing on the first data point that crosses a threshold, is one of the simplest, highest-value changes to reduce false-positive pages.

Dashboards That Actually Get Used

Grafana is the most widely used tool for visualizing metrics from Prometheus and similar sources, but a dashboard with fifty panels crammed onto one screen gets used far less than a small, focused one built around the golden signals for a specific service. The most effective dashboards answer a specific question at a glance, “is this service healthy right now”, rather than trying to display everything that could theoretically be measured, which tends to produce dashboards nobody actually looks at during an incident because they’re too dense to parse quickly under pressure.

Distributed Tracing for Microservices

Once an application is split across multiple services, covered in the context of orchestration in our Kubernetes guide, a single user request might touch five or six different services before returning a response. Distributed tracing, using tools like Jaeger or OpenTelemetry, follows a single request across every service it touches, showing exactly which hop added the most latency, which is far faster than manually correlating logs across five separate services trying to reconstruct the same picture by hand.

Where to Actually Start

If you’re setting up observability for a system that currently has none, start with metrics and the four golden signals before investing in logs aggregation or distributed tracing. Metrics are cheapest to implement and give the fastest, broadest visibility into system health. Add centralized log aggregation next, so logs from every service are searchable in one place rather than requiring SSH access to individual servers. Add distributed tracing last, once you’re specifically debugging cross-service latency issues that metrics and logs alone aren’t resolving quickly enough.

SLIs, SLOs, and Error Budgets

Beyond raw metrics and dashboards, mature reliability practice formalizes targets: a Service Level Indicator (SLI) is a specific measured metric, like the percentage of requests completing successfully. A Service Level Objective (SLO) is the target for that indicator, say, 99.9% of requests succeed over a rolling 30-day window. The gap between actual performance and the SLO is the error budget, the amount of unreliability you’re allowed before breaching the target.

This framing does something genuinely useful beyond just measurement: it turns an abstract, endless debate (“should we prioritize reliability work or new features this sprint”) into a data-driven decision. If the error budget is nearly exhausted, that’s a clear, objective signal to prioritize reliability work over new features until it recovers; if there’s plenty of budget remaining, teams can justifiably take on more deployment risk and ship faster. This is the same underlying discipline referenced briefly in our enterprise SaaS guide as a tool for balancing shipping speed against reliability risk at scale.

Structured Logging: Making Logs Actually Searchable

Freeform text logs ("User 4821 failed to checkout") are readable by a human scanning a single file, but become nearly impossible to search or aggregate reliably at scale, across thousands of log lines from dozens of services. Structured logging outputs each log entry as a consistent, machine-parseable format, typically JSON, with well-defined fields, making it possible to filter, aggregate, and alert on log data the same way you would with metrics.

{
  "timestamp": "2026-07-12T14:32:01Z",
  "level": "error",
  "service": "checkout-api",
  "user_id": 4821,
  "event": "checkout_failed",
  "reason": "payment_declined",
  "trace_id": "a1b2c3d4"
}

With this structure, a query like “show me every failed checkout with reason payment_declined in the last hour, grouped by service” becomes a fast, reliable aggregation rather than a fragile text-pattern search across raw log files. The trace_id field also links this log entry directly to the distributed trace covering the same request, connecting all three observability pillars for a single incident investigation.

Runbooks: Turning Incident Response Into a Repeatable Process

An alert firing is only useful if the person paged knows what to actually do next. A runbook is a written, step-by-step guide tied to a specific alert, explaining what the alert means, the most likely causes, and the concrete steps to investigate and resolve it. Good monitoring platforms link an alert directly to its runbook, so an engineer paged at 3am isn’t starting their investigation from scratch under pressure, they’re following a process the team already thought through calmly, in advance.

Runbooks also compound in value over time: every real incident is an opportunity to update the relevant runbook with what was actually learned, so the next person to hit the same alert (possibly a less experienced engineer, possibly the same one six months later having forgotten the details) resolves it faster than the first time around.

Managing the Cost of Observability at Scale

Metrics, logs, and traces all cost money to store and query, and that cost grows with system scale, sometimes surprisingly fast once a service is generating verbose logs at high request volume. Common cost-control practices include sampling traces (recording a representative percentage of requests rather than every single one), setting shorter retention periods for high-volume, low-value logs (debug-level logs might only need a few days of retention, while audit logs might need months or years), and being deliberate about which metrics get high-cardinality labels, since a label like a unique user ID attached to every metric can multiply storage costs dramatically compared to a small, bounded set of label values.

How This Connects to the Rest of DevOps

Monitoring and observability are the feedback loop for everything else covered in this curriculum: they tell you whether a CI/CD pipeline’s deployment actually succeeded in production, whether a Kubernetes autoscaler is responding correctly to real load, and whether the security controls covered in our DevSecOps guide are catching real issues. Without this feedback loop, every other DevOps practice is operating on assumption rather than evidence.

Frequently Asked Questions

Building This Skill Hands-On

The most effective way to build real comfort with monitoring and observability is instrumenting a personal project end to end: expose Prometheus-style metrics from a small application, run Prometheus and Grafana locally (both have simple Docker Compose setups, covered in our Docker guide), build a dashboard around the four golden signals, then write an alerting rule and deliberately trigger it to see the full loop work. That hands-on practice, seeing a metric spike, watching an alert fire, and tracing it back to the cause, builds the debugging instincts that reading about observability alone never quite delivers.

Related reading: continue to Real-world DevOps projects, review our monitoring, alerting, and incident response guide, or explore database observability best practices.