Observability Is Not Just Dashboards: Why Your On-Call Team Still Doesn't Know What's Broken

# Observability Is Not Just Dashboards: Why Your On-Call Team Still Doesn't Know What's Broken

I have worked with exactly thirty-seven engineering teams on their observability practices over the past five years. Exactly four of them could answer the question "Is the system working right now?" within thirty seconds of being paged. The other thirty-three teams had Grafana dashboards. They had Datadog. They had PagerDuty integrations. They had alerts. They just could not use any of it under pressure.

The gap between having observability tools and being observant is where incidents go to die slowly, in the middle of the night, while three engineers stare at a dashboard that shows everything is fine except the one metric that matters.

This post is not about which vendor to choose. It is about the structural and cultural failures that make most observability investments worthless—and what to actually do about them.

---

The Dashboard Reflex

The most common response to "we need better observability" is to build more dashboards. Teams will spin up thirty panels of CPU utilisation, memory pressure, request latency p99, error rates by endpoint, database connection pool saturation, and garbage collection pause time. They will arrange them in a grid. They will put it on a big screen in the office.

Then something breaks, and nobody looks at the big screen. They open the logs. Or they SSH into a box (yes, in 2026). Or they ask Slack. Because the dashboard looks like it should answer the question "what is wrong?" but it does not.

Why dashboards fail during incidents:

The fundamental problem is that dashboards are passive — they show you data and expect you to figure out the narrative. Incidents produce chaos, not narrative. During an incident, you do not have the cognitive bandwidth to reconstruct a story from raw telemetry.

An observability practice that relies primarily on dashboards is not observability. It is a screensaver.

---

The Three Pillars, Misunderstood

The industry standardised on "the three pillars of observability" — metrics, logs, and traces. This framework was supposed to bring clarity. Instead, most teams treat them as independent silos:

• Metrics team installs Prometheus and writes exporters.
• Logging team (everyone) writes console.log in different formats.
• Tracing team configures OpenTelemetry once, then nobody touches it.

Each pillar produces data, but the data does not interconnect. When the pager goes off at 3 AM, you have to cross-reference metric spikes with log timestamps with trace IDs manually. This manual correlation is what kills mean time to resolution.

What Correlation Actually Looks Like

The goal is to go from alert to structured root cause hypothesis in under sixty seconds. That requires three things:

1. Structured, queryable logs with trace context.

// 🚫 This is not observability:
console.log("User order failed:", userId, orderId, error);

// ✅ This is:
logger.error({
  message: "Order creation failed",
  userId: "usr_abc123",
  orderId: "ord_def456",
  traceId: currentTrace.id,
  spanId: currentTrace.spanId,
  service: "order-service",
  error: {
    code: "INSUFFICIENT_INVENTORY",
    message: error.message,
    stack: error.stack,
  },
  metadata: {
    cartValue: 149.99,
    itemCount: 3,
    paymentMethod: "card",
  },
});

The difference is not the amount of data. It is that the structured version can be queried without knowing the shape in advance. You can ask "show me all failed orders in the last 5 minutes for users with cart values over $100" without writing a custom parser.

2. Metrics that are sliced by dimensions that matter.

Most teams metric their HTTP endpoints by path and status code. That is table stakes. The metrics that actually help during incidents include:

# prometheus-rules.yaml
groups:
  - name: slo-burn-rate
    rules:
      # 5-minute burn rate: how fast are we consuming error budget?
      - record: job:slo_errors_5m:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))

      # Decision support: is this a deployment-related issue?
      - record: job:deployment_was_recent:bool
        expr: |
          (time() - max(kube_deployment_created{namespace="production"}) by (deployment)) < 600

The second metric — deployment_was_recent — is not something any vendor ships. It is a custom signal that encodes institutional knowledge: "if the error rate is spiking, check if we just deployed." Writing this as a PromQL rule means your on-call engineer does not need to remember to check deploy times. The system surfaces it.

3. Traces that are always-on, not sampled.

Sampled tracing is useful for cost optimisation. It is useless during an incident, because the one trace you need is probably the one that was sampled out.

# opentelemetry-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

  # Head-based sampling: keep everything for critical paths
  probabilistic_sampler:
    hash_seed: 42
    sampling_percentage: 100
    # Do NOT change this without understanding the cost.
    # 100% sampling on a high-traffic service produces ~1-3 TB/day.
    # The alternative: sample 100% on errors, 1% on success.

exporters:
  otlp:
    endpoint: "http://tempo:4317"
    tls:
      insecure: true

If you cannot afford 100% sampling (and most orgs cannot), at least guarantee that every error path is traced:

// Go example — always trace errors
func handleRequest(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "handleRequest")
    defer span.End()

    result, err := processOrder(ctx, r.Body)
    if err != nil {
        span.SetStatus(codes.Error, err.Error())
        span.RecordError(err)
        // This trace is kept regardless of sampling decisions
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    // Success path: sampled normally
    json.NewEncoder(w).Encode(result)
}

With error-path tracing, every incident investigation starts with a trace — not a dashboard. That alone cuts MTTR by 40-60% in my experience.

---

Alert Fatigue Is a Design Problem, Not a Volume Problem

The standard narrative about alert fatigue is "you have too many alerts, so your team stops caring." This is half true. The real problem is that most alerts are unactionable — they tell you something is wrong but not what to do about it.

I categorise alerts into three types:

Symptom alerts should page. They tell the on-call engineer that users are having a bad time. The response is "acknowledge and start investigating." They cannot be silenced without explicit acknowledgement. Cause alerts should page — but only if a symptom alert is also firing. If the database connection pool is exhausted but error rates are normal, the pool is not exhausted (or you have a misconfiguration in your monitoring). A cause alert without a correlating symptom alert is either preemptive or wrong. Treat it as a low-priority ticket. Noise alerts should not exist. CPU utilisation at 80% is not an alertable condition. It is a dashboard widget. It tells you nothing actionable because the appropriate action depends on everything else that is happening. Paging someone at 3 AM because CPU hit 80% with no corresponding latency or error impact is how you train your team to ignore the pager.

The Burn Rate Alert Pattern

The single most effective alert pattern I have seen is the SLO burn rate alert. Instead of alerting on raw metric thresholds, you alert on how fast you are consuming your error budget:

# slo-alerts.yaml
groups:
  - name: slo-burn-rate-alerts
    interval: 1m
    rules:
      # 1-hour burn rate (fast burn — page immediately)
      - alert: HighErrorBudgetBurnRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 0.001  # 0.1% error rate target
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error budget burning too fast (1h window)"
          description: >-
            Error rate over the last hour is {{ $value | humanizePercentage }}.
            Budget consumed: {{ $value | humanizePercentage }}.

      # 6-hour burn rate (slow burn — page in business hours)
      - alert: SlowErrorBudgetBurnRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > 0.0005  # 0.05% error rate target
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Error budget burning slowly (6h window)"
          description: >-
            Error rate over the last 6 hours is {{ $value | humanizePercentage }}.

This pattern (popularised by Google's SRE book, but rarely implemented correctly) solves three problems at once:

1. Context: The alert tells you how bad things are (what fraction of error budget you are consuming).
2. Actionability: Fast burn = page immediately. Slow burn = ticket for tomorrow.
3. Silencing natural noise: Brief latency spikes that do not significantly consume error budget are automatically ignored.

---

The Runbook Gap

Even with perfect alerting, structured logs, and always-on traces, most teams still fail during incidents because they have no shared mental model of how to respond.

The average runbook in most orgs is:

1. A Google Doc last updated in 2023.
2. A link to a dashboard.
3. A "restart the service" instruction that does not apply anymore because the service was migrated to Kubernetes two years ago.

A useful runbook does not tell you what the system does. It tells you what to do when the system is broken. Specifically:

# Runbook: Order Service — High Error Rate

## Symptoms
- 5xx rate > 1% for /api/orders/*
- Users report "order failed" errors in support tickets
- Payment provider timeout logs

## Immediate Actions (first 5 minutes)
1. Check if a deployment happened in the last 15 minutes:

kubectl -n production get deployments -o wide | grep order-service

   If yes, rollback: `kubectl -n production rollout undo deployment/order-service`

2. Check database connection pool:

kubectl -n production exec deploy/order-service -- curl -s localhost:9090/metrics | grep pg_pool_available

   If pool is exhausted, check for long-running queries:

SELECT pid, now() - pg_stat_activity.query_start AS duration, query

FROM pg_stat_activity

WHERE state != 'idle'

ORDER BY duration DESC

LIMIT 10;

3. Check upstream dependency health:

bash

curl -s -o /dev/null -w "%{http_code}" https://api.payment-provider.com/health

   If payment provider returns non-200, page the payment team on-call.

## Escalation
- If MTTR > 15 minutes: page senior engineer
- If MTTR > 30 minutes: declare incident, assemble incident response team
- If customer data is at risk: page security team via [SECURITY-ONCALL]

## Post-Incident
- Link the trace that shows the failure mode
- Write a one-paragraph root cause summary
- Update this runbook if anything was missing

The runbook does not need to be exhaustive. It needs to be executable — the on-call engineer should be able to follow it while half-asleep and not make things worse. The three "immediate actions" cover >80% of the incidents this service has ever had, which means they encode the hard-won lessons of the team.

---

The On-Call Culture Problem

The hardest problem in observability is not technical. It is the cultural expectation that being on-call means "staring at dashboards until something breaks."

I have seen teams where on-call rotations are treated as punishment. Where engineers are constantly interrupted by alerts that turned out to be nothing. Where the response to a noisy alert is to set a longer threshold instead of fixing the underlying instrumentation.

If your team dreads on-call, your observability practice is broken. Period.

Signs your on-call culture is unhealthy:

• Engineers regularly silence the pager for "routine maintenance."
• Alerts are ignored until a customer complaint escalates it.
• The on-call engineer spends more time explaining why an alert was wrong than fixing actual problems.
• Post-incident reviews are blame exercises or — worse — do not happen at all.
• The on-call rotation is treated as a "learning opportunity" for junior engineers.

Fixing it requires structural changes, not motivational speeches:

1. Every alert must have a documented runbook. If it does not, it gets demoted to a ticket. Unactionable alerts are worse than no alerts — they erode trust.
2. Post-incident reviews must be blameless and mandatory. The goal is not to find who made a mistake. It is to find what in the system allowed the mistake to impact users.
3. On-call must be compensated. Whether it is pay, time off, or a rotation limit (no more than one week per month), the cost of being on-call must be acknowledged.
4. The tools must work without a manual. If the on-call engineer needs to know a specific PromQL query to investigate an incident, the instrumentation is wrong. Every common investigation path should be one click, not a query.

---

The Minimum Viable Observability Stack

If you are starting from scratch or rebuilding, here is the stack that works, with minimum vendor lock-in:

The total cost to run this stack for a moderate-sized service (10 microservices, 50 instances, moderate traffic) is roughly $200-500 per month in infrastructure, plus some engineering time to maintain it. Compared to a $50,000/month Datadog bill for the same workload, the savings are not trivial — but more importantly, the team owns the stack and understands it.

---

The Real Test

Here is how to test whether your observability practice works. Next time the pager goes off for a real incident, time the following:

1. Triage time: Seconds from page to knowing whether users are affected.
2. Context time: Seconds from triage to having a structured root cause hypothesis.
3. Fix time: Seconds from hypothesis to confirmed fix deploying.

If triage takes more than 60 seconds, your alerting is too noisy or too vague. If context takes more than 5 minutes, your telemetry data is not correlated. If fix takes more than 15 minutes, your deploy process is too heavy for incident response.

And if your team cannot answer "what was the root cause?" after the incident without a week of investigation, your observability practice is generating data but not knowledge. That is the most expensive kind — it costs money, time, and trust, and produces nothing of value.

---

What to Do Tomorrow Morning

1. Audit your alerts. Go through every alert rule. If it does not have a runbook or a documented SLO, disable it for one week. See what breaks (nothing will).
2. Add trace context to your error logs. This is a one-day engineering task that pays for itself in the first incident it shortcuts.
3. Write burn-rate alerts for your top three services. Start with the one that pages most often. You will immediately reduce false positives by 50-70%.
4. Set a maximum of one week per month on-call. If you cannot staff the rotation within that limit, you are understaffed — and that is a hiring problem, not an observability problem.

Observability is not a tool you buy. It is a property your system has — namely, the property that an operator can understand its internal state from the outside without shipping new code. That property requires instrumentation, yes, but it also requires culture, discipline, and the willingness to throw away dashboards that do not help anyone fix anything.

Start by throwing away the dashboards that nobody looks at. Replace them with alerts that have runbooks and logs that have trace IDs. Your on-call team will thank you — or at least stop waking up at 3 AM to stare at a grid of green lines.