Building Observability Into Microservices From Day One

# Building Observability Into Microservices From Day One

Every team I've joined had a "we'll add observability later" phase. It always ends the same way: a midnight Slack message — "Is anyone else seeing errors? No? Must be my machine?" — followed by four hours of SSH-ing into boxes, grep-ing logs, and wondering if anyone ever defined a health check endpoint.

Observability is infrastructure, not polish. You wouldn't ship a service without a database driver. You shouldn't ship one without structured logging, runtime metrics, and distributed tracing wired in at the framework level.

This post covers what that looks like in practice: the scaffolding decisions, the one dashboard that replaces twenty, how to trace across async boundaries, and the specific OpenTelemetry setup that doesn't make you hate your life.

The Three Pillars — And One You're Ignoring

The industry settled on three pillars, but one of them is a trap.

The pillar most teams ignore? Profiles. Continuous profiling (via eBPF or runtime sampling) shows you why your CPU is spiking or where that memory leak lives. Pyroscope and Parca have made this practical, and OpenTelemetry's profiling signal is maturing fast. For this post I'll focus on the first three, but keep an eye on continuous profiling — it catches things metrics never will.

The Scaffold: Standardise at the Framework Level

Observability shouldn't be every developer's responsibility. It should be in the framework, the middleware, the base class. If your service template doesn't export structured logs and trace context automatically, you're asking each team to reinvent (and half-ass) the same thing.

Here's the TypeScript scaffold I drop into every new microservice:

// telemetry.ts — framework-level observability init, copy-paste into every service
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-grpc";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-grpc";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { WinstonInstrumentation } from "@opentelemetry/instrumentation-winston";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { ExpressInstrumentation } from "@opentelemetry/instrumentation-express";
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";

const serviceName = process.env.OTEL_SERVICE_NAME || "unknown-service";
const otlpEndpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://localhost:4317";

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV || "development",
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.GIT_COMMIT_HASH || "local",
  }),
  traceExporter: new OTLPTraceExporter({ url: `${otlpEndpoint}` }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({ url: `${otlpEndpoint}` }),
    exportIntervalMillis: 30_000,
  }),
  logRecordProcessor: new BatchLogRecordProcessor(new OTLPLogExporter({ url: `${otlpEndpoint}` })),
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
    new PgInstrumentation(),
    new WinstonInstrumentation(),
  ],
});

sdk.start();
process.on("SIGTERM", () => sdk.shutdown());
process.on("SIGINT", () => sdk.shutdown());

This single file gives every service:

• Automatic HTTP trace context propagation — every incoming request gets a trace ID, every outgoing HTTP call inherits it
• Database call instrumentation — Postgres queries appear as child spans with full SQL
• Correlated logs — Winston logs carry the active trace ID and span ID
• Metrics with consistent resource attributes — service name, version, environment — so your dashboards don't break when someone renames a deployment

The key investment here is one hour per service max. Wire this in on day one. Don't wait until p99 latency drifts past 2 seconds and nobody knows why.

Structured Logging Is Not an Ornament

I still see teams shipping logs like this:

// ❌ This is 2005 logging
console.log(`User ${userId} created order ${orderId}`);

This is a string. You can't filter it. You can't query it. You can't alert on it. If userId contains a special character, your log parser crashes. This is not logging — it's performance art.

Structured logging means every log line is key-value pairs that your observability platform can index, filter, and aggregate.

import winston from "winston";
import { trace } from "@opentelemetry/api";

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || "info",
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  defaultMeta: { service: serviceName },
  transports: [new winston.transports.Console()],
});

// Usage — every log entry carries structured context
export function createOrderLogger(requestId: string, userId: string) {
  const span = trace.getActiveSpan();
  return {
    info: (message: string, meta?: Record<string, unknown>) =>
      logger.info(message, {
        requestId,
        userId,
        traceId: span?.spanContext().traceId,
        spanId: span?.spanContext().spanId,
        ...meta,
      }),
    error: (message: string, error?: Error, meta?: Record<string, unknown>) =>
      logger.error(message, {
        requestId,
        userId,
        traceId: span?.spanContext().traceId,
        spanId: span?.spanContext().spanId,
        errorMessage: error?.message,
        errorStack: error?.stack,
        ...meta,
      }),
    warn: (message: string, meta?: Record<string, unknown>) =>
      logger.warn(message, {
        requestId,
        userId,
        traceId: span?.spanContext().traceId,
        spanId: span?.spanContext().spanId,
        ...meta,
      }),
  };
}

The output looks like this — queryable, correlate-able, and alert-able:

{
  "level": "error",
  "message": "Payment provider declined transaction",
  "timestamp": "2026-07-11T08:23:14.002Z",
  "service": "order-service",
  "requestId": "req_abc123",
  "userId": "user_789",
  "traceId": "0af7651916cd43dd8448eb211c80319c",
  "spanId": "b7ad6b7169203331",
  "errorMessage": "card_declined: insufficient_funds",
  "paymentProvider": "stripe",
  "attemptNumber": 2,
  "amount": 4999
}

Now you can write a query in Grafana or your log aggregator: {service="order-service", errorMessage=~"card_declined.*"} — and get every occurrence, grouped by payment provider, in under a second. No grep. No SSH. No "3 AM my machine" moments.

The One Dashboard That Replaces Twenty

Teams love dashboards. They make thirty of them. And then nobody looks at any of them, because each one shows a slightly different view of "everything is green or red."

Here's a hot take: you need one operational dashboard per service. Not per team. Per service. It shows exactly four things:

1. Request rate (RPS, coloured by HTTP status class)
2. Latency (p50, p95, p99 on a heatmap)
3. Error rate (as a percentage of requests, with a trend line)
4. Saturation (CPU, memory, connection pool depth, queue depth)

That's it. Everything else is a drill-down query.

Here's how you define those metrics in OpenTelemetry:

import { metrics } from "@opentelemetry/api";

const meter = metrics.getMeter("order-service");

// Histogram captures latency distribution — not just averages
const httpDuration = meter.createHistogram("http.server.duration", {
  description: "HTTP request duration in milliseconds",
  unit: "ms",
  boundaries: [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000],
});

// Counter tracks request volume by status and route
const httpRequests = meter.createCounter("http.server.requests", {
  description: "Total HTTP requests",
});

export function recordRequest(method: string, route: string, statusCode: number, durationMs: number) {
  const attributes = {
    "http.method": method,
    "http.route": route,
    "http.status_code": statusCode,
    "status_class": `${Math.floor(statusCode / 100)}xx`,
  };

  httpRequests.add(1, attributes);
  httpDuration.record(durationMs, attributes);
}

The histogram boundaries matter. Default OpenTelemetry histograms use exponential buckets, but I prefer the fixed boundaries above because they map directly to SLO tiers. A request under 5ms is "fast", under 250ms is "acceptable", over 2000ms is "we need to talk."

Pro tip: Add exemplars to your metrics. Exemplars link a metric datapoint to a specific trace ID:

const { trace } = require("next/dist/compiled/@opentelemetry/api");

httpDuration.record(durationMs, attributes, {
  exemplars: [{
    filtered_attributes: attributes,
    value: durationMs,
    span_id: trace.getActiveSpan()?.spanContext().spanId,
    trace_id: trace.getActiveSpan()?.spanContext().traceId,
  }],
});

With exemplars, you can click on a p99 latency spike in Grafana and jump directly to the slowest trace. This collapses the "what's slow?" → "why is it slow?" feedback loop from hours to seconds.

Distributed Tracing Across Async Boundaries

HTTP request-response tracing is table stakes. The hard part is tracing across async boundaries — message queues, scheduled jobs, background workers.

Here's the pattern for propagating trace context through a Kafka message:

// Producer side — inject trace context into message headers
import { propagation, context, SpanKind } from "@opentelemetry/api";
import { W3CTraceContextPropagator } from "@opentelemetry/core";

const propagator = new W3CTraceContextPropagator();

async function publishOrderCreated(order: Order): Promise<void> {
  const tracer = trace.getTracer("order-service");
  const span = tracer.startSpan("kafka.produce", {
    kind: SpanKind.PRODUCER,
    attributes: {
      "messaging.system": "kafka",
      "messaging.destination": "orders.created",
      "messaging.destination_kind": "topic",
      "messaging.message_id": `order-${order.id}`,
    },
  });

  const headers: Record<string, string> = {};
  propagator.inject(context.active().setValue(SET_MESSAGE_ATTRIBUTE, headers), headers);

  await kafka.produce({
    topic: "orders.created",
    key: order.id,
    value: JSON.stringify(order),
    headers, // <-- trace_id, span_id, trace_flags propagated via W3C Trace Context
  });

  span.end();
}

// Consumer side — extract trace context from message headers
import { propagation, context, SpanKind } from "@opentelemetry/api";

async function handleOrderCreated(message: KafkaMessage): Promise<void> {
  const extractedContext = propagator.extract(
    context.active(),
    message.headers,
    {
      get: (carrier, key) => carrier[key]?.toString(),
      keys: (carrier) => Object.keys(carrier),
    }
  );

  const tracer = trace.getTracer("order-service");

  await context.with(extractedContext, async () => {
    const span = tracer.startSpan("kafka.consume", {
      kind: SpanKind.CONSUMER,
      attributes: {
        "messaging.system": "kafka",
        "messaging.destination": "orders.created",
        "messaging.operation": "process",
        "messaging.message_id": `order-${message.value.id}`,
      },
    });

    try {
      await processOrder(message.value);
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      span.recordException(err as Error);
      throw err;
    } finally {
      span.end();
    }
  });
}

With this in place, a single trace spans:

api-gateway (HTTP POST /orders)
  └── order-service (createOrder)
       ├── postgres (INSERT INTO orders)
       ├── kafka.produce (orders.created)  ───→
       │                                         └── payment-service (handleOrderCreated)
       │                                              ├── postgres (INSERT INTO invoices)
       │                                              ├── http POST stripe.com/charges  ───→ stripe API
       │                                              └── kafka.produce (payment.succeeded) ───→ ...
       └── http response (201 Created)

One trace ID ties every hop together. When a user's payment fails, you don't guess which service dropped the ball — you open the trace and see exactly where it died.

Sampling Strategies That Don't Bankrupt You

Tracing every request sounds great until your observability bill exceeds your compute bill. At scale, you need to sample.

Here's the pragmatic setup:

// Sampler that traces all errors + 5% of success requests
import { Sampler, SamplingResult, SamplingDecision } from "@opentelemetry/types";

export class PrioritySampler implements Sampler {
  private baseRate = 0.05; // 5%
  private errorRate = 1.0; // 100% of errors

  shouldSample(
    context: Context,
    traceId: string,
    spanName: string,
    spanKind: SpanKind,
    attributes: SpanAttributes,
    links: Link[]
  ): SamplingResult {
    // Always sample health checks at minimal cost
    if (spanName === "GET /health" || spanName === "GET /ready") {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED, traceState: undefined };
    }

    // Use consistent hashing so entire trace is sampled or not
    const hash = hashTraceId(traceId);

    if (hash < this.baseRate) {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED, traceState: undefined };
    }

    // Record but don't export — available for tail-based sampling
    return { decision: SamplingDecision.RECORD_ONLY, traceState: undefined };
  }
}

function hashTraceId(traceId: string): number {
  let hash = 0;
  for (let i = 0; i < traceId.length; i++) {
    const char = traceId.charCodeAt(i);
    hash = ((hash << 5) - hash) + char;
    hash |= 0;
  }
  return Math.abs(hash) / 0x7FFFFFFF;
}

What about errors? With head-based sampling, the chance of catching a 0.01% error event is 0.0005%. You'll miss them.

The real solution is tail-based sampling via an OTel collector that aggregates decisions:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    policies:
      # Always keep error traces
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      # Always keep slow traces (>1s)
      - name: slow-policy
        type: latency
        latency: { threshold_ms: 1000 }
      # Keep 10% of everything else
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp:
    endpoint: "grafana-tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp]

This gives you 100% error coverage, 100% slow-request coverage, and 10% of everything else — at a fraction of the cost of sampling everything.

Health Checks That Actually Tell You Something

A health endpoint that returns { "status": "ok" } is a lie. It tells you the process is running but nothing about whether it can serve traffic.

// ✅ Readiness check that validates actual dependencies
import { Router, Request, Response } from "express";

const healthRouter = Router();

interface HealthStatus {
  status: "healthy" | "degraded" | "unhealthy";
  version: string;
  uptime: number;
  dependencies: Record<string, { status: string; latencyMs: number; lastError?: string }>;
}

healthRouter.get("/health", async (_req: Request, res: Response) => {
  const checks = {
    database: await checkDependency("postgres://...", 2000),
    kafka: await checkDependency("kafka://broker:9092", 2000),
    redis: await checkDependency("redis://cache:6379", 500),
    "payment-provider": await checkDependency("https://api.stripe.com", 5000),
  };

  const failed = Object.values(checks).filter(c => c.status !== "ok").length;
  const degraded = Object.values(checks).filter(c => c.status === "degraded").length;

  const status: HealthStatus = {
    status: failed > 0 ? "unhealthy" : degraded > 0 ? "degraded" : "healthy",
    version: process.env.GIT_COMMIT_HASH || "unknown",
    uptime: process.uptime(),
    dependencies: checks,
  };

  const httpStatus = status.status === "healthy" ? 200 : status.status === "degraded" ? 200 : 503;
  res.status(httpStatus).json(status);
});

async function checkDependency(target: string, timeout: number): Promise<...> {
  const start = Date.now();
  try {
    // Actual connectivity check here
    await ping(target, timeout);
    return { status: "ok", latencyMs: Date.now() - start };
  } catch (err) {
    return { status: "unhealthy", latencyMs: Date.now() - start, lastError: (err as Error).message };
  }
}

Now your load balancer can route around a degraded service, and your on-call dashboard shows exactly which dependency is failing — not just "service is down."

The Observability Budget

Every new dependency (database, cache, queue, external API) adds complexity. Every complexity needs observability. Set a budget:

┌──────────────────────────────┬─────────────┬──────────────┐
│ Component                    │ Instrument  │ Alert If     │
├──────────────────────────────┼─────────────┼──────────────┤
│ HTTP endpoints               │ Traces +    │ p99 > 500ms  │
│                              │ Metrics +   │ or error %   │
│                              │ Logs on 4xx │ > 1%         │
│                              │ & 5xx       │              │
├──────────────────────────────┼─────────────┼──────────────┤
│ Database queries             │ Traces +    │ p99 > 100ms  │
│                              │ Metrics     │ (query-level)│
├──────────────────────────────┼─────────────┼──────────────┤
│ Message queue producers      │ Traces      │ Lag > 1000   │
│                              │ (propagate) │ events       │
├──────────────────────────────┼─────────────┼──────────────┤
│ Message queue consumers      │ Traces +    │ Process time │
│                              │ Logs on     │ p99 > 5s     │
│                              │ retry/drop  │              │
├──────────────────────────────┼─────────────┼──────────────┤
│ External API calls           │ Traces +    │ p99 > 2s     │
│                              │ Metrics     │ or error %   │
│                              │ (histogram) │ > 2%         │
├──────────────────────────────┼─────────────┼──────────────┤
│ Background jobs / cron       │ Traces +    │ Failure rate │
│                              │ Logs        │ > 1%         │
└──────────────────────────────┴─────────────┴──────────────┘

Every new component gets an observability checklist entry in the pull request template. If the PR adds a new gRPC client but doesn't add OpenTelemetry instrumentation, it doesn't merge. This is not bureaucracy — this is the difference between "I wonder why it's slow" and "I can see the exact P99 latency of the new dependency on the dashboard that autocreated when the service started."

The Hard Truth: Invest or Inherit the Pain

Adding OpenTelemetry to a greenfield service takes 30 minutes. Adding it to a service with 50,000 lines of production code, hand-rolled HTTP clients, and raw console.log statements takes weeks — and that's if you have the stomach for the instrumentation surgery.

I've been on both sides. The services that had observability from day one are the ones I sleep through the night on. The ones that didn't are the ones I associate with Slack channel names like #prod-break-glass and #emergency-response.

The pattern is simple:

1. Scaffold once. Drop telemetry.ts into every service template.
2. Log structured, not pretty. JSON key-value pairs, not formatted strings.
3. Propagate trace context everywhere. HTTP headers, Kafka headers, gRPC metadata — if it carries data, it carries a trace ID.
4. Sample intelligently. Errors at 100%, slow traces at 100%, everything else at 5-10%.
5. One dashboard per service. Rate, latency, errors, saturation. Everything else is a drill-down.

Do this on day one. Your 3 AM self will thank you.