Node.js Observability with OpenTelemetry: A Practical Guide to Traces, Metrics, and Logs

Programming tutorial - IT technology blog
Programming tutorial - IT technology blog

Context: Surviving Production Outages with OpenTelemetry

Debugging Node.js on localhost is straightforward. Production under load is another story. When your checkout endpoint’s p99 latency spikes from 45ms to 4.2 seconds during a flash sale, standard console.log output turns into a wall of unreadable text. Timestamps drift across asynchronous callbacks. You find yourself guessing: did a Postgres query stall on an unindexed table, did Stripe’s webhook timeout, or did a 12MB JSON payload block the single-threaded event loop?

OpenTelemetry (OTel) eliminates that guesswork. Backed by the Cloud Native Computing Foundation (CNCF), it delivers a vendor-neutral standard for telemetry. Instead of trapping your codebase inside a proprietary APM agent, OTel standardizes how you emit three fundamental telemetry signals:

  • Traces: Follow an HTTP request across functions, microservices, and databases using injected trace and span IDs.
  • Metrics: Numeric aggregations (request throughput, GC pause duration, event loop lag) computed over rolling intervals.
  • Logs: Structured events tied directly to active trace IDs so you can isolate error context in seconds.

Correlating these three signals turns midnight firefighting into a 30-second root-cause lookup.

Installation: Adding the OpenTelemetry Node.js SDK

OTel uses a modular architecture. You install only the components your application requires. Here is how to set up an Express service with automatic instrumentation and OpenTelemetry Protocol (OTLP) exporters:

mkdir node-otel-demo
cd node-otel-demo
npm init -y

# Install Express and the OpenTelemetry API
npm install express @opentelemetry/api

# Install SDK runtime, auto-instrumentations, and OTLP exporters
npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/sdk-metrics \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions \
  @opentelemetry/exporter-trace-otlp-proto \
  @opentelemetry/exporter-metrics-otlp-proto

Here is what each component handles:

  • @opentelemetry/api: The API package providing tracing and context functions for your application code.
  • @opentelemetry/sdk-node: The runtime engine that initializes tracer providers, metric readers, and background exporters before your app starts.
  • @opentelemetry/auto-instrumentations-node: Community-maintained hooks that wrap modules like http, express, pg, ioredis, and mongodb automatically.
  • @opentelemetry/exporter-*-otlp-proto: High-performance protobuf exporters that transmit telemetry over HTTP/gRPC to collectors such as Grafana Tempo, Jaeger, or SigNoz.

Configuration: Instrumenting Node.js Cleanly

Initialization order matters. OpenTelemetry must patch core Node.js modules before your application loads dependencies. If Express or pg imports ahead of OTel, outbound network sockets and route handlers run uninstrumented.

Step 1: Create the Instrumentation Module

Create an instrumentation.js file in your project root. Keep this logic separate from your web server bootstrap.

// instrumentation.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto');
const { PeriodicExportingMetricReader, ConsoleMetricExporter } = require('@opentelemetry/sdk-metrics');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const traceExporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT || 'http://localhost:4318/v1/traces',
});

const metricReader = new PeriodicExportingMetricReader({
  exporter: new ConsoleMetricExporter(),
  exportIntervalMillis: 15000,
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'order-api-service',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.2.0',
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV || 'production',
  }),
  traceExporter,
  metricReader,
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Flush remaining spans during process termination
const shutdown = () => {
  sdk.shutdown()
    .then(() => console.log('[OTel] SDK shutdown complete'))
    .catch((err) => console.error('[OTel] Error shutting down SDK', err))
    .finally(() => process.exit(0));
};

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

Step 2: Build the Sample Application with Trace-Correlated Logs

Create app.js. We will create custom spans around critical business logic and inject trace metadata into our JSON log payloads.

// app.js
const express = require('express');
const opentelemetry = require('@opentelemetry/api');

const app = express();
const PORT = process.env.PORT || 3000;

// Correlate stdout logs with active spans
function logWithTrace(level, message, data = {}) {
  const activeSpan = opentelemetry.trace.getActiveSpan();
  const spanContext = activeSpan ? activeSpan.spanContext() : null;

  const payload = {
    timestamp: new Date().toISOString(),
    level,
    message,
    trace_id: spanContext ? spanContext.traceId : 'none',
    span_id: spanContext ? spanContext.spanId : 'none',
    ...data,
  };
  console.log(JSON.stringify(payload));
}

app.get('/api/orders/:id', async (req, res) => {
  const orderId = req.params.id;
  const tracer = opentelemetry.trace.getTracer('order-tracer');

  // Create an explicit child span for granular timing
  return tracer.startActiveSpan('process-order-logic', async (span) => {
    try {
      span.setAttribute('app.order.id', orderId);
      logWithTrace('info', 'Processing order request', { orderId });

      // Simulate a 120ms database query
      await new Promise((resolve) => setTimeout(resolve, 120));

      if (orderId === 'invalid') {
        throw new Error('Order lookup failed: record 0x404 missing in primary replica');
      }

      span.setStatus({ code: opentelemetry.SpanStatusCode.OK });
      res.status(200).json({ status: 'success', orderId, totalCents: 4999 });
    } catch (err) {
      span.recordException(err);
      span.setStatus({
        code: opentelemetry.SpanStatusCode.ERROR,
        message: err.message,
      });
      logWithTrace('error', err.message, { orderId });
      res.status(404).json({ error: err.message });
    } finally {
      span.end();
    }
  });
});

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

Verification & Production Deployment

Preload your instrumentation file using Node’s native --require flag before starting the main process. This ensures telemetry hooks attach before Express initializes.

node --require ./instrumentation.js app.js

1. Trigger Test Requests

Send two sample requests to inspect how OTel handles healthy vs. failing code paths:

# 1. Successful request (generates OK span and standard 200 HTTP trace)
curl -i http://localhost:3000/api/orders/order_8829

# 2. Failing request (records an exception and sets span status to ERROR)
curl -i http://localhost:3000/api/orders/invalid

2. Trace-to-Log Correlation in Action

Review the application’s stdout. The log formatter outputs structured JSON records containing matching W3C trace_id and span_id values:

{"timestamp":"2026-11-16T11:50:00.123Z","level":"error","message":"Order lookup failed: record 0x404 missing in primary replica","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","orderId":"invalid"}

When PagerDuty pings you at 2:00 AM, query your log aggregator (such as Loki, Datadog, or Elasticsearch) for the error message. Grab the trace_id and paste it into Jaeger or Grafana Tempo. Immediately, you see the full waterfall: exactly 120ms spent waiting on the database before the error surfaced.

3. Next Production Steps

  • Decouple with an OTel Collector: Avoid sending telemetry directly from production Node.js instances to third-party endpoints. Deploy the OpenTelemetry Collector as a Kubernetes sidecar or local daemonset. It batches spans (e.g., 512-record buffers), compresses payloads, and reduces networking overhead on Node’s event loop.
  • Configure head-based sampling: At 10,000 requests per second, capturing 100% of traces wastes storage and money. Set OTEL_TRACES_SAMPLER=parentbased_traceidratio with an OTEL_TRACES_SAMPLER_ARG=0.05 ratio to record 5% of healthy traffic while capturing 100% of exceptions.
  • Use standard environment variables: Never bake collector endpoints into code. Set OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 and OTEL_SERVICE_NAME=order-api through your deployment manifests.
Share: