The Practical Developer

OpenTelemetry in Node.js: Distributed Tracing That Actually Helps During an Incident

September 30, 2022 by The Practical Developer

Distributed tracing only earns its keep at 3 a.m., when one slow request is hiding in a microservice call graph. Here is the OpenTelemetry setup for Node.js that auto-instruments the boring stuff, lets you add the span attributes that matter, and connects to any backend you point it at.

observability

node.js

reliability

A web of glowing fiber optic strands — the visual that tracing UIs always end up as

The user’s request hits seven services. One of them is slow. The logs say “everything is fine” because each service did its job in 200ms — but the request took 4 seconds end to end and nobody can point at the 3.4 seconds that disappeared. Twenty minutes into the incident, somebody pulls up Honeycomb / Tempo / Datadog APM, finds a 3-second span on a database call inside the auth service, and the whole team groans because it has happened three times this quarter.

Distributed tracing is the difference between “everything is fine and the user is angry” and “we know exactly which call inside which service is slow.” It is also the observability tool that pays back fastest because the auto-instrumented version covers 80% of what you need. This post is the OpenTelemetry-for-Node.js setup that gets you there in about 50 lines of code.

What tracing actually is

A trace is a tree of spans. Each span represents a unit of work — an HTTP handler, a database query, a fetch to another service — and has a start time, end time, and parent span ID. The root span is the original request; everything underneath is what the request caused to happen.

[ HTTP GET /api/orders/42                              ] 4200ms
   ├── [ DB: SELECT * FROM users WHERE id=$1 ]   12ms
   ├── [ HTTP GET auth-service/verify         ] 3400ms ←  the culprit
   │     └── [ DB: SELECT ... slow query      ] 3380ms
   ├── [ DB: SELECT * FROM orders WHERE ...   ]   45ms
   └── [ Redis: GET session:abc                ]    1ms

The point of tracing is that you read this tree once and see the slow call. Without tracing, you grep seven services’ logs by request ID — if you propagated a request ID — and reconstruct the timeline by hand.

OpenTelemetry in 50 lines

OpenTelemetry (OTel) is the vendor-neutral standard. The Node SDK auto-instruments most popular libraries — http, pg, mysql2, mongodb, redis, express, koa, fastify, aws-sdk, grpc — without any code changes. The setup file goes in tracing.ts and is loaded before anything else:

// tracing.ts — must be required before app code.
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: process.env.SERVICE_NAME ?? 'api',
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.GIT_SHA ?? 'dev',
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV ?? 'dev',
  }),
  traceExporter: new OTLPTraceExporter({
    // Where to send spans. Could be your own collector, or vendor's endpoint.
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // Disable a few that produce noise; opt back in selectively.
      '@opentelemetry/instrumentation-fs': { enabled: false },
      '@opentelemetry/instrumentation-dns': { enabled: false },
      '@opentelemetry/instrumentation-net': { enabled: false },
    }),
  ],
});

sdk.start();

process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0));
});

Load it from your entry point before anything else:

// index.ts
import './tracing'; // ← MUST be the first import
import express from 'express';
// ...

The reason it has to be first: OTel monkey-patches modules at require-time. If express loads before the SDK, the auto-instrumentation hook never fires and you get no spans for it. This catches everyone exactly once.

That is the entire setup. Run the app, point an OTel collector at it (Tempo, Jaeger, Honeycomb, Datadog all speak OTLP), and you will see traces for every HTTP request, every DB query, every outgoing fetch, automatically.

Adding the spans the auto-instrumentation misses

Auto-instrumentation knows about libraries. It does not know about your business logic. The two places you almost always want to add manual spans:

  1. The expensive operation that is not a DB or HTTP call — image processing, PDF generation, JWT signing, an in-process search.
  2. A logical boundary you want to see in the trace — “validate request,” “compute pricing,” “persist order” — even if it is not expensive, because it makes the trace readable.
import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('orders');

async function createOrder(req: Request) {
  return tracer.startActiveSpan('orders.create', async (span) => {
    try {
      span.setAttribute('user.id', req.userId);
      span.setAttribute('cart.item_count', req.cart.length);

      const validated = await validateCart(req.cart);
      const priced    = await computePricing(validated);
      const order     = await persistOrder(priced);

      span.setAttribute('order.id', order.id);
      span.setAttribute('order.total_cents', order.totalCents);

      return order;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: 2 /* ERROR */, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

startActiveSpan is the right helper because it makes the new span the current context — any DB calls or HTTP calls inside the callback automatically nest under it. setAttribute adds key/value metadata you can search by (“show me all orders.create spans where cart.item_count > 50”). recordException and setStatus flag failures so the trace UI highlights them.

Span attributes that pay rent

Throwing every variable onto a span as an attribute is a great way to blow up your trace storage cost. The attributes worth setting:

  • Identifiers you’ll search by. user.id, org.id, order.id, tenant.id. These let you filter to “all traces for user 42” in a real outage.
  • High-cardinality but informative facts. db.query.shape (a normalized version of the SQL with parameters stripped), http.route (the matched route, not the literal URL).
  • Decisions and outcomes. feature_flag.new_pricing.enabled, cache.hit, auth.method. These let you correlate behavior with codepaths.

Things to avoid:

  • PII. Email addresses, names, addresses. Tracing data ends up in many places (vendor systems, logs, debug exports). Treat traces as if a stranger will read them.
  • Massive values. A 200KB request body as a span attribute is going to cost you. If you need it, log it with a span ID for correlation.
  • Per-call random-looking values. Generated request IDs, raw timestamps. These create unique attribute combinations that explode the storage backend’s index.

Sampling: do not trace 100%

At low volume, sampling 100% of traces is fine. At ~50 RPS or higher, the cost (network egress, vendor ingest) becomes real. Three sampling strategies, in order of usefulness:

Head-based probability sampling. “Keep 10% of traces, dropping the rest entirely.” Cheap, simple, works.

import { TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-node';
new NodeSDK({ sampler: new TraceIdRatioBasedSampler(0.1) /* 10% */, ... });

The decision is made at the root span and propagated to all children — so if a trace is sampled in service A, every downstream call is also sampled. This is what trace context propagation is for.

Tail-based sampling. “Keep traces with errors or that took >1s; drop the rest.” Strictly better than head-based but requires a buffering collector (the OTel Collector supports it) because the decision happens at the end of the trace.

Always-sample for specific routes / users. “Always trace /checkout, always trace user 42, sample 1% otherwise.” Most production systems end up with a small allowlist of high-value endpoints sampled at 100% and a low rate for everything else.

Pick head-based to start. Move to tail-based when the bill is uncomfortable. Tail-based is harder to operate but gives you the best signal-to-noise ratio.

Trace context propagation across services

A trace only stays connected if every service forwards the trace ID. The OpenTelemetry SDK does this automatically for HTTP fetches via the traceparent header:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
              ^   ^                                ^                ^
              |   |                                |                trace flags
              |   |                                parent span id
              |   trace id
              version

If your services all use OTel SDKs, this is automatic. Edge cases that break it:

  • Custom HTTP clients. If you wrote your own fetch wrapper, the auto-instrumentation may not patch it. Use node:http directly, or wire propagation.inject(...) manually.
  • Message queues. The HTTP propagation does not apply. For Kafka / SQS / Redis Streams, inject the trace context into message headers/attributes and extract it on the consumer side.
  • Background jobs. A worker that pulls a job from a queue and processes it should continue the trace from the producer, not start a new one. Same fix: propagate via the job payload.

The smell of broken propagation is that traces look correct end-to-end inside one service and then “stop” at the network boundary, with the downstream service starting a fresh root trace.

What this earns you

Three concrete payoffs once tracing is in place:

Bottleneck identification. A trace UI sorts spans by duration. The slow span is visually obvious. The “we suspect it is the database” guesswork stops.

Cross-service request shape. “What does a typical POST /orders actually do?” — open one trace and the answer is a tree, not a slack thread.

Capacity planning by business action. Per-attribute aggregations let you ask “how does median checkout latency change with cart size?” or “are enterprise customers experiencing slower searches than free?”. This is the data engineering teams reach for during capacity reviews.

The cost is one config file, ~50 lines, and a backend (Tempo + Grafana works free; Honeycomb / Datadog if you want polish). The payback is the next time a multi-service incident happens — and you find the slow span in 60 seconds instead of 60 minutes.

The takeaway

Distributed tracing is one of the lowest-effort, highest-payoff observability investments you can make in a service that talks to more than one other service. Start with auto-instrumentation, add manual spans at logical boundaries, set the right attributes, sample with intent, and propagate context across queues. The full setup is half a day. The next time someone asks “where does the time go?”, you point at the trace.

Logs and metrics are still useful — but they answer different questions. “What happened to this request?” is what tracing answers, and nothing else does.

A note from Yojji

Wiring up tracing properly — propagating context through HTTP, queues, and background workers, sampling without losing the rare slow request, keeping span cardinality under control — is the kind of cross-cutting backend work that pays for itself the first incident in. It is the kind of engineering Yojji’s teams build into the production systems they deliver for clients.

Yojji is an international custom software development company founded in 2016, with offices across Europe, the US, and the UK. Their teams specialize in the JavaScript stack (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and microservices — including the observability work that decides whether your team is debugging a trace or a hunch at 3 a.m.