OpenTelemetry and instrumentation

The pieces you touch: SDK, Collector, backend

OpenTelemetry (OTel) splits into parts you configure separately. An SDK inside your process produces the telemetry: a TracerProvider and MeterProvider create spans and metrics, tag them with a resource (attributes describing the process, above all service.name), and hand them to an exporter. The exporter speaks OTLP, OTel's wire protocol, over gRPC or HTTP. It usually ships to a Collector, a separate process that receives, processes, and forwards to one or more backends (Jaeger, Prometheus, a SaaS vendor). What a span, metric, or log actually IS lives in the observability module; this module is about how they get produced and shipped.

One resource attribute is load-bearing: service.name. It is required, and if you forget it the SDK falls back to unknown_service:<executable>, so every service you didn't name collapses into one blob on the backend. Set it in code or via OTEL_SERVICE_NAME before anything else.

Why not export straight to the vendor

You can point the SDK's exporter at the backend and skip the Collector. It works until it doesn't. The Collector earns its place by taking four jobs out of every app:

• Batching: group spans and metrics before sending, fewer round trips.
• Retry and queue: hold data when the backend blips instead of dropping it.
• Transform: drop or redact an attribute (a token, a PII field) before it leaves your network.
• Fan-out: send the same data to two backends, or route traces one place and metrics another.

Put that logic in the Collector and you change it by editing config, not by redeploying twenty services. Two shapes: an agent next to each app (sidecar or host daemon) that offloads fast, and a gateway cluster that centralizes processing. Many setups run both.

Wrong "the SDK already retries and buffers, so a Collector is redundant." SDK exporters have limited in-memory retry and no cross-service policy, and there is no general disk buffer you flip on. Right the Collector is where retry, queue, redaction, and fan-out live as configuration shared by every service.

How a trace crosses a service boundary

Observability covers how a traceparent header joins a caller's span to the callee's; the mechanism this module owns is inject and extract. Within one process the SDK tracks the current span in context. Across a boundary a propagator injects that context into the outgoing carrier (for HTTP, the W3C traceparent header) and the receiving side extracts it.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ ^---------- trace-id ----------^ ^--parent-id---^ ^^
           version                                              flags

The trap is any hop that is not plain HTTP client-to-server. A message queue, a background thread pool, a manually spawned goroutine or task: the propagator does not run there unless something calls it. Publish to a queue without injecting the context into the message, and the consumer sees no parent and starts a brand-new root trace. The two halves never join.

Wrong "auto-instrumentation propagates context everywhere." Right it propagates on the library edges it patched (HTTP clients and servers, some queue clients). A plain publish/consume, a ThreadPoolExecutor, or an async job you wrote by hand carries nothing unless you inject on the way out and extract on the way in. Fragmented traces are usually a propagation gap, not a sampling problem.

Naming: semantic conventions keep labels sane

Semantic conventions are OTel's agreed names for common attributes, so a backend can read http.request.method, http.response.status_code, and http.route from any language or library the same way. Use them rather than inventing httpMethod.

One convention protects you from a bill later. An HTTP server span carries both url.path (the real path, /orders/4837) and http.route (the template, /orders/:id). Emit the route as the low-cardinality identifier and let the span name be the route, not the raw path. The reasoning (one series per label value) is in the senior notes and the observability module; the produce-side rule is simple. Raw ids and paths go in span and log attributes, never in the metric labels you aggregate on.

Sampling: one head decision, propagated

Observability covers why head sampling is cheap but outcome-blind and why tail sampling is stateful; the instrumentation job is to get the head decision made once and propagated, not re-rolled per hop.

The decision has to be made once and shared, or traces fragment. The SDK default sampler is ParentBased(root = AlwaysOn), the env value parentbased_always_on: if there is an incoming parent, honor its sampled flag; otherwise ask the root sampler. To sample a percentage, set the root to TraceIdRatioBased (env parentbased_traceidratio). The pairing is the point. Bare TraceIdRatioBased ignores the parent's sampled flag by spec, so using it alone lets each service re-decide and shreds traces into fragments. Wrapped in ParentBased, the root's decision rides in the traceparent sampled flag and every child obeys it.

Wrong "our traces are incomplete, so raise the sampling rate." Raising an independent per-service rate only raises the odds each hop keeps its own dice roll, so the retained traces are still whichever ones happened to line up end to end. Right the fix is propagation, not volume. One decision at the root, carried by the sampled flag.

Tail sampling is where this module's Collector work lives. Because it can only decide once a trace's spans have arrived (observability covers that tradeoff), something must buffer those spans until the trace is mostly complete, which is why it runs in a Collector and never in the SDK.

Running tail sampling across a scaled Collector tier

The tail_sampling processor decides per trace, so every span of a trace must reach the same Collector instance. A round-robin gateway breaks that: spans of one trace scatter across replicas and each replica sees a fragment, so it drops traces it should have kept. The standard fix is two tiers. A first tier runs a load-balancing exporter with routing_key: traceID, which hashes each span by trace id to a fixed second-tier instance; the second tier runs tail_sampling.

Order is not cosmetic. memory_limiter goes first so the process sheds load before it OOMs; tail_sampling before batch so you drop traces before paying to batch and send them; batch last to cut export round trips. OTLP itself is stable for traces, metrics, and logs, so this pipeline is vendor-neutral: swap the exporter, keep everything else.

Where propagation breaks, and baggage

Propagation runs only where a propagator injects and extracts. HTTP client and server instrumentation does both for you. The rest is on you:

• Message queues: inject the context into the message headers on publish, extract on consume. Nothing does this by default for a raw producer or consumer.
• Thread pools and async executors: the context lives on the current execution unit. Hand work to a ThreadPoolExecutor or a detached task and the captured context does not follow unless the instrumentation, or you, re-attaches it.
• Batching your own work: one span around a batch of N unrelated items files N requests under one trace. Relate them with span links instead of forcing them under one parent.

Baggage is a second thing that rides the same context. It is a key-value store propagated across services in the W3C baggage header, separate from span attributes. Two facts trip people. First, baggage is not copied onto your spans automatically; you read a key and set it as an attribute yourself. Second, it leaves your network: baggage travels in outbound headers, so a value like user.tier=gold can reach third-party APIs and anyone reading the traffic, and there is no integrity check on baggage you receive. Keep it small and non-sensitive.

Cardinality economics on the produce side

Observability owns why cardinality is expensive (one series per label-value combination) and what an exemplar is; the instrumentation job is to not manufacture cardinality in the first place. Put user_id on an HTTP metric and you mint a series per user; put raw url.path and you mint one per unique URL. Unbounded labels are what turn a metrics bill into a surprise.

The produce-side rules are narrow. Keep metric labels to bounded dimensions: http.request.method, http.response.status_code, and the templated http.route (/orders/:id), never the raw url.path. Per-request identity (user_id, request_id, a full path) belongs in spans and logs, where one more attribute costs one field, not a new series. When you need to jump from a slow metric bucket to an example request, reach for an exemplar (defined in observability): set a trace_id on the metric data point instead of adding a per-request label.