←Back to Guides
๐Ÿ“ˆObservability & SREโฑ๏ธ 14 min read•Updated November 2026•โœ๏ธBy Chris

Production Observability: Structured JSON Logging, OpenTelemetry Tracing & Prometheus p99 Metrics

Debug distributed outages in minutes instead of hours: emit W3C Trace-Context correlated NDJSON logs without PII leaks, calculate true p95/p99 latency histograms in PromQL, and separate SLO burn-rate alerts from noisy CPU spikes.

#Observability#OpenTelemetry#Prometheus#Structured Logging#NDJSON#SRE

1. Logs, Metrics & Distributed Traces Unified via trace_id

Averages hide tail outages: 99% healthy traffic masks 1% of checkout users hitting a 12-second timeout. Modern observability links lightweight Metrics (cheap aggregated time series), Distributed Traces (sampled request waterfalls via W3C traceparent headers), and Structured NDJSON Logs stamped with the shared trace_id so engineers jump straight from an alert spike to the exact failing database span.
Canonical Production NDJSON Log Schema (1 Line Per Event)
{"ts":"2026-11-17T12:04:11.482Z","level":"error","service":"billing-api","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","route":"/v1/invoices","status":502,"duration_ms":3012,"err":"upstream_timeout"}

Single-line NDJSON streams directly into stdout without breaking multi-line container log collectors.

Telemetry SignalStorage CostPrimary Question AnsweredCopy
Prometheus MetricsUltra-low (aggregated counters/histograms)Is something broken globally right now?
OpenTelemetry TracesMedium (head/tail sampled spans)Which downstream service introduced latency?
Structured NDJSON LogsHighest (contextual payload fields)What exact parameters triggered this failure?

2. Why Averages Lie & How to Compute Prometheus p95/p99 Quantiles

Never compute averages across buckets (`avg(duration)`): always track cumulative Prometheus histogram_quantile() over bounded rate() windows. Watch label cardinality carefullyโ€”placing raw user_id or unbounded UUID paths inside metric labels triggers exponential time-series memory explosion.
PromQL p99 Tail Latency & 5xx Error Budget Rate
# 99th percentile request duration over trailing 5 minutes:
histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

# 5xx Error percentage per service:
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

Aggregates histogram buckets cleanly across pods using "by (le, route)" before quantile interpolation.

3. Log Redaction Discipline: Stripping Authorization Headers & PII

Centralized log indexes quickly turn into compliance liabilities when middleware dumps raw HTTP headers or request bodies containing Authorization: Bearer tokens, session cookies, credit cards, or IBAN numbers. Enforce centralized pino/winston/slog field redaction lists before logs ever hit stdout.
Sensitive FieldRedaction RuleCompliance RiskCopy
headers.authorization / cookieAlways replace with [REDACTED]Session hijacking from read-only log viewers
password / secret / apiKeyBlock at logger serializer levelCredential sprawl across SIEM backups
email / ip / ibanMask last octets or HMAC pseudonymizeGDPR PII retention exposure

4. Alert on User-Facing SLO Burn Rates Rather Than CPU Spikes

Paging an on-call engineer at 3 AM because CPU touched 85% for two minutes causes alert fatigue while users experience zero errors. Trigger wake-up pages strictly on user-facing symptom Service Level Objectives (elevated 5xx ratio or p99 breach) and route resource pressure thresholds into next-morning capacity tickets.
๐Ÿ‘จโ€๐Ÿ’ป
Authored & Reviewed by ChrisMayPixels • Netherlands

Independent software developer focused on client-side architectures, local-first computing, WebAssembly, and privacy-first web systems.

About MayOnlineTools→

๐ŸŽ“Explore More Guides

โœจ