Production Observability: Structured JSON Logging, OpenTelemetry Tracing & Prometheus p99 Metrics
Debug distributed outages in minutes instead of hours: emit W3C Trace-Context correlated NDJSON logs without PII leaks, calculate true p95/p99 latency histograms in PromQL, and separate SLO burn-rate alerts from noisy CPU spikes.
1. Logs, Metrics & Distributed Traces Unified via trace_id
traceparent headers), and Structured NDJSON Logs stamped with the shared trace_id so engineers jump straight from an alert spike to the exact failing database span.{"ts":"2026-11-17T12:04:11.482Z","level":"error","service":"billing-api","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","route":"/v1/invoices","status":502,"duration_ms":3012,"err":"upstream_timeout"}Single-line NDJSON streams directly into stdout without breaking multi-line container log collectors.
| Telemetry Signal | Storage Cost | Primary Question Answered | Copy |
|---|---|---|---|
| Prometheus Metrics | Ultra-low (aggregated counters/histograms) | Is something broken globally right now? | |
| OpenTelemetry Traces | Medium (head/tail sampled spans) | Which downstream service introduced latency? | |
| Structured NDJSON Logs | Highest (contextual payload fields) | What exact parameters triggered this failure? |
2. Why Averages Lie & How to Compute Prometheus p95/p99 Quantiles
histogram_quantile() over bounded rate() windows. Watch label cardinality carefullyโplacing raw user_id or unbounded UUID paths inside metric labels triggers exponential time-series memory explosion.# 99th percentile request duration over trailing 5 minutes:
histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
# 5xx Error percentage per service:
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100Aggregates histogram buckets cleanly across pods using "by (le, route)" before quantile interpolation.
3. Log Redaction Discipline: Stripping Authorization Headers & PII
Authorization: Bearer tokens, session cookies, credit cards, or IBAN numbers. Enforce centralized pino/winston/slog field redaction lists before logs ever hit stdout.| Sensitive Field | Redaction Rule | Compliance Risk | Copy |
|---|---|---|---|
| headers.authorization / cookie | Always replace with [REDACTED] | Session hijacking from read-only log viewers | |
| password / secret / apiKey | Block at logger serializer level | Credential sprawl across SIEM backups | |
| email / ip / iban | Mask last octets or HMAC pseudonymize | GDPR PII retention exposure |
4. Alert on User-Facing SLO Burn Rates Rather Than CPU Spikes
Related Free Online Tools
Put these techniques into practice immediately with our privacy-first browser utilities:
Inspect, pretty-print, and validate single structured JSON telemetry events offline.
Convert structured log arrays into newline-delimited NDJSON streams for shipping & replay.
Resolve Unix millisecond and second timestamps from log batches into human UTC & local time.
Independent software developer focused on client-side architectures, local-first computing, WebAssembly, and privacy-first web systems.