Observability Engineering Series

Code examples and practical implementations for the blog series "Practical Observability with Python", based on concepts from the book Observability Engineering by Charity Majors, Liz Fong-Jones, and George Miranda.

Prerequisites

• Python 3.13+
• uv for dependency management
• Docker with Docker Compose (chapters 3–11)

Getting Started

There is no project at the repository root. Each chapter is its own uv project with its own pyproject.toml, uv.lock and .python-version (3.13), so install and run from inside the chapter:

cd chapters/ch4-distributed-tracing
uv sync

Chapter 1: Structured Logging

Status: Complete

Moving from "grep-based debugging" to "query-based observability" using structured logging.

• Location: chapters/ch1-structured-logging/
• Key Concepts:
  • The limitations of standard text logs.
  • Implementing structured JSON logging with loguru.
  • Context binding for queryable logs.

For detailed usage instructions, including how to run the examples and configure environment variables, please refer to the Chapter 1 README.

Chapter 2: Context Propagation

Status: Complete

Correlating logs across a request lifecycle using context propagation.

• Location: chapters/ch2-context-propagation/
• Key Concepts:
  • Propagating request context through function calls.
  • Correlation IDs for tracing a request across log lines.

For detailed usage instructions, please refer to the Chapter 2 README.

Chapter 3: OpenTelemetry Tracing

Status: Complete

Automatic tracing with OpenTelemetry — three levels of instrumentation within a single service.

• Location: chapters/ch3-opentelemetry/
• Key Concepts:
  • Zero-code, medium-code, and custom instrumentation.
  • Exporting traces to Jaeger via OTLP.
  • Loguru + OTel trace ID patching for log correlation.

For detailed usage instructions, please refer to the Chapter 3 README.

Chapter 4: Distributed Tracing

Status: Complete

Following a request across service boundaries — from single-service tracing to distributed tracing.

• Location: chapters/ch4-distributed-tracing/
• Key Concepts:
  • W3C Trace Context and the traceparent header.
  • Automatic context propagation via httpx instrumentation.
  • Manual context propagation with explicit inject/extract flow.
  • Two-service setup: API Gateway → Order Service.
  • Custom spans for fine-grained visibility across services.

For detailed usage instructions, please refer to the Chapter 4 README.

Chapter 5: Custom Metrics

Status: Complete

Adding custom metrics with the OpenTelemetry Metrics API and Prometheus as the backend.

• Location: chapters/ch5-metrics/
• Key Concepts:
  • OTel instrument types: Counter, Histogram, Observable Gauge.
  • PrometheusMetricReader exposing /metrics for Prometheus scraping.
  • Middleware recording per-request metrics with low-cardinality attributes.
  • Infrastructure: Jaeger (traces) + Prometheus (metrics).

For detailed usage instructions, please refer to the Chapter 5 README.

Chapter 6: Three Pillars Connected

Status: Complete

Correlating all three telemetry signals — logs, traces, and metrics — via the pivot workflow.

• Location: chapters/ch6-three-pillars/
• Key Concepts:
  • Replacing stdlib logging with Loguru + OTel patcher for trace correlation.
  • Structured JSON logs with trace_id, span_id, span_name.
  • Metric exemplars linking Prometheus metrics to specific traces.
  • The pivot workflow: Metrics → Traces → Logs.

For detailed usage instructions, please refer to the Chapter 6 README.

Chapter 7: Error Handling & Semantic Instrumentation

Status: Complete

Three error-handling patterns for production observability — propagate, catch-and-record, catch-and-continue.

• Location: chapters/ch7-error-handling/
• Key Concepts:
  • span.record_exception() + span.set_status(ERROR) for semantic error recording.
  • Business-context span attributes set before the operation (present even on failure).
  • call_external_api() wrapper auto-instrumenting external dependencies.
  • DB-fail → cache-fallback recovery pattern (/products/{id}).

For detailed usage instructions, please refer to the Chapter 7 README.

Chapter 8: OTel Collector Pipeline

Status: Complete

Routing all telemetry through the OpenTelemetry Collector instead of exporting directly to backends.

• Location: chapters/ch8-otel-collector/
• Key Concepts:
  • App → Collector → backends (Jaeger, Prometheus) instead of direct export.
  • Single OTLP endpoint (localhost:4317) for all signals.
  • Removing prometheus_client — metrics are OTel-native via OTLP.
  • Two configurations: MVP (getting started) and Production (hardened).

For detailed usage instructions, please refer to the Chapter 8 README.

Chapter 9: Sampling & PII Scrubbing

Status: Complete

Keeping 100% of error traces while sampling successes, and redacting sensitive data before telemetry leaves your infrastructure.

• Location: chapters/ch9-sampling-pii/
• Key Concepts:
  • Tail sampling: 100% errors, 100% slow traces, 5% normal traces.
  • PII scrubbing via transform processor: redacts llm.prompt, llm.completion, emails, SSNs.
  • Processor ordering: transform → tail_sampling → batch.
  • GenAI-style /ask endpoint demonstrating PII in span attributes.

For detailed usage instructions, please refer to the Chapter 9 README.

Chapter 10: Dashboards & the RED Method

Status: Complete

Building actionable Grafana dashboards using the RED method (Rate, Errors, Duration) for the API Gateway and Order Service.

• Location: chapters/ch10-dashboards/
• Key Concepts:
  • The RED method: Rate, Errors, Duration (P50, P95, P99).
  • Dashboard-as-code: version-controlled JSON auto-loaded by Grafana on startup.
  • Data source provisioning via YAML (Prometheus + Jaeger).
  • Inverted-pyramid layout: stat panels → time series → latency heatmap and top errors.

For detailed usage instructions, please refer to the Chapter 10 README.

Chapter 11: Alerting & Service Level Objectives (SLOs)

Status: Complete

Defining SLIs and SLOs, configuring symptom-based alerts in Grafana, and building error budget burn rate alerts that page you before users notice.

• Location: chapters/ch11-alerting-slos/
• Key Concepts:
  • 99% availability SLO with error budget tracking.
  • Multi-window (1h + 5m) error budget burn rate alert at 14.4x.
  • Severity-based routing template: local dev sends every alert to a dev-null contact point; production routing (critical → PagerDuty, all → Slack) is provided commented out in policies.yaml.
  • promtool unit tests for alert rules.

For detailed usage instructions, please refer to the Chapter 11 README.

Sponsor

If this project helps you, consider sponsoring my work on GitHub. Your support helps fund maintenance, documentation, and continued development.