Engineering for Reliability
Distributed tracing with OpenTelemetry and Cloud Trace
Twenty-three substantive episodes on SLOs, telemetry, alerting, investigation, and operations
Twenty-three 2021–2022 episodes connect user-facing reliability targets to metrics, logs, traces, and alerts. Examples show instrumentation, Prometheus collection, quota monitoring, GKE investigation, and observability cost controls.
Google Cloud Tech 14 of 23
In this collection Browse 23 summaries 14 of 23
When a frontend waits for a backend before responding, either service can contribute to the delay a user sees. This episode uses distributed tracing to follow that request across services and separate its total duration from time spent in individual calls 00:00–02:04(opens in a new tab).
Key Points Covered
- Spans time individual operations within one trace: The parent span records the overall user request. Child spans record the calls made to serve it, so their durations can be compared with the total. OpenTelemetry supplies the instrumentation, and an exporter sends the traces to Cloud Trace 00:00–02:04(opens in a new tab).
- Pass context so the backend joins the same trace: In the manual example, the frontend sends identifying context in HTTP headers. The backend extracts it when creating its own spans, connecting the two services' measurements. Custom spans can also time specific backend tasks 02:04–03:07(opens in a new tab).
- Compare durations in a waterfall view: Cloud Trace places the parent span above its children and shows each duration. Added span events—annotations written by the application—provide detail about what happened during an operation, helping the engineer locate major contributors to latency 03:07–04:14(opens in a new tab).
Full video: https://www.youtube.com/watch?v=NQeK8Kbu6UM(opens in a new tab)