Engineering for Reliability
Creating custom metrics with OpenTelemetry
Twenty-three substantive episodes on SLOs, telemetry, alerting, investigation, and operations
Twenty-three 2021–2022 episodes connect user-facing reliability targets to metrics, logs, traces, and alerts. Examples show instrumentation, Prometheus collection, quota monitoring, GKE investigation, and observability cost controls.
In this collection Browse 23 summaries 23 of 23
Yuri, a site reliability engineer at Google Cloud, adds measurements to a Python web service to answer three questions: how many requests arrive, how many fail, and how long processing takes. OpenTelemetry records those measurements in the application and sends them through a separate Collector to Cloud Monitoring 00:00–02:04(opens in a new tab).
Key Points Covered
- Count events and record a distribution of durations: The application increments counters for requests and errors. It records processing times in a histogram, preserving a distribution of latency measurements rather than only a request count 01:03–02:04(opens in a new tab).
- Attach service information before exporting: The application's metric setup includes information identifying the service, so its measurements can be found and grouped later. Its exporter sends the recorded values to the Collector 01:03–02:04(opens in a new tab).
- Deploy the Collector separately from the application: In the demo, three copies of the application run on Google Kubernetes Engine (GKE). The Collector runs alongside them in the same namespace and forwards their measurements to Cloud Monitoring 02:04–03:10(opens in a new tab).
- Turn measurements into health signals: Cloud Monitoring divides failed requests by total requests to show the error ratio and displays the latency histogram as a heat map. The application records the measurements; the monitoring backend performs the displayed analysis 03:10–04:12(opens in a new tab).
Full video: https://www.youtube.com/watch?v=SCNObWlILqk(opens in a new tab)