Engineering for Reliability
Defining SLIs with custom metrics
Twenty-three substantive episodes on SLOs, telemetry, alerting, investigation, and operations
Twenty-three 2021–2022 episodes connect user-facing reliability targets to metrics, logs, traces, and alerts. Examples show instrumentation, Prometheus collection, quota monitoring, GKE investigation, and observability cost controls.
In this collection Browse 23 summaries 18 of 23
Platform measurements such as load-balancer request counts may not capture the transactions an application needs to track. This episode shows how custom metrics become service-level indicators (SLIs), measurements of reliability, and how service-level objectives (SLOs) set targets for them 00:00–01:02(opens in a new tab).
Key Points Covered
- Prometheus reads an endpoint exposed by the application: The Node.js example counts events and records latency in a histogram. It exposes those values on a metrics endpoint that Prometheus scrapes—reads to collect measurements. A collector added to the Prometheus server forwards the data to Cloud Monitoring in the 2021 setup 01:02–03:08(opens in a new tab).
- A latency distribution can measure compliance with a threshold: A “distribution cut” counts the fraction of measured values within a selected range. The Prometheus example sets a target of 98% of responses completing within 500 milliseconds over a rolling 28-day period 02:04–03:08(opens in a new tab).
- Direct export is a different collection path: The separate OpenCensus example sends measurements from the application directly to Cloud Monitoring without scraping. OpenCensus is a legacy library; this is the video's historical alternative to maintaining a Prometheus server, not current library-selection advice 02:04–04:11(opens in a new tab).
- Choose histogram ranges around the question being asked: Buckets group measurements into ranges. The presenter recommends boundaries that distinguish requests inside and outside the SLO threshold, plus additional buckets for examining slow responses, such as the 99th-percentile latency 03:08–04:11(opens in a new tab).
Full video: https://www.youtube.com/watch?v=eCcsxDln6I0(opens in a new tab)