with-agents

Engineering for Reliability

Defining SLIs with custom metrics

Twenty-three substantive episodes on SLOs, telemetry, alerting, investigation, and operations

Twenty-three 2021–2022 episodes connect user-facing reliability targets to metrics, logs, traces, and alerts. Examples show instrumentation, Prometheus collection, quota monitoring, GKE investigation, and observability cost controls.

Google Cloud Tech 18 of 23
In this collection Browse 23 summaries 18 of 23

Platform measurements such as load-balancer request counts may not capture the transactions an application needs to track. This episode shows how custom metrics become service-level indicators (SLIs), measurements of reliability, and how service-level objectives (SLOs) set targets for them 00:00–01:02(opens in a new tab).

Key Points Covered

  • Prometheus reads an endpoint exposed by the application: The Node.js example counts events and records latency in a histogram. It exposes those values on a metrics endpoint that Prometheus scrapes—reads to collect measurements. A collector added to the Prometheus server forwards the data to Cloud Monitoring in the 2021 setup 01:02–03:08(opens in a new tab).
  • A latency distribution can measure compliance with a threshold: A “distribution cut” counts the fraction of measured values within a selected range. The Prometheus example sets a target of 98% of responses completing within 500 milliseconds over a rolling 28-day period 02:04–03:08(opens in a new tab).
  • Direct export is a different collection path: The separate OpenCensus example sends measurements from the application directly to Cloud Monitoring without scraping. OpenCensus is a legacy library; this is the video's historical alternative to maintaining a Prometheus server, not current library-selection advice 02:04–04:11(opens in a new tab).
  • Choose histogram ranges around the question being asked: Buckets group measurements into ranges. The presenter recommends boundaries that distinguish requests inside and outside the SLO threshold, plus additional buckets for examining slow responses, such as the 99th-percentile latency 03:08–04:11(opens in a new tab).

Full video: https://www.youtube.com/watch?v=eCcsxDln6I0(opens in a new tab)

Search

Search field guides and resources across the site.

    Keyboard shortcuts

    Press ? to open this guide (usually Shift + /). Close it before using other site shortcuts.

    Show keyboard shortcuts
    ?
    Search site
    Ctrl + K
    Close the current dialog or menu
    Esc

    Site shortcuts are ignored while typing. Browser and extension shortcuts may take priority.