Engineering for Reliability
Managing GKE infrastructure at scale
Twenty-three substantive episodes on SLOs, telemetry, alerting, investigation, and operations
Twenty-three 2021–2022 episodes connect user-facing reliability targets to metrics, logs, traces, and alerts. Examples show instrumentation, Prometheus collection, quota monitoring, GKE investigation, and observability cost controls.
Google Cloud Tech 16 of 23
In this collection Browse 23 summaries 16 of 23
When an application on Google Kubernetes Engine (GKE) has a resource problem, operators need to distinguish that application's behavior from problems on the node running it. Two dashboard examples compare application and node measurements to narrow the investigation 00:00–03:11(opens in a new tab).
Key Points Covered
- Start broad, then narrow ownership: Alert timelines and workload search locate a namespace and workload from sparse reports 00:00–01:03(opens in a new tab).
- Compare application memory with node memory: The workload has a memory alert, but its nodes show no corresponding memory problem. The presenter therefore treats this incident as local to the service. That narrows the likely cause rather than proving infrastructure was uninvolved 01:03–02:08(opens in a new tab).
- Separate services during a broader alert: A node has high CPU use. Some services on it show no increase in container CPU use, while at least two do. The comparison identifies which services share the symptom; it does not establish which service caused the contention 02:08–03:11(opens in a new tab).
Full video: https://www.youtube.com/watch?v=lHCB_i_rWAI(opens in a new tab)