Engineering for Reliability
Manage GKE services with Cloud Operations
Twenty-three substantive episodes on SLOs, telemetry, alerting, investigation, and operations
Twenty-three 2021–2022 episodes connect user-facing reliability targets to metrics, logs, traces, and alerts. Examples show instrumentation, Prometheus collection, quota monitoring, GKE investigation, and observability cost controls.
Google Cloud Tech 15 of 23
In this collection Browse 23 summaries 15 of 23
Healthy infrastructure does not necessarily mean users are receiving a reliable service. This Google Kubernetes Engine (GKE) demonstration starts with a service reliability alert, traces it to a recent deployment, and rolls that deployment back to restore service 00:00–03:10(opens in a new tab).
Key Points Covered
- Measure the service users experience: A service-level indicator (SLI) measures reliability; a service-level objective (SLO) sets its target. Infrastructure measurements can help explain a failure, but they do not replace these service-level signals 00:00–01:04(opens in a new tab).
- Correlate symptoms with workload changes: Revision history shows a rollout shortly before the alert, making it a strong suspect 01:04–03:10(opens in a new tab).
- Rollback is a mitigation and a test: In the example, the SLI recovers and the alert stops after rollback 02:07–03:10(opens in a new tab).
- Interpretation: The timing and recovery make the rollout a strong suspect, but the demonstration does not identify the code defect that caused the failure 02:07–03:10(opens in a new tab).
Full video: https://www.youtube.com/watch?v=L6JnAhp3E-c(opens in a new tab)