Architecting with Google Cloud — selected guide
How to migrate a data warehouse to BigQuery
Ten case studies on scale, service boundaries, data platforms, machine learning, and security
Ten 2021–2023 case studies explain how teams handle global storage, concurrent requests, service boundaries, media ingestion, analytics migration, machine learning, healthcare data, and security response. Results describe the participating organizations' systems, not universal performance guarantees.
In this collection Browse 10 summaries 7 of 10
Moving a data warehouse means preserving the reports and applications that depend on it, not just copying tables. Gary Moriali, a health insurer's data services director, describes an in-progress 2021 move to BigQuery that kept the old and new systems running together while the team compared their data.
Key Points Covered
- Assess everything that reads and writes the warehouse. Moriali says the team and a Google business partner inventoried consuming applications and inbound and outbound processes, rather than treating the database as an isolated migration. Networking and data-center connections came before data transfer. 02:06–03:09(opens in a new tab)
- Move data through checks and staged layers. Files first landed in Cloud Storage. Dataflow performed quality checks, Cloud Composer scheduled the work, and BigQuery separated staging, core data, and the information layer exposed to users. 02:06–04:11(opens in a new tab)
- Compare before redirecting consumers. A comparison tool running on Google Kubernetes Engine checked the old and new systems during coexistence. The intended result was for reporting tools and applications to point at BigQuery and continue their existing work; the interview does not report a completed cutover. 03:09–05:14(opens in a new tab)
- The destination also supports ongoing ingestion and modeling. Provider, member, and partner data entered through secure file transfers and APIs, then Dataproc loaded it into BigQuery. Data scientists used Jupyter notebooks and Vertex AI Pipelines, adding resources such as GPUs when needed rather than being limited to fixed on-premises hardware. 04:11–06:19(opens in a new tab)
Full video: https://www.youtube.com/watch?v=lc68XluDeH4(opens in a new tab)