Prepare and test your skills

Prepare and test your skills

Core Google Cloud storage and database services are divided by data shape and type of work. Cloud Storage (opens in a new tab) holds unstructured objects such as files, images, backups, and static website assets. BigQuery (opens in a new tab) holds large analytical datasets for queries and reporting, while relational databases such as Cloud SQL (opens in a new tab) and Spanner hold operational records for day-to-day transactions.
Cloud Storage is low-cost, highly durable object storage for many data types. Data is stored as objects inside buckets in a flat hierarchy, where each object has a globally unique ID plus system-assigned and user-defined metadata. Applications access objects by ID using REST APIs or client libraries, and data can be reached from inside and outside Google Cloud.
Cloud Storage protects data with cross-region replication and version history. Dual-regions and multi-regions automatically replicate data across geographically separated regions for 99.99% availability, and Turbo replication replicates 100% of newly written objects between dual-region pairs within 15 minutes. Object Versioning and soft delete keep previous versions and recently deleted objects so data can be rolled back after accidental overwrites or ransomware events.
Cloud Storage controls cost by matching access frequency to a storage class, and data flows from the bucket to lifecycle controls that manage its lifetime:
Autoclass automatically moves objects between classes based on access patterns, while Object Lifecycle Management deletes or transitions objects using custom rule-based policies. Buckets also connect to other services: buckets serve static HTML, CSS, JavaScript, and media directly for static website hosting, act as backends for Cloud CDN (opens in a new tab) and Cloud Load Balancing for global low-latency delivery, and emit creation, deletion, or metadata events through Eventarc and Pub/Sub to trigger Cloud Functions or Cloud Run services. Signed URLs provide time-limited read or write access to private objects without requiring Google credentials.
BigQuery is a fully managed, highly scalable data warehouse with built-in machine learning (ML). BigQuery storage is optimized for analytic queries over large datasets with high-throughput streaming ingestion and high-throughput reads. Choose BigQuery when the need is analytical — scanning, aggregating, and analyzing large volumes of structured data — rather than processing single operational transactions.
BigQuery decouples storage and compute so each side scales independently. Storage flows into Colossus, where data is compressed and stored in a columnar format called Capacitor, and BigQuery operates directly on compressed data without decompressing it. Compute flows through the Dremel execution engine, where each query uses slots, which are units of computation that include CPU and RAM, with autoscaling for fast petabyte-scale queries.
Data in BigQuery is organized under datasets as the highest-level container for controlling access to tables, with schemas, labels, and partitioning to improve query performance and cost and manage information lifecycle. Storage resources are allocated as data is consumed and deallocated as data is removed or tables are dropped. BigQuery connects directly to Cloud Storage: data flows from Cloud Storage into BigQuery through loads, federated queries, and transfers, and query results flow from BigQuery back to Cloud Storage through exports.
Cloud SQL is a fully managed relational database service for MySQL, PostgreSQL, and SQL Server. Cloud SQL offers separate options including Cloud SQL for MySQL, Cloud SQL for PostgreSQL, and Cloud SQL for SQL Server. Data is stored in structured tables with schemas, which fits operational workloads such as orders, accounts, and inventories that need row-level reads and writes.
Cloud SQL depends on Cloud Storage for bulk movement, where import data flows from a Cloud Storage bucket into the database and export data flows from the database back to Cloud Storage. Existing databases can move to Cloud SQL with Database Migration Service, which provides serverless migration with minimal downtime, and ongoing changes can be captured with Datastream change data capture (CDC) and replication. Choose Cloud SQL when the application needs a regional relational database with a specific MySQL, PostgreSQL, or SQL Server engine and standard transactional consistency.
Spanner is a mission-critical, global-scale relational database service for applications. Spanner is relational like Cloud SQL, but it is designed to scale transaction processing beyond a single region to a global footprint. That global scale supports operational applications that must serve users and writes across widely separated regions while remaining available through regional outages.
Choose Spanner when operational data needs both relational structure and global-scale growth, and choose Cloud SQL when a single-region MySQL, PostgreSQL, or SQL Server instance meets the workload. Both Spanner and Cloud SQL serve operational needs, where applications read and write small sets of current records, while BigQuery serves analytical needs over large historical datasets.
Exam tip: Choose Cloud SQL or Spanner for operational transactions on current records, BigQuery for analytical queries over large datasets, and Cloud Storage for objects such as files, backups, and static assets.
Choosing by structure and workload means matching data shape and access pattern to the right core service. The four core services differ in structure, scale, and whether they fit operational or analytical use, so the choice of storage strategy (opens in a new tab) depends on what the data looks like and how it will be queried:
A decision tree branching on data shape and workload to choose Cloud Storage for objects, BigQuery for analytics, Cloud SQL for regional transactions, or Spanner for global transactions.
| Service | Data structure | Scales to | Choose when |
|---|---|---|---|
| Cloud Storage | Objects in buckets with IDs and metadata | Virtually unlimited objects, with cross-region replication | The data is unstructured files, media, backups, archives, or static web assets |
| BigQuery | Columnar analytical tables in datasets | Petabyte-scale analytics with decoupled storage and compute | The work is analytical queries, reporting, and ML over large datasets |
| Cloud SQL | Relational tables for MySQL, PostgreSQL, or SQL Server | Regional transactional workloads | The work is operational transactions with a specific relational engine |
| Spanner | Relational tables for global applications | Mission-critical global scale | The work is operational transactions that must grow and stay available globally |
Matching business use cases to data management solutions means choosing storage and database services based on performance, cost, and scalability needs. Different workloads need different guarantees for speed, durability, sharing, and geographic reach. Managed services reduce operational work while providing built-in replication, backups, and availability backed by service agreements.
Application backends on VMs, containers, and serverless define where application code runs and how much infrastructure the team manages. Compute Engine (opens in a new tab) provides virtual machines (VMs) where teams control the operating system and install databases or file systems directly. Google Kubernetes Engine (GKE) runs containerized applications and automates deploying, scaling, and managing containers. Cloud Run is a serverless platform where code is deployed to a preconfigured environment with minimal infrastructure management.
The decision involves a trade-off between configuration flexibility and management effort:
Choose VMs or containers when custom configuration or direct access to the operating system is required, and choose serverless when low management effort matters more than low-level control.
Block storage attached to virtual machines provides persistent disks for the operating system, applications, and databases running on a single VM. Persistent Disk in its zonal form stores data within one zone and was used for all tiers in the reference architecture. Persistent Disk in its regional form provides more durable storage through synchronous replication of data across two zones within a region. Synchronous replication means every write is copied to both zones before it completes, so the volume survives a zone outage.
Google Cloud Hyperdisk provides higher performance, flexibility, and efficiency than Persistent Disk for demanding workloads. With Hyperdisk Balanced, IOPS and throughput can be provisioned separately and changed dynamically, which lets one volume be tuned for a wide variety of workloads. Choose zonal disks for the lowest cost when zone-level durability is enough, regional disks when synchronous protection across two zones is needed, and Hyperdisk Balanced when performance must be tuned independently for IOPS and throughput.
Shared file storage across multiple VMs allows many machines in a region to mount and use the same files at the same time. Filestore regional instances store data that is replicated synchronously across three zones within a region, which provides high availability and robustness against zone outages. Data flows from application VMs to the Filestore instance and back, so shared configuration files, common tools and utilities, and centralized logs can be stored once and mounted on every VM in the web or application tier. A Filestore instance can also be replicated to a different region for robustness against a full region outage.
Google Cloud NetApp Volumes provides fully managed file storage for specialized database needs. When the database is Microsoft SQL Server and requires access to the operating system or a failover cluster instance (FCI), NetApp Volumes provides continuous availability SMB storage for the database. Choose Filestore when many VMs need the same shared files in a region, and choose NetApp Volumes when a SQL Server cluster depends on continuously available shared SMB storage.
Object storage for unstructured and distributed data provides low-cost storage that is replicated across zones or regions. Cloud Storage stores objects in regional, dual-region, or multi-region buckets with different replication behavior. Data in regional buckets is replicated synchronously across the zones in the region, so the data remains available if one zone fails. Data in dual-region or multi-region buckets is stored redundantly in at least two separate geographic locations, where metadata is written synchronously across regions and data is replicated asynchronously.
Dual-region buckets can use turbo replication, which ensures that objects are replicated across the region pair with a recovery point objective (RPO) of 15 minutes. RPO measures how much data could be lost, so a 15-minute RPO means replication lags by no more than 15 minutes. Choose regional buckets when synchronous protection within one region is enough, and choose dual-region or multi-region buckets when the business needs survival of a regional disruption and low-cost geographic redundancy.
Managed relational and global databases remove much of the work of installing, patching, monitoring, backing up, and recovering databases. Cloud SQL is the managed relational service, and Cloud SQL for SQL Server is recommended when the database is Microsoft SQL Server. A third-party database deployed on Compute Engine VMs requires the team to perform updates, monitor availability, perform backups, and recover from failures. The managed service depends on Google for those operations, so the team trades direct operating-system control for lower operational cost and built-in resilience.
Spanner is a managed multi-regional service for workloads where data replication across regions is critical. When an active-active topology across two regions is needed, traffic flows to both regions at the same time and Spanner keeps the data replicated between them. Cloud Storage provides the same multi-regional benefit for object data. Choose Cloud SQL when the use case is a standard relational application backend, and choose Spanner when the use case is global transaction management that must stay available and consistent across regions.
Balancing performance, cost, and availability requires distinguishing high availability (HA) (opens in a new tab) from disaster recovery (DR) and weighing cost against downtime. HA generally applies to regional deployments where managed services provide synchronous replication between zones with zero or near-zero recovery time objective (RTO) and RPO. RTO measures how quickly service resumes, so synchronous zone replication supports an active-active design with no single point of failure. DR generally applies to multi-region deployments where replication is typically asynchronous, so teams must define a replication strategy and accept manual or semi-automated failover steps.
Increasing availability generally costs more but helps meet regulatory and business limits. For example, 99.9% availability allows about 86 seconds of downtime per day, while 99% allows about 864 seconds, which is ten times more downtime. The following comparison summarizes storage choices by replication and fit:
| Storage option | Replication behavior | Choose when |
|---|---|---|
| Persistent Disk regional | Synchronous replication across two zones in a region | Durable VM-attached storage must survive a zone outage |
| Google Cloud Hyperdisk Balanced | High-performance volume with separately provisioned IOPS and throughput | Performance must be tuned for demanding or varied workloads |
| Cloud Storage multi-region or dual-region | Redundant storage in two or more locations with synchronous metadata and asynchronous data | Low-cost unstructured data must survive regional disruption |
| Filestore regional | Synchronous replication across three zones with mounting on multiple VMs | Shared files must be highly available to many VMs in a region |
Use managed multi-regional services for critical data, store Persistent Disk snapshots in multi-region locations, and test DR plans regularly to validate RTO and RPO against business and regulatory needs.
Exam tip: HA means synchronous replication between zones in one region with near-zero RTO and RPO, while DR means replication across regions that is typically asynchronous with defined RTO and RPO.
Data processing and transformation is the work of moving raw data through pipelines that clean, reshape, and enrich it so it can be analyzed. Google Cloud splits this work across four services: Pub/Sub ingests streaming events, Dataflow runs unified batch and streaming pipelines, Dataproc runs managed open-source big-data clusters, and Dataplex governs data across storage systems.
Pub/Sub is a messaging service that moves events from producers to consumers without the two sides needing to know about each other. A publisher sends a message to a named topic, and the message is delivered to every subscription attached to that topic. Each subscriber reads from its own subscription, so one slow consumer does not block the others.
This decoupling matters because the publisher does not wait for a subscriber to be online. Messages are held until a subscriber reads them, which lets systems absorb bursts of traffic. Choose Pub/Sub when the job is capturing a continuous stream of events — clicks, sensor readings, log lines — and feeding them to one or more downstream systems.
Dataflow is a serverless service for running data pipelines that handle both batch and streaming data with the same code. A pipeline is written with Apache Beam and then submitted to Dataflow, which provisions the compute, scales it up or down based on load, and shuts it down when the job finishes.
A streaming pipeline moves data through an ordered set of stages:
Because Dataflow manages the workers, teams do not size or patch servers. Choose Dataflow when the same transformation logic must run over both real-time streams and stored batches, or when pipeline load varies unpredictably.
Dataproc is a managed service for running open-source big-data tools such as Apache Spark, Apache Hadoop, and Apache Hive. A user creates a cluster, submits jobs to it, and the cluster runs the work using familiar frameworks. Dataproc handles the cluster setup, and clusters can be sized up or down or shut down when idle.
Dataproc is a good fit when existing Spark or Hadoop jobs need to move to the cloud without being rewritten. Choose Dataproc for batch processing that uses those frameworks, and choose Dataflow when the pipeline should be serverless and unified across batch and streaming.
Dataplex is a data governance service that provides a single view across data stored in different systems. It organizes data into lakes, zones, and assets, and it applies metadata, quality checks, and lineage tracking across those layers. Governance policies apply to the assets inside a zone, so the same rules follow the data regardless of where it physically sits.
Dataplex does not move or transform data itself. Instead, it sits above storage and processing services and answers questions such as where a dataset lives, who owns it, and whether it meets quality rules. Choose Dataplex when the requirement is to discover, classify, and govern data spread across multiple storage systems.
Exam tip: Dataflow and Dataproc both process data, but Dataflow is serverless and unifies batch with streaming, while Dataproc runs managed Spark and Hadoop clusters for existing big-data jobs.
Choosing the right tool means matching each stage of the pipeline to the service built for that role. The four services divide the pipeline by role:
| Service | Primary role | Choose when |
|---|---|---|
| Pub/Sub | Streaming ingestion | Events must be captured continuously and delivered to one or more consumers |
| Dataflow | Unified batch and streaming pipelines | The same transformation logic must run on streams and stored data without managing servers |
| Dataproc | Managed Spark and Hadoop clusters | Existing big-data framework jobs need to run with minimal rewriting |
| Dataplex | Data governance | Data spread across systems needs discovery, metadata, quality, and lineage control |
The services compose into one pipeline: Pub/Sub ingests events and passes them to Dataflow, Dataflow transforms and writes results to storage, Dataproc can process stored batches with Spark, and Dataplex governs the datasets that all three produce.
A left-to-right pipeline showing producers sending events to Pub/Sub, which delivers messages to Dataflow for transformation; Dataflow writes results to Cloud Storage, from where loads and federated queries feed BigQuery and stored batches feed a Dataproc Spark job whose output also reaches BigQuery, all governed by Dataplex for metadata, quality, and lineage.
Choose Cloud SQL when the application needs a regional relational database with a specific MySQL, PostgreSQL, or SQL Server engine and standard transactional consistency. Choose Spanner when operational data needs both relational structure and global-scale growth with availability through regional outages. Both Cloud SQL and Spanner serve operational needs where applications read and write small sets of current records, while BigQuery serves analytical needs over large historical datasets.
Cloud Storage holds unstructured objects such as files, images, backups, and static website assets in buckets, while BigQuery holds large analytical datasets for queries and reporting. Choose BigQuery when the need is analytical, scanning, aggregating, and analyzing large volumes of structured data rather than processing single operational transactions. BigQuery decouples storage and compute so each side scales independently, with storage in Colossus in a columnar format called Capacitor and compute through the Dremel execution engine using slots.
Choose Dataflow when the same transformation logic must run over both real-time streams and stored batches without managing servers. Dataflow is a serverless service for running unified batch and streaming pipelines with Apache Beam that provisions, scales, and shuts down compute automatically. Choose Dataproc when existing Apache Spark, Apache Hadoop, or Apache Hive jobs need to move to the cloud without being rewritten on managed clusters.
A retail company collects real-time telemetry and purchasing event streams from millions of mobile devices. The engineering team needs a serverless data processing service that can execute complex streaming transformations, windowing, and data enrichment before loading the processed output into BigQuery.
Which Google Cloud service should the company implement for this processing pipeline?
Dataflow
Datastream
Cloud Storage
Pub/Sub
Last updated