Prepare and test your skills

Prepare and test your skills

Choosing and deploying data products means matching each workload to the Google Cloud database, warehouse, messaging, or caching service that fits its structure, scale, and latency needs. Relational services enforce tables and SQL for transactions, non-relational services trade fixed schemas for automatic scale, and messaging plus processing services move data toward analytics. Caching with in-memory stores then speeds up repeated reads in front of slower databases.
Relational database services on Google Cloud are fully managed systems that store data in tables with rows, columns, and enforced schemas for online transaction processing (OLTP). Cloud SQL (opens in a new tab) provides managed MySQL, PostgreSQL, and SQL Server with automated backups, binary logging for point-in-time recovery, high availability configuration, and cross-region replicas. The Enterprise Plus edition for MySQL and PostgreSQL adds a 99.99% availability service-level agreement (SLA), near-zero downtime maintenance, optimized hardware and software, intelligent data caching for read-intensive work, and 35 days of log retention.
Spanner (opens in a new tab) is the mission-critical, global-scale relational service that keeps transactional data strongly consistent across regions. Spanner supports GoogleSQL with query statements that scan tables, data definition language statements that create tables and views, data manipulation statements that insert and update data, access control statements for privileges, and transaction control statements. AlloyDB for PostgreSQL is the fully managed PostgreSQL-compatible service for demanding transactional workloads, while AlloyDB Omni is the downloadable edition that runs in a data center, on a laptop, at the edge, and in any cloud. Spanner databases can be exported in full through Dataflow templates to Avro files in Cloud Storage (opens in a new tab) and re-imported when a full copy is needed.
Non-relational database services store schemaless data that scales automatically without fixed tables or joins. Firestore is the highly scalable NoSQL document database built for automatic scaling, high performance, and ease of application development, and it is the successor to Datastore. Firestore in Datastore mode preserves capabilities for legacy App Engine applications, while new applications use Firestore APIs, with redundancy across Google data centers and automatic encryption at rest.
Firestore maps differently from relational tables because a kind maps to a table, a document or entity maps to a row, a field or property maps to a column, and a document ID maps to a primary key. Entities of the same kind can have different properties, so the model is schemaless and flexible for catalogs and user profiles. Because all queries use prebuilt indexes, Firestore provides atomic transactions, high availability, massive scalability, and strong consistency, but it does not support joins, inequality filtering on multiple properties, or filtering on subquery results. Bigtable is the NoSQL option for data that is not highly structured and for workloads that do not require ACID transactions, with asynchronous replication across zones in a region and backup by export to Cloud Storage as Hadoop sequence files.
The analytics pipeline separates durable messaging, in-flight transformation, and SQL analysis into services that hand data forward in order. Producers publish events once to a Pub/Sub topic or to Google Cloud Managed Service for Apache Kafka, and consumers subscribe independently so ingestion keeps running when downstream jobs restart or fall behind. A Dataflow job built with the Apache Beam software development kits (SDKs) for Java, Python, or Go then reads from the topic, applies filtering and mapping, and writes processed rows to BigQuery for near real-time analytics.
A left-to-right process flow showing producers publishing to a Pub/Sub topic, a Dataflow Beam job filtering and mapping, and BigQuery storing rows for SQL analytics.
BigQuery (opens in a new tab) is the central data warehouse where curated data lands for large-scale SQL analysis. In an extract, load, transform (ELT) flow, data is loaded into BigQuery first and then transformed with SQL, views, routines, and DataFrames. The BigQuery Data Transfer Service automates recurring scheduled loads from sources such as software-as-a-service apps, Cloud Storage buckets, Amazon S3, Google Ads, Google Play, and YouTube Channel reports. Choose messaging when producers and consumers must operate independently, choose Dataflow when data must be cleaned or joined in motion, and choose BigQuery when the goal is centralized SQL analytics.
| Service | Data model and role | Choose when |
|---|---|---|
| Cloud SQL | Managed MySQL, PostgreSQL, SQL Server tables | The workload needs full SQL for OLTP with backups and replicas |
| Spanner and AlloyDB for PostgreSQL | Global-scale or high-performance relational tables | The workload needs strong consistency at global scale or demanding PostgreSQL performance |
| Firestore | Schemaless NoSQL documents | The application needs automatic scaling with flexible fields |
| Bigtable | NoSQL for less-structured data | The workload does not need ACID transactions |
Memorystore for Memcached is the fully managed, distributed in-memory key-value store that keeps frequently used data in memory instead of on disk. Applications connect with standard open source Memcached client libraries, so existing applications can often lift and shift with little code change. Data flows from the primary database into the cache on first read, and later reads flow from the cache to the application to reduce database load.
A Memcached instance represents one Memcached cluster and can contain one node or many nodes with the same memory and vCPUs per node. Keys are divided across all nodes with no replication, so loss of a node means loss of the keys on that node. The instance runs in a single region and the application must run in the same region because cross-region access is not supported, and nodes should be spread across zones for fault tolerance. Memorystore for Memcached will shut down on January 31, 2029, and Google recommends migrating to Memorystore for Valkey.
Exam tip: Choose Cloud SQL when the requirement states full SQL support for OLTP, choose Firestore when the requirement states automatic scaling with a schemaless model, and choose Bigtable when the requirement states no ACID transactions are needed.
Choosing and deploying storage products means matching shared file systems and object storage to access frequency, location, and cost. Filestore and NetApp Volumes provide shared file systems that many clients mount at once, while Cloud Storage classes price object data by how often it is read. Automation and monitoring then keep objects in the right class and region as use changes.
Shared file storage provides a network file system that multiple clients mount and use at the same time. Filestore is the managed service for general shared workloads such as an NFS file share that clients reach over a Virtual Private Cloud (VPC) network through private services access. Google Cloud NetApp Volumes (opens in a new tab) is the enterprise file service for demanding workloads such as electronic design automation, high-performance computing, and legacy migrations with large datasets and intensive metadata operations.
Filestore on a Shared VPC network requires an ordered setup where each step depends on the previous one. First, create a global peering address range named google-service-range in the host project. Then, connect the Shared VPC network to servicenetworking.googleapis.com using that range. Only after the private connection succeeds, create the Filestore instance with its capacity, zone, file share name, IP range, and access controls. Basic-tier instances need a /29 prefix and zonal-tier instances need a /26 prefix, but a larger prefix such as /20 is recommended when many instances or other managed services share the range.
NetApp Volumes organizes capacity into storage pools before volumes are created, so the storage pool is the container that holds performance and capacity for large-capacity volumes. The service is designed to stay available through system failures and to handle very large file counts with consistent performance. Choose Filestore for a straightforward shared file system on a VPC network, and choose NetApp Volumes when many files, large datasets, and heavy metadata work require enterprise scale and resilience.
Cloud Storage classes set storage cost based on how often object data is accessed. Standard storage starts all new objects and fits frequently read data, Nearline storage also fits frequently accessed data, while Coldline storage and Archive storage fit infrequently accessed and long-term retention data at lower storage cost. Storage Intelligence advisor (opens in a new tab) flags a Spike in Class A or B operations on Coldline or Archive data, which means cold data is being read or written frequently and belongs in Standard or Nearline storage.
Autoclass is the bucket-level feature that moves each object automatically based on its own access pattern. Every new object starts in Standard storage even if the write requests another class, and unaccessed objects cool toward a terminal class that is Nearline storage by default or optionally Archive storage. Reading object data moves the object back to Standard storage, while reading or editing only metadata does not trigger a transition. Objects smaller than 128 KiB stay in Standard storage, and soft-deleted objects keep their class until retention ends. Object Lifecycle Management is the alternative bucket policy that deletes objects or moves them to colder classes by fixed rules, such as removing noncurrent versions that accumulate after new uploads.
Location controls latency, egress cost, and residency for Cloud Storage data. Data flows from a bucket in one region to compute in another region when they are separated, which creates cross-region egress charges and added latency. The advisor reports this as a Spike in cross-region egress, for example when a Compute Engine instance in us-central1 frequently reads a bucket in us-east1. The fix is bucket relocation so the bucket and the services that use it sit in the same region.
Growth and throttling signals show when storage is oversized or access patterns exceed sustainable rates. A Total bytes stored increased above trend finding often means noncurrent versions are retained indefinitely, which points to lifecycle rules or Autoclass. A Spike in 429 errors means request throttling from operation spikes, which requires smoothing I/O or aligning hot data with the right class and location rather than simply adding capacity. Centralized oversight across projects, folders, or organizations replaces custom pipelines, with project-level resources protected inside a VPC Service Controls perimeter and broader oversight relying on Identity and Access Management (IAM).
Exam tip: In Autoclass, reading object data returns the object to Standard storage, but reading or editing object metadata does not trigger a transition.
Loading data means staging raw files in Cloud Storage and then moving them into warehouses or processing services with the right tool for the source and size. Small local uploads use command-line tools, recurring warehouse loads use scheduled transfers, and very large external migrations use managed transfer services. Permissions and connectors then control which identities can read buckets and write tables.
Command-line uploads copy local files and folders into Cloud Storage buckets that act as a central landing zone for big data. Cloud Storage (opens in a new tab) buckets feed BigQuery loads, Dataflow staging files, and Managed Service for Apache Spark jobs, and data should stay in the cloud once uploaded because transfers to services in the same location are very fast. Before starting, install the Google Cloud CLI and run gcloud init.
The gcloud storage cp command copies a single file with gcloud storage cp local-file gs://example-bucket and copies between buckets with source and destination gs:// addresses. Add --recursive to copy a whole directory tree, such as gcloud storage cp top-level-dir gs://example-bucket --recursive, because only the flag tells the tool to walk subdirectories. Wildcards select groups of objects in one operation, such as image* to copy or delete only names that start with image. The gcloud storage rsync command synchronizes a directory so the destination matches the source, and adding --delete-unmatched-destination-objects deletes extra files at the destination to make an exact mirror. Large transfers are resumable based on bytes already received, so interrupted uploads and downloads avoid resending completed bytes.
gcloud storage is the recommended current CLI with support for newer features such as soft delete and managed folders and with less manual tuning for speed. gsutil is the legacy tool that lacks those features, needs more manual optimization, and after March 2027 will ship only as a standalone tool through PyPI instead of with the Google Cloud CLI. The legacy tool supports parallel transfers with -m and bulk input with -I, but new work should use gcloud storage cp and gcloud storage rsync.
Loads from Cloud Storage into BigQuery move bucket objects such as Parquet, JSON, and CSV files into BigQuery tables. A load job can replace a table with truncate mode or add to it, and a JSON load can use an explicit schema that defines column names and types. BigQuery can also query bucket files without loading by using a temporary table, an external table, or Hive partitioning, and the BigQuery DataFrames API can turn a CSV file into a DataFrame for Python analysis.
The BigQuery Data Transfer Service (DTS) schedules those loads to run on a recurring managed basis. DTS can load from Cloud Storage on a schedule and can replace extract-and-load pipelines for supported sources including software-as-a-service apps, Teradata, Amazon Redshift, Amazon S3, Google Play, YouTube Channel reports, and Google Ads. A DTS configuration supports backfill runs for historical data, sends run notifications to Pub/Sub on success or failure, and produces log messages for each run. Alternatives include Cloud Storage triggers with Cloud Functions and Managed Airflow with Composer, plus pull models such as SFTP and push models where data flows through Pub/Sub or Apache Kafka into a bronze tier.
Storage Transfer Service (opens in a new tab) is the fully managed service for moving large amounts of object and file data without managing infrastructure or custom code. The service is optimized for transfers over 1 TiB and can move petabytes with parallelism, retries, and validation, but it provides no SLA for transfer performance or latency. Data flows from a configured source to a configured sink after the operator defines both sides once through the console, gcloud, REST APIs, or client libraries.
Supported movements include Amazon S3, Azure Blob Storage, or Cloud Storage to Cloud Storage, on-premises storage to or from Cloud Storage, transfers between on-premises systems, publicly accessible URLs to Cloud Storage, and HDFS to Cloud Storage. Common uses are cloud migration, backup copies to another region, feeding BigQuery or Spark processing, and moving cold data to Cloud Storage for archival. The network path can be Google-managed for simpler cloud-to-cloud moves or self-hosted agents for hybrid cases where the customer needs control over routing and bandwidth. Security uses end-to-end encryption with TLS 1.3, plus integrity validation, automatic retries, and load balancing for reliability.
Managed Service for Apache Spark (opens in a new tab) is the fully managed service for running Hadoop, Spark, and Flink jobs that integrate Cloud Storage and BigQuery. The service uses the BigQuery connector to write Hadoop and Spark data directly to BigQuery and the Cloud Storage connector to read and write bucket data. It favors many ephemeral clusters over one monolithic cluster, so each job gets its own configuration and scale, pays only while running, avoids long-term maintenance, and can keep separate versions for development, testing, and production.
Database ingestion keeps a data lake synchronized through change data capture (CDC), where operational databases emit change events to a Pub/Sub topic. A Spark job subscribes to that topic, processes the stream, and applies changes to the raw datastore in Cloud Storage. Secure access depends on Identity and Access Management roles: a loading identity commonly needs Storage Object Admin for bucket objects and Dataproc Worker for processing jobs, while project setup may need Project Creator, Service Usage Admin, and Create Service Accounts with Project IAM Admin. Access to BigQuery datasets is granted and revoked separately for users, groups, or service accounts.
Exam tip: Storage Transfer Service moves outside data into Cloud Storage, while BigQuery Data Transfer Service moves data into BigQuery on a scheduled, managed basis.
Maintaining multi-region redundancy means keeping data available for global reads and regional failover without losing track of where primary data lives. Cloud SQL uses manual replica promotion across regions, while managed multi-regional services distribute data automatically. Analytical and object storage then add location controls, replication timing, and backups to balance availability, residency, and cost.
Cross-region replication for managed databases keeps a secondary copy ready for global reads and disaster recovery. Cloud SQL high availability protects against a zonal failure by placing a primary instance and a standby instance in two zones of one primary region, synchronized through regional persistent disks. If the whole primary region fails, the Cloud SQL database becomes unavailable because Cloud SQL is a regional service and high availability failover is automatic only within that region.
A Cloud SQL cross-region read replica is an instance in a secondary region that synchronizes from the primary by asynchronous replication. Data flows from the primary instance in the primary region to the replica in the secondary region for reads closer to users, but the replica can lag so the recovery point objective (RPO) after failover is likely non-zero. Operators watch Lag Bytes in monitoring to estimate unreplicated writes that could be lost. Disaster recovery follows an ordered sequence: first the primary region fails, then the team declares a disaster, then the replica in the secondary region is promoted to the new primary, and then clients are reconfigured to the new primary. Promotion is manual, unlike automatic high availability failover, and the database can stay in the secondary region or fall back later. In Enterprise Plus, one replica can be designated as the disaster recovery replica for failover and switchover with role reversal.
Managed multi-regional services avoid that manual promotion because Google distributes the data as part of the service. Cloud Spanner, Cloud Storage, Datastore, and Firestore provide high availability and disaster recovery capable storage without configuring a cross-region replica. For zonal or regional resources, redundancy needs extra work such as snapshots to a multi-regional resource. Compute in another region must still be started and pointed at the surviving data copy.
BigQuery (opens in a new tab) dataset location controls where warehouse data is stored and queried. A regional location is a single geographic place such as Iowa (us-central1), while a multi-region location is a large area such as the United States or Europe that contains two or more places. Data loaded into BigQuery is stored synchronously in two zones of the specified region, so a zonal outage does not cause loss, and data stays within the specified region for residency.
Synchronous replication returns success only after writing to multiple locations, so no committed data is lost but latency can rise by tens or hundreds of milliseconds. Asynchronous replication returns success after one write and replicates afterward, which gives lower latency and higher throughput but risks losing recent writes. BigQuery cross-region replication creates a continuous read-only copy in a secondary region near consumers for collocated joins, but it is not a disaster recovery plan for total-region outages and it also copies corruption. For recovery, use snapshots or exports to Cloud Storage on a schedule set by Cloud Scheduler or Managed Service for Apache Airflow, with formats such as Avro with SNAPPY or JSON with GZIP. If corruption is found within 7 days, snapshot decorators can query the table as of a time before corruption.
Pub/Sub message storage policies keep pipeline messages in specified locations, and Cloud Logging can store log data in a specified region with Admin Activity logs by default and Data Access logs for BigQuery. Together they let ingestion events and audit trails follow the same residency rules as warehouse data. If encryption can mitigate locality limits, Cloud Key Management Service (Cloud KMS) and external key management allow multi-regional services with standard recovery techniques. Otherwise single-region warehouses must plan for size in terabytes to petabytes, append-only history, and controlled extract, transform, load (ETL) paths that can reproduce recent updates.
Dual-region and multi-region buckets in Cloud Storage are location types that keep data redundant across two separate geographic places in one active-active bucket. Applications keep using the same storage path during a regional outage, which is designed for 99.99% availability and a recovery time objective (RTO) of zero for data that finished replicating. Availability means data can be accessed immediately, while durability means data stays intact, and Cloud Storage is designed for at least 99.999999999% annual durability through erasure coding, redundant pieces, checksums, and automatic repair.
A regional bucket stores data redundantly in at least two zones of the selected region and tolerates loss of one zone. Dual-region and multi-region buckets first confirm writes after storing across at least two zones in one region, then replicate asynchronously across regions for protection against region outages. Strong consistency ensures stale versions are not served if a region fails before replication finishes. A dual-region bucket stores objects in two customer-selected regions at higher storage cost, while a multi-region bucket lets Cloud Storage choose data centers within the boundary, separated by at least 100 miles, at lower cost. When a needed pair is not available as a dual-region, separate buckets in each region can be kept in sync with event-driven Storage Transfer Service transfers or cross-bucket replication.
Default cross-region replication targets redundancy for 99.9% of newly written objects within one hour and 100% within 12 hours, and most objects finish within minutes. Turbo replication is the premium option for dual-region buckets that targets 100% of newly written objects replicated to both regions within a 15-minute RPO regardless of size. Any data that has not finished replicating before a region fails stays inaccessible until that region returns, with loss only in the very unlikely case of physical destruction.
Exam tip: Turbo replication applies only to dual-region buckets with a 15-minute RPO, while default cross-region replication targets 99.9% in one hour and 100% in 12 hours.
Continue with this study guide: Maintaining multi-region redundancy across data solutions
Use Cloud SQL when the workload needs full SQL support for online transaction processing (OLTP) with backups and replicas. Use Firestore when the application needs automatic scaling with a schemaless document model, and use Bigtable when the workload involves less-structured data and does not require ACID transactions.
Producers publish events once to a Pub/Sub topic and consumers subscribe independently so ingestion keeps running when downstream jobs restart or fall behind. A Dataflow job then reads from the topic, applies filtering and mapping, and writes processed rows to BigQuery, which serves as the central data warehouse for large-scale SQL analysis.
Every new object starts in Standard storage even if the write requests another class, and unaccessed objects cool toward Nearline storage by default or optionally Archive storage. Reading object data moves the object back to Standard storage, while reading or editing only metadata does not trigger a transition. Objects smaller than 128 KiB stay in Standard storage.
Turbo replication is the premium option for dual-region buckets that targets replication of 100 percent of newly written objects to both regions within a 15-minute recovery point objective (RPO) regardless of size. Default cross-region replication targets redundancy for 99.9 percent of newly written objects within one hour and 100 percent within 12 hours.
Last updated