Prepare and test your skills
Prepare and test your skills
Storage service scalability is the ability of a storage system to handle growing data volume and read-write speed without loss of performance or cost control. BigQuery (opens in a new tab) scales analytical workloads by separating storage from compute, Cloud Bigtable (Bigtable) (opens in a new tab) scales high-throughput NoSQL workloads by adding nodes and clusters, Cloud Spanner (Spanner) (opens in a new tab) scales relational workloads through configured nodes, and Cloud Storage (opens in a new tab) scales object storage automatically with usage. Matching long-term data projections to these limits prevents forced migrations and over-provisioning.
BigQuery scaling for analytical growth is serverless scaling that decouples data storage in Colossus from query execution in Dremel. Data flows from ingestion into Colossus where it is compressed and stored in a columnar format called Capacitor, and query requests then flow from Dremel compute slots to that stored data without moving or resizing storage. BigQuery operates directly on compressed Capacitor data, gathers column statistics during load, and allocates compute in units called slots that bundle CPU and RAM. Storage resources are allocated as data is consumed and deallocated when data is removed or tables are dropped, so dozens of petabytes can be stored without adding compute.
BigQuery manages query performance at the query level with in-memory execution, a multilevel Dremel tree, automatic Capacitor optimization, and autoscaling over a 1 petabit per second network. Each query uses a number of slots based on its type and complexity, and slots are assigned by a fair scheduler that continuously calculates demand during execution. Projects are limited by default to a maximum of 2,000 slots under on-demand pricing, where charges are based on bytes processed, while capacity-based pricing reserves slots for predictable throughput. Datasets organize access to tables, and schemas, labels, and partitioning further organize data and improve query performance, cost, and lifecycle management.
Bigtable and Spanner scaling for operational growth is the way throughput grows by adding Bigtable nodes and clusters or by adding Spanner compute nodes. Bigtable production instances require a minimum of two clusters in unique zones within a region, and replication between the clusters is automatically managed by Google to create a separate copy in each zone. Incoming traffic flows from clients to the primary cluster and can fail over to another cluster in the same instance when needed. When low latency depends on storing indexes in RAM, Bigtable machines must be scaled vertically together with storage capacity to hold the primary index, and a single node can handle data chunks of several hundred terabytes or even petabytes.
Spanner read and write throughput depends on the number of configured compute nodes, which also controls backup and restore speed. Data flows from Spanner through a Dataflow connector or export template into Cloud Storage as Avro files for backup, and data flows back from Cloud Storage into a new Spanner database during re-import. Spanner supports consistent managed backups retained for up to 1 year that restore by mounting the backup without copying data, point-in-time recovery to any point in the past up to 7 days, and commit timestamps that allow selection of only rows added or modified since the last full backup for incremental growth.
Block, file, and object capacity limits define the ceiling and scaling model that determine how each storage type handles data growth. Persistent Disk provides network block storage, Google Cloud Hyperdisk (Hyperdisk) provides network block storage with storage pools and dynamic resizing, and Local SSD provides ephemeral locally attached block storage that is fixed at creation. The three block options differ in per-disk limits, per-VM limits, and whether they can grow after creation.
| Service | Capacity limit | Scaling behavior |
|---|---|---|
| Persistent Disk | 10 GiB to 64 TiB per disk, up to 257 TiB per VM | Scale up, add and remove disks, autoscale |
| Google Cloud Hyperdisk (Hyperdisk) | 4 GiB to 64 TiB per disk, up to 512 TiB per VM, 10 TiB to 1 PiB per storage pool | Scale up with dynamic resizing |
| Local SSD | 375 GiB per disk, up to 12 TiB per VM | Not scalable |
File storage mounts to multiple clients and scales by instance or pool rather than automatically with usage. The three managed file options differ by ceiling and flexibility:
Cloud Storage provides object storage with no lower or upper limit that scales automatically based on usage. Objects are stored in a flat hierarchy of buckets and accessed by globally unique IDs from inside or outside Google Cloud, with optional multi-region or dual-region redundancy for maximum reliability. Because Cloud Storage grows without provisioning disks or instances, it fits unpredictable growth, static content, backups, and data lakes where block or file ceilings would require re-architecture.
Exam tip: Cloud Storage scales automatically with no capacity limit, while Local SSD is fixed at creation and cannot be scaled.
Cost-efficient growth with storage classes is the practice of matching Cloud Storage price to access frequency and minimum retention as retained data grows. Data flows from the application into a bucket assigned to a class, and the Autoclass feature moves each object between classes automatically based on its access pattern when retention is unpredictable. Frequently accessed data for analytics, lakes, websites, and streaming uses Standard, with Cloud CDN caching near clients and Rapid Cache improving read-heavy machine learning and analytics reads. BigQuery provides a parallel benefit where a table unedited for 90 consecutive days drops to long-term storage pricing by 50 percent without loss of performance.
| Class | Use when | Minimum retention |
|---|---|---|
| Autoclass | Access frequency varies or retention is unknown | Automatic transitions per object |
| Standard | Data is accessed frequently for analytics, lakes, websites, and streaming | No minimum |
| Nearline | Low-cost storage for infrequently accessed backups and long-tail content | At least 30 days |
| Coldline | Low-cost storage for infrequently accessed disaster recovery data | At least 90 days |
| Archive | Lowest-cost storage for regulatory archives | At least 365 days |
Data lifecycle and tiering is the practice of keeping fresh data in high-performance storage while automatically moving aging or rarely used data to lower-cost storage. BigQuery (opens in a new tab) manages active analytic data with partitioning and automatic storage optimization, while Cloud Storage (opens in a new tab) provides tiered classes and lifecycle rules for backups and exports. Together they let growth in volume be absorbed without growth in cost, because unchanged data is backed up once and only new or updated partitions move through the pipeline.
BigQuery partitioning schemes divide a large table into smaller segments by a date, timestamp, or integer column to improve query performance, lower cost, and manage information lifecycle. When a query filters on the partitioning column, BigQuery scans only the matching partitions instead of the whole table, and storage resources are allocated as data is consumed and deallocated when data or tables are removed. Partitioning can be combined with clustering on up to four different columns, where data inside each partition is sorted into blocks by the clustering values so filters and aggregations skip unnecessary blocks.
BigQuery partitioning directly supports tiering because lifecycle work can target partitions rather than full tables. BigQuery handles reclustering automatically without using allocated slots, so maintenance does not compete with queries. For an append-only table partitioned by date, such as an orders table where new orders land in the current-date partition, a backup or export only needs the new partition plus any recent partitions that can still change. If returns allow updates within 7 days, the export must include the partitions for the last 7 days to avoid missing updates.
Cloud Storage classes for tiered retention store BigQuery export files in progressively cheaper classes as access frequency drops and retention length grows. The Autoclass feature automatically transitions objects in a bucket to the appropriate class based on each object's access pattern when the retention period is unknown. When the pattern is known, an explicit class is chosen by minimum storage duration and a lifecycle policy deletes objects after the retention requirement ends. Export files stored in Nearline Storage or Coldline Storage cost less than Standard storage, and columnar formats such as Parquet and ORC add high compression plus interoperability with engines such as Hive and Presto.
| Storage class | Minimum duration | Choose when |
|---|---|---|
| Standard | None stated | Data is accessed frequently for analytics, data lakes, websites, or streaming |
| Nearline Storage | At least 30 days | Data is infrequently accessed, such as backups and long-tail multimedia content |
| Coldline Storage | At least 90 days | Data is rarely accessed, such as disaster recovery copies |
| Archive Storage | At least 365 days | Data is rarely accessed and must be kept long term, including regulatory archives |
Formats such as Avro with SNAPPY compression or JSON with GZIP compression are also used for scheduled BigQuery exports.
Exam tip: Choose Nearline Storage for 30-day backups, Coldline Storage for 90-day recovery copies, and Archive Storage only when data can stay for at least 365 days.
Automated export and lifecycle workflows move data from BigQuery tables to Cloud Storage buckets on a schedule and then apply retention rules to the bucket. The most cost-effective backup for single-region needs is a BigQuery export scheduled by Cloud Scheduler or Managed Airflow (formerly Cloud Composer), because one export backup of a table or partition after the load or extract, transform, and load (ETL) operation finishes is enough and unchanged partitions do not need repeated export. Multi-region dataset storage is simpler for availability when locality rules allow it, while a regional dataset uses exports or a dataset copy with BigQuery Data Transfer Service to another region.
A date-partitioned backup follows an ordered sequence:
A four-step left-to-right process flow showing BigQuery partitions copied to a staging table, exported to a Cloud Storage bucket, transferred to backup location, then tiered by lifecycle rule.
gcloud storage command, where egress charges might apply for movement out of Google Cloud.Data flows from the BigQuery table to the staging table, then to the Cloud Storage bucket, then to the on-premises or secondary-cloud backup location. For on-premises columnar recovery, data can flow from BigQuery to an Apache Spark job on Managed Service for Apache Spark for processing and schema evolution, then to Cloud Storage, then across Cloud Interconnect to the on-premises network.
Storage billing and retention controls determine how stored bytes are measured and how long historical data is kept without changing file format or performance. Logical billing charges on uncompressed data size and is the default, while Physical billing charges on compressed size on disk. The model is set at the dataset level, takes 24 hours to take effect, and cannot be changed again for 14 days.
Retention cost is controlled through time travel and fail-safe storage plus metadata inspection. BigQuery retains historical snapshots for time travel up to seven days and allows restore of deleted tables within two days using snapshot syntax, while fail-safe provides additional protection with a window that cannot be modified. Under physical billing, time travel and fail-safe storage are charged separately at active storage rates, while under logical billing they are included in the base rate, so shortening the configurable time travel window lowers physical storage cost. To find candidates for tiering, query INFORMATION_SCHEMA.TABLE_STORAGE and INFORMATION_SCHEMA.TABLE_STORAGE_BY_ORGANIZATION for table storage metadata, then partition, export, or delete data based on age and access.
Manage Capacity Planning and Resource Quotas is the practice of sizing storage, memory, and compute for current data plus future growth, then tracking use over time so systems are expanded before they run out of room. Cloud Monitoring collects utilization metrics from database and cache systems, and those trends drive decisions to add disk, memory, vCPU, shards, or replicas. Good planning keeps performance steady, preserves headroom for maintenance, and avoids outages caused by full disks, exhausted memory, or project quota limits.
Defining the workload baseline means estimating how much data and traffic the storage system must handle now and later. AlloyDB Omni (opens in a new tab) sizing starts from the source database when migrating, using matching CPU, RAM, and disk as the baseline and reducing resources only after testing. Storage capacity must then cover data volume plus indexes, Write-Ahead Log (WAL), backups, and projected growth, while input/output operations per second (IOPS) are estimated from read and write patterns.
The workload definition covers four peer dimensions:
Hardware is matched to that workload with 8 GB of RAM per vCPU for shared buffers and query working memory, multiple CPU cores for concurrency, and a storage type chosen for performance or sharing. In public clouds, a larger provisioned capacity may be needed to reach a specific IOPS target because throughput depends on the storage type.
Maintaining headroom for Cloud SQL (opens in a new tab) means keeping free space and table counts within limits that allow maintenance, upgrades, and log retention to succeed. Instances should keep at least 20% available space for critical maintenance operations, stay under 10,000 tables per instance to limit upgrade time, and be sized to hold transaction and binary log retention, especially for high-write workloads. When disk throughput or CPU becomes the bottleneck, higher vCPU provides more IOPS and throughput, and a disk size of 4 TB or greater provides more throughput and IOPS.
Growth controls prevent files from expanding in large, disruptive jumps:
autogrow in megabytes rather than as a percentage, using increments appropriate to the requirement.autogrow threshold is reached.If performance issues appear, relief flows from immediate actions to long-term fixes in this order:
Sharding splits one instance into multiple smaller instances by customer or subject to scale writes and reads, but it increases application complexity and can create unbalanced hotspots.
Monitoring memory and CPU thresholds for Memorystore for Redis Cluster (opens in a new tab) means watching Cloud Monitoring metrics so shard or replica count is increased before write load or failover pressure causes errors. Memory pressure is tracked with /cluster/memory/maximum_utilization, where very high write loads can experience issues at 65% or higher and moderately high write loads can experience issues at 85% or higher. In either case, the cluster size should be scaled up to improve performance, and shard scaling should occur during periods of low writes because replication and slot migration add memory overhead during scaling.
Scale-down planning uses /cluster/memory/total_used_memory to measure data stored in the cluster. For use cases that must not lose keys, the new target shard count must provision at least 1.5 times the memory used by data. Key-eviction use cases can scale to a smaller size without data-loss concern, but the cache hit ratio may drop. CPU headroom protects availability when a zonal outage removes nodes in the unavailable zone, and high-availability clusters with up to five replicas per shard provide extra CPU during an outage.
CPU use is tracked with /cluster/cpu/maximum_utilization for primaries and replicas, and the target depends on replica count:
| Cluster configuration | Primary target | Replica target |
|---|---|---|
| One replica per node | 0.5 seconds | 0.5 seconds |
| Two or more replicas per node | 0.9 seconds | 0.5 seconds for each replica |
When values exceed those targets, the cluster depends on additional capacity, so scale up the number of shards, or scale up replicas up to the maximum of five if fewer than five exist. High CPU, too many connections, or exhausted resources can cause misbehavior and missing external metrics.
Exam tip: Scale up Memorystore at 65% memory utilization for very high write loads, but wait until 85% for moderately high write loads.
Sizing persistent compute and disk for Omni deployments means provisioning durable block storage and balanced CPU-to-memory nodes that meet minimum IOPS and throughput per vCPU. AlloyDB Omni nodes require an attached data disk with an ext4 or xfs file system sized at 2 times the data size, with minimum hardware of 2 vCPUs, 16 GB RAM, and 10 GB disk. Recommended hardware is 64 vCPUs, 8 GB RAM per vCPU, and 20 GB disk plus the 2-times data disk, with a dedicated solid-state drive (SSD) attached directly to the host. Singleton configuration uses one disk per container, while consolidated configuration has multiple containers share the same disk.
Spanner Omni (opens in a new tab) requires a dedicated, persistent, and attachable SSD with an ext4 file system for each virtual machine (VM), with 500 GB of storage per vCPU and disk performance of 500 IOPS and 30 MB per second per vCPU. Local disks are not supported for Spanner Omni, and Dell PowerFlex block storage is recommended for production. On Google Cloud, Google Kubernetes Engine (GKE) nodes and Compute Engine VMs require at least 4 vCPU and 16 GB RAM with zonal persistent disk (pd-ssd) or Hyperdisk Balanced storage. On Amazon Web Services (AWS), Elastic Kubernetes Service (EKS) and Elastic Compute Cloud (EC2) deployments require 4 vCPU and 16 GB RAM with SSD-backed Elastic Block Store (EBS) zonal io2 Block Express or gp3 volumes. Choose local Non-Volatile Memory Express (NVMe) when absolute performance and lowest latency matter, choose Storage Area Network (SAN) when large-scale shared storage matters, and consider SAN with NVMe over Fabrics when both sharing and faster access are needed.
BigQuery scales by separating storage in Colossus from query execution in Dremel, storing compressed columnar Capacitor data that can reach dozens of petabytes without adding compute. Queries run in memory over a multilevel Dremel tree with compute allocated in slots that bundle CPU and RAM, with projects limited by default to a maximum of 2,000 slots under on-demand pricing.
Persistent Disk provides 10 GiB to 64 TiB per disk and up to 257 TiB per VM with scale up, adding and removing disks, and autoscaling. Google Cloud Hyperdisk (Hyperdisk) provides 4 GiB to 64 TiB per disk, up to 512 TiB per VM, and 10 TiB to 1 PiB per storage pool with scale up through dynamic resizing, while Local SSD provides 375 GiB per disk and up to 12 TiB per VM and is fixed at creation and not scalable.
Standard is for data accessed frequently for analytics, data lakes, websites, or streaming with no minimum retention. Nearline Storage is for infrequently accessed backups and long-tail content kept at least 30 days, and Coldline Storage is for rarely accessed disaster recovery copies kept at least 90 days. Archive Storage is the lowest-cost storage for rarely accessed regulatory archives that must be kept at least 365 days.
Memorystore for Redis Cluster should be scaled up at 65 percent memory utilization for very high write loads and at 85 percent for moderately high write loads, tracked with the cluster memory maximum utilization metric. Shard scaling should occur during periods of low writes because replication and slot migration add memory overhead during scaling.
An enterprise is planning its multi-year data architecture on Google Cloud to handle projected data growth across three critical business workloads:
Which combination of Google Cloud storage and database services best satisfies the scalability and architectural requirements of these workloads?
Workload 1: Cloud SQL; Workload 2: Cloud Spanner; Workload 3: BigQuery
Workload 1: Cloud Bigtable; Workload 2: Cloud Spanner; Workload 3: Cloud Storage with external tables
Workload 1: Cloud Spanner; Workload 2: Cloud Bigtable; Workload 3: BigQuery
Workload 1: Cloud Spanner; Workload 2: BigQuery; Workload 3: Cloud Bigtable
Last updated