Prepare and test your skills
Prepare and test your skills
Specialized storage for high-throughput and low-latency workloads separates massive durable datasets from fast in-memory data. Bigtable (opens in a new tab) is the fully managed database service for large analytical and operational workloads that need high throughput at petabyte scale. Memorystore is the fully managed in-memory storage service for low-latency caching and scratch space. The choice depends on data volume, required input/output rate and read-write throughput, and whether the workload needs durable storage or a cache in front of a primary database.
Bigtable is a fully managed, high-performance database service that implements a multidimensional key-value store for large analytical and operational workloads. A Bigtable instance can support tens of thousands of queries per second with consistent single-digit millisecond latency for reads and writes, and storage that scales up to petabytes while tolerating node failure. Write throughput and latency are as important as read performance, which makes Bigtable fit workloads with frequent writes and reads or frequent writes with occasional reads. Bigtable instances are zonal resources, and replication can be configured across multiple zones in the same region or across multiple regions for increased reliability.
Choose Bigtable when one or more of the following patterns apply:
When replication is enabled, Bigtable synchronizes data between clusters so each cluster holds a separate, independent copy. If an outage affects one zone, traffic automatically fails over from the affected cluster to another available cluster in the same Bigtable instance.
Memorystore is the scalable, secure, and highly available in-memory storage service for low-latency caching and scratch space. An application sends frequent read requests to a Memorystore instance instead of the primary database, so repeated reads are served from memory and load on the persistent store drops. Memorystore supports data backends for Redis, Memcached, and Valkey, and each backend has different availability behavior. The selection of tier, region, and zone placement controls how the cache survives zonal outages.
Memorystore for Redis uses service tiers that define availability, node size, and bandwidth. Instances in the Standard tier and the Standard tier with read replicas automatically replicate data across multiple zones in a region. Memorystore for Memcached does not replicate data across zones or regions, so zone selection controls placement only. For Memcached, the instance is deployed in a selected region with nodes distributed across zones, and letting the service automatically distribute nodes avoids placing all nodes in the same zone. Memorystore for Valkey offers zonal and multi-zonal configurations where availability and reliability options are chosen at creation.
Exam tip: Memorystore for Redis in Standard tiers automatically replicates across zones, while Memorystore for Memcached does not replicate across zones or regions.
The decision between Bigtable and Memorystore depends on performance-to-volume ratio and access pattern. Bigtable persists massive datasets with durable replication and failover, while Memorystore holds smaller, frequently accessed data in memory to answer reads with low latency.
| Option | Best fit | Volume and performance |
|---|---|---|
| Bigtable | Durable store for massive analytical and operational data with frequent writes and reads | Petabytes of storage, tens of thousands of queries per second, single-digit millisecond latency |
| Memorystore | In-memory cache or scratch space in front of a primary database for frequent reads | Memory-sized working set, low-latency reads, availability depends on Redis, Memcached, or Valkey configuration |
Use Bigtable when the workload must durably store and serve very large volumes with high read and write throughput. Use Memorystore when the workload needs low-latency caching or scratch space and can refetch data from the primary store if the cache is lost, especially with Memcached where cross-zone replication is not provided.
Assess Scalability Thresholds and Capacity Planning is the practice of matching data volume and request rate to the hard limits and autoscaling behavior of a storage service. A service with a fixed maximum size or per-node limit requires added nodes, disks, or caches as data grows from gigabytes to petabytes, while a service with virtually unlimited capacity still has request-rate thresholds that affect latency and stability. Planning compares capacity per unit, simultaneous client support, regional boundaries, and how query and read performance changes under load.
Persistent Disk (PD) is POSIX-compliant block storage for Compute Engine with a maximum size of 64 TB per disk. A single instance can attach up to 128 Persistent Disks and can dynamically resize them, which sets the practical ceiling for scaling on one virtual machine. A Persistent Disk can attach in read-write mode to only one instance at a time, but the same disk can attach in read-only mode to hundreds or thousands of instances such as render workers.
Persistent Disk storage can be accessed only by resources in the same region, so capacity planning must keep compute and disks together. Standard drives suit sequential work and solid-state drives (SSDs) suit higher random performance. Choose Persistent Disk when the workload needs familiar file mounts and infrequently updated data, such as software or common libraries served read-only to many workers, and move to object or clustered file storage once total data exceeds tens of terabytes per disk or needs multi-region access.
Cloud Storage (opens in a new tab) is highly redundant, highly durable unstructured object storage that keeps files in buckets and is accessible worldwide. Unlike traditional file systems, Cloud Storage has virtually unlimited capacity and a single API across storage classes, but it is not POSIX-compliant and cannot be mounted as a logical volume by an operating system. Applications must read and write through command-line utilities such as the gcloud CLI or through the Cloud Storage API, and in a pipeline data must be transferred locally before use.
Cloud Storage is multi-tenant and autoscales by distributing bucket load across multiple servers. Each bucket starts with about 1000 object write requests per second for uploads, updates, and deletes and about 5000 object read requests per second for listing, reading data, and reading metadata. For 1 MB objects this averages to about 2.5 PB written and 13 PB read in a month. As request rate grows, Cloud Storage increases input/output (IO) capacity automatically, but redistribution takes on the order of minutes, so a spike faster than redistribution causes higher latency and error rates.
Exam tip: If the rate stays below 1000 writes or 5000 reads per second no ramp-up is needed; above those thresholds start at or below the threshold and double no faster than every 20 minutes.
Key distribution and hotspotting describe how object key order affects Cloud Storage performance under heavy writes. Cloud Storage maintains an index of object keys for each bucket in lexicographical order to support consistent object listing. The index is updated on every write or delete, so adding or deleting many objects whose keys fall in a small range increases contention, also called hotspotting. Cloud Storage detects contention and redistributes load for that index range across servers, but the same gradual ramp-up is needed when writing under a new prefix.
Capacity planning for billions of objects therefore includes both throughput and key layout. A naming convention and bulk-operation order that spreads load evenly across key ranges keeps any single index range from becoming a bottleneck. Combined bandwidth limits also apply to buckets in the same location and project, so growth planning must consider per-bucket request rate plus project-level throughput.
Shared file systems for high-client counts bridge Persistent Disk and object storage when thousands of clients need shared access. Filestore is a clustered file system that supports thousands of simultaneous Network File System (NFS) connections and can synchronize with an on-premises network-attached storage (NAS) cluster, but it has no selective sync and no bidirectional sync. PixStor is a scale-out file system that supports thousands of simultaneous NFS or POSIX clients, caches data on demand from on-premises NAS, and sends updates back to on-premises storage automatically.
Google Cloud NetApp Volumes is fully managed storage that supports NFS, Server Message Block (SMB), and multiprotocol environments with point-in-time snapshots and instance recovery, but it is not available in all Google Cloud regions. Cloud Storage FUSE mounts Cloud Storage buckets as file systems at low cost, but it is not POSIX-compliant and can be difficult to configure and optimize. The choice depends on protocol, management burden, and region availability as client count and data volume grow.
| Storage option | Capacity and scale threshold | Access boundary and tradeoff |
|---|---|---|
| Persistent Disk | Up to 64 TB per disk, up to 128 disks per instance | Same region only; single writer or many read-only readers |
| Cloud Storage | Virtually unlimited bucket capacity | Worldwide access by API; data must be copied locally before use |
| Filestore and PixStor | Hundreds of terabytes for thousands of NFS or POSIX clients | Shared file mount; sync limits or third-party cost and support |
Caching and synchronization as data and nodes grow control how on-premises data reaches cloud workers without overload. Directly mounting on-premises storage on cloud workers requires at least 10 Gbps connectivity and close proximity to a Google Cloud region. Data written back to on-premises counts as egress, and the maximum number of connected instances depends on bandwidth and connection type, so cost and latency rise quickly with scale. Synchronize-on-demand avoids constant mounting by pushing data to the cloud on publish or pulling data to workers on request, triggered by a watch script, a Pub/Sub event handler, or job-script commands using scp, rsync, or UDP-based transfer protocols.
A cloud-based read-through cache scales further by placing a caching layer on the same Virtual Private Cloud (VPC) network as the workers. The KNFSD caching solution is a scale-out cache that reduces load on the primary file-sharing service and reduces overload when many nodes request the same files, supporting hundreds or even thousands of nodes across multiple regions and hybrid pools. Data flows from on-premises storage to the cloud cache on first read, then from the cache to each worker, which lowers read latency and bandwidth. Cached data stays until it ages out after being untouched for a set time or is removed by age when space is needed, and operators can pre-warm the cache with a read or stat before a large job.
Cloud Bigtable scales by adding nodes, and autoscaling uses central processing unit (CPU) utilization, storage utilization, and minimum and maximum node counts. The maximum number of nodes cannot be more than 10 times the minimum, which bounds how far a cluster can grow automatically during a surge. If latency is unacceptably high, the CPU utilization target should be lowered, and after setting initial values the cluster should be monitored and adjusted.
Storage per node has fixed ceilings of 5 TB per node for SSD storage and 16 TB per node for HDD storage. Autoscaling targets are set in bytes per node:
Latency-sensitive applications should keep storage utilization below 60%, which is 3 TB per node on SSD and 9.6 TB per node on HDD. Less sensitive applications can use 70% or more, which is 3.5 TB per node on SSD and 11.2 TB per node on HDD, with 80% equal to 4 TB on SSD and 12.8 TB on HDD. Tiered storage with infrequent access does not change SSD autoscaling, but autoscaling also enforces a combined SSD plus infrequent-access limit of 32 TB per node for Enterprise edition and 32 TB or 64 TB per node for Enterprise Plus edition.
Relational database services store data in tables with rows and columns and a fixed schema, while non-relational services store flexible documents or entities without requiring every record to look the same. On Google Cloud, Cloud SQL (opens in a new tab) and Cloud Spanner provide SQL-based relational access with tables, joins, and transaction controls, while Firestore in Datastore mode (Datastore) provides a NoSQL document model that scales automatically. The choice depends on ACID needs, how data must scale, and whether data is highly structured or semi-structured.
Cloud SQL is a fully managed relational database service that offers MySQL, PostgreSQL, and SQL Server database engines. Data is stored in row-based tables, so related data in different tables can be connected without storing the same data multiple times. This structure fits online transaction processing (OLTP) systems that need full SQL support, business intelligence, and consistent relationships across tables.
Cloud SQL protects data through automated backups with binary logging for point-in-time recovery to a fresh instance. For availability, Cloud SQL supports a high-availability (HA) configuration and cross-region replicas to stay online during zonal or regional failure. The Enterprise Plus edition provides a 99.99% availability SLA with near-zero downtime maintenance, optimized hardware and software, intelligent data caching for read-intensive transactional workloads, and 35 days of log retention.
Cloud Spanner is a SQL-based database service that uses GoogleSQL, an ANSI-compliant Structured Query Language dialect. Applications work with tables, views, and roles through Data Definition Language (DDL) statements that create and modify database objects, Data Query Language (DQL) statements that scan tables and return result rows, and Data Manipulation Language (DML) statements that insert, update, and delete data. Data Access Control statements grant and revoke access at the table and column level, while Transaction Control statements manage transactions for data modifications.
Spanner integrates with data pipelines for backup and movement at large volume. A full export writes the database to a set of Avro files in a Cloud Storage bucket using Dataflow templates, and another template re-imports those files into a new Spanner database. For more controlled movement, the Dataflow connector lets code read and write Spanner data in a pipeline, for example to copy data out to Cloud Storage as a backup target.
Firestore in Datastore mode (Datastore) is a NoSQL document database built for automatic scaling, high performance, and ease of application development. Datastore is schemaless, so entities of the same kind can have different properties, and the same property name can hold different value types in different entities. This model maps naturally to object-oriented and scripting languages and fits semi-structured data such as product catalogs with real-time inventory and user profiles based on past activity.
Datastore provides managed behavior for scale and reliability:
Datastore writes scale by automatically distributing data as necessary. Datastore reads scale with the size of the result set, not the size of the dataset, so a query returning 100 entities performs the same whether it searches over a hundred entities or a million. Because all queries are served by previously built indexes, Datastore does not support join operations, inequality filtering on multiple properties, or filtering based on results of a subquery.
The names for the same ideas differ between models:
| Concept | Datastore | Firestore | Relational database |
|---|---|---|---|
| Category of object | Kind | Collection group | Table |
| One object | Entity | Document | Row |
| Individual data for an object | Property | Field | Column |
| Unique ID for an object | Key | Document ID | Primary key |
Exam tip: Choose Cloud SQL when the workload needs a relational database with full SQL support for OLTP; Datastore does not support joins, inequality filters on multiple properties, or subquery-based filtering.
Selecting by volume, consistency, and data shape starts with data shape and query pattern. Choose Cloud SQL for highly structured transactional data that needs relational tables, full SQL, and point-in-time recovery, with HA and cross-region replicas for resilience. Choose Spanner when the application needs SQL statements for queries, definitions, manipulations, access control, and transaction control, plus pipeline-based export and import for large datasets. Choose Datastore for highly available structured and semi-structured data at very large scale, including ACID-based transactions such as transferring funds, where automatic distribution of writes and result-size-based read scaling matter more than joins.
Datastore is not the right fit for every workload. If ACID transactions are not required or data is not highly structured, consider Bigtable. If the need is interactive querying for online analytical processing (OLAP), consider BigQuery. If the need is large immutable blobs such as large images or movies, consider Cloud Storage.
Throughput and latency requirements decide whether a storage system can keep up with an application workload. Throughput measures how much data moves per second, input/output operations per second (IOPS) measures how many small reads and writes complete per second, and latency measures how long each request takes to return. A system that meets volume needs but misses latency or throughput targets will still cause slow responses, queued jobs, or failed recovery goals.
IOPS, throughput, and latency describe three different performance limits that must be checked together. IOPS matters most when a workload makes many small requests, such as billions of small files, where the system becomes QPS-bound. Throughput matters most when objects are large and the network bandwidth becomes the bottleneck. Latency also depends on distance between storage and compute, which is why storing data in the same zone that processes the data reduces communication time, outbound transfer cost, and I/O operations between layers.
Block storage for Compute Engine workloads includes Persistent Disk and Google Cloud Hyperdisk. Zonal Persistent Disk volumes store data in one zone, while regional Persistent Disk volumes synchronously replicate writes across two zones in a region, so a backup virtual machine (VM) in the secondary zone can force-attach the disk after a zonal outage. Google Cloud Hyperdisk provides better performance, flexibility, and efficiency than Persistent Disk, and Hyperdisk Balanced lets teams provision IOPS and throughput separately and dynamically to tune one volume for a wide variety of workloads. Choose regional Persistent Disk when synchronous two-zone protection is the priority, and choose Hyperdisk Balanced when independent control of IOPS and throughput is needed for changing workload shapes.
Cloud Storage is a worldwide object store that offers multiple storage classes with different cost and retrieval models. Standard storage fits high-frequency access, Nearline and Coldline fit low-frequency access, and Archive fits lowest-frequency access. Nearline is designed for backup cases with access at most once a month, which allows regular recovery testing while keeping storage cost low. Nearline, Coldline, and Archive lower storage cost compared to Standard storage, but they add retrieval costs for data or metadata and charge for minimum storage durations if data is deleted early.
| Storage class | Access pattern | Cost tradeoff |
|---|---|---|
| Standard | High-frequency access | Higher storage cost, no low-access retrieval penalty |
| Nearline | Low-frequency access, at most once a month | Lower storage cost, retrieval costs and minimum duration apply |
| Coldline | Low-frequency access | Lower storage cost than Nearline, retrieval costs and minimum duration apply |
| Archive | Lowest-frequency access | Lowest storage cost, highest retrieval cost and longest minimum duration |
Because retrieval fees and minimum durations apply to infrequent-access classes, moving active or real-time data into Nearline, Coldline, or Archive increases latency cost and early-delete charges.
Storage location controls how data is replicated and how it survives zone or region failures. Data in Cloud Storage regional buckets is replicated synchronously across zones in the region, while data in dual-region or multi-region buckets is stored redundantly in at least two geographic locations with metadata written synchronously and data replicated asynchronously. For dual-region buckets, turbo replication replicates objects across region pairs with a recovery point objective (RPO) of 15 minutes. Filestore regional instances store data shared across multiple VMs in a region, replicate synchronously across three zones, mount on multiple VMs for shared configuration files, tools, and logs, and can replicate to a different region for protection against region outages.
Exam tip: Regional buckets and Filestore regional instances replicate synchronously within a region, while dual-region and multi-region buckets replicate data asynchronously across locations except for synchronous metadata writes.
Matching storage to processing type means aligning performance tiers with real-time, batch, or archival needs. Real-time and operational workloads need low latency and high read-and-write throughput, which fits provisioned block storage and Bigtable, a fully managed NoSQL key-value service for large analytical and operational workloads with fast access to large amounts of data. Batch pipelines move large datasets for analytics in BigQuery or Managed Service for Apache Spark, so they depend more on sustained throughput and co-location of storage and compute in the same zone and region. Archival and backup workloads prioritize low storage cost over retrieval speed, which fits Nearline, Coldline, and Archive, plus protections such as Soft Delete and object holds against deletion.
Data movement time determines whether a migration, backup copy, or batch ingest can meet its deadline without overloading the network. Ideally, 1 GB transfers in eight seconds over a 1 Gbps network, while 100 TB takes about 12 days, before accounting for real-world losses. Bandwidth limits are set at the region level and fairly allocated across projects, and Storage Transfer Service (opens in a new tab) can complete around 1000 tasks per transfer job per second when sufficient bandwidth is available.
Time estimates should include four peer factors:
When estimates show a bandwidth or request-rate bottleneck, avoid peak work hours, increase allocated bandwidth for the transfer window, or use a seed transfer followed by a smaller incremental sync transfer that moves only changed data.
Performance characteristics of object storage describe how quickly unstructured data can be read and written, measured as throughput and latency. In Cloud Storage, throughput and latency are shaped by three factors working together: the storage class chosen for the access pattern, the rate and distribution of requests, and the total volume and location of the data.
Cloud Storage storage classes match cost to how often data is accessed, and the wrong match hurts both cost and performance. Standard storage and Nearline storage are intended for frequent access, while Coldline storage and Archive storage are intended for infrequent access with lower storage cost but higher operation cost. A Spike in Class A or B operations on Coldline or Archive data finding means data stored in Coldline or Archive is being read or modified frequently, so that data is better suited for Standard or Nearline. Aligning the class to the real access pattern keeps latency predictable and avoids paying operation charges for cold data that behaves like hot data.
Request rate performance in Cloud Storage depends not only on total requests per second but on how evenly those requests spread across objects. Cloud Storage can scale to a very large number of requests per second, but a large spike in write volume can temporarily overload a single bucket, so spreading burst writes across multiple buckets avoids throttling. Object naming also controls distribution, because sequential names concentrate load on a narrow key range while non-sequential file names distribute load evenly. For the high-burst loads required by grid and parallel computing, request distribution matters as much as raw request count, which is why sizing follows the Request Rate and Access Distribution Guidelines.
Data volume and region placement determine how much data must be managed and how far each request must travel. The Key charts in Storage Intelligence advisor track Cumulative object count and Storage volume over time across locations and storage classes, so growth in objects and bytes can be seen together. A Total bytes stored increased above trend finding means consumption grew beyond historical trends over the last 30 days, for example when noncurrent object versions are retained indefinitely after new versions are uploaded. A Spike in cross-region egress finding means data flows from a Cloud Storage bucket in one region to Google Cloud services in another region, such as a Compute Engine instance in us-central1 frequently accessing a bucket in us-east1, which adds latency and egress cost until compute and storage resources are colocated in the same region using bucket relocation.
To control volume growth, Object Lifecycle Management deletes noncurrent versions or moves them to colder storage, while Autoclass automatically manages storage classes based on access patterns.
Parallel reads and writes between Dataflow (opens in a new tab) workers and Cloud Storage buckets use Apache Beam TextIO or AvroIO connectors that handle gs:// filenames. Reads from Cloud Storage to Dataflow gain parallelism in two ways: individual files are keyed separately so multiple workers can read different files at the same time, and uncompressed files can be split further so workers read separate sub-ranges of the same file. Sub-range splitting delivers very high parallelism only when each line is a meaningful record, so it is not available by default for formats like JSON. Writes from Dataflow to Cloud Storage gain parallelism primarily from sharding, where the runner automatically selects the shard count and each shard produces an output file.
Faster transfer between the Dataflow workers and the Cloud Storage bucket uses gRPC, a high-performance remote procedure call framework. Enabling the Apache Beam input/output connector to use gRPC requires Apache Beam SDK version 2.55.0 or later and the pipeline option --additional-experiments=use_grpc_for_gcs from the command line or --experiments=use_grpc_for_gcs in the SDK, which speeds up read and write requests and can emit client-side gRPC metrics to Cloud Monitoring for tuning and troubleshooting. Measured write performance on one e2-standard2 worker with the Java SDK reached 130 MBps and 130,000 elements per second for 100 million 1 kB records, though real pipelines vary with VM type, data, external sources and sinks, and user code.
Efficient writes to Cloud Storage follow several related practices:
.withAutoSharding(), not .withNumShards(0), and if tuning manually aim for 100 MB to 1 GB per shard.@ followed by a number or * in file names because @* and @N are reserved sharding specs.Exam tip: Frequent Class A or B operations on Coldline or Archive data means move that data to Standard or Nearline; Autoclass or Object Lifecycle Management fixes over-retained data, not hot access to cold classes.
The Storage Intelligence advisor (opens in a new tab) observes Cloud Storage performance without custom pipelines, BigQuery scripts, or external visualization tools. The advisor centralizes oversight of projects, folders, or organizations in a single interface and surfaces findings such as Spike in 429 errors when requests are rate limited. A spike in 429 errors signals that the input/output pattern from clients to buckets needs smoothing, such as spreading load, reducing burst concurrency, or adjusting sharding and naming. Together, advisor findings and key charts connect throughput drops and latency increases to their cause: wrong storage class, uneven request distribution, cross-region access, or unchecked volume growth.
Use Bigtable when the workload must durably store and serve very large volumes with high read and write throughput, reaching hundreds of terabytes to petabytes with tens of thousands of queries per second at consistent single-digit millisecond latency. Use Memorystore when the workload needs low-latency in-memory caching or scratch space in front of a primary database and can refetch data from the primary store if the cache is lost.
Each Cloud Storage bucket starts with about 1000 object write requests per second and about 5000 object read requests per second, and load above those thresholds must ramp gradually to avoid higher latency and errors. If the rate stays below those thresholds no ramp-up is needed, while above them start at or below the threshold and double no faster than every 20 minutes.
Use Cloud SQL when the workload needs a relational database with full SQL support for online transaction processing systems, with row-based tables, point-in-time recovery, high-availability configuration and cross-region replicas. Use Firestore in Datastore mode (Datastore) for highly available structured and semi-structured data at very large scale with automatic scaling, because it does not support joins, inequality filters on multiple properties, or subquery-based filtering.
Last updated