Professional Cloud Developer
Professional Cloud Developer
Intermediate
Gauge your current knowledge
Intermediate
Gauge your current knowledge
Storing and accessing data
...
Professional Cloud Developer
Gauge your current knowledge
Gauge your current knowledge
Choosing between database types depends on your data’s structure and how you need to scale. Relational databases (SQL) organize data into fixed tables with rows and columns, while non-relational databases (NoSQL) handle semi-structured data like documents or key-value pairs. Cloud SQL is a fully managed service for traditional relational engines like MySQL and PostgreSQL. It provides ACID compliance and uses vertical scaling, meaning you increase the power of a single machine to handle more load. This service is ideal for standard business applications that require high data integrity but do not need massive global scale.
Cloud Spanner combines the structure of a relational database with the ability to scale like a NoSQL system. It offers horizontal scalability, allowing it to handle massive data volumes by adding more servers. It maintains strong consistency across different geographic regions, making it perfect for global applications. Firestore is a non-relational document database designed for automatic scaling and ease of application development. It stores data in flexible collections rather than rigid tables, which is great for semi-structured data like user profiles. Firestore is serverless, meaning Google handles all the infrastructure management and scaling for you, and it is highly effective for mobile and web apps that need real-time data sync across many devices.
Cloud Storage is a powerful service designed to store unstructured data such as images, videos, and backups. It is built for massive scalability and high performance, offering low latency and high durability. Developers can store an unlimited amount of data and access it globally with a consistent set of tools and APIs. Choosing the right storage class depends on your specific throughput and latency requirements and how often you access data.
Performance is also heavily influenced by the location type you select for your data buckets. Regional provides the best performance when your compute resources are in the same geographic area. Dual-region offers higher availability and redundancy by storing data in two separate regions. Multi-region is designed for data that needs to be served to users all around the world with high uptime. To optimize both performance and cost, you can use the Autoclass feature to manage your data automatically. This automation tool moves objects between classes based on how often they are used, reducing manual management effort.
When designing applications, developers must choose storage based on data volume and performance requirements. This involves evaluating whether the workload needs block, file, or object storage. Key factors include read-write patterns, consistency needs, and the geographic location where the data must be stored.
For massive analytical datasets, Bigtable is the preferred NoSQL database service because it handles petabytes of data with ease. It provides high throughput at low latency, making it ideal for large-scale workloads that require fast writes and reads. Bigtable scales linearly, meaning performance increases directly as you add more nodes to a cluster. Common use cases include storing time-series data, Internet of Things (IoT) reports, and large-scale marketing or financial histories.
When applications require sub-millisecond latency, Memorystore provides a fully managed in-memory caching service for rapid data access. For workloads needing the fastest possible local performance, Local SSD offers high-speed scratch space physically attached to the virtual machine. These options are critical for real-time data processing where even slight delays can negatively impact the user experience.
Moving large datasets into the cloud requires choosing between online and offline transfer methods based on your available network bandwidth. The Storage Transfer Service automates data movement from other clouds or on-premises systems over the internet for a seamless transition. For massive volumes where bandwidth is limited, the Transfer Appliance provides a physical hardware solution to ship data securely to a data center.
Selecting the appropriate storage system requires understanding how data volume, access patterns, and performance needs influence scalability and long-term reliability. As your application grows from gigabytes to petabytes, it becomes essential to evaluate how each storage option handles throughput, latency, and autoscaling behavior. Capacity planning ensures that storage stays performant under changing workloads while preventing bottlenecks and unnecessary costs.
Different storage services offer unique strengths. Persistent Disk and Hyperdisk provide low-latency block storage ideal for databases and IOPS-heavy workloads, scaling performance by adjusting disk size, IOPS, or throughput settings. Filestore and NetApp Volumes offer shared file systems with features like snapshots and NFS/SMB support, suitable for workloads needing consistent shared access. In contrast, Cloud Storage excels in massive scalability for unstructured data, supporting petabyte-to-exabyte capacity with high-throughput object access.
To handle rapid data growth, developers must understand scaling thresholds and how storage systems respond to increased traffic. For example, Cloud Storage begins with baseline limits like 1,000 writes per second and 5,000 reads per second, but its autoscaling activates as demand increases—provided workloads ramp up gradually. This prevents hotspotting, where too many requests target a narrow object-key range. Likewise, file-based systems like Filestore scale by increasing capacity or selecting higher service tiers to maintain stable throughput.
When planning for performance, consider how each storage service manages high-volume access. Cloud Storage supports up to 1 TB/s throughput with proper parallelization, but comes with higher latency than block or file storage. Systems like Managed Lustre deliver ultra-low latency and very high IOPS for AI/ML training at scale. Understanding these architectural limits helps ensure that your storage choice aligns with both current performance demands and future data growth.
To build scalable applications, developers must analyze throughput and latency requirements for their specific workloads. Throughput refers to the amount of data moved over a network in a given time, while latency is the time delay before a data transfer begins. Choosing the right storage tier ensures that real-time or batch workloads perform efficiently without unnecessary costs.
For high-performance needs, Google Cloud Hyperdisk and Filestore are essential tools. Hyperdisk allows developers to provision IOPS (Input/Output Operations Per Second) and throughput separately to match the exact demands of an application. Filestore provides managed NFS servers, with regional instances offering high availability by replicating data across three zones within a region.
Cloud Storage offers different classes based on how often data is accessed and how quickly it must be retrieved. Standard storage is best for high-frequency access and real-time processing, while other classes are designed for data that is rarely touched. Key classes include Standard for immediate needs, Nearline and Coldline for infrequent access, and Archive for the lowest-cost, long-term storage.
Moving large volumes of data requires a careful evaluation of network bandwidth and total transfer time. Using the Storage Transfer Service for online moves or a physical Transfer Appliance for massive datasets helps manage these large-scale transfers reliably. Finally, selecting the best storage system involves balancing cost, performance, and location. Colocating compute resources with storage in the same region reduces network latency and helps avoid expensive data transfer charges.
Row keys are the main way Bigtable organizes and sorts all its data, similar to how words are sorted in a dictionary. This order is crucial because it determines how your data is spread across the different servers, or nodes, in the cluster. A good design prevents hotspotting, which happens when one node gets overloaded with all the new traffic while others sit idle. This often occurs if you use a simple, sequential key like a timestamp by itself, as all new data will go to the same place.
To avoid hotspots and ensure even data distribution, you should design your row keys to be more random. Strategies include using a hash of a value, reversing common strings (like using com.google.maps instead of maps.google.com), or avoiding predictable, increasing numbers. For time-series data, you have a choice between tall tables (many rows) and wide tables (fewer rows with more data in each). Using time buckets, where you store multiple events from the same hour in one row, can reduce the total number of rows and speed up reads for analytical workloads.
You must never put sensitive personally identifiable information (PII), like names or emails, into a row key. Row keys are considered service data and may appear in logs. For privacy, use internal IDs or hashed values instead. Finally, for flexible data that changes often, you can store it as a serialized protocol buffer (protobuf) inside a single column. This gives you schema flexibility without needing many column families, but you lose the ability to filter on individual fields inside that data.
Designing for Cloud Spanner is different because it’s a globally distributed database. The most important goal is to spread data evenly to prevent hotspots. You achieve this by choosing non-sequential primary keys, like UUIDs or bit-reversed sequences, instead of simple, increasing numbers. This ensures writes and reads are distributed across all servers, maintaining performance at scale.
A key feature for performance in Spanner is table interleaving. This creates a parent-child relationship where child rows are stored physically next to their parent row. This acts like a pre-join, keeping related data together for much faster queries that involve both tables. To further speed up queries, you should create secondary indexes on columns that are often searched but aren’t part of the primary key. You can make these indexes even more efficient by using a covering index, which uses the STORING clause to include extra data right in the index, so the database doesn’t need to fetch the main table.
AlloyDB is a PostgreSQL-compatible database optimized for heavy workloads. It includes a special columnar engine that can make complex analytical queries run much faster by scanning data more efficiently. To get the best performance from AlloyDB, you need to properly size your instances and monitor CPU utilization. Overall, schema design is an iterative process; you should use tools like Query Insights and run realistic load tests to find and fix bottlenecks.
Firestore is a NoSQL document database, storing data as documents within collections. It doesn’t use fixed tables, which gives you flexible, schemaless design. To model relationships, you use a hierarchical structure by nesting subcollections inside documents. This groups related data logically and optimizes it for the specific ways your app needs to query it.
Indexing is critical in Firestore for fast reads. Firestore automatically creates indexes for single fields, but for complex queries that filter or sort on multiple fields, you must create composite indexes. Firestore can sometimes merge indexes for equality filters, which helps reduce the total number of indexes you need to manage. A common technique to improve read speed is denormalization, which means duplicating data across different documents. This avoids the need for complex joins, which Firestore doesn’t support natively, leading to highly scalable applications.
You must also be aware of Firestore’s limits to keep your application reliable. Key constraints include a 1 MB maximum size for a single document and a limit of 40,000 index entries per document. A single composite index can have up to 100 fields, and each index entry itself cannot be larger than 7.5 KiB. Understanding these quotas helps you design your data model to stay within the system’s capabilities.
Strong consistency means that as soon as data is written, anyone reading it will see the latest version, no matter where they access it from. Eventual consistency means that after a write, it may take a short time for all copies of the data to update, so different users might see slightly different versions for a moment. Choosing between them is a tradeoff between having perfectly accurate data immediately and having a faster, more scalable system.
A NoSQL database like Bigtable is built for massive scale and often uses eventual consistency. When data is written to one cluster, it is replicated to others asynchronously. This can cause replication lag, meaning a user might write data and then not see their own change if they immediately read from a different cluster. This "read-your-writes" inconsistency is a known tradeoff for gaining high performance and availability.
Cloud Storage uses strong consistency for object uploads and metadata. When you upload a file or change its details, the system guarantees that the next read—from anywhere—will return that new data. This simplifies application development because programmers don’t have to write extra code to handle temporary mismatches between data copies.
Services use multi-regional replication to protect data and keep it available during an outage in one location. Cloud Storage can replicate objects across regions for high availability, and it maintains strong consistency even in this setup. Bigtable uses multi-cluster deployments for the same reason, but its eventual consistency model means the tradeoff for this availability is the potential for temporary data inconsistency across regions.
The CAP Theorem states that a distributed system can only prioritize two of three properties: Consistency (all data is the same everywhere), Availability (every request gets a response), and Partition Tolerance (the system works even if parts of it can’t communicate). In cloud databases, this often means choosing between strong consistency (C) and high availability (A) when network partitions (P) occur.
For applications where data must be perfectly accurate, like financial transactions, you choose services that prioritize strong consistency. Cloud Spanner provides this globally by using synchronous replication; it waits for all copies to update before confirming a write. Cloud SQL and AlloyDB also provide strong consistency for their primary databases, ensuring every transaction is complete before it’s visible.
When handling enormous traffic, like for user activity logs, high availability and performance are often more critical than immediate consistency. Bigtable and Cloud Storage favor availability and partition tolerance, using asynchronous replication. This allows them to scale horizontally and remain fast, accepting that reads might temporarily return stale data.
When using an eventually consistent system, you must design your application to handle stale data. Strategies include routing a user’s requests to the same cluster where they wrote data, or designing features where slight delays in updates are acceptable. This allows you to optimize for low-latency global performance while mitigating the user-facing impact of replication lag.
Cloud Spanner is unique because it delivers strong external consistency across the globe. It uses a technology called TrueTime to synchronize clocks across all its servers, ensuring transactions are processed in the exact order they occurred. This means an application can read and write from any location worldwide without worrying about data conflicts or stale reads, simplifying global application design.
Cloud SQL and AlloyDB use a primary instance that handles all writes. They can replicate data to read replicas or secondary clusters. This replication can be synchronous (waiting for confirmation) for high integrity, or asynchronous (sending data and continuing) for better performance. The asynchronous method is common for read replicas, which creates the possibility of replication lag.
Replication lag is the delay between a write on the primary database and its appearance on a read replica. Applications that use read replicas to scale must account for this. For example, showing a user their own profile update might need to read from the primary, while showing a public feed can safely read from a replica. This is the core tradeoff: read replicas increase read scalability but can temporarily reduce transactional integrity for users reading from those replicas.
The choice depends on the application’s needs. Financial systems require the strong consistency of Cloud Spanner, Cloud SQL primary, or AlloyDB primary. Features like comment threads or product catalogs can often tolerate the slight delay of reading from an asynchronous replica to gain scalability and lower latency for users around the world.
A signed URL is a custom link that provides temporary, secure permission to access Cloud Storage resources. This mechanism is essential for sharing private data with users who do not have a Google identity. By using a digital signature, the URL proves that the requester has been granted access by an authorized account, typically a Service Account.
The V4 signing process is the modern standard for creating secure, authenticated requests to the Cloud Storage XML API. This process begins when the system generates a canonical request, which is a standardized version of the HTTP request. This standardization ensures that the cloud platform can verify the request exactly as it was intended by the creator without any alterations.
A V4 signed URL contains several key parameters in its query string to ensure security and validity. These parameters include X-Goog-Algorithm for the cryptographic method, X-Goog-Credential for the service account details, and X-Goog-Date for the creation timestamp. It also features X-Goog-Expires to define the duration in seconds and X-Goog-Signature as the encrypted string that validates the entire request.
When constructing these URLs, you must specify the HTTP method to control what actions the user can perform. Common options include GET for downloading, PUT for uploading, and DELETE for removing objects. To maintain precise resource control, the signature is linked to a specific bucket and object path, which prevents the URL from being used to access other files. Security is further improved by setting a short expiration timestamp and disabling public access on the bucket.
Identity delegation allows developers to safely share resources with external parties without requiring them to log in. To make this work, a service account acts as the signing identity that delegates its authority to the external user. Before a URL can be generated, this service account must possess the necessary permissions for the target action, such as having the Storage Object Viewer role to allow file downloads.
To generate a valid signature, the identity performing the signing process must have specific administrative roles. The Service Account Token Creator role must be assigned because it allows the system to generate short-lived credentials. Specifically, the signing identity needs the iam.serviceAccounts.signBlob permission to invoke the IAM signBlob method, which generates the actual digital signature.
The signing process relies on a secure cryptographic workflow to protect the request. First, the program constructs the canonical request and a string-to-sign based on the request details. Then, an RSA signature is created using the SHA-256 algorithm and attached to the final URL. Using this API method is highly secure because it avoids the operational risks of managing long-lived, private service account keys.
Developers can generate signed URLs either programmatically using Cloud Storage client libraries or manually using command-line tools like gsutil. Programmatic generation is the preferred choice for applications because it supports automated security checks and dynamic expiration settings. In contrast, command-line tools are ideal for quick, manual administrative tasks performed directly by developers.
Access control policies are maintained by applying Identity and Access Management (IAM) roles and Access Control Lists (ACLs). Implementing uniform bucket-level access is recommended because it simplifies how you manage these security policies across your entire project. For scenarios requiring high security, you can use customer-supplied encryption keys to protect files, meaning recipients must possess the correct decryption key to read the downloaded data.
To protect sensitive resources, you must always apply strict constraints to your delegated links. This includes setting the shortest practical expiration limit to minimize the window of exposure. Additionally, you must use canonical headers to verify the integrity of the request and ensure it has not been tampered with while traveling over the network.
BigQuery organizes data using a columnar format instead of traditional rows, which completely changes how developers design tables. In this architecture, denormalization is the preferred method for organizing large datasets because it reduces the need for complex joins. To achieve this, developers use flattening to combine multiple tables into a single flat table structure. This approach speeds up queries and improves storage efficiency, making it much easier to run machine learning and AI tasks on massive amounts of data.
To maintain relationships without slowing down performance, BigQuery supports nested fields and repeated fields inside a single row. This feature allows developers to keep a high level of relational integrity without relying on fully normalized tables. A nested field stores related data together inside one record, while a repeated field allows a single column to hold an entire list of values. Together, these fields provide an efficient way to model complex "one-to-many" relationships without the performance cost of traditional database joins.
As business needs change, schema evolution allows developers to update a table’s structure without rebuilding it from scratch. You can easily add new columns to introduce new data points or convert data types to support evolving machine learning feature requirements. To keep the pipeline healthy, a data validation process must check raw data against the defined schema contract before it is processed. If any incoming data fails this validation, it is routed to a separate table for debugging and forensics to protect the integrity of downstream AI models.
The BigQuery Storage Write API is a high-performance tool that combines both streaming and batching ingestion into a single, unified interface. It is designed to handle high-throughput and low-latency data delivery, making it ideal for real-time analytics and AI workloads. Depending on the design of your cloud application, you can configure this API to focus on extreme speed or strict data accuracy. Choosing the correct ingestion method requires analyzing your network throughput and managing your project quotas.
When developers need data to be available for queries immediately, they use streaming ingestion. A common architectural pattern sends events first to Pub/Sub, which acts as a temporary buffer to absorb sudden spikes in traffic. From there, a Dataflow pipeline processes the events and writes them directly into BigQuery. To prevent failures from stopping the pipeline, developers use truncated exponential backoff to retry failed writes and send broken "poison records" to a dead-letter queue for troubleshooting.
Developers must choose between different stream types based on how they want to handle duplicate data during network interruptions. A committed stream offers exactly-once semantics by using client-provided offsets to track the exact position of every written record. This guarantees that duplicate records are never created, even if a write operation must be retried. If the application can tolerate occasional duplicates, you can choose a default stream instead, which provides higher throughput and does not count against certain project quotas.
For scenarios where data is processed in large groups rather than real-time events, batch loading is the most efficient choice. This method utilizes a pending stream to buffer records in a temporary state while the rest of the batch is processed. Once the entire batch of data is ready, the system commits the changes as a single atomic transaction. This ensures that downstream analytics and machine learning models only see the new data once the entire import has succeeded.
Table partitioning is a highly effective way to cut costs and speed up queries by splitting massive datasets into smaller, physical segments. BigQuery allows you to set up partitioning using time-unit columns, integer ranges, or ingestion-time partitioning. When a query runs, partition filters force the system to read only the specific segments that match the query criteria. This simple design choice prevents BigQuery from scanning the entire table, which dramatically reduces processing costs and query times.
While partitioning splits data into physical segments, clustering organizes and sorts the data within those segments. This strategy is best for columns that users frequently use to filter or group query results, such as customer IDs or product categories. As new data is written to the table over time, BigQuery automatically performs re-clustering to maintain optimal sorting. Grouping similar rows together on the physical storage disks increases storage efficiency and makes downstream analytical queries run much faster.
To write optimized data structures reliably, a Dataflow pipeline can manage ingestion using the BigQuery Storage Write API. This managed pipeline simplifies complex streaming configurations while ensuring that exactly-once processing rules are followed. To keep the ingestion pipeline highly available, the system uses exponential backoff to pause and retry when BigQuery is busy. Any records that repeatedly fail to write are redirected to dead-letter queues so the rest of the pipeline can continue running without interruption.