StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Selecting storage systems
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
Selecting the right Google Cloud storage system requires evaluating whether your workload is operational or analytical. Operational workloads require low latency and balanced transactional reads and writes. To meet these demands, developers should select Cloud Spanner or Cloud SQL. However, if an application requires high-speed, single-row access at a massive scale, Cloud Bigtable is the ideal database choice.
For analytical workloads, BigQuery acts as a serverless data warehouse that separates compute resources from physical storage. It stores data using a columnar format called Capacitor, which allows the query engine to scan compressed data directly. Compute capacity is dynamically allocated using slots, which are flexible units of CPU and RAM that scale based on query complexity. When applications require file or block storage instead of databases, developers can select Cloud Storage for object storage, Filestore for shared network filesystems, or Persistent Disk for virtual machine storage.
To monitor and diagnose data access issues, BigQuery offers an execution graph that visually maps how each processing stage runs. This graph helps developers find bottlenecks by calculating the exact number of query slots used by each stage. Performance insights can highlight issues like inefficient joins, missing clustering, or queries that scan more data than necessary. For transaction databases, Cloud Spanner monitors performance by sending availability and latency metrics to Cloud Monitoring. To handle heavy reporting workloads without affecting live transactions, administrators can enable Spanner Data Boost to process analytical queries on separate, dedicated compute capacity.
Other storage systems use specific service level indicators to track performance trends over time. For instance, Cloud Storage records availability by measuring successful versus failed requests using request count metrics. Meanwhile, Cloud Bigtable sends metrics such as request count, error count, and latencies to Cloud Monitoring to ensure the system maintains high throughput. Developers should establish baseline metrics to spot performance anomalies when access patterns change. Utilizing partitioning and caching query results for up to 24 hours can significantly reduce compute costs and improve latency.
Designing an optimized schema ensures physical data organization aligns with how applications run queries. In BigQuery, partitioning physically divides a large table into smaller segments based on a date or timestamp column to limit the amount of data scanned. Clustering then sorts the data inside each partition, allowing the system to filter results efficiently without using traditional database indexes. This combination reduces both execution time and query costs.
In Cloud Bigtable, row key design is the most critical factor for distributing reads and writes evenly across server nodes. If a row key starts with a sequential value like a timestamp, all incoming writes will hit a single server node and cause hotspotting. To prevent this performance bottleneck, developers use strategies like key salting to add a hash prefix or reverse domain names to spread the load. Choosing schema designs always involves balancing performance against architectural trade-offs, such as how storing all row data in a single protocol buffer column saves space but prevents column filtering.
Designing a cost-effective data lakehouse in Google Cloud requires separating storage from compute so both layers can scale independently. BigQuery serves as the analytical warehouse, using the Colossus file system to run fast, parallel queries completely in memory. For raw, unstructured, or semi-structured data, Cloud Storage acts as a highly durable object store that organizes files into regional, dual-region, or multi-region buckets. To minimize network latency and egress costs, teams should store data in the same region where the data processing actually occurs.
Traditional external tables let users query files directly, but this setup requires permissions on both the table and the underlying files. To solve this security gap, BigLake acts as a unified storage engine that enables secure data virtualization across formats like Parquet, Avro, and JSON without moving the data. It supports access delegation, which means data consumers only need access to the table itself and not the raw Cloud Storage bucket. This architecture protects sensitive data while still allowing multi-cloud analytics over a cost-effective lakehouse. To maximize query performance and control costs in analytical workloads, architects can implement table partitioning, materialized views, and clustering. For disaster recovery, Cloud Storage provides dual-region or multi-region options that replicate data asynchronously, where turbo replication guarantees a recovery point objective of 15 minutes.
Cloud Bigtable is Google Cloud's fully managed NoSQL database designed for high-throughput workloads that handle massive amounts of data. It can store petabytes of data across distributed nodes while providing sub-millisecond read and write latencies. The service automatically scales by adding or removing nodes based on demand, separating storage from compute for maximum flexibility. Users must make a key choice between SSD and HDD storage options, where SSD storage is chosen when applications require the lowest possible latency. Designing effective row-keys is critical for Bigtable performance because it determines how data is distributed across nodes and prevents hotspotting.
Cloud Memorystore provides fully managed Redis and Memcached services that deliver microsecond-level latencies for frequently accessed data. The service handles maintenance tasks like patching and updates automatically, reducing overall operational overhead. To ensure high availability, it supports automatic failover and replication across zones so cached data remains accessible during a failure. Applications use this service to cache database query results, user sessions, or frequently accessed records. Implementing the right caching strategy with Memorystore can dramatically improve application speed by reducing the load on backend databases. Teams can choose the cache-aside pattern, where the application checks the cache first before querying the database, or the write-through pattern, where writes update both the cache and the database simultaneously.
Relational databases are the best choice for transactional applications that require strong consistency, complex queries, and a fixed database schema. Cloud SQL is a fully managed service supporting MySQL, PostgreSQL, and SQL Server, making it easy to set up with automated backups. For highly demanding workloads, AlloyDB offers superior performance and integrated analytics while remaining fully compatible with PostgreSQL. These options excel in traditional online transaction processing workloads, such as e-commerce or financial systems, where ACID compliance is critical.
When applications need to scale horizontally across the globe while maintaining strong consistency, Cloud Spanner is the ideal choice. It combines the structured schema and SQL query capabilities of a relational database with the horizontal scalability of a NoSQL system. Spanner provides external consistency across multiple regions, making it perfect for mission-critical systems like global financial trading or inventory management. This service ensures high availability and low latency across continents without sacrificing data integrity.
Non-relational databases provide flexible schemas and are optimized for specific operational access patterns. Cloud Firestore is a serverless document database designed for mobile and web applications that need hierarchical data structures and real-time updates. Selecting the right database requires analyzing trade-offs between vertical scaling, where you increase machine size as in Cloud SQL, and horizontal scaling, where you add more nodes as in Spanner or Bigtable. You must also evaluate consistency models, choosing strong consistency for financial transactions or eventual consistency for less critical updates.
Data access patterns describe how often and in what way your applications read or write data. Choosing the right Cloud Storage class depends on these patterns, with Standard storage serving frequently accessed data, while Nearline, Coldline, and Archive handle progressively colder data. You can use Object Lifecycle Management policies or the Autoclass feature to automatically move data to cheaper storage classes as it ages and is accessed less. This automated tiering optimizes costs without requiring manual intervention from operations teams.
Network egress costs are fees charged when data leaves a Google Cloud region. To reduce these costs and improve speed, you should store data close to where it is used. Placing a Cloud Storage bucket in the same region as your application servers avoids cross-region transfer fees. Configuring Private Google Access for your VPC network lets resources like virtual machines reach Google services without using the public internet, which also helps manage egress costs.
Multi-regional distribution means copying data to multiple geographic locations for high availability and disaster recovery. You can use Cross-bucket replication with the Storage Transfer Service to automatically copy objects from a source bucket to a destination in another region. This provides durability and a failover location, but you must plan for replication latency. Setting up this distribution requires configuring VPC networks, firewall rules, and DNS, while ensuring service accounts have specific roles like Storage Legacy Bucket Writer on both buckets.
Object storage tiering uses different Cloud Storage classes to match cost with how often data is accessed. Each lower-cost class has specific trade-offs, such as higher retrieval fees and minimum storage durations. Nearline is good for backups accessed monthly, while Coldline has lower storage costs but higher retrieval fees. Archive offers the lowest storage cost but the highest retrieval cost and typically requires data to be stored for at least 365 days.
Lifecycle management policies automate moving data between these storage classes based on rules like age. A critical factor when using these policies is early deletion penalties. If you delete or move data before its minimum storage duration, you will be charged a penalty. The pricing model creates a clear trade-off: storage classes with lower monthly costs have higher fees for retrieving data and stricter minimum storage durations. This encourages you to put long-term, rarely accessed data into Coldline or Archive, while keeping hot data in Standard storage.
Balancing performance and cost for structured and semi-structured data involves optimizing specific configurations for different Google Cloud services. A key advantage is the separation of compute and storage, which services like BigQuery use. This means you can store petabytes of data without paying for expensive, idle compute resources, scaling each part independently. In BigQuery, you optimize query speed and cost by organizing data efficiently using table clustering and partitioning to reduce bytes scanned.
For high-throughput operational databases, the approach to cost-performance balancing differs. Cloud Bigtable scales performance linearly with the number of nodes in its cluster, which you can scale up or down based on traffic. Enabling autoscaling does this dynamically to optimize the compute-to-storage cost ratio. Bigtable offers SSD-backed storage for low-latency performance and lower-cost HDD storage for infrequently accessed data, allowing further tiering within the service. When using Cloud Storage as a data lake, you must balance geographic redundancy with cost, as multi-regional storage classes offer higher availability but incur higher replication charges.
Object Lifecycle Management in Cloud Storage automates data management by moving objects between storage classes or deleting them based on rules you set. You create these rules at the bucket level to transition data from expensive classes like Standard to cheaper ones like Nearline or Coldline as the data ages. Retention policies enforce that objects must be kept in a bucket for a minimum period of time, which is crucial for regulatory compliance. For immutable compliance, you can lock this policy using Bucket Lock, which permanently prevents anyone from shortening the retention period or removing the policy.
Object holds provide a finer level of control by preventing specific objects from being deleted or overwritten. A temporary hold can be manually applied and released, while an event-based hold automatically releases when a predefined condition is met. To protect against accidental deletions, soft delete keeps deleted objects in a recoverable state for a configurable period, which defaults to seven days. Additionally, object versioning maintains multiple generations of an object, and you can set lifecycle policies to automatically expire old, non-current versions.
For large-scale management, storage batch operations let you perform actions on billions of objects efficiently with a single job. You can delete objects, update metadata, apply or remove holds, or change encryption keys across massive datasets. Finally, the actual deletion of data in Cloud Storage is a secure, multi-stage process. After the soft delete recovery window, data undergoes logical deletion from active systems, achieved either through traditional overwriting or cryptographic erasure, which destroys the encryption keys needed to read the data.
BigQuery uses two main storage billing models: logical storage billing and physical storage billing. Logical billing charges for the uncompressed size of your data, while physical billing charges for the compressed size and includes fees for extended data retention features like time travel. You choose the model at the dataset level, and selecting the right one is a key decision for controlling costs and aligning with your data access patterns. You can automate data deletion to control storage costs by setting expiration rules at the dataset, table, or partition level.
For partitioned tables, partition expiration automatically drops individual time-based data chunks once their age exceeds the set limit. Under the physical billing model, you can also reduce the default time travel window from seven days to a shorter period to reduce the amount of historical data retained, further lowering storage costs. BigQuery also automatically provides a long-term storage discount. If a table or partition is not modified for 90 consecutive days, its storage cost is automatically reduced by 50% without changing its performance, availability, or durability.
Designing an effective multi-storage tiering strategy requires analyzing your data access patterns to balance monthly storage costs against data retrieval fees and access times. For example, you might keep current month's application logs in Standard storage for quick analysis, move logs from the previous quarter to Nearline, and archive logs older than a year to Coldline or Archive. The strategy should be automated with lifecycle policies so data flows to the appropriate cost tier without manual intervention.
To implement tiering, you architect your platform with a separate data storage layer and a data computation layer, such as Cloud Dataflow. You then configure buckets and their lifecycle rules using the Google Cloud console, gcloud CLI, or infrastructure-as-code tools like Terraform. For advanced optimization, consider data location to colocate storage with compute resources and reduce network transfer costs. Regularly monitor storage usage and costs with Cloud Monitoring and refine your policies as business needs and access patterns change to maintain the optimal balance between cost, performance, and compliance.
Selecting the right Google Cloud storage system requires analyzing the specific read/write ratios and latency tolerances of your workload. Operational workloads demand low latency and balanced tra…
Selecting the right Google Cloud storage system requires analyzing the specific read/write ratios and latency tolerances of your workload. Operational workloads demand low latency and balanced tra…
Cloud Storage Object Lifecycle Management and Compliance
Cloud Storage Object Lifecycle Management and Compliance
Designing a cost-effective data lakehouse in Google Cloud requires separating storage from compute so both layers can scale independently. BigQuery serves as the analytical warehouse, using a colu…
Designing a cost-effective data lakehouse in Google Cloud requires separating storage from compute so both layers can scale independently. BigQuery serves as the analytical warehouse, using a colu…
Data access patterns describe how often and in what way your applications read or write data. Choosing the right Google Cloud Storage class depends on these patterns. Standard storage is for d…
Data access patterns describe how often and in what way your applications read or write data. Choosing the right Google Cloud Storage class depends on these patterns. Standard storage is for d…