Choosing appropriate storage types (e.g., object, file, and databases)

Select Relational and Non-Relational Database Services

Relational Database Options

Cloud SQL is a fully managed service that supports MySQL, PostgreSQL, and SQL Server. Choose this option when transactional workloads fit within a single region and require standard ACID compliance with a fixed schema. When data must scale horizontally across multiple regions while maintaining strong consistency, choose Cloud Spanner. Spanner is designed for global scale, such as large financial systems, handling massive growth without sacrificing transactional integrity.

Non-Relational Database Options

For applications requiring a flexible schema, Firestore provides a NoSQL document database that scales automatically. It is ideal for storing user profiles or product catalogs, and it supports ACID transactions for mobile and web applications. In contrast, Bigtable is a high-performance NoSQL wide-column database optimized for sub-millisecond latency and massive analytical workloads. It easily handles time-series data or IoT sensor streams, though it does not support complex joins or multi-row transactions.

Decision Criteria for Database Selection

Architects must evaluate the tradeoffs between latency, consistency, and structure when choosing a service. While databases manage structured and semi-structured queries, unstructured data like images and backups should reside in Cloud Storage.

• Key storage types include:
  • Relational Databases: Best for structured data and strict consistency using Cloud SQL or Spanner.
  • Non-Relational Databases: Best for flexible schemas and high-speed scaling using Firestore or Bigtable.
  • Object Storage: Best for unstructured data and long-term file retention using Cloud Storage.

Differentiate Between Object, File, and Block Storage

Object and File Storage

Cloud Storage serves as the primary object storage solution, designed to store unlimited amounts of unstructured data cost-effectively. It supports parallel read access and global availability, making it ideal for serving web assets, videos, and large archives. Since it scales automatically, users do not need to pre-provision space, though it lacks the low latency of a local disk.

When multiple servers must share access to a single file system, Filestore provides shared file storage using the Network File System (NFS) protocol. For extreme workloads like High Performance Computing, Managed Lustre acts as a parallel file system that supports thousands of concurrent clients. These file systems offer POSIX compliance and sub-millisecond latency, allowing existing applications to read and write files without code changes.

High-Performance Block Storage

For virtual machines that need dedicated, high-performance storage, Persistent Disk provides reliable block storage. This storage attaches directly to individual Compute Engine instances, acting like a traditional physical hard drive. You can choose Zonal Persistent Disk for single-zone workloads, or Regional Persistent Disk to replicate data across multiple zones for high availability.

• Disk performance tiers include:
  • Balanced Persistent Disk: Best for general-purpose workloads, balancing cost and performance.
  • SSD Persistent Disk: Best for high-speed input/output operations, such as active databases.
  • Extreme Persistent Disk: Best for the most demanding database workloads requiring maximum throughput.

Assess Analytical and Lifecycle Requirements

Serverless Analytics with BigQuery

BigQuery is a serverless data warehouse designed to run complex analytical queries over petabytes of data. It separates compute from storage, meaning you only pay for the queries you run and the data you store, with each scaling independently. BigQuery organizes data in a columnar storage format called Capacitor, which speeds up analysis by reading only the specific columns needed for a query.

Data enters BigQuery through several pathways depending on the business requirements. You can use batch loading for large periodic transfers from Cloud Storage, or streaming ingestion for real-time data analysis. Additionally, BigQuery can perform federated queries, which allow you to analyze data directly in external sources like Cloud Storage or Cloud Spanner without importing it first.

Cost Optimization and Lifecycle Policies

Storage costs in BigQuery are managed automatically through a long-term storage model. If a table or partition is not modified for 90 consecutive days, its storage cost drops by 50 percent automatically, with no change in query performance. To further optimize costs and speed up query execution, developers should organize their data.

• Key organization strategies include:
  • Partitioning: Dividing a large table into smaller segments based on dates or integer ranges to limit the data scanned.
  • Clustering: Sorting and grouping data within partitions based on specific columns to make lookups more efficient.
  • Materialized Views: Caching query results to avoid running expensive calculations repeatedly.

Workload Assessment and Performance Tuning

To manage performance and control costs, administrators must monitor slots, which represent the virtual CPUs used to execute queries. Organizations can choose between on-demand pricing, which charges by the gigabyte scanned, or capacity-based reservations to secure a dedicated number of slots. Using Cloud Monitoring helps track slot utilization and identify slow queries, ensuring the warehouse scales efficiently.