StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Defining architecture to support data access patterns
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
BigQuery stores data in a columnar format called Capacitor, which separates storage from compute and allows operations on compressed data. This design makes optimization techniques especially effective for analytical workloads. Table partitioning divides a large table into smaller segments based on a column value like a date, so queries scan only the relevant partitions instead of the entire table. Clustering sorts data within each partition by one or more columns, which helps BigQuery skip irrelevant data blocks when queries filter on those columns. Together, partitioning and clustering reduce the amount of data processed and lower both query time and slot consumption.
Materialized views are precomputed snapshots of query results that refresh incrementally, allowing BigQuery to answer complex queries faster without recomputing large joins or aggregations. BigQuery can automatically rewrite queries to use these views when appropriate. BigQuery BI Engine is an in-memory service that caches frequently accessed data for sub-second query responses, making it useful for dashboards and reports. Configuring BI Engine based on historical query patterns ensures the most critical data stays in memory. Traditional database indexes are unnecessary in BigQuery and are not offered because the combination of partitioning, clustering, and materialized views provides the needed performance improvements. For workloads requiring heavy transactional processing, Cloud SQL or Spanner are more appropriate choices.
Cloud Bigtable is designed for low-latency, high-throughput operational workloads, offering petabyte-scale NoSQL storage with single-digit millisecond response times. It works well for applications like ad tech, fintech, IoT tracking, and real-time analytics that need frequent reads and writes with predictable performance. Bigtable scales horizontally automatically and supports autoscaling and tiered storage for cost control. Row key design is critical for performance; related data should be co-located in the same row, and keys should distribute reads and writes evenly to avoid hot spots.
Cloud Spanner provides a globally distributed, strongly consistent relational database that combines traditional SQL capabilities with horizontal scalability. It supports ANSI 2011 SQL with extensions, schema management, and automatic synchronous replication for up to 99.999% availability. Spanner is suitable for financial services, inventory management, and global e-commerce platforms that need transactional consistency across multiple regions. The architecture separates compute from storage, allowing independent scaling. Spanner supports interleaved tables and secondary indexes for optimizing parent-child relationships.
Cloud Storage handles object-based access patterns with different storage classes: Standard for frequently accessed data, Nearline for infrequent access with 30-day minimum retention, Coldline for 90-day retention, and Archive for 365-day retention. For real-time access needs, Cloud Storage integrates with Cloud CDN to cache objects closer to users, reducing latency and transfer costs. Choosing between these services depends on the workload: Spanner for ACID-compliant transactions, Bigtable for low-latency key-value access, and Cloud Storage for object-based storage. Many organizations use a polyglot approach, combining Bigtable for real-time serving with BigQuery for analytics.
Secure data access for multiple tenants requires isolating data through tenant-specific projects within a structured resource hierarchy. Identity and Access Management (IAM) enforces least-privilege access control, and separating compute resources into dedicated tenant projects prevents unauthorized cross-tenant access. Organizations should disable service account key creation and use predefined IAM roles instead of basic roles to reduce credential leakage risks.
Sharing data products with tenants or decentralized business units securely uses authorized views and authorized datasets, which let consumers query analytical data without direct read permissions on the underlying tables. This protects sensitive schema structures. Organizations can version these interfaces by either recreating and renaming the parent dataset while keeping view names identical, or by revising individual view names within a single dataset to reflect each update. These approaches let organizations expose data dynamically while maintaining centralized governance over base tables.
BigLake tables enable querying external data in Cloud Storage while applying fine-grained access control down to the file level, so users can run SQL queries without direct storage access. Row-level security and column-level security restrict data access on a granular basis, such as filtering account data by user role. For multi-tenant environments, BigQuery reservations isolate compute workloads so one tenant's intensive queries do not degrade performance for others. Analytics Hub allows decentralized business units to securely publish and subscribe to shared datasets. These tools together ensure predictable performance, robust security, and accurate cost attribution across the enterprise.