StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Managing the lake (configuring data discovery, access, and cost controls)
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
BigQuery manages data warehouse workloads by separating compute capacity from storage. When running analyses, users can choose between two query execution modes depending on speed and priority needs. Interactive queries run immediately and consume resources right away, while batch queries queue up and run when idle resources become available, which helps optimize capacity. To manage costs predictably instead of paying per-query, organizations can purchase slot commitments, which are dedicated virtual CPUs starting at a minimum of 100 slots. These slots are allocated using reservations assigned to projects, folders, or organizations. By default, any unused slots in one reservation are shared automatically with other workloads that need them, ensuring no purchased capacity goes to waste.
To optimize storage costs in a data lake, organizations use Cloud Storage lifecycle management policies. These policies automatically transition data to cheaper storage classes as it ages and becomes less frequently accessed. When querying data in BigQuery, engineers can lower costs by using partitioning and clustering to restrict the amount of data scanned. Partitioning divides tables by date or integer ranges, while clustering groups related data together within those partitions. During query execution, BigQuery's engine dynamically tunes query plans and shuffles intermediate data using a remote memory service, which speeds up processing and minimizes costs.
Structuring an enterprise data lake begins with Dataplex, which acts as a centralized governance layer across Google Cloud storage environments. Dataplex logically organizes raw data into lakes, zones, and assets. Once configured, its automated discovery processes scan Cloud Storage buckets and BigQuery datasets to harvest, catalog, and register metadata without manual effort. This system then utilizes the Dataplex Universal Catalog and Data Catalog tag templates to assign structured metadata, also known as aspects, to your data. These tags define custom fields for business classification, which makes files and tables easily searchable across the entire enterprise while facilitating compliance tracking.
To query this discovered data without exposing the underlying physical storage, administrators use BigLake. BigLake unifies warehouses and data lakes by utilizing access delegation, which decouples query execution privileges from direct storage access. This architecture allows security administrators to enforce granular security policies on open file formats. These controls include row-level security to restrict table access based on user attributes, column-level security to shield sensitive fields, and dynamic data masking to hide confidential values in real-time. For continuous data safety, Sensitive Data Protection automatically scans storage environments to generate data profiles, identify sensitive information, and trigger quarantine or de-identification pipelines for newly uploaded files.
Fine-grained access control protects sensitive datasets by restricting access to the specific rows, columns, or cells a user is authorized to see. Applying the principle of least privilege ensures that users and automated services only have the absolute minimum access required to perform their jobs. Administrators enforce these controls in BigQuery using row-level security to filter rows dynamically, and column-level security to restrict access to sensitive fields like financial records. When users need to work with tables containing sensitive data, dynamic data masking can obscure specific characters, such as masking a credit card number to display only the last four digits. To maintain consistent security across both Cloud Storage and BigQuery, Dataplex security policies centralize these rules and apply them uniformly across the entire lake.
Robust governance couples granular access with strict identity management and encryption systems. Identity and Access Management (IAM) serves as the baseline, allowing organizations to configure custom IAM roles that tailor permissions more precisely than broad predefined roles. For data encryption, organizations manage their keys using Cloud Key Management Service (Cloud KMS), utilizing customer-managed encryption keys (CMEK) to maintain complete control over data access. Classification is driven by policy tags within Data Catalog, which automatically trigger security controls like masking based on data sensitivity levels. Finally, VPC Service Controls construct a secure perimeter around services to block data exfiltration, while Access Transparency and Access Approval provide visibility and control over administrative access by Google personnel.