Managing the lake (configuring data discovery, access, and cost controls)

Optimize Storage and Query Costs Across Lake Environments

This focus area covers strategies to manage and reduce expenses for storing data in the lake and processing it with queries. The goal is to leverage platform features to automate cost savings while maintaining performance.

BigQuery Workload Management and Slot Reservations

BigQuery provides controls to manage how and when your queries run, which directly impacts cost. Queries can run in two modes: interactive queries execute immediately, while batch queries are queued and run when idle resources become available, typically costing less. To gain predictable pricing and capacity, organizations can purchase slot commitments, which are units of query processing power. Commitments start at a minimum of 100 slots and can be purchased on flex, monthly, or yearly terms, offering cost savings over on-demand pricing. By default, a reservation can automatically use idle slots from other reservations, maximizing the efficiency of your purchased capacity.

Storage Cost Optimization Strategies

Reducing storage costs involves automatically moving data to cheaper tiers as it ages. Cloud Storage lifecycle management policies enable this by defining rules to transition objects between storage classes, such as moving infrequently accessed data from Standard storage to Nearline or Coldline. Within BigQuery, you can minimize the amount of data scanned per query—and thus the cost—by using partitioning and clustering in your table designs. Partitioning divides a table into segments based on a column (like date), allowing queries to scan only relevant partitions. Clustering sorts the data within a partition based on one or more columns, which further reduces the data scanned for filtered queries.

Query Execution Optimization

Understanding how BigQuery executes queries helps design cost-effective operations. BigQuery’s execution engine uses dynamic optimization, meaning it can adjust the query plan while the query is running based on real-time data characteristics. For complex operations, it uses a remote memory service to shuffle intermediate data between workers efficiently. To optimize costs, monitor query patterns and use the workload management controls to assign the right type of reservation (interactive vs. batch) to each workload, balancing speed against expense.

---

Configure Automated Data Discovery and Metadata Management

This focus area is about creating a searchable, well-organized catalog of all data assets in the lake by automatically harvesting and enriching their metadata.

Dataplex Organization and Automated Discovery

Setting up a governed data lake starts with Dataplex. You first organize your resources by creating lakes (logical containers for data), zones within lakes (like raw, curated, or production zones), and assets (the actual datasets and storage buckets). Dataplex then performs automated discovery, scanning Cloud Storage buckets and BigQuery datasets to harvest, catalog, and register technical metadata (like schema, location, and format) without manual effort. This process makes data immediately discoverable and queryable.

Enriching Metadata with Data Catalog Tags

To add business context, Dataplex Universal Catalog uses Data Catalog tag templates. These templates allow you to define custom fields—such as data owner, sensitivity classification, or department—and apply them as tags to your discovered assets. This systematic tagging enriches metadata, ensuring enterprise-wide searchability and supporting compliance tracking. Data stewards can use these tags to define ownership and context, helping users quickly find trustworthy datasets through a centralized search interface.

Unified Security with BigLake

Access to these cataloged assets is secured through BigLake. This storage engine provides a unified layer for data in warehouses and lakes, supporting consistent security across different open file formats. It uses access delegation to separate the permission to run a query from direct access to the underlying storage files. BigLake enables fine-grained security policies that administrators can enforce uniformly, including row-level security, column-level security, and dynamic data masking.

Automated Classification with Sensitive Data Protection

Integrating Sensitive Data Protection (formerly Automatic DLP) adds automated risk management. This service scans your storage environments to build detailed data profiles and identify sensitive information like PII. By setting up automated pipelines, you can implement systems to quarantine or de-identify sensitive data in newly uploaded files. Together, these tools create a secure, compliant, and highly discoverable data ecosystem.

---

Implement Fine-Grained Access Control and Governance Policies

This focus area details how to enforce precise, multi-layered security controls across your data lake to protect sensitive information and meet compliance requirements.

Principles of Fine-Grained Access

Implementing fine-grained access control means applying security at a detailed level—such as specific rows, columns, or even masked values within a dataset—rather than granting broad table or bucket access. The guiding principle is least privilege, ensuring users and services have only the minimum access necessary for their tasks. This is critical for protecting sensitive data like personal and financial information in shared analytical environments.

BigQuery and Dataplex Security Controls

BigQuery provides several native tools for granular control. Row-level security (RLS) filters which rows a user can see based on their identity or other attributes. Column-level security (CLS) restricts access to specific sensitive columns. Dynamic data masking can be applied to columns to obfuscate sensitive values in real-time for unauthorized users (e.g., showing only the last four digits of a social security number). Dataplex security policies allow you to define and enforce these types of unified access rules consistently across both Cloud Storage and BigQuery resources, centralizing governance.

Integrating IAM, Encryption, and Policy Tags

A robust governance framework integrates fine-grained controls with core identity and encryption services. Identity and Access Management (IAM) is the foundation, where custom IAM roles can be crafted to provide more precise permissions than predefined roles allow. For encryption, Cloud Key Management Service (Cloud KMS) manages keys, including customer-managed encryption keys (CMEK) for enhanced control over data-at-rest encryption. Policy tags from Data Catalog classify data by sensitivity level; these tags can then automatically trigger the appropriate security controls (like masking or access restrictions), creating a policy-driven governance model.

Monitoring, Auditing, and Perimeter Security

Advanced governance requires visibility and protection against data exfiltration. Audit logs record all user and system activity for compliance and forensic analysis. Access Transparency provides logs of Google's own administrative access to your data, while Access Approval gives you control to approve such access requests. For high-security environments, VPC Service Controls establish a secure perimeter around your resources, preventing data from being accessed or copied outside of trusted networks and services, completing a defense-in-depth strategy.

Key Facts

• BigQuery slot commitments start at 100 slots and offer cost savings over on-demand pricing, with reservations allowing capacity to be shared efficiently across projects.
• Cloud Storage lifecycle management policies automatically transition data to lower-cost storage classes as it ages, while BigQuery partitioning and clustering reduce query costs by limiting scanned data.
• Dataplex uses automated discovery to harvest metadata from storage assets, and Data Catalog tag templates enrich this metadata for enterprise-wide searchability and compliance.
• BigLake provides a unified security layer, enabling fine-grained access controls like row-level security, column-level security, and dynamic data masking across diverse data formats.
• A comprehensive governance strategy integrates fine-grained BigQuery controls, custom IAM roles, customer-managed keys in Cloud KMS, policy tags, audit logs, and VPC Service Controls for defense-in-depth security.