Using a data lake

Managing the Lake

Metadata Organization and Automated Discovery with Dataplex

Dataplex Universal Catalog organizes distributed data assets into logical structures and automates technical metadata discovery across storage locations. The service arranges resources in a three-tier hierarchy that standardizes governance across different business domains:

1. Lakes: Logical containers that represent high-level organizational domains or administrative boundaries.
2. Zones: Sub-containers within a lake that group data assets by readiness or purpose, such as raw, curated, or production zones.
3. Assets: The physical storage resources mapped to a zone, specifically Cloud Storage buckets and BigQuery datasets.

Dataplex scans mapped storage assets to harvest schemas, file formats, and partition structures without manual data entry. To enrich this technical metadata with business context, administrators create Data Catalog tag templates to assign custom fields—such as data owners, business units, or classification levels—to cataloged assets. Sensitive Data Protection (formerly Cloud Data Loss Prevention) scans lake storage to profile datasets, detect sensitive information such as personally identifiable information (PII), and trigger automated de-identification or quarantine pipelines.

Lakehouse Governance and Fine-Grained Security with BigLake

BigLake is a storage engine that extends BigQuery governance to files stored in Cloud Storage, enabling unified access controls across open formats. It uses access delegation to separate a user's query execution permission from direct object permissions on the underlying storage bucket. This separation allows administrators to grant users access to query tables while preventing them from downloading or viewing raw files directly from the storage layer.

BigLake enforces fine-grained security policies directly during query processing:

• Row-level security (RLS): Filters specific rows from query results based on user identity, role, or attribute conditions.
• Column-level security (CLS): Restricts access to sensitive columns using policy tags assigned through Dataplex Universal Catalog.
• Dynamic data masking: Obfuscates sensitive column values at query time for unauthorized users while preserving the underlying raw data.

Underlying data security is enforced through Identity and Access Management (IAM) custom roles, Cloud Key Management Service (Cloud KMS) with customer-managed encryption keys (CMEK), and VPC Service Controls perimeters that prevent data exfiltration across trusted boundaries. Access Transparency records logs when Google staff access customer data, while Access Approval requires administrative authorization before that access occurs.

Storage and Query Cost Management

Lake cost management balances query execution performance with storage expenses across different object lifecycles and query tiers. Cloud Storage lifecycle management policies automatically transition aging objects from Standard storage to Nearline or Coldline storage classes based on age or access rules. Within BigQuery, table partitioning divides data by date or column segments so queries scan only relevant partitions, and table clustering sorts data within partitions to reduce scanned bytes further.

BigQuery manages compute expenses through query modes and capacity reservations:

Queries run in either interactive mode, which executes immediately with dedicated resources, or batch mode, which queues queries until idle slots become available at lower operational priority. Slot reservations can share unused compute capacity across projects automatically, ensuring purchased commitments remain fully utilized.

Processing Data

Streaming Ingestion and Real-Time Processing Pipelines

Streaming data processing pipelines continuously ingest, transform, and output real-time event streams to deliver low-latency analytics. The data flows sequentially from source systems through managed processing layers to analytical storage:

1. Cloud Pub/Sub: Ingests high-throughput messages from distributed data producers and buffers the stream for downstream consumers.
2. Cloud Dataflow: Executes processing logic using the Apache Beam framework, applying windowing strategies (fixed, sliding, or session windows) and triggers to aggregate unbounded data.
3. Analytical and storage sinks: Writes processed records out to BigQuery for SQL analytics, Cloud Bigtable for low-latency operational lookups, or Cloud Storage for long-term lake retention.

[Event Producers] ──> [Cloud Pub/Sub] ──> [Cloud Dataflow] ──> [BigQuery / Cloud Bigtable / Cloud Storage]

To handle late-arriving and out-of-order events, Dataflow tracks watermarks, which estimate data completeness for an event-time window. Pipeline developers configure allowed lateness thresholds to define how long Dataflow waits for late records before closing a window and finalizing aggregations. Alternatively, teams with existing Apache Kafka investments can use the Google Cloud Managed Service for Apache Kafka to ingest real-time event streams into Google Cloud pipelines.

Batch Processing Engines and Lake Transformations

Batch data processing transforms accumulated datasets stored in Cloud Storage data lakes into structured formats suitable for analysis. Google Cloud provides several distributed execution engines tailored to different operational requirements:

Processed tabular machine learning features can be published to Vertex AI Feature Store to support feature reuse and reduce training-serving skew. Supporting ingestion tools include Datastream for serverless change data capture (CDC) from operational databases, Cloud Run functions for event-driven file processing, and Cloud Data Fusion for graphical, code-free ETL development.

Pipeline Orchestration and Workload Optimization

Workflow orchestration coordinates task dependencies, schedules, and executions across complex data lake architectures. Cloud Composer, built on Apache Airflow, schedules and monitors multi-step data pipelines by executing containerized tasks across services. Vertex AI Pipelines orchestrates machine learning workflows by linking data extraction, model training, evaluation, and endpoint deployment into automated Directed Acyclic Graphs (DAGs). Dataform manages transformation workflows natively within BigQuery, providing version control, automated testing, and SQL dependency management.

Data processing cost and performance optimization rely on precise compute sizing and monitoring:

• Resource right-sizing: Matches worker virtual machine (VM) memory and CPU allocations to specific pipeline task requirements.
• Autoscaling: Dynamically adjusts worker node capacity based on pipeline throughput and resource consumption.
• Preemptible and Spot VMs: Lowers compute costs for fault-tolerant, non-critical batch workloads on Dataproc and Dataflow.
• Backlog metrics: Cloud Monitoring tracks backlog bytes and backlog elements in Dataflow to identify pipeline bottlenecks where processing stages stall.

Monitoring the Data Lake

Data Quality, Freshness, and Lineage Tracking

Data quality and freshness monitoring verify that data lake assets remain accurate, structurally consistent, and up-to-date for downstream applications. Automated quality monitoring operates across three core dimensions:

• Data completeness: Confirms all expected source records are present without omissions caused by pipeline failures.
• Schema consistency: Validates incoming records against predefined schemas and flags unexpected column or type modifications.
• Data freshness: Measures the duration between data arrival and consumption to verify pipelines meet latency requirements.

Dataplex Universal Catalog runs automated data quality scans to score assets against declarative business rules. TensorFlow Data Validation (TFDV) analyzes machine learning datasets to detect distribution drift and schema anomalies. End-to-end data lineage in Dataplex Universal Catalog tracks transformations from raw storage buckets through processing pipelines to downstream BigQuery tables, allowing teams to isolate where data issues originated.

Centralized Security, Auditing, and Access Monitoring

Centralized security monitoring protects data lake assets by auditing API calls, enforcing identity boundaries, and detecting misconfigurations. Security Command Center tracks security posture across Google Cloud storage resources, identifying threat patterns and unauthorized modifications.

Cloud Audit Logs track administrative and user interactions across three distinct categories:

Audit logs are stored in regional log buckets with configurable retention periods ranging from 1 to 3,650 days. Administrators protect audit logs with CMEK and export log streams to BigQuery or external security information and event management (SIEM) systems for analysis. Storage bucket security is strengthened by enforcing uniform bucket-level access and locking retention policies to prevent unauthorized data modification.

Performance and Cost Monitoring

Performance and cost monitoring tracks resource utilization across data processing engines to identify bottlenecks and eliminate unnecessary cloud spend. Cloud Monitoring collects operational metrics and issues proactive alerts when system consumption exceeds defined thresholds.

Different data services require targeted monitoring metrics:

• BigQuery: Administrators track query slot utilization to detect resource contention, setting alerts when usage exceeds 90% capacity.
• Dataproc: Teams monitor cluster CPU utilization, memory consumption, and job runtimes to resize clusters and eliminate idle worker nodes.
• Cloud Dataflow: Engineers track system lag, data freshness, and worker throughput to resolve bottlenecked pipeline execution stages.

Cost data from Cloud Storage and compute engines can be consolidated into Looker Studio dashboards to track spending trends across departments. Teams use the Google Cloud Pricing Calculator to model the financial impact of storage class transitions and query reservation changes before deploying infrastructure updates.

Key Facts

• Dataplex Universal Catalog organizes storage resources into a hierarchy of lakes, zones, and assets while running automated scans to harvest technical metadata and enforce data quality rules.
• BigLake uses access delegation to allow queries against Cloud Storage files through BigQuery while enforcing row-level security, column-level security, and dynamic data masking without direct bucket permissions.
• Cloud Dataflow processes real-time streams using Apache Beam windowing, watermarks, and allowed lateness settings to handle out-of-order data before exporting results to BigQuery, Cloud Bigtable, or Cloud Pub/Sub.
• Cloud Audit Logs capture lake activity across Admin Activity, Data Access, and System Event categories, with Data Access logs tracking user-initiated read and write interactions.
• BigQuery slot commitments start at a minimum of 100 slots with flex, monthly, or yearly terms, and reservations can automatically share idle slots across workloads to optimize query processing costs.