Dataplex Lakehouse Multi Tier Staging Architecture

What this architecture is

This architecture establishes a multi-tiered Dataplex lake topology dividing storage assets into distinct Raw and Curated zones. It pairs Cloud Storage object lifecycle management with columnar BigLake storage formats and automated metadata discovery.

How it meets the requirements

• Multi-tiered staging: Dataplex organizes storage buckets and BigQuery datasets into Raw (unstructured/semi-structured landing) and Curated (standardized, cleaned, and modeled) asset zones, ensuring logical isolation and access boundary enforcement.
• Schema discovery and evolution: Enabling automated metadata harvesting and schema discovery in the Raw zone allows Dataplex to detect incoming file structures without breaking downstream pipelines. Enforcing schema contracts and data quality scans before promotion protects Curated zone tables from unvalidated schema drift.
• Storage cost and performance optimization: Object Lifecycle Management on raw Cloud Storage buckets automatically shifts objects to Nearline or Coldline storage classes after defined retention intervals. Partitioning the curated BigLake Parquet tables by date ensures query pruning and maximizes scan efficiency in analytical workloads.

Key advantages

• Granular Governance: Applies uniform IAM security policies across distributed buckets and tables within specific Dataplex zones.
• Data Integrity: Blocks malformed or non-conforming data records at the Raw-to-Curated transformation stage.
• Query Optimization: Partitioning and columnar Parquet format drastically reduce BigQuery bytes scanned and query execution latencies.
• Automated Lifecycle: Automatically reduces storage costs without manual engineering intervention.