Configure Dataplex Discovery Metadata Governance

What Google Cloud Dataplex and Universal Catalog Are

Google Cloud Dataplex is an intelligent data fabric that enables organizations to centrally manage, monitor, and govern distributed data across Cloud Storage buckets and BigQuery datasets. Within Dataplex, data is organized logically into lakes, zones (such as raw or curated zones), and assets pointing to physical cloud storage resources. Dataplex Universal Catalog (incorporating Data Catalog capabilities) provides automated discovery, metadata harvesting, and schema management.

How it Meets the Requirements

• Automated Schema Harvesting and Registration: When storage resources are registered as Dataplex assets, Dataplex Discovery automatically crawls files and datasets, infers table schemas and partition definitions, and publishes metadata entries into BigQuery datasets and the catalog.
• Fine-Grained Security on Data Lake Storage: Upgrading Cloud Storage assets to use BigLake tables replaces standard external tables with BigLake-managed tables. This unlocks fine-grained access controls including column-level security, column data masking, and row-level security on data residing in Cloud Storage.
• Metadata Classification and Enrichment: Defining aspect types (the evolutionary metamodel corresponding to Data Catalog tag templates) allows data stewards to attach custom, strongly typed attributes (e.g., PII indicators, data ownership, retention periods) to discovered entries to power faceted search.
• Unified Access Propagation: Dataplex translates and automatically propagates data roles (such as roles/dataplex.dataReader and roles/dataplex.dataWriter) and metadata roles (roles/dataplex.metadataReader) down the lake and zone hierarchy to underlying BigQuery and Cloud Storage IAM policies.

Key Advantages

• Zero Data Movement: Manages distributed data in place without copying or consolidating files.
• Automated Metadata Cataloging: Removes the maintenance overhead of manual schema registration.
• Centralized Security Enforcement: Simplifies policy management by managing IAM at logical domain boundaries.
• Enterprise Discoverability: Standardizes metadata tagging across structured and semi-structured assets.

Why This Is Optimal

This architecture leverages fully managed native Google Cloud services designed specifically for automated data lake governance, combining unified access inheritance with automated schema discovery and fine-grained BigLake security.