StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Data staging, cataloging, profiling, and discovery (data governance)
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
To implement effective data governance, organizations must maintain a centralized, searchable inventory of all data assets across their environment. Dataplex Universal Catalog serves as the primary intelligent metadata management platform, succeeding the older, deprecated Data Catalog service. This service automatically ingests and indexes technical metadata—such as table names, schemas, and configurations—directly from BigQuery, Cloud Storage, and Pub/Sub. By automating this collection, organizations establish a single source of truth that keeps metadata current without requiring manual tracking or data consolidation.
To make metadata useful for discovery, data stewards must enrich raw technical metadata with business meaning. They implement schematized Tag Templates to attach structured business, operational, and financial labels to individual data assets. For example, these tags define data sensitivity, ownership, quality scores, cost metrics, and retention rules. This enrichment transforms technical schemas into a rich, semantic layer that explains what the data means and how it should be governed.
Once metadata is centralized and enriched, users can perform semantic searches using everyday business terms rather than strict technical database names. Access to these search results is strictly governed by Identity and Access Management (IAM) policies and sensitivity classifications. Integrating this catalog with frameworks like the Cloud Data Management Capabilities (CDMC) allows systems to automate compliance checks. This process ensures that sensitive datasets are properly tagged and protected before anyone can discover or query them.
To understand the structure and quality of stored data, organizations deploy automated data profiling. Within the Dataplex Universal Catalog, profiling runs continuously or on demand to analyze statistical characteristics, such as unique values, null counts, and data distributions, across project datasets. For sensitive data, Sensitive Data Protection scans data lakes and warehouses using more than 150 predefined detectors to identify patterns like government identifiers, credit card numbers, and personally identifiable information (PII). The results of these scans help security teams identify high-risk areas and recommend automated quality rules at the project, table, and column levels.
Data classification involves automatically tagging data assets based on their discovered sensitivity levels and applying granular security rules. In BigQuery, security teams enforce column-level security by creating taxonomies of policy tags, such as 1_Sensitive, 2_Private, and 3_Confidential. To read data in a protected column, a user must have the Fine-Grained Reader role. This taxonomy follows a hierarchical access model: a user granted access to a high-level tag, like 3_Confidential, is automatically allowed to view columns with lower-level tags, like 2_Private, while untagged columns remain open to anyone with general table access.
For users who need to query tables but lack permission to see raw sensitive values, dynamic data masking modifies query results in real time. Based on the associated policy tags and user roles, the system applies default masking rules to obscure values like credit card numbers or physical addresses at query time. For non-production or development use cases, organizations can completely de-identify data by building Dataflow pipelines. These pipelines read raw files from Cloud Storage, use Sensitive Data Protection to apply transformation methods like tokenization (pseudonymization) or date shifting, and write the safe, altered records into BigQuery.
A robust lakehouse architecture relies on structuring Dataplex lakes into distinct virtual asset zones to control how data transitions through its lifecycle. To maintain data integrity, engineers establish logical boundaries by moving data through three primary layers: a raw zone for unprocessed files, a staging zone for validation, and a curated zone for clean, consumption-ready data. Within these zones, organizations enforce distinct schema evolution rules and partition strategies to optimize storage layouts and prevent schema mismatches. Dataplex Universal Catalog automatically captures technical metadata across these zones, using aspects to add lineage and ownership details so consumers can easily discover clean data.
When policy tags and masking are not enough to meet strict security standards, organizations establish cryptographic boundaries using field-level encryption. Engineers use AEAD encryption functions to encrypt sensitive data columns at a granular level before writing them to storage. This process relies on customer-managed encryption keys (CMEKs) hosted securely in Cloud HSM. Because keys are managed independently, unauthorized users cannot decrypt and read the underlying plaintext data, even if they have read permissions on the database tables or storage buckets.
Managing a multi-tiered lakehouse requires balancing high performance with storage costs. Engineers run continuous profiling using Sensitive Data Protection to identify where high-risk data lives, allowing them to apply strict residency and privacy rules. For aging data in the staging zone, organizations configure object lifecycle management or Autoclass within Cloud Storage. These tools automatically transition older staging files to lower-cost cold storage tiers based on their age and access patterns, keeping storage spend optimized without disrupting active pipelines.