Designing Governed Data Platforms with Dataplex

Building a data platform on Google Cloud means selecting storage, processing, and governance tools to match cost, performance, freshness, and compliance requirements. BigQuery (opens in a new tab) serves as the governed analytical warehouse, Cloud Storage (opens in a new tab) serves as low-cost raw and archival object storage, and data mesh (opens in a new tab) with Dataplex Knowledge Catalog provides federated governance across domains. Data flows from source systems through ingestion and processing into serving tables, while catalog, security, and lifecycle controls apply at every stage.

1.1Storage and Processing Tiers with BigQuery and Cloud Storage

Storage and processing tiers separate low-cost object storage from high-performance analytical serving. Cloud Storage holds raw, high-throughput, and archival data in open formats decoupled from compute, while BigQuery consolidates processed results into governed tables for analysis and AI actions. A distributed processing tier connects the two by joining borderless sources and writing a unified result to BigQuery for serving.

Cloud Storage is an object store with no practical limit that holds diverse data types for access from inside and outside Google Cloud. Cloud Storage stores high-throughput data such as clickstream event logs as Apache Parquet files in Apache Iceberg tables, so Spark, BigQuery, and other engines read the same files without migration or duplication. For backup, BigQuery table exports are written to Cloud Storage as Avro files with SNAPPY compression or JSON files with GZIP compression, and lifecycle controls manage retention and deletion over time.

Location controls determine redundancy and residency for each bucket. Regional buckets replicate synchronously across multiple zones within one region, while dual-region and multi-region buckets replicate asynchronously across regions for higher availability. BigQuery data is stored within the specified region and stored synchronously in two zones within that region to protect against a zone outage.

BigQuery is an enterprise data warehouse with built-in machine learning, geospatial analysis, and business intelligence features. BigQuery consolidates the output of distributed jobs into a unified customer profile with complex denormalized types, so analysts query one consistent view instead of repeating large joins. BigQuery also queries live operational data in place through BigQuery AlloyDB federation, where the query flows from BigQuery directly to live transactional data in AlloyDB for PostgreSQL for needs such as real-time inventory. This in-place path eliminates change data capture latency, and connections to external Iceberg catalogs extend the same principle across clouds.

Managed Service for Apache Spark (opens in a new tab) is the distributed processing engine between raw storage and BigQuery serving. The service uses Lightning Engine to run massive multi-way joins, complex windowing, and behavioral aggregations across borderless sources. Data flows from source systems into the Spark engine for transformation, then flows from Spark output into BigQuery tables that form the unified profile.

The Spark engine in this architecture ingests from three distinct paths:

• Parquet files from Amazon S3 that back external Iceberg tables
• Cloud Storage event logs for high-throughput activity data
• Live transactional rows from AlloyDB for PostgreSQL through BigQuery connections

Cost, performance, and access tradeoffs decide when data stays in place and when it moves. Static catalog data, event logs, and append-only history fit Cloud Storage because storage is low-cost and accessible to multiple engines, while frequently queried denormalized profiles fit BigQuery because BigQuery provides governed, high-performance analysis. Querying over the public internet between clouds can incur high egress fees and unpredictable latency, so Cross-Cloud Interconnect establishes a private connection between clouds for reliable performance.

The three tiers divide responsibility by access pattern and freshness:

1.2Ingestion, Transformation, and Lifecycle Pipelines

Ingestion, transformation, and lifecycle pipelines move data from sources into governed storage, reshape it for quality and privacy, and retain or delete it on schedule. Cloud Storage holds raw objects, Pub/Sub buffers streaming messages, BigQuery serves curated tables, and Knowledge Catalog provides discovery, lineage, and quality governance. Data flows from publishers to Pub/Sub topics, from buckets to processing jobs such as Dataflow jobs and Cloud Run functions, and finally into BigQuery tables through streaming or batch uploads.

Automated ingestion depends on secure connectivity and identity before any bytes move. Traffic flows from on-premises networks over Dedicated Interconnect, or from another cloud over Cross-Cloud Interconnect or Virtual Private Cloud networking with Cloud NAT and Cloud Router, into a Virtual Private Cloud protected by VPC Service Controls and firewall rules that deny all traffic by default. Credentials for external systems flow from Secret Manager to the ingestion engine, which verifies them first and aborts with an error if verification is denied.

The hardened ingestion sequence preserves confidentiality from source to warehouse through encryption, private transfer, buffering, and separated key storage:

1. Encrypt outside Google Cloud: Encrypt on-premises or multicloud data with the Tink library and Cloud Key Management Service keys, using deterministic encryption when join or lookup preservation is required.
2. Transfer over private connectivity: Send encrypted payloads over Dedicated Interconnect or Cross-Cloud Interconnect into the VPC Service Controls perimeter.
3. Land and buffer: Land objects in Cloud Storage and messages in Pub/Sub, where Pub/Sub flow control limits publish rates during transient spikes.
4. Load into BigQuery: Upload encrypted data and wrapped encryption keys into separate BigQuery tables, or load internal data into staging tables for de-identification.

Transformation for quality and privacy reshapes raw output into trusted datasets. Sensitive Data Protection (opens in a new tab) discovers, classifies, and de-identifies sensitive fields through pseudonymization, redaction, and generalization such as bucketing. Validation checks verify formats, ranges, and distributions before training or analytics, and anomaly detection in BigQuery flags values that might indicate data poisoning.

Teams choose a transformation pattern based on where compute should run and how much custom logic is needed:

Lifecycle policies control how long data persists and when it is removed to support efficiency and data minimization. Cloud Storage lifecycle controls automatically transition or expire objects, while Pub/Sub message storage policies pin message data to specific locations when residency is required. Cloud Logging keeps Admin Activity audit logs that record configuration changes by default and Data Access audit logs that track reads and writes to user data in the specified region. Data minimization requires not collecting or keeping data beyond strict business necessity and never logging personally identifiable information in experiments or log streams.

1.3Data Governance and Cataloging with Dataplex

Dataplex Knowledge Catalog is the organization-wide governance platform for centrally discovering, managing, monitoring, and governing data and AI artifacts. The catalog sits above distributed storage and processing systems such as BigQuery, Cloud Storage, and BigLake and grants access to trusted users while maintaining logs for access and manipulation. This design lets a central platform team enforce consistent policy while domain teams retain ownership of their own data products.

Centralized governance works without requiring data movement into a single store. The catalog connects metadata from lakes, warehouses, data marts, and AI artifacts into one control plane, so discovery, monitoring, and policy apply consistently across organization boundaries. Data flows from source systems to the catalog as metadata, lineage, quality results, and classification tags, while access decisions flow back from governance policy to the source systems.

Knowledge Catalog unifies governance tasks that otherwise fragment across engines:

• Manage data lineage across pipelines and AI artifacts, including BigQuery column-level lineage
• Manage data quality checks, data profiles, and automatic data quality validation
• Manage data discovery, exploration, and processing across different data marts
• Manage feature metadata and model artifacts for machine learning reproducibility
• Create a business glossary and enrich metadata with aspects and aspect types
• Unify governance across BigLake and open-format tables like Iceberg and Delta Lake, integrated with Dataproc Metastore
• Build a data mesh that decentralizes ownership among domain teams while preserving security

Metadata management and discovery depend on entries, aspects, and integrated metastores. An entry represents a table, file set, feature group, or model, and aspects and aspect types attach ownership, sensitivity, or domain context to that entry. The business glossary maps technical entries to standard business terms, so teams use the same vocabulary. Because governance attaches to catalog metadata rather than to one engine, an Iceberg table in Cloud Storage can be discovered and governed the same way as a native BigQuery table.

Lineage, quality, and classification answer where data came from, whether it can be trusted, and what controls it requires. Lineage is managed end-to-end in Knowledge Catalog, while BigQuery additionally provides column-level lineage within its own queries and transformations. Classification connects Sensitive Data Protection to the catalog: discovery scans identify sensitive data, inspection results flow from BigQuery into the catalog as aspects, and a table tagged with personally identifiable information can trigger tighter access and de-identification.

Access control uses role-based access with least privilege, so users receive only the no-access, read-only, or write permissions their role needs. Identity and Access Management roles map to a user, group, or service account, with finer control from resource tags, conditional policies, principal access boundary policies, and Privileged Access Manager for just-in-time elevation. Enforcement depends on location: Cloud Storage uses Identity and Access Management on buckets and managed folders, BigQuery adds dataset roles plus row-level and column-level security with policy tags and dynamic masking, and tags with conditions apply across products. Audit completes the loop because Admin Activity logs record configuration changes and Data Access logs track reads and writes, all flowing to Cloud Logging.

Building a federated governance model means defining ownership, security, and compliance centrally while letting distributed domains manage their own data products. Unified discovery depends on knowing where each partition or replica lives, which direction data flows, and whether reads return the latest state. Access control and auditability depend on the same facts: who can write in each location, how changes propagate, and how lineage is preserved in order.

2.1Federated Discovery, Replication Patterns, and Access Across Locations

Federated discovery and access is the ability to find and use data across distributed databases while each location keeps local ownership. Discovery metadata must record the system scope, because scope decides whether data stays discoverable during an outage. A zonal system is tied to a zone and is inaccessible if that zone fails, a regional system is tied to a region and remains accessible when at least one zone is accessible, and a cross-regional system is tied to two or more regions and functions when at least one region is available.

Deployment patterns define how data flows from source databases to destination databases and what a reader sees. Asynchronous unidirectional replication flows writes from a primary database to one or more secondaries, where the secondary supports reads but with replication lag. Bidirectional replication (opens in a new tab) flows writes both ways through two independent replications, so concurrent changes to the same item create a write-write conflict that needs resolution. A fully active-active synchronized system acts as one distributed database in a primary-primary setup where every update commits transactionally and synchronously, so both locations immediately see the same committed state.

The four patterns compare directly for federated decisions:

Application placement decides whether access crosses a network boundary on every request. In cloud bursting, the database lives only in the primary location and the bursting application accesses it across locations. In a primary-and-standby deployment, each location has an instance and each application accesses its local instance, with the databases transactionally or asynchronously synchronized. If the standby accepts updates, those updates must propagate back to the primary, and concurrent updates in both locations require conflict resolution. The model assumes a secure connection exists between clouds; networking generally does not affect correctness, but throughput and latency affect database latency and throughput.

Change capture and migration consistency provide the lineage and audit trail that discovery depends on. Change data capture (CDC) based on a transaction log reads the log continuously, such as the binary log in MySQL, so every insert, update, and delete is observed in order without loss and with least load impact. Differential querying polls through the query interface using a timestamp or sequence number, but deletes require special design with a logical delete flag or separate delete tables because polling cannot detect a deleted row. Migration consistency requires three properties: complete means all specified data is migrated, duplicate-free means each piece is migrated once and only once, and ordered means source changes apply to the target in the same order they occurred.

2.2Central Policy Definition and Automated Cross-System Enforcement

Cross-system policy enforcement is the practice of defining security, privacy, and compliance rules once at the center and applying them automatically across every data product. The self-service data infrastructure platform team builds shared components and guardrails, while distributed domain teams build data products under a shared responsibility model. Even when a domain team uses its own infrastructure, that infrastructure must still comply with organization-wide security and governance policies.

Central ownership divides work to reduce cognitive load without creating a bottleneck. The platform team creates reusable infrastructure for ingestion, storage, processing, and consumption, embeds governance into templates and blueprints, and publishes best practices on a prioritized roadmap. Limiting domain teams to a small set of approved options early can speed adoption, but too much centralization blocks scaling. The goal is consistent compliance where producers choose from constrained, pre-governed options and any deviation still inherits global observability and compliance automation.

Automated enforcement turns central definitions into metadata that systems evaluate without manual review. The platform team builds tooling to automate metadata annotations, and the data governance engineer extends that tooling for all domains. Annotations travel with the data product from producer to every storage, processing, and consumption system, so the same decision applies consistently across boundaries.

Enforcement follows an ordered flow from definition to detection:

1. Annotate: Tools auto-generate governance annotations for sensitivity, ownership, and handling requirements.
2. Enforce: Policy engines evaluate annotations at access and deployment time to allow, deny, encrypt, or mask data.
3. Verify and alert: Monitoring checks annotation consistency, verifies compliance, and raises alerts, reports, and dashboards when a product violates policy.

Identity, access, and network controls establish who can reach data and from where. Central policies restrict public access to BigQuery datasets and Cloud Storage buckets, enforce public access prevention by default, and require uniform bucket-level access so Identity and Access Management (IAM) is the single control plane instead of mixed ACLs. Least privilege narrows access further with least-privilege service accounts for Google Kubernetes Engine (GKE) clusters and restricted Firestore roles. Network policies block public IP addresses for AlloyDB instances, restrict Compute Engine external IPs through the compute.vmExternalIpAccess constraint, skip the default network with compute.skipDefaultNetworkCreation, and require TLS 1.2 or higher.

Encryption governance ensures protection with customer-controlled keys. Central controls require customer-managed encryption keys (CMEK), enforce CMEK for supported services, and restrict non-CMEK services. Key governance depends on Cloud Key Management Service (KMS) for protection level, purpose, algorithm, rotation, and asymmetric key management. The requirement to use CMEK establishes that a governed key must protect the resource, while the gcp.restrictCmekCryptoKeyProjects constraint restricts which projects may supply that key.

Lifecycle and geographic controls enforce retention, disposal, and residency. Central definitions include retention periods for Cloud Storage buckets, locked retention policies that cannot be reduced, object versioning, audit-record retention, and storage-class actions. Secure disposal requires methods where data is not recoverable by forensic means. The Restrict Flow of Sensitive Data control uses the allowedRegions parameter to define which regions may access data stores; if allowedRegions is empty, all access is flagged as a violation.

Continuous verification proves policies remain enforced as systems change. Controls operate in Audit and Detective modes and report findings to Security Command Center with severity, finding category, revision, tiers, and target resources. Evidence comes from Cloud Asset Inventory, bucket logging, firewall rule logging, network monitoring, and database flag logging. Remediation closes the loop when owners update the organization policy, IAM grant, or resource configuration and validate the change with tools such as Binary Authorization.

2.3Data Domains, Ownership, and Reshaping Through Migration

Data domains and ownership structures divide distributed data into clear partitions with a designated owner that can write and manage each partition. A domain is a partition, shard, or separate database that one application or location owns, while replication and deployment boundaries define how domains interact. Decentralized management works when each domain evolves independently with explicit rules for sharing and conflict handling.

Partitioned domains give each application part its own separate partition with no direct database connection. Partitioning can be two applications sharing one database with completely separate datasets, where a change to one dataset does not affect another, or each part owning its own database. In the separate-database case, the owning application writes shared data twice so ownership stays with the producer. Multi-instance sharding extends this for scale by assigning each disjoint partition to a separate instance, where more shards can be added dynamically, but cross-shard queries might not be possible in all systems.

Deployment boundaries shape who can own and operate a domain. A zonal system concentrates ownership in one zone but creates a single point of failure, while regional and cross-regional domains survive zone or region loss. A regional system may share a disk between two zones so it continues in the remaining zone if either zone fails, but it becomes unavailable if both of those zones fail even when other zones remain online. Strong consistency (opens in a new tab) choices must therefore combine the replication pattern with network reality: allow local reads from secondaries when lag is acceptable, and route reads requiring the latest state to a primary or transactionally synchronized replica.

Write ownership defines which domains can modify data, and replication enforces it through update flow. A primary owns writes while secondaries are read-only in unidirectional replication, while bidirectional and active-active designs allow writes in multiple locations. Choose partitioned ownership when domains must evolve independently, unidirectional replication when one domain owns the system of record and another needs reads such as analytics, and bidirectional or active-active only when multiple locations must accept writes.

Migration cardinality reshapes domains when ownership must consolidate or split:

• Consolidation (n:1): data flows from several source databases into fewer targets to simplify management
• Distribution (1:n): data flows from one centralized source into several targets with separate owners
• Re-distribution (n:m): data flows from several sources into several targets, such as rebalancing uneven shards

During reshaping, active-passive migration keeps ownership centralized by allowing writes only in sources while targets allow read-only access. Active-active migration allows clients to write into both sources and targets, but then all conflicts must be resolvable by resolution rules or inconsistency results.

Key Facts

• Cloud Storage holds raw and archival data in open formats decoupled from compute, while BigQuery serves consolidated denormalized tables for governed analysis.
• BigQuery AlloyDB federation sends queries from BigQuery directly to live AlloyDB for PostgreSQL data and eliminates change data capture latency.
• Regional buckets replicate synchronously across zones, while dual-region and multi-region buckets replicate asynchronously across regions.
• Change data capture reads the database transaction log continuously to capture changes in order, while differential querying polls and cannot detect deletes without special design.
• Customer-managed encryption keys enforcement requires that a customer-managed key protects the resource, while the restrict key projects constraint controls which projects may supply that key.