Dataplex, BigQuery and Cloud Storage Data Platform

Designing storage and processing tiers with BigQuery (opens in a new tab) and Cloud Storage (opens in a new tab) means separating low-cost object storage from high-performance analytical serving. Cloud Storage holds raw, high-throughput, and archival data in open formats, while BigQuery consolidates processed results into governed tables for analysis and AI actions. A distributed processing tier connects the two tiers by joining borderless sources and writing a unified result to BigQuery for serving.

1.1Cloud Storage as Raw and Archival Tier

Cloud Storage is a low-cost object store with no practical limit that holds diverse data types for access from inside and outside Google Cloud. Cloud Storage decouples data from compute by storing high-throughput data, such as clickstream event logs, in open formats like Apache Parquet within Apache Iceberg tables. This decoupling lets Spark, BigQuery, and other engines read the same files without costly migration or duplication. For backup and archival use, BigQuery table exports are written to Cloud Storage as Avro files with SNAPPY compression or JSON files with GZIP compression, and Cloud Storage lifecycle controls manage retention and deletion over time.

Cloud Storage controls redundancy and residency through the regions specified for each bucket. Buckets can be created in three location types: regional buckets replicate synchronously across multiple zones within one region, while dual-region and multi-region buckets replicate asynchronously across regions for higher availability. Regional placement therefore determines both where data is allowed to reside and how it survives zone or regional disruptions.

1.2BigQuery as Analytical Serving Tier

BigQuery is an enterprise data warehouse for managing and analyzing data with built-in machine learning, geospatial analysis, and business intelligence features. BigQuery serves as the central governed datastore by consolidating the output of distributed Spark jobs into a unified customer profile. That BigQuery table transforms fragmented borderless metrics into complex denormalized data types, so analysts and agents query one consistent view instead of repeating large joins at query time. BigQuery data is stored within the specified region and stored synchronously in two zones within that region to protect against a zone outage.

BigQuery also queries live operational data in place through BigQuery AlloyDB federation. The federation sends a query from BigQuery directly to live transactional data stored in AlloyDB for PostgreSQL (opens in a new tab), which is a fully managed database for hybrid transactional and analytical processing such as real-time inventory or live customer profiles. This in-place query path eliminates the latency and overhead of change data capture pipelines. Connecting to external Iceberg catalogs and object stores extends the same principle across clouds by letting teams analyze data where it resides.

1.3Processing Tier Connecting Storage to Serving

Managed Service for Apache Spark is the distributed processing engine that handles complex analytical workloads between raw storage and BigQuery serving. The service uses Lightning Engine to execute massive multi-way joins, complex windowing, and behavioral aggregations efficiently across borderless sources. Data flows from source systems into the Spark engine for transformation, then flows from Spark output into BigQuery tables that form the unified customer profile for analysis.

The Spark engine ingests data from three distinct paths in this architecture:

• Parquet files from Amazon S3 that back external Iceberg tables
• Cloud Storage event logs for high-throughput activity data
• Live transactional data from AlloyDB for PostgreSQL through BigQuery connections

1.4Cost, Performance, and Access Tradeoffs

Cost, performance, and access-pattern tradeoffs determine when data stays in Cloud Storage, when it moves to BigQuery, and when it is queried in place. Static catalog data, event logs, and historical append-only data fit Cloud Storage because storage is low-cost and accessible to multiple engines, while frequently queried denormalized customer profiles fit BigQuery because BigQuery provides governed, high-performance analysis. Data warehouses commonly grow to terabytes to petabytes, so recovery time, recovery point, and recovery cost depend on data size, whether older data is static, and whether updates arrive through controlled extract, transform, and load or extract, load, and transform paths that can be replayed.

Network and compute choices further shape cost and latency for borderless access. Querying data over the public internet between Google Cloud and AWS can incur high egress fees and unpredictable latency, so Cross-Cloud Interconnect establishes a private connection between clouds for reliable performance. A modern lakehouse does not rely on monolithic compute, so teams select compute per workload and use federation or direct external connections to avoid physical data movement. When disaster recovery requires another region, multi-region storage is the simplest and most cost-effective option, while the BigQuery Data Transfer Service copies datasets across regions and scheduled exports to Cloud Storage provide a lower-cost backup when only one region can be used.

The three tiers divide responsibility by access pattern and freshness:

1.5Reliability, Security, and Lifecycle Controls

Reliability, security, and lifecycle controls apply differently to each tier but share regional control and encryption defaults. BigQuery encrypts all data at rest and in transit by default and supports customer-managed encryption keys, while Cloud Storage encrypts stored data by default with Google-owned keys and also supports customer-managed encryption keys or customer-supplied encryption keys. Cloud Storage grants access through Identity and Access Management at bucket and project levels in most cases rather than access control lists, and BigQuery provides Identity and Access Management controls plus governance features for access control, sensitive-data protection, accuracy, and consistency. Sensitive data loaded through Cloud Storage can be discovered, classified, and de-identified with Sensitive Data Protection (opens in a new tab) before broader use.

Implementing data ingestion, transformation, and lifecycle pipelines means moving data from source systems into governed storage, reshaping it to meet quality and privacy standards, and retaining or deleting it on schedule. On Google Cloud, Cloud Storage holds raw objects and events, Pub/Sub buffers streaming messages, BigQuery (opens in a new tab) serves as the analytical warehouse, and Knowledge Catalog (opens in a new tab) provides centralized discovery, lineage, and quality governance. Data flows from external or internal sources through ingestion services into storage, then through transformation steps into curated tables, while lifecycle and audit controls apply at every stage.

2.1Automated Batch and Streaming Ingestion

Automated batch and streaming ingestion moves data from operational and external systems into the platform without manual copying or fragile point-to-point scripts. The Managed Service for Apache Spark (opens in a new tab) with Lightning Engine handles large-scale joins, windowing, and behavioral aggregations by ingesting Parquet files from Amazon S3, event logs from Cloud Storage, and live transactional rows from AlloyDB for PostgreSQL through BigQuery connections. BigQuery AlloyDB federation sends a query from BigQuery directly to live AlloyDB data, which eliminates the latency and overhead of a change data capture pipeline when real-time operational context is needed. Connecting to external catalogs such as Databricks Unity Catalog for Apache Iceberg metadata lets teams analyze data in place and avoids costly migration.

Ingestion depends on secure connectivity and identity before any bytes move. Traffic flows from on-premises networks over Dedicated Interconnect, or from another cloud over Cross-Cloud Interconnect or Virtual Private Cloud networking with Cloud NAT and Cloud Router, into a Virtual Private Cloud protected by VPC Service Controls and firewall rules that deny all traffic by default. Credentials for external systems flow from Secret Manager to the ingestion engine, which verifies them first and aborts the operation with an error if verification is denied. Once inside the perimeter, data moves from publishers to Pub/Sub topics, from buckets to processing jobs such as Dataflow jobs and Cloud Run functions, and finally into BigQuery tables through streaming or batch uploads.

2.2Hardened Ingestion Sequence from Source to Warehouse

The hardened ingestion sequence preserves confidentiality from source to warehouse through encryption, private transfer, buffering, and separated key storage. Each step depends on the one before it, so a failure in verification or connectivity stops the flow before sensitive data lands in the warehouse.

1. Encrypt outside Google Cloud: Encrypt on-premises or multicloud data with the Tink library and Cloud Key Management Service keys, using deterministic encryption when join or lookup preservation is required.
2. Transfer over private connectivity: Send encrypted payloads over Dedicated Interconnect or Cross-Cloud Interconnect into the VPC Service Controls perimeter.
3. Land and buffer: Land objects in Cloud Storage and messages in Pub/Sub, where Pub/Sub flow control limits publish rates during transient spikes.
4. Load into BigQuery: Upload encrypted data and wrapped encryption keys into separate BigQuery tables, or load internal Google Cloud data into staging tables for de-identification.

2.3Transformation Patterns for Quality and Privacy

Transformation for quality and privacy reshapes raw ingestion output into trusted datasets that can be analyzed consistently. Sensitive Data Protection (opens in a new tab) discovers, classifies, and de-identifies sensitive fields while preserving utility through pseudonymization, de-identification, and generalization such as bucketing. Data labeling jobs add informative tags to raw records to create structured training data, which reduces ambiguity and mitigates bias before model training. Robust validation checks verify formats, ranges, and distributions before data is used for training or analytics, and anomaly detection in BigQuery uses statistical or machine learning methods to flag values that might indicate data poisoning.

Teams choose a transformation pattern based on where compute should run and how much custom logic is needed:

Knowledge Catalog enforces quality after transformation with automatic data quality capabilities, data profiles, and column-level lineage from BigQuery. Feature Store monitoring schedules checks on features and feature groups, while inspection results sent from BigQuery to Knowledge Catalog enrich entries with sensitivity context. Together these controls let the platform detect inconsistent behavior, unexpected access, or distribution drift across data and machine learning pipelines.

2.4Lifecycle Policies for Retention and Deletion

Lifecycle policies for retention and deletion control how long data persists, where it resides, and when it is removed to maintain efficiency and compliance. Cloud Storage lifecycle controls automatically transition or expire objects to protect resources from prolonged access or manipulation, which directly supports data minimization. Data minimization requires teams to not collect, keep, or use data beyond strict business necessity, and to prefer synthetic or fully anonymized data where possible. Never logging personally identifiable information in experiments, Cloud Logging streams, or separate log servers prevents lifecycle sprawl of sensitive data.

Storage location determines durability, availability, and residency. Cloud Storage buckets can be regional with synchronous replication across zones in one region, or dual-region and multi-region with asynchronous replication across regions for higher availability. BigQuery stores data synchronously in two zones within the specified region, and Pub/Sub message storage policies pin message data to specific locations when residency is required. Cloud Logging regionalized logs keep Admin Activity audit logs and Data Access audit logs in the specified region, where Admin Activity logs record configuration changes by default and Data Access logs track reads and writes to user data.

Encryption and access boundaries travel with data through its lifecycle. Cloud Storage encrypts by default with Google-owned keys and supports customer-managed encryption keys in Cloud Key Management Service and customer-supplied encryption keys for added control, while Pub/Sub encrypts messages at rest and in transit and supports customer-managed encryption keys at the application layer. Cloud Storage grants access through Identity and Access Management at bucket and project levels in most cases rather than access control lists, and BigQuery adds column-level security with policy tags, dynamic data masking, and row-level security. These layered controls ensure that archival retains recoverability without retaining excessive privilege or plaintext exposure.

2.5Governance and Observability Across Pipelines

Governance and observability across pipelines provide a single layer to monitor data flows, transformations, and access across teams and domains. Knowledge Catalog acts as the organization-wide governance platform that centrally discovers, manages, monitors, and governs data and artificial intelligence artifacts while granting access to trusted users. The platform team delivers this capability through infrastructure-as-code templates that provision projects with standard Identity and Access Management, logging and monitoring, networking, security guardrails, billing tags, and registration for storage, transformation, and publishing. Domain teams then assemble these composable solutions instead of building custom orchestrators or continuous integration and continuous deployment systems, using managed services such as Cloud Build to reduce overhead.

Knowledge Catalog unifies governance tasks that otherwise fragment across data marts and engines:

• Manage data lineage, including BigQuery column-level lineage, to trace data from trigger to connector step to destination table
• Manage data quality checks and data profiles to validate accuracy and consistency
• Manage data discovery, exploration, and processing across different data marts
• Manage feature metadata and model artifacts for machine learning reproducibility
• Create a business glossary and enrich metadata with aspects and aspect types to standardize vocabulary
• Unify governance across BigLake and open-format tables like Iceberg and Delta Lake, integrated with Dataproc Metastore and Lakehouse runtime catalog for an open lakehouse
• Build a data mesh that decentralizes ownership among domain teams to improve accessibility while preserving security

Real-time observability depends on Cloud Logging, Cloud Monitoring, and Cloud Audit Logs capturing pipeline metrics, API calls, and access events. Logs from data pipelines flow to Cloud Logging for live tailing and alerting, while audit trails record who created, modified, or read resource data. This lineage plus logging combination lets auditors detect unauthorized attempts, alert consumers to producer issues, and prove that ingestion, transformation, and lifecycle actions stayed within policy.

Dataplex Knowledge Catalog (opens in a new tab) is the organization-wide data governance platform for centrally discovering, managing, monitoring, and governing data and AI artifacts across data platforms. The catalog sits above distributed storage and processing systems such as BigQuery, Cloud Storage, and BigLake, and it grants access to trusted users while maintaining logs for data access and manipulation. This design lets central platform teams enforce consistent policy while domain teams retain ownership of their own data products.

3.1Centralized Governance Platform

Dataplex Knowledge Catalog provides centralized governance over fragmented data estates without requiring data movement into a single store. The catalog connects metadata from data lakes, warehouses, data marts, and AI artifacts into one control plane, so discovery, monitoring, and policy apply consistently across organization or department boundaries. Data flows from source systems to the catalog as metadata, lineage, quality results, and classification tags, while access decisions flow back from governance policy to the source systems.

The catalog supports centralized management across platforms through peer governance functions:

• Manage data lineage across pipelines, datasets, and AI artifacts, with column-level detail available in BigQuery
• Manage data quality checks, data profiles, and automatic data quality validation before data is used for training or analytics
• Manage data discovery, exploration, and processing across different data marts
• Manage feature metadata and model artifacts alongside tabular data
• Create a business glossary to manage metadata and establish a standardized vocabulary
• Enrich metadata with context through aspects and aspect types
• Unify governance across BigLake and open-format tables like Iceberg and Delta
• Send Sensitive Data Protection inspection results from BigQuery to the catalog for classification

Together these functions let auditors trace who accessed data, detect unauthorized attempts, and prevent unwanted access without inspecting each engine separately.

3.2Metadata Management and Discovery

Dataplex Catalog performs centralized metadata management and discovery by unifying technical metadata, business context, and operational metadata in one searchable inventory. An entry in the catalog represents a table, file set, feature group, or model, and aspects and aspect types attach additional context such as ownership, sensitivity, or domain to that entry. The business glossary then maps those technical entries to standard business terms, so analysts in different teams use the same vocabulary when they search and consume data products.

This metadata layer depends on integrated metastores and storage formats to stay complete. Managed metastore services like Dataproc Metastore and Lakehouse runtime catalog integrate lakes and warehouses into a well-governed open lakehouse, where open table formats remain compatible with different processing engines. Because governance is attached to catalog metadata rather than to a single engine, an Iceberg or Delta table stored in Cloud Storage (opens in a new tab) can be discovered, classified, and governed the same way as a native BigQuery table.

3.3Lineage, Quality, and Classification

Data lineage, quality, and classification in Dataplex answer where data came from, whether it can be trusted, and what sensitivity controls it requires. Lineage is managed in Knowledge Catalog across systems and transformations, while BigQuery (opens in a new tab) also provides column-level lineage for tracing a sensitive column through queries, views, and feature pipelines. Quality is managed through data quality checks, data profiles stored and analyzed in BigQuery, and automatic data quality capabilities that validate formats, ranges, and distributions before training.

Classification connects Sensitive Data Protection (opens in a new tab) to the catalog for privacy enforcement. Discovery scans at the organization, folder, or project level identify sensitive data, and inspection results flow from BigQuery into Knowledge Catalog as aspects on the affected entries. That flow lets governance policy depend on classification output: a table tagged as containing personally identifiable information can trigger de-identification by pseudonymization, redaction, or generalization such as bucketing, plus tighter access and monitoring. Real-time pipeline telemetry from Cloud Logging (opens in a new tab) and Cloud Monitoring, plus API audit trails from Cloud Audit Logs, then records how classified data is collected, stored, and transformed.

3.4Logical Data Domains and Data Mesh

Dataplex enables logical data domains by supporting a data mesh in which ownership is decentralized among data owners from different teams or domains. In this model, the central data platform team provides self-service platform solutions and common services, while domain teams build, publish, and maintain their own data products. The mesh adheres to security principles by keeping stewardship close to the domain that understands the data, which improves accessibility and operational efficiency without losing central oversight.

Common services provide the federated governance controls that hold the mesh together. A metadata catalog enables discovery of data products, a data marketplace facilitates sharing across domain teams, and a centrally defined set of computational policies can be applied globally across the mesh. Observability services build consumer trust by exposing uptime checks, custom metrics, product scorecards, and operational alerts when producers change a product that consumers depend on. Choose decentralized domain ownership when teams need autonomy over definitions and quality, and choose global computational policies and a central catalog when the organization needs consistent discovery, interoperability, and compliance.

3.5Access Control and Audit for Governed Data

Access control for governed data uses role-based access control with the principle of least privilege, so users receive only no-access, read-only, or write permissions needed for their role. Identity and Access Management roles are mapped to a user, group, or service account, and granularity is increased with resource tags, Identity and Access Management policies with conditions, principal access boundary policies that limit what a principal can reach, and Privileged Access Manager for just-in-time temporary elevation that is recorded in audit logs. This layered model prevents data theft, preserves model integrity, and limits the surface area for accidents or attacks in lakes, feature stores, and training configurations.

The choice of enforcement point depends on where the governed data resides:

Audit completes the control loop by making every grant verifiable. Admin Activity audit logs record configuration changes to governed resources, while Data Access audit logs track reads, writes, and metadata access to user data. Because logs flow from each engine to Cloud Logging, governance teams can audit access, detect unauthorized attempts, and tie temporary elevation or policy changes back to the principal that requested them.

Key Facts

• Cloud Storage holds raw and archival data in open formats decoupled from compute, while BigQuery serves consolidated denormalized tables for governed analysis.
• BigQuery AlloyDB federation sends queries from BigQuery directly to live AlloyDB for PostgreSQL data and eliminates change data capture latency.
• Regional Cloud Storage buckets replicate synchronously across zones, while dual-region and multi-region buckets replicate asynchronously across regions.
• Choose extract, transform, and load in Dataflow, Data Fusion, or Spark when data must be cleaned before landing, and choose extract, load, and transform when BigQuery SQL can perform the final transform.
• Dataplex Knowledge Catalog centralizes discovery, lineage, quality, and classification, while BigQuery additionally provides column-level lineage within its own queries.