Designing for a data platform

Building a Data Platform Based on Requirements

Data Ingestion Tools and Sources

Data ingestion is the process of bringing data from external sources into a Google Cloud data platform. The choice of ingestion tool depends on the source type, whether the data arrives in batches or streams, and the required latency.

Google Cloud offers several tools to collect data from diverse sources. Dataflow processes both batch and streaming data from databases such as Cloud SQL, Spanner, AlloyDB, and Firestore, as well as from external APIs and streaming platforms like Pub/Sub and Apache Kafka. Cloud Storage serves as a centralized data lake for raw files, and BigLake tables allow those files to be queried directly by BigQuery without moving the data. For simpler, event-driven ingestion, Cloud Run functions can be triggered by changes in data sources for real-time processing. BigQuery handles classical tabular data that is accessed frequently. Dataplex Universal Catalog governs metadata across all these ingestion sources, providing a single view of the data landscape.

Data Transformation for Quality and Reuse

Once data is ingested, it often requires transformation to meet quality standards or prepare it for analysis and machine learning. Dataflow and Dataform automate transformations such as feature scaling, encoding categorical variables, and creating new features in batch, streaming, or real-time modes. The choice between batch, streaming, or real-time transformation depends on latency requirements and the nature of the data. Vertex AI Feature Store centralizes features so they can be reused across different models and projects, ensuring consistency in ML workflows.

Data Lifecycle Management

Data lifecycle management establishes policies for when data should be moved to cheaper storage or deleted. Cloud Storage object lifecycle policies automatically transition objects to lower-cost storage classes such as Nearline, Coldline, or Archive based on age, or delete them entirely. In BigQuery, dataset time travel and table snapshots allow point-in-time recovery beyond the default window. The platform must track data provenance and lineage — understanding where data originates, how it transforms, and when it should be archived or deleted — to maintain compliance and cost control.

Data Governance and Cataloging

Dataplex Universal Catalog provides an intelligent data fabric that unifies distributed data without physically moving it. It automatically ingests metadata from supported Google Cloud sources and allows enrichment with business metadata through aspects, making data discoverable and trustworthy. BigQuery sharing enables secure exchange of data assets across organizations through data exchanges and listings, and data clean rooms allow secure collaboration on AI and ML projects with external teams.

Security and Compliance

Security must be built into every stage of the pipeline. This includes implementing identity and access management (IAM) with granular permissions, using customer-managed encryption keys (CMEK) for additional data protection, and applying Sensitive Data Protection to discover, classify, and de-identify sensitive data. Audit logs through Cloud Logging track who accesses data and what changes are made. Within BigQuery, column-level security, row-level security, and dynamic data masking provide additional layers of protection for confidential information.

Architecting the Storage Tiers

A multi-tiered storage and processing strategy balances cost, performance, and data access patterns by using the right service for each stage of the data lifecycle. Cloud Storage handles raw, archival, and low-cost storage due to its extreme durability, expansive scale, and low-cost storage classes (Nearline, Coldline, Archive). For data that requires frequent analytical queries, you load curated subsets into BigQuery, a serverless data warehouse designed for petabyte-scale SQL analytics. A key design pattern is using BigLake tables, which create a unified logical layer over data stored in Cloud Storage, separating storage from compute and allowing direct querying without data movement.

Optimizing BigQuery for Performance and Cost

Within BigQuery, you optimize tables for analytical workloads by designing appropriate physical layouts. Table partitioning divides large tables into segments, often by date, so that queries scan only relevant partitions, improving performance and reducing cost. Clustering orders data within each partition based on one or more columns, enabling faster filtering and aggregation. For frequently accessed complex queries, materialized views precompute results and provide sub-second latency for repetitive analytical patterns.

Governing and Securing the Data Platform

Governance and security must span both storage tiers. Dataplex unifies policy management across BigQuery and Cloud Storage, automating governance. Sensitive data discovered during ingestion into Cloud Storage can be classified and de-identified using Sensitive Data Protection. Within BigQuery, you enforce fine-grained access using column-level security with policy tags and dynamic data masking to protect confidential information at query runtime.

Building a Federated Governance Model

Enable Federated Data Discovery and Access

Understanding Distributed Data Environments

In a federated data governance model, organizations must manage data that lives across multiple distributed systems. Data can be stored in different deployment locations depending on availability and residency needs. Zonal systems are databases tied to a single zone that become unavailable if that zone fails. Regional systems are databases tied to a region that remain accessible if at least one zone is online. Cross-regional systems span multiple regions to ensure high availability. Understanding these deployment architectures is essential because each model presents different challenges for governance and access.

Metadata Management and Data Cataloging

Implementing metadata management and data cataloging systems enables unified data discovery across distributed sources. Organizations must establish clear data ownership, security policies, and compliance controls that work across different distribution models. These include partitioned databases without cross-database dependencies, asynchronously replicated databases, and fully active-active synchronized distributed systems. A data catalog serves as the central registry that allows users to find relevant data assets regardless of where they are physically stored.

Access Controls and Audit Trails

Access controls and audit trails are fundamental components of federated data governance. Organizations must track data lineage and usage to maintain compliance with regulatory requirements. The choice between optimistic concurrency and pessimistic concurrency controls impacts how access is managed and monitored. Optimistic concurrency assumes conflicts are rare and checks for them only when committing, while pessimistic concurrency locks data before any operation to prevent conflicts. Transactionally synchronized databases provide the strongest consistency guarantees but require careful planning to ensure audit trails accurately reflect data movement across deployment locations.

Implement Cross-System Policy Enforcement

The Federated Governance Model

A federated governance model manages data across separate, distributed systems while ensuring consistent rules are followed. In this model, a central data platform team creates shared tools and infrastructure that different data domain teams use, while these distributed teams still have the freedom to manage their own data products. The key challenge is enforcing organization-wide security, privacy, and compliance policies across all these independent teams and systems without creating a central bottleneck.

Centralized Policy Definition and Tooling

Centralized policy definitions are the core rules for data security, privacy, and access that everyone must follow. The central platform team provides tools like automated metadata annotation systems that help apply these policies. A data governance engineer builds software that automatically tags data with required classifications and then uses those tags to automatically check if data products comply with policies. This automation is essential for scaling governance across many teams.

Resource Hierarchy and Policy Inheritance

Policy enforcement relies on understanding the resource hierarchy in Google Cloud, which includes the Organization, folders, and projects. Policies set at a high level like the Organization are inherited by everything underneath them. Tools like Organization Policy Constraints and IAM (Identity and Access Management) enforce compliance. For instance, a policy could require multi-factor authentication for all users or prevent certain network configurations. Effective logging and monitoring with tools like Cloud Audit Logs track user actions and detect policy violations.

Network Segmentation for Compliance

Network segmentation places sensitive systems like a payment card environment (CDE) into a strictly isolated, dedicated Virtual Private Cloud (VPC). VPC Service Controls and precise firewall rules enforce the policy that only authorized, trusted systems can communicate with it. This physically enforces data access policies by removing network routes to untrusted systems, which is a requirement for standards like PCI DSS. The network design must align with the resource hierarchy to maintain a clear and enforceable scope.

Establish Data Domains and Ownership Structures

Data Distribution Models

When designing a distributed data platform, establishing clear boundaries across distributed data systems is essential for maintaining data consistency. Organizations can deploy databases using various deployment locations to meet residency and availability requirements. Choosing the right distribution model allows teams to manage data boundaries effectively across different environments.

Assigning Data Ownership

Establishing data ownership requires assigning specific IAM roles to enforce security policies and compliance. Administrators must ensure access control policies are strictly enforced across all database systems to prevent unauthorized access. Key roles used to establish authority include BigQuery Data Owner, Dataplex Storage Data Owner, and Databases Admin. Organizations should never grant service agent roles to regular users, as these are reserved solely for automated service agents.

Stewardship Personas and Responsibilities

A federated governance model relies on defining distinct personas to divide stewardship responsibilities across the platform. Assigning explicit roles ensures each domain has designated specialists accountable for specific operations. The Cluster administrator is responsible for deploying clusters and configuring permissions, while the Network administrator maintains secure network connectivity and VPC routing rules. The Application owner focuses on developing and monitoring domain applications to ensure high performance and low latency.

Data Movement and Conflict Resolution

As data domains interact, managing data movement and resolving conflicts becomes critical to federated governance. During data migrations, systems can perform active-active migration where writes happen on both source and target databases, or active-passive migration which allows only read-only access on target systems. To prevent inconsistencies during these parallel processes, teams must implement strict conflict resolution rules. Database systems use concurrency controls — either optimistic or pessimistic — to resolve data contention when multiple operations compete to modify the same document.

Key Facts

• Dataflow processes both batch and streaming data from databases, APIs, and message queues, while Cloud Run functions handle simpler event-driven ingestion.
• BigLake tables create a unified logical layer over data in Cloud Storage, allowing BigQuery to query it directly without moving the data, separating storage from compute.
• BigQuery table partitioning and clustering reduce query costs and improve performance by limiting scans and ordering data within partitions.
• Dataplex uses lakes and zones to logically organize distributed data and enforce consistent governance across both Cloud Storage and BigQuery.
• A federated governance model balances centralized policy enforcement with decentralized data management, where a central platform team provides tools while domain teams manage their own data products.
• The resource hierarchy in Google Cloud (Organization → folders → projects) enables policy inheritance, where Organization-level policies cascade down to all underlying resources.
• Concurrency control options — optimistic (checks conflicts at commit time) or pessimistic (locks data before operation) — affect how access is managed in federated environments.