Data ingestion is the process of bringing data from external sources into a Google Cloud data platform. The choice of ingestion tool depends on the source type, whether the data arrives in batches or streams, and the required latency. After ingestion, data must be transformed to meet quality standards, and lifecycle policies must control its retention, archival, and deletion to maintain efficiency and compliance.
Google Cloud offers several tools to collect data from diverse sources. Dataflow processes both batch and streaming data from databases such as Cloud SQL, Spanner, AlloyDB, and Firestore, as well as from external APIs and streaming platforms like Pub/Sub and Apache Kafka. Cloud Storage serves as a centralized data lake for raw files, and BigLake tables allow those files to be queried directly by BigQuery without moving the data. For simpler, event-driven ingestion, Cloud Run functions can be triggered by changes in data sources for real-time processing. BigQuery itself handles classical tabular data that is accessed frequently. Dataplex Universal Catalog governs metadata across all these ingestion sources, providing a single view of the data landscape.
Once data is ingested, it often requires transformation to meet quality standards or prepare it for analysis and machine learning. Dataflow and Dataform automate transformations such as feature scaling, encoding categorical variables, and creating new features in batch, streaming, or real-time modes. The choice between batch, streaming, or real-time transformation depends on latency requirements and the nature of the data. Vertex AI Feature Store centralizes features so they can be reused across different models and projects, ensuring consistency in ML workflows.
Data lifecycle management establishes policies for when data should be moved to cheaper storage or deleted. Cloud Storage object lifecycle policies automatically transition objects to lower-cost storage classes such as Nearline, Coldline, or Archive based on age, or delete them entirely. In BigQuery, dataset time travel and table snapshots allow point-in-time recovery beyond the default window. The platform must track data provenance and lineage — understanding where data originates, how it transforms, and when it should be archived or deleted — to maintain compliance and cost control.
Dataplex Universal Catalog provides an intelligent data fabric that unifies distributed data without physically moving it. It automatically ingests metadata from supported Google Cloud sources and allows enrichment with business metadata through aspects, making data discoverable and trustworthy. BigQuery sharing enables secure exchange of data assets across organizations through data exchanges and listings, and data clean rooms allow secure collaboration on AI and ML projects with external teams.
Security must be built into every stage of the pipeline. This includes implementing identity and access management (IAM) with granular permissions, using customer-managed encryption keys (CMEK) for additional data protection, and applying Sensitive Data Protection to discover, classify, and de-identify sensitive data. Audit logs through Cloud Logging track who accesses data and what changes are made. Within BigQuery, column-level security, row-level security, and dynamic data masking provide additional layers of protection for confidential information.
A multi-tiered storage and processing strategy balances cost, performance, and data access patterns by using the right service for each stage of the data lifecycle. Cloud Storage handles raw, archival, and low-cost storage, while BigQuery provides high-performance analytical processing. The two services are integrated through BigLake tables, which let you query Cloud Storage data directly from BigQuery without moving it.
Cloud Storage is ideal for ingesting and storing raw data, semi-structured files, and long-term archival due to its extreme durability, expansive scale, and low-cost storage classes (Nearline, Coldline, Archive). For data that requires frequent analytical queries, you load curated subsets into BigQuery, a serverless data warehouse designed for petabyte-scale SQL analytics. A key design pattern is using BigLake tables, which create a unified logical layer over data stored in Cloud Storage, separating storage from compute and allowing direct querying without data movement.
Within BigQuery, you optimize tables for analytical workloads by designing appropriate physical layouts. Table partitioning divides large tables into segments, often by date, so that queries scan only relevant partitions, improving performance and reducing cost. Clustering orders data within each partition based on one or more columns, enabling faster filtering and aggregation. For frequently accessed complex queries, materialized views precompute results and provide sub-second latency for repetitive analytical patterns.
Governance and security must span both storage tiers. Dataplex unifies policy management across BigQuery and Cloud Storage, automating governance. Sensitive data discovered during ingestion into Cloud Storage can be classified and de-identified using Sensitive Data Protection. Within BigQuery, you enforce fine-grained access using column-level security with policy tags and dynamic data masking to protect confidential information at query runtime.
Cloud Storage object lifecycle policies automatically transition data to cheaper storage classes or delete it based on age, optimizing costs. For business-critical data in BigQuery, table snapshots enable point-in-time recovery beyond the standard time-travel window. To ensure high availability, you architect for redundancy using BigQuery’s inherent replication and consider cross-region dataset copies for geo-redundancy, weighing trade-offs between replication latency, recovery point objectives (RPO), and cost.
Dataplex is an intelligent data fabric that unifies distributed data without requiring physical consolidation. It organizes data assets into logical structures called lakes and zones, abstracting underlying storage like Cloud Storage and BigQuery. Data flows from raw landing areas (raw zones) into curated zones after refinement and quality checks. This logical domain model enforces consistent governance and simplifies regulatory compliance across diverse storage environments.
The Dataplex Catalog is a fully managed, scalable metadata management service that provides a single source of truth for all enterprise data assets. It automatically ingests metadata from Google Cloud sources, and custom entry systems allow cataloging external data repositories. Technical metadata is enriched with business context using aspects, which tag datasets with ownership, sensitivity levels, and lineage. This unified catalog ensures that data scientists, analysts, and ML practitioners can search and reuse trusted data products across the entire lifecycle.
Robust governance requires strict access control policies across the unified platform. You can use Google Groups and Dataplex roles to administer permissions to lakes and zones. VPC Service Controls mitigate the risk of data exfiltration. For highly sensitive data, policy tags enforce column-level security, restricting access to authorized users with roles such as Fine-Grained Reader. Dynamic data masking selectively obscures sensitive fields at query runtime based on the user’s role, keeping data secure even when shared across organization boundaries.
Dataplex integrates with other Google Cloud tools to automate operations and simplify sharing. Auto data quality validates datasets against predefined business rules, building trust without manual engineering. Curated data products can be securely exposed through BigQuery sharing and data exchanges, allowing consumers to access analytical assets across different domains. Unstructured data in Cloud Storage can be governed through BigQuery object tables and accessed via signed URLs. This toolset enables reliable, compliant, and cost-effective data architectures that scale to meet diverse analytical demands.
professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Dataflow processes both batch and streaming data from databases, APIs, and streaming platforms like Pub/Sub and Apache Kafka, while Cloud Run functions handle simpler, event-driven ingestion triggered by changes in data sources for real-time processing.
BigLake tables create a unified logical layer over data stored in Cloud Storage, allowing BigQuery to query that data directly without moving it, which separates storage from compute and enables direct querying of raw files.
Table partitioning divides large tables into segments, often by date, so queries scan only relevant partitions, improving performance and reducing cost; clustering orders data within each partition based on one or more columns, enabling faster filtering and aggregation.
Dataplex Catalog is a fully managed, scalable metadata management service that provides a single source of truth for all enterprise data assets; it automatically ingests metadata from Google Cloud sources and allows enrichment with business context through aspects, making data discoverable and trustworthy.
An enterprise is architecting a multi-tiered data storage and analytics platform on Google Cloud to handle hundreds of terabytes of transactional log data. The solution must satisfy the following architectural requirements:
StudyCerts
StudyCertscustomer_id, store_id, and product_category.Which multi-tiered storage and processing architecture should you implement?