An enterprise e-commerce platform is designing a real-time event ingestion architecture on Google Cloud for order processing and inventory updates. The system must satisfy the following architectural and operational requirements: Ordering Guarantees: Ensure strict in-order message delivery per merchant account (merchant_id). Operational Overhead: Eliminate cluster provisioning, broker patching, ZooKeeper/KRaft maintenance, and manual partition capacity planning. Elastic Scalability: Automatically scale ingestion throughput up or down to handle sudden traffic spikes without manual partition rebalancing. Delivery Semantics: Provide built-in retry mechanisms, dead-letter support, and granular message acknowledgment. Which messaging framework should the enterprise implement?

Google Cloud Pub/Sub with ordering keys configured using the `merchant_id`: ### What Google Cloud Pub/Sub with Ordering Keys Is **Google Cloud Pub/Sub** is a fully managed, serverless, globally distributed messaging service. When configured with **ordering keys**, Pub/Sub guarantees that messages published with the same key (such as `merchant_id`) are delivered to subscribers in the exact order they were received by the service, while still scaling horizontally across independent keys without requiring manual partition management. ### How It Meets All Requirements * **Ordering Guarantees:** Pub/Sub ordering keys ensure strict, FIFO delivery for all messages sharing the same key string (e.g., `merchant_id`). When a subscriber client receives an ordered message, subsequent messages for that key are withheld until the earlier message is acknowledged. * **Operational Overhead:** Pub/Sub is completely serverless. It eliminates operational tasks such as cluster provisioning, broker sizing, storage rebalancing, replication configuration, and Kafka metadata/KRaft cluster maintenance. * **Elastic Scalability:** In standard Apache Kafka architectures, throughput scaling is constrained by the static number of partitions assigned to a topic. Pub/Sub decouples storage and serving dynamically, scaling seamlessly to absorb massive, unpredictable traffic spikes without manual partition re-allocation or partition rebalancing. * **Delivery Semantics:** Pub/Sub offers native individual message acknowledgments (`ack`), negative acknowledgments (`nack`), automatic exponential backoff retries, and integrated dead-letter topics. ### Key Advantages * **Zero Infrastructure Management:** Fully automated infrastructure scaling and automated high-availability across multiple zones. * **Key-Level Concurrency:** Allows massive concurrency across thousands of distinct `merchant_id` values while maintaining strict serialization within each specific merchant stream. * **Native Cloud Integration:** Seamlessly integrates with Dataflow, Cloud Run, Cloud Functions, and BigQuery without custom connectors. ### Why This Is Optimal Compared to managing Apache Kafka clusters, Pub/Sub delivers enterprise-grade ordering guarantees and elastic scale with zero operational overhead, making it the superior choice for cloud-native ingestion.

Compare Pubsub Kafka Event Ingestion Architecture | professional-cloud-data-engineer

AdvancedProfessional Cloud Data Engineer

An enterprise financial organization is designing an automated ingestion and processing architecture to stream high-volume transaction records from Pub/Sub into BigQuery. The architecture must satisfy the following technical requirements: Ingest records directly into BigQuery storage while strictly guaranteeing exactly-once processing semantics. Isolate and extract individual malformed or schema-mismatched rows into a dead-letter collection without failing the entire streaming pipeline. Maintain high throughput without relying on legacy streaming buffer inserts. Which ingestion pathway and processing approach should the organization implement?

Identifying the services (e.g., Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Apache Spark, Hadoop ecosystem, and Apache Kafka)

AdvancedProfessional Cloud Data Engineer

An enterprise is establishing a data analytics platform on Google Cloud and needs to assign the appropriate processing engine for three distinct pipeline workloads: Workload 1: Migrating an existing portfolio of legacy Apache Spark jobs and Hadoop dependencies with minimal code changes while leveraging ephemeral clusters to optimize compute costs. Workload 2: Processing real-time unbounded event streams alongside bounded batch datasets using a fully managed serverless runtime with exactly-once processing semantics. Workload 3: Enabling business analysts to visually design, deploy, and manage ETL/ELT pipelines using a point-and-click graphical interface backed by open-source CDAP. Which service mapping should the enterprise select for Workload 1, Workload 2, and Workload 3, respectively?

AdvancedProfessional Cloud Data Engineer

An organization is designing an automated Dataflow ingestion pipeline to de-identify sensitive structured records containing Personally Identifiable Information (PII) before storing them in BigQuery for analytics. The privacy compliance team establishes the following technical constraints: Identifier Tokenization: The 10-digit numerical patient_account_id column must retain referential integrity for analytical joins, remain reversible for authorized re-identification pipelines, and preserve its original 10-digit numeric character set to maintain backward compatibility with legacy schema validators. Quasi-Identifier Generalization: The patient_age column must be abstracted into discrete 10-year cohorts to mitigate re-identification risk while preserving population distribution metrics. Key Security: Plaintext cryptographic keys must not be exposed in configuration templates; all token encryption keys must be encrypted using Cloud Key Management Service (Cloud KMS). Which transformation configuration in Cloud Sensitive Data Protection (Cloud DLP) meets all of these requirements?

AdvancedProfessional Cloud Data Engineer

An enterprise is designing a multi-environment data processing architecture across separate Development and Production Google Cloud environments. The architecture must satisfy the following governance and compliance requirements: Data Classification and Access Control: A centralized taxonomy must classify sensitive personally identifiable information (PII). Production analysts with explicit clearance must be able to view cleartext values, while developers and testers in the development environment must only see masked data. Cryptographic Security: All persistent data in BigQuery datasets and Cloud Storage buckets must be encrypted using Customer-Managed Encryption Keys (CMEK) via Cloud KMS/Cloud HSM, with strict cryptographic key isolation between environments. Lifecycle Management: Development storage buckets must automatically delete intermediate test data older than 30 days to limit exposure and reduce costs, whereas production audit data must be retained under an immutable compliance policy. Which combination of architectural controls and services best satisfies these requirements?

AdvancedProfessional Cloud Data Engineer

Your financial enterprise processes sensitive customer transaction records subject to strict European Union data residency and sovereignty regulations. The compliance policy mandates the following requirements: All data stored at rest in Cloud Storage must be encrypted using Customer-Managed Encryption Keys (CMEK). Key management cryptographic material and operations must reside exclusively within the europe-west3 (Frankfurt) region. Data processing pipelines running on Cloud Dataflow must execute workers exclusively within europe-west3, while maintaining resilience against potential single-zone capacity constraints or outages. No cross-region network egress or unapproved resource location provisioning is permitted. How should you configure the architecture to enforce compliance and data sovereignty while ensuring job execution resilience?

IntermediateProfessional Cloud Architect

An enterprise is designing a hybrid network architecture to connect its on-premises data center to Google Cloud Virtual Private Cloud (VPC) networks. The organization has identified the following requirements and operational constraints: The expected steady-state throughput is 500 Mbps, with future growth capped below 2 Gbps. The enterprise does not maintain a presence in a Google Cloud colocation facility (point of presence). The enterprise wants private RFC 1918 connectivity backed by an SLA without the overhead of provisioning and managing physical cross-connects or optical routing hardware. Which connectivity solution should the enterprise choose?

Professional Cloud Data Engineer

Compare Pubsub Kafka Event Ingestion Architecture

Related questions