professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
An e-commerce company ingests high-throughput user clickstream events via Cloud Pub/Sub and processes them using a streaming Dataflow pipeline before persisting the records into BigQuery. Upstream mobile application updates have occasionally introduced unannounced schema changes, unexpected null values in mandatory telemetry attributes, and malformed JSON payloads, causing unhandled pipeline exceptions and data corruption in downstream reporting tables.
The data engineering team needs to implement an automated validation framework that detects schema drift and corrupted records in real time, isolates non-conforming payloads without interrupting the processing of valid events, and alerts engineers when anomaly rates exceed predefined thresholds.
Which architecture should the team deploy?
Configure an Cloud Monitoring alert on Dataflow system lag that automatically triggers a Cloud Function to stop the streaming job when lag increases
Implement schema validation within a Dataflow DoFn using dead-letter side outputs for malformed or non-conforming records, increment custom pipeline metrics, and emit alerts via Cloud Monitoring
Enable schema auto-detection and ignore unknown values on the BigQuery streaming sink within the Dataflow pipeline to coerce missing and drifted fields automatically
Deploy a Cloud Function subscribed to Pub/Sub to call the Cloud Data Loss Prevention (Cloud DLP) API for schema structural validation before forwarding records to Dataflow
Configure an Cloud Monitoring alert on Dataflow system lag that automatically triggers a Cloud Function to stop the streaming job when lag increases
Implement schema validation within a Dataflow DoFn using dead-letter side outputs for malformed or non-conforming records, increment custom pipeline metrics, and emit alerts via Cloud Monitoring
The dead-letter pattern with side outputs is an architectural best practice in Apache Beam and Google Cloud Dataflow for building fault-tolerant streaming pipelines. In this pattern, incoming payloads are parsed and validated inside a DoFn. Valid records are emitted to the standard main PCollection for downstream transformations and storage, while invalid, malformed, or schema-deviant records are caught and routed to a side output PCollection (the dead-letter queue) without halting pipeline execution.
StudyCerts
StudyCertsDoFn explicitly parses the JSON payloads against expected schema models, detecting missing mandatory fields, incorrect data types, or unrecognized attributes as each event arrives.NullPointerException or serialization errors.This solution isolates corrupt data at the ingestion stage, protecting downstream analytical data warehouses while maintaining continuous stream availability.
Enable schema auto-detection and ignore unknown values on the BigQuery streaming sink within the Dataflow pipeline to coerce missing and drifted fields automatically
Deploy a Cloud Function subscribed to Pub/Sub to call the Cloud Data Loss Prevention (Cloud DLP) API for schema structural validation before forwarding records to Dataflow