Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An e-commerce company ingests high-throughput user clickstream events via Cloud Pub/Sub and processes them using a streaming Dataflow pipeline before persisting the records into BigQuery. Upstream mobile application updates have occasionally introduced unannounced schema changes, unexpected null values in mandatory telemetry attributes, and malformed JSON payloads, causing unhandled pipeline exceptions and data corruption in downstream reporting tables.
The data engineering team needs to implement an automated validation framework that detects schema drift and corrupted records in real time, isolates non-conforming payloads without interrupting the processing of valid events, and alerts engineers when anomaly rates exceed predefined thresholds.
Which architecture should the team deploy?
Cloud Data Loss Prevention (Sensitive Data Protection) is designed to discover, classify, mask, and tokenize sensitive information such as Personally Identifiable Information (PII) and credentials. Placing a Cloud Function in front of Dataflow introduces an intermediate serverless compute hop to inspect payloads.
The primary objective was scanning user clickstream text fields for exposed credit card numbers or Social Security Numbers prior to pipeline ingestion.
The dead-letter pattern with side outputs is an architectural best practice in Apache Beam and Google Cloud Dataflow for building fault-tolerant streaming pipelines. In this pattern, incoming payloads are parsed and validated inside a DoFn. Valid records are emitted to the standard main PCollection for downstream transformations and storage, while invalid, malformed, or schema-deviant records are caught and routed to a side output PCollection (the dead-letter queue) without halting pipeline execution.
DoFn explicitly parses the JSON payloads against expected schema models, detecting missing mandatory fields, incorrect data types, or unrecognized attributes as each event arrives.NullPointerException or serialization errors.This solution isolates corrupt data at the ingestion stage, protecting downstream analytical data warehouses while maintaining continuous stream availability.
BigQuery schema auto-detection and setting ignoreUnknownValues or lenient schema parameters allows streaming ingestion pipelines to adapt table schemas dynamically or silently drop undeclared columns when inserting data into BigQuery tables.
The workload was an exploratory batch load where schema flexibility is preferred over strict data integrity and governance.
Dataflow system lag measures the time difference between the current time and the timestamp of the oldest unprocessed element in the pipeline. Using Cloud Monitoring and Cloud Functions to stop a streaming job halts pipeline execution when processing bottlenecks or unhandled exceptions cause data backlogs.
The primary goal was emergency circuit-breaking to prevent extreme downstream quota exhaustion during severe infrastructure outages.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated