Pipeline Anomaly Detection Custom Metrics Alerting

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise data analytics platform ingests high-frequency transaction records through a distributed data processing pipeline before storing them in Google Cloud databases. During recent production runs, upstream schema mutations and data source drift caused unexpected null distributions and value outliers, degrading downstream analytical fidelity.

You need to design an automated validation and monitoring architecture that accomplishes the following:

Emits custom validation metrics (such as null ratios and out-of-bounds record rates) from the pipeline
Detects statistical anomalies against historical baseline observation periods
Triggers automated alerting and remediation workflows to quarantine malformed records without halting the entire ingestion pipeline

Which architecture should you implement?

Deploy the AlloyDB Omni Kubernetes Operator to poll DBCluster status YAML files for critical incident error codes, and execute heap dumps to automatically isolate corrupted data rows.

Instrument the pipeline to log structured validation errors to Cloud Logging, create log-based custom metrics in Cloud Monitoring with anomaly detection alerting policies against baseline observation thresholds, and route alert notifications via Pub/Sub to a Cloud Run remediation service that isolates drifting records to a dead-letter quarantine table.

Enable Spark dynamic allocation on Dataproc with dynamic resource scaling up to 100% scaleUpFactor to automatically drop pending task containers whenever data validation thresholds fail in YARN.

Configure Cloud SQL to automatically scale up compute size and memory when lock contention spikes, and use the database error log to trigger near-zero downtime scale-down events once outlier records are committed.

Professional Cloud Data Engineer

Pipeline Anomaly Detection Custom Metrics Alerting

Related questions