Automated Pipeline Recovery Pubsub Gcs Composer

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise ingestion pipeline processes streaming transactional data that is staged in Cloud Storage before undergoing complex transformations orchestrated by Cloud Composer. Recent schema deviations and malformed payloads have corrupted downstream analytics tables, leading to pipeline halts and unrecoverable data.

You need to architect an automated recovery and reprocessing workflow that satisfies the following operational requirements:

Automatically quarantine unparseable or corrupted streaming messages without crashing primary pipelines
Enable point-in-time recovery to a clean state if corrupted files are written to Cloud Storage
Automatically orchestrate reprocessing from validated checkpoints while maintaining security governance and execution auditability

Which architecture should you implement?

Deploy an AlloyDB Omni instance to replicate streaming transactions, enable PostgreSQL Write-Ahead Logging (WAL) archiving to Cloud Storage, and execute ad-hoc pg_restore scripts from Cloud Shell to reprocess data.

Configure a Pub/Sub dead-letter topic to isolate invalid streaming messages, enable Object Versioning on the Cloud Storage staging bucket to restore prior uncorrupted states, and deploy Cloud Composer DAGs via CI/CD to orchestrate automated reprocessing from validated checkpoints.

Configure an exponential backoff retry policy on the primary Pub/Sub subscription, use Cloud Storage Object Retention Locks to prevent overwrite errors, and allow DAGs to query the Cloud Composer Airflow metadata database directly to identify failed tasks.

Configure Pub/Sub message filtering to drop malformed messages immediately, enable Cloud Storage soft delete with daily snapshots, and grant data engineers direct IAM permissions to edit DAG code directly in the Cloud Composer storage bucket during recovery.

Professional Cloud Data Engineer

Automated Pipeline Recovery Pubsub Gcs Composer

Related questions