Reusable Data Fusion Pipeline Dynamic Quality Checks

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise is designing a centralized data preparation and cleaning architecture using Cloud Data Fusion. Data files from multiple external partners are uploaded to separate Cloud Storage buckets, and each dataset requires distinct data quality validation rules, formatting directives, and output destinations.

Creating and maintaining individual static pipelines for dozens of source feeds has caused significant operational overhead and deployment delays. The data engineering team requires a single, maintainable pipeline structure that dynamically determines its input source, validation checks, and output target during each scheduled execution.

How should you configure the Cloud Data Fusion pipeline to support these dynamic validation and cleaning requirements?

Implement an Action plugin within a single static pipeline that compiles dynamic Apache Spark Scala scripts from Cloud Storage and executes them within the pipeline execution container

Develop a custom Cloud Function that modifies the underlying pipeline JSON deployment specification and invokes the Cloud Data Fusion REST API to redeploy the pipeline before every run

Construct a reusable pipeline with parameterized macro arguments and use the Cloud Storage Argument Setter plugin to dynamically pass input paths, quality validation directives, and output configurations at runtime

Create separate static pipelines for each partner feed and use Cloud Composer DAGs to orchestrate and execute each pipeline concurrently based on file arrival triggers

Professional Cloud Data Engineer

Reusable Data Fusion Pipeline Dynamic Quality Checks

Related questions