Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Reusable Pipelines with Argument Setter Macros
Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise is designing a centralized data preparation and cleaning architecture using Cloud Data Fusion. Data files from multiple external partners are uploaded to separate Cloud Storage buckets, and each dataset requires distinct data quality validation rules, formatting directives, and output destinations.
Creating and maintaining individual static pipelines for dozens of source feeds has caused significant operational overhead and deployment delays. The data engineering team requires a single, maintainable pipeline structure that dynamically determines its input source, validation checks, and output target during each scheduled execution.
How should you configure the Cloud Data Fusion pipeline to support these dynamic validation and cleaning requirements?
Explanations
Develop a custom Cloud Function that modifies the underlying pipeline JSON deployment specification and invokes the Cloud Data Fusion REST API to redeploy the pipeline before every run
Develop a custom Cloud Function that modifies the underlying pipeline JSON deployment specification and invokes the Cloud Data Fusion REST API to redeploy the pipeline before every run
What Modifying Pipeline JSON Specifications Does
Deploying a pipeline via the Cloud Data Fusion REST API involves submitting a serialized JSON workflow representation that defines all stages, plugins, connections, and properties. Automating this via Cloud Functions reconfigures the pipeline definition inside the namespace programmatically.
Why It Doesn't Meet the Requirement
- Deployment and Locking Conflicts: Repeatedly deploying and recompiling pipelines before every execution creates race conditions if multiple runs overlap, and introduces unnecessary latency into scheduling.
- High Maintenance Overhead: Maintaining custom scripting logic that parses, mutates, and validates raw JSON pipeline graphs introduces technical debt and high failure risks compared to native runtime macro substitution.
- Version Control Disruption: Continuous redeployment overwrites the pipeline definition, making historical audits and lineage tracking of pipeline versions difficult to manage.
Key Characteristics and Limitations
- The Cloud Data Fusion REST API is designed for CI/CD release management, not per-job dynamic parameter passing.
- Modifying pipeline graph specifications at runtime bypasses built-in parameterization frameworks.
- REST-based deployment introduces unnecessary API quotas and credential management requirements.
This Answer Would Be Correct If
You needed to automate initial environment provisioning, stage promotion, or CI/CD deployments across development, staging, and production environments.
Construct a reusable pipeline with parameterized macro arguments and use the Cloud Storage Argument Setter plugin to dynamically pass input paths, quality validation directives, and output configurations at runtime
Construct a reusable pipeline with parameterized macro arguments and use the Cloud Storage Argument Setter plugin to dynamically pass input paths, quality validation directives, and output configurations at runtime
What Reusable Pipelines and Argument Setters Are
Cloud Data Fusion reusable pipelines decouple the pipeline execution DAG from static node configurations by utilizing runtime macros and dynamic argument plugins. Rather than hardcoding source paths, transformation directives, or sink targets, the pipeline nodes evaluate parameter variables during runtime initialization.
How It Meets the Requirements
- Dynamic Quality Validation Rules: The Cloud Storage Argument Setter plugin reads runtime configuration files (such as JSON or property definitions) from a designated bucket before pipeline execution begins and populates macro variables that dictate the specific data quality validation logic applied to that batch.
- Dynamic Source and Destination: Node properties for Cloud Storage sources and output sinks are configured with macro placeholders (e.g.,
${input.path},${output.path}), allowing a single deployed pipeline to ingest from and write to different endpoints on each run. - Operational Efficiency: Eliminates the operational overhead of managing dozens of individual static pipelines, consolidating maintenance into a single pipeline template.
Key Advantages
- Centralized Pipeline Governance: Schema changes, engine upgrades, or logging improvements only need to be applied to one template pipeline.
- Configuration-Driven Ingestion: New source feeds or customized data quality rules can be onboarded by supplying a new configuration file without modifying or redeploying the core pipeline code.
- Auditing and Traceability: Runtime arguments and execution parameters are tracked per run, ensuring full visibility into which validation rules were applied to specific datasets.
Why This Is Optimal
Using Cloud Data Fusion's native dynamic argument resolution provides an out-of-the-box, serverless mechanism to orchestrate varied validation and transformation logic without developing custom orchestration middleware.
Create separate static pipelines for each partner feed and use Cloud Composer DAGs to orchestrate and execute each pipeline concurrently based on file arrival triggers
Create separate static pipelines for each partner feed and use Cloud Composer DAGs to orchestrate and execute each pipeline concurrently based on file arrival triggers
What Static Pipeline Proliferation Does
This approach builds discrete, dedicated Cloud Data Fusion pipelines for every distinct business unit or partner source, hardcoding the specific transformations, Wrangler directives, and output locations inside each pipeline graph.
Why It Doesn't Meet the Requirement
- Operational Overhead: Fails the explicit requirement to reduce operational overhead. Managing dozens of distinct static pipelines requires duplicating boilerplate logic across every instance.
- Maintenance Bottlenecks: Any bug fix, security patch, or shared cleansing standard (such as timestamp normalization or null imputation) must be manually updated across dozens of individual pipelines.
- Resource Inefficiency: Provisioning dozens of static pipelines increases administrative surface area within the Cloud Data Fusion instance and complicates central monitoring.
Key Characteristics and Limitations
- Static pipelines hardcode configuration properties directly into plugin nodes.
- Pipeline duplication scales linearly with the number of partners, degrading developer productivity.
- Orchestrating multiple duplicate pipelines through Cloud Composer adds orchestration complexity without addressing the underlying lack of pipeline reusability.
This Answer Would Be Correct If
Different partner feeds required fundamentally divergent DAG topologies, completely different source/sink technologies (e.g., Kafka to Spanner vs. Cloud Storage to BigQuery), or strict isolated compute profiles.
Implement an Action plugin within a single static pipeline that compiles dynamic Apache Spark Scala scripts from Cloud Storage and executes them within the pipeline execution container
Implement an Action plugin within a single static pipeline that compiles dynamic Apache Spark Scala scripts from Cloud Storage and executes them within the pipeline execution container
What Action Plugins and Dynamic Script Compilation Do
Action plugins in Cloud Data Fusion execute custom operational tasks (such as running database scripts, invoking external webhooks, or running command-line tasks) either before the pipeline executes or after it terminates.
Why It Doesn't Meet the Requirement
- Bypasses Native Visual Quality Tooling: Dynamically compiling custom Spark Scala code eliminates the fidelity and visibility benefits of Cloud Data Fusion’s visual transformations, data quality plugins, and native lineage tracking.
- Reliability and Debugging Risks: Runtime compilation of unmanaged code within pipeline action nodes introduces runtime syntax errors, missing dependency issues, and unpredictable resource allocation failures.
- Not Designed for Core Data Processing: Action plugins do not process distributed pipeline data records in-stream; they run on the pipeline driver to initiate pre- or post-execution actions.
Key Characteristics and Limitations
- Action plugins are intended for setup/teardown tasks (e.g., moving files, triggering alerts, truncating staging tables), not for processing and validating streaming or batch data flow.
- Custom Spark scripts executed in action plugins do not integrate with Data Fusion’s native data quality, error collector, or Wrangler directives.
- Introduces complex code maintenance into a visual ETL platform.
This Answer Would Be Correct If
You needed to perform an external administrative task, such as creating an external database table or sending a custom webhook notification prior to running a batch load.
Try 10 interactive questions like this one, free
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
- 1,353 practice questions across all 5 exam domains
- Every answer explained: why it is right, and why the others are not
- Scout, an AI tutor that answers your follow-up questions
- Readiness tracking by domain, so you know when to book the exam
Related questions
Keep the momentum going with these hand-picked practice scenarios
AdvancedProfessional Data Engineer
Directive Prompt Design Framework
Preparing and cleaning data (e.g., Dataform, Dataflow, and Cloud Data Fusion, prompting LLMs for query generation)
AdvancedProfessional Data Engineer
Dataform BigQuery SQL ELT
Preparing and cleaning data (e.g., Dataform, Dataflow, and Cloud Data Fusion, prompting LLMs for query generation)
AdvancedProfessional Data Engineer
Dataplex Sensitive Data Protection Dataflow Freshness Monitoring
Monitoring the data lake
AdvancedProfessional Data Engineer
BigQuery Nested and Repeated Fields
Designing the data model
AdvancedProfessional Data Engineer
Cloud Composer Ephemeral Dataproc Clusters
Deciding between persistent or job-based data clusters (e.g., Dataproc)
FundamentalAssociate Data Practitioner
gcloud storage cp --recursive
Load data into Google Cloud storage systems using the appropriate tool (e.g., gcloud and BQ CLI, Storage Transfer Service, BigQuery Data Transfer Service, client libraries)
Last updated Validated