professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Your team orchestrates an enterprise ETL workflow in Cloud Composer that processes analytical datasets across Google Cloud. The workflow uses built-in Google operators for BigQuery, but also includes custom PythonOperator and BashOperator tasks that transform intermediate data before writing the final output to BigQuery.
The Data Lineage API is enabled in your project, and Dataplex data lineage integration is enabled on the Cloud Composer environment. However, the data governance team notices that while lineage is automatically captured for native BigQuery operators, the intermediate transformations performed by the PythonOperator and BashOperator tasks are missing from the Dataplex lineage graphs.
What should you do to ensure end-to-end data lineage is captured in Dataplex for the entire workflow?
Cloud Composer integrates with Dataplex Universal Catalog by utilizing the apache-airflow-providers-openlineage package to capture workflow execution events and report them to the Data Lineage API. While native operators (such as BigQueryInsertJobOperator) automatically extract and report lineage metadata out of the box, generic operators like BashOperator and PythonOperator do not have built-in SQL parsing or dataset inspection logic. To report lineage for these operators, users must explicitly annotate tasks with custom lineage entities.
StudyCerts
StudyCertsinlets (input datasets) and outlets (output datasets) on the BashOperator and PythonOperator definitions informs the OpenLineage backend of the data assets being read and modified.AUTO for the inlets parameter dynamically links the task's inputs to the outlets of its immediately preceding upstream task, bridging any gaps in the lineage graph across mixed operator types.AUTO keyword minimizes redundant dataset definitions between adjacent workflow stages.This approach directly leverages the native Dataplex data lineage integration mechanism built into Cloud Composer without requiring additional infrastructure, third-party libraries, or custom lineage backend implementations.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.