Cloud Composer Custom Lineage Inlets Outlets

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

Your team orchestrates an enterprise ETL workflow in Cloud Composer that processes analytical datasets across Google Cloud. The workflow uses built-in Google operators for BigQuery, but also includes custom PythonOperator and BashOperator tasks that transform intermediate data before writing the final output to BigQuery.

The Data Lineage API is enabled in your project, and Dataplex data lineage integration is enabled on the Cloud Composer environment. However, the data governance team notices that while lineage is automatically captured for native BigQuery operators, the intermediate transformations performed by the PythonOperator and BashOperator tasks are missing from the Dataplex lineage graphs.

What should you do to ensure end-to-end data lineage is captured in Dataplex for the entire workflow?

Enable the --experiments=enable_stackdriver_agent_metrics pipeline option and assign the roles/datacatalog.tagEditor role to the Cloud Composer service account.

Configure a custom LineageBackend in the airflow.cfg configuration overrides to route execution metadata directly to Cloud Monitoring.

Migrate all BashOperator and PythonOperator tasks to the KubernetesPodOperator, as containerized pods automatically detect input and output paths.

Define the inlets and outlets parameters on the custom BashOperator and PythonOperator task definitions using lineage entity definitions, utilizing AUTO where applicable to inherit upstream task outlets.

Professional Cloud Data Engineer

Cloud Composer Custom Lineage Inlets Outlets

Related questions