An enterprise analytics team orchestrates daily data pipeline workflows in Cloud Composer to refresh BigQuery datasets and publish operational dashboards. The data engineering team needs to operationalize these reporting pipelines under strict governance requirements: Automatically track end-to-end data lineage from source data to downstream BigQuery reporting tables in Dataplex Universal Catalog. Ensure that lineage metadata is accurately captured for both native BigQuery operators and custom Python transformation tasks. Maintain least-privilege security and operational resilience without risking reporting task failures if lineage metadata submission experiences transient issues. Which solution should the team implement?

Enable the Data Lineage API and activate Dataplex data lineage integration on the Cloud Composer environment. Use standard BigQuery operators for automated lineage tracking, define explicit `inlets` and `outlets` with `BigQueryTable` entities for custom `PythonOperator` tasks, and assign the `roles/composer.worker` role to the environment's user-managed service account.: ### What the Solution Is This solution establishes automated metadata tracking, governance, and secure pipeline orchestration by combining **Cloud Composer**, **Dataplex Universal Catalog**, and the **Data Lineage API** using native OpenLineage integration. ### How It Meets All Requirements - **Automated lineage tracking**: Enabling the **Data Lineage API** and setting the Cloud Composer environment's integration flag (`--enable-cloud-data-lineage-integration`) automatically activates the preinstalled `apache-airflow-providers-openlineage` library. Standard operators like `BigQueryInsertJobOperator` automatically generate and emit lineage events to Dataplex without requiring custom code. - **Custom Python transformation tracking**: By populating the `inlets` and `outlets` attributes of `PythonOperator` tasks with `BigQueryTable` objects (or `AUTO` for upstream dependencies), custom execution code explicitly registers lineage relations in the catalog. - **Least-privilege security**: Applying the **Composer Worker** (`roles/composer.worker`) role to a dedicated user-managed service account provides the minimum IAM permissions required for environment operation, PyPI package handling, and Data Lineage API write operations. - **Operational resilience**: Cloud Composer executes lineage reporting via asynchronous RPC calls at the end of task execution. Lineage emission takes roughly 1–2 seconds, and DAG tasks will not fail if lineage cannot be reported due to transient network issues. ### Key Advantages and Capabilities - **Native OpenLineage integration**: Eliminates custom instrumentation boilerplate for standard analytical queries. - **Robust fault isolation**: Ensures that metadata telemetry failures never interrupt critical business report refreshes. - **Granular catalog discovery**: Surfaces complete DAG-to-table lineage graphs directly within the Dataplex Universal Catalog console. - **Adherence to least privilege**: Protects execution nodes from excessive default Compute Engine permissions. ### Why This Is Optimal This architecture leverages fully managed Google Cloud services to achieve automated pipeline tracking with zero custom backend maintenance. It guarantees strict governance while maintaining robust operational SLAs for analytical reporting.

Operationalize Composer Lineage Bigquery Reporting Pipelines | professional-cloud-data-engineer

Operationalize Composer Lineage Bigquery Reporting Pipelines

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise analytics team orchestrates daily data pipeline workflows in Cloud Composer to refresh BigQuery datasets and publish operational dashboards. The data engineering team needs to operationalize these reporting pipelines under strict governance requirements:

Automatically track end-to-end data lineage from source data to downstream BigQuery reporting tables in Dataplex Universal Catalog.
Ensure that lineage metadata is accurately captured for both native BigQuery operators and custom Python transformation tasks.
Maintain least-privilege security and operational resilience without risking reporting task failures if lineage metadata submission experiences transient issues.

Which solution should the team implement?

Configure a custom Airflow Lineage Backend pointing to Cloud Logging, parse task logs using Cloud Functions to extract SQL queries, and assign the roles/editor basic role to the Cloud Composer service account.

Deploy DAGs using only BashOperator invoking the bq CLI tool, configure Airflow task execution settings with execute_tasks_new_python_interpreter = True, and assign roles/owner to the default Compute Engine service account.

Enable Dataplex data lineage integration, write custom Bash scripts within DAGs to push metadata records directly to the Airflow Cloud SQL metadata database, and grant the environment service account roles/cloudsql.admin access.

Enable the Data Lineage API and activate Dataplex data lineage integration on the Cloud Composer environment. Use standard BigQuery operators for automated lineage tracking, define explicit inlets and outlets with BigQueryTable entities for custom PythonOperator tasks, and assign the roles/composer.worker role to the environment's user-managed service account.

Professional Cloud Data Engineer

Operationalize Composer Lineage Bigquery Reporting Pipelines

Related questions