Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
Your team orchestrates an enterprise ETL workflow in Cloud Composer that processes analytical datasets across Google Cloud. The workflow uses built-in Google operators for BigQuery, but also includes custom PythonOperator and BashOperator tasks that transform intermediate data before writing the final output to BigQuery.
The Data Lineage API is enabled in your project, and Dataplex data lineage integration is enabled on the Cloud Composer environment. However, the data governance team notices that while lineage is automatically captured for native BigQuery operators, the intermediate transformations performed by the PythonOperator and BashOperator tasks are missing from the Dataplex lineage graphs.
What should you do to ensure end-to-end data lineage is captured in Dataplex for the entire workflow?
LineageBackend in the airflow.cfg configuration overrides to route execution metadata directly to Cloud Monitoring.Configuring a custom LineageBackend in airflow.cfg is a legacy Apache Airflow method for routing lineage metadata to external systems, while Cloud Monitoring is Google Cloud's infrastructure and application performance monitoring service.
LineageBackend is not configured. Setting a custom lineage backend prevents Cloud Composer's default OpenLineage provider from sending events to the Data Lineage API.apache-airflow-providers-openlineage integration.You needed to export custom numerical execution metrics or performance telemetry to Cloud Monitoring dashboards rather than tracking data asset lineage in Dataplex.
--experiments=enable_stackdriver_agent_metrics pipeline option and assign the roles/datacatalog.tagEditor role to the Cloud Composer service account.The --experiments=enable_stackdriver_agent_metrics flag is a Dataflow worker pipeline option used to collect VM-level system metrics (such as disk and process telemetry) into Cloud Monitoring. The roles/datacatalog.tagEditor IAM role provides permissions to create and attach metadata tag templates in Dataplex/Data Catalog.
enable_stackdriver_agent_metrics flag applies to Dataflow jobs, not Cloud Composer environments or Airflow task lineage emission.roles/composer.worker) role assigned to the environment service account. Assigning tag editor roles does not cause unsupported Airflow operators to automatically emit lineage events.You were optimizing Dataflow compute workers to report low-level operating system metrics to Cloud Monitoring while managing tag templates.
inlets and outlets parameters on the custom BashOperator and PythonOperator task definitions using lineage entity definitions, utilizing AUTO where applicable to inherit upstream task outlets.Cloud Composer integrates with Dataplex Universal Catalog by utilizing the apache-airflow-providers-openlineage package to capture workflow execution events and report them to the Data Lineage API. While native operators (such as BigQueryInsertJobOperator) automatically extract and report lineage metadata out of the box, generic operators like BashOperator and PythonOperator do not have built-in SQL parsing or dataset inspection logic. To report lineage for these operators, users must explicitly annotate tasks with custom lineage entities.
inlets (input datasets) and outlets (output datasets) on the BashOperator and PythonOperator definitions informs the OpenLineage backend of the data assets being read and modified.AUTO for the inlets parameter dynamically links the task's inputs to the outlets of its immediately preceding upstream task, bridging any gaps in the lineage graph across mixed operator types.AUTO keyword minimizes redundant dataset definitions between adjacent workflow stages.This approach directly leverages the native Dataplex data lineage integration mechanism built into Cloud Composer without requiring additional infrastructure, third-party libraries, or custom lineage backend implementations.
BashOperator and PythonOperator tasks to the KubernetesPodOperator, as containerized pods automatically detect input and output paths.The KubernetesPodOperator executes arbitrary containerized tasks as isolated pods within a Kubernetes cluster (such as the GKE cluster running the Airflow workloads).
KubernetesPodOperator runs containerized workloads in isolation and cannot inspect arbitrary application binaries, scripts, or network sockets to infer which datasets are read or written.BashOperator and PythonOperator, if an operator is not among the automatically supported provider operators, it still requires manual metadata annotation or custom OpenLineage event dispatching.KubernetesPodOperator introduces substantial operational complexity without solving the lineage reporting requirement.KubernetesPodOperator does not provide automatic schema or data access discovery.You needed isolated dependencies, custom runtime binaries, or separate resource allocations that could not be satisfied inside the standard Cloud Composer worker image.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated