Orchestrate Dataproc Spark Bigquery Cloud Composer

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise data engineering team is designing a daily data pipeline in Cloud Composer to orchestrate a multi-step analytical workflow. The pipeline must load external files from Cloud Storage into BigQuery, execute an aggregation query, spin up compute resources to process a large Apache Spark batch transformation job, clean up compute resources immediately after completion to minimize costs, and adhere to Google Cloud security best practices for identity and access management.

How should the team design this Cloud Composer orchestration workflow?

Author a DAG that connects to a persistent, 24/7 Dataproc cluster using custom SSH operators and retains the cluster between pipeline runs, while granting all Airflow users the Project Owner IAM role.

Author a DAG using the PythonOperator and BashOperator to run Spark scripts and query executions directly inside the Cloud Composer Airflow worker containers using the default Compute Engine service account.

Author a DAG using specialized Google Cloud operators to orchestrate BigQuery jobs and create an ephemeral Dataproc cluster that runs the Spark job and is deleted afterward, assigning a dedicated user-managed service account with the Composer Worker role and scoped resource permissions.

Author a DAG that triggers Google Cloud Workflows via Cloud Functions, while deploying long-running Spark daemon pods directly into the Cloud Composer system GKE cluster namespaces.

Professional Cloud Data Engineer

Orchestrate Dataproc Spark Bigquery Cloud Composer

Related questions