An enterprise is designing a centralized orchestration solution for complex batch ETL/ELT pipelines spanning multiple Google Cloud services and on-premises data sources. The architecture must satisfy the following operational, security, and resilience requirements: Workflows must be authored and maintained natively in Python as Directed Acyclic Graphs (DAGs) with automated dependency resolution. Database connection strings, API keys, and sensitive service credentials must be stored securely and retrieved dynamically without hardcoding them in pipeline files. The orchestration infrastructure must be protected against single-zone failures through multi-zone scheduler redundancy and automated database failover. Service access must adhere to the principle of least privilege using a custom identity rather than overly broad default accounts. Which architecture and configuration should the organization deploy?

Deploy Cloud Composer in a Highly Resilient Private IP configuration using a dedicated user-managed service account granted the Composer Worker role, and configure Secret Manager as the Airflow backend for connections and variables.: ### What Cloud Composer in a Highly Resilient Configuration Is **Cloud Composer** is Google Cloud's managed workflow orchestration service built on **Apache Airflow**. In a **Highly Resilient Private IP configuration**, Cloud Composer deploys Airflow components across multiple availability zones with automated failover capabilities and private network isolation. ### How It Meets the Requirements - **Python DAG Authoring**: Cloud Composer natively manages workflows defined in Python as Directed Acyclic Graphs (DAGs), enabling automatic dependency resolution and seamless integration with over 150 Google Cloud operators. - **Secret Manager Integration**: Cloud Composer supports configuring **Secret Manager** as the backend for Airflow connections and variables, allowing workers to fetch sensitive credentials dynamically without storing secrets in DAG code or plain environment variables. - **Operational Resilience**: The Highly Resilient architecture provides a regional **Cloud SQL** metadata database with primary and standby instances distributed across zones, alongside redundant Airflow schedulers, DAG processors, and web servers running in separate zones. - **Least-Privilege IAM**: Assigning a dedicated user-managed service account with the **Composer Worker** (`composer.worker`) role guarantees that Airflow tasks only execute with the exact permissions assigned, preventing the security risks associated with default accounts. ### Key Advantages and Capabilities - **Multi-Zone High Availability**: Automatically recovers from zonal infrastructure failures without dropping pipeline scheduling or corrupting metadata state. - **Centralized Telemetry**: Streams component and DAG execution logs directly to **Cloud Logging** and metrics to **Cloud Monitoring** for immediate observability. - **VPC Service Controls Compatibility**: Private IP environments can be placed inside security perimeters to mitigate data exfiltration risks. ### Why This Is Optimal This architecture directly fulfills all requirements for Python-native data orchestration, enterprise secret management, high availability, and least-privilege access control without requiring custom infrastructure management.

Cloud Composer High Resilience Secret Manager Architecture | professional-cloud-data-engineer

Cloud Composer High Resilience Secret Manager Architecture

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise is designing a centralized orchestration solution for complex batch ETL/ELT pipelines spanning multiple Google Cloud services and on-premises data sources. The architecture must satisfy the following operational, security, and resilience requirements:

Workflows must be authored and maintained natively in Python as Directed Acyclic Graphs (DAGs) with automated dependency resolution.
Database connection strings, API keys, and sensitive service credentials must be stored securely and retrieved dynamically without hardcoding them in pipeline files.
The orchestration infrastructure must be protected against single-zone failures through multi-zone scheduler redundancy and automated database failover.
Service access must adhere to the principle of least privilege using a custom identity rather than overly broad default accounts.

Which architecture and configuration should the organization deploy?

Deploy Cloud Composer in a Highly Resilient Private IP configuration using a dedicated user-managed service account granted the Composer Worker role, and configure Secret Manager as the Airflow backend for connections and variables.

Deploy Cloud Workflows using YAML definitions, inject sensitive connection strings via static workflow environment variables, and grant the workflow's service account the Editor role to access target data services.

Deploy Cloud Workflows to orchestrate containerized Python tasks on Google Kubernetes Engine Autopilot, using Kubernetes Secrets for credential storage and standard cluster auto-repair for resilience.

Deploy Cloud Composer in a standard single-zone Public IP configuration using the default Compute Engine service account, and store Airflow connections inside the Airflow metadata database.

Professional Cloud Data Engineer

Cloud Composer High Resilience Secret Manager Architecture

Related questions