Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Cloud Composer
Topic in 2 exam domains
Google Cloud's managed Apache Airflow service that orchestrates workflows and allows managing user workload secrets and ConfigMaps.
Where this appears on the exam
2 objectives in 2 of 5 domains
Related topics
FoundationsThis topicWorks alongsideBuilds on theseAppliedVPC Service ControlsServiceIdentity and Access ManagementServiceIdentity-Aware ProxyServiceCloud BuildServicePub/SubService
Cloud ComposerService
Compute EngineServiceCloud StorageServiceSecret ManagerServiceCloud RunServiceCloud Run functionsServicePrivate Service ConnectFeaturePub/Sub topicConceptManaged Service for Apache SparkServiceCloud SQLServiceGKEServiceManaged Service for Apache AirflowServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Cloud Composer works with Secret Manager From the official documentation: “Once you configure Secret Manager for your Composer environment, you can store an SMTP password in Secret Manager:”
- Cloud Build works with Cloud Composer From the official documentation: “Cloud Build synchronizes your development Managed Airflow environment with these new changes.”
- Cloud Composer is an alternative to Cloud Run From the official documentation: “If you already have a Managed Airflow environment, you can orchestrate your DPMS metadata exports and imports by running an Airflow DAG on that environment (as shown in Figure 2). This use of an Airflow DAG would be in place of the Cloud Run method mentioned earlier.”
- Cloud Composer works with Cloud Storage From the official documentation: “When you create an environment, Managed Airflow [creates a Cloud Storage bucket](/composer/docs/composer-1/cloud-storage) and associates this bucket with your environment.”
- Cloud Composer depends on Identity and Access Management From the official documentation: “Managed Airflow uses [Identity and Access Management (IAM)](/iam) for access control.”
- Cloud Composer depends on Identity-Aware Proxy From the official documentation: “Managed Airflow (Legacy Gen 1) uses [Identity-Aware Proxy](/iap) for this purpose.”
- Cloud Composer works with Managed Service for Apache Spark From the official documentation: “This tutorial shows how to use Managed Service for Apache Airflow to create an [Apache Airflow](http://airflow.apache.org) DAG (Directed Acyclic Graph) that runs an Apache Hadoop wordcount job on a Managed Service for Apache Spark cluster.”
- Cloud Composer works with Private Service Connect From the official documentation: “In Managed Airflow (Gen 2), you can create environments that [use Private Service Connect](/composer/docs/composer-2/configure-private-service-connect) instead of VPC networks.”
- Cloud Composer works with Pub/Sub topic From the official documentation: “This page guides you through creating an event-based push architecture by triggering Managed Airflow DAGs in response to Pub/Sub topic changes.”
- VPC Service Controls secures Cloud Composer From the official documentation: “If you want to protect your environment with VPC Service Controls, then make sure that both host and service projects are located in the same service perimeter For more information, see [Perimeters for environments with Shared VPC and…”
- Cloud Run works with Cloud SQL From the official documentation: “Learn how to connect to Cloud SQL from Cloud Run”
- Cloud SQL works with Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
- Cloud SQL works with Private Service Connect From the official documentation: “You can use Private Service Connect to connect to a Cloud SQL instance from multiple VPC networks.”
- Cloud Run works with Pub/Sub From the official documentation: “Route Cloud Pub/Sub events to Cloud Run”
- Cloud SQL works with Compute Engine From the official documentation: “Create a Compute Engine VM Instance to host a sample web app that connects to Cloud SQL.”
- Cloud Run works with Cloud Storage From the official documentation: “Route Cloud Storage events to Cloud Run”
- Cloud Run works with Secret Manager From the official documentation: “You can integrate Secret Manager with other Google Cloud products such as Cloud Run.”
- Managed Service for Apache Airflow works with Secret Manager From the official documentation: “You can [configure Secret Manager](/composer/docs/composer-3/configure-secret-manager) as your Airflow secrets backend.”
- Cloud Build works with Secret Manager From the official documentation: “You can substitute secrets from [Secret Manager](/secret-manager/docs) in Cloud Build.”
- Pub/Sub works with Secret Manager From the official documentation: “Receive notifications via Pub/Sub when Secret Manager resources change.”
- Cloud Run functions works with Cloud Storage From the official documentation: “Set up functions in response to events in Cloud Storage:”
- Compute Engine is an alternative to GKE From the official documentation: “To use TPU7x, you can use Google Kubernetes Engine (GKE) or Compute Engine.”
- Cloud Run depends on Pub/Sub From the official documentation: “Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.”
- Cloud Run functions works with Pub/Sub From the official documentation: “Use Cloud Run functions to publish messages on Pub/Sub.”
- Cloud SQL works with Identity and Access Management From the official documentation: “This page describes how Cloud SQL is integrated with IAM .”
- Compute Engine works with Secret Manager From the official documentation: “To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the `cloud-platform` OAuth scope.”
- Identity-Aware Proxy secures Cloud Run From the official documentation: “There are two ways to enable Identity-Aware Proxy for your Cloud Run services.”
- Cloud Build works with Cloud Run From the official documentation: “Cloud Build - create container images and deploy to Cloud Run.”
- Cloud Build works with Cloud Storage From the official documentation: “Cloud Build then stores Terraform files and the Terraform State File in a Cloud Storage bucket.”
- Cloud Run is an alternative to GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
- Cloud Run functions depends on Pub/Sub From the official documentation: “The Pub/Sub message triggers Cloud Run functions.”
- Cloud SQL depends on Compute Engine From the official documentation: “Cloud SQL uses Compute Engine virtual machines (VMs) with persistent storage disks.”
- Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow associates a Cloud Storage bucket with your environment.”
- GKE works with Secret Manager From the official documentation: “To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the `cloud-platform` OAuth scope.”
- Cloud SQL works with Secret Manager From the official documentation: “Secret Manager attempts to update the credentials in your Cloud SQL instance.”
- Cloud Storage works with Identity and Access Management From the official documentation: “To learn how to use IAM with Cloud Storage, see [Using IAM Permissions](/storage/docs/access-control/using-iam-permissions).”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Identity and Access Management secures Secret Manager From the official documentation: “Secret Manager uses Identity and Access Management (IAM) for authorization.”
- Identity-Aware Proxy secures Compute Engine From the official documentation: “Securing Compute Engine apps Enable IAP for Compute Engine apps.”
- VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
- Cloud Build works with Cloud SQL From the official documentation: “This page contains information and examples for connecting to a Cloud SQL instance from a service running in Cloud Build.”
- Cloud Run is an alternative to Cloud Run functions From the official documentation: “To create remote functions, you must [grant required roles](/bigquery/docs/remote-functions#grant_permission_on_function) to Cloud Run functions or Cloud Run.”
- Cloud SQL works with GKE From the official documentation: “[From an application running on GKE](/sql/docs/postgres/connect-kubernetes-engine)”
- Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
- Identity and Access Management secures Cloud Storage From the official documentation: “Cloud Storage uses Identity and Access Management (IAM) for authorization.”
- Managed Service for Apache Airflow depends on Cloud SQL From the official documentation: “Managed Airflow uses Cloud SQL to run Airflow database.”
- VPC Service Controls secures Managed Service for Apache Airflow From the official documentation: “Managed Airflow can be selected as a secured service inside a VPC Service Controls perimeter.”
- Cloud Build works with GKE From the official documentation: “Remove the permission granted to Cloud Build to connect to GKE.”
- Cloud Run works with Identity-Aware Proxy From the official documentation: “On the Cloud Run security page, find the checkbox for Identity-Aware Proxy (IAP).”
- Cloud Run functions depends on Cloud Build From the official documentation: “Cloud Run functions leverages Cloud Build when building and deploying your Cloud Run function.”
- Cloud Storage works with Compute Engine From the official documentation: “You can use a web browser, the Google Cloud CLI, or a programming language to transfer files between Cloud Storage and Compute Engine VMs.”
- Compute Engine works with Identity-Aware Proxy From the official documentation: “Set up a Compute Engine instance with IAP.”
- GKE depends on Compute Engine From the official documentation: “For example, GKE deploys clusters on Compute Engine VMs.”
- GKE works with Pub/Sub From the official documentation: “Route Cloud Pub/Sub events to GKE”
- Identity and Access Management works with Identity-Aware Proxy From the official documentation: “The IAP API lets you to apply IAM permissions to individual resources in an IAP-secured project.”
- Managed Service for Apache Airflow depends on GKE From the official documentation: “Managed Airflow runs your workloads on GKE.”
- Managed Service for Apache Airflow works with VPC Service Controls From the official documentation: “When used with Managed Airflow, VPC Service Controls have several [known limitations](/vpc-service-controls/docs/supported-products#composer).”
- Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
- Private Service Connect works with VPC Service Controls From the official documentation: “VPC Service Controls that use Private Service Connect”
How learners score on this topic
Last updated