Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Managed Airflow
Topic in 4 exam domains
Google Cloud's managed Apache Airflow service (Cloud Composer) for workflow orchestration and dependency management.
Where this appears on the exam
4 objectives in 3 of 5 domains
Related topics
FoundationsThis topicBuilds on theseBuilds on theseAppliedApplication Default CredentialsFeature
Managed AirflowService
FirestoreServiceBigQueryServiceDataflowServiceCloud Run functionsServicePub/SubServiceCloud SchedulerServiceCloud RunServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Cloud Scheduler is an alternative to Managed Airflow From the official documentation: “Use [Cloud Scheduler](/scheduler) or [Managed Airflow](/composer) to periodically schedule an export job to write to Cloud Storage.”
- Managed Airflow depends on Application Default Credentials From the official documentation: “To authenticate to Managed Airflow, set up Application Default Credentials.”
- Firestore depends on Application Default Credentials From the official documentation: “To authenticate to Firestore, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Firestore From the official documentation: “To authenticate to Firestore, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Cloud Run works with Cloud Scheduler From the official documentation: “Create a Cloud Scheduler job that triggers the Cloud Run service daily.”
- Cloud Run works with Pub/Sub From the official documentation: “Route Cloud Pub/Sub events to Cloud Run”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Application Default Credentials works with Cloud Run functions From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- Cloud Run functions depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Run functions works with Cloud Scheduler From the official documentation: “Create a Cloud Scheduler job to trigger the Cloud Run function on a regular schedule.”
- BigQuery works with Cloud Run functions From the official documentation: “BigQuery supports external function calls through Cloud Run functions.”
- Cloud Run depends on Pub/Sub From the official documentation: “Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.”
- Cloud Run functions works with Pub/Sub From the official documentation: “Use Cloud Run functions to publish messages on Pub/Sub.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Run depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Cloud Run functions depends on Pub/Sub From the official documentation: “The Pub/Sub message triggers Cloud Run functions.”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Cloud Run From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Run functions From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- Cloud Run is an alternative to Cloud Run functions From the official documentation: “To create remote functions, you must [grant required roles](/bigquery/docs/remote-functions#grant_permission_on_function) to Cloud Run functions or Cloud Run.”
- Cloud Scheduler works with Pub/Sub From the official documentation: “Create a Pub/Sub topic to set up as your Cloud Scheduler job target.”
- Cloud Run works with Firestore From the official documentation: “The Cloud Run app stores the result in Firestore.”
- Cloud Run functions works with Firestore From the official documentation: “The Cloud Run function stores the result in Firestore.”
- Application Default Credentials secures Firestore From the official documentation: “To authenticate to Firestore, set up Application Default Credentials.”
- BigQuery works with Cloud Run From the official documentation: “You can then use Cloud Run to transcode data present in the bucket and move it to BigQuery.”
- BigQuery works with Firestore From the official documentation: “You can load data from a Firestore with MongoDB compatibility export into BigQuery, but only if you specified a `collection-ids` filter.”
- Cloud Run functions depends on Cloud Run From the official documentation: “Cloud Run functions (2nd gen) is based on Cloud Run and has a limit of 100 database connections per instance.”
- Application Default Credentials configures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Application Default Credentials configures Cloud Run From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- BigQuery works with Cloud Scheduler From the official documentation: “To learn more about using Cloud Scheduler with BigQuery, see [Scheduling workflows with Cloud Scheduler](/scheduler/docs/tut-workflows).”
- BigQuery is an alternative to Dataflow From the official documentation: “To automate the EDA process, use BigQuery or a managed data processing service like Dataflow.”
- Cloud Run works with Dataflow From the official documentation: “Route Dataflow events to Cloud Run”
- Cloud Run functions is part of Cloud Run From the official documentation: “| Cloud Run functions | Standalone functions on Cloud Run. |”
- Cloud Run functions depends on Firestore From the official documentation: “Grant the Compute Engine service account (`your-project-number-developer.gserviceaccount.com`) the **Cloud Datastore User** role, so the newly deployed Cloud Run functions can access Firestore.”
- Cloud Scheduler works with Dataflow From the official documentation: “A Cloud Scheduler service account to run the batch Dataflow job that creates the Dataflow pipeline.”
- Cloud Scheduler works with Firestore From the official documentation: “You can use Cloud Scheduler to export data from Firestore on a regular cadence or date.”
- Cloud Scheduler depends on Pub/Sub From the official documentation: “Cloud Pub/Sub (required for Cloud Scheduler to invoke functions)”
- Dataflow works with Firestore From the official documentation: “Running a Firestore workload in Dataflow incurs costs for Firestore usage and Dataflow usage.”
- Pub/Sub depends on Cloud Run functions From the official documentation: “After you have set up Cloud Run functions to process model output and generate events, you can set up event notification with Pub/Sub.”
How learners score on this topic
Last updated