Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Concept
DAG
Topic in 4 exam domains
Also called: Airflow DAGs, DAGs, directed acyclic graph, Directed Acyclic Graphs
A directed acyclic graph defining an Airflow workflow that can connect to and run tasks on remote resources.
Where this appears on the exam
2 objectives in 1 of 5 domains
Related topics
FoundationsBuilding blocksThis topicAppliedApplication Default CredentialsFeatureCustomer-managed encryption keysFeatureDataflowServiceManaged Service for Apache AirflowServicePub/SubServiceCloud MonitoringServiceCloud StorageServiceCloud LoggingService
DAGConcept
Cloud Run functionsServiceBigQueryServicetaskConceptPoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Cloud Run functions works with DAG From the official documentation: “For example, you can [use Cloud Run functions to trigger Airflow DAGs](/composer/docs/composer-2/triggering-with-gcf) when a specified event occurs.”
- DAG works with Dataflow From the official documentation: “Now you will reference the files you created earlier to create a DAG that kicks off the Dataflow workflow.”
- DAG depends on Managed Service for Apache Airflow From the official documentation: “This guide shows you how to write an Apache Airflow directed acyclic graph (DAG) that runs in a Managed Service for Apache Airflow environment.”
- DAG works with Pub/Sub From the official documentation: “You can then create a DAG that pulls Pub/Sub messages and then handles these messages.”
- task is part of DAG From the official documentation: “A *DAG* is a collection of tasks that you want to schedule and run, organized in a way that reflects their relationships and dependencies.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Logging works with Customer-managed encryption keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
- BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Application Default Credentials works with Cloud Run functions From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- Cloud Run functions depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Run functions works with Cloud Storage From the official documentation: “Set up functions in response to events in Cloud Storage:”
- BigQuery works with Cloud Run functions From the official documentation: “BigQuery supports external function calls through Cloud Run functions.”
- Cloud Monitoring depends on Application Default Credentials From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Cloud Run functions works with Pub/Sub From the official documentation: “Use Cloud Run functions to publish messages on Pub/Sub.”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- Customer-managed encryption keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Logging works with Managed Service for Apache Airflow From the official documentation: “You can use [Cloud Monitoring](/monitoring/docs) and [Cloud Logging](/logging/docs) with Managed Service for Apache Airflow.”
- Cloud Run functions depends on Pub/Sub From the official documentation: “The Pub/Sub message triggers Cloud Run functions.”
- Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow associates a Cloud Storage bucket with your environment.”
- Customer-managed encryption keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
- Customer-managed encryption keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
- Cloud Monitoring works with Managed Service for Apache Airflow From the official documentation: “You can use [Cloud Monitoring](/monitoring/docs) and [Cloud Logging](/logging/docs) with Managed Service for Apache Airflow.”
- Cloud Storage works with Customer-managed encryption keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Application Default Credentials secures Cloud Run functions From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- Cloud Logging depends on Application Default Credentials From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
- Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Monitoring works with Dataflow From the official documentation: “Dataflow is fully integrated with Cloud Monitoring.”
- Customer-managed encryption keys secures Managed Service for Apache Airflow From the official documentation: “This page describes how to use [Customer Managed Encryption Keys (CMEK)](/kms/docs/cmek) to protect Managed Airflow environments.”
- BigQuery depends on Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Cloud Monitoring works with Pub/Sub From the official documentation: “Use Cloud Monitoring and Pub/Sub to set up custom monitoring and alerting policies for exported logs.”
- Application Default Credentials secures Cloud Monitoring From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Application Default Credentials works with Managed Service for Apache Airflow From the official documentation: “To authenticate to Managed Airflow, set up Application Default Credentials.”
- Cloud Logging works with Cloud Monitoring From the official documentation: “If a log entry matches the filter of the alerting policy, then Cloud Logging extracts labels and forwards the data to Cloud Monitoring.”
- Cloud Logging works with Cloud Run functions From the official documentation: “Cloud Run functions is configured to use Cloud Logging automatically.”
- Cloud Logging works with Dataflow From the official documentation: “Information stored in [Cloud Logging](/logging/docs/region-support) is primarily generated by the code in your Dataflow program.”
- Cloud Monitoring works with Cloud Run functions From the official documentation: “This hands-on lab shows you how to view your Cloud Run functions with their execution times, execution counts, and memory usage in the Google Cloud console. These metrics are also available in Cloud Monitoring.”
- Cloud Run functions depends on Cloud Storage From the official documentation: “Cloud Run functions uses the config in Cloud Storage to determine what actions to take for each stateful MIG.”
How learners score on this topic
Last updated