Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Dataform
Topic in 10 exam domains
A Google Cloud service used to build, version control, and run SQL-based data pipelines that transform raw data into clean datasets.
Where this appears on the exam
7 objectives in 3 of 5 domains
~22% of exam
~25% of exam
~18% of exam
Related topics
FoundationsBuilding blocksThis topicWorks alongsideAppliedIdentity and Access ManagementServiceVPC Service ControlsServiceApplication Default CredentialsFeatureBigQueryServiceSecret ManagerServiceCloud MonitoringServiceCloud StorageServiceKnowledge CatalogServiceCloud RunServiceCompute EngineService
DataformService
Cloud SQLServiceOrganization Policy ServiceServiceCloud LoggingServiceGKEServiceManaged Service for Apache AirflowServiceBigQuery StudioFeaturerelease configurationFeatureworkflow configurationFeaturePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- BigQuery works with Dataform From the official documentation: “You can define custom SQL commands for Dataform to run in BigQuery.”
- Dataform works with Secret Manager From the official documentation: “To connect a remote repository through HTTPS, you need to create a Secret Manager secret with a personal access token, and share the secret with your default Dataform service agent.”
- Identity and Access Management secures Dataform From the official documentation: “Dataform uses [Identity and Access Management (IAM)](/iam/docs) for access control.”
- VPC Service Controls secures Dataform From the official documentation: “If you use VPC Service Controls, you must configure the perimeter to protect Dataform and BigQuery.”
- BigQuery Studio depends on Dataform From the official documentation: “To use CMEK to protect your BigQuery Studio code assets, you must set a default Dataform CMEK key for the Google Cloud project that contains your code assets.”
- Cloud Logging works with Dataform From the official documentation: “To enhance security, Dataform checks if the `iam.serviceAccounts.actAs` permission is missing on the service accounts used by Dataform resources. The results of these checks, including any potential permission issues, are recorded in Cloud Logging.”
- Cloud Monitoring works with Dataform From the official documentation: “2. **[Cloud Monitoring integration](/monitoring/docs)**: Track Dataform pipeline metrics, such as execution durations, active compilations, and workflow job failure rates, through custom dashboard panels.”
- Cloud Storage works with Dataform From the official documentation: “Dataform writes the output of scheduled notebook runs to the [Cloud Storage bucket](/storage/docs/buckets) selected during schedule creation.”
- Dataform depends on BigQuery From the official documentation: “Dataform uses the following Google Cloud services that have their own quota limits: - [BigQuery](/bigquery/quotas)”
- Dataform depends on Cloud Monitoring From the official documentation: “Dataform uses the following Google Cloud services that have their own quota limits: - [BigQuery](/bigquery/quotas) - [Identity and Access Management](/iam/quotas) - [Cloud Monitoring](/monitoring/quotas)”
- Dataform depends on Identity and Access Management From the official documentation: “Dataform uses the following Google Cloud services that have their own quota limits: - [BigQuery](/bigquery/quotas) - [Identity and Access Management](/iam/quotas)”
- Dataform works with Identity and Access Management From the official documentation: “Dataform uses Identity and Access Management (IAM) for authorization.”
- Dataform works with Knowledge Catalog From the official documentation: “Dataform repositories are available in Knowledge Catalog by default, without any additional configuration.”
- Dataform works with Organization Policy Service From the official documentation: “This document shows you how to use the [Organization Policy Service](/resource-manager/docs/organization-policy/overview) to restrict the pool of third-party remote Git repositories that Dataform repositories can connect to.”
- Dataform depends on Secret Manager From the official documentation: “Dataform uses the following Google Cloud services that have their own quota limits: - [BigQuery](/bigquery/quotas) - [Identity and Access Management](/iam/quotas) - [Cloud Monitoring](/monitoring/quotas) - [Secret Manager](/secret-manager/quotas)”
- release configuration is part of Dataform From the official documentation: “A production release in Dataform ensures that your environment is consistently updated with the results of your data transformations.”
- workflow configuration is part of Dataform From the official documentation: “This section shows you how to create a workflow configuration in Dataform to schedule and configure workflow runs.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials works with Secret Manager From the official documentation: “To authenticate to Secret Manager, set up Application Default Credentials.”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- Cloud Run works with Cloud SQL From the official documentation: “Learn how to connect to Cloud SQL from Cloud Run”
- Application Default Credentials works with Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Knowledge Catalog depends on Application Default Credentials From the official documentation: “To authenticate to Knowledge Catalog, set up Application Default Credentials.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- BigQuery works with Knowledge Catalog From the official documentation: “Some Knowledge Catalog functionalities trigger job execution using BigQuery.”
- Cloud SQL works with Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
- BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
- Cloud SQL works with Compute Engine From the official documentation: “Create a Compute Engine VM Instance to host a sample web app that connects to Cloud SQL.”
- Cloud Run works with Cloud Storage From the official documentation: “Route Cloud Storage events to Cloud Run”
- Cloud Run works with Secret Manager From the official documentation: “You can integrate Secret Manager with other Google Cloud products such as Cloud Run.”
- Managed Service for Apache Airflow works with Secret Manager From the official documentation: “You can [configure Secret Manager](/composer/docs/composer-3/configure-secret-manager) as your Airflow secrets backend.”
- Application Default Credentials secures Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- Cloud Logging works with GKE From the official documentation: “Enable Cloud Logging on GKE Clusters”
- Compute Engine is an alternative to GKE From the official documentation: “To use TPU7x, you can use Google Kubernetes Engine (GKE) or Compute Engine.”
- Secret Manager depends on Application Default Credentials From the official documentation: “To authenticate to Secret Manager, set up Application Default Credentials.”
- Cloud Monitoring depends on Application Default Credentials From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Cloud Monitoring works with Cloud SQL From the official documentation: “You can also get more details by viewing the Cloud Monitoring logs for your Cloud SQL destination instance.”
- Cloud Monitoring works with GKE From the official documentation: “Enable Cloud Monitoring on GKE Clusters”
- Cloud SQL works with Identity and Access Management From the official documentation: “This page describes how Cloud SQL is integrated with IAM .”
- Cloud SQL works with Knowledge Catalog From the official documentation: “You can use Knowledge Catalog to organize and enrich the metadata of Cloud SQL resources.”
- Compute Engine works with Secret Manager From the official documentation: “To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the `cloud-platform` OAuth scope.”
- Cloud Logging works with Cloud SQL From the official documentation: “Cloud SQL uses Cloud Logging.”
- Cloud Logging works with Managed Service for Apache Airflow From the official documentation: “You can use [Cloud Monitoring](/monitoring/docs) and [Cloud Logging](/logging/docs) with Managed Service for Apache Airflow.”
- Cloud Run depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Cloud Run is an alternative to GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
- Cloud SQL depends on Compute Engine From the official documentation: “Cloud SQL uses Compute Engine virtual machines (VMs) with persistent storage disks.”
- Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow associates a Cloud Storage bucket with your environment.”
- GKE works with Secret Manager From the official documentation: “To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the `cloud-platform` OAuth scope.”
- Application Default Credentials secures Knowledge Catalog From the official documentation: “To authenticate to Knowledge Catalog, set up Application Default Credentials.”
- BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
- Cloud Monitoring works with Managed Service for Apache Airflow From the official documentation: “You can use [Cloud Monitoring](/monitoring/docs) and [Cloud Logging](/logging/docs) with Managed Service for Apache Airflow.”
- Cloud SQL works with Secret Manager From the official documentation: “Secret Manager attempts to update the credentials in your Cloud SQL instance.”
How learners score on this topic
Last updated