Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Dataproc
Topic in 1 exam domain
Also called: Cloud Dataproc, Dataproc on Compute Engine, Google Cloud Serverless for Apache Spark
A data processing service supported in Google Distributed Cloud.
Where this appears on the exam
1 objective in 1 of 5 domains
Related topics
FoundationsThis topicWorks alongsideAppliedCMEKFeaturecustomer-managed encryption keysFeatureApplication Default CredentialsFeatureCloud SQLService
DataprocService
Cloud LoggingServiceCloud StorageServicePub/SubServiceGoogle Cloud StorageServiceDataflowServiceWorkflowsServiceCloud Data FusionServiceBigQueryServiceManaged Service for Apache AirflowServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- CMEK secures Dataproc From the official documentation: “Require CMEK on Dataproc Clusters”
- Dataproc works with Google Cloud Storage From the official documentation: “# Dataproc job output is saved to the Cloud Storage bucket”
- Application Default Credentials secures Dataproc From the official documentation: “To authenticate to Managed Service for Apache Spark, set up Application Default Credentials.”
- BigQuery works with Dataproc From the official documentation: “the built-in BigQuery Connector for Spark that bridges BigQuery and Cloud Dataproc.”
- Cloud Data Fusion works with Dataproc From the official documentation: “Cloud Data Fusion translates the pipeline that you build in the Studio into an Apache Spark program that executes transformations in parallel on an ephemeral Managed Service for Apache Spark cluster.”
- Cloud Storage works with Dataproc From the official documentation: “Dataproc job output gets saved to the Google Cloud Storage bucket # allocated to the job.”
- customer-managed encryption keys works with Dataproc From the official documentation: “A Dataproc cluster with customer-managed encryption keys (CMEK) gives you more control over data encryption and key management.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials works with Google Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Logging works with customer-managed encryption keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
- Cloud SQL works with Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- CMEK secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
- BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- CMEK secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
- customer-managed encryption keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
- Application Default Credentials works with Workflows From the official documentation: “To authenticate to Workflows, set up Application Default Credentials.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Logging works with Cloud SQL From the official documentation: “Cloud SQL uses Cloud Logging.”
- Cloud Logging works with Managed Service for Apache Airflow From the official documentation: “You can use [Cloud Monitoring](/monitoring/docs) and [Cloud Logging](/logging/docs) with Managed Service for Apache Airflow.”
- Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow associates a Cloud Storage bucket with your environment.”
- CMEK secures BigQuery From the official documentation: “Enable CMEK for BigQuery Tables”
- customer-managed encryption keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
- customer-managed encryption keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
- customer-managed encryption keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
- Cloud Storage works with customer-managed encryption keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- CMEK secures Cloud Storage From the official documentation: “Enable CMEK for Cloud Storage Buckets”
- Cloud Logging depends on Application Default Credentials From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
- Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
- customer-managed encryption keys secures Managed Service for Apache Airflow From the official documentation: “This page describes how to use [Customer Managed Encryption Keys (CMEK)](/kms/docs/cmek) to protect Managed Airflow environments.”
- Managed Service for Apache Airflow depends on Cloud SQL From the official documentation: “Managed Airflow uses Cloud SQL to run Airflow database.”
- BigQuery works with Cloud Data Fusion From the official documentation: “Cloud Data Fusion pulls data from the storage bucket and writes it to BigQuery.”
- BigQuery depends on Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Cloud Data Fusion works with Cloud Logging From the official documentation: “Starting with Cloud Data Fusion version 6.11, pipeline logs and service logs are available in Cloud Logging.”
- Cloud SQL works with Google Cloud Storage From the official documentation: “Imports data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage.”
- Google Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
How learners score on this topic
Last updated