Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Batch
Topic in 1 exam domain
- Batch · Service · Google Cloud
- Batch · Concept · Managed Service for Apache Spark
Also called: Batch API, Batch for Google Cloud, Batch resource, batch workload, Batches
A Google Cloud service for scheduling, queuing, and executing batch workloads on cloud resources.
Where this appears on the exam
1 objective in 1 of 5 domains
Related topics
FoundationsBuilding blocksThis topicWorks alongsideAppliedIdentity and Access ManagementServiceApplication Default CredentialsFeatureVPC Service ControlsServiceManaged Service for Apache SparkServiceCompute EngineServiceCloud LoggingServiceCloud MonitoringServiceBigQueryServicePub/SubServiceDataflowServicePersistent DiskService
BatchService
Cloud Run functionsServiceCloud RunServiceFilestoreServiceCloud SchedulerServiceGKEServiceGoogle Cloud StorageServiceJobConceptPoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Batch depends on Application Default Credentials From the official documentation: “To authenticate to Batch, set up Application Default Credentials.”
- Batch depends on Compute Engine From the official documentation: “After you submit the job, Batch automatically queues, schedules, and executes the job on Compute Engine VMs.”
- Batch works with Compute Engine From the official documentation: “After you submit the job, Batch automatically queues, schedules, and executes the job on Compute Engine VMs.”
- Batch works with Persistent Disk From the official documentation: “This document explains how to create and run a Batch job that uses one or more external storage volumes. External storage options include new or existing persistent disk, new local SSDs, existing Cloud Storage buckets, and an existing network file system (NFS) such as a…”
- Batch works with VPC Service Controls From the official documentation: “This document explains how to use VPC Service Controls with Batch.”
- VPC Service Controls secures Batch From the official documentation: “Protect Batch resources and data by using VPC Service Controls.”
- Application Default Credentials works with Batch From the official documentation: “To authenticate to Batch, set up Application Default Credentials.”
- Application Default Credentials secures Batch From the official documentation: “To authenticate to Batch, set up Application Default Credentials.”
- Batch works with Cloud Logging From the official documentation: “This document describes how to enable, generate, and view [logs](/logging/docs/overview) from [Cloud Logging](/logging/docs) for a Batch job.”
- Batch works with Cloud Monitoring From the official documentation: “This document describes how to monitor and try to optimize the resources for a Batch job by viewing metrics in Cloud Monitoring.”
- Batch works with Cloud Scheduler From the official documentation: “Specifically, you can use Terraform to schedule and run a Cloud Scheduler cron job that targets the Batch API to create and run Batch jobs.”
- Batch works with Filestore From the official documentation: “You can create a job that uses an existing network file system (NFS), such as a [Filestore file share](/filestore/docs/create-instance-gcloud), using the Google Cloud console, gcloud CLI or Batch API.”
- Batch is part of Managed Service for Apache Spark From the official documentation: “In Managed Service for Apache Spark, a `Batch` resource represents a batch workload.”
- Batch works with Pub/Sub From the official documentation: “Batch supports Pub/Sub notifications for changes to job and task states, which you can use for alerts, observability, or analysis.”
- Identity and Access Management secures Batch From the official documentation: “Batch uses Identity and Access Management (IAM) for authorization.”
- Job is part of Batch From the official documentation: “Batch consists of the following components: - **Job:** a scheduled program that runs a set of tasks to completion without any user interaction, typically for computational workloads.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Application Default Credentials works with Google Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
- Cloud Run works with Cloud Scheduler From the official documentation: “Create a Cloud Scheduler job that triggers the Cloud Run service daily.”
- Cloud Run works with Pub/Sub From the official documentation: “Route Cloud Pub/Sub events to Cloud Run”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Application Default Credentials works with Cloud Run functions From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- Cloud Run functions depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- Persistent Disk is part of Compute Engine From the official documentation: “Compute Engine offers different types of persistent disks.”
- Application Default Credentials secures Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Logging works with GKE From the official documentation: “Enable Cloud Logging on GKE Clusters”
- Cloud Run functions works with Cloud Scheduler From the official documentation: “Create a Cloud Scheduler job to trigger the Cloud Run function on a regular schedule.”
- Compute Engine is an alternative to GKE From the official documentation: “To use TPU7x, you can use Google Kubernetes Engine (GKE) or Compute Engine.”
- BigQuery works with Cloud Run functions From the official documentation: “BigQuery supports external function calls through Cloud Run functions.”
- BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”
- Cloud Monitoring depends on Application Default Credentials From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Cloud Monitoring works with GKE From the official documentation: “Enable Cloud Monitoring on GKE Clusters”
- Cloud Run depends on Pub/Sub From the official documentation: “Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.”
- Cloud Run functions works with Pub/Sub From the official documentation: “Use Cloud Run functions to publish messages on Pub/Sub.”
- BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Run depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Cloud Run is an alternative to GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
- Cloud Run functions depends on Pub/Sub From the official documentation: “The Pub/Sub message triggers Cloud Run functions.”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Filestore works with GKE From the official documentation: “See how to access Filestore instances from GKE clusters”
- Application Default Credentials works with Cloud Run From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Run functions From the official documentation: “To authenticate to Cloud Run functions, set up Application Default Credentials.”
- Application Default Credentials works with Identity and Access Management From the official documentation: “To authenticate to IAM, set up Application Default Credentials.”
- BigQuery is an alternative to Managed Service for Apache Spark From the official documentation: “Run queries against your federated data using BigQuery or Managed Service for Apache Spark.”
- Cloud Logging depends on Application Default Credentials From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
How learners score on this topic
Last updated