Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Concept
batch processing
Topic in 3 exam domains
- batch processing · Concept
- batch processing · Feature
- batch processing · Task
Also called: batch processing capability
A way to process many rows asynchronously using the summarization function.
Where this appears on the exam
3 subdomains in 3 of 5 domains
Related topics
This topicBuilds on theseApplied
batch processingConcept
Application Default CredentialsFeatureVPC Service ControlsServiceCloud Storage bucketConceptCompute EngineServicePub/SubServiceCloud MonitoringServiceBatchServiceDataflowServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Batch works with batch processing From the official documentation: “Run these high-volume workloads on services like [Batch](/batch/docs/get-started). The service automatically provisions and manages the necessary infrastructure, which helps to ensure optimal resource utilization.”
- batch processing works with Dataflow From the official documentation: “Managed services like Batch and [Dataflow](/dataflow/docs/overview) automatically handle resource provisioning, scheduling, and monitoring.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Application Default Credentials works with Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- Cloud Monitoring depends on Application Default Credentials From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Application Default Credentials secures Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Monitoring From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Batch depends on Application Default Credentials From the official documentation: “To authenticate to Batch, set up Application Default Credentials.”
- Batch depends on Compute Engine From the official documentation: “After you submit the job, Batch automatically queues, schedules, and executes the job on Compute Engine VMs.”
- Batch works with Compute Engine From the official documentation: “After you submit the job, Batch automatically queues, schedules, and executes the job on Compute Engine VMs.”
- Cloud Monitoring works with Dataflow From the official documentation: “Dataflow is fully integrated with Cloud Monitoring.”
- Cloud Monitoring works with Pub/Sub From the official documentation: “Use Cloud Monitoring and Pub/Sub to set up custom monitoring and alerting policies for exported logs.”
- Dataflow depends on Compute Engine From the official documentation: “When Dataflow runs a pipeline, the processing is distributed across multiple Compute Engine virtual machines (VMs), also known as workers.”
- VPC Service Controls secures Compute Engine From the official documentation: “To further protect your Compute Engine resources, you can protect them using [VPC Service Controls](/vpc-service-controls/docs/overview).”
- Application Default Credentials works with Batch From the official documentation: “To authenticate to Batch, set up Application Default Credentials.”
- Application Default Credentials secures Batch From the official documentation: “To authenticate to Batch, set up Application Default Credentials.”
- Application Default Credentials configures Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- Batch works with Cloud Monitoring From the official documentation: “This document describes how to monitor and try to optimize the resources for a Batch job by viewing metrics in Cloud Monitoring.”
- Batch works with Cloud Storage bucket From the official documentation: “Create and submit a sample Batch job that runs a script that accesses an existing Cloud Storage bucket.”
- Batch is an alternative to Compute Engine From the official documentation: “3. Do you want to manage VMs or bare metal instances yourself? - **Yes**: [use Compute Engine](#compute-engine). - **No**: [use Batch](#batch).”
- Batch works with Pub/Sub From the official documentation: “Batch supports Pub/Sub notifications for changes to job and task states, which you can use for alerts, observability, or analysis.”
- Batch works with VPC Service Controls From the official documentation: “This document explains how to use VPC Service Controls with Batch.”
- Cloud Monitoring works with Compute Engine From the official documentation: “Many Google services like Compute Engine send telemetry to Cloud Monitoring.”
- Cloud Monitoring works with VPC Service Controls From the official documentation: “Cloud Monitoring performs the VPC perimeter check when a project is *added* to a metrics scope:”
- Cloud Storage bucket works with Dataflow From the official documentation: “Begin by creating a Cloud Storage bucket using Google Cloud CLI. This bucket is used as a temporary storage location by the Dataflow pipeline.”
- Dataflow depends on Cloud Storage bucket From the official documentation: “Create a Cloud Storage bucket by using the gcloud CLI. This bucket is used as a temporary storage location by the Dataflow pipeline.”
- Dataflow works with VPC Service Controls From the official documentation: “To customize DNS resolution when using Dataflow with VPC Service Controls, use Cloud DNS [private zones](/dns/docs/overview#dns-private-zones) instead of using custom BIND servers.”
- VPC Service Controls secures Batch From the official documentation: “Protect Batch resources and data by using VPC Service Controls.”
- VPC Service Controls secures Cloud Monitoring From the official documentation: “VPC Service Controls provides additional security for Cloud Monitoring to help mitigate the risk of data exfiltration.”
- VPC Service Controls secures Dataflow From the official documentation: “Dataflow VPC Service Controls help protect against accidental or targeted action by external entities or by insider entities, which helps to minimize unwarranted data exfiltration risks.”
- VPC Service Controls secures Pub/Sub From the official documentation: “If you use [VPC Service Controls](/vpc-service-controls/docs/overview) and your Pub/Sub topic is part of a project inside a service perimeter, you must grant access to projects in order to create notifications.”
How learners score on this topic
Last updated