Prepare and test your skills
Prepare and test your skills
Evaluate Managed Data Processing Services means choosing the Google Cloud service that runs batch and streaming work without managing servers or clusters. Managed Service for Apache Spark preserves existing Hadoop and Spark work, Dataflow (opens in a new tab) runs unified batch and streaming pipelines written with Apache Beam, and Cloud Data Fusion provides a no-code graphical option that compiles pipelines into Spark jobs. Data flows from sources such as on-premises systems, event streams, and Cloud Storage through transforms on managed workers into sinks such as Cloud Storage and BigQuery.
Managed Service for Apache Spark is the managed option for running Apache Hadoop and Apache Spark jobs without operating a Hadoop cluster. The service runs jobs on ephemeral clusters that exist only for the workload, which supports cost-effective execution for Sqoop imports, Spark SQL, and Hive extract, transform, and load (ETL) jobs. Data flows from on-premises or Hadoop sources through Spark transforms into Google Cloud sinks such as Cloud Storage and BigQuery. Choose this service when the architecture must keep Hadoop and Spark application programming interfaces (APIs), libraries, and connectors rather than rewriting the pipeline.
Migration with Managed Service for Apache Spark centers on decoupling storage from compute and using connectors. Sqoop output files can move to Cloud Storage with Hadoop DistCp or be written directly with the Cloud Storage connector, which is an open source Java library that lets Hadoop or Spark jobs read and write Cloud Storage directly. Apache Hive (opens in a new tab) running on Google Cloud can use Cloud Storage as its warehouse by setting hive.metastore.warehouse.dir to a Cloud Storage bucket instead of Hadoop Distributed File System (HDFS). Sqoop jobs can also submit directly to import data into BigQuery without operating a cluster.
Existing Hive ETL work has four practical migration paths for data pipelines (opens in a new tab) that trade rewrite effort for managed execution:
The batch SQL translator or interactive SQL translator can automatically translate Apache Spark SQL or HiveQL to GoogleSQL for BigQuery. Teams that want to keep Spark can use the BigQuery connector for Apache Spark to read from and write to BigQuery. Multi-step work can be managed with Managed Service for Apache Airflow workflow orchestration, including a tool for converting Oozie workflows, and Managed Service for Apache Spark Workflow Templates.
Dataflow is a fully managed service for executing data processing pipelines written with the open source Apache Beam framework. Apache Beam provides a unified programming model, so the same pipeline code can process bounded batch data and unbounded streaming data. A pipeline is represented as a directed acyclic graph (DAG) where immutable collections called PCollections flow from a source through PTransforms into a sink, and each PTransform contains processing logic in a DoFn that acts on one data element at a time. The Dataflow Runner for Apache Beam distributes that DAG to parallel workers, brings parallel stages back together with side inputs or joins, and autoscales resources as data volume moves through stages.
A process flow of a Dataflow Apache Beam pipeline showing a source feeding PCollections through PTransforms (each containing a DoFn) on parallel autoscaled workers, converging via side inputs or joins, and ending at a sink such as BigQuery or Cloud Storage.
Dataflow supports standard movement, machine learning, and highly parallel high-performance computing (HPC) work in a single pipeline. For standard movement, Google-provided Dataflow templates can read text from Cloud Storage and write to BigQuery or read JSON messages from Pub/Sub and write to BigQuery without writing code. Beam SQL further simplifies processing with SQL-like statements over Beam data. For HPC work, Dataflow can also run compiled external binaries, such as C++ code, inside a DoFn as a subprocess call, with binaries staged in Cloud Storage and pulled to workers during setup.
An end-to-end highly parallel workload (opens in a new tab) keeps ingest, preparation, distributed compute, and post-analysis in one Dataflow pipeline rather than separate orchestrated systems. Keeping all stages in one system preserves data locality between stages, exposes in-system telemetry such as total bytes per stage, and lets autoscaling respond to data as it flows. A typical highly parallel workflow (opens in a new tab) follows an ordered sequence:
Exam tip: Choose Managed Service for Apache Spark when the workload must stay on Hadoop, Spark, HiveQL, or Sqoop, and choose Dataflow when the workload can use Apache Beam for unified batch and streaming in one autoscaled pipeline.
Integrating storage with Dataflow pipelines means connecting Dataflow workers to Google Cloud storage services that can absorb high-concurrency reads and large fan-out writes. Data flows from the source service through Beam transforms on Dataflow workers to the destination service, and large driving elements are passed by Cloud Storage uniform resource identifier (URI) reference rather than embedded in the PCollection. The core integrations are:
DoFn processingData size controls how driving data and joining data reach workers. Driving data is primary source data, such as trades or image files, while joining data is secondary data acquired during processing, such as market data joined to trades. Low-megabyte driving elements travel directly in a PCollection, while high-megabyte or gigabyte elements stay in Cloud Storage with only the URI in the PCollection. Joining data of a few hundred megabytes or less travels to every worker that needs it as a side input, while gigabyte- or terabyte-scale joining data is merged from Bigtable for key-value lookups or from Cloud Storage.
BigQuery reads through BigQueryIO follow two different paths with different quota and performance tradeoffs. Table export is the default path, while direct read uses the BigQuery Storage API over gRPC for high-throughput row reads.
| Read method | How data flows | Quota and timing effect |
|---|---|---|
| EXPORT | Pipeline sends an export request to BigQuery, BigQuery writes table data to temporary Cloud Storage, then the pipeline reads from that temporary location | Consumes BigQuery export quotas and the export must complete before processing, which adds job run time |
| DIRECT_READ | Pipeline reads table rows directly from BigQuery through the BigQuery Storage API | Avoids the export step and export quotas and can decrease job run time |
Writes carry matching quota choices that affect pipeline type. Batch pipelines that write with BigQuery load jobs consume table- and project-level load quotas, while streaming pipelines use streaming inserts or the Storage Write API. Avoid appending with load jobs thousands of times per day into one table and use a streaming pipeline with streaming inserts or the Storage Write API for near real-time appends. Reads from Cloud Storage with the Beam TextIO or AvroIO connectors add parallelism because separate files are read by separate workers, and uncompressed files with line-oriented records can be split into sub-ranges for even higher parallelism.
Implement Resource Optimization and Provisioning Strategies means matching machine type, purchase model, and scaling behavior to what each workload actually needs. Compute Engine (opens in a new tab) provides machine recommendations and custom shapes to right-size virtual machines (VMs), multiple provisioning models to balance cost against capacity assurance, and autoscaling in stateless managed instance groups to follow demand without over-provisioning. The architect chooses low-cost, interruptible capacity for flexible work and reserved or on-demand capacity for critical work.
Optimize Machine Type Selection means choosing the vCPU and memory shape that fits actual utilization so the workload is neither over-provisioned nor under-provisioned. Compute Engine analyzes existing VM utilization and produces machine type recommendations that identify when a workload should move to a different size. For workloads with predictable but uneven needs, custom machine types allow CPU cores to be provisioned independently of memory instead of accepting a fixed predefined ratio. This independence also limits third-party licensing cost because licenses tied to CPU cores are paid only for the cores that are actually needed.
Spot VMs are excess Compute Engine capacity sold under the spot consumption model (opens in a new tab) at discounts of up to 91% on vCPUs, memory, graphics processing units (GPUs), and attached Local solid-state drive (SSD) disks compared to on-demand prices. A Spot VM runs with the same machine types, options, and performance as a standard VM and has no minimum or maximum runtime, but Compute Engine can preempt the VM at any time to reclaim capacity for other tasks. On preemption, Compute Engine either stops the VM by default or deletes it, depending on the termination action configured for that VM. Choose Spot VMs when the workload is fault-tolerant and can slow down or retry, such as batch processing, data analytics, fault-tolerant distributed training, high performance computing simulation, and media encoding.
Spot VMs carry tradeoffs that make them unsuitable for critical or stateful work. Spot VM availability varies with overall Compute Engine usage and consumes preemptible quota, so a Managed Instance Group (MIG) using Spot VMs might not be able to scale out to its target size when zone capacity is limited. Spot VMs do not support the A4X machine series or bare metal instances, cannot use live migration to become standard VMs while running, cannot be set to automatically restart on a host event, are excluded from the Compute Engine service level agreement (SLA), and do not receive Free Tier credits. Because preemption probability varies by day and zone, architects check Spot availability before creation and design jobs to fail gracefully when host VMs are preempted.
Flex-start is a provisioning model for densely allocated capacity needed for up to seven days, requested through clustered MIG resize requests. When Compute Engine makes the requested capacity available, it provisions the VMs together for low-latency, tightly packed placement, which suits small model pre-training, fine-tuning, batch inference, and short high performance computing (HPC) (opens in a new tab) runs. Flex-start receives a discount of up to 53% on vCPUs, memory, GPUs, and Local SSDs for the A4, A3, A2, and G4 machine series, while other supported series are billed at standard on-demand rates.
Standard VMs using the on-demand provisioning model provide immediate availability without scheduling or approval and run until they are stopped or deleted. On-demand VMs are billed at standard rates with no long-term commitment and support any GPU machine series in the general GPU path, which suits inference and model serving plus prototyping and experimentation. When capacity (opens in a new tab) must be guaranteed, standard reservations hold a pool of zonal VMs with very high assurance that resources will be available, are billed for the reservation period at standard rates, and can attach resource-based committed use discounts for savings. Choose on-demand when flexibility matters most and choose reservations when a critical production workload cannot tolerate a stock-out.
For future GPU needs, future reservations provide exclusive, densely allocated capacity through a reservation-bound model that increases the chance of obtaining high-demand GPUs. Future reservations in calendar mode support A4, A3 Ultra, A3 Mega, or A3 High with 8 GPUs for up to 80 VMs and a maximum of 90 days, while capacity-block reservations support A4X Max, A4X, A4, A3 Ultra, A3 Mega, or A3 High for long-running, large-scale distributed training. At the end of the reservation period, Compute Engine deletes the reservation and then stops or deletes any VMs using it based on their configured termination action. Dynamic Workload Scheduler (DWS) offers another path by queuing GPU requests in a dedicated pool to avoid on-demand stock-outs, Spot preemption risk, and the cost of holding a reservation.
Apply Autoscaling to Match Fluctuating Demand means letting a stateless MIG add and remove VMs as traffic rises and falls. The autoscaler monitors demand and then adds VMs to the MIG when traffic rises so requests are handled gracefully, and removes VMs when demand falls so idle capacity is not paid for. This dependency means the application tier depends on the MIG for elasticity, and the MIG depends on available zone capacity to fulfill scale-out. Stateful MIGs cannot be autoscaled because their persistent disks must be preserved when VMs are repaired or recreated, so only stateless tiers in the application and web layers should rely on autoscaling for cost control.
Exam tip: Use Spot VMs for batch jobs that tolerate preemption with no SLA, and use standard reservations for critical production workloads that require very high capacity assurance.
Choosing a Provisioning Model means trading cost against capacity assurance and interruption risk. The models differ in when to use them, what they cost, and how capacity behaves:
| Model | Choose when | Cost | Capacity behavior |
|---|---|---|---|
| Spot VMs | Workload is fault-tolerant and stateless, such as batch processing | Deeply discounted, up to 91% off vCPUs, memory, GPUs, and Local SSDs, pay as you go | Best-effort availability, Compute Engine can preempt at any time |
| Flex-start | Workload needs densely allocated GPUs for up to 7 days via MIG resize | Discounted up to 53% for A4, A3, A2, and G4 series, pay as you go | Best-effort but densely allocated once provisioned |
| On-demand VMs | Workload has no fixed duration and needs immediate start, such as inference serving | Standard on-demand rates, pay as you go | Best-effort availability based on current zone capacity |
| Standard reservations | Workload is critical and needs assured capacity | Standard rates for reservation period, can attach committed use discounts | Very high assurance with exclusive access to reserved VMs |
The table shows the core tradeoff: lower price requires accepting preemption or best-effort scheduling, while guaranteed capacity requires paying for the reservation period even if VMs sit idle.
Analyze Workload Requirements for Compute Selection means matching what an application needs for control, scaling, and management effort to the right Google Cloud compute environment. Compute in Google Cloud (opens in a new tab) refers to virtualized processing power and memory, ranging from raw virtual machine (VM) instances where you customize the operating system (OS), networking, and software stack, to fully managed platforms where Google handles the underlying infrastructure. The core tradeoff is configuration flexibility versus management effort: VMs and containers give more flexibility but require you to manage resources, while serverless platforms run on a preconfigured platform with minimal management.
Compute Engine is the Infrastructure as a Service (IaaS) service for creating and running configurable VM and bare metal instances on Google infrastructure. With Compute Engine you manage the guest OS, middleware, and applications while Google manages virtualization, servers, storage, and networking, and you choose machine series, storage services, accelerators, and guest OSs. You can add graphics processing units (GPUs) or tensor processing units (TPUs) to a customized instance, scale manually or automatically by using managed instance groups (MIGs), and distribute traffic with load balancing. Choose Compute Engine when the workload needs full OS and kernel control, lift-and-shift migration without re-architecting, custom machine shapes, or license-restricted software that must run on specific VMs.
Google Kubernetes Engine (GKE) is the managed Kubernetes environment for deploying, scaling, and orchestrating containerized applications and general-purpose microservices. In GKE, Google manages the Kubernetes control plane for security and availability while you deploy and manage container workloads, and GKE builds on top of Compute Engine so many Compute Engine reliability practices still apply. GKE supports all available NVIDIA GPUs and Google TPUs, adjusts container capacity with Horizontal and Vertical Pod Autoscalers and node capacity with the cluster autoscaler, and orchestrates batch jobs through Kueue. Choose GKE in Autopilot mode when you want Google to manage nodes and capacity automatically, and in Standard mode when you want to manage node pools yourself while keeping Kubernetes orchestration.
Event-Driven Scaling and Batch Processing With Managed Platforms means using serverless and queued services that scale automatically without managing VMs or clusters. Cloud Run is the fully managed serverless platform for developing and deploying highly scalable containerized applications without managing servers or clusters. Containers on Cloud Run respond to web requests or events and scale automatically based on requests or CPU consumption, because the platform abstracts away infrastructure management. Batch is the managed service for scheduling, queueing, and running large-scale asynchronous batch jobs, which provisions Compute Engine resources to run jobs and manages their lifecycle so you do not manage VMs directly. Choose Cloud Run when the priority is focusing on data and applications with event-driven scaling, and choose Batch when the work is large-scale queued jobs that run to completion rather than serving live requests.
Comparing Compute Options By Workload Needs (opens in a new tab) means looking at who controls the OS, how work is orchestrated, and how scaling is triggered. The compute options differ by those dimensions, so workload requirements point directly to a fit:
| Compute option | OS and kernel control | Orchestration | Scaling |
|---|---|---|---|
| Compute Engine | Self-managed | Manual | Manual or automatic by using MIGs |
| Google Kubernetes Engine (GKE) | Google managed in Autopilot or semi-managed in Standard | Kubernetes | Automatic pod and node scaling |
| Cloud Run | Google managed | Serverless | Automatic based on requests or CPU consumption |
| Batch | Google managed | Job queue | Automatic based on queued jobs |
This comparison shows the control-to-abstraction spectrum in practice: control moves from self-managed VMs to Google-managed containers to serverless and queued jobs. When granular OS, storage, and accelerator choices matter most, Compute Engine fits; when distributed containers need Kubernetes orchestration and autoscaling, GKE fits; when minimal operations and automatic scale-to-demand matter most, Cloud Run or Batch fits.
Exam tip: Choose Compute Engine for self-managed OS control, GKE for Kubernetes pod and node orchestration, and Cloud Run for serverless scaling on requests — Batch scales instead on queued jobs.
Use Managed Service for Apache Spark when the workload must stay on Hadoop, Spark, HiveQL, or Sqoop without rewriting the pipeline, because it runs Hadoop and Spark jobs on ephemeral clusters that preserve existing APIs, libraries, and connectors. Use Dataflow when the workload can use Apache Beam for unified batch and streaming in one autoscaled pipeline, where immutable PCollections flow from a source through PTransforms into a sink.
EXPORT sends an export request to BigQuery, which writes table data to temporary Cloud Storage for the pipeline to read, while DIRECT_READ reads table rows directly from BigQuery through the BigQuery Storage API. EXPORT consumes BigQuery export quotas and the export must complete before processing, which adds job run time, while DIRECT_READ avoids the export step and export quotas and can decrease job run time.
Use Spot VMs for fault-tolerant and stateless work such as batch processing, data analytics, and media encoding that can slow down or retry, and use standard reservations for critical production workloads that cannot tolerate a stock-out. Spot VMs offer up to 91% discount on vCPUs, memory, GPUs, and attached Local SSD disks but can be preempted at any time with a stop or delete termination action and carry no Compute Engine service level agreement, while standard reservations hold zonal VMs for the reservation period at standard rates with very high assurance.
Choose Compute Engine when the workload needs full operating system and kernel control, lift-and-shift migration, or custom machine shapes, choose Google Kubernetes Engine when containerized applications need Kubernetes orchestration with automatic pod and node scaling, and choose Cloud Run or Batch when minimal operations matter most. Cloud Run scales automatically based on requests or CPU consumption for event-driven applications, while Batch provisions resources automatically based on queued jobs for large-scale asynchronous work that runs to completion.
An analytics organization is designing a compute architecture on Google Cloud to process incoming data batches. The data processing pipeline has the following requirements:
Which compute provisioning and optimization strategy should you implement?
Create a stateless Managed Instance Group (MIG) configured with Spot VMs using a custom machine type (6 vCPUs, 22 GB RAM) and enable autoscaling
Create a stateful Managed Instance Group (MIG) configured with Spot VMs using predefined e2-standard-8 machine types and enable schedule-based autoscaling
Purchase calendar-mode future reservations for predefined n2-standard-8 instances with resource-based committed use discounts
Create a stateless Managed Instance Group (MIG) configured with on-demand n2-standard-8 VMs and configure an autohealing health check policy without autoscaling
Last updated