Prepare and test your skills
Prepare and test your skills
Compare Interactive Prototyping and Managed Experiment Tracking Tools is the division of exploratory machine learning work between fast iteration and reproducible capture. Interactive prototyping uses graphical and notebook environments to iterate quickly on data connections, transforms, and inference logic, while managed experiment tracking captures runs, metrics, and artifacts with lineage and telemetry outside the notebook cell. On Google Cloud, Cloud Data Fusion and Dataflow notebooks (opens in a new tab) provide the prototyping surface, and Dataflow, GKE Inference Gateway (opens in a new tab), and managed storage and monitoring services provide the tracking surface.
Interactive prototyping in graphical and notebook environments is code-free pipeline building paired with code-level testing of dataflow patterns. Interactive prototyping in Cloud Data Fusion builds a directed acyclic graph (DAG) in the UI by connecting sources to transformations, sinks, and other nodes, and the Cloud Data Fusion planner transforms this DAG into parallel computations executed as an Apache Spark job on Managed Service for Apache Spark. Data flows from the source database through JDBC drivers and plugin transforms to the destination such as BigQuery, so the same UI graph defines both logic and parallel execution.
Cloud Data Fusion exposes customizable plugins for sources, transforms, aggregates, sinks, error collectors, alert publishers, actions, and post-run actions, and JDBC drivers can be uploaded and configured to read from source databases. Dataflow notebooks complement this UI with experimentation for Apache Beam patterns, including enrichment, embeddings, preprocessing, and inference. Representative notebook patterns include:
Choose graphical prototyping when the work connects known sources to known sinks with standard transforms, and choose notebook prototyping when the work must test Beam transforms, vocabulary handling, scaling, or per-key and generative inference behavior before formalizing the pipeline.
Managed tracking of runs, metrics, and artifacts is the capture of what ran, what it consumed, and how it performed in systems that persist telemetry outside the notebook cell. Dataflow provides in-system telemetry for the pipeline itself, including total bytes per stage to inform later stage design, data locality that avoids explicit materialization between stage boundaries, and autoscaling where resources scale as data moves through pipeline stages. Because a single system owns all stages, Dataflow removes the need for an external orchestration system to coordinate multiple pipelines.
Serving-layer tracking comes from Kubernetes gateways and monitoring. GKE Inference Gateway offers built-in observability into GPU or Tensor Processing Unit (TPU) utilization, key-value (KV) cache hits and utilization, request queue length, and model latency, while GKE Gateway with custom metrics relies on application-reported metrics viewed in Cloud Monitoring. For durable artifacts, Dataflow integrates with managed services that absorb massive parallel input and output: Pub/Sub for global event stream ingestion, Bigtable for wide-column caching and serving, Cloud Storage for unified object storage, and BigQuery for petabyte-scale warehousing. Bigtable complements Dataflow with low-latency reads and writes at 6 ms on the 90th percentile and support for many thousands of concurrent clients, which makes Bigtable suitable as both a source and a sink inside the processing-phase DoFn function.
Formalizing exploratory work into reproducible runs is the shift from task-based execution on manually managed hosts to data-based execution in a single DAG with managed lineage. The highly parallel Dataflow pattern runs the full workflow in one pipeline in four ordered stages:
A single Apache Beam DAG defines this shape, where a PTransform contains a DoFn whose process function takes a data element with one or more properties. Parallel pipelined stages rejoin with side-inputs or joins such as CoGroupByKey, and data stays loaded in the system for the entire workflow without intermediate output to databases or storage. This contrasts with the task-farming pattern where each virtual machine (VM) in a managed instance group (MIG) runs third-party pipeline software, a lightweight agent on each VM pulls messages from Pub/Sub topics, starts a pipeline, waits for completion, and then retrieves the next message. Triggers flow from Cloud Scheduler, Managed Service for Apache Airflow, Cloud Storage triggers, Cloud Endpoints, Cloud Functions, or the Pub/Sub API into the Pub/Sub topic, and the workflow depends on the agent and the vendor distribution facility rather than on a unified DAG runner.
Choosing the serving layer for experimental inference is the choice between AI-aware routing and general-purpose metric routing, both built on the GKE Gateway API. GKE Inference Gateway extends the Gateway API with InferencePool and InferenceModel custom resource definitions (CRDs) and routes with native AI signals, while GKE Gateway with custom metrics uses Application Load Balancers and Open Request Cost Aggregation (ORCA) load reporting over HTTP headers with policies such as WEIGHTED_ROUND_ROBIN for endpoint weighting within a locality.
| Feature | GKE Inference Gateway | GKE Gateway with Custom Metrics |
|---|---|---|
| Primary use case | Optimize latency-sensitive GPU or TPU large language model (LLM) inference with fair access to model resources | Distribute general HTTP(S) traffic on application-reported utilization for services with uniform request cost |
| Supported metrics | Native KV cache utilization, pending queue length, GPU or TPU utilization, LoRA adapter availability, per-request cost, plus optional application headers | Application-reported ORCA metrics including CPU, memory, and custom-named constrained resources |
| Request handling | Handles non-uniform LLM costs from varying prompt complexity with criticality levels and prioritization | Best for uniform processing costs with no native request prioritization |
| Autoscaling and security | Scales on AI signals such as KV cache utilization and filters content with Model Armor at the gateway | Scales with Horizontal Pod Autoscaler on custom signals and uses Google Cloud Armor, TLS termination, and Identity and Access Management (IAM) |
Traffic flows from clients through the selected gateway to the least-loaded model server replica, and the gateway choice depends on the experiment. Choose GKE Inference Gateway when routing must consider LoRA affinity, queue length, KV cache state, variable prompt cost, request criticality, or built-in AI observability, such as serving vLLM variants during comparison. Choose GKE Gateway with Custom Metrics when one or two ORCA signals describe load, requests have uniform cost, and consistency with existing Application Load Balancer deployments avoids unnecessary complexity.
Exam tip: GKE Inference Gateway natively supports LoRA affinity routing and request criticality for non-uniform LLM costs, while GKE Gateway with Custom Metrics does not and relies on ORCA-reported metrics with uniform-cost assumptions.
Evaluate Pipeline Orchestration Architectures Between Vertex AI and Self-Managed Kubeflow is the choice between managed orchestration and self-managed container orchestration for reproducible machine learning workflows. Gemini Enterprise Agent Platform Pipelines (opens in a new tab) is the managed orchestration service for ML workflow orchestration, while self-managed Kubeflow Pipelines runs its steps as container workloads on Google Kubernetes Engine (GKE) (opens in a new tab). The managed path reduces infrastructure ownership and integrates directly with governance services, while the GKE path keeps control of the cluster, network, accelerators, and storage performance tuning.
Managed orchestration with Gemini Enterprise Agent Platform Pipelines is serverless execution where a pipeline definition triggers managed runs without operating the orchestrator control plane. A pipeline definition triggers managed execution where data flows from training and evaluation steps into governed artifacts, and those artifacts flow into deployment and monitoring without the team managing the orchestrator. The pipeline depends on surrounding platform services for lineage, registration, and oversight, which preserves reproducibility across teams.
The managed ecosystem around pipelines includes peer services that each own a lifecycle stage:
Choose this managed orchestration when framework dependencies are standard, collaboration requires shared lineage and registry, and operational requirements favor no cluster management over low-level control.
Self-managed orchestration on GKE runs each pipeline step as containers scheduled by Kubernetes, with the team owning cluster configuration and workload packaging. The GKE cluster manages the compute nodes where training and serving processes run, including the control plane, nodes, and system components. The Kubernetes scheduler in the control plane manages workload lifecycle, scaling, and upgrades, while the kubelet agent on each node starts and runs the scheduled containers.
The GKE execution environment consists of connected infrastructure that a pipeline depends on:
Traffic flows from users through Cloud Load Balancing to serving containers, data flows from Cloud Storage through Cloud Storage FUSE volumes into training containers, and control signals flow from the control plane through the kubelet to each node. This architecture preserves Kubernetes-native operation but retains overhead for node images, VPC design, accelerator selection, driver installation, and volume integration that the managed pipeline service abstracts away.
Storage and accelerator dependencies for pipeline steps determine whether a step meets throughput, latency, and concurrency requirements on GKE. Cloud Storage Rapid Bucket provides zonal buckets with very high throughput up to 15 TB/s and sub-millisecond latency for open files at optimal cost, while Managed Lustre provides sub-millisecond latency and parallel data access as a dedicated ultra-fast workspace. Cloud Storage Rapid Cache acts as a booster that pre-warms model weights in the same zone as inference nodes to reduce cold start.
Pipeline artifact patterns map to different primary storage choices:
| Pipeline artifact pattern | Primary choice | Why it fits |
|---|---|---|
| Training and preparing datasets | Cloud Storage Rapid Bucket | Capacity, throughput scale, cost efficiency, and durability for massive dataset volumes |
| Moving or saving model weights for checkpointing | Managed Lustre | Parallel access lets thousands of rollout workers pull the same weight file simultaneously |
| Storing and downloading models for inference | Cloud Storage Rapid Cache or Rapid Bucket | Pre-warmed zonal weights let new inference instances download quickly and serve the first request faster |
| Key-value (KV) cache offloading for chat resumption | Managed Lustre | Parallel access lets different nodes pull KV cache and resume chats without reprocessing history |
Secondary choices reverse the tradeoff when the bottleneck changes. Managed Lustre is the secondary choice for training datasets when metadata concurrency is the priority, Cloud Storage Rapid Bucket with GCSFS through fsspec or tuned Cloud Storage FUSE is the secondary choice for asynchronous multi-tiered checkpointing, and Managed Lustre is the secondary choice for inference when many concurrent GPUs download the same model simultaneously. For model serving, Run:ai Model Streamer for vLLM is recommended for peak download performance, while optimizing Cloud Storage FUSE parallel download parameters reduces cold-start latency for other inference stacks.
Exam tip: Use Rapid Bucket as primary for training datasets and Managed Lustre as primary for checkpoint weight transfers — the high-throughput zonal bucket fits dataset scale, while the parallel file system fits concurrent checkpoint reads.
Choosing the orchestration platform balances infrastructure overhead, execution control, and enterprise security policy. Gemini Enterprise Agent Platform Pipelines fits when the priority is reproducible collaboration with built-in lineage in Vertex ML Metadata, registration in Model Registry, and monitoring and evaluation in Agent Platform. Self-managed execution on GKE fits when the priority is control over the VPC boundary, scheduler behavior, GPU or TPU selection, and storage tuning with Rapid Bucket, Rapid Cache, and Managed Lustre.
Security policy further separates the options. The GKE foundation uses TLS, Identity and Access Management (IAM), role-based access control (RBAC), and namespaces, with Google Cloud Armor and Model Armor filtering at the gateway and load balancer layers. For sealed AI workloads, linked runner instances run as Trusted Execution Environments with Titanium Intelligence enclave for TPUs and NVIDIA Confidential Computing for GPUs, a minimal OS image with no SSH or container shell access, workload and firmware attestation, and container image signature validation. Protecting model weights in that sealed pattern follows an ordered sequence where access is granted only to attested workloads:
Choose managed pipelines when serverless operation and integrated governance outweigh custom infrastructure needs, and choose self-managed Kubeflow Pipelines on GKE when VPC isolation, Kubernetes control, accelerator and storage optimization, or sealed attested execution are required by cost or security policy.
Select Compute Infrastructure and Framework-Specific Execution Environments is the practice of matching a machine learning framework and workload shape to the right runtime, container, and accelerator. Gemini Enterprise Agent Platform provides managed training and serving with framework-specific pre-built containers, Google Kubernetes Engine (GKE) provides container orchestration with direct control over node pools and topologies, and Compute Engine and Cloud Run provide virtual-machine and serverless container options. Framework code that targets accelerators flows from the framework graph through the runtime to the hardware, so the choice of container, machine type, and accelerator determines performance, scalability, and reproducibility.
Framework-specific containers and custom runtimes package a framework with its dependencies, drivers, and accelerator libraries so experiments run reproducibly across development and managed training. Gemini Enterprise Agent Platform custom training accepts pre-built containers for TensorFlow, PyTorch, and JAX and also accepts custom containers when the workload needs additional libraries, custom TensorFlow operations written in C++, or Hugging Face serving stacks such as Optimum TPU (opens in a new tab), JetStream, or vLLM. Code that runs on Cloud TPUs (opens in a new tab) must be compiled by the accelerated linear algebra (XLA) compiler, which takes the graph emitted by the framework application and compiles the linear algebra, loss, and gradient components into TPU machine code while the rest of the program runs on the TPU host machine. Because the XLA compiler is part of the TPU virtual machine (VM) image that runs on the TPU host, the container and host image must expose a compatible XLA and framework version or TPU execution will fail.
Optimum TPU illustrates a framework-specific serving runtime for open-source large language models from Hugging Face on GKE. The Optimum TPU container downloads models from Hugging Face and serves them on a GKE Standard cluster with a TPU node pool sized to the recommended TPU topology for the model characteristics. Optimum TPU supports the following serving features:
Choose JetStream when the priority is the latest Google optimizations for LLM serving, choose vLLM or quantized serving with Ollama on Cloud Run for portable Graphics Processing Unit (GPU)-based inference, and choose Optimum TPU when the workflow must stay in the Hugging Face transformers ecosystem on TPUs.
Matching workloads to CPUs, GPUs, and Cloud TPUs decides whether an experiment is bottlenecked by flexibility, programmability, or matrix throughput. Cloud TPUs are Google custom-developed application-specific integrated circuits (ASICs) designed for large matrix operations found in machine learning algorithms, with on-chip high-bandwidth memory (HBM) that supports larger models and batch sizes and with slices that connect TPUs in groups to scale workloads with little to no code changes. The XLA compiler tiles matrix multiplies into smaller blocks for the 128x128 systolic array matrix unit (MXU), and the TPU memory subsystem prefers dimensions that are multiples of 8 for tiling efficiency. A program whose computation is dominated by non-matrix operations such as add, reshape, or concatenate will likely not achieve high MXU utilization, so model layout directly controls TPU efficiency.
The three processor families split by workload shape:
| Option | Best for | Avoid when |
|---|---|---|
| Central Processing Units (CPUs) | Quick prototyping that requires maximum flexibility, simple models that do not take long to train, small models with small effective batch sizes, and models limited by host I/O or networking bandwidth | Large matrix-heavy training that needs weeks of accelerated throughput |
| Graphics Processing Units (GPUs) | Medium-to-large models with larger effective batch sizes, models with TensorFlow operations not available on Cloud TPU, and models with significant custom PyTorch or JAX operations that must run at least partially on CPUs | Ultra-large matrix workloads where TPU slices and HBM would give higher throughput per watt |
| Cloud TPUs | Models dominated by matrix computations, models with no custom PyTorch or JAX operations inside the main training loop, models that train for weeks or months, large models with large effective batch sizes, and models with ultra-large embeddings for ranking and recommendation | Linear algebra with frequent branching or many element-wise operations, workloads requiring high-precision arithmetic, and networks with custom operations in the main training loop |
Exam tip: Cloud TPU execution requires XLA compilation of the main training loop, so any custom PyTorch, JAX, or TensorFlow operation inside that loop disqualifies the workload from TPUs even if the model is large.
Managed environments for training and experimentation separate interactive development from repeatable, distributed execution. Vertex AI Workbench instances provide per-member notebook development for prototyping, small-dataset training inside the notebook, and model understanding with feature attributions, while Experiments on Gemini Enterprise Agent Platform logs parameters, code versions, dataset versions, and results for comparison, debugging, and reproducibility. Gemini Enterprise Agent Platform custom training then moves code from the notebook to a managed job where the worker pool specification defines the machine type, the type and number of accelerators, and the number of worker pools. The managed training service depends on Cloud Storage buckets for datasets, checkpoints, and model artifacts and on Model Registry for versioned artifacts, so storage redundancy and artifact organization determine recoverability.
Infrastructure as code makes accelerator selection repeatable across development, staging, and production. Terraform defines GKE clusters with node pools optimized by workload, such as A100, H100, H200, or B200 GPUs for training and L4 GPUs for inference, plus Agent Platform endpoints with defined machine types and scaling policies and Cloud Storage buckets for data and artifacts. Cluster Toolkit accelerates this workflow with Google-curated Terraform blueprints for ready-to-use high-performance computing (HPC), artificial intelligence (AI), and machine learning (ML) clusters in Slurm or GKE, and Config Connector manages Agent Platform datasets, models, endpoints, Cloud SQL instances, Pub/Sub topics, and Cloud Storage buckets declaratively as Kubernetes objects. For jobs that tolerate interruptions and where training code implements checkpointing, Spot VMs reduce cost by allowing the scheduler to preempt workers and resume from saved checkpoints.
Scaling distributed experimentation and serving connects autoscaling, high availability (HA), and fault tolerance across the chosen compute. Gemini Enterprise Agent Platform endpoints scale online inference by adjusting replica count between a configured minimum and maximum based on traffic and metrics such as CPU utilization and replica utilization, with a minimum of two replicas for HA. Dataflow pipelines use horizontal autoscaling to adjust worker instances based on CPU utilization, pipeline parallelism, and pending data, while GKE configures the cluster autoscaler and node auto-provisioning to adjust node count based on pending Pod requests for CPU, memory, GPU, and TPU. The Horizontal Pod Autoscaler (HPA) then scales deployments based on CPU and memory utilization or custom AI and ML metrics such as GPU or TPU utilization and prediction requests per second.
Serverless and multi-location designs extend the same scaling model to variable inference traffic. Cloud Run scales container instances between configured minimum and maximum limits based on incoming requests and scales efficiently to zero instances when there are no requests, with GPU-enabled instances scaled by specifying the accelerator type for agents and third-party workloads such as quantized models with Ollama and LLM inference with vLLM. For global availability and low latency, models deploy to multiple Agent Platform endpoints across regions or to the global endpoint with global load balancing routing traffic across locations. Training on GKE or Compute Engine managed instance groups (MIGs) adds hardware fault handling by monitoring for Xid errors and then resetting GPUs, resetting Compute Engine instances, or triggering hardware replacement with the gcloud report faulty host command, complemented by elastic training with the Google Resiliency Library for GPUs and resilient training with Pathways for TPU workloads.
Graphical prototyping in Cloud Data Fusion fits when the work connects known sources to known sinks with standard transforms, while notebook prototyping with Dataflow notebooks fits when the work must test Beam transforms, vocabulary handling, scaling, or per-key and generative inference behavior before formalizing the pipeline. Cloud Data Fusion builds a directed acyclic graph in the UI that the planner executes as an Apache Spark job on Managed Service for Apache Spark, while Dataflow notebooks support experimentation for enrichment, embeddings, preprocessing, and inference patterns.
Google Kubernetes Engine Inference Gateway extends the Gateway API with InferencePool and InferenceModel custom resource definitions and routes with native artificial intelligence signals including key-value cache utilization, queue length, LoRA affinity, and request criticality for non-uniform large language model costs. GKE Gateway with Custom Metrics uses Application Load Balancers and Open Request Cost Aggregation load reporting over HTTP headers with policies such as WEIGHTED_ROUND_ROBIN and fits uniform-cost traffic where one or two application-reported signals describe load.
Gemini Enterprise Agent Platform Pipelines fits when the priority is reproducible collaboration with built-in lineage in Vertex ML Metadata, registration in Model Registry, and monitoring and evaluation without managing the orchestrator control plane. Self-managed Kubeflow Pipelines on Google Kubernetes Engine fits when the priority is control over the Virtual Private Cloud boundary, scheduler behavior, graphics processing unit or Tensor Processing Unit selection, and storage tuning with Rapid Bucket, Rapid Cache, and Managed Lustre or sealed attested execution.
Cloud Storage Rapid Bucket is the primary choice for training and preparing datasets because capacity, throughput scale, cost efficiency, and durability fit massive dataset volumes with very high throughput up to 15 TB/s. Managed Lustre is the primary choice for moving or saving model weights for checkpointing and for key-value cache offloading because parallel access lets thousands of workers pull the same weight file simultaneously and lets different nodes resume chats without reprocessing history.
Last updated