Scaling Prototypes Into Production ML Systems

Building models given the task means matching data shape, problem type, and operational limits to the model family, training product, deployment path, and explanation method. Ordered numeric series point to forecasting, high-volume high-dimensional data point to deep networks, and text plus image, video, and audio point to large language and multimodal models. The documented rule is to choose the most affordable model that still meets response quality and latency.

1.1Forecasting, Deep Learning, and Multimodal Model Families

Model family choice starts from modality and data structure. BigQuery ML ARIMA_PLUS is the model type for univariate time-series forecasting where each row pairs a time_series_timestamp_col with a numeric time_series_data_col from a BigQuery table. Training follows an ordered SQL workflow: first run CREATE OR REPLACE MODEL with OPTIONS(model_type = 'ARIMA_PLUS', time_series_timestamp_col, time_series_data_col, auto_arima = TRUE, data_frequency = 'AUTO_FREQUENCY', decompose_time_series = TRUE), then run SELECT * FROM ML.FORECAST(MODEL, STRUCT(HORIZON AS horizon, CONFIDENCE_LEVEL AS confidence_level)), then read forecasted values for downstream decisions. HORIZON is an INT64 count of future points and CONFIDENCE_LEVEL is a FLOAT64 percentage expected inside the prediction interval, while missing timestamps or irregular frequency cause weak performance.

Deep learning foundation models are large models trained on multi-terabyte datasets of text, images, code, and other media to learn complex patterns and generate new content. The dominant architecture is the Transformer, developed by Google in 2017, which enables emergent abilities where one prompt can translate, answer questions, write a poem, or write code. Large language models (LLMs) specialize this capability for translation, summarization, question answering, and code generation, while Gemini models extend it to joint reasoning over text, images, video, audio, and code in one model. The serving path flows from model discovery in Model Garden to customization and deployment in Vertex AI.

Operational demands separate SQL forecasting from accelerator-intensive serving. ARIMA_PLUS training and ML.FORECAST run as SQL in BigQuery, while LLM inference often needs graphics processing units (GPUs) and tensor processing units (TPUs) with serving frameworks such as vLLM on single-host GPU nodes in Google Kubernetes Engine (GKE) (opens in a new tab).

1.2Training Products from BigQuery ML to Custom Training and Pipelines

Training product choice depends on data location, control, velocity, cost, and latency. Vertex AI Custom Training runs user-owned code of any type on serverless infrastructure, Vertex AI AutoML automates training for common tasks without code, BigQuery ML trains in the warehouse with SQL, and Vertex AI Pipelines (opens in a new tab) orchestrates the full workflow into repeatable production.

Vertex AI Custom Training, delivered through the Vertex AI Managed Training service, is the path to scale beyond single-notebook execution. Training inside a Vertex AI Workbench instance fits small datasets or subsets, while the managed service is recommended for larger datasets, distributed training, and any training that runs on a schedule or in response to new data. The Vertex AI SDK for Python submits PyTorch, TensorFlow, XGBoost, and scikit-learn code from Workbench to managed compute, which provisions resources, runs the job, and returns metrics and artifacts. Custom Training integrates with Vertex AI Vizier for hyperparameter tuning and Vertex AI TensorBoard for tracking loss, accuracy, model graphs, and embeddings.

Vertex AI Pipelines makes training, evaluation, and deployment repeatable and auditable. Data flows from preparation in BigQuery, Dataflow, or Vertex AI Feature Store to the Custom Training job, then to evaluation, then to Model Registry, and finally to Vertex AI Inference. Because the pipeline triggers each step in order, retraining can run automatically on schedule or when new data lands. Lineage is preserved through Vertex ML Metadata and Model Registry, which store parameters, metrics, dataset versions, and approved model versions.

1.3Deployment Strategies for Online, Batch, and Monitored Serving

Deployment strategy means selecting batch, online, or streaming inference plus the scaling, traffic, and monitoring controls that meet cost, latency, and reliability targets. Online inference on GKE hosts models in containers on GPU or TPU node pools for interactive requests. Traffic flows from the client to a load balancer to GKE Inference Gateway to a model server Pod with accelerators, and the response flows back along the same path. The gateway adds model-aware routing, affinity-based picking for fine-tuned models sharing low-rank adaptation files, health checks, and Model Armor policies.

Model weight loading controls startup when many Pods load multi-gigabyte weights at once. Cloud Storage FUSE mounts buckets with caching for simple access with limited scaling, Google Cloud Managed Lustre is a parallel file system that can attach to 10,000 or more Pods for high-throughput scale-out, and Google Cloud Hyperdisk ML is block storage that can attach to up to 2,500 Pods. Traffic distribution uses model-metric routing to the Pod with the best metrics, priority and fallback profiles from reserved capacity to Spot VMs, centralized routing that sends low-GPU requests to Cloud Run and GPU-intensive tasks to GKE, and a global external Application Load Balancer that sends users to the nearest healthy backend.

Latency and throughput metrics separate prompt processing from output generation:

• Time to First Token (TTFT) in ms: Time from request arrival to first output token.
• Time Per Output Token (TPOT) in ms: (request_latency - time_to_first_token) / (total_output_tokens - 1).
• Inter-token latency (ITL) in ms: Time between two output tokens, reported as mean, median, and p90.
• Normalized Time Per Output Token (NTPOT) in ms: request_latency / total_output_tokens.
• Request latency in ms: End-to-end time including queueing, prefill, decode, and post-processing.
• Tokens per second: Output, input, and total tokens per second; requests per second alone is unreliable because it varies with context length.

Versioning and monitoring close the lifecycle. Vertex AI Model Registry stores artifacts from Cloud Storage with lineage to parameters, code, dataset, and evaluation results in Vertex AI Experiments and ML Metadata. Canary releases and rollbacks use Cloud Deploy with Cloud Monitoring, while endpoints can run a new version alongside a stable version for comparison. Prediction-quality monitoring compares live behavior to a baseline to detect input feature drift, output drift, and feature attribution drift, with TensorFlow Data Validation establishing a golden-dataset schema and the Gen AI evaluation service scoring response quality, safety, grounding, and verbosity.

1.4Interpretability Methods for Auditable Predictions

Interpretability balances predictive power against transparency, cost, and latency. Post-hoc methods explain a trained black-box model without changing predictions, while intrinsically interpretable families build explanation into the model structure. ML.EXPLAIN_PREDICT returns predictions plus per-row explanations, including Shapley values for logistic regression, ordered by absolute attribution so the top features appear first with STRUCT(3 AS top_k_features) limiting output to three features. ML.GLOBAL_EXPLAIN identifies the most important features across the dataset, and ML.EXPLAIN_FORECAST returns forecasted values with trend, seasonality, holiday effect, spikes and dips, step changes, and prediction intervals ordered by time_series_timestamp.

Managed explanations with Vertex Explainable AI work for online and batch paths. For online inference the client calls endpoint.explain(instances=[{...}]) and receives a prediction plus explanation per instance, while batch prediction runs the same explained prediction over stored inputs. Each explanation compares baselineOutputValue to instanceOutputValue and assigns scores in a featureAttributions map for features such as store_id, dept_id, item_id, date, and sales. Intrinsically explainable families use Integrated gradients for dnn_classifier, dnn_regressor, dnn_linear_combined_classifier, and dnn_linear_combined_regressor, and Sampled shapley for boosted_tree_classifier, boosted_tree_regressor, random_forest_classifier, and random_forest_regressor. Because Explainable AI adds cost and compute, the explanation method directly affects whether the endpoint meets objectives such as 95th percentile latency below 300 ms.

Training models on Google Cloud means separating operational sources from training-optimized copies, ingesting diverse sources through the right landing and processing services, choosing the training SDK, and diagnosing failures with logs, metrics, and lineage. Data flows from logs, devices, or documents into BigQuery (opens in a new tab) tables or Cloud Storage objects, then into registered datasets, training jobs, and Model Registry. This separation protects production performance while giving training high-throughput, reproducible inputs.

2.1Organizing Tabular and Unstructured Training Inputs

Training-data preparation extracts data from live systems into formats and locations tuned for machine learning. Training jobs must not read directly from operational databases because scans would contend with production traffic. Structured data stays in BigQuery as materialized tables rather than views or subqueries, with partitioning so queries over a date range scan only matching partitions. Unstructured image, video, and audio data stays in Cloud Storage buckets as large sharded containers, using sharded TFRecord for TensorFlow and Avro for other frameworks.

Throughput depends on layout, location, and access path. Data flows from Cloud Storage objects through parallel streams to each GPU or TPU worker, and each worker reads a disjoint subset of shards. Buckets and directories group shards so jobs list and stream a coherent prefix.

Rapid Cache is an SSD-backed zonal read cache that accelerates reads to dual-region or multi-region buckets at up to 2.5 TB/s without code changes. Hierarchical namespace (HNS) enables file-system structure with faster lookups and 8 times higher queries per second, and HNS is required for zonal buckets with Rapid Bucket. Cloud Storage FUSE mounts buckets as local file systems with architectures that can exceed 1 TB/s aggregate throughput when tuned with parallel downloads and caching.

A managed Vertex AI Dataset registers the table, query result, or storage manifest as a versioned input that links preprocessed data to custom-trained models. The lifecycle runs in order: first prepare and store data in BigQuery or sharded Cloud Storage, then register it as a managed dataset, then split it into training, validation, and test subsets, then consume the named version in AutoML, custom training, or pipelines, then compare runs in Experiments and TensorBoard. Multimodal datasets extend the same pattern to mixed text, image, audio, and video stored in BigQuery for both supervised fine-tuning and batch prediction.

2.2Ingesting Batch and Streaming Sources into Pipelines

Ingestion matches each source to a landing service and a processing service based on volume, velocity, and format. BigQuery is the landing zone for structured and semi-structured tabular data, while Cloud Storage is the landing zone for unstructured text, images, video, and audio. Pub/Sub, Dataflow, and Dataproc move and transform data between operational sources and those stores, and training pipelines then read from Cloud Storage or BigQuery rather than live systems.

Pub/Sub is the fully managed messaging service that decouples producers from ingestion when velocity is high or bursty. Data flows from publishers such as logs, sensors, or APIs to a Pub/Sub topic, then to subscribers such as a Dataflow pipeline or Cloud Storage sink. Because the topic buffers messages independently, spikes do not overwhelm transformation. With Eventarc, traffic flows from the event source to Eventarc to a Dataflow job, and only after the trigger fires does Dataflow read, transform, and write the micro-batch for retraining.

Validation enforces schema, format, and statistical quality before training. TensorFlow Data Validation (TFDV) validates schemas, formats, and distributions and monitors shifts, with results flowing to BigQuery for analysis and Cloud Storage for archiving. BigQuery handles large-scale SQL checks for missing values, out-of-range values, and duplicates, while Dataflow handles programmatic checks for malformed records and late events. Agent Platform Feature Store centralizes features with BigQuery as the offline store for historical values and an online store instance with feature views for low-latency serving.

2.3Training with Custom, AutoML, and Kubeflow SDKs

Model training with different SDKs means choosing between serverless custom training, managed AutoML, and containerized Kubeflow workflows on GKE. Vertex AI custom training runs user-supplied code on managed virtual machines (VMs) defined by a CustomJob, AutoML automates architecture and tuning for supported formats, and Kubeflow on GKE runs multi-step containerized workflows on Kubernetes. Training data flows from Cloud Storage or BigQuery into the training service, and outputs flow back to Cloud Storage and Model Registry.

Custom container packaging bundles training code into a Docker image that Vertex AI runs directly. The prerequisite workflow runs in order: first create an Artifact Registry Docker repository, then build and push the image URI, then configure the service account and buckets for inputs and outputs, then submit a CustomJob and monitor in TensorBoard. A CustomJob carries a job_spec with worker_pool_specs, where each pool defines a machine_spec, a replica_count, and either a container_spec with image_uri, command, and args or a python_package_spec with an executor image and Python tarball. The SDK submits the specification to a regional endpoint such as us-central1-aiplatform.googleapis.com with a parent of projects/{project}/locations/{location}.

Distributed configuration lives in worker_pool_specs, where machine_spec selects the VM family and accelerators and replica_count selects how many identical VMs run. Accelerator selection pairs machine_type with accelerator_type and accelerator_count, and code must already be written to use GPUs or the GPUs remain idle. GPU training pools must use an A2, N1, or G2 machine type, and small machines such as n1-highmem-2 with GPUs can fail logging because of central processing unit (CPU) constraints. Kubeflow on GKE instead runs pipeline steps as containers that read data and write checkpoints and models to Cloud Storage, with Terraform, Cluster Toolkit, and Config Connector provisioning clusters and Cloud Logging, Cloud Monitoring, Cloud Trace, and Cloud Profiler tracking health.

2.4Troubleshooting Data, Runtime, and Resource Failures

Troubleshooting training failures finds why training cannot ingest data, why model code fails, or why compute and quota are insufficient. TFDV generates a schema from training data and validates each new batch, while distribution tracking exposes drift that can trigger retraining. When jobs report permission denied, missing objects, or slow reads, checks cover least-privilege service accounts, bucket-level IAM policies, and Cloud Audit Logs streamed to Cloud Logging. Knowledge Catalog tracks accuracy, freshness, completeness, validity, and consistency, with lineage showing which pipeline run produced the bad partition.

Resource failures are diagnosed by combining Cloud Logging signatures with Cloud Monitoring signals. In Managed Service for Apache Spark, data flows through executors on worker nodes under YARN while shuffle data flows to attached disks. Out-of-memory failures appear as Lost executor, java.lang.OutOfMemoryError: GC overhead limit exceeded, and Container killed by YARN for exceeding memory limits, with follow-on FetchFailedException when crashed executors cannot answer fetches. Fixes include raising spark.executor.memoryOverhead, moving to high-memory machine types, or using smaller partitions. Disk pressure causes slow shuffles and UNHEALTHY YARN nodes when shuffle directories exceed 90% utilization, so standard persistent disks should be at least 1 terabyte per worker. Memory Protection kills appear as Driver received SIGTERM/SIGKILL signal and exited, confirmed by the google.dataproc.oom-killer log.

Dialogflow to Insights notification failures follow an ordered diagnostic flow: first verify notifications in Cloud Monitoring with Metrics Explorer using Resource Type Customer Experience Insights Environment and Metric Notifications From Contact Center AI Runtime, then isolate the failure in Logs Explorer, then escalate to a Google representative with diagnostic API outputs for securitySettings, agents/AGENT-ID, and conversationProfiles.

2.5Tuning Hyperparameters Under a Fixed Budget

Hyperparameter tuning defines which hyperparameters Vertex AI may vary, what values each may take, and which search algorithm chooses the next trial. A HyperparameterTuningJob runs many trials of the same training application with different values, reports a numeric metric from each trial, and returns the best configuration. Hyperparameters such as learning rate stay constant during a single trial, while model parameters such as neural network weights change during training. Trial values flow from the StudySpec to code as command-line arguments, and metrics flow back through the cloudml-hypertune package.

Scaling reshapes how DOUBLE and INTEGER types are sampled by mapping the interval to 0 through 1. UNIT_LINEAR_SCALE preserves equal distances, UNIT_LOG_SCALE spreads small values and requires a strictly positive space, and UNIT_REVERSE_LOG_SCALE spreads values near the top and also requires a strictly positive space. If no algorithm is specified, default Bayesian optimization balances exploration against exploitation. GRID_SEARCH performs exhaustive search and requires all parameters to be INTEGER, CATEGORICAL, or DISCRETE, while RANDOM_SEARCH provides a simple baseline. Budget controls are maxTrialCount for total trials, parallelTrialCount for concurrent trials, and maximum failed trial count for early stopping. In BigQuery ML, tuning uses NUM_TRIALS with HPARAM_RANGE and HPARAM_CANDIDATES.

Vertex AI Pipelines embeds tuning as modular components in order: first data preparation produces the versioned dataset, then training and tuning with HyperparameterTuningJobRunOp produce the best model, then evaluation checks thresholds, then deployment and monitoring follow only after evaluation passes. A custom tuning container must install cloudml-hypertune, parse hyperparameters with argparse, and pre-install CUDA and cuDNN for GPU training. Prediction autoscaling separately allocates replicas between MIN_REPLICA_COUNT and MAX_REPLICA_COUNT to hold CPU utilization, GPU duty cycle, vLLM cache usage, or queue signals near target, with CPU and GPU utilization defaulting to 60 percent.

2.6Fine-Tuning Foundation Models from Model Garden

Fine-tuning adapts a proprietary Gemini model or an open-weight model such as Llama 3.1 to a downstream task. Data flows from JSON Lines (JSONL) datasets in Cloud Storage into a managed tuning job, and the job writes a tuned model, checkpoints, and experiment metadata to a Cloud Storage output directory and deploys a servable endpoint. The tuned adapter attaches task-specific behavior to frozen base weights.

Supervised tuning is the right option when output is well-defined with labels, such as classification, entity extraction, extractive question answering, strict summarization, or persona control, and it is the only supported tuning option for code models. Start with prompting, then choose supervised tuning when 100 or more high-quality labeled examples exist and prompting still fails, since quality matters more than quantity. Datasets are JSONL files at gs:// URIs with at least 100 to 500 examples reflecting production distribution, using prompt-completion, turn-based chat, or GenerateContent objects with multimodal file_data in JPEG, PNG, and WEBP. Training token cost is tokens multiplied by epoch count, with up to 131,072 input and output tokens per example. Hyperparameters include epochs for complete passes, adapter size including 1, 2, 4, 8, and 16 for trainable parameters, and learning rate multiplier defaulting to 1.

Execution binds base model, datasets, hyperparameters, and output location into a managed pipeline. Supported bases include Gemini 3.5 Flash, Gemini 3.1 Flash-Lite, Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite, while open-model tuning can start from meta/llama3_1@llama-3.1-8b or a same-architecture checkpoint. A maximum of 10 checkpoints are stored, with one per epoch when epochs are fewer than 10. Deployment registers the adapter in Model Registry, validates with Cloud Build tests and bulk evaluations, releases in controlled stages with rollback, and schedules continuous training on new data. Evaluation uses the Gen AI evaluation service and automatic side-by-side comparison, while Model Monitoring watches drift, toxicity, coherence, and hallucination.

Choosing appropriate hardware for training means matching central processing units (CPUs), graphics processing units (GPUs), and Cloud Tensor Processing Units (TPUs) (opens in a new tab) to math patterns, batch size, cost, and operational constraints. Training traffic flows from job queues and autoscalers to colocated accelerator hosts, and the choice of processor, machine series, orchestration system, and pricing model determines throughput, stability, and price per training hour.

3.1Matching Processors to Architecture and Batch Size

Processor selection starts from whether the model is dominated by large matrix multiplications and how large an effective batch the hardware can exploit. Cloud Tensor Processing Units (TPUs) are Google's custom application-specific integrated circuits purpose-built for large-scale matrix multiplication, while CPUs offer maximum flexibility for prototyping and GPUs offer broad operation coverage for medium-to-large models. Models that train for weeks or months, models dominated by matrix math, and models with ultra-large embeddings for ranking and recommendation fit TPUs well. Programs dominated by add, reshape, concatenate, frequent branching, or many element-wise operations achieve low matrix unit utilization and do not suit TPUs.

CPUs map to quick prototyping and small effective batch sizes, especially when limited by host input/output or networking bandwidth. GPUs map to medium-to-large models with larger effective batch sizes, especially when code depends on operations not available on Cloud TPU. TPUs map to large matrix-dominated models with large batches that train for weeks or months and contain no custom operations inside the main loop. Cloud TPU provides on-chip high-bandwidth memory (HBM) for larger models and batches, and TPUs connect in groups called slices that scale data-parallel work with little to no code changes.

3.2Precision, Shapes, and Compilation for TPUs

TPU efficiency depends on whether code can compile through the accelerated linear algebra (XLA) compiler and keep the matrix hardware full. The XLA compiler is a just-in-time compiler that takes the graph emitted by the framework and compiles linear algebra, loss, and gradient components into TPU machine code, while the rest of the program runs on the TPU host machine. Workloads with custom operations in the main training loop or requiring high-precision arithmetic are excluded. When lower precision is acceptable, quantization from 32-bit floating-point to 8-bit integers can significantly decrease memory footprint and power consumption without noticeable accuracy loss.

Shape rules follow directly from the hardware. The matrix unit (MXU) is a 128x128 systolic array, and the memory subsystem prefers dimensions that are multiples of 8. The XLA compiler tiles matrix multiplies into smaller blocks and compiles just in time for the first batch, so tensors with dynamic shapes are not well suited because recompiling on every shape change is too slow. When computation cannot fill an entire MXU, the compiler pads tensors with zeros, which underutilizes the core and increases on-chip memory that can lead to out-of-memory errors. To maximize use, either the batch size or a feature dimension must be a multiple of 128 or the compiler will pad it to 128, and ideally both batch and feature dimensions are multiples of 8.

3.3Accelerator Families, Pricing, and Orchestration

Accelerator families map directly to workload intensity, from pre-training to small-model serving. Accelerator-optimized A-series machines including A4X Max with NVIDIA GB300, A4X with NVIDIA GB200, A4 with NVIDIA B200, A3 Ultra with NVIDIA H200 141GB, A3 Mega and A3 High with NVIDIA H100 80GB, and A2 cover pre-training, while fine-tuning adds G4 and inference adds A3 Edge and G4 for lower-cost deployment. Graphics-intensive work uses G4, G2, and N1 with T4. Within GPUs, General GPUs are managed as independent units with asynchronous maintenance, while Clustered GPUs are managed as a single tightly coupled system with synchronized maintenance for multi-host training. For pre-training on AI Hypercomputer, Google Cloud recommends A4X Max, A4X, A4, A3 Ultra, A3 Mega, or A3 High as orchestrated clusters through GKE or Slurm with Cluster Director or Cluster Toolkit.

Cost per training hour is the sum of attached GPUs plus virtual CPUs (vCPUs), memory, and bundled Local SSD, with only one discount type applying at a time. Choose Future reservations or Calendar mode when training cannot tolerate interruption, choose Flex-Start when batch training can wait for capacity and finish within seven days, and choose Spot VMs only when checkpoints and retries can absorb preemption. Cluster Director deploys and manages accelerator groups with colocation for multi-GPU training on A3 Ultra H200 and A4 B200, Cluster Toolkit deploys blueprints for GKE or Compute Engine, and GKE provides autoscaling, high availability, and hardware-aware scheduling. Storage completes the path with Managed Lustre for 10,000 or more Pods, Cloud Storage Rapid and Cloud Storage FUSE for high-throughput weight access, and Hyperdisk ML block storage for up to 2,500 Pods.

3.4Distributed Training with Data and Model Parallelism

Distributed training design chooses how to replicate models, shard weights, and place computation across GPU and TPU workers. The TPU execution foundation is a matrix processor specialized for multiply-and-accumulate operations. A CPU loads values from memory, performs one calculation, and stores the result back for every operation, creating the von Neumann bottleneck. A GPU increases throughput with 2,500 to 5,000 Arithmetic Logic Units that execute thousands of operations at once, but each calculation must still access registers or shared memory. Cloud TPU v3 contains two systolic arrays of 128x128 units where each multiplication result passes directly to the next accumulator with no memory access during multiplication.

Data flow through the TPU worker follows a fixed host-to-TPU-to-host path:

1. Infeed staging: The TPU host streams input data into the infeed queue.
2. HBM staging: The TPU loads data from the infeed queue into High Bandwidth Memory and loads parameters into the matrix unit.
3. Systolic execution: The TPU loads data into the Matrix Multiplication Unit, where results pass between accumulators.
4. Outfeed return: The TPU loads completed results into the outfeed queue, then the host reads results into host memory.

The TPU7x worker pool building block is a full-host, 4-chip virtual machine with 224 vCPUs, 960 GB of RAM, and access to 2 Non-Uniform Memory Access (NUMA) nodes. Adding workers means adding virtual machines or slices connected by high-bandwidth interconnects rather than subdividing a virtual machine. A TPU v2 slice can scale to 512 chips in one fabric and is requested as v2-8 for 8 cores or v2-128 for 128 TensorCores. Managed TPU capacity holds back part of the reservation so Google can replace faulty machines automatically, while All Capacity mode exposes the full reservation and makes the user responsible for failures.

Data parallelism replicates the same model on every worker and splits the training data among replicas. Pure data parallelism shards only the data batch and synchronizes gradients with all-reduce over the data center network, while fully sharded data parallelism shards data, weights, and optimizer state using all-gather for weights and reduce-scatter for gradients. Fully sharded mode decreases memory per slice as slice count grows but increases network traffic. Both modes require scaling the global batch linearly with slice count, with Transformer models implying a practical margin of at least 350k tokens per Pod for two Pods to 700k tokens for many Pods. Model parallelism instead partitions a large model across accelerators: tensor parallelism shards individual layers across dimensions and pipeline parallelism places consecutive layer groups on different slices and streams micro-batches through stages. Tensor parallelism across slices is not recommended because the data center network cannot sustain fine-grained communication, while pipeline parallelism incurs a pipeline bubble where chips sit idle waiting for upstream or downstream stages.

Start with pure data parallelism, use fully sharded data parallelism to relieve memory pressure, and use pipeline parallelism only when those approaches require too large a global batch for convergence. The open-source MaxText framework provides JAX large language model recipes optimized for Cloud TPU, while vLLM with the tpu-inference plugin provides optimized kernels and tensor parallelism for serving throughput.

Key Facts

• BigQuery ML ARIMA_PLUS forecasts univariate time series from a timestamp column and numeric value column using HORIZON as INT64 future points and CONFIDENCE_LEVEL as FLOAT64 prediction-interval percentage.
• Vertex AI AutoML automates training without code but text, video, or tabular models require tolerance for inference latency greater than 100ms, while Custom Training is required when the problem does not match BigQuery ML or AutoML types.
• Training data must be stored as materialized BigQuery tables for structured data and as sharded TFRecord or Avro containers of at least 100 MB with 100 to 10,000 shards for unstructured data.
• GRID_SEARCH requires all parameters to be INTEGER, CATEGORICAL, or DISCRETE and cannot use a DOUBLE range, while supervised tuning needs 100 to 500 high-quality JSONL examples and stores a maximum of 10 checkpoints.
• TPUs require XLA-compiled graphs with static shapes and batch or feature dimensions sized to multiples of 128 to avoid zero-padding of the 128x128 matrix unit, and tensor parallelism across TPU slices is not recommended.