Choosing Models for Cost and Scale

Choosing the model type means matching data structure, problem formulation, and operational limits to the model family that can learn and serve the task. Ordered timestamped numbers point to forecasting, high-volume high-dimensional data points to deep neural networks, and text or combined media points to large language and multimodal models.

1.1Time-Series Forecasting with ARIMA_PLUS in BigQuery ML

BigQuery ML (BigQuery ML (opens in a new tab)) ARIMA_PLUS is the model type for univariate time-series forecasting where each row pairs a time point with a numeric value to predict. The problem requires a time_series_timestamp_col that supplies ordered time points and a time_series_data_col that supplies numeric values, both selected from a BigQuery table. This structure learns trend, seasonality, and dependence over time, so it fits future sales and other ordered numeric forecasts rather than fixed-feature classification or language generation.

Training and forecasting follows an ordered SQL workflow:

1. Create model: Run CREATE OR REPLACE MODEL with OPTIONS(model_type = 'ARIMA_PLUS', time_series_timestamp_col, time_series_data_col, auto_arima = TRUE, data_frequency = 'AUTO_FREQUENCY', decompose_time_series = TRUE) over a SELECT of timestamp and value columns.
2. Generate forecast: Run SELECT * FROM ML.FORECAST(MODEL, STRUCT(HORIZON AS horizon, CONFIDENCE_LEVEL AS confidence_level)) where HORIZON is an INT64 count of future time points and CONFIDENCE_LEVEL is a FLOAT64 percentage of future values expected inside the prediction interval.
3. Serve insights: Read forecasted values for future dates from the query results and pass them to downstream applications for decisions.

Setting auto_arima = TRUE lets BigQuery ML search autoregressive and moving-average structure automatically, data_frequency = 'AUTO_FREQUENCY' infers time granularity, and decompose_time_series = TRUE separates components for interpretation. Because output depends directly on the timestamp column and horizon, missing timestamps, irregular frequency, or inaccurate inputs cause weak performance or false outputs.

1.2Deep Neural Networks and Foundation Models for Complex Patterns

Deep learning foundation models are large models trained on multi-terabyte datasets of text, images, code, and other media to find complex patterns and build contextual understanding. Data volume plus model architecture lets the model produce new content such as text, images, music, or video shaped by its training data. This capability fits problems where inputs are high-volume, high-dimensional, and non-temporal, and where relationships are too complex for explicit time-series equations.

The dominant architecture for these models is the Transformer, developed by Google in 2017, which underlies large-scale models trained on billions of text samples and other content. Foundation models built on Transformers show emergent abilities where one text prompt lets the same model translate languages, answer questions, write a poem, or write code without separate training for each task. A model can then adapt to a domain with prompt techniques or fine-tuning on minimal additional data, so data volume shifts from training from scratch to supplying enough representative examples for adaptation.

1.3Generative Language and Multimodal Reasoning with Large Language Models

Large language models (LLMs) (generative AI applications (opens in a new tab)) are foundation models trained on text that specialize in translation, question answering, summarization, and code generation. An LLM is the right choice when the task requires understanding context, following instructions, or generating fluent new text rather than forecasting a numeric series or classifying fixed features. LLMs can be customized for specific domains where vocabulary, style, and factual grounding matter.

Multimodal models extend LLM ability beyond text to joint reasoning over multiple input types:

• text
• images
• video
• audio
• code

A multimodal prompt combines formats, for example sending an image and asking the application to list or describe objects in the image. Gemini models are built for multimodality and reason across text, images, video, audio, and code in one model. Vertex AI connects discovery to serving, so the serving path flows from model discovery in Model Garden to customization and deployment in Vertex AI.

1.4Matching Modality, Cost, and Operational Constraints to Model Choice

Model selection balances modality fit, size, cost, response quality, latency, traffic, and physical resources. The documented rule is to choose the most affordable model that still meets response quality and latency requirements. Modality must align with the use case, so text-to-image generation depends on a model trained on both text and image data.

Operational demands further separate lightweight forecasting from large-model serving. ARIMA_PLUS training and ML.FORECAST run as SQL in BigQuery, while LLM inference often needs graphics processing units (GPUs) and tensor processing units (TPUs) and serving frameworks such as vLLM on single-host GPU nodes in Google Kubernetes Engine (GKE) (GKE serving (opens in a new tab)). After choosing a deployment environment, anticipated traffic, latency needs, and budget must be balanced against physical resources, where lower cost can mean accepting higher latency on lower-cost machines in test environments.

Choosing the product means selecting the Google Cloud training and orchestration service together with the model type based on data location, control, velocity, cost, and latency. Vertex AI Custom Training (training methods (opens in a new tab)) runs user-owned code of any type on serverless infrastructure, Vertex AI AutoML automates training for common tasks without code, BigQuery ML (BigQuery ML for warehouse users (opens in a new tab)) trains in the warehouse with SQL, and Vertex AI Pipelines orchestrates the full workflow into repeatable production.

2.1When Vertex AI Custom Training Is Required

Vertex AI Custom Training is required when the problem cannot be solved with predefined routines and needs full control over code, dependencies, and infrastructure. Choose Custom Training when the use case does not match the model types and data formats supported by BigQuery ML or Vertex AI AutoML, or when training already runs on-premises or on another cloud and must stay consistent across platforms. The service runs custom routines and deploys models of any type on serverless architecture, with added services for hyperparameter tuning and monitoring. The tradeoff is higher operational ownership in exchange for unlimited framework choice and portability.

The three training environments divide by data location, skill, and control:

Custom Training depends on code portability across environments, so the same training package that runs locally in a Vertex AI Workbench instance can be submitted as a managed job without rewriting the framework logic.

2.2Production Scale with Distributed Training and Hyperparameter Tuning

Vertex AI Custom Training, delivered through the Vertex AI Managed Training service, is the path to scale prototypes beyond single-notebook execution. Training inside a Vertex AI Workbench instance is sufficient for small datasets or subsets of larger datasets, while the training service is recommended for larger datasets or for distributed training. Production use also requires the managed service even for small datasets when training runs on a schedule or in response to new data, because only the managed service provides serverless provisioning, repeatable job submission, and operational monitoring.

The Vertex AI SDK for Python provides the development interface for these jobs and works with major frameworks:

• PyTorch for custom deep learning training submitted from Workbench to managed compute
• TensorFlow for custom deep learning training submitted from Workbench to managed compute
• XGBoost for gradient-boosted tabular training submitted from Workbench to managed compute
• scikit-learn for classical machine learning training submitted from Workbench to managed compute

The SDK sends code from the Workbench development environment to the managed training control plane, which then provisions compute, runs the job, and returns metrics and artifacts to the developer. To maximize predictive accuracy, Custom Training integrates with Vertex AI Vizier for hyperparameter tuning, which tests different configurations across many runs on Google Cloud processing infrastructure. For analysis, Vertex AI TensorBoard tracks experiment metrics such as loss and accuracy over time, visualizes the model graph, and projects embeddings for comparison across runs.

2.3Vertex AI Pipelines for Repeatable MLOps Orchestration

Vertex AI Pipelines is the service for machine learning workflow orchestration that makes training, evaluation, and deployment repeatable and auditable. A pipeline defines the full workflow as code, so data flows from preparation stages in BigQuery, Dataflow, or Vertex AI Feature Store to the Custom Training job, then to model evaluation, then to the Model Registry, and finally to Vertex AI Inference for serving. Because the pipeline triggers each step in order, retraining can run automatically on a schedule or when new data lands, with the same preprocessing, training, and validation applied every time.

This orchestration reduces operational complexity for production-grade MLOps by centralizing dependencies, parameters, and execution order in one versioned definition. The pipeline depends on the underlying services for execution but owns the ordering and data passing between them, so failures, retries, and outputs are tracked per step. Choose Pipelines when the goal is continuous retraining, consistent promotion from experiment to production, and a single system that coordinates Feature Store reads, managed training jobs, and evaluation checks. Lineage is preserved through Vertex ML Metadata, which records linkages across parameters, metrics, and dataset and model artifacts, and through Model Registry, which stores approved versions for deployment and governance.

2.4Evaluate BigQuery ML for In-Database SQL Workflows

BigQuery ML is an in-database machine learning capability that trains and scores models with SQL where warehouse data already lives. Training data flows from BigQuery tables directly to the BigQuery ML training engine, and predictions flow back to BigQuery tables, so no extract to an external cluster is required. This co-location minimizes data movement, egress cost, and batch-inference latency for tabular workloads that start and end in the warehouse.

BigQuery decouples storage in Colossus from query execution in Dremel, storing data compressed in the columnar Capacitor format and operating directly on compressed data without decompressing it. Because storage scales as data is added and compute scales per query through autoscaling, a model can train on terabytes to petabytes without provisioning nodes. The supported SQL-declared options described are linear regression, binary logistic regression, multiclass logistic regression, k-means clustering, and TensorFlow model imports for scoring externally trained networks inside BigQuery.

A complete SQL-driven lifecycle runs in order from untrained to trained to evaluated to registered to served:

1. Create table: Create a local BigQuery table in the project to hold curated training features.
2. Train: Train a BigQuery ML model with a CREATE MODEL statement over that table.
3. Evaluate: Evaluate the model with evaluation functions to produce metrics.
4. Export: Export the model as a cloud model artifact when outside serving is needed.
5. Upload: Upload the exported model as a Vertex AI model resource for managed serving.
6. Tune: Hyperparameter tune with Vertex AI Vizier when defaults are insufficient.
7. Register: Automatically register the model to Model Registry for versioning and discovery.

2.5Assess Vertex AI AutoML for Rapid Prototyping

Vertex AI AutoML is a managed training environment that produces production-ready models for common tabular, image, text, and video problems without custom architecture code. AutoML owns architecture selection, hyperparameter tuning, machine provisioning, training, and serving setup, so a team supplies a labeled dataset that matches a supported format and objective and receives a deployable model. Data flows from its source system into AutoML, where built-in preprocessing applies before AutoML searches architectures and hyperparameters on managed infrastructure. AutoML tabular models can also be initiated from the BigQuery ML environment, so tabular data in BigQuery can flow directly into AutoML training without export.

Deployment supports serving from Google Cloud as a Vertex AI endpoint and deployment to an external device through AutoML Edge. For text, video, or tabular AutoML models, the application must tolerate inference latencies greater than 100ms, which rules out ultra-low-latency use cases that require custom optimization. Custom training with Vertex AI Vizier and Vertex Explainable AI is preferred when the problem falls outside AutoML formats, when cross-platform consistency is required, or when latency, cost, interpretability, and architecture control outweigh development velocity.

Choosing the deployment strategy means selecting batch, online, or streaming inference and the platform, scaling, traffic, and monitoring controls that meet cost, latency, and reliability targets. On Google Cloud, Google Kubernetes Engine (GKE) (orchestrator choice (opens in a new tab)) is the primary platform for custom accelerator-intensive online inference, with Vertex AI Model Registry storing versions, Vertex AI Model Monitoring watching drift, and Cloud Monitoring watching endpoint and infrastructure signals.

3.1Model Versioning, Monitoring, and Lifecycle Control

Model versioning on Vertex AI centers on Vertex AI Model Registry, which stores trained artifacts from Cloud Storage after each training run and records lineage for reverting to the most recent stable version. Each version links to its parameters, code version, dataset version, and evaluation results tracked in Vertex AI Experiments and Vertex AI ML Metadata, so comparisons across iterations are reproducible. Model signing and verification protect this chain, with Cloud Build signing artifacts, Binary Authorization verifying containerized models, and custom checks verifying models deployed directly to Vertex AI endpoints.

Deployment strategy for lifecycle control uses staged releases and automated rollback tied to monitoring signals. Cloud Deploy integrated with Cloud Monitoring tracks the status of canary releases and rollbacks, while endpoints support deploying a new model version alongside a stable version for comparison and traffic shifting. A robust rollback plan uses Model Registry lineage to restore the prior version when Model Monitoring detects significant drift, when generative AI evaluation plus Cloud Logging and Cloud Monitoring flag output-quality shifts, or when Model Armor or custom rules flag malicious inputs and unsafe outputs.

Prediction-quality monitoring compares live behavior against a baseline such as training data or past production traffic:

Continuous evaluation for traditional models uses accuracy, precision, recall, F1-score, AUC-ROC, and log loss, with training-serving skew detection starting from a golden dataset analyzed with TensorFlow Data Validation (TFDV) to establish a baseline schema. Continuous evaluation for generative models uses the Gen AI evaluation service for response quality, safety, grounding, and verbosity, plus human review for subjective quality and LLM-specific signals such as time to first token (TTFT), token throughput, and queries per second.

3.2Design for Scalability and Cost Efficiency

Design for scalability and cost efficiency keeps deployments responsive under variable load while paying only for needed capacity. Capacity planning starts by forecasting users, request rate, historical trends, seasonal variation, and growth, then translating that forecast into Compute Engine managed instance group (MIG) sizing, machine type, persistent disks, and quota headroom checked in the Pricing Calculator. A scalable plan adds N+1 redundancy, load testing under normal and peak conditions, quota review for zonal, regional, and global limits, Cloud Monitoring analysis of historical utilization, and Compute Engine reservations with peak-event drills.

Autoscaling lets serving add capacity when traffic rises and release it when demand falls. Stateless MIGs support autoscaling for graceful handling of increases and lower cost when need is low, while stateful MIGs cannot be autoscaled. On GKE, autoscaling works with node pools and custom compute classes so pods scale across preferred hardware and fall back to alternate capacity when preferred accelerators are unavailable. Effective autoscaling depends on observability of request rate, error rate, latency, and saturation of CPU, GPU, TPU, memory, network, and disk, plus rightsizing with Active Assist recommendations.

Consumption options trade stability and obtainability against discount depth and preemption risk:

Machine-type selection balances latency, throughput, memory, and hourly price, with high-end GPUs such as A100, H100, H200, and B200 suiting training and L4 GPUs suiting cost-sensitive inference. For fluctuating needs, vLLM can dynamically use TPUs and GPUs, an approach described as GPU/TPU fungibility.

3.3Online Inference Serving and Traffic Distribution on GKE

Online inference serving on GKE hosts models in containers on GPU or TPU node pools to answer interactive requests with millisecond-scale latency. A production deployment uses the GKE inference reference architecture, which provisions a standardized environment with Infrastructure as Code using Terraform and deploys single-host GPU nodes running a serving framework such as vLLM, NVIDIA Triton Inference Server, Text Generation Inference, TorchServe, Ray Serve, or MaxDiffusion. Traffic flows from the client to a load balancer to GKE Inference Gateway to a model server Pod equipped with accelerators, and the response flows back along the same path. The gateway adds model-aware routing, affinity-based picking for fine-tuned models sharing low-rank adaptation files, detailed health checks, and Model Armor policies.

Inference latency and throughput metrics separate prompt processing from output generation:

• Time to First Token (TTFT) (ms): Time from request arrival to first output token; captures prefill and scheduling delay.
• Time Per Output Token (TPOT) (ms): Average decode speed as (request_latency - time_to_first_token) / (total_output_tokens - 1); captures streaming smoothness.
• Inter-token latency (ITL) (ms): Time between two output tokens, aggregated as mean, median, and p90; exposes stalls within a single request unlike TPOT.
• Normalized Time Per Output Token (NTPOT) (ms): request_latency / total_output_tokens; allows comparison across different output lengths.
• Request latency (ms): End-to-end time combining queueing, prefill, decode, and post-processing.
• Output, input, and total tokens per second: total_output_tokens / elapsed_time, total_input_tokens / elapsed_time, and total_tokens / elapsed_time; requests per second alone is unreliable for LLMs because it varies with context length.

Model weight loading controls startup and scale-out consistency when many Pods load multi-gigabyte weights simultaneously:

Traffic distribution preserves low latency and high availability under fluctuating demand. Model-metric routing with GKE Inference Gateway sends traffic from the load balancer to the Pod with the best model metrics, prioritizing GPU and TPU instances for generative inference. Priority and fallback profiles with custom compute classes send scheduling first to reserved capacity with fallback to Spot VMs, centralized routing sends low-GPU requests to Cloud Run (single-zone and Compute Engine planning (opens in a new tab)) and GPU-intensive tasks to GKE, and a global external Application Load Balancer sends global users to the nearest healthy backend. Reliability goals tie the four golden signals of latency, traffic, error rate, and saturation to objectives such as 99.9% successful responses, 95th percentile latency below 300 ms, TTFT below 500 ms for 99% of requests, and harmful output below 0.1%.

Modeling techniques given interpretability requirements balance predictive power against transparency, cost, and latency by choosing explainable architectures and explanation methods that stakeholders and regulators can audit. Post-hoc methods explain a trained black-box model without changing predictions, while intrinsically interpretable families build explanation into the model structure itself.

4.1Post-Hoc Explanations with BigQuery ML and Vertex Explainable AI

Post-hoc explainability explains a trained black-box model without changing the model or its predictions. BigQuery ML adds explanations through SQL functions, and Vertex Explainable AI adds explanations through online and batch prediction services that return per-feature attribution scores alongside each prediction. Both paths preserve model performance while producing feature importance scores and local explanations for governance review.

ML.EXPLAIN_PREDICT is an extended version of ML.PREDICT that returns predictions plus per-row explanations for individual instances. The function outputs all input feature columns and then adds attribution columns that quantify how much each feature contributed to that row's outcome. For logistic regression models, the attributions are Shapley values, which measure relative feature attribution. A typical query selects rows from an input table, calls ML.EXPLAIN_PREDICT(MODEL census.census_model, input_query, STRUCT(3 AS top_k_features)), and reads the top three attributions ordered by absolute attribution value in descending order, so the most influential features for that instance appear first.

ML.GLOBAL_EXPLAIN explains the model as a whole by identifying which features are most important across the dataset, answering which features drive decisions such as income bracket overall. ML.EXPLAIN_FORECAST forecasts future time-series values and returns separate components using the same horizon and confidence controls as ML.FORECAST, for example STRUCT(30 AS horizon, 0.8 AS confidence_level) for 30 future points with an 80% interval. Output rows are ordered chronologically by time_series_timestamp and include time_series_data, time_series_adjusted_data, standard_error, prediction interval bounds, and components such as trend, holiday effect, spikes and dips, step changes, residual, and yearly, quarterly, monthly, weekly, and daily seasonality.

Managed explanations with Vertex Explainable AI work for online and batch paths. For online inference, the client calls endpoint.explain(instances=[{...}]) with the same format used for prediction, and the endpoint returns a prediction plus an explanation for each instance. For batch use cases, Vertex AI Batch Prediction runs the same explained prediction over a stored input set for audit workflows. Each explanation compares a baseline to the current instance: baselineOutputValue is the reference output, instanceOutputValue is the output for the explained instance, and the featureAttributions map assigns each feature such as store_id, dept_id, item_id, date, and sales to its numeric score. Forecasting models also return context_columns, horizon_columns, and horizon_weights for importance per horizon.

4.2Trade-Offs Between Complexity, Cost, and Explainability in Architecture Search

Trade-off analysis weighs accuracy gains from larger, more opaque architectures against higher training cost, longer search time, and harder-to-trace decisions. Agent Platform Neural Architecture Search illustrates this trade-off directly, because expanding the search space and model size can improve accuracy but increases compute cost, trial duration, and difficulty in explaining why one architecture was selected. Managing the trade-off requires controlling baseline training cost, using scaled-down proxy tasks, and budgeting total and parallel trials before launching a full search.

Controlling baseline complexity starts with an optimized baseline model before search, because every inefficiency is multiplied across thousands of trials. Teams separate block design from scaling by searching first with a scaled-down model and then running a separate search to scale up, which keeps proxy-task runtime much lower. If certain blocks are independent of the search space, teams load the same pretrained checkpoints for those blocks across all trials, for example using a fixed encoder while only the decoder is searched, so training is faster without bias. Efficiency also depends on data in a Cloud Storage bucket in the same region as the job, tuned TensorFlow pipelines with TFRecord where helpful, distributed training across accelerators or machines, mixed precision training to reduce memory and speed training, and batch size tuned to maximize GPU utilization above 50% without causing out-of-memory failures.

Search budgeting balances convergence confidence against days of compute. Convergence typically requires approximately 2,000 trials, although the exact number varies with search-space size. The recommended practice is to set --max_nas_trial very high, approximately 5,000 to 10,000, then cancel earlier if the search plot shows convergence. The full search runs as batch processing controlled by --max_parallel_nas_trial, with expected duration as (trials-to-converge / max-parallel-nas-trial) * (avg-trial-duration-in-hours / 24). Initial planning can use 3,000 as a rough upper bound for trials and 2 hours for average proxy-task duration. A previous search can be resumed once with search_resume, but a resumed search cannot itself be resumed again.

Resource choice per trial trades per-trial speed against startup overhead:

4.3Built-In Interpretability by Model Family and Serving Design

Designing architectures with built-in interpretability means choosing families, training paths, and serving configurations where explanation is part of the structure. Balancing performance and interpretability starts with optimizing metrics that improve with higher values, including accuracy and f-measure for classification, mean absolute percentage error for forecasting, discounted cumulative gain for ranking, and perplexity and BLEU for language models, versus satisficing metrics that reject any model violating a threshold such as 200 ms prediction latency or mobile model size. Production latency turns satisficing thresholds into service-level objectives such as 95th percentile latency below 300 ms and TTFT below 500 ms for 99% of requests. Because Vertex Explainable AI incurs minor additional cost and compute for attribution, the explanation method directly affects whether the endpoint still meets its latency objective.

Intrinsically explainable families on Gemini Enterprise Agent Platform are supervised tabular models that natively support feature attributions at serving time:

Choose the deep network family when the task needs learned nonlinear representations where Integrated gradients can trace output back to inputs, and choose the tree ensemble family when the task needs partitioned decision logic where Sampled shapley can summarize contributions. Vertex Explainable AI also supports custom-trained tabular and image models, while a Vertex AI Workbench instance (notebook tutorials (opens in a new tab)) provides the What-If Tool for bias exploration and the Language Interpretability Tool for natural language behavior. BigQuery ML complements this with ML.GLOBAL_EXPLAIN for mean absolute attribution across the evaluation set and ML.ADVANCED_WEIGHTS for inspecting learned weights.

Serving explanations under production constraints means deploying to an Agent Platform endpoint with logging and monitoring wired to the same release. The deployment depends on Model Registry, which stores the version with training parameters, evaluation metrics, dataset lineage, code version, limitations, intended uses, and known biases. Deploying a BigQuery ML model with explainability follows an ordered workflow of registering the model to Model Registry, opening the version detail page, starting Deploy to endpoint, creating an endpoint name, selecting a machine type such as n1-standard-2, enabling explainability under Model settings logging, and confirming deployment. Only after deployment can online explanations be requested alongside predictions. For generative systems where token attribution is challenging, process transparency uses trace logs of model calls, tool calls, and retrieved context, citations grounding retrieval-augmented generation claims in source documents, and Gen AI evaluation (evaluation metrics (opens in a new tab)) for grounding, safety, and style plus Model Monitoring for toxicity, coherence, and hallucination rates.

Key Facts

• BigQuery ML ARIMA_PLUS forecasts univariate time series from a timestamp column and numeric value column using HORIZON as INT64 future points and CONFIDENCE_LEVEL as FLOAT64 prediction-interval percentage.
• Vertex AI AutoML trains custom models without training code but requires tolerance for inference latency greater than 100ms for text, video, or tabular models, while Vertex AI Custom Training is required when the problem does not match BigQuery ML or AutoML types.
• LLM inference latency splits into Time to First Token for prefill delay and Time Per Output Token for decode speed, while LLM throughput should be compared with output tokens per second or total tokens per second, not requests per second.
• Large-model scale-out on GKE depends on weight-loading storage, where Cloud Storage FUSE suits limited scaling, Google Cloud Managed Lustre attaches to 10,000 or more Pods, and Hyperdisk ML attaches to up to 2,500 Pods.
• ML.EXPLAIN_PREDICT returns per-row predictions with top-n attributions ordered by absolute Shapley value, ML.GLOBAL_EXPLAIN aggregates mean absolute attribution across the dataset, and Integrated gradients explains DNNs while Sampled shapley explains boosted-tree and random-forest models.