Choosing BigQuery ML, AutoML and Pipelines

Vertex AI Custom Training (opens in a new tab) is the managed option for running user-owned training code of any model type on serverless infrastructure, while Vertex AI Pipelines (opens in a new tab) is the orchestration service that turns training into a repeatable production workflow. Custom Training provides full control over frameworks, dependencies, and distributed compute, and Pipelines connects data preparation, training, evaluation, registration, and serving into one automated system. Together they support continuous retraining on a schedule or when new data arrives, with lineage tracked from dataset to deployed model.

1.1When Vertex AI Custom Training Is Required

Vertex AI Custom Training is required when the problem cannot be solved with predefined routines and needs full control over code, dependencies, and infrastructure. Choose Custom Training when the use case does not match the model types and data formats supported by BigQuery ML or Vertex AI AutoML, or when training already runs on-premises or on another cloud and must stay consistent across platforms. The service runs custom routines and deploys models of any type on serverless architecture, with added services for hyperparameter tuning and monitoring. The tradeoff is higher operational ownership in exchange for unlimited framework choice and portability.

The three training environments divide by data location, skill, and control:

Custom Training depends on code portability across environments, so the same training package that runs locally in a Vertex AI Workbench instance can be submitted as a managed job without rewriting the framework logic.

1.2Production Scale, Distributed Training, and Hyperparameter Tuning

Vertex AI Custom Training, delivered through the Vertex AI Managed Training service, is the path to scale prototypes beyond single-notebook execution. Training inside a Vertex AI Workbench instance is sufficient for small datasets or subsets of larger datasets, while the training service is recommended for larger datasets or for distributed training. Production use also requires the managed service even for small datasets when training runs on a schedule or in response to new data, because only the managed service provides serverless provisioning, repeatable job submission, and operational monitoring.

The Vertex AI SDK for Python provides the development interface for these jobs and works with major frameworks:

• PyTorch for custom deep learning training submitted from Workbench to managed compute
• TensorFlow for custom deep learning training submitted from Workbench to managed compute
• XGBoost for gradient-boosted tabular training submitted from Workbench to managed compute
• scikit-learn for classical machine learning training submitted from Workbench to managed compute

The SDK sends code from the Workbench development environment to the managed training control plane, which then provisions compute, runs the job, and returns metrics and artifacts to the developer. To maximize predictive accuracy, Custom Training integrates with Vertex AI Vizier (opens in a new tab) for hyperparameter tuning, which tests different configurations across many runs on Google Cloud processing infrastructure. For analysis, Vertex AI TensorBoard tracks experiment metrics such as loss and accuracy over time, visualizes the model graph, and projects embeddings for comparison across runs.

1.3Vertex AI Pipelines for Repeatable MLOps Orchestration

Vertex AI Pipelines is the service for machine learning workflow orchestration that makes training, evaluation, and deployment repeatable and auditable. A pipeline defines the full workflow as code, so data flows from preparation stages in BigQuery, Dataflow, or Vertex AI Feature Store to the Custom Training job, then to model evaluation, then to the Model Registry, and finally to Vertex AI Inference for serving. Because the pipeline triggers each step in order, retraining can run automatically on a schedule or when new data lands, with the same preprocessing, training, and validation applied every time.

This orchestration reduces operational complexity for production-grade MLOps by centralizing dependencies, parameters, and execution order in one versioned definition. The pipeline depends on the underlying services for execution but owns the ordering and data passing between them, so failures, retries, and outputs are tracked per step. Choose Pipelines when the goal is continuous retraining, consistent promotion from experiment to production, and a single system that coordinates Feature Store reads, managed training jobs, and evaluation checks.

1.4Lineage Tracking and Artifact Organization

Vertex ML Metadata (opens in a new tab) and Model Registry provide lineage tracking and artifact organization for models built with Custom Training and Pipelines. Vertex ML Metadata records the linkages across parameters, metrics, and dataset and model artifacts, so each trained model can be traced back to the exact data, code, and hyperparameters that produced it. The Model Registry then stores approved model versions as central artifacts for deployment and governance, separating experimental outputs from production candidates.

Lineage starts with data through managed datasets and Vertex AI Experiments. Managed datasets in Vertex AI create a clear link between data and custom-trained models and provide descriptive statistics and automatic or manual splits into train, test, and validation sets. Experiments integrates with Vertex ML Metadata to log and connect those dataset versions with training parameters, TensorBoard metrics, and resulting model artifacts. Use managed datasets when lineage between data and model is critical, and retain custom splitting in training code only when full control over the split logic outweighs the benefit of automatic tracking.

BigQuery ML (opens in a new tab) is an in-database machine learning capability that trains and scores models with SQL where warehouse data already lives. Training data flows from BigQuery tables directly to the BigQuery ML training engine, and predictions flow back to BigQuery tables, so no extract to an external cluster is required. This co-location minimizes data movement, egress cost, and batch-inference latency for tabular workloads that start and end in the warehouse.

2.1In-Database Execution and Data Movement

In-database execution in BigQuery (opens in a new tab) means compute comes to the data instead of copying data to compute. BigQuery decouples storage in Colossus from query execution in Dremel, storing data compressed in the columnar Capacitor format and operating directly on compressed data without decompressing it. Because storage scales as data is added and compute scales per query through autoscaling, a model can train on terabytes to petabytes without provisioning nodes, resizing clusters, or managing a separate training fleet. Choose in-database training and batch scoring when the source tables, features, and consumers are already in BigQuery and the goal is fast iteration without pipeline and transfer overhead.

2.2Supported Model Architectures

BigQuery ML supports SQL-declared models for core tabular and clustering tasks plus imported custom models. The documented architectures cover regression, classification, unsupervised grouping, and reuse of externally trained networks. SQL-native training keeps feature engineering, training, and evaluation in one GoogleSQL workflow, while import and export paths connect to specialized serving.

The supported options described are:

• linear regression for continuous tabular targets
• binary logistic regression for two-class tabular classification
• multiclass logistic regression for multi-class tabular classification
• k-means clustering for unsupervised segmentation
• TensorFlow model imports for bringing an externally trained TensorFlow model into BigQuery for scoring

Converting Oracle Data Mining jobs to these options requires rewriting PL/SQL-based mining code into GoogleSQL CREATE MODEL, ML.EVALUATE, and ML.PREDICT patterns. Use BigQuery ML when one of the listed architectures fits the task, and use Vertex AI Training when the workload needs custom training and scoring at scale beyond those architectures.

2.3SQL-Native Preprocessing and Warehouse Organization

SQL-native preprocessing in BigQuery is an extract, load, and transform (ELT) pattern executed with GoogleSQL. Source data flows into a staging table through load options, then a SQL query reads the staging table, applies cleaning, joins, aggregations, and feature expressions, and writes its output into a final production or training table. GoogleSQL supports the SQL 2011 standard with extensions for nested and repeated data, so semi-structured features can be transformed without leaving the warehouse.

Organization and pruning directly affect machine learning cost and speed. BigQuery datasets (opens in a new tab) are the highest-level abstraction for access control, with schemas and labels for further organization and partitioning to improve query performance, lower scanned bytes, and manage information lifecycle. While columns are encoded, BigQuery persists column statistics that are later used to compile optimal plans and choose efficient runtime algorithms, plus query plan and timing diagnostics for tuning heavy feature queries. Design features as selective, partitioned SQL views or tables so training and batch prediction scan only the columns and partitions they need.

2.4Performance, Cost, and Scaling Mechanics

BigQuery performance for machine learning comes from serverless, query-level scaling rather than pre-provisioned clusters. Compute is metered in slots, which are units of computational capacity managed by Google, memory is provided by a remote distributed service connected over the petabit network, and storage is automatically compressed, encrypted, replicated, and distributed. This separation lets storage grow to dozens of petabytes without buying additional compute for queries that do not need it.

The execution optimizations that matter for large training and scoring scans are:

• in-memory query execution for iterative access without repeated disk reads
• multilevel tree architecture based on the Dremel execution engine for parallel aggregation
• automatic storage optimization within Capacitor for compressed columnar scans
• 1 petabit per second total bisection bandwidth with Jupiter for shuffle and data delivery
• autoscaling resource management to provide fast petabyte-scale queries without manual resizing

Unlike node-based systems that require classic or elastic resizes that pause queries or copy data for hours to days, BigQuery reservations assign slots to projects and scale per workload. Control cost by limiting scanned data with partitioning and column selection, and control latency by keeping batch inference as SQL scoring inside the same dataset and region as the features.

2.5Operational Lifecycle From Training to Registry

The BigQuery ML lifecycle moves a model from untrained to trained to evaluated to registered to served, all anchored to warehouse tables. A repeatable SQL-driven workflow preserves lineage because the training table, SQL text, evaluation output, and prediction table remain in BigQuery. Hyperparameter search and governance connect outward without moving the training data.

A complete workflow runs in this order:

1. Create table: Create a local BigQuery table in the project to hold curated training features.
2. Train: Train a BigQuery ML model with a CREATE MODEL statement over that table.
3. Evaluate: Evaluate the BigQuery ML model with evaluation functions to produce metrics.
4. Export: Export the BigQuery ML model as a cloud model artifact when outside serving is needed.
5. Upload: Upload the exported model as a Vertex AI model resource for managed serving.
6. Tune: Hyperparameter tune a BigQuery ML model with Vertex AI Vizier when defaults are insufficient.
7. Register: Automatically register a BigQuery ML model to Model Registry for versioning and discovery.

BigQuery ML model monitoring tracks deployed behavior using training statistics saved during training, which enable skew detection without retaining the original training data. Data skew means feature value distributions differ between training and serving data, while data drift means production serving data changes over time away from the training distribution and degrades performance. Monitor for standard skew and drift continuously, and enable advanced data skew or drift when fine-grained, feature-level statistics are required.

2.6Choosing BigQuery ML Versus Specialized Training Environments

Choice of training environment depends on where data lives, what model is needed, and where predictions must be served. BigQuery ML keeps tabular training and batch inference inside the warehouse with SQL velocity and minimal operations, while Vertex AI Training runs custom training and scoring workloads at scale and Vertex AI Workbench provides a notebook development environment for data scientists. The export path bridges them: a model trained in SQL can be exported and uploaded as a Vertex AI model resource for online endpoints.

Vertex AI AutoML (opens in a new tab) is a managed training environment in Vertex AI that produces production-ready models for common tabular, image, text, and video problems without custom architecture code. AutoML owns architecture selection, hyperparameter tuning, machine provisioning, training, and serving setup, so a team supplies a labeled dataset that matches a supported format and objective and receives a deployable model. This ownership model gives superior development velocity and strong baseline quality when machine learning expertise, engineering time, or iteration speed is the constraint. The tradeoff is reduced control over training logic, training budget, latency behavior, and model internals compared with custom training.

3.1What Vertex AI AutoML Automates

Vertex AI AutoML automates nearly all aspects of training and serving, including choosing a model architecture, tuning hyperparameters, provisioning training machines, and preparing a serving artifact. Data flows from its source system into AutoML, where built-in preprocessing transformations apply before AutoML searches architectures and hyperparameters on managed Google Cloud infrastructure. Because the platform depends on Google Cloud compute rather than user-managed clusters, teams avoid distributed-training code, cluster sizing, and manual tuning loops. Choose Vertex AI AutoML when rapid prototyping and repeatable baseline quality matter more than explicit control of the training routine.

3.2When Vertex AI AutoML Accelerates Prototyping

Vertex AI AutoML accelerates prototyping when the data and problem match one of the formats, data types, and model objectives that AutoML supports, such as image classification and tabular regression plus supported text, video, and tabular objectives. The workflow depends on a compatible managed dataset in BigQuery for structured data or Cloud Storage for image, video, audio, and unstructured data, with AutoML applying its own preprocessing instead of requiring a custom pipeline. AutoML tabular models can also be initiated from the BigQuery ML environment, so tabular data that already lives in BigQuery can flow directly into AutoML training without export. Choose Vertex AI AutoML when the goal is to validate feasibility, establish a quality baseline, and deploy quickly without building custom model code.

3.3Latency, Serving, and Deployment Constraints

Vertex AI AutoML deployment supports two destinations: serving from Google Cloud as a Vertex AI endpoint and deployment to an external device through AutoML Edge. For text, video, or tabular AutoML models, the application must tolerate inference latencies greater than 100ms, which rules out ultra-low-latency use cases that require custom optimization or specialized serving stacks. This latency boundary separates AutoML cloud serving on one side from strict real-time requirements on the other, where custom training, quantization, or dedicated serving infrastructure may be required. Evaluate serving destination and latency tolerance before committing to AutoML, because a model that trains well but cannot meet the latency budget cannot scale into production.

3.4Trade-Offs Against Custom Training

Vertex AI custom training (opens in a new tab) trades automated convenience for explicit control over code, search space, hardware, and interpretability. Custom training runs user-owned code on serverless Vertex AI (opens in a new tab) infrastructure and uses services such as Vertex AI Vizier for explicit hyperparameter search, so the team controls the search space, training budget, hardware, and stopping criteria across many architecture, data, hyperparameter, and hardware combinations. Custom-trained tabular and image models explicitly support Vertex Explainable AI feature attributions that detail the importance of each input feature, which helps diagnose behavior and build trust when interpretability is required. Choose custom training when the problem falls outside AutoML supported formats and objectives, when cross-platform consistency with on-premises or other-cloud training is required, or when latency, cost, interpretability, and architecture control outweigh development velocity.

3.5Choosing Between BigQuery ML, Vertex AI AutoML, and Custom Training

The three machine learning environments differ by where data lives, what interface the team uses, and how much training control the team retains:

The choice controls downstream cost, complexity, and scalability because BigQuery ML keeps processing and training inside BigQuery, AutoML moves data through managed AutoML preprocessing and search, and custom training moves data through team-owned preprocessing and training code before deployment.

Key Facts

• Vertex AI Custom Training runs user-owned code of any model type on serverless infrastructure and is required when the problem does not match BigQuery ML or AutoML model types, or when consistency with on-premises training is needed.
• Vertex AI Pipelines orchestrates data preparation, Custom Training, evaluation, Model Registry, and serving into an automated workflow that supports scheduled retraining and lineage tracking through Vertex ML Metadata.
• BigQuery ML trains linear regression, binary and multiclass logistic regression, k-means, and imported TensorFlow models with GoogleSQL where the data lives, minimizing movement and batch-inference latency.
• Vertex AI AutoML automates architecture selection, hyperparameter tuning, and provisioning for tabular, image, text, and video problems but requires tolerance for inference latency greater than 100ms for text, video, or tabular models.