Collaborative Data and Model Management for ML

Organizing and preprocessing data for machine learning means placing each data type where distributed training can read it at high speed. BigQuery (opens in a new tab) holds tabular data as materialized tables, while Cloud Storage holds images, video, audio, and text as large binary containers. Source data flows from service logs, on-device images, or web documents into a dedicated training store that is separate from production, so repeated experiments do not compete with production writes.

1.1Tabular Data in BigQuery and Unstructured Data in Cloud Storage

Tabular storage in BigQuery (opens in a new tab) uses materialized tables instead of views or subqueries, so training jobs do not recompute query logic on every read. Large tables are partitioned by date, timestamp, or ingestion time so queries scan only relevant partitions, and data flows from tables to workers through the BigQuery Storage API for high-throughput parallel reads. Training pipelines can reference the data as a managed tabular dataset in Vertex AI to track lineage.

Unstructured storage in Cloud Storage packages many small files into fewer large sequential files for higher throughput. Dataflow can convert raw objects into binary containers such as TFRecord, and buckets and directories group shards for management and parallel discovery. BigLake can present lake data for governed analytics, while BigQuery object tables can reference Cloud Storage objects without copying bytes and signed URLs grant time-limited access.

Unstructured datasets follow three sharding rules: use sharded TFRecord for TensorFlow and Avro for other frameworks, size each shard to at least 100 MB, and keep 100 to 10,000 shards per dataset. When shards are smaller than 100 MB or more numerous than 10,000, workers spend more time opening objects than reading bytes.

1.2Preprocessing at Scale with SQL, Dataflow, and Spark

Scalable preprocessing (opens in a new tab) matches dataset size and complexity to the engine that controls cost. Single-node Python with pandas fits in-memory prototyping on small datasets in a Vertex AI Workbench notebook, while BigQuery SQL transforms enterprise-scale tables where the data already lives. Cloud Dataflow fits distributed batch and streaming with custom logic, and Managed Service for Apache Spark fits existing PySpark and Hadoop code with minimal rewrite.

Cloud Dataflow is the fully managed service for running Apache Beam pipelines. Data flows from sources such as Pub/Sub, Apache Kafka, Cloud Storage, or databases through Beam transforms that run in parallel across Dataflow workers to sinks such as BigQuery. A Beam pipeline is built from a pipeline definition, PCollections that carry distributed data, transforms that apply parallel processing, and a runner that executes the work on Dataflow workers. One pipeline can process historical backfill and live events with identical logic, so training features stay consistent with serving features.

Consistency across training and serving depends on windowing, embedded transforms, and validation. Windowing groups unbounded streaming elements by event time into bounded collections, and alignment logic correlates elements from different streams that arrive at different times. TensorFlow Transform embeds full-pass analysis and per-example transformation into both training and serving graphs to prevent training-serving skew, while TensorFlow Data Validation checks schemas and BigQuery quality scans check staged tables. Managed Service for Apache Spark favors many short-lived ephemeral clusters over one long-lived cluster, with storage in Cloud Storage or BigQuery because destroying the cluster destroys its local HDFS. Movement from Spark to BigQuery uses the Spark-BigQuery connector in fixed order: first Spark writes the DataFrame to temporary Cloud Storage, then the connector triggers a BigQuery Load Job, then BigQuery ingests the data into the target table.

1.3Shared Features, Lineage, and Enterprise Search Features

Vertex AI Feature Store (opens in a new tab) is the central location to create, maintain, share, and serve features so teams reuse the same definitions. Feature data itself lives in a BigQuery table or view that holds history for offline training and point-in-time correctness. To serve online, first create a FeatureOnlineStore instance and then define feature views, so an online prediction request flows from the endpoint to the online store instead of scanning BigQuery. With optimized online serving, the online store also supports vector similarity search for approximate nearest neighbors.

Reproducibility links every model to its exact data, features, code, and parameters. Managed Vertex AI Datasets link preprocessed data to custom-trained models with train, test, and validation splits, Vertex ML Metadata logs sources and transformations, and Vertex AI Pipelines lineage tracks each artifact. The versioning strategy assigns each artifact to its store:

Scalable feature pipelines in Gemini Enterprise connect enterprise sources to apps through governed stores. Data moves in order: first the pipeline connects a source such as Confluence, Jira, SharePoint, or ServiceNow, then ingestion creates a collection of data stores with one store per entity type, then each store changes from Creating to Active. Only an Active store can serve an app, and apps have a many-to-many relationship with stores. When one app connects to many stores, Gemini Enterprise sends the query to all connected stores and blends the results into one comprehensive answer.

1.4Privacy Controls for Sensitive Data and Models

Data privacy for machine learning combines access controls, de-identification, encryption, and perimeters to protect personally identifiable information (PII). Least-privilege Identity and Access Management (IAM) gives every user and service account only minimum permissions, with dedicated service accounts per pipeline stage and table-level IAM for specific tables. BigQuery column-level security (opens in a new tab) hides entire columns with policy tags unless the caller holds both data access and the Fine-Grained Reader role, while row-level security filters rows at query time and authorized views allow querying without direct access to base tables.

The production taxonomy is hierarchical: 3_Confidential for data such as credit card numbers grants 2_Private and 1_Sensitive, 2_Private for sensitive PII such as first name grants 1_Sensitive, and 1_Sensitive for non-public data such as credit limit grants nothing higher. Cloud Sensitive Data Protection discovers PII with a data profiler that scans BigQuery tables for more than 200 infoTypes, then de-identifies with tokenization, pseudonymization, generalization, masking, or encryption. Ingestion-time tokenization flows in order from Cloud Storage or Pub/Sub to a Dataflow pipeline to Sensitive Data Protection and then to BigQuery, with confidential and de-identified data in separate projects. Customer-Managed Encryption Keys (CMEK) must reside in the same location as the protected resource and rotate every 30 days by default, while VPC Service Controls creates a service perimeter where data flows only inside unless explicit ingress and egress rules allow movement.

Collaborative notebook development separates per-user experimentation from governed promotion to production. Traffic flows from the notebook to BigQuery, Cloud Storage, Vertex AI Feature Store, and Vertex AI training and serving, with IAM, Virtual Private Cloud (VPC), and VPC Service Controls (opens in a new tab) checking every hop. Vertex ML Metadata, Git, container images, Cloud Build, and Vertex AI Pipelines connect prototypes to tracked deployment.

2.1Notebook Workspaces, Standards, and Promotion to Pipelines

Notebook options divide by control versus managed collaboration. Vertex AI Workbench (opens in a new tab) provides per-user customizable instances for writing code, starting jobs, running queries, and checking status with custom VPC, IAM, and container images. Colab Enterprise provides a managed collaborative environment that inherits Google Cloud security and compliance controls without managing instances. Team isolation starts with one Workbench instance per data scientist, with separate instances per project when dependencies differ, and instances are stopped when idle to control cost.

Shared standards keep isolated workspaces compatible from notebook to pipeline. Teams use the Vertex AI SDK for Python across PyTorch, TensorFlow, XGBoost, and scikit-learn, so notebook code calls the same dataset, training, and endpoint objects that pipelines will call later. Vertex AI TensorBoard tracks loss and accuracy, visualizes model graphs, and projects embeddings, while Vertex AI Experiments links parameters, metrics, and artifacts through Vertex ML Metadata. Reproducible environments depend on containerized kernels, with separate containers for feature engineering, training, and inference when dependencies differ, and private Artifact Registry storage with vulnerability scanning.

Promotion to production moves tested notebook code into Vertex AI Pipelines through Cloud Build in fixed order:

1. Source commit: A Git commit triggers a Cloud Build job.
2. Build and scan: Cloud Build builds the image, Artifact Analysis scans dependencies, and Binary Authorization enforces policy.
3. Pipeline execution: Vertex AI Pipelines runs versioned components built with the Kubeflow Pipelines or TFX SDK.
4. Evaluation gate: Re-execution performs explainability and responsible AI checks before deployment.
5. Versioned registration: Successful models register as new versions in Model Registry.

2.2Custom Training, Experiment Tracking, and Serving Transition

Vertex AI Custom Training is the managed service that runs modular training code outside the notebook on Google Cloud infrastructure. Training inside the Workbench instance fits small datasets or subsets for rapid iteration, while the managed service fits larger datasets, distributed training, and production retraining on new data arrival. The same managed service is recommended even for small datasets when training runs on a schedule or in response to new data, because the run becomes repeatable. Data flows from BigQuery or Cloud Storage into the training job, and the trained artifact flows from the job to Model Registry.

Vertex AI Experiments logs parameters, metrics, and dataset and model linkages across runs and integrates with Vertex ML Metadata for lineage. Vertex AI TensorBoard visualizes those runs with loss and accuracy curves, model graphs, and embedding projections, while Vertex AI Vizier tests hyperparameter configurations to maximize accuracy. Vertex AI Pipelines then automates the lifecycle where output from one step becomes input to the next: data preparation into BigQuery or Cloud Storage, training and tuning as a custom step, evaluation against test data, and deployment preparation through Model Registry. The production transition flows in order from saving the model locally to staging in Cloud Storage to uploading to Model Registry to creating a Vertex AI Endpoint to deploying with a selected machine type, a long-running operation that can take more than five minutes. Online prediction scales between minimum and maximum nodes, with at least two nodes for high availability.

2.3Identity, Network, Encryption, and Model Garden Prototyping

Notebook identity separates human access from workload access. Every user receives only minimum permissions, while a separate notebook service account is the workload identity that code uses to call Vertex AI and read training data. Teams restrict downloadable service account keys, block key creation with the constraints/iam.disableServiceAccountKeyCreation policy, and add IAM Conditions, principal access boundaries, Privileged Access Manager just-in-time elevation, and Identity-Aware Proxy controls. Network controls place a trust boundary around Vertex AI and data APIs with private IPs, Private Service Connect, deny-all firewall policies, and Shielded VM or Confidential Computing. Encryption uses customer-managed keys in Cloud KMS that must match the resource location and rotate every 30 days by default, while Sensitive Data Protection tokenizes PII in Dataflow before notebooks read it.

Vertex AI Model Garden (opens in a new tab) is the central catalog for discovering and prototyping with foundation models. First-party Google models such as Gemini (opens in a new tab) for text, code, and multimodal work plus Imagen and Veo are served as managed APIs, third-party models such as Claude, Llama, and Mistral are served as managed APIs with publisher terms, and open-source models such as Gemma and Falcon can be deployed to a dedicated endpoint for full control. Prototyping flows from the model card into Workbench or Colab Enterprise (opens in a new tab) where prompts run against candidate APIs. Notebook code authenticates without keys through Agent Identity, where the runtime obtains a short-lived token from the metadata server and the client library attaches it to API calls. Deploying an agent prototype with Agent Identity flows from initializing vertexai.Client to wrapping the agent in AdkApp to calling client.agent_engines.create() with identity type set to Agent Identity. Default Context-Aware Access binds tokens with Demonstrating Proof of Possession and mutual TLS so stolen tokens cannot be replayed elsewhere.

Choosing the experimentation environment is the decision between fast interactive prototyping and managed, reproducible execution with the right accelerator. Interactive work uses Cloud Data Fusion and Dataflow notebooks to test logic quickly, while managed execution uses Dataflow pipelines, Gemini Enterprise Agent Platform Pipelines, or self-managed Kubeflow Pipelines on Google Kubernetes Engine (GKE) to capture runs, lineage, and scale. Framework code flows from the development container through the runtime to CPUs, GPUs, or Cloud TPUs.

3.1Interactive Prototyping and Managed Orchestration Choices

Interactive prototyping with Cloud Data Fusion builds a directed acyclic graph in the user interface by connecting sources to transforms, sinks, error collectors, and actions, and the planner converts that graph into parallel computations executed as an Apache Spark job on Managed Service for Apache Spark. Data flows from source databases through JDBC drivers and plugin transforms to destinations such as BigQuery. Dataflow notebooks complement this by testing Beam patterns such as enrichment, text embeddings, vocabulary handling with MLTransform, side-input model updating, multi-model inference, and generative inference with Gemma before formalizing a pipeline.

Managed tracking captures what ran, what it consumed, and how it performed. Dataflow provides in-system telemetry including total bytes per stage, data locality that avoids materialization between stages, and autoscaling as data moves through stages. Durable artifacts flow into Pub/Sub for event ingestion, Bigtable for low-latency caching with 6 ms reads and writes at the 90th percentile, Cloud Storage for objects, and BigQuery (opens in a new tab) for warehousing. Managed orchestration with Agent Platform Pipelines triggers serverless runs from a pipeline definition without operating the control plane, while self-managed Kubeflow on GKE runs each step as containers scheduled by Kubernetes where the team controls VPC, GPU or TPU selection, and storage tuning.

Matching frameworks to processors decides whether an experiment is limited by flexibility or matrix throughput. Cloud TPUs are Google ASICs for large matrix operations with high-bandwidth memory and slices that scale with little code change, but code must be compiled by the XLA compiler that tiles multiplies for the 128x128 systolic array matrix unit and prefers dimensions that are multiples of 8.

3.2Evaluation Pipelines, Quality Metrics, and LLM-as-a-Judge

Standardized evaluation measures model quality on fixed data plus continuous watching in production. Evaluation starts by separating training loss from business success, such as training a fraud model with cross-entropy loss but evaluating on recall. Agent Platform Pipelines (opens in a new tab), built on Kubeflow Pipelines, packages evaluation as a versioned component that loads a model version and fixed dataset, computes metrics, checks thresholds, and logs results to ML Metadata, Experiments, and Model Registry. Model Monitoring then compares live distributions to a baseline from training data analyzed with TensorFlow Data Validation or past production traffic.

Predictive quality uses scores such as F1 that balance precision against recall, while generative retrieval-augmented generation (RAG) quality judges factual accuracy grounded in retrieved data and relevance to the request. An automated quality subsystem flows in order from a Pub/Sub trigger to a Cloud Run function to configuration in AlloyDB for PostgreSQL to fetching prompts to scoring responses to loading scores into BigQuery for offline comparison. LLM-as-a-judge uses a high-quality model such as Gemini to score coherence, relevance, and safety where string-overlap metrics such as BLEU or ROUGE fail. AutoSxS compares Model A against Model B with an autorater that emits choice, confidence from 0 to 1, and explanation, then aggregates win rates. When human labels exist, accuracy, precision, recall, F1, and Cohen's Kappa measure agreement, where Kappa corrects for chance from -1.0 to 1.0: at or below 0.0 is chance, 0.0 to 0.2 slight, 0.2 to 0.4 fair, 0.4 to 0.6 moderate, 0.6 to 0.8 substantial, 0.8 to 1.0 nearly perfect, and 1.0 perfect. A high win rate with low Kappa means the autorater prompt must be revised.

3.3Artifact Versioning, Lineage Graphs, and Run Comparison

Tracking lineage (opens in a new tab) assigns each artifact type to the store that provides storage, versioning, and access control. Code versions flow from Git into builds, data versions flow from storage into training, and both converge in ML Metadata and Model Registry where pointers, parameters, and binaries are linked. For unstructured Cloud Storage data, combine clips into large containers with at least 100 MB per file and 100 to 10,000 shards to improve throughput.

Registration follows an ordered lifecycle: first training writes model files to Cloud Storage and images to Artifact Registry, then the pipeline registers the version with semantic versioning, then registration links parameters, metrics, dataset, and code in ML Metadata, then CI/CD pulls versioned artifacts for testing and deployment. A lineage graph traces a model backward to its dataset and pipeline and forward to its predictions. A backward walk starts from a bad prediction to the serving version in Model Registry to the training run in ML Metadata to the dataset version and raw source to check drift, while a forward walk starts from a changed dataset to downstream runs and dependent versions to decide retraining. Agent Platform ML Metadata tracks the full MLOps pipeline, Knowledge Catalog tracks individual pipeline artifacts for debugging, Model Registry manages model versions, and Cloud Audit Logs record who accessed or modified data and when. Vertex AI TensorBoard compares visual training behavior with loss curves and embeddings, while Experiments compares linked runs with parameters, metrics, and artifacts, and Vertex AI Vizier automates hyperparameter trials. Scaling uses Agent Platform endpoints with at least two replicas for high availability, Dataflow worker autoscaling, GKE autoscaling, and Cloud Run (opens in a new tab) scaling to zero, with Spot VMs for fault-tolerant training that checkpoints.

Key Facts

• BigQuery stores structured training data as materialized and partitioned tables read through the BigQuery Storage API, while Cloud Storage stores unstructured data in sharded TFRecord files for TensorFlow and Avro files for other frameworks sized to at least 100 MB with 100 to 10,000 shards per dataset.
• Vertex AI Feature Store keeps feature history in BigQuery and serves current values from a FeatureOnlineStore instance through feature views for low-latency online prediction.
• Training inside a Workbench instance fits small datasets, while managed Custom Training is required for larger datasets, distributed training, scheduled runs, or retraining on new data arrival.
• BigQuery column-level security uses a hierarchy where 3_Confidential grants 2_Private and 1_Sensitive, while column-level encryption requires permission to both the table and the encryption key, and CMEK keys must match the resource location and rotate every 30 days by default.
• Model Registry versions models and links each version to training, evaluation, dataset, and code metadata, while ML Metadata tracks the full MLOps pipeline, Knowledge Catalog tracks individual pipeline artifacts, and skew detection compares production to training data while drift detection compares production over time.