Tracking ML Experiments and Model Lineage

Choosing the development and experimentation environment is the decision between fast interactive prototyping and managed, reproducible execution with the right accelerator. Interactive work uses Cloud Data Fusion and Dataflow notebooks to test logic quickly, while managed execution uses Dataflow (opens in a new tab) pipelines, Gemini Enterprise Agent Platform Pipelines, or self-managed Kubeflow Pipelines on Google Kubernetes Engine (GKE) to capture runs, lineage, and scale. Framework code flows from the development container through the runtime to central processing units (CPUs), graphics processing units (GPUs), or Cloud tensor processing units (TPUs), so container, machine type, and accelerator choice control performance and reproducibility.

1.1Interactive Prototyping with Graphical Pipelines and Notebooks

Interactive prototyping with graphical pipelines and notebooks is code-free pipeline building paired with code-level testing of dataflow patterns. Interactive prototyping in Cloud Data Fusion builds a directed acyclic graph (DAG) in the user interface by connecting sources to transforms, sinks, error collectors, and actions, and the planner converts that DAG into parallel computations executed as an Apache Spark job on Managed Service for Apache Spark. Data flows from source databases through Java Database Connectivity (JDBC) drivers and plugin transforms to destinations such as BigQuery, so the same graph defines both logic and parallel execution.

Dataflow notebooks (opens in a new tab) complement the user interface by testing Apache Beam patterns before formalizing a pipeline. Representative patterns include enrichment with the Beam enrichment transform and Vertex AI Feature Store, text embeddings with the Vertex AI text-embeddings API or Hugging Face SentenceTransformers, vocabulary and scaling with MLTransform, model updating with side inputs, multi-model inference with KeyedModelHandler, remote inference with the Vision API or custom spaCy models, and generative inference with Gemma 2B, Hugging Face large language models (LLMs), and ensemble or cascade models. Choose graphical prototyping when sources, transforms, and sinks are known and standard, and choose notebook prototyping when vocabulary handling, scaling, per-key logic, or generative inference must be validated first.

1.2Managed Tracking of Pipeline Runs and Serving Telemetry

Managed tracking of pipeline runs and serving telemetry is the capture of what ran, what it consumed, and how it performed outside the notebook cell. Dataflow provides in-system telemetry for the pipeline itself, including total bytes per stage, data locality that avoids materialization between stages, and autoscaling where resources scale as data moves through stages. Because one system owns all stages, Dataflow removes the need for an external orchestrator to coordinate multiple pipelines.

Durable artifacts and serving signals flow into surrounding managed services. Dataflow integrates with Pub/Sub for event ingestion, Bigtable for low-latency wide-column caching and serving, Cloud Storage for object storage, and BigQuery for petabyte-scale warehousing. Bigtable supports 6 ms reads and writes at the 90th percentile with many thousands of concurrent clients, which makes it usable as both source and sink inside a DoFn. For inference, GKE Inference Gateway reports GPU or TPU utilization, key-value (KV) cache hits and utilization, queue length, and model latency, while GKE Gateway with custom metrics relies on application-reported metrics viewed in Cloud Monitoring.

1.3Managed Orchestration Versus Self-Managed Kubeflow on GKE

Managed orchestration versus self-managed orchestration is the tradeoff between serverless governance and direct control of cluster, network, and accelerators. Managed orchestration with Gemini Enterprise Agent Platform Pipelines triggers serverless runs from a pipeline definition, where data flows from training and evaluation steps into governed artifacts and then into deployment and monitoring without operating the orchestrator control plane. The pipeline depends on surrounding services for lineage, registration, and oversight:

• Gemini Enterprise Agent Platform Managed Training runs custom PyTorch, TensorFlow, XGBoost, and scikit-learn training.
• Vertex ML Metadata records artifact lineage and organization.
• Model Registry on Gemini Enterprise Agent Platform registers versioned models.
• Feature Store, evaluation, inference with Vector Search, Model Monitoring, Explainable AI, and Ray supply features, scoring, serving, and observability.

Self-managed orchestration on GKE runs each pipeline step as containers scheduled by Kubernetes. The GKE cluster provides the control plane, nodes, and system components, the Kubernetes scheduler assigns workloads and manages lifecycle and upgrades, and the kubelet on each node starts the containers. Traffic flows from users through Cloud Load Balancing to serving containers, data flows from Cloud Storage through Cloud Storage FUSE volumes into training containers, and control signals flow from the control plane to each kubelet. The team retains control of the Virtual Private Cloud (VPC), GPU or TPU selection, and storage tuning with Cloud Storage Rapid Bucket, Rapid Cache, and Managed Lustre, but also retains overhead for images, networking, drivers, and volumes.

Storage choice follows the artifact bottleneck:

1.4Matching Frameworks to CPUs, GPUs, and Cloud TPUs

Matching frameworks to processors decides whether an experiment is limited by flexibility, programmability, or matrix throughput. Cloud TPUs are Google application-specific integrated circuits (ASICs) for large matrix operations, with high-bandwidth memory (HBM) for larger models and TPU slices that scale with little code change. Code for TPUs must be compiled by the accelerated linear algebra (XLA) compiler, which compiles the framework graph into TPU machine code while the rest runs on the TPU host, so container and host images must expose compatible XLA and framework versions. The XLA compiler tiles multiplies for the 128x128 systolic array matrix unit (MXU) and prefers dimensions that are multiples of 8.

Framework containers make the choice reproducible. Agent Platform custom training accepts pre-built TensorFlow, PyTorch, and JavaScript Extension (JAX) containers and custom containers for extra libraries, custom C++ ops, or Hugging Face stacks such as Optimum TPU, JetStream, or vLLM. Optimum TPU serves Hugging Face models on GKE with continuous batching, token streaming, and greedy and sampling decoding. Choose JetStream for latest Google LLM optimizations, vLLM or quantized Ollama on Cloud Run for portable GPU inference, and Optimum TPU to stay in the transformers ecosystem on TPUs.

1.5Scaling Development, Training, and Serving

Scaling development, training, and serving connects interactive notebooks to distributed jobs and autoscaled endpoints. Vertex AI Workbench provides per-member notebooks for prototyping and small-dataset training, Experiments on Gemini Enterprise Agent Platform logs parameters, code and dataset versions, and results, and custom training moves code to a managed job where the worker-pool specification defines machine type, accelerator type and count, and pool count. Managed training depends on Cloud Storage for datasets, checkpoints, and artifacts and on Model Registry for versioned outputs, while Terraform, Cluster Toolkit, and Config Connector define GKE node pools, endpoints, and buckets declaratively. Spot virtual machines (VMs) reduce cost when training code checkpoints and tolerates preemption.

Autoscaling and availability differ by plane. Agent Platform endpoints scale replicas between a minimum and maximum on traffic and CPU or replica utilization, with at least two replicas for high availability (HA). Dataflow scales workers on CPU, parallelism, and pending data, GKE cluster autoscaler scales nodes on pending CPU, memory, GPU, and TPU requests, and Horizontal Pod Autoscaler scales deployments on utilization or custom AI metrics. Cloud Run scales containers to zero when idle and supports GPU instances for agents and vLLM or Ollama workloads, while global endpoints route across regions for low latency. Hardware faults on GKE or Compute Engine are handled by monitoring Xid errors, resetting GPUs or instances, reporting faulty hosts, and using elastic or Pathways resilient training.

Evaluating predictive and generative AI solutions is the repeatable measurement of model quality on fixed data plus continuous watching of quality in production. Predictive tasks compare predictions to labeled ground truth with numeric scores, while generative tasks judge free-form text for usefulness, truthfulness, and safety. In Google Cloud, Agent Platform Pipelines (opens in a new tab) orchestrates evaluation steps, Model Monitoring on Agent Platform tracks drift and skew, and Cloud Monitoring fires alerts that flow back to retraining or rollback.

2.1Standardized Evaluation Pipelines and Production Monitoring

Standardized evaluation pipelines and production monitoring create a closed loop from datasets to pipeline components to dashboards to remediation. Standardized evaluation starts by separating training loss from business success: a fraud model may train with cross-entropy loss but be evaluated on recall because recall measures coverage of fraud. Computation-based metrics are calculated by formulas, while qualitative indicators such as fluency require LLM-as-a-judge or human labeling (opens in a new tab). Evaluation data must be large and varied, using a golden dataset that reflects production, a judge LLM where an LLM can judge the task, and user feedback from production traffic. Data is stored in BigQuery for SQL-based tasks or Cloud Storage for large cost-efficient datasets.

Agent Platform Pipelines, built on Kubeflow Pipelines, packages evaluation as a versioned component that loads a model version and fixed dataset, computes metrics, checks thresholds such as minimum accuracy or F1-score, and logs parameters, code, dataset versions, and results to Agent Platform ML Metadata, Experiments, and Model Registry. Building the process follows an ordered sequence:

1. Define indicators: select computation-based metrics for predictive quality and LLM-as-a-judge or human review for generative quality, safety, grounding, and style.
2. Choose data and techniques: select golden datasets, judge models, and user feedback hosted in BigQuery or Cloud Storage.
3. Standardize the pipeline: implement evaluation as a versioned component with fixed datasets, thresholds, and metadata logging.

Model Monitoring on Agent Platform then compares live input and output distributions to a baseline from training data analyzed with TensorFlow Data Validation (TFDV) or past production traffic. Version 2 monitors managed endpoints and self-hosted GKE serving at model level, tracks input feature drift, output drift, and feature-attribution change, and applies to structured numerical and categorical features rather than unstructured images. Feature attribution from Vertex Explainable AI is an early signal when the model relies on different features than in training.

The Gen AI evaluation service extends monitoring to response quality, safety, instruction adherence, grounding, style, verbosity, toxicity, and user ratings, with grounding checks against source text and human-in-the-loop review. Model Armor screens outputs for unsafe or off-topic content. Serving health is tracked separately through time to first token (TTFT), queries per second, token throughput, latency, error rate, and service-level objectives (SLOs) such as 99.9% successful calls, 95th percentile latency below 300 ms, TTFT below 500 ms for 99% of requests, and harmful-output rate below 0.1%. Alerts flow from Cloud Monitoring through Pub/Sub to Slack, email, SMS, or PagerDuty, triggering retraining pipelines or rollback to the last stable registry version.

2.2Predictive and Generative Quality Metrics for RAG

Predictive and generative quality metrics for retrieval-augmented generation (RAG) match the metric to the prediction type and business risk. The F1 Score balances precision against recall in one score for predictive output, so a model cannot score well by favoring missed positives over false positives. Information extraction needs this balance because both finding all relevant facts and avoiding incorrect facts affect quality. Factual accuracy and relevance judge generative RAG responses from the serving subsystem: factual accuracy measures grounding in retrieved domain data such as electronic health records, PubMed, statutes, or product reviews, while relevance measures whether the response answers the request in context.

An automated quality subsystem decouples scoring from serving so evaluation runs continuously. Evaluation data flows from storage through compute to analysis in order:

1. Trigger: Pub/Sub triggers a Cloud Run functions instance on an evaluation request.
2. Configure: The function reads configuration from an AlloyDB for PostgreSQL database.
3. Fetch prompts: The function pulls evaluation prompts previously loaded by ingestion.
4. Score responses: The function scores serving-subsystem responses for factual accuracy and relevance.
5. Store for analysis: The function loads scores, prompts, and responses to BigQuery.

BigQuery depends on function output for offline comparison, which lets teams track quality over time and justify promotion.

2.3LLM-as-a-Judge with AutoSxS and Human Alignment

LLM-as-a-judge with AutoSxS and human alignment is automated side-by-side scoring validated against people. LLM-as-a-judge uses a high-quality foundation model such as Gemini to score candidate outputs for coherence, relevance, and safety, which solves the problem where prompts and outputs are too complex for string-overlap metrics such as Bilingual Evaluation Understudy (BLEU) or Recall-Oriented Understudy for Gisting Evaluation (ROUGE). AutoSxS is the Google Cloud implementation that compares Model A against Model B with an autorater LLM and benchmarks the autorater against systematic human evaluation.

Custom evaluation datasets are required because a foundation model's pre-training distribution is unknown and generic. The dataset curates essential, average, and edge prompts plus RAG context, versioned prompt templates, rewritten queries, and vector-database embeddings and chunking. LLMs can generate and augment rare cases, but humans must check synthetic data. Manual review works for prototyping but becomes a bottleneck, so automation improves speed and reproducibility while human preferences remain ground truth.

An AutoSxS run flows in order:

1. Inference: Prompts flow to Model A and Model B, including tuned models from Vertex Model Registry, which each produce a response.
2. Arbitration: Both responses plus instruction and context flow to the arbiter, which emits a choice, confidence score, and explanation.
3. Aggregation: Judgments flow to the metrics component, which computes win rates and, when human labels exist, alignment metrics.
4. Inspection: Artifacts flow to Agent Platform Pipelines and to Cloud Storage JSONL or BigQuery tables for row-level review.

Each judgment row preserves inference_instruction, inference_context, response_a, response_b, choice of Model A, Model B, or Error, confidence from 0 to 1, and explanation. Aggregate win rates report the percentage of examples where the autorater chose each model, but wins alone do not prove correctness. When human preferences exist, accuracy, precision, recall, F1, and Cohen's Kappa measure agreement, where Kappa corrects for chance from -1.0 to 1.0: at or below 0.0 is chance, 0.0 to 0.2 slight, 0.2 to 0.4 fair, 0.4 to 0.6 moderate, 0.6 to 0.8 substantial, 0.8 to 1.0 nearly perfect, and 1.0 perfect. A high win rate with low accuracy or Kappa means the autorater prompt or model must be revised.

Tracking and comparing model artifacts, versions, and lineage is the system that logs, stores, versions, and links every machine learning artifact so any model can be reproduced and audited. Agent Platform ML Metadata records lineage and relationships, Gemini Enterprise Agent Platform Experiments (opens in a new tab) logs parameters, metrics, and dataset and model linkages per run, and Model Registry applies immutable versions. Together they provide traceability from raw data to prediction.

3.1Artifact Versioning by Store and Model Registry Lifecycle

Artifact versioning by store assigns each artifact type to the store that provides storage, versioning, and access control. Code versions flow from Git into builds, data versions flow from storage into training, and both converge in ML Metadata and Model Registry where pointers, parameters, and binaries are linked. Metadata and pointers are stored in version control or a data catalog so locations resolve without copying data.

For unstructured Cloud Storage data, combine clips into large container files with a target of at least 100 MB per file and 100 to 10,000 shards to improve throughput. Managed datasets strengthen lineage by linking custom models directly to training data, providing statistics and train, test, and validation splits, with BigQuery Storage API reads supporting tabular tracking.

Model Registry preserves provenance for each version, including training parameters and code version, evaluation metrics, dataset version traced to the Knowledge Catalog, framework and fine-tuning parameters, and generative quality and safety results. Registration follows an ordered lifecycle:

1. Store binaries: training writes model files to Cloud Storage and images to Artifact Registry.
2. Register version: the pipeline registers the version with semantic versioning through the SDK or API.
3. Attach lineage: registration links parameters, metrics, dataset, and code in ML Metadata.
4. Control promotion: CI/CD pulls versioned artifacts for testing and deployment under registry access policies.

Immutability and auditability protect the chain. Encrypt artifacts with Cloud Key Management Service (Cloud KMS), enforce least-privilege Identity and Access Management (IAM) with dedicated pipeline service accounts and IAM Conditions, sign approved builds, and use multi-region or dual-region buckets for durability. Audit logs for Agent Platform record endpoint creation, deployment, and configuration changes, with analysis in Observability Analytics, real-time analysis in Cloud Logging, long-term retention in BigQuery, and routing to security information and event management (SIEM) systems.

3.2Lineage Graphs for Governance and Debugging

A lineage graph is a connected record that traces a model backward to its dataset and pipeline and forward to its predictions. Agent Platform ML Metadata stores graph-based lineage for models, datasets, and artifacts encrypted at rest, including data sources, transformations, parameters, and experiment results, plus resources and runs from Agent Platform Pipelines. A processed dataset points back to its raw source and creating pipeline, a training run points back to dataset and code versions, and a registered version points back to its run and forward to its endpoint.

Complete lineage requires automatic registration during execution rather than manual entry. Model files and weights land in Cloud Storage, custom images land in Artifact Registry, datasets are tagged with ownership, sensitivity, and source links, and Knowledge Catalog discovers inventories from BigQuery, Cloud Storage, and Pub/Sub while tracking pipeline-artifact lineage. Track an individual artifact in Knowledge Catalog when debugging a single step, and track the entire MLOps pipeline in ML Metadata when auditing or reproducing the full lifecycle.

Interpreting lineage means traversing in two directions. A backward walk starts from a bad prediction, identifies the serving version in Model Registry, follows the edge to its training and evaluation run in ML Metadata, then follows the edge to dataset version and raw source in Knowledge Catalog to check drift or upstream change. A forward walk starts from a changed dataset, finds downstream runs that consumed it, then lists dependent registered versions to decide retraining or rollback. Audit coverage depends on Cloud Audit Logs to record who accessed or modified data and models and when.

3.3Comparing Experiment Runs Across Teams and Platforms

Comparing experiment runs across teams is the practice of building many variants and contrasting metrics, hyperparameters, and artifacts to find the optimal model. Data scientists vary architectures, input datasets, hyperparameters, and hardware, then compare aggregate metrics such as accuracy, precision, and recall overall and on important data subsets, versions, and architectures.

Vertex AI TensorBoard provides visual, collaborative comparison of training behavior over time. Data flows from training jobs into TensorBoard, where loss and accuracy curves, model graphs, and low-dimensional embedding projections from different runs appear side by side. Teams compare hyperparameter choices and training dynamics directly without exchanging isolated charts. Choose TensorBoard when the question is how training evolved and how visual differences explain the gap.

Experiments on Agent Platform provides linked tracking of parameters, metrics, and artifacts through Vertex ML Metadata. Each run records hyperparameters, metrics, dataset pointers, and model outputs, so different accuracy numbers can be traced to parameters, data, or code rather than undocumented change. Choose Experiments when provenance and reproducibility across Agent Platform and Gemini Enterprise Agent Platform matter.

Hyperparameter-driven comparison automates the search. The Gemini Enterprise Agent Platform Managed Training service runs jobs on Google processing infrastructure while Vertex AI Vizier selects and tests configurations across trials. Each trial flows back into Experiments and TensorBoard as a comparable run, which removes manual tuning and lets teams promote only the configuration with verified lineage.

Key Facts

• Cloud Data Fusion builds a directed acyclic graph in the user interface that executes as an Apache Spark job, while Dataflow provides single-system telemetry, data locality, and autoscaling without external orchestration.
• Cloud TPUs require XLA compilation of the main training loop, so custom operations inside that loop disqualify a workload even if the model is large.
• Agent Platform Pipelines standardizes evaluation as a versioned component that logs to ML Metadata and Model Registry, with skew detection when training data is available and drift detection when only production data over time is available.
• AutoSxS compares Model A against Model B with an autorater LLM, and autorater validity requires agreement with human preferences measured by accuracy and Cohen's Kappa.
• Model Registry versions models and links each version to training, evaluation, dataset, and code metadata, while ML Metadata tracks the full MLOps pipeline and Knowledge Catalog tracks individual pipeline artifacts.