A machine learning engineering team at a financial institution needs to build a scalable and auditable machine learning workflow on Google Cloud. The architecture must satisfy several key operational requirements: Standardize the end-to-end lifecycle—from data ingestion and preprocessing to model training, evaluation, and endpoint deployment—using modular, reusable components. Automatically track execution history, hyperparameters, and full artifact lineage (datasets, metrics, and model binaries) to ensure end-to-end reproducibility and compliance auditability. Minimize infrastructure management overhead by orchestrating the workflow using a serverless execution platform. Which architectural approach should the team implement?

Define modular pipeline steps using the Kubeflow Pipelines (KFP) SDK and prebuilt Google Cloud Pipeline Components, compile the workflow definition, and execute it using Vertex AI Pipelines to automatically capture lineage in Vertex ML Metadata.: What Vertex AI Pipelines and KFP are Vertex AI Pipelines is a serverless orchestration service in Google Cloud that automates, monitors, and governs end-to-end machine learning workflows. It supports domain-specific language (DSL) frameworks such as the Kubeflow Pipelines (KFP) SDK and TensorFlow Extended (TFX). Developers can construct modular components as standalone containerized steps or lightweight Python functions, which are compiled into a unified pipeline specification. How it meets the requirements Modular, reusable components: The Kubeflow Pipelines SDK and the Google Cloud Pipeline Components library provide standardized, modular factory functions for common ML operations—such as dataset creation, custom or AutoML training, model registry uploads, and endpoint deployments. Artifact lineage and auditability: When running on Vertex AI Pipelines, the platform automatically captures all component inputs, outputs, execution states, parameters, and produced artifacts in Vertex ML Metadata without requiring custom tracking scripts. Serverless orchestration: Vertex AI Pipelines executes workflow DAGs in a fully managed, serverless environment, eliminating the operational overhead of provisioning, scaling, or patching underlying compute clusters. Key advantages and capabilities Automatic lineage tracking: Captures complete directed acyclic graph (DAG) execution history and artifact provenance for regulatory auditability. Component reusability: Modular components can be shared across multiple team pipelines, accelerating development lifecycles. Native Google Cloud integration: Direct integration with services like Cloud Storage, BigQuery, Vertex AI Model Registry, and Vertex AI Endpoints. Serverless efficiency: Resources are provisioned dynamically for individual pipeline steps and released immediately upon task completion. Why this is optimal This approach directly satisfies all architectural goals by combining the modular authoring model of the KFP SDK with the managed execution and governance features of Vertex AI Pipelines and Vertex ML Metadata.

StudyCerts

Professional Cloud Architect

Intrigued by the art of cloud architecture? Discover how to design, develop, and manage robust, secure, scalable, and dynamic solutions on Google Cloud as you prepare for the Professional Cloud Architect exam!

Intermediate

Prepare and test your skills

Track Readiness

Set Exam Goal

Intermediate

Prepare and test your skills

Track Readiness

Set Exam Goal

Vertex Ai Pipelines Kfp Ml Metadata

Section: Using Vertex AI pipelines to automate and orchestrate the ML lifecycle

A machine learning engineering team at a financial institution needs to build a scalable and auditable machine learning workflow on Google Cloud. The architecture must satisfy several key operational requirements:

Standardize the end-to-end lifecycle—from data ingestion and preprocessing to model training, evaluation, and endpoint deployment—using modular, reusable components.
Automatically track execution history, hyperparameters, and full artifact lineage (datasets, metrics, and model binaries) to ensure end-to-end reproducibility and compliance auditability.
Minimize infrastructure management overhead by orchestrating the workflow using a serverless execution platform.

Which architectural approach should the team implement?

Package each pipeline step as an independent Cloud Run microservice, orchestrate step execution with Cloud Composer, and write pipeline parameters and artifact paths directly to BigQuery tables.

Define modular pipeline steps using the Kubeflow Pipelines (KFP) SDK and prebuilt Google Cloud Pipeline Components, compile the workflow definition, and execute it using Vertex AI Pipelines to automatically capture lineage in Vertex ML Metadata.

Execute individual training and deployment tasks on Compute Engine virtual machines managed by cron jobs, and register model versions and metadata in Dataproc Metastore.

Deploy a self-managed Kubeflow cluster on Google Kubernetes Engine (GKE), deploy pipelines via the Kubeflow UI, and maintain an external Cloud SQL database for intermediate container state logging.

Professional Cloud Architect

Intermediate

Prepare and test your skills

Track Readiness

Set Exam Goal

Intermediate

Prepare and test your skills

Track Readiness

Set Exam Goal

Vertex Ai Pipelines Kfp Ml Metadata

Section: Using Vertex AI pipelines to automate and orchestrate the ML lifecycle

Standardize the end-to-end lifecycle—from data ingestion and preprocessing to model training, evaluation, and endpoint deployment—using modular, reusable components.
Automatically track execution history, hyperparameters, and full artifact lineage (datasets, metrics, and model binaries) to ensure end-to-end reproducibility and compliance auditability.
Minimize infrastructure management overhead by orchestrating the workflow using a serverless execution platform.

Which architectural approach should the team implement?

Package each pipeline step as an independent Cloud Run microservice, orchestrate step execution with Cloud Composer, and write pipeline parameters and artifact paths directly to BigQuery tables.

Execute individual training and deployment tasks on Compute Engine virtual machines managed by cron jobs, and register model versions and metadata in Dataproc Metastore.

Deploy a self-managed Kubeflow cluster on Google Kubernetes Engine (GKE), deploy pipelines via the Kubeflow UI, and maintain an external Cloud SQL database for intermediate container state logging.

Explanations

What Vertex AI Pipelines and KFP are

Vertex AI Pipelines is a serverless orchestration service in Google Cloud that automates, monitors, and governs end-to-end machine learning workflows. It supports domain-specific language (DSL) frameworks such as the Kubeflow Pipelines (KFP) SDK and TensorFlow Extended (TFX). Developers can construct modular components as standalone containerized steps or lightweight Python functions, which are compiled into a unified pipeline specification.

How it meets the requirements

Modular, reusable components: The Kubeflow Pipelines SDK and the Google Cloud Pipeline Components library provide standardized, modular factory functions for common ML operations—such as dataset creation, custom or AutoML training, model registry uploads, and endpoint deployments.
Artifact lineage and auditability: When running on Vertex AI Pipelines, the platform automatically captures all component inputs, outputs, execution states, parameters, and produced artifacts in Vertex ML Metadata without requiring custom tracking scripts.
Serverless orchestration: Vertex AI Pipelines executes workflow DAGs in a fully managed, serverless environment, eliminating the operational overhead of provisioning, scaling, or patching underlying compute clusters.

Key advantages and capabilities

Automatic lineage tracking: Captures complete directed acyclic graph (DAG) execution history and artifact provenance for regulatory auditability.
Component reusability: Modular components can be shared across multiple team pipelines, accelerating development lifecycles.
Native Google Cloud integration: Direct integration with services like Cloud Storage, BigQuery, Vertex AI Model Registry, and Vertex AI Endpoints.
Serverless efficiency: Resources are provisioned dynamically for individual pipeline steps and released immediately upon task completion.

Why this is optimal

This approach directly satisfies all architectural goals by combining the modular authoring model of the KFP SDK with the managed execution and governance features of Vertex AI Pipelines and Vertex ML Metadata.

View source documentation

Professional Cloud Architect

Vertex Ai Pipelines Kfp Ml Metadata

Professional Cloud Architect

Vertex Ai Pipelines Kfp Ml Metadata

What Vertex AI Pipelines and KFP are

How it meets the requirements

Key advantages and capabilities

Why this is optimal

Related questions