Prepare and test your skills
Prepare and test your skills
Data Ingestion and Preparation Foundations are the first stages of the machine learning lifecycle that turn scattered raw data into clean training sets. Data ingestion moves data from source systems into Google Cloud, while data preparation cleans, transforms, and validates that data for training. In Google Cloud, Cloud Storage (opens in a new tab) stages raw and unstructured files, BigQuery (opens in a new tab) holds cleaned structured tables, Knowledge Catalog (opens in a new tab) makes distributed data searchable, and Dataflow (opens in a new tab) transforms data between storage and analytics.
Centralized storage gives machine learning teams one trusted place to find and reuse data instead of copying files between isolated systems. Cloud Storage holds unstructured data such as images, files, raw staged data, and shared pipeline templates. BigQuery is a fully managed and highly scalable platform for structured tables, large-scale analysis, and machine learning, including building models directly with SQL. Knowledge Catalog unifies distributed data without forcing everything into one place by organizing logical lakes and zones, automatically ingesting metadata, and making assets searchable.
| Central store | Type of data held | Role in the ML lifecycle |
|---|---|---|
| Cloud Storage | Unstructured data, raw staged files, and shared pipeline templates | Landing zone for ingestion and flexible file storage governed through BigQuery object tables |
| BigQuery | Structured, curated tables and listings | Central analytics platform where validated data is queried, shared, and used directly for training |
Data flows from source systems into Cloud Storage buckets for staging, and then from Cloud Storage into BigQuery tables for analysis and training. Because Knowledge Catalog tracks metadata from supported Google Cloud sources, data scientists can discover assets in both locations through search instead of managing separate inventories.
Data ingestion is the process of bringing outside data into Google Cloud on a schedule or continuously as events occur. Batch ingestion copies larger, bounded sets of data at intervals, while streaming ingestion captures continuous event streams for near-instant access. The choice depends on how fresh the training or analytics data must be, with scheduled transfers for periodic updates and streaming pipelines for real-time needs.
Common Google Cloud ingestion paths include:
After the initial load, teams can reroute upstream feeds directly to BigQuery to reduce dependence on an older warehouse. Streaming setups often pair Pub/Sub or Kafka as the entry point with Dataflow templates that load continuously from the stream into BigQuery.
Data preparation converts raw staged data into clean, enriched, and correctly shaped training sets through extract, transform, and load (ETL) work. Dataflow defines and runs complex transformation and enrichment graphs using the Apache Beam model. The service is serverless and fully managed, so processing capacity scales automatically as data volume changes.
A typical preparation flow moves data through three ordered stages:
A left-to-right process flow showing data moving from source systems through ingestion tools into Cloud Storage for raw staging, through a Dataflow pipeline that cleans and transforms records, into BigQuery as curated structured tables, with validation via count, sum, and average checks and Knowledge Catalog tracking metadata across all assets.
This sequence separates durable raw storage from repeatable transformation logic, so the same pipeline can run again as new data arrives. For heavy batch work, Managed Service for Apache Spark provides managed Apache Spark and Hadoop processing as an alternative compute option.
Validation and reuse ensure that prepared data stays accurate and can serve many models without repeated engineering work. Validation compares source data to the target in BigQuery using aggregations such as count, sum, and average on selected columns. When validation passes, teams can confidently use BigQuery features for analysis and training. Reusable assets reduce duplication because prepared datasets, features, and models are published once and discovered by other teams through Knowledge Catalog search, BigQuery sharing with data exchanges and listings, and centralized dataset management.
Exam tip: Use Cloud Storage to stage raw and unstructured files, and use BigQuery to hold the cleaned, structured tables that models train on — Dataflow moves and transforms data from the first to the second.
Model Training and Deployment Execution is the stage of the machine learning lifecycle where a model is built or tuned and then connected to computing resources to serve predictions. Vertex AI (opens in a new tab) is the central machine learning platform that lets teams train and deploy models and AI applications. In a secure on-premises context, Vertex AI on Distributed Cloud (opens in a new tab) runs services inside an organization's data center for data sovereignty and compliance.
Training in Vertex AI follows two broad paths: a more automated path using pre-built models and algorithms, and a custom model training path using your own data. Pre-built models provide ready-to-use models for common machine learning and AI tasks, which accelerates time to value and avoids building from scratch. Custom model training tailors a model to specific needs and gives more control over behavior for a specialized problem.
For specialized tasks, model tuning can improve performance beyond what prompt design alone achieves. A tuning workflow follows an ordered process:
The size of the dataset and the tuning method depend on the model and what is being optimized, and niche tasks may need only a smaller dataset for significant improvement. Vertex AI Workbench provides a managed JupyterLab notebook environment for development, while Cloud TPU (opens in a new tab) and GPUs on Compute Engine can accelerate training workloads.
Deploying a model associates the model with an endpoint and physical machine resources so it can serve online, low-latency predictions. Not all models require deployment because Google's foundation models in generative AI on Vertex AI already have endpoints that are specific to a Google Cloud project and immediately available. However, if any of those foundation models are tuned, the tuned version must be deployed to an endpoint before use.
When a model is deployed, teams choose between a fully managed environment and a self-managed environment. The two options differ in who manages resources and how much control the team keeps:
| Deployment environment | Resource management | Control |
|---|---|---|
| Fully managed | Vertex AI instantiates and manages the physical resources selected, such as machine type and accelerator type | Less fine-grained control, but simpler operation |
| Self-managed | The team manages resources on its own, for example by serving on Google Kubernetes Engine (GKE) | More fine-grained control over resources |
After choosing the environment, teams must balance anticipated traffic, latency requirements, and budget against physical resources. If lower cost is a priority, a team might tolerate higher latency with lower-cost machines, and test environments are a good example of this tradeoff. For serving, Online Prediction is used for online, low-latency predictions, while batch prediction is listed as a deployment option in the generative AI development flow.
Exam tip: Google's foundation models already have project-specific endpoints and are immediately available, but any tuned version of those models must be deployed to an endpoint before use.
Google Cloud tools for training and deployment connect model creation to model serving through Vertex AI as the central platform. Training or tuning produces a model, deployment associates that model with an endpoint and resources, and prediction services send requests to the endpoint for serving. The choice between pre-built and custom training affects how much model work is done, while the choice between fully managed and self-managed deployment affects who manages serving resources.
The tools that support specific parts of the process include:
Model Management and Operational Governance are the practices and Google Cloud tools that keep machine learning models organized, reusable, monitored, and compliant after training. Model management follows data ingestion, data preparation, and model training in the lifecycle, and it continues through model deployment and ongoing operations. It covers model versioning, deployment, reuse, monitoring, and the data governance rules that control the data and features models depend on.
A data governance framework is a set of policies, procedures, and roles that manages data throughout its lifecycle. A framework helps improve data quality, enhance security, improve utilization, and reduce redundancy. The framework should be chosen and adapted to an organization's industry, size, needs, and priorities.
Google Cloud implements data governance with Knowledge Catalog (opens in a new tab), an intelligent data fabric that unifies distributed data and automates governance without consolidating datasets in one place. To organize data, teams set up logical data lakes and zones. To administer access, they use Google Groups and Knowledge Catalog roles. To streamline quality, they enable auto data quality. Knowledge Catalog also provides a catalog that automatically ingests metadata from supported Google Cloud sources, while sources outside Google Cloud require custom entries. Aspects enrich technical metadata with business metadata, and permissions let data scientists and practitioners search and use assets. BigQuery sharing (opens in a new tab) lets organizations exchange data assets securely through data exchanges and curated listings, while data clean rooms securely manage access to sensitive data for projects with external partners.
Vertex AI Model Registry (opens in a new tab) stores, manages, and deploys trained models after training. The registry lets teams reuse models in subsequent pipelines or for online prediction, so previous training work is not lost. This makes the registry the central place for model management after training.
Versioning tracks and controls different versions of pipelines and components to ensure reproducibility and auditability. Kubeflow pipeline templates are stored in a Kubeflow Pipelines repository in Artifact Registry (opens in a new tab). Custom containers package training code and dependencies into containers and store them in Artifact Registry, giving consistent and reproducible training environments across pipelines and projects.
Monitoring is the practice of watching pipeline execution and deployed models so teams can identify and address issues. Cloud Logging and Cloud Monitoring (opens in a new tab) monitor pipeline execution. For deployed models, Vertex AI MLOps capabilities include model deployment, monitoring, and management. Monitoring tracks model performance and data drift over time so teams can keep models reliable as data changes.
Vertex AI on Distributed Cloud provides an air-gapped deployment for organizations with strict data sovereignty, security, and privacy requirements. It offers MLOps capabilities for model deployment, monitoring, and management inside the organization's data center. Its available services include Generative AI, Online Prediction (Preview), Optical Character Recognition, Speech-to-Text, Translation, and Vertex AI Workbench.
MLOps automation reduces technology and personnel costs by avoiding duplicated activities and reducing workload for data scientists and engineers. Agent Platform Pipelines or Kubeflow Pipelines create and orchestrate pipelines for data transformation, training, deployment, and versioning, letting teams define dependencies between steps, manage data flow, and automate the workflow. Managed Service for Apache Airflow handles complex operational tasks with heavy CI/CD and extract, transform, and load (ETL) requirements. Workflows handles infrastructure-focused tasks managed outside Agent Platform Pipelines, such as starting and stopping VMs or integrating with external systems. Cloud Build with Pub/Sub automates CI/CD with notifications and triggers when code is pushed or a model needs retraining, while Cloud Data Fusion provides a fully managed solution for pipeline management.
Reusability extends beyond pipelines to features, datasets, models, and code that teams publish once and share:
| Resource | What it manages | Governance or reuse benefit |
|---|---|---|
| Vertex AI Model Registry | Trained models | Store, manage, deploy, and reuse models in pipelines or online prediction |
| Agent Platform Feature Store | ML features | Central repository for organizing, storing, and serving features for online and offline use |
| Agent Platform datasets | Datasets | Create and manage datasets centrally and discover them through Knowledge Catalog |
| Custom containers | Training code and dependencies | Package code in Artifact Registry for consistent, reproducible training environments |
Pipeline templates can also be stored in Cloud Storage and shared across an organization. Data products can be published to maximize reuse and discovered through Knowledge Catalog and BigQuery sharing. Unstructured data can be stored in Cloud Storage and governed with BigQuery object tables and signed URLs, while vector embeddings can be managed by including metadata in Vector Search indexes.
Model evaluation and experimentation should begin before an AI project scales up to reduce time to production and wasted resources. During experiments, teams track metadata such as dataset version, model parameters, and model type. Google Cloud tools streamline and automate evaluation and tuning to support continuous training and monitoring. This helps teams improve performance with fewer expensive iterations, achieve faster results, and minimize wasted compute.
Exam tip: Model Registry stores and deploys trained models, while Feature Store stores and serves ML features for reuse across projects.
Cloud Storage holds unstructured data such as images, files, raw staged data, and shared pipeline templates as a landing zone for ingestion. BigQuery is a fully managed and highly scalable platform for structured, curated tables where validated data is queried, shared, and used directly for training. Data flows from source systems into Cloud Storage buckets for staging, then through Dataflow pipelines for transformation into BigQuery tables for analysis and training.
Batch ingestion copies larger, bounded sets of data at intervals while streaming ingestion captures continuous event streams for near-instant access. The choice depends on how fresh the training or analytics data must be, with scheduled transfers for periodic updates and streaming pipelines for real-time needs. Common streaming entry points are Pub/Sub or Google Cloud Managed Service for Apache Kafka paired with Dataflow templates that load continuously into BigQuery.
Google's foundation models in generative AI on Vertex AI already have endpoints that are specific to a Google Cloud project and are immediately available. A tuned version of those models must be deployed to an endpoint before use. Deploying a model associates the model with an endpoint and physical machine resources so it can serve online, low-latency predictions.
Vertex AI Model Registry stores, manages, and deploys trained models after training so they can be reused in subsequent pipelines or for online prediction. Agent Platform Feature Store is a central repository for organizing, storing, and serving machine learning features for online and offline use.
An enterprise is deploying several custom machine learning and generative AI models into production. The AI governance team requires a centralized solution to organize, track, and version models throughout their lifecycle, while operations engineers need automated alerts if the distribution of production input data shifts or model performance degrades over time.
Which combination of Google Cloud capabilities addresses these model management and governance requirements?
Cloud Storage for storing model artifact versions, paired with Cloud Logging for capturing prediction error rates
Vertex AI Feature Store for model version control, paired with Vertex AI Pipelines for continuous drift alerting
Vertex AI Model Garden for custom model versioning, paired with Vertex AI Evaluation for live production drift detection
Vertex AI Model Registry for central versioning and lifecycle tracking, paired with Vertex AI Model Monitoring for drift and performance tracking
Last updated