Securing IaaS and PaaS Training Environments

Secure PaaS machine learning in Vertex AI (opens in a new tab) means Google manages the training compute while the customer controls the network perimeter, encryption keys, and identity boundaries around it. Custom training jobs, Vertex AI Pipelines, managed datasets, Model Registry, and Vertex AI Workbench notebooks all run as Google-managed resources that must be attached to the customer Virtual Private Cloud (VPC). Security comes from combining a service perimeter to block exfiltration, private connectivity to remove public paths, and per-resource encryption for data at rest.

1.1Network Perimeters That Block Exfiltration During Custom Training

VPC Service Controls (opens in a new tab) creates a service perimeter around Vertex AI application programming interfaces (APIs), training jobs, and sensitive data to prevent data exfiltration from the VPC. The Vertex AI training project, Cloud Storage training buckets, and related APIs are placed inside the same perimeter, and any call that tries to leave the perimeter is denied unless an explicit ingress or egress rule allows it. Firewall rules add a second layer by denying all inbound and outbound traffic by default and allowing only traffic that meets defined criteria such as specific ports and sources.

For compute isolation, Vertex AI managed training supports private-only communication through a Private Service Connect (PSC) interface and private Internet Protocol (IP) addressing. Traffic flows from the custom training job to Cloud Storage, Vertex AI datasets, and other authorized Google Cloud services through the PSC interface inside the perimeter, and the training workers are not exposed to public traffic. Use dedicated projects or dedicated VPCs to isolate training environments and limit access, and use Shielded VMs or Confidential Computing when sensitive code and data must be protected during execution.

Choose VPC Service Controls when the risk is copying data to an outside project or service, and choose PSC with private IP when the risk is exposure of the training workers themselves to the internet.

1.2Private and Context-Aware Access for Workbench Sessions

Vertex AI Workbench instances are managed notebook environments that must be prevented from becoming an exfiltration path through browsers, terminals, or public networking. Harden Workbench networking by blocking public IP addresses, blocking use of the default VPC network, requiring attachment to a designated private VPC subnet, and enabling Private Google Access so notebooks reach Google APIs without internet routes. Detective controls check for risky states, including file downloading in the JupyterLab console, root access, terminal access, and internet access for runtime templates in Colab Enterprise.

Access to Workbench instances is re-evaluated on every authentication by Access Context Manager combined with Chrome Enterprise Premium and Identity-Aware Proxy (IAP). Access Context Manager enforces context-aware rules based on IP address and device attributes, so a user who passed yesterday is checked again on each login to the instance. Additional posture controls include automatic upgrades on a schedule, integrity monitoring and Secure Boot for runtimes, delete-to-trash protection, and OS Login to tie virtual machine (VM) access to Identity and Access Management (IAM) identities.

1.3Customer-Managed Encryption Across Buckets, Datasets, Registry, and Metadata

Customer-managed encryption keys (CMEK) in Cloud Key Management Service (Cloud KMS) give the customer control over the keys that encrypt Vertex AI data at rest, replacing Google-managed default encryption for regulated workloads. Keys can be software-based or hardware-backed in Cloud Hardware Security Module (HSM), can be auto-provisioned with Autokey, or can be sourced externally with Cloud External Key Manager (Cloud EKM). Encryption in transit is separate and is enforced with Hypertext Transfer Protocol Secure (HTTPS) for all artificial intelligence (AI) and machine learning (ML) API calls, with HTTPS load balancers used to enforce HTTPS for applications.

CMEK must be enabled per resource type and does not automatically inherit from the bucket to the model, so each layer of the pipeline needs its own key assignment:

• Cloud Storage training buckets that hold input data, staging files, and exported artifacts
• Vertex AI datasets, Feature Store, and BigQuery (opens in a new tab) tables used for training inputs
• Vertex AI models, model endpoints, custom jobs, training pipelines, and hyperparameter tuning jobs
• Vertex AI ML Metadata stores, TensorBoard resources, Model Registry artifacts, Workbench instance disks, and notebook runtime templates

The Vertex AI platform, Cloud Storage, BigQuery, Cloud SQL, and Spanner are CMEK-integrated services that encrypt data at rest with the assigned key once configured. Organization policy gcp.restrictNonCmekServices can require CMEK protection for services such as BigQuery, and detective controls flag any dataset, model, endpoint, metadata store, feature store, custom job, or training pipeline that is missing CMEK.

1.4Least-Privilege Identity Boundaries for Pipelines

Vertex AI Pipelines manage ML operations (MLOps) stages across services and support low-level access control through dedicated service accounts and attribute-based conditions. Create a dedicated, least-privilege service account for each pipeline so a training pipeline service account can read data from only one specific Cloud Storage bucket and write model artifacts only to Model Registry. IAM Conditions enforce conditional access, for example allowing a service account to trigger a pipeline only if the request originates from a trusted Cloud Build trigger.

The same boundary model applies to users and operators through role-based access control with the principle of least privilege. Map predefined or custom Vertex AI IAM roles to users, groups, or service accounts, add IAM Conditions for data, models, code, and hyperparameters, and layer principal access boundary policies and Privileged Access Manager for just-in-time elevation that is later visible in audit logs. For data-level granularity, set IAM on Cloud Storage buckets and managed folders and use row-level and column-level security in BigQuery.

1.5Lineage, Audit, and Continuous Compliance for PaaS Artifacts

Vertex AI ML Metadata provides lineage and tracking for models, datasets, and artifacts by logging data sources, transformations, model parameters, and experiment results. Pipeline artifact lineage from Vertex AI Pipelines can be searched and analyzed with Knowledge Catalog for debugging and root-cause analysis, while Model Registry applies and manages versions for each stored model. Cloud Audit Logs for Vertex AI record endpoint creation, model deployment, configuration updates, and data access, and logs can be analyzed with Observability Analytics or routed to third-party destinations.

Continuous assurance comes from Security Command Center (opens in a new tab) with AI Protection against the Google Recommended AI Essentials framework for Vertex AI. That framework bundles preventative and detective policies for Workbench public IPs, default VPC use, file downloads, root access, audit logging, automatic upgrades, and CMEK across custom jobs, datasets, models, endpoints, feature stores, metadata stores, training pipelines, TensorBoard, and Workbench disks. Model artifacts themselves are further protected by storing them in Cloud Storage and Model Registry with fine-grained access control, encrypting them with Cloud KMS, signing approved packages and containers, and scanning images with Artifact Analysis.

A hardened Infrastructure as a Service (IaaS) training environment isolates self-managed model training on Compute Engine (opens in a new tab) and Google Kubernetes Engine (GKE) from unauthorized infrastructure access. The training VMs hold sensitive training data and model weights in memory, on disk, and on accelerators, so controls must cover network perimeters, boot integrity, in-use memory encryption, operating system baselines, and accelerator selection. Together these controls keep training traffic private, prove the boot chain is trusted, and limit who can reach the hosts and graphics processing units (GPUs) or Tensor Processing Units (TPUs).

2.1Isolate Training Projects, Networks, and Perimeters for IaaS

Isolation for IaaS training starts by placing training in dedicated projects or dedicated VPC networks. Traffic flows from the training VMs to storage, logging, and Google Cloud services only through private paths, and the project boundary separates training identities and quotas from development or serving projects. To enforce this globally, organization policies can disable external IP allocation to VMs, prevent automatic creation of the default network, restrict resource creation to specific locations, and disable creation of service accounts or keys.

Network enforcement then controls what can enter or leave the isolated environment. Custom firewall policies deny all traffic by default and explicitly allow only required ports, protocols, and sources, which prevents training nodes from accepting public connections or exfiltrating data. For stronger data-movement control, a VPC Service Controls perimeter creates a security boundary around the training VPC and included AI resources, with ingress and egress rules that govern data flow across the perimeter.

2.2Protect Boot Integrity and In-Use Memory with Shielded VMs and Confidential Computing

Shielded VMs protect the boot chain of Compute Engine training VMs by authenticating boot components such as the bootloader and kernel during startup. Secure Boot blocks rootkits and bootkits that would otherwise load before the operating system and then steal training data or tamper with model weights. Secure Boot does not apply to all configurations, including GKE instances covered by node-level controls, Compute Engine disks with GPU accelerators that do not use Container-Optimized OS, guest operating systems that do not use Unified Extensible Firmware Interface (UEFI), or Serverless VPC Access resources.

Confidential Computing, implemented as Confidential VMs, protects data while it is in use in system memory. Google Cloud encrypts data at rest and data in transit by default, but data in memory is not encrypted by default, so an attacker with host-memory access could undermine the confidentiality and integrity of training data and code during execution. A Confidential VM runs the workload in a trusted execution environment that encrypts memory, so sensitive datasets and intermediate model state remain protected even while the training process is actively using them.

2.3Harden GKE Clusters Used for Self-Managed Training

A hardened GKE (opens in a new tab) cluster for training isolates containerized training jobs at the node, pod, and network layers. Private nodes remove public IP addresses from nodes so inbound traffic flows only from the VPC or authorized private paths, while network policies and pod security controls restrict which pods can communicate with each other and which privileges they can use. Access to applications running in the cluster is verified through IAP integrated with GKE Ingress, so only authenticated users with the correct IAM role reach the endpoint.

Container and runtime controls further limit blast radius if a training container is compromised. Training workloads must use trusted and verified container images, and Artifact Analysis scans those images for operating system and language-package vulnerabilities before deployment. GKE Sandbox adds a second kernel-level isolation boundary around pods to contain container escapes, and workload separation places training jobs with different sensitivity levels on separate node pools.

Remediation for a noncompliant boot configuration follows a fixed order:

1. Create a new node pool with Secure Boot enabled under the corrected Shielded VM settings.
2. Migrate training workloads from the old pool to the new pool until the old pool is empty.
3. Delete the original pool only after migration is complete so no workload is left on untrusted nodes.

2.4Lock Down Host OS, Access, and Traffic Paths for Training VMs

Compute Engine gives more granular operating system-level control than managed training, which is both an advantage and a hardening responsibility. Administrators select, configure, and manage the VM operating system environment to meet workload requirements, tailor the VM machine type to exact central processing unit (CPU), memory, and accelerator needs, and rebuild from a new hardened image when anything changes rather than patching a running host. For container-based training on GKE, require Container-Optimized OS as the node operating system baseline because it presents a minimal, auto-updating surface designed for containers.

Host access and packet-forwarding controls reduce lateral movement to and from training VMs:

• Require OS Login on Compute Engine instances to centralize Secure Shell (SSH) identity and authorization.
• Restrict external IP addresses to only the specific VM instances that require them and disable the default network.
• Prevent IP forwarding on training VMs so a compromised VM cannot route packets on behalf of another host.
• Disable hardware-accelerated nested virtualization with the compute.disableNestedVirtualization constraint to reduce attack surface.
• Apply deny-by-default firewall policies that allow only explicitly approved training, storage, and management traffic.

To remediate a Compute Engine VM with Secure Boot disabled, first stop the instance, then edit the instance and enable Secure Boot under the Shielded VM settings, and finally start the instance again. An existing VM cannot simply toggle forwarding or boot trust while running without recreating or restarting under the corrected configuration, so noncompliant training hosts must be rebuilt or replaced.

2.5Allocate GPU and TPU Capacity for Isolated Training

Secure accelerator allocation on Compute Engine uses the accelerator-optimized machine family to bind specific GPUs or TPUs to isolated training VMs. The VM machine type determines how much CPU, memory, and accelerator capacity the training job receives, and a workload manager such as Slurm controls how those compute resources are scheduled across the training fleet. Because Slurm is highly configurable but requires expertise in Slurm administration and Linux system management, GKE automates cluster management while Compute Engine with Slurm provides maximum control for proprietary intellectual property and specialized performance tuning.

GPU and TPU accelerators differ in optimization and best fit:

Accelerator hosts inherit the same isolation requirements as the rest of the training fleet, including private addressing, firewall deny-by-default, and memory protection. GPU-based VMs that do not use Container-Optimized OS cannot rely on the standard Compute Engine Secure Boot control, so those hosts need compensating image-hardening, network isolation, and vulnerability scanning.

Least-privilege IAM and artifact protection for training means giving automated pipelines only the identity, network path, and storage access they need, and locking down the containers and model weights they produce. The division of work changes with the hosting model: self-managed IaaS leaves operating system, cluster, and runtime isolation to the customer, while fully managed PaaS training has Google secure the underlying infrastructure and provide controls for data, code, and models. Across both models, protection depends on dedicated service accounts, Artifact Registry controls for training containers, Model Registry and storage controls for weights, and audit logging for every pipeline action.

3.1Shared Responsibility Across Training Hosting Models

Shared responsibility for training defines which security layers Google manages and which layers the customer must configure. In PaaS training with Gemini Enterprise Agent Platform (opens in a new tab) serverless training and managed custom jobs, Google secures the underlying infrastructure while the customer protects training data, code, models, and access policies, using tools such as private IP, Private Service Connect interfaces, CMEK, VPC Service Controls, and Access Transparency. In IaaS and container training with Compute Engine, GKE, Cloud Run, and Managed Service for Apache Spark, the customer additionally hardens the execution environment itself, including operating system selection, cluster network policies, pod security, and workload isolation.

IaaS control brings a tradeoff: Compute Engine with Slurm allows granular operating system control and proprietary intellectual property integration for specialized accelerators, but it requires Slurm and Linux expertise, while GKE automates cluster management at the cost of less operating system flexibility. Network boundaries apply in both models through dedicated projects or VPCs for training isolation, firewalls that deny by default, VPC Service Controls perimeters with ingress and egress rules, and Shielded VMs or Confidential Computing to protect code and data during execution.

3.2Dedicated Least-Privilege Service Accounts for Training Pipelines

Dedicated least-privilege service accounts are per-pipeline identities that carry only the minimum IAM permissions needed for one automated stage. A training pipeline service account, for example, has permission to read data from only a specific Cloud Storage bucket and to write model artifacts to Model Registry, but it does not have write access to production serving endpoints. This separation limits blast radius so compromise or misconfiguration of training cannot directly overwrite production models or read unrelated datasets.

Apply IAM policies consistently across every asset the pipeline touches, including Cloud Storage buckets that hold training data and checkpoints, BigQuery datasets used for training inputs, Agent Platform resources such as pipelines and endpoints, and compute resources such as GKE clusters and Cloud Run services. Refine those grants with IAM Conditions for conditional, attribute-based access control, for example allowing a service account to trigger a pipeline only if the request originates from a trusted Cloud Build trigger. Centralize human and group lifecycles in IAM, combine predefined or custom roles for different user groups, and use principal access boundary policies and Privileged Access Manager for just-in-time elevation with auditable history.

3.3Secure Training Containers in Artifact Registry

Artifact Registry (opens in a new tab) is the universal package manager that centralizes custom ML packages, vetted dependencies, and training container images behind access control, VPC Service Controls perimeters, and organization policies. Remote repositories cache external artifacts from sources such as Docker Hub, Maven Central, Python Package Index (PyPI), Debian, or CentOS to improve availability and enable scanning, while virtual repositories consolidate same-format repositories behind one endpoint and prioritize private packages to reduce dependency-confusion risk. Data flows from the build in Cloud Build to the registry repository and then to the training cluster or Cloud Run service, and every hop depends on IAM and perimeter checks.

Protect that flow by verifying what is stored and blocking what fails checks. Artifact Analysis provides on-demand or automated scanning of base images and language packages in containers, generates a software bill of materials (SBOM), and accepts Vulnerability Exploitability eXchange (VEX) statements for images in Artifact Registry. Binary Authorization then enforces deployment policy so GKE and Cloud Run run only images that pass security checks, and dashboards expose vulnerabilities, build provenance, digests, source locations, toolchain, and build duration.

3.4Protect Model Weights and Artifacts from Unauthorized Access and Export

Model artifact protection treats weights, files, deployment configurations, and training data as intellectual property with different access levels per artifact based on user roles and needs. Model Registry on Gemini Enterprise Agent Platform enables fine-grained access control and encryption at rest for versioned models, links each version to training parameters, dataset version, code version, and evaluation metrics, and supports digital signatures for packages and containers from approved builds. Before import, store raw model files and pretrained weights in Cloud Storage with IAM access controls and Cloud KMS CMEK encryption, and use organization policies and Model Garden access controls to limit which model types and versions a project can use.

Reduce exfiltration and tampering risk by layering encryption, posture, and re-execution checks around those stores. Server-side encryption with CMEK protects data at rest even if underlying storage is compromised, while HTTPS and HTTPS load balancers protect data in transit for AI API calls. When Agent Platform Pipelines re-executes, require explainability evaluations such as feature attributions and responsible AI checks before deploying artifacts, which helps reveal unauthorized changes indicating tampering, cross-site scripting indicating compromised images, and insecure endpoints indicating misconfigured serving infrastructure.

3.5Monitor Pipeline Execution with Logging and Audit Trails

Pipeline monitoring for training captures who invoked what, what data and code were used, and where artifacts moved. Enable Cloud Audit Logs for Agent Platform to record API calls and administrative actions such as endpoint creation, model deployment, and configuration updates, and stream those logs to Cloud Logging for real-time analysis, alerting, and visualization with Observability Analytics. Agent Platform ML Metadata and Knowledge Catalog extend that trail with lineage for datasets, transformations, parameters, experiments, and pipeline artifacts, so an individual artifact can be debugged and the full MLOps run can be audited from raw data to prediction.

This logging dependency closes the least-privilege loop: the service account defines what the pipeline is allowed to do, Binary Authorization and registry scans define what code is allowed to run, Model Registry defines where weights are allowed to go, and audit logs prove what actually happened. Analyze logs to identify who accessed or modified data and models and when, route logs to third-party destinations where required, and track provenance, SBOM dependencies, and deployment history to sustain Supply-chain Levels for Software Artifacts (SLSA) integrity over time.

Key Facts

• VPC Service Controls creates a service perimeter around Vertex AI APIs, training jobs, and Cloud Storage buckets that denies data movement outside the perimeter except through configured ingress and egress rules.
• Vertex AI custom training with Private Service Connect and private IP keeps job traffic private to authorized Google Cloud services with no public exposure.
• Shielded VMs with Secure Boot verify boot integrity against rootkits and bootkits, while Confidential VMs encrypt data in use in memory.
• Customer-managed encryption keys (CMEK) must be assigned per Vertex AI resource, including Cloud Storage buckets, datasets, models, endpoints, pipelines, ML Metadata, and Workbench disks.
• A training pipeline service account should read only its assigned input bucket and write only to Model Registry, with IAM Conditions able to require a trusted Cloud Build trigger as the request origin.