Automated VM and Container Image Hardening

Design and Automate Image Maintenance and Lifecycle Management is the practice of keeping virtual machine (VM) and container images current, secure, and compliant without manual rebuilds. In Google Cloud, Cloud Build (opens in a new tab) executes the image build, Artifact Registry (opens in a new tab) stores the image with its security metadata, and Cloud Scheduler (opens in a new tab) starts rebuilds on a schedule. Together they pull base-image fixes, operating system patches, and organizational policies into new images, then rotate old images out before they are used by mistake.

1.1Scheduled Rebuild Pipeline With Cloud Build and Cloud Scheduler

A scheduled rebuild pipeline is an automated chain where Cloud Scheduler sends an authenticated HTTP request to a manual Cloud Build trigger, and the trigger builds and publishes a fresh image to Artifact Registry. The workflow runs in a fixed order:

1. Store image: Create an Artifact Registry repository to hold the custom image and enable scanning.
2. Store configuration: Keep the cloudbuild.yaml orchestration file, imagebuilder.yaml recipe, and build scripts in GitHub connected to Cloud Build.
3. Create trigger: Create a manual Cloud Build trigger that points to the target cloudbuild.yaml file.
4. Schedule execution: Create a Cloud Scheduler HTTP job that calls the trigger on a cron schedule.
5. Review results: Check build history in Cloud Build and vulnerability results in Artifact Registry.

Cloud Scheduler invokes the trigger with a POST to the Cloud Build run-trigger endpoint for the project ID and trigger ID. The example schedule 0 1 * * * runs once per day at 01:00, but production frequency should match how quickly base images and patches change. Because the scheduler calls the trigger directly rather than on a Git event, the manual trigger does not use push or pull-request filters.

Two service-account dependencies must succeed for the scheduled call to work. The service account that Cloud Scheduler acts as needs roles/cloudbuild.builds.editor so it can run the trigger, using --oauth-service-account-email and scope https://www.googleapis.com/auth/cloud-platform. The Cloud Build service account that performs the build needs roles/artifactregistry.admin so it can upload the finished image. An administrator setting up the system typically needs Cloud Build Editor, Cloud Scheduler Job Runner or Cloud Scheduler Admin, and Service Account User (roles/iam.serviceAccountUser).

1.2Artifact Registry Storage, Scanning, and Provenance

Artifact Registry is the immutable system of record for published images, security metadata, and build provenance. For custom operating system (OS) images, Image Builder (opens in a new tab) uses a repository with format generic, created with gcloud artifacts repositories create and --repository-format=generic. When an Artifact Registry destination is configured, Image Builder exports the finalized VM boot disk as a .tar.gz file, uploads that file to the generic repository, generates and signs Supply-chain Levels for Software Artifacts (SLSA) build provenance attestations that link the artifact to its source metadata, and registers the production-ready Compute Engine image from the Artifact Registry tar-file URI.

Artifact Registry also supports continuous compliance for container images through automatic vulnerability scanning. When the Container Scanning API is enabled, Artifact Registry automatically scans each pushed image digest and shows totals in the Vulnerabilities column. Selecting an image opens the finding list, which shows severity level, whether a fix is available, and the package name that contains the vulnerability. Rebuilding on schedule therefore pulls in upstream fixes, while scanning verifies whether outdated dependencies or new vulnerabilities remain in the latest digest.

1.3Image Builder Orchestration and Network Dependencies

Image Builder is the Compute Engine pipeline that customizes, validates, and publishes a hardened OS image from a cloudbuild.yaml orchestration file. The orchestration file defines substitutions for staging bucket, recipe path, service account, output folder, and Artifact Registry resource URI, then runs three container steps in strict order. First, the imagebuilder-customize step runs /build on a temporary worker VM to parse imagebuilder.yaml and apply OS customization. Then, the imagebuilder-validate step runs /validate on a temporary test VM to perform system boot checks. Only after validation succeeds, the imagebuilder-publish step runs /publish to register the image in Compute Engine and upload tar files to Artifact Registry as generic_artifacts.

The build depends on Virtual Private Cloud (VPC) connectivity because Image Builder provisions temporary worker and test VMs during build and validation. By default those VMs attach to the default VPC network with ephemeral external IP addresses. If the recipe sets a custom network or subnetwork or sets externalIP: none, the subnetwork must have Private Google Access (opens in a new tab) enabled so instances without external IPs can still reach Google APIs such as Cloud Storage and Artifact Registry. If customization downloads packages from external internet repositories, the subnetwork also needs Cloud NAT and firewall rules must allow egress to Google APIs and software repositories. For interactive debugging with debug: true, firewall rules must allow ingress on TCP port 22, and VMs without external IPs must allow ingress from the Identity-Aware Proxy (IAP) TCP-forwarding range 35.235.240.0/20.

1.4Image Lifecycle Rotation With Families and Deprecation

Image lifecycle rotation is the process of grouping related images, pointing users to the current version, and blocking or deleting stale versions. Compute Engine image families group related images so deployments can roll forward to a new image and roll back to a prior image without hard-coding a specific image name. Rotation depends on the build pipeline using the latest successful output and on administrators deprecating the images the family no longer points to.

Deprecation in Compute Engine uses explicit states that control whether an image can still launch instances:

Lifecycle policy is enforced with the gcloud compute images deprecate command, which can be added as a final check in the image-build pipeline. Administrators attach future deletion metadata with --delete-in or --delete-on and future obsolescence metadata with --obsolete-in or --obsolete-on. Deprecated and deleted images are hidden from API and console results by default, but --show-deprecated reveals them for audit, and complete removal requires an explicit gcloud compute images delete call. This pattern prevents stale-image sprawl while preserving a clear rollback path through the image family.

Automated patch management for virtual machines in Google Cloud is delivered by VM Manager Patch (opens in a new tab), which keeps long-running Compute Engine VMs protected against defects and vulnerabilities. The service has two cooperating parts: patch deployment, which automates operating system and software updates, and patch compliance reporting, which shows patch status across Windows and Linux fleets. A patch deployment schedules patch jobs, and a patch job runs across VM instances and applies patches.

2.1How Patch Deployment Moves Updates to VMs

Patch deployment is the execution path that starts with an API call and ends with OS-native tools installing updates on each target VM. A call to the VM Manager API, also known as the OS Config API, from the Google Cloud console, Google Cloud CLI, or direct API notifies the OS Config agent running on target VMs to start patching. The agent runs patching with the native tool for each distribution, retrieves updates from the distribution source, and reports progress back to the VM Manager API as patching proceeds.

The update tools used by the OS Config agent are:

• Red Hat Enterprise Linux (RHEL), Rocky Linux, and CentOS use yum upgrade
• Debian and Ubuntu use apt upgrade
• SUSE Linux Enterprise Server (SLES) uses zypper update
• Windows uses Windows Update Agent

The Patch service does not host or maintain package updates. Updates are pulled from package repositories, called the distribution source, or from a local repository, so each VM must have network access to its update source. VMs without public IPs or in a private VPC network need extra access, and Google recommends hosting a local repository or Windows Server Update Service for full control over the patch baseline, or making external sources reachable with Cloud NAT or other proxy services. A local repository cannot be hosted when running Pay-As-You-Go (PAYG) RHEL, RHEL for SAP, SLES, and SLES for SAP images.

2.2How Patch Compliance Status Is Generated

Patch compliance reporting is the continuous visibility path that turns installed-package inventory into a severity-ranked compliance state. After VM Manager is set up, the OS Config agent reports OS inventory data about every 10 minutes, the patch compliance backend reads that data and cross-references it with package metadata from the OS distribution, and the Google Cloud console displays the result. This lets operators see which VMs need critical, security, or other updates without logging into each machine.

The patch compliance backend generates compliance data in an ordered sequence:

1. Read inventory reports: Read the OS inventory reports collected from the VM.
2. Scan vulnerability sources: Scan classification data from the vulnerability source for each operating system and order it by severity from highest to lowest.
3. Map to Google status: Map those vendor classifications to Google's patch compliance status.
4. Select highest severity: Select the highest severity for each available update and show it on the dashboard, with the full per-VM update list available on the VM details page.

Vulnerability sources differ by operating system, and the backend orders their data before mapping:

• RHEL and CentOS use Red Hat security data, based on the latest minor version for each major version, which can be inaccurate for older minor versions
• Debian uses the Debian security tracker
• Ubuntu uses the Ubuntu CVE tracker
• SLES uses SUSE update data
• Rocky Linux uses Rocky errata data
• Windows uses classification data from the Windows Update Agent API

The mapping from vendor categories to Google status is:

For example, if a RHEL 7 VM reports installed version 1.4 with update version 2.0 available, and the source lists version 1.5 as Critical for CVE-001 plus versions 1.8 and 1.9 as Low, the dashboard counts the VM as having one Critical update covering three CVEs.

2.3Targeting VMs With Patch Jobs and Patch Deployments

Targeting controls which VMs receive patches, while jobs and deployments control when patching runs. A patch job (opens in a new tab) runs on demand and starts patching simultaneously on all instances matched by its instance filter, with a maximum of 500 VMs in a single patch job. A patch deployment (opens in a new tab) creates automatically scheduled jobs with either a one-time schedule at a specific date and time or a recurring schedule at a specified interval, with a maximum of 500 VMs in a single patch deployment.

Instance filters are evaluated at the time each job runs, so fleet changes are captured without editing the schedule. If a deployment targets a zone that has 20 instances at creation time and 40 instances are added before the run, all 60 instances are updated. In the console, filtering is limited to zone selection plus name prefix or label, such as name prefix test- with labels env=dev and app=web, while the CLI and REST support additional filter types. Targeting all VMs at once can reduce availability, so operators should plan maintenance windows carefully or patch in small batches with filters.

Handling for managed instance groups (MIGs) and unpatchable operating systems is controlled by two flags. By default VM Manager does not patch VMs that are part of a MIG unless migInstancesAllowed is set to true, and the console exposes this as Allow patching MIG instances. When skipUnpatchableVms is set to true, the job skips VMs whose OS does not support patching, such as Container-Optimized OS, and VMs in a MIG when MIG patching is not allowed, reporting those VMs as SKIPPED and reporting COMPLETED_WITH_INACTIVE_VMS instead of COMPLETED_WITH_ERRORS when no other errors occur. After a job starts, the Patch dashboard populates in about 30 minutes.

2.4Maintenance Windows and Controlled Rollouts

A maintenance window is the total time allowed for a patch job to run before the job times out. If the window is set to 60 minutes, no new patch activity is initiated 60 minutes after the start time. Some in-flight work such as downloading a file or rebooting can occur outside the window, but no new patch operations start after the window expires.

Rollout options limit blast radius by controlling zone order and concurrency. Operators choose whether to patch one zone at a time or patch zones concurrently, then set a disruption budget as the number or percentage of VMs in a zone that can be disrupted at one time. For example, a rollout can patch all VMs zone-by-zone with no more than 10 VMs disrupted per zone, or patch zones concurrently with no more than 50 percent or 25 percent disrupted per zone. Patch deployment names used for recurring automation must be unique within a project, use only lowercase letters, numbers, and hyphens, start with a letter, end with a letter or number, and be 1 to 63 characters.

2.5Reboot Control and Pre-Patch and Post-Patch Health Checks

Reboot control and hook scripts determine whether updates fully apply and whether applications are safely quiesced and verified. Reboot behavior for a patch job is one of three options:

• Default: the agent decides if reboot is necessary from well-known signals on each OS, and multiple reboots can occur, including before patches install
• Always: the machine reboots after the update completes
• Never: the machine does not reboot after the update completes, which can leave some patches incompletely applied

Pre-patch and post-patch scripts provide the health-check and application handling around patching. Pre-patch scripts run before patching starts, and before a pre-patching reboot if one is required, which supports tasks such as shutting down an application. Post-patch scripts run after patching completes, and after a patching reboot if one occurs, which supports tasks such as health checks. A patch job accepts one pre-patch and one post-patch script for Linux and one pre-patch and one post-patch script for Windows, where Linux scripts run only on Linux VMs and Windows scripts run only on Windows VMs.

Linux scripts must begin with a shebang line such as #!/bin/bash to select the executing shell. Script files can be stored on the VM or in a versioned Cloud Storage bucket. When scripts come from Cloud Storage, the instance service account needs Identity and Access Management (IAM) permission to read non-public objects, the console uses the latest object version by default, and buckets must sit in regions and zones permitted when the Resource Location Restriction organization policy is enforced.

2.6Permissions, Monitoring, and Operational Limits

Permissions for patch automation separate who can run jobs from who can manage schedules. Project owners have full access, while other users need granular roles:

The executor role carries osconfig.patchJobs.exec, which can manage software packages on VMs, so access should follow least privilege and usage should be monitored to avoid unintended code execution on VMs. The Patch dashboard centralizes monitoring with an operating system overview and a per-OS compliance breakdown into Critical, Important or security, Other, and Up-to-date. A VM appears as No data when it is unresponsive, the OS Config agent is not installed, OS inventory management is not enabled, or the OS is unsupported. For Windows fleets, Google recommends disabling Windows automatic updates to reduce conflicts with Patch, and Microsoft application updates for products such as SQL Server, SharePoint Server, or .NET Framework are enabled separately through Windows Update settings or PowerShell. Patch jobs run only within a single Google Cloud project and cannot span projects even with Shared VPC, although compliance data can be viewed across projects, and patching MIG members carries the risk that MIG repair recreates VMs from the instance template or that autoscaling replaces patched VMs with new unpatched VMs.

Automated hardening and baseline security for images is the practice of defining a minimal, secure starting image in code and rebuilding it through a pipeline so every workload starts compliant. In Google Cloud, Packer (opens in a new tab) with Cloud Build (opens in a new tab) produces hardened Compute Engine VM images, while Cloud Build triggers with Artifact Registry rebuild container images from versioned Dockerfiles to pull in base-image fixes. Both paths integrate vulnerability scanning and image review so only validated, immutable images reach production.

3.1Hardened Images as Immutable Baseline

Hardened base images implement immutable infrastructure with strictly defined functionality and privileges. The organization defines the operating system, packages, users, and privileges once in code, then redeploys new images instead of patching running systems. Google Cloud controls (opens in a new tab) for this model require teams to review hardened Compute images, review Artifact Registry images and Cloud Run images, and verify Google Kubernetes Engine (GKE) Container Optimized OS on nodes.

This baseline reduces drift because no manual change persists outside the build definition. A VM baseline starts from a trusted source family such as ubuntu-2004-lts from ubuntu-os-cloud and adds only required software through provisioners. A container baseline starts from a pinned, minimal base image and adds only required layers. When a new vulnerability or benchmark update appears, the team updates the definition and rebuilds rather than logging into running VMs or containers.

3.2Building Hardened VM Images With Packer and Cloud Build

Packer is an open source tool that builds identical VM images from a single template, and Cloud Build automates that build on Google Cloud. The Packer template packer.pkr.hcl declares variables for project_id, image_name, image_family, and image_zone, declares a googlecompute source with source_image_family, source_image_project_ids, zone, ssh_username, and machine_type, and declares a build block with shell provisioners that apply hardening. The example provisioner runs apt-get update and installs only the approved package, which is where Center for Internet Security (CIS) benchmark steps, removal of unneeded users, disabling of services, and configuration validation commands belong.

The build depends on a Packer builder container stored in Artifact Registry. The team clones cloud-builders-community, builds the packer directory, and pushes it to [LOCATION]-docker.pkg.dev so Cloud Build can invoke packer build as a build step. Data flows from source code and the Packer template in the build workspace to the temporary Compute Engine instance that Packer launches, then to a new Compute Engine image in the image family, with the builder image pulled from Artifact Registry at step start.

The pipeline requires specific APIs and identity boundaries. The project must enable compute.googleapis.com, servicemanagement.googleapis.com, storage-api.googleapis.com, and artifactregistry.googleapis.com. The Cloud Build service account must hold the permissions that let it create the ephemeral build VM and publish results:

3.3Automating Container Rebuilds for Patch Synchronization

Automated container rebuilds keep custom images synchronized with upstream base-image security updates and patches. The image definition, typically a Dockerfile and Cloud Build config, lives in GitHub as versioned configuration, while Artifact Registry stores and scans the built image. Cloud Scheduler initiates rebuilds on a regular schedule and Cloud Build triggers execute them, so fixes flow from the base-image publisher to deployed workstations and services without manual rebuilds.

The pipeline moves through an ordered lifecycle where each stage depends on the prior stage:

1. Store: Push the custom image to an Artifact Registry repository with scanning enabled, so the image moves from built to stored.
2. Version: Commit the Dockerfile and build config to GitHub connected to Cloud Build, so changes move from local to reviewable.
3. Trigger: Create a Cloud Build trigger on the GitHub repository that builds the container and pushes it to Artifact Registry on commit or API call.
4. Schedule: Configure Cloud Scheduler to call the trigger on a cadence, so the image moves from current to rebuilt even with no code change.
5. Review: Inspect build history, scan findings, and deployed image digests to confirm the new image moved from scanned to approved for use.

This design enforces the trust boundary between mutable source and immutable output. Developers can change GitHub definitions, but only the Cloud Build service account can push to the protected Artifact Registry repository. External build integrations for Cloud Build should be explicitly defined and limited to the required GitHub connection and worker pool, and private build pools with Private Google Access keep build traffic inside the Virtual Private Cloud (VPC) boundary.

3.4Integrating Vulnerability Scanning and Configuration Validation

Vulnerability scanning and configuration validation turn the rebuild pipeline into a security gate for both VM and container baselines. Artifact Analysis with Container Scanning automatically scans images pushed to Artifact Registry for known operating system and language-package vulnerabilities. Agent and application best practices extend this with runtime vulnerability scanning for deployed artifacts and cleanup policies that remove old, unpatched tags so teams cannot accidentally redeploy them.

Validation confirms the image matches its hardened baseline before promotion. Controls include reviewing Artifact Registry images for unexpected packages or privileges, verifying Container-Optimized OS on GKE nodes (opens in a new tab), enabling GKE auto-repair and auto-upgrade to replace nodes running vulnerable images, and enforcing Binary Authorization to allow only signed, scanned images. When scanning reports a high-severity finding, the fix flows from an updated base pin or hardening script in GitHub through Cloud Build to a newly scanned image in Artifact Registry, and only after the finding clears does the image digest advance to production workstations, Cloud Run, or GKE. Compliance frameworks (opens in a new tab) such as CIS benchmarks provide the baseline rules that this validation checks against.

Hardened image builds, scheduled image maintenance, and VM patch management form one lifecycle with different targets. Hardened pipelines create the trusted starting point for new VMs and containers, scheduled rebuilds and family rotation keep that starting point current, and VM Manager Patch protects long-running VMs that cannot simply be replaced. Scanning, provenance, deprecation states, maintenance windows, and reboot controls are the guardrails that keep each path compliant without expanding blast radius.

Key Facts

• Cloud Scheduler triggers scheduled image rebuilds by calling a manual Cloud Build trigger, and Cloud Build publishes the fresh image to Artifact Registry for scanning and deployment.
• Compute Engine image families allow roll forward and rollback without hard-coding image names, while DEPRECATED images still launch with a warning and OBSOLETE images block new launches.
• A patch job runs on demand across a maximum of 500 VMs, while a patch deployment creates one-time or recurring scheduled jobs with a maximum of 500 VMs per deployment.
• The OS Config agent installs patches using native tools such as yum upgrade, apt upgrade, zypper update, and Windows Update Agent, pulling updates from the distribution source rather than from Google-hosted packages.
• Packer with Cloud Build produces hardened Compute Engine VM images, while Cloud Build triggers with Dockerfiles produce container images stored in Artifact Registry where Container Scanning applies.