Safe Patching and GKE Upgrade Rollouts

Orchestrating OS patching for Compute Engine VMs means automating patch discovery, targeting, rollout, and verification across a fleet without taking production down. In Google Cloud, VM Manager (opens in a new tab) divides this work into patch deployment, which schedules and controls patch jobs, and patch compliance reporting, which shows which VMs still need updates. Both depend on the OS Config API in the control plane and the OS Config agent running on each VM to apply updates with native OS tools.

1.1Patch Deployment Flow from Control Plane to VM Agents

Patch deployment is the control flow that starts when an operator calls the OS Config API from the console, Google Cloud CLI, or direct API call. The OS Config API then notifies the OS Config agent on each targeted VM, the agent invokes the distribution-native updater to pull packages from the OS distribution source, and the agent reports progress back to the OS Config API where job status becomes visible. A patch deployment schedules one or more patch jobs, while a patch job (opens in a new tab) is the single execution that patches the VMs matched at start time.

The agent does not use a Google-hosted patch feed. It uses yum upgrade on Red Hat Enterprise Linux (RHEL), Rocky Linux, and CentOS, apt upgrade on Debian and Ubuntu, zypper update on SUSE Linux Enterprise Server (SLES), and Windows Update Agent on Windows. Setup is strict: the OS Config API must be enabled and the agent installed with OS inventory management enabled, or the VM will not appear correctly and calls return NO_AGENT_DETECTED or TIMED_OUT. For Windows fleets, disable Windows automatic updates to avoid conflicts between the native scheduler and VM Manager.

1.2Package Sources and Network Paths for Isolated VMs

Package sources are the external distribution repositories or local mirrors that actually supply OS updates, because the Patch service does not host patches. VMs with public IPs normally reach external repositories directly, while VMs without public IPs or in a private Virtual Private Cloud (VPC) network need an extra path. Google recommends hosting a local repository or Windows Server Update Service (WSUS) for full control over the patch baseline, or exposing external sources through Cloud NAT or another proxy.

Local mirrors are not available on all images. Pay-As-You-Go (PAYG) images for RHEL, RHEL for SAP, SLES, and SLES for SAP cannot host a local repository. This boundary matters for isolated designs: if production VMs have no internet egress, the architecture must include an approved mirror or NAT path before any patch job can succeed.

1.3Fleet Targeting Filters and Per-OS Patch Content

Fleet targeting is the instance filter that selects which VMs a patch job patches, evaluated when the job starts rather than when it is scheduled. Because filtering happens at execution time, a recurring deployment automatically includes VMs created or deleted after the schedule was created. Console filtering is limited to zone, name prefix, and labels such as env=dev and app=web, while gcloud and REST support richer filters. A single patch job patches a maximum of 500 VMs simultaneously, so large fleets must be split into smaller batches by labels or zones.

Patch content is selective by operating system, and a job can also update Google agents installed as standard packages for that distribution:

By default, Windows jobs patch only the Windows OS and do not patch Microsoft applications such as SQL Server, SharePoint Server, or .NET Framework. Application patching must be explicitly enabled on the VM through Windows Update settings or PowerShell registration of the Microsoft Update service. Patch approvals, pre-patch scripts, and post-patch scripts further narrow exactly what runs on each VM.

1.4Schedules, Maintenance Windows, and Zone Rollout Budgets

Schedules and rollout budgets are the controls that limit when patching starts and how many VMs change at once. Patch deployments (opens in a new tab) can run on demand with Start now, at a future one-time slot, or on a recurring schedule, and an in-progress job can be cancelled. If a job is cancelled or times out, the OS Config agent finishes its current task before stopping rather than aborting mid-package. Targeting all VMs at once risks availability loss, so production practice is to patch in small batches staggered across zones and regions.

Two timing controls enforce safety. A maintenance window sets the total duration in which new per-VM patch tasks may start, for example 60 minutes; downloads or reboots already in flight may extend past the window, but no new VM patching starts after it expires. An installation deadline sets a date by which targeted VMs must be patched; already-compliant VMs take no action unless a pre-script, post-script, or reboot is required. Rollout options then control concurrency: zone-by-zone mode completes one zone before starting the next, while concurrent-zones mode patches selected zones in parallel, and the disruption budget limits how many VMs in a zone may be disrupted at one time as a count such as 10 or a percentage such as 25% or 50%.

1.5Reboot Options and Pre-Patch and Post-Patch Scripts

Reboot handling determines whether a patch fully takes effect on the VM. The reboot options are Default, where the agent checks well-known OS signals and may reboot multiple times including before patches install; Always, where the VM reboots after updates complete; and Never, where the VM does not reboot and some patches may remain incompletely applied. Choose Never only when an external orchestrator handles reboots, because security updates often require a restart.

Pre-patch and post-patch scripts run application-aware steps around the OS update in a fixed order: first pre-patch script, then patching and reboots, then post-patch script. Pre-patch scripts run before patching starts and before any pre-patch reboot if one is required, which suits draining connections or stopping services. Post-patch scripts run after patching completes and after any patch reboot, which suits health checks or restarting services. A job accepts one Linux pre-patch script, one Linux post-patch script, one Windows pre-patch script, and one Windows post-patch script, with Linux scripts running only on Linux VMs and Windows scripts only on Windows VMs.

Linux scripts must start with a shebang line such as #!/bin/bash. Scripts can live on the VM or in a versioned Cloud Storage bucket, and the console uses the latest object version by default. If the object is not public, the VM service account needs read permission on the object, and buckets must sit in regions allowed by any Resource Location Restriction organization policy.

1.6Patch Compliance Reporting and Severity Mapping

Patch compliance reporting shows patch debt independently of whether a job recently ran. After setup, the OS Config agent reports OS inventory data about every 10 minutes, the patch compliance backend reads that inventory and cross-references it with distribution package metadata, and the console displays the result on the Patch dashboard. Dashboard data lags job start by about 30 minutes, so immediate verification should use per-instance job details instead.

The backend generates compliance status in an ordered sequence:

1. Read inventory: Collect the installed and available package versions reported by the OS Config agent.
2. Fetch vulnerability classifications: Pull severity data from the distribution-specific vulnerability source and order it from highest to lowest severity.
3. Map to Google status: Translate distribution severities into Critical (RED), Important/Security (ORANGE), Other (YELLOW), or Up-to-date (GREEN).
4. Publish highest severity: Display the highest severity for each available update on the dashboard and full CVE details on the VM details page.

Distribution severities map consistently: Critical, Urgent, and WINDOWS_CRITICAL_UPDATE become Critical; Important, High, and WINDOWS_SECURITY_UPDATE become Important/Security; everything else becomes Other; and no available updates becomes Up-to-date. A single available package can carry multiple CVEs, so a RHEL VM offered version 2.0 that fixes one Critical CVE plus two Low CVEs still appears as Critical with three CVEs listed. Vulnerability sources differ by OS and include Red Hat security data for RHEL and CentOS, Debian security tracker, Ubuntu CVE tracker, SUSE update feeds, Rocky Linux errata, and the Windows Update Agent API for Windows.

1.7In-Place Patching Limits for Managed Instance Group VMs

Managed instance group (MIG) patching is special because the MIG controller owns VM lifecycle separately from Patch. By default, VM Manager skips MIG members; a job patches them only when migInstancesAllowed is set to true through the Allow patching MIG instances option. A second flag, skipUnpatchableVms, skips VMs whose OS does not support patching, such as Container-Optimized OS, and MIG members when MIG patching is not allowed. Skipped VMs report SKIPPED, and a job with only skipped VMs and no other errors ends as COMPLETED_WITH_INACTIVE_VMS rather than an error state.

In-place OS patching does not update the MIG instance template, so MIG repair and autoscaling can undo patch work. When a MIG repairs a VM, it recreates the VM from the template and may revert it to an unpatched state. When autoscaling scales in, it can delete patched VMs, and when it scales out, it creates new VMs from the unpatched template. For durable upgrades, update the instance template and roll the MIG rather than relying only on in-VM patching, tuning surge and unavailable limits to keep capacity during the roll.

MIG configuration rollouts use a separate update type from Patch disruption budgets. A proactive MIG update (opens in a new tab) automatically applies a new template to existing VMs, while an opportunistic update (opens in a new tab) applies the new configuration only to new VMs and to existing VMs an operator explicitly selects or recreates. Use opportunistic updates to control timing and sequence for stateful, repaired, or autoscaled workloads, then selectively update chosen instances and let scale-in preferentially remove old-configuration VMs while scale-out creates new-configuration VMs.

1.8Identity Access and Single-Project Operational Limits

Identity access for patching separates who can execute changes from who can only observe them. Project owners have full patch rights, while other users need explicit Identity and Access Management (IAM) roles. The roles/osconfig.patchJobExecutor role can run, cancel, get, and list patch jobs plus view instance details, while roles/osconfig.patchJobViewer is read-only for get, list, and instance details. The executor permission includes osconfig.patchJobs.exec, which can manage software and run code on VMs, so grant it under least privilege and audit its use.

Patch deployments use roles/osconfig.patchDeploymentAdmin for create, delete, get, and list, and roles/osconfig.patchDeploymentViewer for read-only access. Operational scope is also bounded: patch jobs run only within a single Google Cloud project and cannot span projects even with Shared VPC, although compliance data can be viewed across projects. Deployment names must be unique per project, use lowercase letters, numbers, and hyphens, start with a letter, end with a letter or number, and stay within 1 to 63 characters.

Multi-environment lifecycle promotion is the practice of validating an operating system patch in a lower environment such as staging and then promoting the same patch baseline sequentially to production, with fast reversion if health degrades. In Google Cloud, VM Manager (opens in a new tab) divides this work into two cooperating services: patch deployment executes the change on target virtual machines (VMs), and patch compliance verifies the resulting state for promotion decisions. Progressive rollout controls, compliance status, and infrastructure health monitoring together determine whether promotion continues or a rollback is triggered.

2.1Progressive Promotion Sequence from Staging to Production

Progressive promotion is the ordered expansion of a change from a small canary scope to full production only after workload behavior is verified. A safe sequence first patches a few VMs in one zone, monitors the effects, and only after verifying that the workload runs as expected expands zone by zone to production. If any issue is observed, the operator reverts quickly to the previous stable state, diagnoses and resolves the issue, and then restarts the progressive process from the beginning.

Patch deployments and patch jobs enforce this sequence through instance filters, scheduling, and rollout options. Instance filters select targets by zone, name prefix such as test-, and labels such as env=dev and app=web, which allows the same patch configuration to promote from staging labels to production labels without redefining the job. Filters are evaluated at the time each scheduled job runs, so VMs added after the deployment was created are still included; for example, a deployment created when a zone had 20 VMs will patch all 60 VMs if 40 more are added before run time. Rollout options then control blast radius by selecting whether to patch one zone at a time or patch zones concurrently, plus a disruption budget that limits the number or percentage of VMs in a zone disrupted at one time.

2.2Compliance Gates That Control Promotion Decisions

Patch compliance is the verification service that determines whether a VM still needs updates after a patch run and assigns a severity-based status used as a promotion gate. After VM Manager is set up, the OS Config agent reports OS inventory data about every 10 minutes, and the patch compliance backend periodically reads that inventory, cross-references it with package metadata from the operating system distribution, and saves the result. The Google Cloud console then displays the saved compliance data on the dashboard and on the VM details page, which shows the full report of available updates for that VM.

The patch compliance backend generates compliance status through an ordered sequence:

1. Read inventory: The backend reads the OS inventory reports collected from the VM.
2. Scan vulnerability sources: The backend scans classification data from the vulnerability source for each operating system and orders the findings from highest to lowest severity.
3. Map to Google status: The backend maps each distribution classification to a Google patch compliance status.
4. Select highest severity: The backend selects the highest severity update for each VM for the dashboard view.

Vulnerability classifications come from the authoritative source for each operating system:

• RHEL and CentOS use Red Hat security data, based on the latest minor version for each major version, which can make results for older minor versions inaccurate.
• Debian uses the Debian security tracker.
• Ubuntu uses the Ubuntu CVE tracker.
• SLES uses SUSE update data.
• Rocky Linux uses Rocky Linux errata.
• Windows uses classification data from the Windows Update Agent API.

The mapping from distribution categories to Google status preserves severity so staging and production can share one gate:

Promotion should advance only when the lower environment reaches Up-to-date (GREEN) or an approved residual risk level, because RED and ORANGE indicate missing critical or security updates that must not be promoted to production.

2.3Health Monitoring and Rollback Decisions for Global Resources

Health monitoring for promotion combines workload health, patch progress, and Google Cloud infrastructure availability to decide between continued rollout and rollback. The Patch dashboard tracks per-VM patch execution, while infrastructure health monitoring (opens in a new tab) through Personalized Service Health and the Google Cloud Service Health Dashboard tracks infrastructure incidents relevant to the project, including incident duration, affected zones and regions, affected services, and workarounds. Service Health also exposes incident data through an API and alerts and through an RSS feed, so an operator can correlate a failed canary with a platform incident before deciding to roll forward or revert.

Reversion restores the previous stable infrastructure state when monitoring shows degradation after a patch or configuration change. For VM machine-type or similar changes, the documented recovery is to revert quickly, diagnose the failure, and restart progressive deployment only after the fix is verified. For global resources such as VPC networks and global load balancers, extra control is required because a single global instance can become a single point of failure; for example, a misconfigured forwarding rule can stop the frontend from receiving user requests while the backend remains intact. Change review should therefore classify modifications to global resources as high-risk changes that require additional reviewer verification and approval before production promotion.

2.4Deployment Scheduling and Safe Handling of Special VMs

Patch deployment scheduling is the object that runs patch jobs automatically on a one-time or recurring schedule, while a patch job (opens in a new tab) can also be run on demand for immediate validation. Each patch deployment (opens in a new tab) name must be unique within the project, contain only lowercase letters, numbers, and hyphens, start with a letter, end with a letter or number, and be between 1 and 63 characters. Scheduling supports a duration or maintenance window that bounds when patching may disrupt the VM, plus reboot options and optional pre-patch and post-patch scripts for quiescing, validation, or service restart around the update. The Patch service does not host package updates, so staging and production networks must each provide a trusted path from VMs to the approved source via public egress, local repository or WSUS, Cloud NAT, or proxy.

Safe execution also depends on correct handling of managed and unpatchable VMs and on least-privilege access. Patch jobs can optionally allow patching of managed instance group (MIG) instances and can skip unpatchable VMs such as VMs running Container-Optimized OS, which prevents a promotion job from failing on images that cannot be patched in place. For Windows VMs, disabling automatic updates reduces conflicts between Windows automatic updates and the Patch service. Access is governed by granular Identity and Access Management roles: roles/osconfig.patchDeploymentAdmin creates and manages deployments, roles/osconfig.patchDeploymentViewer provides read-only deployment access, roles/osconfig.patchJobExecutor runs and manages jobs, and roles/osconfig.patchJobViewer provides read-only job access, with the executor role requiring least-privilege care because osconfig.patchJobs.exec permits managing software packages and therefore running code on VMs.

Secure Google Kubernetes Engine (GKE) control plane and node pool upgrades are managed by qualifying a new version in one environment before rolling it to the next. In Google Kubernetes Engine (GKE) (opens in a new tab), rollout sequencing groups clusters into ordered stages so early stages validate the version while later stages wait. The automated rollout is driven by release channel targets, constrained by node upgrade strategy and available maintenance time, and controlled with pause, resume, and notification signals.

3.1Release Channels as Upgrade Targets in Rollout Sequences

Release channels in GKE rollout sequencing are the source that selects the upgrade target for each cluster. All clusters in a rollout sequence should be enrolled in the same release channel to keep targets consistent across stages. The sequence advances only after a version becomes the designated auto-upgrade target for that channel. Manually upgrading clusters in the first stage updates those clusters but does not qualify the version or trigger the rollout to proceed to later stages.

3.2Node Pool Strategies and Thirty-Day Soak Timing

Node pool upgrade strategy in GKE determines how long node upgrades take and whether a stage can finish before its soak phase. GKE uses surge upgrades for Autopilot nodes, while Standard clusters can use strategies such as blue-green upgrades that may take longer, especially for a large node pool. Blue-green upgrades safeguard running workloads by preparing new nodes before shifting workloads, while surge upgrades add extra capacity during the change, but both consume maintenance time. If node upgrades cannot complete within 30 days, the group enters its soak phase even when some clusters have not finished upgrading.

This incomplete upgrade is more likely when the chosen strategy is slow and when maintenance windows are not large enough for the node upgrade to complete. Maintenance windows and exclusions therefore directly bound upgrade safety: narrow windows leave less time for node replacement, and staging versus production windows must each allow the chosen surge or blue-green strategy to finish. Safe planning matches the strategy to the window size, validates the slower blue-green path in staging first, and only then promotes the same version to production.

3.3Concurrent Versions, Notifications, and Pause Controls

Concurrent versions across a sequence occur because each stage follows the current release channel target at the time it starts. If a new version becomes available while upgrades to a previous target are still in progress, the first stage can begin the newer rollout while later stages still receive the older upgrade. For example, the third group in a sequence can be rolling out version 1.31.12-gke.1265000 while the first group concurrently rolls out version 1.31.13-gke.1008000. This behavior lets qualification continue without waiting for every downstream stage to catch up.

Rollout control and observation in GKE depend on cluster notifications, deprecation checks, and explicit rollout actions. GKE sends cluster-level upgrade notifications plus sequence-level RolloutEvent and RolloutSequenceEvent signals when a rollout stage starts, completes, is blocked, or is misconfigured. GKE deprecation usage detection still applies during sequencing and can pause upgrades on clusters that use a deprecated API. Together these signals let operators see which version each stage is receiving and why a rollout has stopped.

Pausing a rollout in GKE temporarily stops new upgrade work without stopping work already in progress. When a rollout is paused, GKE will not start new upgrade operations to that version, including operations for subsequent stages, so teams can investigate a potential issue with clusters and the new version. After pausing, the rollout can either resume or be canceled, and a rollout can remain paused for up to 90 days.

Key Facts

• VM Manager Patch does not host OS packages, so each Compute Engine VM must reach its distribution source directly or through a local repository, WSUS, Cloud NAT, or proxy.
• A single VM Manager patch job patches a maximum of 500 VMs simultaneously and evaluates its instance filter at job start time.
• Patch compliance maps Critical, Urgent, and WINDOWS_CRITICAL_UPDATE to Critical (RED) and Important, High, and WINDOWS_SECURITY_UPDATE to Important/Security (ORANGE).
• In-place OS patching does not update a managed instance group (MIG) instance template, so MIG repair, autoscaling, and scale-out can revert or replace patched VMs.
• A Google Kubernetes Engine (GKE) rollout sequence advances only after a version becomes the auto-upgrade target for the release channel, and a paused rollout can remain paused for up to 90 days.