Ephemeral Projects, Policy Controls and GKE Fleets

Managing ephemeral environments is the practice of defining short-lived, isolated Google Cloud environments in code and creating them on demand through automated pipelines. Infrastructure as Code (IaC) declares virtual machines, networks, databases, storage, and identity in versioned files, and continuous integration and continuous deployment (CI/CD) executes those files to build, test, and tear down the same environment repeatedly. Each environment is created by a trigger or schedule and destroyed when work completes, so development, testing, and preview work cannot leak into staging or production.

1.1Automated Provisioning with Infrastructure as Code and Terraform Tooling

Automated provisioning with Infrastructure as Code replaces manual console clicks with human-readable configuration that declares desired state. The same versioned configuration can be deployed many times to produce matching behavior for a feature branch, pull request, or load test, and every change keeps history that can be audited and reverted. Because configuration lives in source control, infrastructure changes follow the same review and validation discipline as application code, with one repository as the single source of truth.

The Terraform toolchain on Google Cloud centers on authoring, native execution, and organization-wide change management:

HashiCorp Terraform (opens in a new tab) workspaces connect to GitHub, GitLab, and Bitbucket so commits can trigger plans, while Infra Manager depends on the Terraform configuration as its input. Curated Google Kubernetes Engine (GKE) (opens in a new tab) blueprints, Cluster Toolkit blueprints, and Google Cloud modules for BigQuery, Pub/Sub, Cloud SQL, and load balancing provide ready starting points that teams customize in version control. In Kubernetes designs, Config Connector manages Google Cloud resources as Kubernetes objects, so application and infrastructure update together.

1.2Isolated Identity, Network, and Policy Design for Short-Lived Projects

Isolated design for ephemeral use gives each short-lived environment its own boundary for identity, network, and data. A common pattern provisions a dedicated project per instance with the project-factory module, which creates Shared VPC attachment, Identity and Access Management (IAM) bindings, and enabled APIs. The network module then creates a new Virtual Private Cloud (VPC) network inside that project, while folders, organization policy, and VPC Service Controls modules place the project under the correct hierarchy with perimeter guardrails.

Trust flows from identities to resources through IAM, where the pipeline runs as a dedicated service account with least-privilege roles and workloads receive their own service accounts. Secrets and encryption are declared in code with secret-manager and kms modules, and traffic from the ephemeral VPC flows only to explicitly allowed Google APIs, with Cloud NAT, Cloud DNS, and Secure Web Proxy controlling egress. Organization Policy constraints constraints/iam.disableServiceAccountKeyCreation and constraints/iam.disableServiceAccountKeyUpload block downloadable keys, forcing short-lived credentials. VPC Service Controls checks whether resource and caller are inside the same perimeter for users and service accounts, while Chrome Enterprise Premium checks per-request user and device context only for organization users. Hardened access uses OS Login with 2-Step Verification for Compute Engine SSH, while Privileged Access Manager (PAM) provides time-bound elevation with audit logs, and emergency access uses two to five super-admin accounts protected by FIDO keys in a dedicated organizational unit.

1.3Triggered Lifecycle, Cleanup, and Cost Attribution for Ephemeral Work

The triggered lifecycle for ephemeral work preserves order from request to cleanup through Cloud Build, Artifact Registry, and Cloud Deploy. Cloud Build imports source from Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket and produces artifacts such as containers, Artifact Registry stores versioned packages and images, and Cloud Deploy promotes the same tested artifact through dev to staging to prod without rebuilding:

1. Trigger: A commit, pull request, schedule, or API call invokes a Cloud Build trigger through a second-generation or Developer Connect connection.
2. Validate and plan: The pipeline runs checks such as Terraform validate and plan before any infrastructure changes state.
3. Apply: The pipeline creates the isolated project, network, service accounts, buckets, and workloads such as GKE clusters or Cloud Run services.
4. Test and publish: Tests run inside the ephemeral environment and outputs are stored in Artifact Registry with logs in a Cloud Storage workdir bucket.
5. Promote or destroy: The pipeline promotes the same configuration with distinct dev, staging, and prod variables, then deletes the project to stop cost.

Resource cleanup and cost control depend on project isolation plus scheduled teardown:

• Enable lifecycle management for Cloud Storage buckets to age out test datasets.
• Configure deletion rules with isLive set to False so only noncurrent versions are removed.
• Create cleanup policies in Artifact Registry to remove stale images, and use SetStorageClass rules to move aging data to cheaper classes.

Cost attribution exports Cloud Billing data to a BigQuery dataset in the prj-c-billing-export project and groups charges by the billingcode project label for chargeback. Budgets use alert thresholds for workload projects, firing when forecasted spending reaches 120% of budget, which can also signal a security incident.

Managing configuration and policy means every Google Cloud environment inherits the same organization-level guardrails and desired state without manual checks in each project. The Organization Policy Service centrally defines restrictions that apply to resources and their descendants in the Google Cloud resource hierarchy, while fleet tools distribute configuration and enforce admission. Policies attach at organization, folder, or project level and are inherited by descendants unless inheritance is changed, so a violating create or update fails with an error.

2.1Organization Constraints, Guardrails, and Conditional Exceptions

Organization constraints provide centralized, programmatic control where a constraint defines the restriction and the policy attaches it to a hierarchy node. Managed constraints apply built-in best practices such as blocking external IPs, while custom constraints restrict specific fields and operations before a resource persists, such as machine family, disk type, label values, firewall ports, or sole-tenant requirements. The request flows from caller to Google Cloud API to Organization Policy Service evaluation, and only compliant resources are created.

Custom constraints span compute, containers, data, and automation, including compute.googleapis.com/Instance, container.googleapis.com/Cluster, Spanner instances and databases, BigQuery Data Transfer configs, Cloud Build builds and worker pools, and Backup and DR vaults. Foundational Google Cloud-wide constraints establish residency, encryption, and service boundaries:

Exceptions must be narrow and compensated, such as allowing iam.disableServiceAccountKeyCreation relief for an on-premises server that cannot use Workload Identity Federation, using tags on projects and folders for conditional enforcement. Organization-wide changes should first be validated in a separate test organization, and VPC Service Controls perimeters should start in dry-run mode before enforcement, with break-glass access for pipeline recovery.

2.2Declarative Export and Isolated State with Terraform Bulk Export

Declarative export with Terraform bulk export converts live resources into versioned code with separated state. Each environment, typically a separate project for development, staging, and production, is exported to its own output directory and state, so future changes are managed through code and drift appears as a difference from code. Virtual Private Cloud (VPC) (opens in a new tab) and project boundaries keep each export isolated.

The end-to-end workflow moves resources from live infrastructure into managed code in order:

1. Prepare isolated context: Launch Cloud Shell, set export GOOGLE_CLOUD_PROJECT=PROJECT_ID for one project at a time, and install dependencies.
2. Bulk-export configuration: Create a dedicated directory and run gcloud beta resource-config bulk-export --path=OUTPUT_DIRECTORY --project=PROJECT_ID --resource-format=terraform.
3. Generate reusable modules: Run gcloud beta resource-config terraform generate-import OUTPUT_DIRECTORY to create gcloud-export-modules.tf and an import shell script.
4. Import into isolated state: Run terraform init then execute the import script to bind live IDs, such as instance paths, into state.

Preparation depends on the Config Connector CLI installed with gcloud components install config-connector and on the Cloud Asset API enabled with gcloud services enable cloudasset.googleapis.com. Bulk export is not supported on Windows, so the workflow runs from Cloud Shell.

2.3GitOps Fleet Configuration with Config Sync and Policy Controller

GitOps fleet configuration defines desired cluster state once centrally and continuously pushes it to every environment. GKE attached clusters registers externally created clusters as centrally managed resources through an AttachedCluster object, so staging, production, and external clusters receive the same fleet-wide configuration without per-cluster edits. Once attached, fleet tools operate from Google Cloud toward member clusters and reapply the fleet definition when drift is detected.

Config Sync handles what clusters should look like, while Policy Controller handles what clusters are allowed to do, including PodSecurity admission at scale. The secure control plane uses a regional AttachedCluster resource that keeps metadata in the selected region for data residency, plus a Connect Agent inside the external cluster that connects outward to the Connect API to create a secure tunnel. Management traffic then flows from Google Cloud through the tunnel to the cluster without separate inbound paths, with Connect Gateway providing single-interface access.

Managing Google Kubernetes Engine (GKE) enterprise fleets means joining different Kubernetes clusters into one fleet with shared identity and management. Registration creates a membership record for each cluster, while federation lets workloads exchange Kubernetes credentials for Google Cloud credentials. Together they allow centralized configuration, policy, networking, and secure access across staging, production, and external environments.

3.1Fleet Registration, Membership, and Secure Tunnel Connectivity

Heterogeneous registration with GKE attached clusters (opens in a new tab) joins any conformant Kubernetes cluster, including Amazon EKS and Azure AKS clusters, into the GKE management environment. Registration creates an AttachedCluster resource in the GKE Multi-Cloud API as a regional resource, and all cluster metadata is stored in that region to support data residency. Once attached, the cluster is operated through the Google Cloud CLI or console with the same fleet tooling as native GKE clusters.

Fleet membership is the centralized record that makes a registered cluster part of a fleet, where each cluster receives a unique membership name used for operations. Operators verify with gcloud container fleet memberships list or describe MEMBERSHIP_NAME, and the fleet host project acts as the trust boundary across multiple projects. Connect Agent is the workload inside each cluster that connects back to the Connect API to establish a secure tunnel, so management traffic flows from the Google Cloud control plane through the tunnel to the Kubernetes API server without a direct inbound path. The API server authorizes the agent through RBAC impersonation and permissions policies, and Connect Gateway then provides authenticated access through the console and CLI.

Attaching unlocks fleet-wide tools for configuration, policy, networking, and access:

• Config Sync deploys cluster configurations across the fleet.
• Policy Controller automates policy and security across all clusters.
• Cloud Service Mesh monitors and manages reliable service communication.
• Connect Gateway provides access to all clusters through a single interface.

3.2Workload Identity Federation, Service Agents, and Support Access

Workload Identity Federation for GKE lets GKE workloads exchange Kubernetes credentials for Google Cloud credentials for secure authentication. Enabling federation depends on the Kubernetes Engine Service Agent, which manages instances, disks, and load balancers, and fails with DeployPatch failed when its role is missing. Both Google-managed agents must hold their roles on the project, resolved with gcloud projects describe and verified with gcloud projects get-iam-policy.

The agents require roles/container.serviceAgent and roles/container.defaultNodeServiceAgent, restored with gcloud projects add-iam-policy-binding or by re-enabling the Kubernetes Engine API. In organizations created on or after May 3, 2024, the iam.automaticIamGrantsForDefaultServiceAccounts constraint blocks automatic grants, so least-privilege roles must be granted explicitly.

Temporary support access grants Google Support read-only access to registered external clusters after fleet membership exists and connectgateway.googleapis.com is enabled. Enabling follows an ordered workflow:

1. Enable the gateway API: Run gcloud services enable connectgateway.googleapis.com --project=PROJECT_ID.
2. Enable access: Run gcloud container fleet memberships support-access enable MEMBERSHIP_NAME --project=PROJECT_ID, which requires gkehub.rbacrolebindings.create.
3. Verify access: Run gcloud container fleet memberships support-access describe MEMBERSHIP_NAME --project=PROJECT_ID.

The support account can list pods and check image pulls and node status but cannot change the cluster, and access is removed after the case closes.

3.3Cross-Cluster Discovery and Global Ingress Traffic Management

Cross-cluster traffic management combines east-west Pod-to-Pod calls with north-south external access through one global entry point. Multi-cluster Services (MCS) lets Pods discover Services in other fleet clusters, while Multi-cluster Ingress (MCI) exposes applications in multiple clusters through one external load balancer virtual IP (VIP). Both depend on fleet membership as the shared control boundary.

MCS exports a Service across the fleet so it becomes available for highly available, geographically distributed use, with initial sync in about five minutes and later endpoint updates immediately. MCS automates Cloud DNS records for SERVICE_EXPORT_NAME.NAMESPACE.svc.clusterset.local, firewall rules for Pod-to-Pod communication, and Traffic Director endpoint health. Requirements bound where traffic can flow: clusters must be VPC-native and share the same VPC or connected networks via Peering, Interconnect, VPN, or Shared VPC, each exported service belongs to one project and one VPC, the HttpLoadBalancing add-on must stay enabled, and network policies are not supported. Enabling creates the gcp-sa-mcsd service account, requires six APIs including multiclusterservicediscovery.googleapis.com and trafficdirector.googleapis.com, grants roles/compute.networkViewer to the gke-mcs-importer, and uses the same namespace in every cluster. Only ClusterSetIP and headless Services with DNS A records are supported.

MCI uses one config cluster as the central API server, where MultiClusterService objects select Pods by labels and member clusters across the fleet to create Network Endpoint Groups (NEGs), and a MultiClusterIngress object then creates Compute Engine load balancer resources exposing all backends through one VIP. Traffic flows from internet clients to the VIP, then to NEG backends pointing directly to Pod IPs.

3.4Fleet-Wide Synchronization, Enforcement, and Regional Control

Fleet-wide synchronization and enforcement operates many GKE clusters as one fleet with consistent state and security. Config Sync is the fleet service that deploys desired cluster configurations to every member so staging and production converge, correcting drift toward the declared state. Policy Controller is the fleet service that applies declarative security constraints consistently to GKE, EKS, AKS, and on-premises clusters, surfacing noncompliant drift regardless of location.

The AttachedCluster resource represents each external cluster in the hierarchy as a regional resource, keeping metadata in the selected region for compliance. The Connect Agent deployed inside the external cluster connects outward to the Connect API to create the secure tunnel that Config Sync, Policy Controller, and Cloud Service Mesh use, so central teams manage clusters without exposing separate inbound management paths.

Safe and secure patching and upgrading practices validate a change in a lower environment and roll it to production in small batches with fast reversion. In Google Cloud, VM Manager divides operating system work into patch deployment, which schedules and controls jobs, and patch compliance reporting, which verifies remaining updates. Both depend on the OS Config API in the control plane and the OS Config agent on each virtual machine (VM).

4.1Operating System Patch Deployment from Control Plane to Virtual Machines

Patch deployment with VM Manager (opens in a new tab) starts when an operator calls the OS Config API from the console, CLI, or API, which notifies the agent on each targeted VM. The agent invokes the native updater — yum upgrade on RHEL, Rocky, and CentOS, apt upgrade on Debian and Ubuntu, zypper update on SLES, and Windows Update Agent on Windows — then reports progress back where job status is visible. A patch deployment schedules one or more patch jobs, while a patch job is the single execution that patches VMs matched at start time. Setup requires the OS Config API enabled and the agent installed with inventory enabled, or calls return NO_AGENT_DETECTED or TIMED_OUT.

The Patch service does not host patches, so each VM must reach its distribution source directly or through a local repository, Windows Server Update Service (WSUS), Cloud NAT, or proxy. Pay-As-You-Go (PAYG) images for RHEL and SLES cannot host a local mirror, which matters for private VPC designs. Fleet targeting selects VMs by zone, name prefix, and labels such as env=dev at job start time, so recurring jobs include VMs created after scheduling, but a single job patches a maximum of 500 VMs simultaneously. Selectable content differs by OS, with Windows offering definition, driver, feature pack, security, and tool updates, while application patching for SQL Server or .NET must be explicitly enabled.

Schedules and rollout budgets limit when patching starts and how many VMs change at once, using on-demand, one-time, or recurring schedules with cancellable jobs. A maintenance window bounds when new per-VM tasks may start, such as 60 minutes, while an installation deadline sets the date by which targets must be patched. Zone-by-zone mode completes one zone before the next, concurrent-zones mode patches zones in parallel, and the disruption budget limits disrupted VMs per zone by count or percentage. Reboot options are Default, Always, and Never, and pre-patch scripts run before patching and any pre-reboot while post-patch scripts run after patching and any reboot, with Linux scripts requiring a shebang and Cloud Storage objects requiring read permission.

4.2Compliance Gates and Progressive Promotion from Staging to Production

Progressive promotion patches a few VMs in one zone, monitors workload behavior, then expands zone by zone to production, reverting quickly if health degrades. Instance filters by zone, prefix such as test-, and labels let the same configuration promote from staging to production, and filters evaluate at run time so later-added VMs are included. Patch compliance is the verification gate: the agent reports inventory about every 10 minutes, the backend cross-references distribution metadata, and the console shows status on the Patch dashboard with about 30 minutes lag.

The backend generates status in order: read inventory, scan vulnerability sources ordered highest to lowest, map to Google status, and publish the highest severity per VM:

Sources include Red Hat security data, Debian tracker, Ubuntu CVE tracker, SUSE feeds, Rocky errata, and Windows Update API. Promotion should advance only from GREEN or approved residual risk. Health monitoring adds Personalized Service Health and Service Health Dashboard incidents, API alerts, and RSS feeds to distinguish platform incidents from bad patches. For global VPC networks and load balancers, classify changes as high-risk with extra approval because one forwarding-rule error can stop frontends.

In-place patching does not update a managed instance group (MIG) template, so repair and autoscaling can revert work. Create patch jobs (opens in a new tab) with migInstancesAllowed set to true to include MIG members, and skipUnpatchableVms to skip Container-Optimized OS, reporting SKIPPED and ending as COMPLETED_WITH_INACTIVE_VMS when only skips remain. For durability, update the template and roll with proactive updates to existing VMs or opportunistic updates to new VMs only. Access separates roles/osconfig.patchJobExecutor and roles/osconfig.patchDeploymentAdmin for change from read-only viewer roles, jobs run only within one project, and deployment names must be 1 to 63 lowercase letters, numbers, and hyphens starting with a letter.

4.3Secure GKE Control Plane and Node Pool Upgrade Sequencing

Secure GKE control plane and node pool upgrades qualify a new version in one environment before rolling it to the next. Rollout sequencing groups clusters into ordered stages where early stages validate while later stages wait. All clusters in a sequence should share the same release channel, and the sequence advances only after a version becomes the designated auto-upgrade target for that channel. Manually upgrading the first stage does not qualify the version or trigger later stages.

Node strategy and maintenance windows determine whether a stage finishes before its soak phase. Autopilot uses surge upgrades that add capacity, while Standard can use blue-green upgrades that prepare new nodes before shifting workloads but take longer for large pools. If upgrades cannot complete within 30 days, the group enters soak even when some clusters are unfinished, which is more likely with slow strategies and narrow windows. Concurrent versions occur because each stage follows the channel target at start time, so one group can roll 1.31.12-gke.1265000 while an earlier group concurrently rolls 1.31.13-gke.1008000. Schedule patch jobs (opens in a new tab) and GKE notifications expose RolloutEvent signals, deprecation checks can pause upgrades on deprecated APIs, and pausing stops new operations without stopping in-progress work for up to 90 days before resume or cancel.

Key Facts

• Ephemeral environments use versioned Infrastructure as Code and CI/CD triggers to create an isolated project per instance and destroy the project boundary to stop cost.
• Organization Policy constraints constraints/iam.disableServiceAccountKeyCreation and constraints/iam.disableServiceAccountKeyUpload block downloadable service account keys across temporary projects.
• GKE attached clusters registers external clusters as regional AttachedCluster resources with a Connect Agent tunnel from the cluster to Google Cloud for fleet management.
• Multi-cluster Services (MCS) exposes ClusterSetIP and headless Services through SERVICE_EXPORT_NAME.NAMESPACE.svc.clusterset.local, while Multi-cluster Ingress (MCI) exposes Pod backends through one external load balancer VIP with Network Endpoint Groups.
• A single VM Manager patch job patches a maximum of 500 VMs, the Patch service never hosts packages, and a GKE rollout sequence advances only after a version becomes the auto-upgrade target for the release channel.