Deploying to Hybrid and Multi-Cloud GKE Fleets

GitOps and declarative configuration management stores the desired state of clusters and applications as versioned files and uses automation to make every cluster match that state. In hybrid and multi-cloud environments, Google Kubernetes Engine (GKE) (opens in a new tab) clusters on-premises, at the edge, in Google Cloud, AWS, and Azure are grouped as fleets and kept consistent by Anthos Config Management (opens in a new tab) with Config Sync, the Kubernetes Resource Model, and customization tooling such as Kustomize.

1.1Declarative Configuration with KRM and Infrastructure as Code

The Kubernetes Resource Model (KRM) is a declarative approach where Kubernetes-style files define the desired state instead of manual commands. Automation reads KRM files and creates or updates infrastructure through the Kubernetes API, so the same file produces the same namespaces, role-based access control (RBAC), and workloads in every environment. This model supports infrastructure as code for project provisioning alongside tools such as Terraform, and Deployment Manager convert can translate Deployment Manager configurations and templates into Terraform and KRM formats to keep configurations portable. Because containers separate applications from host infrastructure and Kubernetes provides the same APIs in cloud and private environments, KRM manifests stay portable across heterogeneous clusters.

1.2Config Sync and Anthos Config Management as the Synchronization Engine

Config Sync is the Anthos Config Management component that deploys configuration from a source of truth to registered clusters. A commit to the central repository becomes the declared desired state, and Config Sync propagates that state from the centralized management plane in Google Cloud to each fleet member. Synchronization follows a continuous, ordered loop:

1. Declare: Authors commit declarative KRM manifests and environment customizations to the versioned source of truth.
2. Distribute: Config Sync pulls the declared configuration from the source of truth to each connected cluster.
3. Apply: Each cluster applies the manifests through its Kubernetes API server to create or update namespaces, workloads, and cluster-scoped configuration.
4. Reconcile: Agents continuously compare live cluster state to desired state and reapply configuration to correct drift.

Deployment drift is any difference between live state and declared state, such as a manually edited Deployment, an added namespace, or a changed RBAC binding. Automated reconciliation treats drift as a signal to remediate, so clusters converge back without per-cluster kubectl fixes. Centralized monitoring with Google Cloud Observability, including Cloud Logging and Cloud Monitoring, provides visibility into sync status across cloud and edge.

1.3Fleets, Sameness, and Consistent Customization Across Environments

A GKE fleet is a logical grouping that permits management of sets of GKE clusters as a unit with consistent security, configuration, and service management. Fleet membership enables sameness, which means Kubernetes objects with the same name in different clusters are treated as the same thing across namespaces, services, and identity. This normalization lets one declarative package address development, test, and production without renaming objects per cluster. Kustomize-style customization preserves consistency while allowing controlled differences, where a shared base defines common state and small overlays adjust image tags, replica counts, resource requests, or load balancer behavior that varies with compute, networking, storage, or proximity to dependencies. Administrative access flows from operators through the Connect gateway (opens in a new tab), which fetches credentials for multiple clusters with one command and integrates groups and third-party identity providers.

The choice of where a fleet member runs determines which hybrid or multi-cloud construct to use:

1.4Consistent Delivery Pipelines That Feed GitOps

Consistent continuous integration and continuous delivery (CI/CD) pipelines ensure the exact same binaries, packages, or containers deploy across computing environments. The blueprint pattern uses layered pipelines where the foundation infrastructure pipeline deploys the application factory, fleet-scope pipeline, and multi-tenant infrastructure pipeline, and the application CI/CD pipeline builds and deploys application components in a controllable, auditable, and repeatable way. When workloads must deploy identically, a CI system such as Tekton implements a deployment pipeline that works across environments. Pipeline stages flow from source code to fleet-wide desired state through Cloud Build (opens in a new tab), Cloud Deploy, and Artifact Registry for managed build, delivery, and artifact storage, while immutable containers signed with Binary Authorization preserve code provenance from test to promotion. Supporting controls harden that flow with private control-plane access and private node pools, Cloud NAT for outbound pod access without direct exposure, Container-Optimized OS and Shielded GKE Nodes, Kubernetes RBAC with Google Groups for GKE, and Workload Identity Federation for GKE.

Secure hybrid delivery pipelines centralize build artifacts, verify their integrity, and deliver them to clusters inside and outside Google Cloud without embedding secrets in delivery artifacts. Artifact Registry (opens in a new tab) centralizes artifacts, Cloud Build (opens in a new tab) produces verifiable builds with provenance, Cloud Deploy (opens in a new tab) promotes releases to GKE, GKE Enterprise, attached clusters, and Cloud Run, and external secret managers inject credentials at runtime.

2.1Centralize Artifacts with Artifact Registry

Artifact Registry is a universal package manager that stores all build artifacts and dependencies in one place for visibility and control of the software supply chain. Centralization exposes direct and transitive dependencies, which matters because risks such as the Log4j vulnerability spread through indirect dependencies. Artifacts are protected with access control, Virtual Private Cloud (VPC) Service Controls perimeters, organizational policies, and other security features. Repository design further reduces supply-chain risk:

• Remote repositories cache artifacts from preset external sources such as Docker Hub, Maven Central, the Python Package Index (PyPI), Debian or CentOS, and user-defined sources for supported formats, which reduces download time, improves availability, and includes vulnerability scanning if enabled.
• Virtual repositories consolidate repositories of the same format behind a single endpoint and control search order across upstream repositories, so private packages can be prioritized to reduce dependency confusion attacks.
• Perimeter and access controls restrict who and what can reach artifacts, including Identity and Access Management (IAM) permissions, VPC Service Controls perimeters, and organizational policies.

2.2Scan Builds and Enforce Deployment Policy

Artifact Analysis is the service that proactively detects vulnerabilities in artifacts stored in Artifact Registry. Artifact Analysis provides on-demand or automated scanning for base container images and language packages, generates a software bill of materials (SBOM), and supports uploaded Vulnerability Exploitability eXchange (VEX) statements for images. After the initial scan, Artifact Analysis continuously monitors image metadata for new vulnerabilities, updates metadata, creates new vulnerability occurrences for new notes, and deletes occurrences that are no longer valid. Cloud Build runs builds on Google Cloud infrastructure with granular IAM permissions, VPC Service Controls, and isolated and ephemeral build environments. Cloud Build supports Supply-chain Levels for Software Artifacts (SLSA) Level 3 builds for container images and generates authenticated, non-falsifiable build provenance that includes digests of built images, input source locations, toolchain, build steps, and duration. Binary Authorization then enforces policy at deployment time by attaching configurable signatures, called attestations, and checking them before GKE or Cloud Run admits an image:

• Verify that a specific build system or continuous integration pipeline created a container image.
• Validate that a container image complies with a vulnerability signing policy.
• Verify that a container image passes promotion criteria for the next environment, such as from development to quality assurance (QA).

2.3Deliver to Hybrid Targets with Cloud Deploy

Cloud Deploy automates delivery of applications to a series of target environments in a defined sequence, with support for GKE, GKE Enterprise, and Cloud Run. Cloud Deploy provides one-click approvals and rollbacks, enterprise security and audit, built-in delivery metrics, and security insights for deployed applications. For hybrid estates, the same model extends to GKE attached clusters, so external Kubernetes clusters receive releases through the same pipeline, approvals, and audit trail as Google Cloud clusters. Secure hybrid delivery depends on keeping pipeline software, infrastructure, and inputs at the same integrity level as the resources the pipeline manages. That means using trusted sources for Terraform, kubectl, and Docker, verifying authenticity, rebuilding infrastructure regularly, and running pipelines with private ingress or egress and VPC Service Controls. VPC Service Controls restrict the resources and application programming interfaces (APIs) accessible from specific Google Cloud projects, so even compromised code has limited ability to reach other resources and exfiltrate data.

2.4Inject Credentials with Centralized Secret Management

Centralized secret management keeps database credentials, API keys, certificates, and passwords outside deployment artifacts and injects them during multi-cloud deployment phases. Kubernetes Secrets store sensitive data separate from Pods and resemble ConfigMaps, but Kubernetes Secrets are not secure by default without encryption, so hybrid designs use external managers. Secrets for Cloud Deploy applications should be generated, managed, and rotated outside Cloud Deploy, with infrastructure provisioning creating or updating the secret before the application deploys:

Multi-cluster rollout orchestration is built around the fleet (opens in a new tab), a logical grouping of Kubernetes clusters and other resources managed together. The fleet is managed by the Fleet service, also known as the Hub service, and it uplevels management and observability from individual clusters to groups of clusters through a single fleet host project. Without fleets, a production-wide change across ten projects with two clusters each must be applied cluster by cluster with repeated context switching.

3.1Fleet Management for Hybrid and Multi-Cloud Deployments

Fleet management is the set of GKE capabilities for managing clusters, infrastructure, and workloads on Google Cloud and across public cloud and on-premises environments. Organizations adopt multiple clusters to achieve technical and business objectives that a single cluster cannot satisfy. Common drivers include:

• Separating production from non-production environments
• Separating services across tiers, locales, or teams
• Isolating failure domains across zones, regions, and providers
• Enabling team-based ownership while keeping central oversight

Fleet-based management normalizes diverse clusters so operators, workload developers, and security and network engineers can view and change them together. The fleet host project becomes the single place to observe the fleet, while team-based constructs and rollout controls determine how changes propagate from cluster to cluster. Fleet-enabled features abstract away cluster boundaries, such as defining team-owned resources across clusters or automating the same configuration fleet-wide.

3.2Cluster Registration and Secure Connectivity Across Environments

Cluster registration is the process of adding a cluster as a fleet member in a chosen fleet host project. GKE clusters on Google Cloud must be explicitly registered through quick registration in the console, the gcloud CLI, Terraform, Config Connector, or during cluster creation, either in their current project or in a different project with extra permissions. Clusters outside Google Cloud install the Connect Agent to establish control plane connectivity to Google Cloud, and the agent can traverse NATs, egress proxies, VPNs, and other interconnects so clusters and API servers do not need public IP addresses.

Registration behavior depends on where the cluster runs:

Consistent command-line access is provided by the Connect gateway (opens in a new tab), which lets users and service accounts connect with Google Cloud identity and run commands against registered clusters without direct IP connectivity. When third-party identity must be preserved, GKE Identity Service brings existing OpenID Connect (OIDC) providers such as Okta and Microsoft AD FS to Anthos environments, with preview LDAP support in some environments. The gateway therefore depends on fleet membership, while the agent depends on outbound connectivity through existing hybrid links rather than inbound exposure.

3.3Sequenced Rollouts Using Team Scopes and Membership Types

Sequenced rollouts control the order in which changes such as cluster upgrades propagate across fleet members instead of updating every cluster at once. Team scopes are the enterprise fleet-level construct for associating subsets of fleet clusters with specific application teams, and they enable team-based access control, team-scoped observability, and sequencing of upgrade rollouts. By binding a rollout sequence to team scopes, an operator can move a new version through non-production, then limited production scopes, then broad production, which limits blast radius and creates checkpoints for validation and rollback. GKE clusters on Google Cloud also support lightweight memberships, which register the cluster without enabling all fleet-level configurations while still permitting fleet-based rollout sequencing for upgrades. Lightweight memberships prevent clusters from using:

• Fleet-level features
• Fleet-level default logs
• Fleet-level default metrics
• Fleet Workload Identity Federation

3.4Global Traffic Routing, Failover Boundaries, and Service Isolation

Global traffic routing for hybrid deployments controls how requests enter Google Cloud and reach select APIs on-premises or in another cloud without exposing entire private networks. The gated egress pattern exposes only chosen APIs through an API gateway, proxy, load balancer facade, or Private Service Connect endpoint placed in an isolated perimeter segment. Traffic to private APIs is initiated only from within the Google Cloud environment, systems in the private environment cannot be reached directly from Google Cloud, and communication from the private environment to Google Cloud workloads is not allowed. Traffic flows from the Google Cloud VPC to the remote gateway through an ordered path:

1. Workload placement: Workloads deploy into single or multiple VPCs aligned to the organization resource hierarchy.
2. Hybrid extension: VPC networks extend to on-premises or other clouds over hybrid connectivity using internal IP addresses and overlap-free RFC 1918 space.
3. Allowlist enforcement: Traffic from specific VPC addresses toward remote gateways (opens in a new tab) is limited by IP allowlist filtering, with return traffic permitted by stateful firewall rules.
4. Inspection: Filtering is enforced by firewall rules or policies, a network virtual appliance (NVA) with next-generation firewall inspection in the path, or Cloud Next Generation Firewall Enterprise with intrusion prevention for deep packet inspection.

Within clusters, pod-to-pod traffic is open by default and must be explicitly restricted to prevent lateral movement and denial of service. Cloud Service Mesh (opens in a new tab) or Istio provides load balancing, service authorization, throttling, quota, and metrics, while Kubernetes network policies provide basic access control. Failure independence reinforces rollout and routing because Compute Engine (opens in a new tab) zonal resources such as instances and zonal persistent disks can only be used in the same zone, regional resources such as static external IP addresses can be used anywhere in the region, and spreading resources across zones and regions isolates physical and software-service failures.

Key Facts

• Config Sync pulls declared KRM manifests from a versioned source of truth to each fleet member and continuously reapplies them to correct drift between live cluster state and desired state.
• Artifact Registry centralizes artifacts with remote repositories that cache external sources and virtual repositories that consolidate same-format repositories behind one endpoint with controlled search order.
• Cloud Build supports SLSA Level 3 builds with authenticated provenance, and Binary Authorization checks attestations for build origin, vulnerability policy, and promotion criteria before GKE or Cloud Run admits an image.
• Secrets for Cloud Deploy applications must be generated, managed, and rotated outside Cloud Deploy and accessed at runtime from Secret Manager or HashiCorp Vault.
• A lightweight fleet membership supports sequenced upgrade rollouts but does not support fleet-level features, default logs, default metrics, or Fleet Workload Identity Federation.