Professional Cloud Network Engineer
Architecting DNS zones and record management for Google Kubernetes Engine (GKE) services involves configuring Cloud DNS to provide service discovery for workloads within and across clusters. This design integrates Cloud DNS directly into GKE, replacing the default kube-dns, and allows you to scope DNS resolution to a specific cluster or to an entire VPC network.
Cloud DNS scopes define the visibility and reachability of DNS records by attaching private DNS resources, like zones and response policies, to specific Google Cloud objects such as VPC networks or GKE clusters. A GKE cluster scope limits DNS record resolution to within a single cluster, similar to kube-dns behavior. In contrast, a VPC scope makes DNS records resolvable across the entire VPC network, allowing clients connected via Cloud VPN or Cloud Interconnect to discover GKE services. When a Pod issues a DNS query, Cloud DNS checks for a match at the most specific scope first—the GKE cluster—before walking up the hierarchy to the VPC network scope. You bind a managed zone or response policy to a selector within a scope, making it visible to clients within that scope. This hierarchy enables multiple clusters in a single network to have their own isolated cluster.local DNS views.
To configure a GKE cluster-scoped DNS zone, first create a private managed zone in Cloud DNS and bind it to a specific cluster. Use the gcloud dns managed-zones create command with the --visibility=private and --gkeclusters= flags, specifying the cluster's fully qualified resource path. This authorizes the cluster to query the zone. Alternatively, update an existing private zone using gcloud dns managed-zones update with the --gkeclusters= flag to add the cluster binding. The required IAM permissions for these operations include dns.managedZones.create (or update), dns.managedZones.list, and dns.gkeClusters.bindPrivateDNSZone. The DNS Administrator role (roles/dns.admin) includes these permissions. After binding, the GKE controller automatically manages the DNS records for Services within that zone, and any manual changes to those records are overwritten when the controller restarts.
To enable Cloud DNS as the DNS provider for a GKE cluster, set the cluster's DNS provider to Cloud DNS and its DNS scope. For a new Standard cluster, use gcloud container clusters create with --cluster-dns=clouddns and --cluster-dns-scope=cluster. For an existing cluster, update it with gcloud container clusters update using the same flags. Enabling Cloud DNS causes the controller to run on the GKE control plane, but Pods do not use it for resolution until you upgrade all node pools in the cluster, which triggers a node recreation. When Cloud DNS is enabled, the value of /etc/resolv.conf inside Pods using the ClusterFirst DNS policy depends on other features. If NodeLocal DNSCache is also enabled, the resolver is 169.254.20.10. If only Cloud DNS is enabled, it is 169.254.169.254. If Cloud DNS is disabled, the resolver falls back to the kube-dns service IP address.
For service discovery beyond a single cluster, you have three primary patterns. Cloud DNS VPC scope makes a cluster's service records (..svc.cluster.local) resolvable across the entire VPC, and you can configure unique cluster DNS names (like cluster1 and cluster2) for non-GKE clients. Multi-cluster Services provides a single virtual IP and a consistent DNS name (..svc.clusterset.local) discoverable from any GKE cluster using the feature. Service Directory for GKE registers both GKE and non-GKE services in a managed registry, populating Cloud DNS records and allowing resolution via DNS, HTTP, or gRPC.
Zonal Cloud DNS is a private DNS service that exists within each Google Cloud zone, offering a failure domain contained to that zone. This contrasts with the default global Cloud DNS service, where an outage can affect all zones. Create a zonal private zone scoped to a GKE cluster for increased isolation. Zonal Cloud DNS supports GKE-scoped private zones and forwarding zones, but not public zones, peering zones, or VPC-scoped private zones. Billing for zonal zones works the same as for global zones.
In a Shared VPC setup, cross-project binding allows service projects to create and manage their own DNS zones while binding them directly to the host project's shared network. This provides autonomy for service project administrators, eliminates the need for a placeholder VPC network, and maintains IAM roles at the project level. With cross-project binding, all zones are directly associated with the Shared VPC network, enabling any-to-any DNS resolution for any VM in that network without a transitive hop limit, which is suitable for hub-and-spoke designs.
Troubleshooting DNS resolution and synchronization issues in GKE environments requires understanding how Cloud DNS integrates with Kubernetes service discovery, the scope boundaries that control where DNS records are visible, and the configuration paths that determine how Pods resolve names.
When a Pod uses the ClusterFirst DNS policy, the contents of /etc/resolv.conf determine which DNS server handles lookups. The IP address in this file changes based on whether Cloud DNS and NodeLocal DNSCache are enabled on the cluster. When both Cloud DNS and NodeLocal DNSCache are enabled, Pods send queries to 169.254.20.10. When Cloud DNS is enabled but NodeLocal DNSCache is disabled, Pods use 169.254.169.254. If Cloud DNS is disabled entirely, Pods fall back to the kube-dns service IP address regardless of NodeLocal DNSCache status. Misalignment between the expected resolver and the actual configuration is a common cause of resolution failures, particularly when enabling Cloud DNS on existing clusters.
Cloud DNS supports two distinct scopes that control DNS visibility: cluster scope and VPC scope. In cluster scope, DNS records are only resolvable within the GKE cluster using the ..svc.cluster.local schema, matching traditional kube-dns behavior. In VPC scope, DNS records become visible to the entire VPC network, allowing clients connected via Cloud VPN or Cloud Interconnect to resolve GKE service names directly. When troubleshooting resolution failures, first determine which scope is configured, because VPC-scoped zones require network-level visibility that cluster-scoped zones do not. The resolution order follows a hierarchy: Cloud DNS checks the most specific scope first (the cluster), then falls back to network-scoped resources if no match is found.
Synchronization between GKE and Cloud DNS can fail when the GKE cluster lacks authorization to query a Cloud DNS private zone, when the managed zone is not properly bound to the cluster's network, or when IAM permissions are missing. To authorize a cluster, run gcloud dns managed-zones update with the --gkeclusters flag to bind the zone to the cluster. The required IAM permission is dns.gkeClusters.bindPrivateDNSZone, which is included in the DNS Administrator role. If the GKE controller cannot communicate with Cloud DNS, records will not synchronize, and services will be unreachable by name. Additionally, when using zonal Cloud DNS, the GKE controller overwrites any manual record changes when it restarts, so automated synchronization is essential.
VPC network configuration directly impacts DNS query reachability. When using VPC scope or Shared VPC setups, firewall rules must allow UDP and TCP traffic on port 53 between Pods and the Cloud DNS resolver IPs. In Shared VPC environments, cross-project binding allows service project administrators to create and manage their own DNS zones while binding them to the shared network owned by the host project, enabling any-to-any DNS resolution across the Shared VPC. If firewall rules block DNS traffic or if the VPC peering relationship is misconfigured, queries will time out. Additionally, Private Google Access must be enabled on subnets where GKE nodes lack public IP addresses if they need to reach Cloud DNS.
DNS configuration choices affect application latency. NodeLocal DNSCache improves performance by caching DNS responses at the node level, reducing the need to query upstream resolvers for repeated lookups. When NodeLocal DNSCache is enabled with Cloud DNS, Pods point to 169.254.20.10 instead of the metadata server, eliminating a network hop. Zonal Cloud DNS provides resilience against global outages by containing the failure domain within a single Google Cloud zone, though it limits visibility to resources in that zone only. For high-volume DNS workloads, enabling NodeLocal DNSCache and using zonal Cloud DNS together reduces latency while protecting against zone-level failures.
When DNS resolution fails after enabling Cloud DNS on an existing cluster, the issue often stems from the node pool not being upgraded. The Cloud DNS controller runs on the control plane immediately after enabling it, but Pods continue using kube-dns until the node pool is upgraded or a new node pool is created. Upgrading to the same version has no effect; the node pool must be upgraded to a new version to pick up the Cloud DNS configuration. Another common issue arises when using domains ending in .local with additive VPC scope, which can cause resolution failures because .local is reserved for mDNS. Finally, if the additive-vpc-scope-dns-domain value is not unique within the VPC, DNS records from different clusters may conflict, causing intermittent resolution failures.
The external-dns operator automates the creation and synchronization of DNS records in Google Cloud DNS based on exposed Kubernetes resources within Google Kubernetes Engine (GKE). Deploying external-dns replaces manual record updates by monitoring cluster ingress objects and services, then applying matching record sets to specified public or private managed zones. Securing this deployment requires implementing the principle of least privilege across identity systems, restricting operational scope to designated namespaces, and enforcing cluster admission policies to prevent unauthorized record tampering.
Google Cloud DNS provides the managed infrastructure necessary to publish and resolve domain name records created by automated cluster workloads. Administrators must first enable the Cloud DNS API (dns.googleapis.com) within the target project to allow API requests from the cluster to reach the DNS control plane. A Cloud DNS managed zone is then provisioned with either public visibility for external traffic or private visibility linked directly to the target Virtual Private Cloud (VPC) network. Once the managed zone exists, automated processes can initiate and execute transactions to insert, modify, or delete resource records such as A and CNAME records.
Workload Identity is the recommended security mechanism for authenticating external-dns pods running inside GKE to Google Cloud APIs. Workload Identity establishes a secure relationship by binding a Kubernetes Service Account (KSA) used by the external-dns pod to an Identity and Access Management (IAM) Google Service Account (GSA). If Workload Identity is omitted, the pod falls back to using the default service account attached to the underlying GKE node, which broadens access and violates least privilege. Using Workload Identity eliminates the risk of downloading and storing persistent service account keys within cluster secrets or configuration files, thereby mitigating credential exfiltration threats.
Scoping controls define the operational boundaries of the external-dns operator to ensure it only manages authorized DNS records. Configuration flags and annotations restrict the operator to specific namespaces, resource types, or domain filters, preventing accidental overwrites in shared environments. Organizations should also use disjoint DNS domain hierarchies across separate accounts and environments to prevent domain ownership conflicts and accidental record collisions. By limiting the operator's visibility to explicit hostnames and designated managed zones, administrators isolate production routing configurations from unvetted workload changes.
Role-based access control (RBAC) and admission controllers safeguard the Kubernetes cluster against unauthorized DNS record creation and modification. Cluster administrators apply RBAC policies to restrict who can create or modify services, ingresses, and external-dns deployment configurations, limiting administrative permissions to trusted operators. To prevent workloads from bypassing controls or modifying privileged annotations, organizations enforce centralized constraints using admission controllers like Policy Controller. Restricting resource creation through strict validation ensures that only vetted configurations reach the external-dns controller for synchronization with Cloud DNS.
StudyCerts
StudyCerts169.254.20.10 when both are enabled, 169.254.169.254 when only Cloud DNS is enabled, and the kube-dns IP when Cloud DNS is disabled..local domains with additive VPC scope, or having non-unique additive-vpc-scope-dns-domain values are common causes of resolution failures.Cluster scope restricts DNS record resolution to within a single GKE cluster, identical to the default kube-dns behavior. VPC scope makes those records resolvable across the entire VPC network, including clients connected via Cloud VPN or Cloud Interconnect, enabling service discovery from non-GKE resources in the same network.
The Cloud DNS controller runs on the GKE control plane immediately after enabling the feature, but Pods continue to use the kube-dns service IP address until the node pool is upgraded to a new version. Upgrading to the same version does not trigger the change; only a version change forces node recreation and picks up the new resolver configuration.
Workload Identity binds a specific Kubernetes Service Account (KSA) used by the external-dns pod to an IAM Google Service Account (GSA) with minimal permissions. This avoids giving the pod access to the broad permissions of the node's default service account and eliminates the need to store service account keys in the cluster, reducing the risk of credential exfiltration.
The most common causes are the GKE cluster lacking authorization to query the private zone (resolved by binding the zone with --gkeclusters), missing IAM permissions (such as dns.gkeClusters.bindPrivateDNSZone), and firewall rules blocking UDP/TCP port 53 traffic to the Cloud DNS resolver IPs. Additionally, manual record changes are overwritten when the zonal Cloud DNS controller restarts, so automated synchronization is required.
Prepare and test your skills
Prepare and test your skills