Managing Infrastructure with Terraform and GitOps

Infrastructure-as-code tooling and managed services on Google Cloud define infrastructure in versioned files and reconcile Google Cloud to that desired state through automated execution. Infrastructure Manager (opens in a new tab) provides managed Terraform orchestration without a separate toolchain, Config Connector provides Kubernetes-native reconciliation, and Terraform Cloud provides remote SaaS execution. Security Command Center (opens in a new tab) enforces policy before resources are created, while Cloud Build (opens in a new tab) executes the GitOps pipeline that moves changes from commit to deployment.

1.1GitOps Desired State and Pipeline Synchronization

GitOps desired state is a version-controlled repository that holds Terraform files, pipeline definitions, and supporting scripts as the single source of truth. Developer changes flow from a commit or merge to main into a continuous integration and continuous delivery trigger in Cloud Build, GitHub Actions, Jenkins, GitLab Runners, or Terraform Enterprise, and that trigger synchronizes the change to the target project, folder, or organization. Promotion between environments uses separate repositories, branches, or projects per environment so the same definition moves from development to production, while rollback reverts the repository commit and lets the pipeline re-synchronize to restore the prior good state.

Repository and pipeline tooling choices determine where definitions live and what runs them. Cloud Source Repositories provides a managed private Git repository, while GitHub, GitLab, or Bitbucket serve as the version-control system when teams standardize outside Google Cloud. Cloud Build runs automation by default in Google Cloud blueprints, while Terraform Enterprise, GitLab Runners, GitHub Actions, or Jenkins can replace it for orchestration. For data orchestration, a GitHub repository added to the workspace stores pipelines and assets, and merging to main through a GitHub Action deploys the pipeline to the Managed Service for Apache Airflow environment.

1.2Shift-Left Validation and Drift Detection

Shift-left validation checks a Terraform plan against organization policies and Security Health Analytics detectors before any resource is created. The check requires Security Command Center Premium tier or Enterprise tier activated at the organization level, with the securityposture.googleapis.com, securitycentermanagement.googleapis.com, and cloudbuild.googleapis.com application programming interfaces enabled and a deployed security posture that defines policies and detectors. The validator identity needs Security Posture Shift-Left Validator, Log Writer, Storage Writer, and Storage Reader at the organization level, and Terraform code must populate the parent project, folder, or organization on every resource, modify assets and policies separately, use only supported types, and omit passwords or other sensitive fields that are stripped before evaluation.

Creating a validation report from a Terraform plan follows an ordered workflow:

1. Initialize: Run terraform init with provider version v5.5 or later, using terraform init -upgrade when an upgrade is required.
2. Render JSON plan: Run terraform plan -out TF_PLAN_FILENAME and convert it with terraform show -json TF_PLAN_FILENAME > TF_PLAN_JSON_FILENAME.json.
3. Create validation report: Run gcloud scc iac-validation-reports create PARENT --tf-plan-file=TF_PLAN_JSON_FILENAME.json, where PARENT is organizations/ORGANIZATION_ID/locations/LOCATION.

The resulting report flows from Security Command Center back to the pipeline owner, and a violating plan blocks promotion until the definition is fixed in version control. Drift after deployment is handled by background reconciliation because Cloud Asset Inventory (opens in a new tab) provides organization-wide visibility with eventual consistency, periodic synchronization, and possible lag, drops, or out-of-order events. GitOps workflows therefore treat Cloud Asset Inventory Pub/Sub feeds as delay-tolerant signals for dashboards and reconciliation loops that must be resilient and idempotent, while any critical pre-cutover check such as verifying a firewall rule calls the service API directly. Delivery from source services to inventory can lag, so inventory is not a synchronous control plane for real-time enforcement.

1.3Managed Execution Planes for Terraform and Kubernetes

Managed execution planes separate authoring Terraform configuration from running it with consistent state and history. The Terraform provider for Google Cloud defines desired state in human-readable versioned files, Infrastructure Manager automates deployment of those files programmatically on Google Cloud, and Terraform Cloud runs Terraform in a stable remote software as a service environment with stored state and secrets, version control triggers, and API integration. Terraform Enterprise provides a self-hosted distribution of Terraform Cloud for organizations that must keep that control plane inside their own boundary, and troubleshooting failed rollouts (opens in a new tab) depends on deployment history and state tracking rather than local workstation files.

Infrastructure Manager centralizes orchestration through previews that verify planned resources and deployments that create and track resources in a location such as us-central1. Infrastructure Manager depends on the Infrastructure Manager API config.googleapis.com and a service account with roles/config.agent plus resource roles such as roles/compute.networkAdmin. Config Connector uses a different boundary where desired state flows from YAML manifests to a Google Kubernetes Engine cluster and the controller reconciles Cloud SQL instances, Pub/Sub topics, Cloud Storage buckets, datasets, models, endpoints, Identity and Access Management policies, and service accounts. Choose Infrastructure Manager when Terraform is the source of truth but execution should live as a managed Google Cloud control plane, choose Config Connector when teams operate through Kubernetes and want kubectl-style lifecycle with application releases, and choose Terraform Cloud or Enterprise when the requirement is organization-wide change management with remote state and secrets.

Reusable modules and compliance controls sit above all three planes. Cluster Toolkit provides curated Terraform blueprints for high-performance computing, artificial intelligence, and machine learning clusters on Slurm or Google Kubernetes Engine, including A100, H100, H200, or B200 graphics processing units for training and L4 for inference, plus serving endpoints and storage for data and artifacts. The secret-manager module creates Secret Manager secrets with basic permissions, Identity and Access Management bindings flow from modules to datasets, views, or projects, and the policy validation tool blocks noncompliant plans before they reach Google Cloud.

Making infrastructure changes using Google-recommended practices and blueprints means provisioning foundations with Terraform and the Cloud Foundation Toolkit, rolling out changes progressively, and governing every change with identity, network, and policy controls. Terraform provisioning with Cloud Build (opens in a new tab) replaces manual console work with planned, reviewed, and repeatable pipelines where state decides whether to create, update, or leave each resource unchanged. Secure pipeline design (opens in a new tab) separates foundation, infrastructure, and application pipelines with least-privilege identities, while operations best practices (opens in a new tab) define breakglass and response readiness.

2.1Terraform Provisioning and Staged Foundation Deployment

Terraform provisioning is a declarative process where configuration files describe the desired end state and the Terraform provider for Google Cloud translates each block into Google Cloud API calls. The Terraform state file is the JSON mapping that links each configured object to its real object, so later runs create objects in configuration but not in state, update objects where state differs from configuration, and leave matching resources unchanged. The core order is fixed with state as the decision point: first author configuration in a root module, then run terraform plan to preview adds, changes, and destroys, then run terraform apply to provision through APIs and update state, which overwrites out-of-band console drift unless it is imported or reflected in code.

The terraform-example-foundation blueprint deploys the organization in sequential stages where each later stage depends on earlier outputs. The 0-bootstrap stage runs first because it creates the seed project holding Cloud Storage buckets for Terraform state with Object Versioning and highly privileged service accounts, plus the CICD project holding the Cloud Build foundation pipeline where Cloud Build acts as those seed service accounts. The foundation pipeline then deploys in order 0-bootstrap, 1-org for folders, shared-services projects, logging, and organization policies, 2-environments for development, non-production, and production folders and projects, and 3-networks for dual Shared VPC or hub-and-spoke topology. Each stage uses a different service account with minimum Identity and Access Management roles defined in 0-bootstrap/sa.tf, so a trigger for one stage cannot directly affect resources owned by another stage.

Remote state stores that mapping in Cloud Storage rather than locally so runs share one record. The foundation pattern reuses one backend bucket such as gs://tfstate-PROJECT_ID with prefix terraform/state, declared in backend.tf with a terraform { backend "gcs" { bucket = "tfstate-PROJECT_ID" prefix = "terraform/state" } } block, so terraform init must initialize against that bucket before any plan is correct. Redeployment from Cloud Shell depends on that backend in order: download the configuration, verify the bucket and state object exist, reuse the same bucket and prefix, run terraform init, then terraform plan and terraform apply. If the foundation was first deployed with the console Deploy button, lock the Infrastructure Manager deployment, download the state file, unlock, move the file into the backend bucket, and redeploy, while a 409 conflict means the resource already exists and must be deleted or imported into state.

2.2Progressive Rollouts and Pipeline Promotion

Progressive deployment releases an infrastructure change to a small blast radius first with verification before expanding across zones and regions. To change Compute Engine machine types or Cloud Storage bucket storage classes, the change is first applied to a few virtual machines in one zone while workload health is monitored, and only after success is the same change gradually deployed everywhere. If issues appear, infrastructure reverts quickly to the prior stable state for diagnosis before restarting, which creates a natural canary and rollback strategy. Control of global resources follows the opposite rule because resilient application patterns (opens in a new tab) often use a single Virtual Private Cloud network or global load balancer that survives zonal and regional outages but becomes a single point of failure, where a misconfigured forwarding rule can stop the frontend even while backends remain intact, so any global change is classified as high-risk and requires additional reviewers before production.

Automated provisioning with Cloud Build pipelines treats infrastructure changes like application changes through versioned configuration and promotion. Cloud Build executes builds to specification, Artifact Registry manages the full artifact lifecycle in one location, and Cloud Deploy automates delivery through a defined promotion sequence. Data flows from source control to Cloud Build for execution, from Cloud Build to Artifact Registry for storage, and from Artifact Registry to target runtimes through Cloud Deploy promotion.

A Google-recommended blueprint for predictable promotion uses Terraform with Cloud Build connected to GitHub and state stored centrally. The ordered workflow first sets up the GitHub repository as source of truth, configures the Cloud Storage backend, grants the Cloud Build service account permission to create and modify resources, and connects Cloud Build to the repository so commits trigger plans and applies. Changes then move from a feature branch for review and validation to development for verification and finally to production progressively where possible. Policy-compliant delivery adds cloudbuild-tf-plan.yaml for Open Policy Agent validation and terraform plan plus cloudbuild-tf-apply.yaml for terraform apply only after policy passes, with least-privilege stage accounts, optional private pools inside a VPC Service Controls perimeter, pinned builder images from 0-bootstrap/Dockerfile, and optional manual approval before execution. Deployment Manager walkthrough practices (opens in a new tab) reinforce the same principle of small, reviewed, and repeatable changes.

2.3Perimeter Controls and Secure Operations

Governance with policy blueprints combines user-centric checks on every request with resource-centric perimeters around data. Chrome Enterprise Premium is a user-centric control that evaluates an access level and binding on every API request, including IP-based or certificate-based mutual TLS conditions, and rejects requests with compromised credentials that fail those conditions for organization user identities. VPC Service Controls is a resource-centric control that defines perimeters within which resources can communicate, blocking access from untrusted locations and mitigating exfiltration for all user and service account identities. Traffic flows from the user or workload to the Google Cloud API where Chrome Enterprise Premium first checks identity and device attributes and VPC Service Controls then checks whether the target resource may communicate inside its perimeter, so together they reduce the effectiveness of compromised credentials on an attacker-controlled machine.

Strong authentication, key restriction, and endpoint protection close the remaining paths. OS Login with 2-Step Verification enforces reauthentication with password or security key for Secure Shell access to Compute Engine, blocking an attacker with only OAuth tokens, while Remote Desktop Protocol to Windows instances does not support OS Login so coarse controls such as session length, web-session verification, and disabling Allow user to trust the device must shorten compromised sessions. Service account key values can be copied from gcloud configuration files, the file system, or code repositories, so the constraints/iam.disableServiceAccountKeyCreation and constraints/iam.disableServiceAccountKeyUpload organization constraints disable creation and upload, least privilege grants only required roles at the smallest scope, and Policy Intelligence recommendations remove unused roles. Trusted endpoints such as developer workstations or Compute Engine instances can run gcloud commands directly and bypass perimeter checks, so operations must protect physical security, monitor breaches, replace persistent SSH keys and service account keys with short-lived credentials where possible, and pre-provision incident responders with console and Admin console access before an incident.

Breakglass access is pre-designed highly privileged access for outages when normal automation fails. Operations teams use breakglass to restart virtual machines, restore data, or repair pipeline resources in the CI/CD project and external Git repository, moving privileges from locked to temporarily active and back to locked. Common mechanisms include temporary group elevation through privileged access management, dedicated administrator accounts such as dana@example.com for daily use and admin-dana@example.com for breakglass, and just-in-time self-escalation with approval and auditing. Every design must define scope so business units cannot disrupt each other, require approvals or split operations where one person holds credentials and another holds multi-factor authentication, audit use with findings such as a custom Event Threat Detection alert on breakglass account use, and remove access after the incident, preferring automated workflows and safe proxies to support zero-touch production.

Automation with scripting in Python and Go runs custom code as versioned pipeline steps and reactive handlers that call Google Cloud APIs to provision resources and remediate drift. Cloud Build (opens in a new tab) executes builds from Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket and produces Docker containers or Java archives that flow to Artifact Registry and onward through Cloud Deploy. Secret Manager supplies credentials at runtime through Identity and Access Management, so database passwords, API keys, and deployment tokens stay out of source code, logs, and container layers.

3.1Pipeline Execution with Secret Handling and Validation

Pipeline execution with secret handling separates building software once from releasing it safely many times. Cloud Build executes versioned build steps where Python and Go scripts call Google Cloud APIs, orchestrate Terraform, and bootstrap environments using the bootstrap Terraform module and Cluster Toolkit blueprints for high-performance computing, artificial intelligence, and machine learning clusters kept in version control. Repository automation depends on a cloudbuild.yaml orchestration file, an imagebuilder.yaml recipe where operating system images are built, referenced scripts, and a connection to GitHub, GitLab, or Bitbucket through Cloud Build repositories second generation or Developer Connect links. The identity that creates triggers needs Cloud Build Editor and Service Account User, and local Terraform use requires command-line interface version 1.3 or later.

Artifact storage and progressive delivery divide work by stage. Artifact Registry provides a single location for packages and Docker images and integrates with CI/CD tools and runtimes to manage the full lifecycle, while Cloud Deploy promotes versioned artifacts through target environments in a defined sequence to Google Kubernetes Engine for containerized workloads and to App Engine flexible environment, Cloud Run functions, and Cloud Run for serverless workloads. A build therefore flows from repository source to Cloud Build to Artifact Registry, and only then does Cloud Deploy promote that stored artifact through development, staging, and production targets. During the run, the Cloud Build service account retrieves each Secret Manager secret at execution time and passes it only to the step that needs it, with the secret-manager Terraform module creating secrets and basic permissions, so a step without permission fails closed as a pipeline error rather than using a default credential.

Validation testing proves that code and deployed systems behave as expected before promotion. The pipeline first runs static checks such as Terraform validate and plan from an automated script, stores the tested configuration as a versioned artifact, and keeps separate development, staging, and production configurations in version control with automated promotion so environment differences are explicit. The Data Validation Tool, an open source Python tool based on Ibis, compares heterogeneous tables for warehouse, database, or lake migrations with multi-leveled functions, while Artifact Registry container scanning surfaces image vulnerabilities and automated rebuild pipelines pull base-image patches before republishing. Repository-triggered and scheduled rebuilds keep images current through an ordered sequence: create an Artifact Registry repository, connect GitHub source through second generation or Developer Connect, create a Cloud Build trigger on repository events, add a Cloud Scheduler schedule for regular rebuilds, and review build status, published images, and scan findings. Automated image builds on events (opens in a new tab) publish custom images to Compute Engine and Artifact Registry on push to main and on schedule even when no configuration changed.

3.2Client Library Authentication from Workstation to Workload

Programmatic resource management via client libraries lets Python and Go scripts call Google Cloud APIs without hand-building REST requests. For Go, installation uses go get cloud.google.com/go/language/apiv1, and for Python it uses pip install --upgrade google-cloud-language after preparing the development environment. Before the library can succeed, billing must be verified and the relevant API enabled with gcloud services enable such as language.googleapis.com, which requires serviceusage.services.enable through Owner or Service Usage Admin. Authentication flows from a user credential or attached service account through Application Default Credentials into the client library and then to the target API.

Local development uses Application Default Credentials created with gcloud auth application-default login, which the library discovers automatically and which are separate from gcloud command-line credentials used in Authorization: Bearer headers. If the command warns about missing serviceusage.services.use, an administrator grants Service Usage Consumer and the developer sets the billing context with gcloud auth application-default set-quota-project PROJECT_ID. Service account impersonation tests exact service account permissions locally and requires iam.serviceAccounts.getAccessToken through Service Account Token Creator, with gcloud config set auth/impersonate_service_account SERVICE_ACCT_EMAIL for gcloud itself and gcloud auth application-default login --impersonate-service-account=SERVICE_ACCT_EMAIL for an impersonated credentials file supported only for Go, Java, Node.js, and Python client libraries.

Workload authentication on Google Cloud uses the service account attached to the compute resource such as a Compute Engine virtual machine, which is preferred because no local keys are stored or distributed. The trust boundary sits between the compute resource and the API where the control plane provides the attached identity and the API trusts requests carrying it, and for most services the account must be attached at creation while Compute Engine is the exception allowing attachment at any time. Provisioning that automation identity follows an ordered sequence where each step depends on the prior identity configuration: install the CLI and run gcloud init to select the owning project, create the account with gcloud iam service-accounts create requiring Create Service Accounts and Project IAM Admin, bind it to resource access with gcloud projects add-iam-policy-binding to serviceAccount:NAME@PROJECT_ID.iam.gserviceaccount.com, authorize attachment by granting the deploying principal Service Account User on the account, and create the hosting resource with --zone and --service-account set. Creating a project requires Project Creator with resourcemanager.projects.create, and production accounts must use predefined or custom roles rather than basic Owner, Editor, or Viewer.

3.3Event-Driven Remediation from Logs to Fix

Event-driven remediation runs small reactive scripts that detect unwanted changes and fix them without human action. Cloud Functions or Cloud Run services in Python, Go, Java, Node.js, and other runtimes subscribe to signals from Pub/Sub or Cloud Logging events, evaluate each event against desired state, and call Google Cloud APIs to reprovision resources or enforce compliance. Declarative infrastructure with Terraform, Config Controller, and Cloud Build provides the known good state to restore, while Open Policy Agent checks code before deployment and Eventarc or Workflows trigger remedial actions from audit observations with Organization Policy, Identity and Access Management, and VPC Service Controls limiting what remediation identities can change.

How detection flows from logs to remediation code is an ordered pipeline where a break in any link stops the fix:

1. Record: A Google Cloud API call is recorded as an audit log entry in Cloud Logging with method name, resource name, and initiator identity.
2. Route: A log sink with a matching filter forwards the entry from Cloud Logging to a Pub/Sub topic, with aggregated sinks also copying to Cloud Storage or BigQuery for retention.
3. Deliver: Pub/Sub delivers the message to a Cloud Functions handler or to a Cloud Run service through a push subscription.
4. Remediate: The function or service parses the event and calls the relevant API to restore the resource, publish a follow-up message, or write a structured log.

Filtering at the sink controls which drifts trigger work and limits cost and noise, and only after delivery can the script act. Where remediation code runs depends on packaging and duration needs because both Cloud Functions second generation and Cloud Run start on demand, scale instantly with bursts, and scale to zero when idle, unlike Compute Engine autoscaling that needs fine-tuned rules.

Remediation scripts combine focused actions where the script depends on the API it calls and the workflow depends on safe retries. Handlers parse the Pub/Sub or Logging event to identify the resource and drift, call the resource API to restore settings, recreate deleted resources, or restrict access, publish follow-up Pub/Sub messages to chain steps or notify automation, and write structured logs with request correlation for troubleshooting. Retry logic must avoid loops by discarding events older than 10 seconds so a poison message cannot retrigger forever, using explicit retry-on-error only when the remediation call is safe to repeat.

Key Facts

• GitOps stores declarative Terraform configurations in version control as desired state, synchronizes changes through an automated pipeline, and rolls back by reverting the commit and re-synchronizing.
• Shift-left validation by Security Command Center Premium or Enterprise tier checks a Terraform JSON plan created with terraform plan -out and terraform show -json against organization policies before resources are created.
• Infrastructure Manager automates Terraform deployment as a managed service using previews and deployments with roles/config.agent, while Config Connector manages Google Cloud resources as Kubernetes objects reconciled by a GKE cluster controller.
• google_project_iam_policy and google_project_iam_binding are authoritative and overwrite existing access, while google_project_iam_member is additive and preserves other bindings.
• Detection for event-driven remediation flows in order from Cloud Logging to a log sink to a Pub/Sub topic to a Cloud Functions handler or Cloud Run service, which then calls a Google Cloud API to restore drift.