CI/CD (Continuous Integration and Continuous Deployment)

Deploying and operationalizing the pipelines

Automate Testing and Quality Assurance of Data Pipeline Code

Automated testing and quality assurance for data pipelines validate data processing code, SQL scripts, and orchestration workflows before they reach production environments. Modern data pipelines combine programming code for frameworks like Apache Beam with SQL scripts that execute Data Definition Language (DDL) and Data Manipulation Language (DML) commands. Teams use Dataform to compile SQL code, check dependencies, and enforce schema validation rules before queries run against live data warehouses. Storing all pipeline assets in a Git version control repository provides the change history and rollback capabilities needed to recover from errors.

To run automated checks, organizations configure Google Cloud Build (Cloud Build) triggers that listen for code commits and pull requests. When a developer pushes code, Cloud Build fetches the source repository alongside build dependencies stored in Google Artifact Registry (Artifact Registry). The build system then runs static code analysis tools, including linters and analyzers, to verify formatting rules and identify structural flaws before compiling the code.

Testing within continuous integration pipelines is divided into three distinct levels:

Unit testing: Isolates and validates individual functions or components, such as custom transform logic within an Apache Beam pipeline.
Integration testing: Verifies that separate modules, staging datasets, and external connections interact correctly.
End-to-end testing: Simulates complete data workflows from initial ingestion through downstream transformations to guarantee system compatibility.

Orchestration workflows running in Google Cloud Composer (Cloud Composer) require a structured pipeline to validate and deploy Apache Airflow Directed Acyclic Graphs (DAGs):

Commit and pull request: A developer opens a pull request in the source repository containing new or modified Airflow DAG files and dependency definitions.
Presubmit testing: A Cloud Build trigger executes automated unit tests using pytest inside an isolated Python environment to verify DAG syntax and task dependencies.
Merge and synchronize: Once presubmit tests pass and reviewers merge the pull request into the main branch, a release Cloud Build trigger copies the DAG files into the designated Google Cloud Storage (Cloud Storage) bucket connected to the Cloud Composer environment.

Exam tip: Cloud Build uses separate triggers for presubmit testing and deployment, validating Cloud Composer DAGs in an isolated test environment before synchronizing files to the Cloud Storage bucket.

Manage Infrastructure as Code and Multi-Environment Promotion

Infrastructure as Code (IaC) is the practice of defining, provisioning, and managing cloud resources through declarative configuration files rather than manual console interactions. Declarative tools such as HashiCorp Terraform (Terraform) and Google Cloud Infrastructure Manager ensure that cloud infrastructure is reproducible, auditable, and version-controlled. By codifying data infrastructure, teams can deploy identical environments across development, staging, and production tiers while eliminating configuration drift.

A process flow showing a GitOps promotion pipeline: a pull request triggers terraform plan on a feature branch, then merges promote code through dev and staging branches to production, where Cloud Deploy approval gates apply it. Each environment uses its own isolated Terraform remote state bucket in Cloud Storage with object versioning enabled.

Teams manage infrastructure through a GitOps workflow where configuration files reside in a shared Git repository. Changes are submitted through pull requests and applied automatically using branch-aware Cloud Build pipelines. In a branch-aware pipeline, pushes to a development branch update the development infrastructure, while merges to the main or production branch update production systems.

To maintain infrastructure integrity, teams promote changes sequentially across environments:

Feature branch validation: A developer creates a pull request on a feature branch, prompting Cloud Build to run terraform plan to validate syntax and display proposed resource changes without modifying infrastructure.
Development deployment: Merging the approved pull request into the dev branch triggers Cloud Build to execute terraform apply against the development environment.
Staging rollout: Promoting the validated code to the staging branch deploys infrastructure to a pre-production environment that mirrors production for integration testing.
Production release: Promoting code to the prod branch passes through automated and manual approval gates in Google Cloud Deploy (Cloud Deploy) before applying configurations to the live production environment.

Terraform maintains a record of provisioned infrastructure in a remote state file located in a dedicated Cloud Storage bucket. Enabling object versioning on this bucket protects state files from accidental overwrites, provides a history of previous states, and enables fast recovery. Storing separate remote state files in isolated buckets for development, staging, and production ensures that changes in one environment cannot corrupt the state of another.

Exam tip: Terraform remote state must be stored in a dedicated Cloud Storage bucket with object versioning enabled for each separate environment to protect infrastructure state files from corruption and unintended overwrites.

Secure Continuous Deployment and Containerized Artifact Management

Secure continuous deployment protects software delivery pipelines by controlling artifact storage, enforcing identity permissions, and verifying container integrity across release stages. Modern deployment pipelines package data processing applications into containers and manage dependencies within centralized package repositories. Implementing security controls across the build and release lifecycle prevents unauthorized code from entering production environments.

Artifact Registry acts as a universal package manager supporting container images, Python packages, and Java Maven archives. Teams configure specific repository types to balance performance and supply-chain protection:

Repository Type	Primary Purpose	Key Operational Benefit
Standard repository	Stores private build artifacts and container images	Provides centralized, private package versioning with fine-grained access policies
Remote repository	Caches upstream dependencies from external sources like Docker Hub	Reduces download latencies and protects builds against external registry downtime
Virtual repository	Combines multiple repositories behind a single access endpoint	Prioritizes private packages over external ones to stop dependency confusion attacks

To secure artifacts, Google Artifact Analysis (Artifact Analysis) provides automated vulnerability scanning for container images stored in Artifact Registry. It generates a Software Bill of Materials (SBOM) to track open-source dependencies and accepts Vulnerability Exploitability eXchange (VEX) statements to document whether discovered vulnerabilities affect the application. Cloud Build enforces supply-chain security by producing SLSA Level 3 container builds that include non-falsifiable build provenance metadata, verifying the source code repository, commit digest, and build toolchain.

Deployment pipelines enforce security by granting service accounts only the minimum permissions required under Identity and Access Management (IAM). Deployment pipelines must not access raw customer data, and write access is restricted strictly to designated destination resources. VPC Service Controls create security perimeters around Cloud Build, Artifact Registry, and target environments to prevent data exfiltration. Cloud Deploy automates application delivery to Google Kubernetes Engine (GKE), GKE Enterprise, and Google Cloud Run (Cloud Run) using release pipelines that support canary releases to test new software versions on a subset of live traffic before full deployment.

Exam tip: Virtual repositories in Artifact Registry protect pipelines from dependency confusion attacks by explicitly defining the search order and prioritizing internal package repositories over public upstream sources.

Key Facts

Google Cloud Build automates testing for data pipelines by validating SQL schemas with Dataform, running static code analysis, and executing unit tests before synchronizing Directed Acyclic Graphs (DAGs) to Google Cloud Storage.
Multi-environment infrastructure promotion uses branch-aware Google Cloud Build pipelines and HashiCorp Terraform remote state files isolated in Google Cloud Storage buckets with object versioning enabled.
Google Cloud Deploy manages application and infrastructure releases across development, staging, and production environments using configurable approval gates and canary deployment strategies.
Google Artifact Registry uses virtual repositories to combine multiple repositories under one endpoint and prioritize internal private packages over external sources to prevent dependency confusion attacks.
Google Cloud Build supports Supply-chain Levels for Software Artifacts (SLSA) Level 3 container builds by generating verifiable, non-falsifiable build provenance including image digests and build toolchain details.

Frequently Asked Questions

How does a continuous integration and deployment pipeline validate and deploy Apache Airflow Directed Acyclic Graphs (DAGs) to Google Cloud Composer (Cloud Composer)?

The pipeline uses a presubmit Google Cloud Build (Cloud Build) trigger to run automated unit tests with pytest in an isolated environment, verifying DAG syntax and task dependencies during a pull request. Once the pull request is merged into the main branch, a release trigger copies the DAG files into the Google Cloud Storage (Cloud Storage) bucket associated with the Cloud Composer environment.

How should HashiCorp Terraform (Terraform) remote state files be managed across development, staging, and production environments?

Terraform remote state files must be stored in dedicated Google Cloud Storage (Cloud Storage) buckets with object versioning enabled to prevent accidental overwrites and maintain state history. Teams must maintain isolated state buckets for each separate environment to ensure configuration changes in one tier cannot corrupt another.

What is the role of a virtual repository in Google Artifact Registry (Artifact Registry)?

A virtual repository combines multiple repositories behind a single access endpoint. It secures software delivery pipelines by prioritizing internal private packages over external sources to prevent dependency confusion attacks.

Sample Question

Your data engineering team manages cloud data infrastructure—including BigQuery datasets, Cloud Storage buckets, and Dataflow pipelines—across dev and prod environments on Google Cloud. You need to implement a continuous integration and continuous deployment (CI/CD) GitOps workflow using Terraform and Cloud Build that meets the following requirements:

Provide a central, collaborative, and version-controlled mechanism for Terraform state management that prevents concurrency conflicts and state corruption.
Automatically validate configurations and generate plan previews on proposed code changes before merging into environment branches.
Enforce a strict multi-environment promotion sequence where changes are automatically applied only upon merging into the target environment branch.

How should you structure your Terraform configuration, state storage, and Cloud Build triggers?

Configure a Cloud Storage backend with separate object prefixes for each environment and enable Object Versioning. Set up Cloud Build triggers to run terraform init and terraform plan on feature branch pull requests as required status checks, and run terraform apply only on merges to the dev and prod branches targeting their respective environment directories.
Store terraform.tfstate files directly within Git feature and environment branches. Configure Cloud Build triggers to execute terraform apply -auto-approve immediately whenever any commit is pushed to a feature branch, and delete the state file after deployment.

Professional Cloud Data Engineer