Prepare and test your skills
Prepare and test your skills
Incident management and continuous improvement is the practice of limiting harm when a cloud failure happens and using what is learned to make operations stronger. In this practice, incident response handles the immediate failure through clear roles, communication, and automation. Retrospectives and a learning culture then turn each incident into preventive action, creating a closed loop where detection leads to resolution, resolution leads to review, and review leads to better processes and tooling.
Clear incident response procedures define who does what and how information moves during a failure so the response stays coordinated under pressure. The procedures assign roles to incident commanders, investigators, communicators, and technical experts. They also define communication protocols and escalation paths so updates flow promptly from responders to stakeholders. The steps for communication, triage, investigation, and resolution are documented in a runbook or playbook, and teams receive regular training so they can follow that runbook when an incident occurs.
An incident moves through an ordered response sequence where each phase depends on the one before it:
Automated incident response uses Google Cloud compute services to replace slow manual steps with fast, consistent actions. Cloud Run functions (opens in a new tab) (formerly Cloud Functions) or Cloud Run can continuously watch systems, trigger alerts, collect diagnostic data, and run predefined remediation such as isolating an affected system, quarantining a file, or rolling back a change. Because automation connects detection directly to notification and then to remediation, threats are caught before they escalate and recovery starts without waiting for human intervention.
Automated response provides three related benefits that reduce harm and workload:
A shared knowledge base preserves solutions and troubleshooting guides so future incidents are resolved faster and with less downtime. Google Workspace (opens in a new tab) provides collaborative documents with version control and change tracking to create, edit, and share this knowledge. The knowledge base is made accessible through incident management systems or a dedicated portal so responders can find it during an incident. After each incident, teams monitor reports for common issues and trends and then review and update the knowledge base with new fixes, so the stored guidance stays relevant and useful.
Blameless retrospectives are structured reviews held after project milestones, major incidents, or significant testing cycles to improve response and prevent repeat failures. The retrospective gives teams time to reflect on what went well and what needs improvement, learning from both successes and failures without blaming individuals. Findings are turned into actionable steps, including places where change management or testing can be further automated to improve reliability.
An effective way to run the review is the Start-Stop-Continue model, where the team groups feedback into three types of action:
Exam tip: Treat the retrospective as blameless and process-focused: the goal is actionable fixes to systems and runbooks, not assigning fault to a person.
A culture of learning turns incident knowledge into ongoing operational improvement across people, processes, and technology. Teams are encouraged to experiment, share knowledge gained from incidents and tests, and work in a psychologically safe environment where failures are treated as opportunities for growth. That culture depends on data, so teams track key performance indicators (KPIs) and supporting metrics with Cloud Monitoring (opens in a new tab) for real-time observation and BigQuery (opens in a new tab) for longer-term storage and analysis, then visualize trends in dashboards covering CPU, memory, network traffic, and latency. Feedback from surveys, user interviews, in-product feedback forms, and regular team meetings is analyzed for common themes, and priorities for the next improvement cycle are set by impact and feasibility.
Observability and performance measurement is the practice of collecting telemetry about system health and using it to judge whether services meet their reliability targets. Google Cloud Observability (opens in a new tab) is the unified platform that collects, analyzes, and visualizes metrics, logs, and traces from Google Cloud services, applications, and external sources. Cloud Monitoring (opens in a new tab) analyzes metrics related to availability, performance, and latency, while Cloud Logging (opens in a new tab) collects system events for deeper analysis, together supporting Service Level Indicators (SLIs) and Service Level Objectives (SLOs).
Service Level Indicators (SLIs) and Service Level Objectives (SLOs) provide the measurement foundation for operational excellence. An SLI is a specific quantitative measure of the level of service provided, such as average request latency, error rate, or system throughput. An SLO is the target value or range for an SLI, such as average request latency lower than 100 milliseconds, and clearly defining and communicating SLOs and service level agreements (SLAs) for all critical services helps ensure reliability and performance.
Cloud Monitoring collects and analyzes the metrics that track SLIs over time, so teams can verify whether SLOs and SLAs are being met. When an SLI moves away from its SLO, the monitoring data provides the evidence needed to prioritize fixes and manage reliability. This link from raw metric to SLI to SLO turns observability into an operational decision about system health.
Exam tip: An SLI is the measured metric, such as error rate, while an SLO is the target for that metric, such as error rate below 0.1%.
Comprehensive observability in Google Cloud combines Google Cloud Observability tools with third-party solutions tailored to specific needs. Telemetry flows from Google Cloud services, applications, and external sources into the observability platform, where Cloud Monitoring provides insight into resource utilization, performance characteristics, and overall health of resources. Third-party solutions extend this platform with capabilities such as advanced analytics, machine learning-powered anomaly detection, and incident management.
To ensure complete coverage, monitoring must track both system health indicators and business-specific metrics. Important system metrics include:
By tracking these metrics, teams can identify potential bottlenecks, performance issues, and resource constraints before users are affected. Alerts configured on these metrics notify relevant teams proactively about potential issues or anomalies, so response starts when the signal fires rather than after an outage.
Continuous monitoring is the ongoing process of tracking, collecting, and evaluating performance data to optimize resource use and meet performance expectations. Metrics show resource behavior over time, logs provide valuable insight into system events, errors, and warnings, and traces provide detailed information about the flow of requests through an application. Analyzing logs and traces together helps identify potential issues, find root causes, and understand how applications behave under different conditions.
Metrics such as round-trip time between services help locate bottlenecks inside distributed workloads. For long-term analysis, log data flows from Cloud Logging through aggregated sinks to destinations such as Cloud Storage, Pub/Sub, or BigQuery (opens in a new tab). When log data is exported to BigQuery, it can be visualized to identify trends and make predictions about future demand and performance.
Measuring progress depends on distinguishing key performance indicators (KPIs) from supporting metrics. A KPI is a quantifiable measurement that reflects overall performance, while a metric is a specific data point that contributes to the calculation of a KPI. Reviewing metrics regularly supports a data-driven approach that reveals trends, patterns, and anomalies that unsystematic tracking would otherwise miss.
Cloud Monitoring enables real-time monitoring of cloud resources and services, while BigQuery stores and analyzes the data gathered through monitoring. Data flows from monitored resources to Cloud Monitoring for real-time views, then to BigQuery for storage and analysis, then to custom operational dashboards for visualization. An operational dashboard provides a centralized view of the most important metrics, such as CPU utilization, memory usage, network traffic, and latency for a particular application, so teams can quickly identify areas that need attention and take steps to resolve them.
Automation and Infrastructure as Code (IaC) is the practice of defining and managing cloud resources through machine-readable code instead of manual console clicks or one-off scripts. Cloud APIs (opens in a new tab) let code provision, configure, and manage resources, while version control systems store that code as the source of truth. When that code flows through continuous integration/continuous delivery (CI/CD) pipelines, deployments become repeatable, testable, and auditable. The result is fewer manual configuration errors, faster rollbacks, and a known good state the operations team can return to.
Infrastructure as Code is a declarative approach where code describes the desired state of infrastructure and automation brings the environment to that state. Adopting declarative IaC, as opposed to imperative scripting that lists step-by-step commands, creates repeatable infrastructure and gives the operations team a known good state. An IaC process also improves rollbacks, audit of changes, and troubleshooting because every change is a code change that can be reviewed, tested, and reverted. Teams can adopt IaC early so later optimizations are also defined as code, or adopt it gradually to evaluate fit for the environment.
Code moves from version control to Google Cloud through provisioning and configuration tools that divide the work:
| Area | Purpose | Example tools |
|---|---|---|
| Provisioning | Create networks, projects, clusters, and other cloud resources | Terraform (opens in a new tab), Config Controller |
| Configuration | Configure operating systems and software on provisioned resources | Ansible, Chef, Puppet |
| Policy and operations | Enforce constraints and manage operations as code | Open Policy Agent (OPA), Policy Controller, GitOps |
Terraform provisions Google Cloud resources from declarative files, while Ansible, Chef, and Puppet configure those resources after they exist:
Because infrastructure is code, a failed optimization can be rolled back by reverting the code change, and unexpected failures can be avoided by testing changes before they are applied.
Exam tip: Choose declarative IaC over imperative scripting when the goal is repeatable infrastructure with a known good state and efficient rollbacks.
Policy as code is the extension of IaC that expresses governance rules as code and checks infrastructure code before it is deployed. When IaC is combined with CI/CD pipelines and automation, teams can audit infrastructure changes over time and run automatic checks on infrastructure code before changes reach production. Policy Controller enforces IaC constraints in the CI/CD pipeline to avoid cloud misconfigurations, using OPA as its policy engine. This shifts preventive controls left, so customized checks run automatically as part of the pipeline rather than as manual reviews after deployment.
Eliminating toil means using automation to remove repetitive manual work so time can be reinvested in higher-value tasks like optimization. Per the Site Reliability Engineering (SRE) recommendation, the way to eliminate toil is by increasing automation, and not every task needs specialized engineering effort because a short script run on a schedule can save hours per day. Google Cloud provides building blocks that teams combine to automate repetitive tasks, where Cloud Scheduler (opens in a new tab) triggers work on a schedule and services such as Cloud Run and Cloud Composer run the work:
Traffic in this pattern flows from the schedule or event source to the automation runner, and then from the runner through Cloud APIs to the target resources.
Automated provisioning and management is the use of IaC plus CI/CD pipelines to deploy cloud resources and workloads consistently. IaC is not a security tool on its own, but it improves platform security by making infrastructure repeatable, auditable, and recoverable to a known good state. Automation tools such as Config Controller, Terraform, Jenkins, and Cloud Build (opens in a new tab) execute the deployment, while version control preserves history for audit and rollback. The enterprise foundations blueprint is Google's opinionated design for building a secure application environment with IaC and automation, with step-by-step instructions to configure and deploy Google Cloud topology using Terraform and Cloud Build.
A typical IaC-driven change follows an ordered flow from code to production where each step depends on the prior approval:
Teams can start from the enterprise foundations blueprint and modify its scripts to meet their own security requirements, then extend it with additional blueprints for developer platforms, serverless architectures on Cloud Run, generative AI and machine learning, or secured BigQuery data warehouses.
Automated secure application releases use CI/CD pipelines to carry code through the software development lifecycle (SDLC) with consistent security requirements. Without automated tools, deploying, updating, and patching complex environments consistently is difficult and prone to manual error. Automated CI/CD pipelines remove manual errors, provide standardized development feedback loops, and enable efficient product iterations. Automation also helps teams detect and fix security bugs early, fast, and reliably by scanning artifacts when they are created, narrowing the scope of security reviews, and rolling back to a known and safe version.
The pipeline enforces progression rules between environments so only verified artifacts move from development to test to production. Google Cloud tools divide this work by stage, where Cloud Build builds and tests artifacts and Cloud Deploy (opens in a new tab) manages progressive delivery to target environments. Because policies can differ by environment, production can require stronger verification than development while still using the same repeatable pipeline mechanics.
A Service Level Indicator (SLI) is a specific quantitative measure of the level of service provided, such as average request latency, error rate, or system throughput, while a Service Level Objective (SLO) is the target value or range for that SLI, such as average request latency lower than 100 milliseconds. Cloud Monitoring collects and analyzes the metrics that track SLIs over time so teams can verify whether SLOs and service level agreements are being met.
The ordered incident response sequence is communication and triage, followed by investigation, then resolution, and finally review and prevention. During communication and triage the alert is scoped and prioritized with status shared along escalation paths, investigation uses observability data from monitoring and logging to diagnose the cause, resolution restores service through a fix or rollback to a known good state, and review feeds into a retrospective that produces preventive measures.
Declarative Infrastructure as Code (IaC) describes the desired state of infrastructure and automation brings the environment to that state, while imperative scripting lists step-by-step commands. This declarative approach creates repeatable infrastructure with a known good state and improves rollbacks, audit of changes, and troubleshooting because every change is a code change that can be reviewed, tested, and reverted.
Policy Controller enforces Infrastructure as Code (IaC) constraints in the continuous integration and continuous delivery (CI/CD) pipeline to avoid cloud misconfigurations, using Open Policy Agent (OPA) as its policy engine. This shifts preventive controls left so customized checks run automatically as part of the pipeline rather than as manual reviews after deployment.
An enterprise is standardizing its deployment process across multiple Google Kubernetes Engine (GKE) environments (development, staging, and production) to align with the Operational Excellence pillar of the Google Cloud Well-Architected Framework.
Currently, developers manually run kubectl commands from their local workstations, leading to configuration drift, deployment inconsistencies, and elevated credential sprawl.
The team wants to implement an automated CI/CD pipeline using Cloud Build that satisfies the following requirements:
container.deployments.create and container.deployments.update) without broad administrative control over the clusterWhich strategy should the cloud architect recommend?
Last updated