Prepare and test your skills
Prepare and test your skills
Execute Load Testing and Performance Benchmarking is the practice of simulating realistic and extreme traffic to prove that an application meets its Service Level Objectives (SLOs) under normal and peak load. In Google Cloud, load tests validate how backend services, Cloud Load Balancing (opens in a new tab), and autoscaling work together when demand spikes. Traffic flows from clients through Cloud Load Balancing to backends in managed instance groups or Kubernetes clusters, so a bottleneck in any layer can cause slow responses or errors for users.
What load tests measure is the visible behavior clients experience plus the internal health of the servers that serve them. External measurements show how fast and reliably requests complete, while internal measurements show whether hosts are running out of CPU, memory, or connections. A complete test collects both sides so a slow response can be linked to a specific limit.
Connection concurrency is often lower than request concurrency because HTTPS clients reuse one connection for multiple requests. Thread pool limits protect against memory and CPU exhaustion, but defaults are often conservative and can prevent full use of system resources. An Apache server using the Prefork Multi-Processing Model assigns one process per client connection, so the pool size sets the upper bound on connection concurrency.
Capacity planning versus overload testing are two different goals that use the same load-testing tools in different ways. Capacity planning finds the load threshold where performance is still acceptable, while overload testing finds how availability behaves when demand exceeds that threshold. Because careful forecasts can still underestimate real demand, both types are needed to meet SLOs.
| Approach | Question it answers | How success is judged |
|---|---|---|
| Capacity planning | What load can the service handle with acceptable performance | Request rate that keeps 99th-percentile latency below the SLO and keeps resource use at optimal levels, which can be well below 100% |
| Overload testing | What happens to availability when the system is pushed past its limits | How throughput, latency, and error rate change, and whether the service sheds load gracefully or crashes |
Exam tip: For capacity planning, measure the request rate that meets the 99th-percentile latency SLO without exceeding optimal resource use, not the absolute maximum request rate.
Benchmarking application resource efficiency means running load tests to learn the relationship between load on the application and resources such as CPU and memory that it consumes. Efficiency can get worse when load is exceptionally high, and it can change over time as code and traffic patterns change. Tests should cover both normal and peak load conditions and should be repeated at regular intervals.
Benchmark results feed directly into autoscaling and sizing decisions for Compute Engine managed instance groups (MIGs), Google Kubernetes Engine (GKE) clusters, and other scalable backends. If tests show that latency rises or resources saturate before peak traffic, teams can adjust resource allocation, tune application configuration such as thread pool size, or change VM machine type and disk choices. This cycle validates that autoscaling adds capacity fast enough to protect SLOs during spikes.
Planning capacity to support peak load is the workflow that turns forecasts and benchmarks into provisioned, redundant infrastructure. Traffic flows from clients through Cloud Load Balancing to backends in MIGs or GKE, so each layer depends on having enough quota, capacity, and redundancy to absorb failover and spikes. The workflow runs first from forecast to reservation, then repeats as utilization is monitored.
A five-step process flow for planning capacity to support peak load: forecast the application load, estimate capacity requirements, plan N+1 redundancy, benchmark and tune, and manage quotas and reserve capacity, with a dashed feedback loop from monitoring utilization back to forecasting.
When integrating with an Application Load Balancer, first measure the backend service on its own without the load balancer under controlled conditions. This baseline exposes tradeoffs between throughput and latency and shows whether the bottleneck is in the app, thread or connection pools, or host resources. Only after the baseline is known, test with Cloud Load Balancing in place to validate traffic distribution, health checking, and autoscaling under realistic traffic patterns.
Making load and performance testing continuous means running performance and load tests regularly and automating checks inside delivery pipelines. Load testing simulates realistic traffic patterns, while stress testing pushes the system to its limits to find bottlenecks before they affect production. Regular testing with Cloud Load Balancing and load-testing services lets teams adjust infrastructure and applications for optimal performance and scalability.
Conduct Security Assessments and Penetration Testing is the systematic checking of production workloads for weaknesses through vulnerability scanning, configuration audits, and simulated attacks. In Google Cloud, this work validates both security and reliability, because an exploitable flaw, misconfigured firewall rule, or unpatched container can become an outage. Assessments run on the customer's side of the trust boundary under the shared responsibility model, while Google validates its own global infrastructure with separate resilience testing.
Shared responsibility for security testing in Google Cloud divides ownership at the service boundary between Google's infrastructure and the customer's workloads. Google is responsible for the physical infrastructure, control planes, and global services that operate across zones and regions, including periodic zone and region disconnect testing in a production-equivalent internal region. The customer is responsible for monitoring its own applications for security incidents, configuring audits and logging for its projects, and reporting incidents to Google as necessary.
Because global resources such as Virtual Private Cloud (VPC) networks and global load balancers can become single points of failure, security tests that touch them require rigorous control. A misconfigured forwarding rule can stop the frontend from receiving user requests even when the backend is intact. Treat any test or change involving global resources as high-risk, verify it before production, and deploy test changes progressively to a few resources in one zone first, then expand only after monitoring shows expected behavior.
Vulnerability scanning for artifacts and running workloads checks software before and after it runs, so flaws are caught both in the supply chain and in production. Build-time scanning inspects container images and language packages stored as artifacts, while runtime scanning inspects workloads that are already deployed and running. Findings flow from the scanner to centralized monitoring and then to remediation, which depends on an established service-level agreement (SLA) for flaw remediation and an evaluation of the risk and impact of each change.
Agent and application controls for scanning include:
Choose artifact scanning when the decision is whether an image is safe to promote, and choose runtime scanning when the decision is whether a running workload has drifted into a vulnerable state. Both depend on centralized logging and alerting to ensure results reach the team that owns the workload.
Security audits, logging, and centralized monitoring provide the evidence that vulnerability scanning and penetration tests are working. Security Command Center is the central service for security posture and analytics when enabled at the organization level, with alerts configured from its findings. Audit and network telemetry flows from workload projects and VPC networks to Cloud Logging and centralized analysis, where correlation and intrusion detection review it for attack patterns.
Effective audit coverage combines multiple telemetry sources that sit on different sides of the network and identity boundaries:
Security posture checks that audits commonly validate include enforcing two-step verification, ensuring minimum Transport Layer Security (TLS) 1.2 for Google Cloud services, rotating Cloud Key Management Service (Cloud KMS) keys every 90 days, rotating user-managed service account keys every 90 days or less, and restricting application programming interface (API) keys by host, app, and API. When log metric filters and alerts exist for ownership changes, audit configuration changes, custom role changes, and firewall rule changes, a penetration test that attempts privilege escalation or firewall bypass will generate a detectable event.
Exam tip: Artifact scanning checks stored images before deployment, while runtime scanning checks workloads already running in production; enable both so a flaw missed at build time is still caught after deploy.
Penetration testing and load testing for production reliability are complementary parts of a broader reliability practice that also includes performance testing and failure injection. Performance and load testing simulates realistic traffic patterns with tools such as Cloud Load Balancing to confirm that applications handle peak volumes without degradation. Stress testing pushes the system to its limits to identify bottlenecks, while chaos engineering and simulated failure scenarios validate that systems recover gracefully and maintain high availability.
| Test type | Primary question | How it runs in production |
|---|---|---|
| Vulnerability assessment | What known flaws and misconfigurations exist in artifacts and infrastructure? | Automated scanning of images, VMs, buckets, keys, and IAM, plus audit-log review |
| Penetration test | Can an attacker chain those weaknesses to gain access or disrupt service? | Controlled, scoped simulation against the customer's own applications and configurations |
| Load and stress test | Will the system stay available under peak or extreme traffic? | Simulated traffic sent through load balancers in progressive stages with monitoring |
Run these tests as an ordered workflow to protect reliability. First, define incident response roles for commander, investigator, communicator, and technical expert, with communication and escalation paths documented in a runbook. Then, automate unit, integration, and end-to-end tests within continuous integration and continuous delivery (CI/CD) pipelines so changes are validated before production. Only after pipeline tests pass, run load and penetration tests progressively in production, monitor availability with the Google Cloud Service Health Dashboard and Personalized Service Health, and revert quickly to the last stable state if issues appear.
Implement Chaos Engineering and Failure Injection Testing is the practice of intentionally introducing faults into a production or production-equivalent environment to prove that recovery works before a real outage happens. In Google Cloud, experiments simulate events such as zone failures, region failures, and network interruptions, and then measure whether automated recovery meets recovery time objective (RTO), recovery point objective (RPO), and Site Reliability Engineering (SRE) error budgets. The goal is to confirm that traffic reroutes, data survives, and services resume within objectives without manual rescue.
How Google validates zone and region failures is through zone and region disconnect testing in a dedicated, production-equivalent environment. This test environment uses an internal-only Google Cloud region that is architecturally identical to customer-facing regions, and its network connectivity is then physically disconnected to simulate a total zonal or regional failure. The goal is to confirm that local services inside the disconnected area go offline as expected while control planes and services operating in other zones and regions remain unaffected.
Validation runs on a consistent, periodic schedule, which creates a repeatable feedback loop that keeps global infrastructure resilient against large-scale outages. Customers do not run this global disconnect test themselves, but they benefit from the same design principle: test failover regularly in an isolated environment before relying on it in production.
Designing fault-injection experiments for workloads on Google Cloud tests how an application behaves when a zone or region has a 1, 5, 10, or 30 minute outage. A well-designed experiment answers operational questions at many layers rather than only checking whether servers restart:
Business-critical operations that are subject to RTO and RPO must depend only on data plane components that continuously process operations for the service. The data plane carries user traffic and ongoing reads and writes, while the management plane manages configuration state and pushes configuration to the control plane and data plane. A recovery plan therefore must not depend on management-plane operations such as a VM-creation API or an update to an Identity and Access Management permission.
Exam tip: RTO measures how long after a disaster until service resumes, while RPO measures how much data can be lost; recovery that depends on creating VMs or changing permissions will miss RTO because it relies on the management plane.
Validating automated recovery mechanisms proves that Google Cloud can recreate capacity and reroute traffic without manual steps. Compute Engine (opens in a new tab) managed instance groups automatically recreate failed virtual machines (VMs) from instance templates, and Cloud Load Balancing uses health checks to route traffic quickly from an unhealthy zone to healthy VMs in another zone or region. Persistent Disk regional disks synchronously replicate data to another zone in the same region, while multi-regional services such as Cloud Storage and Spanner replicate data across regions to support active-active topologies.
The expected result of an experiment depends on resource scope, because zonal, regional, and multi-regional resources have different failure boundaries:
| Resource scope | Resilience | Recovery expectation during test |
|---|---|---|
| Zonal | Low | Failure in the zone affects resources in that zone; traffic must shift to another zone or region |
| Regional | Medium | Resources stay available from another zone; a region outage affects resources in that region |
| Multi-region or global | High | Service continues from another region; risk shifts to configuration changes |
After a regional mitigation is in place, the mitigation must be tested regularly. There are few worse outcomes than assuming a workload survives a single-region outage and learning during a real event that failover between regional resources in multiple regions does not work.
Measuring resilience with SRE practices and load testing uses service level indicators (SLIs), service level objectives (SLOs), and error budgets to judge whether recovery was fast and complete enough. Google Cloud Observability tools together with Cloud Monitoring collect metrics, traces, and logs, and teams build alerting and dashboards that show whether backup intervals, replication lag, and recovery time stay within RPO and RTO. Google Cloud Service Health Dashboard shows current status across all regions, while Personalized Service Health shows incidents relevant to a specific project and supports API access and alert configuration.
Regular performance, load, and chaos testing provides repeatable evidence that the system can recover gracefully and handle peak demand:
Capacity planning measures the request rate that keeps 99th-percentile latency below the Service Level Objectives (SLOs) without exceeding optimal resource use, while overload testing measures how throughput, latency and error rate change past limits and whether the service sheds load gracefully or crashes. Capacity planning answers what load can be handled with acceptable performance and overload testing answers what happens to availability when demand exceeds that threshold.
Configure artifact vulnerability scanning to detect known vulnerabilities in stored images and packages before deployment and to decide whether an image is safe to promote, and configure runtime vulnerability scanning to detect new vulnerabilities in running containers and virtual machines after deployment and to decide whether a running workload has drifted into a vulnerable state. Enable both so a flaw missed at build time is still caught after deploy, with findings flowing to centralized monitoring for remediation.
Recovery Time Objective (RTO) measures how long after a disaster until service resumes, while Recovery Point Objective (RPO) measures how much data can be lost. Business-critical operations subject to RTO and RPO must depend only on data plane components that continuously process operations, so a recovery plan must not depend on management-plane operations such as a virtual machine creation API or an Identity and Access Management permission update because it will miss RTO.
An e-commerce platform running on Google Cloud uses Regional Managed Instance Groups (MIGs) behind an external Application Load Balancer, with Cloud Monitoring tracking Service Level Indicators (SLIs) against a 99.99% availability Service Level Objective (SLO). The Site Reliability Engineering (SRE) team wants to conduct a chaos engineering experiment in a staging environment that mirrors production to validate that automated recovery mechanisms function properly during infrastructure disruptions without exceeding the allocated error budget.
Which strategy should the team implement to design and validate this failure injection experiment?
Last updated