Professional Cloud DevOps Engineer

Draining/redirecting traffic

When an incident occurs, you can reduce its impact by carefully managing how user traffic reaches your services. Draining traffic means gradually stopping new requests from going to a specific instance or version of your service while letting existing connections finish their work. This is often done before shutting down an unhealthy or outdated server to prevent users from experiencing errors. Redirecting traffic involves shifting requests away from a problematic component to a healthy one, often using a load balancer or a global traffic manager. The decision to drain or redirect depends on whether you need to gracefully stop a single resource or immediately reroute all users to a backup system.

Adding capacity

A common way to mitigate an incident's impact is to add more resources to handle the increased load or to replace failed components. This can be done manually or automatically through autoscaling. Autoscaling policies watch metrics like CPU usage or request latency; when a threshold is crossed, the system automatically adds more virtual machine instances or container pods to the pool. Adding capacity helps maintain performance and availability during a traffic surge or partial failure. However, it's important to understand that scaling takes time—new instances must be provisioned, start up, and pass health checks before they can receive traffic—so planning for this delay is part of incident response.

Rollback strategies

If a recent change caused an incident, rolling back to a previous, known-good version of your application or infrastructure is a key mitigation tactic. A rollback typically involves redeploying an earlier, stable configuration. In a continuous deployment pipeline, this can often be triggered automatically if health checks fail after a new release. The strategy depends on your deployment method: for containerized applications, you might roll back to a previous image version in your registry; for infrastructure as code, you revert to a prior commit in your configuration repository. A successful rollback quickly restores service, but it requires that previous versions are kept available and tested, and it may mean temporarily losing the features or fixes from the failed update.