Decoupling Services with Eventarc Async Events

What graceful degradation is

Graceful degradation is a fault-tolerance design pattern where a system continues to operate, but with reduced functionality, when a component fails. In this context, bypassing the inventory service and logging a metric to Cloud Monitoring allows the main order flow to continue without being blocked by the downstream outage.

Why it doesn't meet the requirement

• Ensure no data is lost: Fails. Writing a summary metric (such as an error count or a dropped request counter) to Cloud Monitoring only records that an event occurred. The actual payload and critical details of the inventory update are discarded and lost.
• Resume from the point of failure: Fails. Because the detailed event data is not logged durably in a messaging system or database, the inventory service has no way to process the missed updates once it recovers.

Key characteristics

• Prioritization: Prioritizes core business flows (like successfully accepting a customer's order) over secondary flows (like updating backend inventory).
• Fallback mechanisms: Often involves returning cached data, default values, or simplified UI elements when a backend service is down.
• Trade-offs: Inherently sacrifices some functionality, data consistency, or completeness to maintain overall system availability.

This answer would be correct if

The business requirements explicitly stated that real-time inventory updates were non-critical, and it was acceptable to permanently drop the data to ensure the order management service remained fast and highly available during an outage.

Additional context

To achieve both high availability and zero data loss, systems should move away from synchronous graceful degradation that drops data, and instead adopt loosely coupled components using asynchronous eventing.

What a circuit breaker is

A circuit breaker is a fault-tolerance design pattern used to detect failures and encapsulate the logic of preventing a failure from constantly recurring. When a downstream service (like the inventory service) fails repeatedly, the circuit breaker "trips" into an open state and immediately returns an error to the caller instead of attempting the actual synchronous call.

Why it doesn't meet the requirement

• Ensure no data is lost: Fails. While a circuit breaker prevents the order management service from blocking, returning an immediate error means the event data is rejected. Unless the order management service implements complex local queuing, durable storage, and retry logic, the inventory update data is permanently lost.
• Resume from the point of failure: Fails. The circuit breaker pattern itself does not store, log, or queue the payload for future processing once the downstream service recovers.

Key characteristics

• Fails fast: Prevents resource exhaustion by failing fast during an outage, returning an error immediately.
• State management: Typically implements a "half-open" state to periodically test if the downstream service has recovered before fully closing the circuit.
• Client responsibility: Requires the calling service to handle the immediate failure gracefully, often by returning a default value or an error to the user.

This answer would be correct if

The scenario required failing fast and rejecting new requests during an outage, and data loss or dropping the request was an acceptable business trade-off to protect system resources and maintain API responsiveness.

Additional context

While a circuit breaker is an excellent pattern for protecting synchronous APIs from cascading failures, it is not a substitute for an asynchronous event-driven architecture when data durability and guaranteed eventual processing are strict requirements.

What the bulkhead pattern is

The bulkhead pattern is a resilient design pattern that isolates elements of an application into separate pools so that if one fails, the others will continue to function. Deploying a service across multiple Google Kubernetes Engine (GKE) clusters and using Cloud Load Balancing is a way to isolate regional, zonal, or cluster-level failures, ensuring that an issue in one environment does not take down the entire service.

Why it doesn't meet the requirement

• Isolate failures between services: Fails. While this isolates failures within the inventory service's infrastructure, it does not decouple the upstream order management service from the downstream inventory service. The communication remains strictly synchronous.
• Prevent cascading effects: Fails. If a global traffic spike overwhelms all GKE clusters simultaneously, or if the backend database goes down, the order management service will still block, wait, and time out, leading to cascading failures.
• Resume from the point of failure: Fails. Load balancing and bulkheads do not durably queue or log requests for later processing if all backends become unavailable.

Key characteristics

• Fault isolation: Divides resources to prevent a single point of failure from taking down the entire system, much like the watertight compartments of a ship.
• High availability: Improves overall availability by distributing traffic across healthy instances and regions.
• Synchronous limitation: Does not change the synchronous nature of the communication between the client and the server.

This answer would be correct if

The primary goal was to protect the inventory service from complete regional outages by distributing its compute resources, rather than decoupling the upstream and downstream services to handle traffic spikes and ensure data durability.