An enterprise web service running on a fleet of Compute Engine Linux VMs generates over 40,000 HTTP access log entries per second per instance during peak traffic. The operations team identifies two major issues: The legacy Cloud Logging agent (google-fluentd) is reaching its single-core CPU limit and dropping log entries due to buffer overflow. Cloud Logging ingestion and storage costs are exceeding budget due to high volumes of redundant HTTP 200 OK access logs. You must design an observability architecture that achieves the following requirements: Ingest 100% of HTTP 5xx error logs for security and root cause analysis. Retain full statistical visibility into total request counts and traffic volume trends across all response codes. Uniformly sample and ingest only 10% of successful HTTP 2xx/3xx access logs into the log bucket. Prevent log dropping and CPU starvation on the VM instances. Which solution should you implement?

Upgrade instances to the Google Cloud Ops Agent; define log-based metrics on the incoming log stream to track overall request counts; configure an exclusion filter on the _Default log sink with sample(insertId, 0.90) for logs with HTTP status codes below 500.: ### What the Solution Is This architecture combines the high-throughput **Google Cloud Ops Agent** with **Log-based Metrics** and **Log Exclusion Filters** using deterministic sampling in Cloud Logging. ### How It Meets the Requirements - **Prevents log dropping and CPU starvation**: The legacy `google-fluentd` agent is CPU-bounded and capped at roughly 5,500 entries per second on a single core. Upgrading to the **Google Cloud Ops Agent** increases throughput capacity to approximately 160,000 log entries per second on Linux, easily handling 40,000 entries per second without buffer overflows. - **Preserves statistical trends and patterns**: In Google Cloud Logging, **Log-based metrics** extract metric data from all incoming log entries before sink exclusion filters are evaluated. Creating a log-based counter metric ensures that 100% of request volumes and latency distributions are recorded in **Cloud Monitoring**. - **Enforces uniform sampling on successful requests**: Configuring an exclusion filter with `sample(insertId, 0.90)` on logs where `httpRequest.status < 500` discards 90% of non-error logs before ingestion into the `_Default` bucket, retaining an exact 10% uniform sample. - **Preserves 100% of error logs**: Logs with HTTP status codes `500` or higher bypass the exclusion filter entirely and are stored in full for debugging and auditing. ### Key Advantages and Capabilities - **Separation of telemetry concerns**: Metrics handle aggregated trends while Logging retains granular context for anomalies. - **Substantial cost optimization**: Reduces ingestion and storage pricing by 90% for high-volume successful transactions. - **Deterministic sampling**: The `sample()` function hashes fields such as `insertId` to produce statistically accurate uniform sampling across distributed instances. - **High performance**: Ops Agent uses a multi-threaded OpenTelemetry and Fluent Bit architecture for lower CPU and memory overhead. ### Why This Is Optimal This design leverages native Cloud Logging pipeline behavior—extracting metric data before exclusion routing—so no monitoring visibility is lost while drastically reducing storage spend and agent compute bottlenecks.

Optimize High Volume Logging Sampling Metrics | Professional Cloud DevOps Engineer

Optimize High Volume Logging Sampling Metrics

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise web service running on a fleet of Compute Engine Linux VMs generates over 40,000 HTTP access log entries per second per instance during peak traffic. The operations team identifies two major issues:

The legacy Cloud Logging agent (google-fluentd) is reaching its single-core CPU limit and dropping log entries due to buffer overflow.
Cloud Logging ingestion and storage costs are exceeding budget due to high volumes of redundant HTTP 200 OK access logs.

You must design an observability architecture that achieves the following requirements:

Ingest 100% of HTTP 5xx error logs for security and root cause analysis.
Retain full statistical visibility into total request counts and traffic volume trends across all response codes.
Uniformly sample and ingest only 10% of successful HTTP 2xx/3xx access logs into the log bucket.
Prevent log dropping and CPU starvation on the VM instances.

Which solution should you implement?

Upgrade instances to the Google Cloud Ops Agent; define log-based metrics on the incoming log stream to track overall request counts; configure an exclusion filter on the _Default log sink with sample(insertId, 0.90) for logs with HTTP status codes below 500.

Modify the logrotate configuration on each VM to use copytruncate with postrotate agent restarts; deploy log-based metrics on the _Default log bucket; route 100% of logs to BigQuery for analysis.

Configure the legacy Logging agent with a record_transformer filter to trim payload sizes below 256 KiB; create a Cloud Monitoring uptime check for traffic metrics; route all logs to a Cloud Storage cold archive bucket.

Keep the legacy Logging agent and assign multiple worker processes on port 24231; apply an exclusion filter of sample(insertId, 0.10) for HTTP 5xx error logs; rely on Logs Analytics SQL queries for request metrics.

Professional Cloud DevOps Engineer

Optimize High Volume Logging Sampling Metrics

Related questions