Monitor Performance and Control Cloud Costs

Collecting performance information in Google Cloud means gathering metrics, logs, and traces into Cloud Monitoring to understand health, performance, and cost. Numeric metric data flows from Google Cloud services and instrumented applications into a project, where charts, dashboards, and alerting policies evaluate it. The same performance signals later guide resource sizing, autoscaling, and recommendations delivered by the Recommender API.

1.1Metric Sources from System Telemetry to Custom Signals

Metric collection in Cloud Monitoring combines automatic system telemetry with agents and instrumentation to cover different layers of the stack. The correct source depends on where the workload runs and whether the needed signal already exists or must be defined. For application code, vendor-neutral OpenTelemetry is recommended because the same instrumentation can collect logs, metrics, and traces and export them to any vendor, including Google Cloud.

Custom metrics capture combined conditions that no single built-in metric describes, such as aggregated workflow duration. Google Cloud Observability provides the unified platform for collecting, analyzing, and visualizing these metrics alongside traces, logs, and profiles, and the metrics scope of a project controls which time-series data can be charted without moving it.

1.2Health Exposure, Autohealing, and Synthetic Monitoring

Health exposure makes system health observable so traffic flows from the load balancer only to healthy resources. Health checks typically use a pull model where the checker pulls data from an application endpoint to determine service status. When a load balancer is associated with a group of VM instances, its health check defines how the load balancer communicates with the VMs to decide which instances should continue to receive traffic.

Load-balancer health checks also drive autohealing for managed instance groups, where unhealthy machines are re-created and removed from service. If external traffic is load-balanced through a GKE ingress resource, GKE automatically creates appropriate health checks for the load balancer. Inside the cluster, Kubernetes liveness and readiness probes help the orchestrator manage pods and requests, so applications on Kubernetes should expose endpoints for those probes. Synthetic monitors test availability the way a user sees it, including uptime checks that probe HTTP, HTTPS, and TCP endpoints and broken-link checkers that crawl web pages.

1.3Golden Signals, SLOs, Dashboards, and Alerting

Key metrics are the small set of descriptive signals that together indicate scaling needs, saturation, or unhealthy services. No single metric usually indicates whether to scale an app or whether a service is unhealthy, so effective strategies combine multiple factors into custom metrics that reflect real workflows. Tolerance for outliers also shapes metric choice because an average can hide wide imbalances, so strategies evaluate the metric distribution where the 99th percentile is often more informative than the average.

• Latency: time taken to serve a request, revealing slow dependencies and regressions.
• Traffic: volume of demand on the system, revealing load that drives scaling.
• Errors: rate of failed requests, revealing broken functionality or downstream faults.
• Saturation: how full constrained resources are, revealing approaching bottlenecks.

These four golden signals guide monitoring of user-facing distributed systems. Service-level objectives (SLOs) turn collected metrics into target levels of performance or reliability, and alerting policies on SLOs warn when burn rate puts an objective in danger. Dashboards track SLOs and KPIs over time, and custom dashboards can display metric charts and tables, log data and error groups, alert charts and alert information, text, and events such as a reboot or crash. Each notification links to an incident that stores status, links to logs, a chart of recorded metric data, labels, and duration.

1.4Diagnosing Bottlenecks with Traces, Logs, and Profiles

Bottleneck diagnosis correlates resource metrics, request traces, log events, and code profiles to distinguish contention, distributed latency, and inefficient code. Cloud Trace records the detailed flow of requests through distributed applications, breaking each request into spans that reveal which service hop, database query, or API dependency dominates total latency. Cloud Profiler (opens in a new tab) is a statistical, low-overhead profiler that continuously gathers CPU and memory-allocation data from production Go, Java, Python, and Node.js applications and attributes cost to source code. Cloud Logging stores, searches, and analyzes log events, and the same trace identifier links slow spans to errors, warnings, retries, and timeouts.

1. Detect saturation in Cloud Monitoring: Confirm elevated CPU, memory, disk I/O, network traffic, or response time and check whether saturation aligns with degraded user experience.
2. Isolate latency in Cloud Trace and logs: Follow the request path to find which hop adds round-trip time, then confirm errors or retries in correlated logs.
3. Pinpoint cost in Cloud Profiler: Attribute the slow path to specific functions with flame graphs and focused views, then verify the fix by comparing profiles before and after the change.
4. Confirm with load testing and tuning: Reproduce the bottleneck under peak, sustained, and spike load, then apply caching, indexing and query optimization, code rewrites, capacity changes, or autoscaling and retest.

Observability metrics for GKE clusters flow into infrastructure views for CPU, memory, cAdvisor, kubelet, control plane, events, startup latency, storage, workload state, and GPU utilization, and into application views for per-second request rate, error rates by operation and response code, and 50th and 95th percentile latency. High request utilization, rising restarts, many unschedulable pods, or sustained 100% GPU use point to over-provisioning, crashes, insufficient resources, or bottlenecks.

1.5Aligning Resources, Autoscaling, and Budgets to Performance Data

Resource optimization uses utilization and performance signals to match resources to actual demand while staying within budget. Cloud Monitoring provides real-time visibility into CPU, memory, network, disk I/O, and response times, while Recommender analyzes utilization history into idle VM, idle resource, and machine-type suggestions. Teams compare resource signals against latency, error rates, and throughput before moving to a less expensive shape, then reduce footprint further with caching, indexing, and code profiling. Continuously monitor and improve performance (opens in a new tab) depends on exporting metrics for multi-quarter analysis because Cloud Monitoring retains metrics for several weeks to support recent investigation.

Discount and budget choices depend on predictability and tolerance for interruption. Committed use discounts provide significant reductions for one-to-three-year usage commitments suited to steady workloads, sustained use discounts apply automatic Compute Engine credits after continuous use passes thresholds, and Spot VMs offer unused capacity at significantly lower price but can be preemptively stopped. Cloud Billing enforces the budget boundary through budgets, alerts, anomaly detection, and reports, and exporting billing data to BigQuery enables attribution by label, project, or time period.

1.6Active Assist Findings, Access Control, and Prioritization

Active Assist insights and recommendations are machine learning-generated findings and actions delivered by the Recommender API (recommender.googleapis.com) to improve usage, cost, and security posture. Insights highlight notable patterns in resource usage, while recommendations propose specific remediation actions with projected impact. Data flows from the Recommender API to the Google Cloud console, to Gemini Cloud Assist chat, and to BigQuery for enterprise-scale analysis, so the same finding can be reviewed interactively or prioritized centrally.

The Active Assist review workflow is the ordered console process for finding, evaluating, and acting on a recommendation for a project, folder, or organization:

1. Open Active Assist: Go to Active Assist in the Google Cloud console.
2. Select scope: Use the resource selector to select the project, folder, or organization to evaluate.
3. List recommendations: Click All recommendations to display findings that can be filtered, sorted, and shared with affected resources and impact.
4. Inspect detail: Click a recommendation to display supporting data for deciding whether to apply or dismiss it.
5. Actuate decision: If permitted, click Apply or Dismiss, or follow console instructions to apply the change manually.

Recommendation access depends on APIs and Identity and Access Management (IAM) roles that separate viewing from changing resources. The Recommender API must be enabled to retrieve guidance, and a caller needs the Recommender Viewer role to list insights and recommendations. Because viewing requires only the viewer role while applying a change requires resource-specific permission, teams can enforce an approval boundary where analysts review findings but only approvers apply them. Filtering narrows the stream with --insight-type and --filter=insightSubtype, such as DATABASE_AUDITING_NOT_ENABLED, and enterprise review moves IAM findings into BigQuery with the BigQuery Data Transfer Service for centralized triage. The lifecycle moves in order from detected to reviewed to approved to applied, and only the final transition mutates infrastructure. Prioritization balances projected savings, risk reduction, and operational impact, so deleting clearly idle resources ranks first while high-risk access removals rank high even with zero savings.

Implementing FinOps practices for optimizing resource utilization and costs means matching infrastructure size and pricing to actual demand while maintaining reliability. Manage and optimize cloud resources (opens in a new tab) starts in the telemetry pipeline where sampling, filtering, and routing control observability spend, and continues with right-sizing, lifecycle management, autoscaling, and commitment discounts. Cost accountability then happens in billing data where exports, tags, and chargeback assign every cost to the team that caused it.

2.1Controlling Trace Sampling and Prometheus Ingestion Costs

Cloud Trace sampling configuration determines which requests produce spans that flow from instrumented applications to the Trace backend for storage and analysis. Cloud Trace costs scale directly with ingested span volume, so the sampling rate is the primary cost control. Lowering the rate reduces ingested spans and cost, while raising it preserves more distributed traces for performance analysis. Keep higher sampling for critical or high-latency paths and lower sampling for high-volume healthy traffic to balance coverage against budget.

Metric collection cost grows with scrape frequency, cardinality, and retention as data flows from scrape targets to an ingestion pipeline and then to a global datastore. Google Cloud Managed Service for Prometheus charges per sample ingested, so total cost equals the number of time series multiplied by samples per series. Every sample sent to the global datastore incurs cost, and very frequent scraping can also degrade workload performance. Cost is controlled with three levers that act before samples reach the global datastore:

• Sampling period: changing the scrape period from 15 seconds to 60 seconds can save 75% without sacrificing cardinality, configured per job, per target, or globally.
• Filtering: use filtering and metric-relabeling configs to drop metrics at ingestion time based on label matchers, so unwanted samples never reach the global datastore.
• Keep high-cardinality, low-value data local: run standard Prometheus alongside the managed service with the same scrape configs and keep exploratory data locally.

Predictability comes from what is not counted as a billable sample. Sparse histograms count only the first non-zero value and then only when the value in bucket n is greater than bucket n-1, which typically means 20% to 40% fewer billed samples than raw bucket counts. Ephemeral containers created by Horizontal Pod Autoscaler (HPA) or GKE Autopilot bill only while running, rather than charging a full month each time a container starts. All user queries, including queries issued by Prometheus recording rules, are charged separately through Cloud Monitoring API calls.

2.2Managing Log Storage, Routing, and Billing Visibility

Cloud Logging storage costs apply only to log data stored in log buckets, so routing and exclusion decisions determine the bill. Log entries flow from sources to log sinks, where exclusion filters decide whether entries are streamed to log buckets or dropped. A sink can exclude all entries matching a filter or only a percentage of matching entries, and excluded entries do not count against storage allotment. Admin Activity audit logs cannot be excluded, configured, or disabled, while Data Access audit logs are optional and very large, so they are the main candidate for exclusion, especially in development projects. Log sinks can also route entries away from log buckets to BigQuery, Cloud Storage, or Pub/Sub for cheaper long-term storage, extended analysis, or streaming.

System and custom buckets divide responsibility by compliance value and retention need. The _Required bucket holds compliance logs for 400 days with no changes allowed, the _Default bucket holds other logs for 30 days by default with changeable retention, and user-defined buckets allow 1 day to 3650 days with full control over location, access, and locking. Neither system bucket can be deleted, and after creation a custom bucket region cannot be changed. Cloud Billing provides the usage and cost data that makes observability spend visible, where billing data flows from Cloud Billing to built-in reports and, when deeper analysis is needed, through export to BigQuery for analysis and to Looker Studio for visualization. Chargeback assigns every cost to the team, product, environment, or cost center that caused it, using the project hierarchy as the starting point plus tags and labels for application, environment, team, and region.

2.3Understanding Spot Pricing, Billing, and Preemption Risk

Spot VMs are excess Compute Engine capacity sold at discounts of up to 91% off on-demand price with no availability guarantee. Compute Engine can reclaim that capacity at any time to run standard VMs, at which point a Spot VM is either stopped or deleted. Spot VMs therefore fit only fault-tolerant work that can lose and replace capacity without corrupting results.

Spot pricing reflects spare capacity rather than a fixed bid, so both price and availability shift with overall platform usage. Spot prices cover many machine types, GPUs, TPUs, and Local SSD disks, and prices can change up to once every day at midnight Pacific Time. Billing follows the VM lifecycle state rather than preemption alone. If a preempted VM was stopped, the VM enters a TERMINATED state and accrues no VM-hour charges while it remains TERMINATED, but attached persistent disks still incur storage charges until deleted. If Compute Engine preempts a Spot VM less than one minute after creation, there is no charge for that VM use, and the same first-minute waiver applies to attached GPUs and Local SSDs. Premium operating systems are the exception because Spot VMs do not reduce premium OS cost and OS charges continue even when preemption occurs in the first minute. Only Spot VMs in a RUNNING state are considered for preemption, while Spot VMs in a TERMINATED state are not. Stopping then restarting a Spot VM resets preemption exposure because the VM passes through TERMINATED, while resetting or rebooting keeps it RUNNING and does not reset exposure.

2.4Following the Preemption Sequence and Building Resilient Fleets

The preemption process is the ordered sequence Compute Engine follows to reclaim a Spot VM while giving the workload a brief chance to react. The workload detects preemption by monitoring instance metadata, handles it during a short shutdown window, and then the VM moves to its configured final state. The sequence runs in this order:

1. Metadata notice: Compute Engine sets the preempted value in VM default metadata to TRUE, after a notice duration of 120 seconds in Preview or 0 seconds by default.
2. ACPI G2 Soft Off and shutdown period: Compute Engine sends an ACPI G2 Soft Off signal, which triggers any configured shutdown script and starts a best-effort shutdown period of up to 30 seconds.
3. ACPI G3 Mechanical Off: If the Spot VM has not stopped after the shutdown period, Compute Engine sends an ACPI G3 Mechanical Off signal to the operating system.
4. Termination action: Compute Engine applies the per-VM termination action to place the VM in its final state.

A Managed Instance Group (MIG) is the control plane for a Spot VM fleet because the MIG continuously attempts to maintain its target size from an instance template. When Compute Engine preempts Spot VMs in the group, the MIG repeatedly tries to recreate those VMs from the same template, and replacements return to RUNNING only when capacity becomes available again. A Terraform configuration provides the repeatable way to provision Spot VMs because the same scheduling block defines the economic model with provisioning_model = "SPOT", preemptible = true, automatic_restart = false, and instance_termination_action set to STOP or DELETE. Resilient applications treat Spot VMs as ephemeral compute, keep state outside the VM, save progress continuously, and spread replicas, using a shutdown script that copies a checkpoint file to a Cloud Storage bucket plus polling of metadata preempted for in-workload handling.

2.5Right-Sizing Compute, Storage Lifecycle, and BigQuery Processing

Right-sizing Compute Engine instances is the continuous practice of monitoring actual demand and adjusting allocation to avoid over-provisioning, which wastes money, and under-provisioning, which causes slowdowns. Utilization data flows from workloads to Cloud Monitoring for real-time visibility, then from Cloud Monitoring to Recommender for analysis, and Recommender returns machine type recommendations and idle VM findings. Effective analysis tracks CPU utilization, memory usage, network traffic, disk I/O, and application response times and custom metrics such as queue depth or request latency to link infrastructure signals to user experience. Compute Engine custom machine types refine the choice when demand is stable but falls between standard sizes, because CPU and memory are selected independently.

Cloud Storage classes separate actively used data from cold data by access frequency and retention need:

Cloud Storage Object Lifecycle Management (OLM) automatically downgrades or deletes objects when age and other conditions are met, while Cloud Storage Autoclass transitions each object based on its observed access pattern. Use OLM when retention rules are known in advance, such as downgrade after 30 or 90 days, and use Autoclass when objects in the same bucket cool at unpredictable rates. BigQuery analysis cost is driven by bytes processed and slot pricing model. BigQuery on-demand pricing charges per bytes scanned for sporadic queries, while BigQuery editions use committed and autoscaling slots for sustained volume. Reduce processing by storing inputs in Parquet or Avro instead of JSON or CSV, retaining only aggregates long term, and running jobs in the same region as source buckets and datasets.

2.6Aligning Container Resources with Autoscaling and Service Choice

Service selection matches traffic patterns, resource use, and execution duration to the lowest-cost compute service. Cloud Run runs code without managing infrastructure and adjusts instances automatically as traffic rises and falls, Compute Engine with MIGs maintains a defined minimum and maximum number of VMs, and GKE dynamically adjusts cluster resources through complementary autoscalers. The goal is to avoid paying for idle capacity while keeping performance stable during bursts and low-use periods.

Container requests and limits control how efficiently Pods pack onto nodes in GKE. A request is the guaranteed CPU and memory the scheduler uses to place a Pod, while a limit is the maximum a container can consume. GKE coordinates scaling in a fixed dependency order where Vertical Pod Autoscaler rightsizes requests, Horizontal Pod Autoscaler changes replicas, Cluster Autoscaler adds or removes nodes, and Node Auto-Provisioning creates new pool shapes when Pods fit no existing pool. A workload is underprovisioned when CPU or memory utilization exceeds 150% for at least 10% of the time over the last 15 days, and overprovisioned when utilization is below 50% for at least 90% of the time over the last 15 days. Cloud Run provides built-in autoscaling including scale to zero by default, and Spot VMs provide unused capacity at more than 80% lower cost for fault-tolerant batch work with on-demand fallback.

2.7Comparing Commitments, Reconciliation, and Network Tiers

Infrastructure cost planning trades commitment, placement, and network path for lower rates without harming performance. Optimize resource usage (opens in a new tab) covers stable baseline with committed use discounts (CUDs), leaves variable peaks to on-demand plus sustained discounts, and uses Standard Tier where backbone performance is not required.

Purchasing a spend-based commitment requires the Billing Account Administrator role, and the commitment cannot be cancelled or moved after purchase. Each hour, Google Cloud reconciles commitments against actual use in a fixed order:

1. Apply resource-based commitments first: Google Cloud utilizes resource-based commitments and applies credits to eligible hourly usage for that region and machine configuration.
2. Apply compute flexible commitments second: Google Cloud utilizes compute flexible commitments to cover remaining eligible Compute Engine, GKE, and Cloud Run usage.
3. Bill overage at on-demand rates: Google Cloud charges additional hourly usage at on-demand rates, which may then qualify for SUDs.

If usage falls below the commitment, the unused portion is still billed as a commitment fee. Network Service Tiers select the path traffic takes between the internet and Google Cloud. Premium Tier routes traffic within the Google Cloud network with 99.99% SLA uptime, while Standard Tier routes traffic over peering, ISP, or transit networks with 99.9% SLA uptime and lower egress cost. Global external Application Load Balancer and global external proxy Network Load Balancer require global IPs and are not supported in Standard Tier, while regional load balancers, VM instances, and Cloud NAT gateways use regional IPs. Cloud CDN is always Premium Tier.

2.8Applying Recommender Findings Through Controlled Automation

Recommender insights are machine-generated findings about resource usage, and recommendations are the suggested actions that follow. In Google Cloud, Active Assist surfaces both through the Recommender API across cost, security, performance, manageability, and reliability domains. The ordered FinOps workflow is first remove idle waste, then rightsize what remains, and only then commit stable usage to discounted pricing. High-impact cost findings include idle VMs, idle clusters, unattended projects, overprovisioned Compute Engine, GKE, and Cloud SQL instances, and Cloud Run services that should switch to CPU always-allocated.

The controlled lifecycle enforces review, explicit state transitions, and rollback readiness before production change:

1. List: List recommendations for the target project, location, and recommender ID with gcloud recommender recommendations list or GET on the recommendations resource.
2. Claim or dismiss: Mark a recommendation the team intends to apply as claimed, or mark one it will not apply as dismissed, so ownership and intent are recorded before mutation.
3. Apply: Apply the change manually with the resource-specific console action, gcloud command, or REST call identified by the resource field, or automatically through Active Assist.
4. Confirm outcome: Mark the recommendation as succeeded or failed only after verifying resource state and spend effect, preserving the audit trail.

Programmatic ingestion depends on the Recommender API being enabled and on an automation identity with viewer roles, and pipelines must handle pagination with pageSize and pageToken and filter by recommenderSubtype or insightSubtype. To avoid races, mark a recommendation as CLAIMED with its current etag before applying it, execute the operationGroups.operations exactly as described, then verify and allow the recommender to clear the finding. Reliability insights may increase cost, so the FinOps Recommendations dashboard excludes VM findings with negative savings and underprovisioned GKE and Cloud SQL findings, while Database Center can validate Cloud SQL for PostgreSQL changes by cloning the instance and comparing baseline versus recommended throughput and latency.

Key Facts

• The four golden signals for user-facing systems are latency, traffic, errors, and saturation, with 99th percentile distribution often more revealing than averages for tail latency.
• Cloud Monitoring retains metrics for several weeks unless exported, and the metrics scope of a project controls which projects' time-series data can be charted.
• Spot VMs offer discounts up to 91% with no availability guarantee, and only Spot VMs in a RUNNING state can be preempted.
• Google Cloud reconciles commitments each hour in fixed order: resource-based CUDs apply first, compute flexible CUDs apply second, and remaining usage is billed at on-demand rates that may earn sustained use discounts.
• Premium Tier routes traffic within the Google Cloud network with a 99.99% SLA, while Standard Tier routes over peering or ISP networks with a 99.9% SLA and lower egress cost.