Collecting Performance Metrics, Traces and Profiles

Application performance monitoring in Google Cloud is the practice of collecting metrics, logs, and traces into Cloud Monitoring (opens in a new tab) to understand health, performance, and cost. Cloud Monitoring automatically ingests system metrics, events, and metadata from most Google Cloud services and can serve as a backend for open-source tools. Numeric metric data stored in a project is visualized in charts and dashboards and evaluated by alerting policies. The metrics scope of a project controls which time-series data can be charted, and by default the scope is limited to time-series data stored by that project alone, while a project whose scope includes other projects can chart their data without moving it.

1.1Collect Metrics from Infrastructure and Applications

Metric collection in Cloud Monitoring combines automatic system telemetry with agents and instrumentation to cover different layers of the stack. The right source depends on where the workload runs and whether the needed signal already exists or must be defined. For application code, vendor-neutral OpenTelemetry is recommended because the same instrumentation can collect logs, metrics, and traces and export them to any vendor, including Google Cloud.

Custom metrics capture combined conditions that no single built-in metric describes, such as aggregated workflow duration. Google Cloud Observability (opens in a new tab) provides the unified platform for collecting, analyzing, and visualizing these metrics alongside traces, logs, and profiles from Google Cloud services, applications, and external sources.

1.2Expose Health and Validate User Experience

Health exposure makes system health observable so traffic flows from the load balancer only to healthy resources. Health checks typically use a pull model where the checker pulls data from an application endpoint to determine service status. When a load balancer is associated with a group of VM instances, its health check defines how the load balancer communicates with the VMs to decide which instances should continue to receive traffic.

Load-balancer health checks also drive autohealing for managed instance groups, where unhealthy machines are re-created and removed from service. If external traffic is load-balanced through a GKE ingress resource, GKE automatically creates appropriate health checks for the load balancer. Inside the cluster, Kubernetes liveness and readiness probes help the orchestrator manage pods and requests, so applications on Kubernetes should expose endpoints for those probes. Monitor application health and performance (opens in a new tab) with synthetic monitors that test availability, consistency, and performance the way a user sees it, including uptime checks that probe HTTP, HTTPS, and TCP endpoints and broken-link checkers that crawl web pages.

1.3Define Golden Signals, SLOs, Dashboards, and Alerts

Key metrics are the small set of descriptive signals that together indicate scaling needs, saturation, or unhealthy services. No single metric usually indicates whether to scale an app or whether a service is unhealthy, so effective strategies combine multiple factors into custom metrics that reflect real workflows. Tolerance for outliers also shapes metric choice because an average can hide wide imbalances across operations or shards, so strategies evaluate the metric distribution where the 99th percentile is often more informative than the average.

• Latency: time taken to serve a request, revealing slow dependencies and regressions.
• Traffic: volume of demand on the system, revealing load that drives scaling.
• Errors: rate of failed requests, revealing broken functionality or downstream faults.
• Saturation: how full constrained resources are, revealing approaching bottlenecks.

These four golden signals guide monitoring of user-facing distributed systems. Service-level objectives (SLOs) turn collected metrics into target levels of performance or reliability, and alerting policies on SLOs warn when burn rate puts an objective in danger. Dashboards track SLOs and KPIs over time, and custom dashboards can display metric charts and tables, log data and error groups, alert charts and alert information, text, and events such as a reboot or crash. Each notification links to an incident that stores status, links to logs, a chart of recorded metric data, labels, and duration, and many integrations provide pre-configured alerts where operators only specify who should be notified.

1.4Diagnose Bottlenecks Across Metrics, Traces, and Profiles

Bottleneck diagnosis correlates resource metrics, request traces, log events, and code profiles to distinguish contention, distributed latency, and inefficient code. Cloud Trace records the detailed flow of requests through distributed applications, breaking each request into spans that reveal which service hop, database query, or API dependency dominates total latency. Cloud Logging stores, searches, and analyzes log events, and the same trace identifier links slow spans to errors, warnings, retries, and timeouts. Cloud Profiler is a statistical, low-overhead profiler that continuously gathers CPU and memory-allocation data from production Go, Java, Python, and Node.js applications and attributes cost to source code.

1. Detect saturation in Cloud Monitoring: Confirm elevated CPU, memory, disk I/O, network traffic, or response time and check whether saturation aligns with degraded user experience.
2. Isolate latency in Cloud Trace and logs: Follow the request path to find which hop adds round-trip time, then confirm errors or retries in correlated logs.
3. Pinpoint cost in Cloud Profiler: Attribute the slow path to specific functions with flame graphs and focused views, then verify the fix by comparing profiles before and after the change.
4. Confirm with load testing and tuning: Reproduce the bottleneck under peak, sustained, and spike load, then apply caching, indexing and query optimization, code rewrites, capacity changes, or autoscaling and retest.

View observability metrics (opens in a new tab) for GKE clusters and workloads through infrastructure views for CPU, memory, cAdvisor, kubelet, control plane, events, startup latency, storage, workload state, and GPU utilization, and through application views for per-second request rate, error rates by operation and response code, and 50th and 95th percentile latency. High request utilization, rising restarts, many unschedulable pods, or sustained 100% GPU use point to over-provisioning, crashes, insufficient resources, or bottlenecks.

1.5Optimize Resource Allocation and Cost from Performance Data

Resource optimization uses utilization and performance signals to match resources to actual demand while staying within budget. Cloud Monitoring provides real-time visibility into CPU, memory, network, disk I/O, and response times, while Recommender analyzes utilization history into idle VM, idle resource, and machine-type suggestions. Teams compare resource signals against latency, error rates, and throughput before moving to a less expensive shape, then reduce footprint further with caching, indexing, and code profiling. Cloud Monitoring retains metrics for several weeks to support recent investigation, so export metrics for multi-quarter or multi-year analysis of bottlenecks, scaling events, and architecture trends.

Discount and budget choices depend on predictability and tolerance for interruption. Committed use discounts provide significant reductions for one-to-three-year usage commitments suited to steady workloads, sustained use discounts apply automatic Compute Engine credits after continuous use passes thresholds, and Spot VMs offer unused capacity at significantly lower price but can be preemptively stopped. Cloud Billing enforces the budget boundary through budgets, alerts, anomaly detection, and reports, and exporting billing data to BigQuery enables attribution by label, project, or time period.

Active Assist insights and recommendations are machine learning-generated findings and actions delivered by the Recommender API (opens in a new tab) (recommender.googleapis.com) to improve usage, cost, and security posture. Insights highlight notable patterns in resource usage, while recommendations propose specific remediation actions with projected impact. Data flows from the Recommender API to the Google Cloud console, to Gemini Cloud Assist chat, and to BigQuery for enterprise-scale analysis, so the same finding can be reviewed interactively or prioritized centrally.

2.1Review and Apply Remediation Actions

The Active Assist review workflow is the ordered console process for finding, evaluating, and acting on a recommendation for a project, folder, or organization. Visibility follows ownership boundaries, so organization or folder recommendations are visible only with permissions for that level. Remediation means changing allocation to match demand through resizing, deletion of idle resources, or configuration changes such as autoscaling policies, where over-provisioning creates unnecessary cost and under-provisioning creates bottlenecks.

1. Open Active Assist: Go to Active Assist in the Google Cloud console.
2. Select scope: Use the resource selector to select the project, folder, or organization to evaluate.
3. List recommendations: Click All recommendations to display findings that can be filtered, sorted, and shared with affected resources and impact.
4. Inspect detail: Click a recommendation to display supporting data for deciding whether to apply or dismiss it.
5. Actuate decision: If permitted, click Apply or Dismiss, or follow console instructions to apply the change manually.

Custom metrics extend this model to automated response when an operator defines a workload-relevant metric, configures an alert threshold, and routes the alert to administrators who adjust allocation. Validation then compares CPU utilization, memory usage, network traffic, disk I/O, application response times, and business-specific metrics before and after the change to confirm that utilization improved without degrading user experience.

2.2Configure Access, Filtering, and Enterprise Review

Recommendation access depends on APIs and Identity and Access Management (IAM) roles that separate viewing from changing resources. The Recommender API must be enabled to retrieve guidance, and a caller needs the Recommender Viewer role to list insights and recommendations. Gemini Cloud Assist (opens in a new tab) uses the same dependency to answer prompts such as Give me recommendations to reduce my resource costs or Which of my VMs are idle. Because viewing requires only the viewer role while applying a change requires resource-specific permission, teams can enforce an approval boundary where analysts review findings but only approvers apply them.

Filtering narrows the insight stream to one recommender family and one pattern so queues stay actionable. Insights flow from the Recommender service to the console, to gcloud, and to the REST API with the same filter concept. For example, Cloud SQL security work uses --insight-type=google.cloudsql.instance.SecurityInsight with --filter=insightSubtype=DATABASE_AUDITING_NOT_ENABLED to surface only instances without auditing in a region such as us-central1. IAM rightsizing uses recommender = "google.iam.policy.Recommender" and recommender_subtype = "REMOVE_ROLE" to select only least-privilege removals.

Enterprise review moves recommendations into BigQuery for centralized triage. The BigQuery Data Transfer Service pulls data from the IAM recommender into a dataset and table, and data then flows from the dataset to analysts and to visualization tools such as Looker. The transfer depends on the BigQuery API being enabled and on the export being configured before queries can run. A scoped query selects recommendation_details where recommender = "google.iam.policy.Recommender" and recommender_subtype = "REMOVE_ROLE" to isolate removal candidates for approver review in Sheets.

2.3Apply Changes and Integrate with Monitoring

Applying a recommendation changes the target resource only after approval, and the method depends on the resource type. In the console, the operator opens the recommendation panel and clicks Edit Instance to enable database auditing on a Cloud SQL instance (opens in a new tab). With gcloud and the API, the operator applies the same outcome by enabling auditing directly on the instance. The lifecycle therefore moves in order from detected to reviewed to approved to applied, and only the final transition mutates infrastructure.

Monitoring integration places findings alongside health, performance, and deployment signals so cost and reliability tradeoffs are visible together. Cloud Monitoring creates custom dashboards, defines custom metrics, and routes alerts to PagerDuty or Slack, while Cloud Deploy status for canary releases and infrastructure health from Compute Engine, GKE, TPUs, and GPUs flow into the same views. Supporting telemetry completes the incident context: Cloud Logging collects logs for root-cause review, Error Reporting captures errors linked to degraded resources, Cloud Trace identifies latency between components, and Cloud Profiler analyzes production code for CPU or memory bottlenecks. When volume is high and remediation is well understood, a scheduled BigQuery transfer plus a saved filtered query continuously surfaces repeat patterns for dashboards, ticketing, or remediation scripts.

2.4Prioritize by Savings, Risk, and Operational Impact

Prioritization of Active Assist recommendations balances projected savings, risk reduction, and operational impact before any production change. Each criterion measures a different kind of value, so a high score on one does not guarantee a high rank overall. High savings with low impact, such as deleting clearly idle resources, ranks first, while high-risk access removals rank high even with zero savings because they reduce privilege-escalation paths.

Cost and performance types include idle VM recommendations, BigQuery partitioning and clustering recommendations that reduce scanned bytes, and near-limit quota findings that predict availability failures. Security posture work focuses on unused orphaned accounts and abandoned service accounts through the IAM recommender, which uses machine learning to recommend removal or rightsizing and supports least privilege. Policy Analyzer answers the complementary question of who currently has sensitive access, such as who can read a BigQuery dataset containing personally identifiable information (PII) or delete specific VM instances. Database Center assessments (opens in a new tab) validate Cloud SQL for PostgreSQL changes by cloning the instance, replaying benchmark load on the clone, and comparing baseline throughput and latency against compute expansion, Enterprise to Enterprise Plus edition upgrade, or data cache activation.

Key Facts

• Cloud Monitoring collects metrics, logs, and traces into dashboards and alerting policies, with metrics scope controlling which projects' time-series data can be charted and with metrics retained for several weeks unless exported.
• The four golden signals for user-facing systems are latency, traffic, errors, and saturation, with 99th percentile distribution often more revealing than averages for tail latency.
• Load-balancer health checks determine which VM instances receive traffic and drive autohealing of managed instance groups, while Kubernetes liveness and readiness probes manage pods and uptime checks probe HTTP, HTTPS, and TCP endpoints.
• Active Assist recommendations are delivered by the Recommender API, require the Recommender Viewer role to view, and move through a lifecycle from detected to reviewed to approved to applied.
• Database Center validates Cloud SQL for PostgreSQL changes by cloning the instance and comparing baseline throughput and latency against performance after the recommended change.