Observability Cost Control in Google Cloud

Managing application performance management (APM) tooling costs means controlling how much telemetry is ingested, stored, and queried while keeping enough visibility to troubleshoot reliably. In Google Cloud, Cloud Trace (opens in a new tab), Google Cloud Managed Service for Prometheus, and Cloud Logging drive most observability spend through span ingestion, per-sample metric ingestion, and log-bucket storage. Cloud Billing (opens in a new tab) dashboards, exports, and alerts provide the governance layer that connects that technical usage to teams, budgets, and chargeback models. Cost control therefore happens in the telemetry pipeline first, and cost accountability happens in billing data second.

1.1Configure Cost-Effective Sampling for Cloud Trace

Cloud Trace sampling configuration determines which requests produce spans that flow from instrumented applications to the Trace backend for storage and analysis. Cloud Trace costs scale directly with ingested span volume, so the sampling rate is the primary cost control. Lowering the rate reduces ingested spans and cost, while raising it preserves more distributed traces for performance analysis. Balance diagnostic coverage against cost tolerance by keeping higher sampling for critical or high-latency paths and lower sampling for high-volume, healthy traffic.

1.2Optimize Collection Frequency and Filtering for Metrics and Profiles

Metric and profile collection cost grows with scrape frequency, cardinality, and retention, so collection frequency and filtering control both cost and performance. Data flows from scrape targets to an ingestion pipeline and then to a global datastore, and every sample sent to that datastore incurs cost. A metric with high sample frequency can degrade workload performance as well as increase spend, so sample intervals must be evaluated before extending monitoring. Sampling and aggregation retain representative signals for troubleshooting without storing excessive raw telemetry.

Google Cloud Managed Service for Prometheus charges on a per-sample basis, which makes cost controllable and predictable. Cost is controlled with three levers that act before samples reach the global datastore:

• Sampling period: changing the metric-scraping period from 15 seconds to 60 seconds can save 75% without sacrificing cardinality, configured on a per-job, per-target, or global basis.
• Filtering: use filtering and metric-relabeling configs to drop metrics at ingestion time based on label matchers, so unwanted samples never reach the global datastore.
• Keep high-cardinality, low-value data local: run standard Prometheus alongside the managed service with the same scrape configs and keep exploratory data locally.

Predictability comes from what is not counted as a billable sample. Sparse histograms count only the first non-zero value and then only when the value in bucket n is greater than bucket n-1; values 10 10 13 14 14 14 count as three samples, which typically means 20% to 40% fewer billed samples than raw bucket counts. Ephemeral containers created by Horizontal Pod Autoscaler (HPA) or GKE Autopilot bill only while running, rather than charging a full month of cardinality each time a container starts. All user queries, including queries issued by Prometheus recording rules, are charged separately through Cloud Monitoring API calls.

1.3Reduce Cloud Logging Storage Costs That Support Observability

Cloud Logging storage costs apply only to log data stored in log buckets, so routing and exclusion decisions determine the bill. Log entries flow from sources to log sinks, where exclusion filters decide whether entries are streamed to log buckets or dropped. A sink can exclude all entries matching a filter or only a percentage of matching entries, and excluded entries are not streamed and do not count against storage allotment. Exclusion filtering is therefore the direct way to remove low-value, high-volume logs while preserving audit and troubleshooting signals.

Log sinks can also route entries away from log buckets to BigQuery (opens in a new tab), Cloud Storage, or Pub/Sub for cheaper long-term storage, extended analysis, or streaming.

1.4Track Observability Spend With Billing Dashboards and Alerts

Cloud Billing provides the usage and cost data that makes observability spend visible across projects and workloads. Billing data flows from Cloud Billing to built-in reports and, when deeper analysis is needed, through export to BigQuery for analysis and to Looker Studio for visualization. Automated notifications send cost anomalies, budget thresholds, and optimization opportunities by email or chat directly to owners who can act. Tailored periodic reports, anomaly alerts, and interactive dashboards together create continuous visibility instead of a single monthly spreadsheet.

Three dashboard patterns cover different governance needs. Built-in views are fastest for service breakdowns, the FinOps hub tracks realized savings, and custom exports support chargeback and multicloud analysis.

The Cost Explorer complements billing views with current and historical visualizations of cost and utilization metrics. Exporting detailed billing data to BigQuery and joining it with Monitoring logs lets teams structure spend with business context and extract patterns such as rising log bytes, metric samples, or trace spans per workload.

1.5Establish Chargeback Models and FinOps Governance

Chargeback and showback models assign every observability cost to the team, product, environment, or cost center that caused it. The Google Cloud project hierarchy is the natural starting point because identity and access management permissions, security policies, and costs are attributed to projects and folders visible in billing exports. Tags and labels add granularity below the project level for application, environment such as development, staging, and production, team, and region. A robust metadata strategy rests on validity, automation, and simplicity: tags such as team=banking_appdev, region=emea, and product=frontend must map to business indicators, tagging must be enforced automatically with tools like Terraform, and rules must stay simple enough for operational teams to follow.

Governance depends on a centralized FinOps team that builds processes, tooling, and policies for mandatory tagging, budget reviews, and optimization, and intervenes when engineering teams do not own their costs. That team aggregates requirements across the organization to maximize commitment-based discounts, while product teams own day-to-day resource optimization with tools such as autoscaling, rightsizing insights from Active Assist, and GKE cost-allocation metrics. Executive sponsorship from CTO, CFO, and CIO roles, incentives such as reinvested savings and public recognition, and transparent rules for shared-cost allocation keep accountability durable. The result is that Trace sampling, Prometheus scraping, and Logging retention decisions are made by the teams that own both the reliability outcome and the budget impact.

Optimizing Cloud Logging (opens in a new tab) ingestion, routing, and retention means controlling which logs are stored, where they flow, and how long they are kept to lower cost while keeping operational visibility. Log entries flow from a Google Cloud project, folder, or organization through sinks and exclusion filters into log buckets or export destinations, and each bucket applies one retention rule to everything it holds. Cost control comes from excluding noisy low-value logs before storage, using tailored buckets, and routing long-term data to cheaper destinations.

2.1Control Ingestion With Exclusion Filters

Log exclusion filters are rules on a sink or bucket that discard matching log entries before they are stored or routed, so excluded volume incurs no ingestion or storage overhead in Cloud Logging. Exclusion is the primary control for high-volume Data Access audit logs, which can be very large and expensive, and development projects are a common place to exclude them entirely. Admin Activity audit logs cannot be excluded, configured, or disabled, and they are still generated even if the Cloud Logging API is disabled, so exclusion planning must leave compliance and admin activity intact.

Manage costs by estimating bills from usage data, configuring alerting policies on usage, and excluding audit logs that are not useful for troubleshooting or security. Data flows from the emitting resource to the routing layer first, then exclusion filters drop unwanted entries, and only the remaining entries flow to the destination bucket or sink. Because filtering happens before storage, a precise filter on resource type, log name, or severity directly reduces ingestion overhead without losing required audit coverage.

2.2Understand System-Provided Log Buckets

System-provided log buckets are the _Required and _Default buckets that Cloud Logging automatically creates in every project, folder, billing account, and organization. Neither bucket can be deleted, and both support an upgrade to use Observability Analytics for SQL querying without additional upgrade cost. The two buckets divide responsibility by compliance value, which determines what can be filtered and how long entries are kept. The _Required bucket stores only entries that originate in the same resource and only entries required for compliance, while the _Default bucket stores all other entries and its contents can be modified with filters.

2.3Design Custom Log Buckets With Tailored Retention

Custom log buckets are user-defined Cloud Logging buckets created in a chosen project and location to isolate workloads with different compliance, access, and cost needs. Custom retention rules apply to all logs in the bucket for 1 day to 3650 days, regardless of log type or whether the log was copied from another location. After creation the bucket region cannot be changed, so location selection controls data residency for the life of the bucket.

A custom bucket depends on its project for Identity and Access Management control, encryption, and lifecycle protection. Access flows from IAM roles to the bucket or to narrower constructs, including log views for subsets of entries, tags for group-based edit access, and field-level access for sensitive fields in an entry. Protection options include customer-managed encryption keys (CMEK) instead of Google-managed keys, and a lock against updates to prevent deletion of entries still within retention, which is critical when the bucket holds audit data.

2.4Route With Cost-Efficient Sinks to Cloud Storage and BigQuery

Log sinks are routing rules that send some or all logs from a source project, folder, or organization to a supported destination such as Cloud Storage (opens in a new tab), BigQuery (opens in a new tab), or Pub/Sub. An aggregated sink collects logs from across an organization into a single central Logging bucket or a single BigQuery dataset, which avoids duplicating storage in every project. Routing to Cloud Storage favors low-cost long-term preservation, while routing to BigQuery favors partitioned analysis with configurable partition expiration.

Centralized routing follows an ordered workflow where each step depends on the prior step:

1. Set up a single destination: Create the central Logging bucket with gcloud logging buckets create or the central BigQuery dataset with bq mk in the aggregation project.
2. Create an aggregated log sink: Define the sink at the organization or folder with a filter that selects only needed logs, so unwanted volume never flows downstream.
3. Grant access to the sink: Grant the sink writer identity write permission on the destination bucket or dataset.
4. Configure read access to the destination: Grant analysts Logs Viewer or Private Logs Viewer for Logging buckets, or dataset read roles for BigQuery.
5. Verify routing: Confirm entries arrive in the destination before adding exclusion filters that drop the original copies.

The _Default bucket typically grants Logs Viewer access to its default view without Data Access entries, while Private Logs Viewer is required for Data Access entries, so central destinations need explicit read grants to preserve least privilege. If a central Logging bucket or BigQuery dataset with an aggregated sink already exists, update that sink filter instead of creating a parallel export that would duplicate ingestion and storage cost.

Control of Cloud Monitoring custom metrics and ingestion rates is the practice of limiting how much metric data is written to Cloud Monitoring (opens in a new tab) so operational visibility stays within budget. Google Cloud Managed Service for Prometheus (opens in a new tab) charges for the number of samples ingested into Cloud Monitoring and for read requests to the Monitoring API, and the number of samples ingested is the primary contributor to cost. Total ingestion grows with two dimensions: the number of time series created by metrics and label combinations, and the number of samples collected from each series over time.

3.1How Ingestion Billing Works

Ingestion billing for Google Cloud Managed Service for Prometheus is based on per-sample pricing, which makes charges scale with the time that targets are actually running. Pricing is designed to be predictable for two difficult cases. Sparse histograms are not fully counted: only the first non-zero bucket value and then buckets where the value is greater than the previous bucket are counted, so a histogram with values 10 10 13 14 14 14 counts as three samples. Depending on histogram use, this typically results in 20% to 40% fewer billable samples than the absolute bucket count would indicate.

Per-sample pricing also avoids penalizing rapidly scaled, preemptible, or ephemeral containers, such as those created by Horizontal Pod Autoscaler or GKE Autopilot. If billing were per-metric, each new container would incur a full month of cardinality at once; with per-sample billing, charges apply only while the container is running. All queries, including queries issued by Prometheus recording rules, are charged separately through Cloud Monitoring API calls. Running the managed service outside Google Cloud can also incur data-transfer fees into Google Cloud and potentially out of the other cloud, which can be minimized with gzip compression.

To view the bill for the managed service, follow the Billing Reports path in order:

1. Go to Billing: In the Google Cloud console, go to the Billing page.
2. Select account: If there is more than one billing account, select the linked billing account for the current project or choose a different account under Manage billing accounts.
3. Open Reports: In the Cost management section of the Billing navigation menu, select Reports.
4. Select service: From the Services menu, select the Cloud Monitoring option.
5. Select SKUs: From the SKUs menu, select Prometheus Samples Ingested and Monitoring API Requests.

3.2Reduce Time Series by Filtering Custom Metrics

Filtering custom metrics reduces the number of time series sent to the managed service by dropping metrics and targets that have no operational value. Open-source Prometheus guidance rarely recommends filtering because cost is bounded by machine cost, but with a managed service billed per unit, sending unlimited data can create unnecessarily high bills. The exporters in the kube-prometheus project, especially the kube-state-metrics service, can emit hundreds of metrics, and a fresh three-node cluster using kube-prometheus sends approximately 900 samples per second before filtering.

Filtering flows from the scrape configuration to the collector to Cloud Monitoring. An operator first narrows scrape configs to scrape fewer targets, then applies metric filters in the collection configuration for managed collection or self-deployed collection. For kube-state-metrics, a Prometheus relabeling rule with a keep action in PodMonitoring or ClusterPodMonitoring retains only wanted metrics and drops the rest before ingestion. Data that is useful locally but not worth sending to the global datastore can be kept in standard Prometheus running alongside the managed service with the same scrape configs.

3.3Adjust Scraping Intervals to Control Sample Volume

Scraping-interval tuning controls how often each time series is sampled, which directly controls how many samples each series contributes to the bill. Total samples ingested equals the number of active time series multiplied by samples per series, so lengthening the interval from frequent to less-frequent scrapes proportionally lowers ingestion while shortening it increases cost and resolution. The tradeoff is visibility delay: a longer interval lowers cost but can miss short spikes and slows detection, while a shorter interval improves fidelity for latency-sensitive or fast-changing signals.

The scraping interval is set in the collector configuration that feeds Google Cloud Managed Service for Prometheus. Managed collection sets the interval in PodMonitoring or ClusterPodMonitoring resources, while self-deployed collection sets it in the Prometheus scrape configuration. Choose a longer interval for high-volume, low-value infrastructure metrics and reserve short intervals for golden signals, service-level indicators, and alerting inputs where missed points change decisions.

3.4Limit High-Cardinality Labels and Aggregate Locally

High-cardinality control limits how label dimensions multiply one metric name into many billable time series. Each unique combination of label values, such as pod name, container ID, request path, or user ID, creates a separate time series that is then scraped on every interval. A misconfigured or overly detailed label can therefore cause runaway cardinality where sample volume grows far faster than the number of metrics suggests.

Self-deployed collection provides two additional controls for this problem: local aggregation and limits on samples from high-cardinality metrics. Local aggregation combines raw samples on the collector before export, so fewer samples flow from the collector to Cloud Monitoring. Cardinality limits cap samples from misconfigured high-cardinality metrics before they reach the global datastore. Both controls sit between the scrape targets and Cloud Monitoring, and both preserve budget by discarding dimensional detail that is not needed for dashboards or alerts.

3.5Identify and Attribute High-Volume Sources

Cost attribution identifies which metrics, namespaces, and projects drive Prometheus Samples Ingested so filtering and interval changes target the largest sources first. The Billing Reports view isolates Cloud Monitoring SKUs for Prometheus Samples Ingested and Monitoring API Requests, which separates ingestion cost from query cost. The Cloud Ops Ingestion view and the Metrics Management (opens in a new tab) page then connect that billing total to specific telemetry inside the project.

The Metrics Management page in Cloud Monitoring reports ingestion volumes for byte- and sample-based billing across metric domains and for individual metrics. The same page reports label and cardinality data for metrics, the number of reads for each metric, use of metrics in alerting policies and custom dashboards, and the rate of metric-write errors. Operators use that linkage to exclude unneeded metrics entirely, which eliminates their ingestion cost, while retaining metrics that support alerts and dashboards.

Key Facts

• Google Cloud Managed Service for Prometheus bills per sample ingested, so total cost equals the number of time series multiplied by samples per series.
• Changing a Prometheus scraping period from 15 seconds to 60 seconds can save 75% without sacrificing cardinality.
• Cloud Logging _Required buckets retain logs for 400 days and cannot be changed or deleted, while _Default buckets retain logs for 30 days by default and custom buckets allow 1 to 3650 days.
• Admin Activity audit logs cannot be excluded, while Data Access audit logs are optional and are the main candidate for exclusion to control Cloud Logging cost.
• Chargeback for observability uses the Google Cloud project hierarchy plus tags and labels to assign Trace, Prometheus, and Logging costs to teams and environments.