Set up alerts for errors and performance metrics

Log-based alerting for error detection is a monitoring method that watches database logs for error patterns and creates an alert when a match appears. Log data flows from Cloud Logging to an alerting policy in Cloud Monitoring, and Cloud Monitoring forwards the alert to configured notification channels. This path catches error patterns, system failures, and security events that numeric thresholds alone can miss.

1.1How Log-Based Alerts Detect Database Problems

Log-based alerts detect database problems by matching a Logging query filter against each incoming log entry. When a database writes an error, failed execution, or system failure to its logs, the filter identifies the pattern and Cloud Monitoring opens an alert. Cloud Monitoring then sends notifications through channels such as email, SMS, or PagerDuty. Multiple types of notification channels should be configured so a single channel failure does not block a critical notification.

1.2Scope and Requirements for Log Filters

Log filter scope defines the boundary that controls which logs an alerting policy can evaluate. The policy evaluates logs at the Google Cloud project level and evaluates only included logs. The policy cannot reach across other scopes or evaluate data types outside that boundary.

• The policy operates only on included logs and ignores excluded logs.
• The policy operates at the Google Cloud project level and cannot filter by log buckets or by other resources such as Cloud Billing accounts or organizations.
• The query must use correct field names and regular expressions from the Logging query language, or the query will return no matching logs.

Database Center (opens in a new tab) does not send notifications about health issues and requires manual dashboard checks, so log-based alerts provide the automatic notification path for critical operational issues.

1.3Creating and Validating Error Filters

Creating an error filter means writing a Logging query that is specific enough to catch real failures without creating noise. The filter is tested in the Query pane in Logs Explorer or with the Preview logs button in the alert-configuration interface. Testing confirms that the query returns the intended error entries before the policy is saved. If the policy creates too many alerts, the query must be made more restrictive by tightening field values or regular expressions. If the policy creates no alerts, the Alerts page in Cloud Monitoring can be filtered by policy name to check whether past alerts exist and whether the query still finds matches.

1.4Notification Delivery and Limits

Notification delivery for log-based alerts follows an ordered path from match to message, and built-in limits control alert volume. First the log entry matches the filter, then Cloud Monitoring creates the alert, then the alert sends notifications to configured channels until daily limits are reached. These limits determine how aggressive a filter should be.

• A policy can create a maximum of 2 new alerts per minute, and label extraction can cause some alerts to be skipped under this limit.
• A policy is limited to 20 alerts per day and 20 notifications per day, and the daily alert limit resets at a random time rather than at midnight.
• The minimum time between notifications can be reduced to receive repeat notices sooner, or increased to reduce noise.

1.5SQL-Based Alerting with Observability Analytics

SQL-based alerting with Observability Analytics (opens in a new tab) is an extension that queries log data stored in BigQuery instead of filtering the live log stream. The log bucket must be upgraded to use Observability Analytics and must have a linked BigQuery dataset. The query must not use a WHERE clause to filter by timestamp because the service automatically applies the correct timestamp filter during evaluation. The Monitoring Service Account must exist and must hold permissions on both the project and the linked dataset for the query to run.

The required permissions for SQL-based alerting connect the project, the service account, and the dataset:

• Monitoring Editor (roles/monitoring.editor) on the project to create and manage policies.
• Logging SqlAlertWriter (roles/logging.sqlAlertWriter) on the project to write SQL alert results.
• Monitoring Service Agent (roles/monitoring.notificationServiceAgent) for the Monitoring Service Account on the project.
• BigQuery Data Viewer (roles/bigquery.dataViewer) for the Monitoring Service Account on the linked BigQuery dataset.

A SQL alerting policy is created through an ordered workflow that moves from query to saved policy:

1. Write query: Enter a SQL query against the log view in the query editor on the Observability Analytics page.
2. Run on BigQuery: Run the query with the BigQuery engine to verify results in the Results table.
3. Create alert: Click Create alert in the Results table to open the Create sql alert policy window with the query included.
4. Set condition: Configure the alert condition and evaluation schedule for the query.
5. Add notifications: Add notification channels, custom subject content, labels, and documentation.
6. Save policy: Review the configuration and save the policy.

A SQL policy can fail when the Monitoring Service Account no longer exists or lacks permission to read the queried log data, when query execution exceeds five minutes, or when an internal error occurs. A failed query writes a log entry containing the alert policy ID and error status. A separate log-based alerting policy can watch for that entry to notify operators that the SQL alert itself has failed.

Metric-based alerting policies in Cloud Monitoring are rules that watch numeric database signals and open an incident when a signal crosses a defined limit. The policy selects a performance metric from the Cloud SQL (opens in a new tab) instance, sets a quantitative threshold and duration window, and defines how notifications and incident closure behave. Metric data flows from the Cloud SQL instance to Cloud Monitoring to the incident and notification path, which lets operations detect resource exhaustion before an outage occurs.

2.1Selecting Critical Performance Metrics

Selecting critical performance metrics means choosing the Cloud SQL signals that show whether an instance is properly sized for its load. Cloud Monitoring collects CPU utilization, memory use, and disk and storage signals from the Cloud SQL instance, and the alerting policy evaluates those time-series values against its conditions. Memory monitoring uses two related signals: the usage metric for memory used by the database process and the total_usage metric for memory used by the database container plus operating system cache.

The core metrics for resource-exhaustion alerts cover compute, memory, and storage pressure:

• CPU utilization, which shows when vCPU demand is consistently high and the instance is undersized.
• Memory usage and total usage, which together show process pressure versus cache use and risk of out-of-memory errors.
• Disk utilization and available space, which show storage growth, binary log retention pressure, and risk of a storage-full condition.
• Disk throughput and IOPS behavior, which show when the I/O subsystem is throttled by disk size or vCPU limits.

2.2Establishing Quantitative Thresholds

Quantitative thresholds are specific numeric limits that separate normal operation from action-required conditions. A Cloud SQL threshold policy keeps at least 20% available disk space for maintenance operations and keeps the usage memory metric under 90% to avoid out-of-memory errors. Design guidance also treats sustained CPU and RAM use near 100% as a high-availability failure risk, aims for under 10,000 tables per instance to control upgrade time, and uses 4 TB or greater disk size when higher throughput and IOPS are required.

2.3Defining Conditions and Duration Windows

An alert condition combines a selected metric, a threshold comparison, and a duration window that the comparison must remain true before an incident opens. The duration window prevents a brief spike from paging on-call staff, while a sustained breach signals overload, undersizing, or a tuning problem such as missing indexes, stale statistics, or heavy updates. The policy also defines when Cloud Monitoring closes incidents in the absence of metric data, so a silent instance does not leave an incident open indefinitely.

When both memory metrics appear high together, the condition indicates the instance is low in memory because of custom configuration, an undersized instance, or both. The response path then flows from the alert to scaling or tuning: first scale up vCPU, RAM, or disk throughput to restore service, then investigate slow queries, locks, index scans, and maintenance history in a test environment. Changing memory size requires an instance restart, and an instance already at maximum memory must be sharded across multiple instances.

2.4Finalizing the Alerting Policy

Finalizing the alerting policy records the response context and creates the rule in Cloud Monitoring. The Documentation field carries runbook information that is included in the notification message, such as which instance, metric, and threshold fired and whether to scale, fail over, or tune. The Alert name identifies the policy, and Create Policy activates evaluation so future metric data flows from the Cloud SQL instance to Cloud Monitoring to the incident path.

The final steps follow in order:

1. Add documentation: Enter the information to include in the notification message, such as severity and remediation steps.
2. Enter alert name: Click Alert name and type a unique name that names the database, metric, and threshold.
3. Create policy: Click Create Policy to save the policy and begin continuous evaluation of the condition.

Manage Notification Channels and Incident Workflows is the practice of connecting Cloud Monitoring (opens in a new tab) alerting policies to people and tools so database issues reach the right stakeholders quickly. An alerting policy detects when a condition is met and Cloud Monitoring creates an incident and sends notifications to the configured communication paths. The incident then moves through a managed workflow of triage, investigation, and resolution inside a centralized system. This connection between detection, notification, and ownership keeps status updates accurate and reduces downtime.

3.1Establish Clear Incident Response Procedures

Clear incident response procedures define who responds to an incident, how information is shared, and when an incident is escalated. The procedures assign roles such as incident commanders, investigators, communicators, and technical experts, so each team member knows their expectations during triage, investigation, and resolution. Communication and escalation paths identify important contacts, set up communication channels, and define the process for escalating incidents to higher levels of management when necessary. Regular training and preparation ensure teams can follow these paths under pressure.

The procedures are documented in a runbook or playbook that serves as the standardized reference guide during an incident. The runbook outlines the steps for communication, triage, investigation, and resolution, and it includes relevant tools and resources plus contact information for important personnel. The runbook depends on current contacts, channels, and tool references, so the runbook must be regularly reviewed and updated to remain effective.

3.2Centralize Incident Management

A centralized incident management system (opens in a new tab) provides a single location for tracking an incident throughout its lifecycle. The system consolidates incident-related data so teams do not search across channels for context, and it integrates with other systems such as monitoring tools and ticketing systems. The system also supports customized workflows, priorities, task assignment, and communication management.

A centralized incident management system provides three related advantages:

• Improved visibility: consolidated incident data gives stakeholders a comprehensive view of the incident status, impact, and progress, which saves time and reduces confusion.
• Better coordination and collaboration: a unified platform for communication and task management gives all departments access to up-to-date information, which reduces miscommunication and misalignment.
• Enhanced accountability and ownership: task allocation to specific individuals or teams makes responsibilities clearly defined and tracked, which encourages proactive problem-solving.

3.3Investigate an Incident

Investigation in Cloud Monitoring starts on the Incident details page, which gathers evidence about why an alerting policy created the incident. The Logs pane shows log entries that match the query of the log-based alerting policy, and those entries can be opened in Logs Explorer or explored further in Metrics Explorer to find patterns and recurring issues. The Labels section identifies the specific monitored resource from the log entry that caused the incident, while the Documentation section shows the user-defined description and mitigation tips from the alerting policy. The Related incidents section helps discover underlying issues by listing incidents created by another condition of the same alerting policy or incidents that share a label with the current incident.

The Incident details page depends on precomputed data that Cloud Monitoring uses to determine when to create an incident. Do not modify the associated alerting policy while working with Google Cloud support, because modification causes Cloud Monitoring to discard that precomputed data and lose the historical record of the incident.

3.4Manage the Incident Lifecycle

Cloud Monitoring incidents move through an ordered lifecycle from detection to closure, and each transition controls whether new notifications are sent. Acknowledging signals ownership without changing detection, while snoozing and closing control notification flow. Acknowledging is recommended when investigation begins and is done from the Incidents table or the Incident details page.

The lifecycle states are:

1. Open: the condition of the alerting policy was met and the incident is still open; if the same condition is met again while an incident is open, a new incident is not opened.
2. Acknowledged: the open incident has been manually marked as acknowledged, which typically indicates that investigation has begun.
3. Closed: the incident was manually closed or was automatically closed after the auto-close period expired.

Snoozing the alerting policy prevents Cloud Monitoring from creating new incidents and sending notifications during a specific period, which is useful during outages and troubleshooting, but open incidents remain open and still close based on the auto-close duration. Monitoring automatically closes an incident when the auto-close duration expires, with a default of 7 days and a minimum of 30 minutes.

Key Facts

• Log data flows from Cloud Logging to Cloud Monitoring, and Cloud Monitoring creates an alert and sends notifications when a log entry matches the alert filter.
• A log-based alerting policy is limited to 2 new alerts per minute, 20 alerts per day, and 20 notifications per day.
• Cloud SQL memory pressure is judged with the usage metric under 90%, compared with total_usage to see memory held in operating system cache.
• Cloud Monitoring incidents move in order through Open, Acknowledged, and Closed, with automatic closure after a default of 7 days and a minimum of 30 minutes.