Integrating Monitoring with PagerDuty and Webhooks

Customizing alert payloads and troubleshooting delivery failures is the practice of adding actionable context to notifications and tracing failed deliveries from source to receiver. Cloud Monitoring (opens in a new tab) evaluates log or metric data from Cloud Logging and Cloud Monitoring, then sends a payload through a notification channel to email, Slack, PagerDuty, SMS, Pub/Sub, or a webhook. When delivery fails, diagnosis follows the same path in reverse, from the receiver back through the channel, the alert policy, and the originating log.

1.1Enriching Alert Payloads with Critical Context

Enriching an alert payload means ensuring the notification carries the facts a responder needs to triage without opening multiple consoles. A complete notification typically carries the same core context:

• The affected Google Cloud services and projects
• A description of the issue or incident
• The time the issue was detected
• A unique identifier for the incident
• Links to relevant dashboards or support resources

Cloud Monitoring alert policies are the mechanism within Google Cloud Observability that delivers these notifications to a team. For network telemetry, Cloud Network Insights (opens in a new tab) uses log-based alerting where an alarm writes a structured log entry named networkmanagement.googleapis.com/insights_alarm to Cloud Logging, and the alert policy fires from that log. Predefined alert policy templates simplify configuration for common scenarios such as a Critical Network Alarm, and each policy connects to existing notification channels to deliver the enriched payload by email, Slack, PagerDuty, SMS, or Pub/Sub.

1.2How Alert Data Flows to Third-Party Tools

Alert data flows from the monitored system to the third-party incident platform through an ordered chain where each step depends on the previous step:

1. Source event writes to Cloud Logging: A threshold deviation writes networkmanagement.googleapis.com/insights_alarm, while a structural change such as a Monitoring Point going offline and online again or a network route change writes networkmanagement.googleapis.com/insights_event.
2. Alert policy evaluates the log: The log-based alert policy in the Google Cloud console evaluates the structured log entry and opens an incident when its conditions match.
3. Notification channel delivers the payload: The alert policy sends the payload through the configured channel to the destination, where data flows from the channel to PagerDuty, Slack, webhook, SMS, email, or Pub/Sub.
4. Downstream automation triggers: A Pub/Sub topic can fan out to additional automation, where the topic delivers to a Cloud Run function that runs remediation or enrichment logic.

Ingestion on the receiving side depends on correct ownership of contacts and visibility. An incident response group email is configured for the Security or All category in Essential Contacts, and security administrators are granted the Advisory Notifications Viewer role (roles/advisorynotifications.viewer) so advisories are visible in the Google Cloud console.

1.3Troubleshooting Delivery Failures and Network Path Issues

Troubleshooting delivery failures means isolating whether the break is in log generation, policy evaluation, channel configuration, or the network path to the receiver. Cloud Logging audit logs provide the diagnostic record for what was written, what was evaluated, and what access or configuration change may have blocked delivery. Diagnosis must check each dependency in order because Cloud Monitoring depends on Cloud Logging for the trigger and the third-party platform depends on the notification channel for the payload.

• Audit log analysis shows whether the expected insights_alarm or insights_event entry reached Cloud Logging and whether calls such as SetIamPolicy changed who can view or manage alerts.
• Channel and contact checks verify that Essential Contacts has the correct group email, that the notification channel points to the correct webhook URL or PagerDuty service, and that the Advisory Notifications Viewer role still permits console visibility.
• Network path checks use Connectivity Tests to test connectivity between a source and destination endpoint across the VPC network, VPC Network Peering, and VPN tunnels to an on-premises receiver, including identifying when a firewall rule is blocking connectivity.
• Agent and flow evidence confirms source health by checking whether Monitoring Points report Active or Offline, by reviewing VPC Flow Logs for traffic sent or received by VM instances, and by enabling Firewall Rules Logging at the individual rule level to record TCP and UDP connections inside the VPC.

When the path crosses a trust boundary from Google Cloud to a third-party receiver, the channel depends on firewall, routing, and authentication remaining open. Fixing the upstream log or the downstream path alone does not restore delivery until both sides succeed in order.

Integrating native incident response platforms automates alert response by sending Monitoring incidents directly to an external on-call service. Cloud Monitoring (opens in a new tab) moves data from a Cloud Monitoring alerting policy to a PagerDuty notification channel to a PagerDuty service, where the event creates a PagerDuty alert that follows that service's escalation policy. This design keeps detection in Google Cloud while ownership for paging, escalation, and resolution tracking lives in the external platform for high-availability operations.

2.1Native PagerDuty Notification Channel

The native PagerDuty notification channel connects Cloud Monitoring to a PagerDuty service through the PagerDuty Events API v1 integration. The operator first creates the endpoint in PagerDuty and copies the integration key, then creates the channel in Google Cloud, and only after the channel is saved can alerting policies select PagerDuty in the Notifications section. The channel is therefore a dependency for every policy that pages PagerDuty.

1. Create the PagerDuty endpoint: Add an Events API v1 integration to a new or existing PagerDuty service from the Service Directory and copy the integration key.
2. Create the Monitoring channel: In Monitoring Alerting, open Edit notification channels, add a new PagerDuty channel, set the Display Name to match the PagerDuty service, and paste the integration key into Service Key.
3. Attach the channel to policy: In each alerting policy, select PagerDuty in Notifications and choose the saved PagerDuty configuration.

The integration key is the trust link between the two systems because possession of the key authorizes Cloud Monitoring to post events into that specific PagerDuty service. Keeping the Display Name aligned with the PagerDuty service name preserves clear ownership when many services and channels exist, including App Hub configurations where the channel is managed in the App Hub host project or management project.

2.2Automated Incident Creation and Lifecycle Payload

The PagerDuty payload carries the full Monitoring incident context needed for automated incident creation and lifecycle synchronization. Cloud Monitoring sends a JSON packet with a description string and a details object that uses schema version 1.2, and the alerting backend attempts to generate that version by default. Data flows from the Monitoring alerting backend to the PagerDuty service endpoint, where an opened Monitoring incident triggers a PagerDuty alert and a later resolved or closed Monitoring incident updates that external alert with end time and state.

The details.incident object identifies both sides of the relationship:

• Identity and navigation: incident_id, scoping_project_id, scoping_project_number, and url link the PagerDuty alert back to the originating Monitoring incident in the Google Cloud console.
• Lifecycle timing and state: started_at, ended_at, state, and renotify describe when the incident started, when it ended, and whether it is still active or closed.
• Affected target: resource, resource_id, resource_name, resource_display_name, resource_type_display_name, metric, and metadata describe the monitored resource type such as gce_instance, its labels, the metric type such as compute.googleapis.com/instance/cpu/utilization, and system and user labels.
• Policy cause: policy_name, policy_user_labels, documentation, condition, condition_name, severity, threshold_value, and observed_value identify the alerting policy and condition that opened or resolved the incident and the value that crossed the threshold.

The payload schema is subject to the Google Cloud deprecation policy, but the formats of generated field values can change without notice. Consume values such as incident.summary, incident.documentation.content, and incident.url as a whole and do not rely on parsing inside those generated strings, because the schema guarantees the field exists but not its internal text format.

2.3Mapping Monitoring Severity to PagerDuty Escalation

Severity mapping aligns the Monitoring incident severity with the PagerDuty alert severity that drives escalation policies. By default, PagerDuty notifications show all Monitoring severity levels as critical, so without extra configuration a low-severity warning and a critical outage escalate the same way. The Monitoring incident.severity field carries the policy severity, and if no severity was defined during alerting policy creation, Cloud Monitoring sets the value to no severity.

To preserve the intended urgency, configure event orchestration rules in PagerDuty that read event.custom_details.incident.severity and set the corresponding PagerDuty alert severity before escalation runs:

1. Open orchestration: In PagerDuty, go to Services, select the service receiving Google Cloud notifications, open Settings, then open Event Management and View Orchestration Rules.
2. Create a conditional rule: Click New Rule and choose If events match certain conditions for when the rule applies.
3. Match Monitoring severity: In the code editor If section, enter event.custom_details.incident.severity matches part 'VALUE', where VALUE is one Monitoring severity value, then continue.
4. Set PagerDuty severity: Under What actions should be applied, select Alert Data and set the alert severity to the value matching the Monitoring value, then save.
5. Repeat and chain: Create one rule for each Monitoring severity value to convert, and optionally add an Else rule to handle remaining values such as no severity.

A custom webhook notification channel in Cloud Monitoring (opens in a new tab) is a delivery target that posts alert data as an HTTP request to any public endpoint you control. An alerting policy or log-based alert evaluates a condition, opens a Monitoring incident, and then Cloud Monitoring sends a notification through the channel to a third-party incident platform such as PagerDuty, ServiceNow, DataDog, OpsGenie, or Rootly. Use webhooks when no built-in integration exists for the destination and you need automated alert response outside Google Cloud.

3.1Alert Flow From Policy to Webhook Endpoint

The webhook channel sits at the end of the alerting flow in Cloud Monitoring. Data flows from the monitored source to the alerting policy, from the policy to an incident, and from the incident to each configured notification channel, including the webhook. The webhook then posts the payload from Cloud Monitoring to the external public endpoint, where the third-party tool creates or updates its own ticket, page, or event.

Each notification carries context for troubleshooting and links back to Google Cloud. A notification includes relevant failure details and a link to its incident, and an incident is a persistent record that stores status, links to logs, a chart of recorded metric data, labels, and duration. For Personalized Service Health log alerts, each notification is also associated with a Monitoring incident and includes a link to the incident page and a direct link to the Service Health dashboard. Cloud Monitoring creates and manages alerting policies through the Google Cloud console, the Cloud Monitoring API, the Google Cloud CLI, or Terraform, but alert delivery depends on the notification channel existing before it is selected in a policy.

3.2Choosing a Delivery Path to Third-Party Tools

Log alerts and metric alerts in Cloud Monitoring use notification channels to reach people or on-call teams when a policy triggers. The supported channels form a set of peer delivery options:

• Email
• Pub/Sub
• PagerDuty
• Slack
• SMS
• Webhooks

The choice of channel defines a trust and network boundary. Webhooks send to public endpoints over the internet, while Pub/Sub sends to private endpoints that do not need public exposure. Native PagerDuty and Slack channels use their built-in Cloud Monitoring integrations, with email sent from alerting-noreply@google.com and Slack messages posted by the Cloud Monitoring application.

3.3Configuring Policies That Use Webhook Channels

Configuration of an alert that notifies a webhook requires a channel first and a policy second. The principal creating the policy must hold the Monitoring Editor (roles/monitoring.editor) IAM role on the Google Cloud project. Only after the webhook channel exists can it be selected during policy creation, and channel creation itself depends on Identity and Access Management (IAM) permission to manage monitoring configuration.

A metric-based alerting policy for Application Integration (opens in a new tab) or Integration Connectors (opens in a new tab) follows an ordered workflow:

1. Select resource and metric: Go to Monitoring > Alerting, click + Create Policy, click Select a metric, and filter by Integration Version or Connectors, then select the metric type and Active Metrics and click Apply.
2. Configure alert trigger: Click Next and set the threshold value that opens an incident when exceeded or breached.
3. Attach notification channels: Click Next, open Notification channels, select one or more channels including the webhook, and click OK.
4. Set closure behavior: Check Notify on incident closure only if closure notices are needed, since by default notifications are sent only when incidents are opened.
5. Add documentation and name: Use the Documentation field for runbook text to include in the notification, enter the Alert policy name, and click Create Policy.

A log-based alert follows a parallel path through Cloud Logging. The policy defines an inclusion filter written in the Logging query language that determines which ingested logs fire the alert, such as maintenance events, failed integration executions, or severity-matched messages. Data flows from Logs Explorer or the Monitoring API into the alerting policy, and the policy can extract labels from log fields for third-party routing, set minimum Time between notifications, and set Incident autoclose duration before the notification channels are attached. Alerts are subject to Cloud Monitoring limits, for example 20 alerts per policy per day per project, and essential contacts configured on the project also receive alert copies.

3.4Securing and Controlling Webhook Notifications

Securing a webhook notification channel in Cloud Monitoring means controlling who can create the channel, what data leaves the project, and which endpoint can receive it. Channel management depends on IAM, so only identities with Monitoring Editor or equivalent permission can create, list, or attach webhook channels through the console, API, CLI, or Terraform. This boundary keeps control of external delivery on the Google Cloud side, while authentication and verification of the incoming POST remain the responsibility of the public endpoint owner.

Control over payload content comes from filters, documentation, and labels. Logging filters decide which logs are sent to third-party tools, so a narrow inclusion filter limits exposure to only matching events such as SCHEDULED maintenance for a specific resource type or location. The Documentation field and extracted log labels further define what context travels with the webhook, which lets the receiver route and verify the event without receiving unrelated project data. For planned maintenance, this means separate policies can fire when a new event is scheduled, when state changes from SCHEDULED to STARTED, when start or end time changes, or when a specific location is impacted.

Key Facts

• Cloud Monitoring alert data flows in order from the monitored source to the alerting policy to the incident to the notification channel to the external endpoint.
• A webhook notification channel posts to any public HTTPS endpoint, while a Pub/Sub notification channel delivers to a private endpoint without public exposure.
• A PagerDuty notification channel requires an Events API v1 integration key pasted into the Service Key field before alerting policies can select PagerDuty.
• PagerDuty shows all Cloud Monitoring severities as critical unless orchestration rules match on event.custom_details.incident.severity to restore the original severity.
• Cloud Monitoring limits log-based alerts to 20 alerts per policy per day per project.