VPC Flow Logs, NGFW and IDS Telemetry

Implementing and monitoring Cloud Next Generation Firewall (Cloud NGFW) (opens in a new tab) and Cloud Intrusion Detection System (Cloud IDS) threat telemetry means combining firewall decision logs with deep threat inspection signals in Cloud Logging. Cloud NGFW records which firewall policy rules allowed or denied traffic and can block threats inline, while Cloud IDS detects threats passively from mirrored traffic. Together they provide audit proof of enforcement, visibility into lateral movement, and alerts for malware, spyware, and command-and-control activity.

1.1Firewall Rules Logging for Allowed and Denied Traffic

Firewall Rules Logging (opens in a new tab) is the per-rule logging control that audits, verifies, and analyzes the effect of firewall rules in a Virtual Private Cloud (VPC) (opens in a new tab) network. Logging is enabled individually for each firewall rule whose connections need to be logged, and it is available for any rule regardless of action (allow or deny) or direction (ingress or egress). The logs record TCP and UDP connections inside the VPC, which makes it possible to verify that a deny rule is functioning as intended, count how many connections a given rule affects, and provide early warning of unapproved network use.

Firewall Rules Logging applies to both VPC firewall rules and firewall policy rules in hierarchical firewall policies and global network firewall policies. Hierarchical policies defined at the organization, folder, and VPC network levels centralize control across projects, including Shared VPC environments, while network firewall policies apply fine-grained controls with tags, service accounts, and network tags. For Cloud NGFW Enterprise, firewall policy rules also control which Layer 3 and Layer 4 traffic is intercepted for Layer 7 inspection. Because logging can generate a large volume of data with associated charges, enable it only on the rules under investigation and disable it on rules that no longer require it.

1.2Cloud NGFW Enterprise Intrusion Detection and Prevention Service

The Cloud NGFW intrusion detection and prevention service is the Cloud NGFW Enterprise capability that continuously monitors workload traffic for malicious activity and takes preemptive action to prevent it. The service creates Google-managed zonal firewall endpoints that use packet intercept technology to transparently inspect workloads for configured threat signatures. Threat prevention is powered by Palo Alto Networks threat prevention technologies and supports the following threat signature categories:

• Anti-spyware: signatures for spyware and command-and-control activity
• Vulnerability protection: signatures for intrusion and exploit attempts
• Antivirus: signatures for malware payloads

The service processes traffic in an ordered sequence where each stage depends on the previous stage:

1. Firewall policy application: Firewall policy rules are applied to traffic to and from virtual machine (VM) instances or Google Kubernetes Engine (GKE) clusters in the network.
2. Interception: Matched traffic is intercepted and packets are sent to the firewall endpoint for Layer 7 inspection.
3. Signature scan: The firewall endpoint scans the packets for configured threat signatures.
4. Enforcement: If a threat is detected, the action configured in the security profile is performed on that packet.

Enforcement depends on two related policy objects and one inspection resource. Security profiles are generic policy structures configured at the organization or project level; for this service the profile type is threat-prevention and it overrides specific threat prevention scenarios and actions. Security profile groups are also configured at the organization or project level and can contain a maximum of one security profile per threat-prevention type per group. Firewall policy rules reference those security profile groups to enable detection and prevention, and the referenced firewall endpoint is a zonal resource that can inspect traffic in the same zone and applies either the default action or the configured profile action when a threat is found.

1.3Cloud IDS Detection from Mirrored Traffic

Cloud IDS is a managed intrusion detection service that provides threat detection for intrusions, malware, spyware, and command-and-control attacks without sitting inline in the traffic path. Cloud IDS works by creating a Google-managed peered network containing VMs that receive mirrored traffic, and that mirrored traffic is then inspected by Palo Alto Networks threat protection technologies. The mirroring happens on the VM instances, not on the network, using packet mirroring built into the Google Cloud networking stack that forwards traffic from Compute Engine VMs or GKE clusters to the Cloud IDS endpoint.

Cloud IDS endpoints are created in the VPC network and monitor ingress and egress traffic to and from that network as well as intra-VPC traffic. Because Cloud IDS has full visibility into intra-subnet traffic, it can monitor VM-to-VM communication and detect lateral movement, covering both north-south and east-west traffic. Detected threat activity includes exploit attempts, port scans, buffer overflows, protocol fragmentation, command-and-control traffic, and malware. This passive model provides visibility where inline prevention is not deployed and complements next-generation firewall (NGFW) network virtual appliances that deliver protection in situations not covered by Cloud NGFW or provide consistency with on-premises NGFW installations.

1.4Routing and Analyzing Threat Telemetry in Cloud Logging

Threat telemetry from Cloud NGFW and Cloud IDS becomes actionable when firewall logs and intrusion alerts flow from the network to Cloud Logging for reporting, alerting, and auditing. Firewall Rules Logging entries show which rule matched, whether the verdict was allow or deny, and how much traffic was affected, while Cloud IDS logs carry intrusion details including threat type and signature match for malware, spyware, and command-and-control communications. IDS logs fall in compliance scope and must be sent to Cloud Logging and Cloud Monitoring, where they can be retained centrally, routed to sinks such as BigQuery or Pub/Sub, and correlated with other signals in Security Command Center Enterprise or Google Security Operations.

Operational monitoring depends on centralized logging and alerting built on those log streams. Teams establish baselines for normal network behavior and continuously analyze traffic flows to identify suspicious patterns such as unusual protocols, unexpected connections, or sudden spikes in data transfer. Alerts are configured so that any intrusion attempt detected by Cloud IDS, whether successful or unsuccessful, triggers notification, and Firewall Rules Logging triggers alerts in response to attempts to violate specific network policies. The choice between the two inspection services shapes response:

Firewall Insights and Network Analyzer extend this analysis by using firewall logs to expose misconfigurations, shadowed rules, and packet-drop signals, helping teams tighten overly permissive rules and confirm that dropped packets reflect intended security protections rather than misconfiguration.

Configuring VPC Flow Logs (opens in a new tab) and Packet Mirroring for deep traffic inspection means choosing between sampled connection telemetry and full-packet capture for security analysis. VPC Flow Logs aggregates sampled packets by IP connection 5-tuple and writes flow records to Cloud Logging for monitoring, forensics, and cost analysis. Packet Mirroring copies all packets from virtual machine (VM) instances to collector instances running third-party tools, so analysts can inspect raw payloads that sampled flow records cannot provide.

2.1What VPC Flow Logs Capture

VPC Flow Logs samples packets sent from and received by VM instances, including instances used as Google Kubernetes Engine (GKE) nodes. VPC Flow Logs also samples packets from Cloud Run or App Engine resources configured with Direct VPC egress and packets sent through VLAN attachments for Cloud Interconnect and Cloud VPN tunnels. Sampled TCP, UDP, ICMP, ESP, GRE, and Remote Direct Memory Access (RDMA) flows are aggregated by connection over a configured interval, and the resulting record estimates total bytes and packets to account for missed packets.

Flow records cover flows inside Google Cloud and flows between Google Cloud and other networks, in both inbound and outbound directions except for RDMA over Converged Ethernet (RoCE), where only outbound flows are sampled. VPC Flow Logs introduces no delay or performance penalty and works with Virtual Private Cloud (VPC) (opens in a new tab) networks, not legacy networks. Flow logs identify packet direction but do not indicate which endpoint initiated the flow.

2.2Controlling Visibility and Cost With Aggregation, Sampling, and Filters

VPC Flow Logs configuration balances network visibility against log volume and storage cost through four peer controls that change how much data flows from the sampled packets to Cloud Logging. Shorter intervals and higher sampling preserve detail for security analysis, while longer intervals and lower sampling reduce charges for long-lived connections and high-volume networks.

• Aggregation interval: sampled packets for one connection over 5 seconds (default), 30 seconds, 1 minute, 5 minutes, 10 minutes, or 15 minutes are combined into a single log entry.
• Secondary sampling rate: flow entries produced by the fixed primary sampling stage are kept at a configurable rate from 1.0 (100%) to 0.0 (0%).
• Metadata annotations: entries are annotated by default with source and destination names inside Google Cloud or geographic region for external endpoints, and annotations can be kept in full, limited to selected fields, or turned off.
• Filtering: by default every sampled flow generates a log, but filters generate logs only for flows that match selected criteria.

Defaults for secondary sampling depend on the management API that created the configuration. Configurations created with the Compute Engine API keep 50% of entries by default, while configurations created with the Network Management API keep 100% by default and cannot be set to 0.0. You cannot change how the primary sampling stage collects packets, so use Packet Mirroring with collector instances when every packet must be analyzed.

2.3Where VPC Flow Logs Are Enabled and Stored

VPC Flow Logs scope determines which resources report traffic and which Google Cloud project pays for and stores the logs. An organization-level configuration enables flow logs for all subnets, VLAN attachments, and Cloud VPN tunnels in all VPC networks in the organization, with each project billed separately. A project-level configuration enables flow logs for a specific VPC network, which includes all of its subnets, VLAN attachments, and Cloud VPN tunnels, or for a specific subnet, VLAN attachment, or Cloud VPN tunnel.

Enabling flow logs for a subnet collects data from all VM instances in that subnet, and final log entries are written to Cloud Logging in the project of the reporting VPC network. For Shared VPC networks, logs are reported in the host project, not the service project. VMs with multiple network interfaces require VPC Flow Logs to be enabled for each subnet that contains a network interface, and flows between Pods on the same GKE node require intranode visibility to be enabled for the cluster.

2.4How Firewall Rules and Platform Limits Shape Interpretation

VPC Flow Logs interaction with firewall rules (opens in a new tab) determines whether denied traffic appears in Cloud Logging, which directly affects forensic conclusions. Egress packets are sampled before egress firewall rules are applied, so outbound packets denied by an egress rule can still be sampled. Ingress packets are sampled after ingress firewall rules are applied, so inbound packets denied by an ingress rule are not sampled.

Several platform boundaries also limit coverage. VPC Flow Logs is not supported for subnets with purpose INTERNAL_HTTPS_LOAD_BALANCER because those proxy-only subnets have no VM instances or serverless endpoints. Attaching multiple overlapping configurations to the same resource generates duplicate logs, with each configuration producing a separate set of flow records.

2.5Packet Mirroring for Full-Payload Forensic Capture

Packet Mirroring copies complete packets, including payloads, from source VM instances to collector destinations for out-of-band security inspection. The mirroring happens on the VM instances, not on the network, so the mirrored stream does not sit inline with production traffic. Traffic flows from the mirrored VM instance to collector instances running third-party software, where analysts can perform forensic reconstruction, malware analysis, and intrusion investigation that sampled flow metadata cannot support.

Cloud Intrusion Detection System (Cloud IDS) shows the same pattern in managed form, because Cloud IDS works by creating a Google-managed peered network containing VMs that receive mirrored traffic. Mirrored traffic sent to that peered network is then inspected by Palo Alto Networks threat protection technologies to detect intrusions, malware, spyware, command-and-control activity, and VM-to-VM lateral movement. Choose this model when inspection must see intra-subnet and east-west traffic content rather than only connection summaries.

2.6Choosing Between Flow Logs and Packet Mirroring

The choice depends on whether connection-level visibility or packet-level evidence is required and how much volume the team can store and analyze:

Use VPC Flow Logs for continuous coverage such as top talkers, traffic between regions and zones, traffic to specific countries, and which IP addresses communicated and when. Use longer 15-minute aggregation for networks with mostly long-lived connections to reduce log count and simplify capacity analysis, use sampling to estimate usage and expense, and remove metadata when only IP addresses and ports are needed for forensics. Use Packet Mirroring when investigation requires every packet, such as HTTP request detail or payload signatures, and accept the higher collection and processing cost that full capture creates.

Analyzing and correlating network telemetry using Log Analytics (opens in a new tab) and SQL is the practice of storing firewall, flow, DNS, and intrusion detection logs in a queryable bucket and using BigQuery-compatible SQL to find threats. Cloud Logging collects the raw logs from across an organization, and an upgraded log bucket with Log Analytics makes those logs queryable as structured columns for investigation. Analysts run SQL to filter by time, address, rule, and severity, join signals from different sources, and export results to dashboards and SIEM systems for hunting and compliance.

3.1Upgrading Log Buckets for Log Analytics

An upgraded log bucket is a Cloud Logging storage bucket that has Log Analytics enabled and exposes its logs through SQL views including _AllLogs. The upgrade preserves routing and retention while adding the ability to run ad hoc investigations in Logs Explorer and to query the same data through a linked BigQuery dataset for reports and views. Data flows from enabled sources such as VPC Flow Logs, Firewall Rules Logging, Cloud DNS logging, and Cloud Intrusion Detection System (Cloud IDS) into the bucket, where fields like timestamp, resource.type, log_id, and json_payload remain structured for filtering and aggregation. Choose the upgraded bucket when centralized, organization-wide network analysis is needed without maintaining a separate warehouse.

The end-to-end security log analytics workflow moves data through three ordered stages:

1. Enable logs: Prioritize and turn on the network and audit logs needed for coverage, since some logs are on by default and others incur ingestion cost.
2. Route logs: Send organization, folder, project, and billing account logs through an aggregated sink to a single destination for analysis.
3. Analyze logs: Query collated logs with SQL in Log Analytics or BigQuery, with YARA-L rules in Google Security Operations, or with the query language of a third-party SIEM.

3.2Routing Network Telemetry to a Single Analytics Destination

Routing to a single destination aggregates distributed network telemetry so firewall decisions, flow records, and IDS alerts can be queried together. A Cloud Logging aggregated sink applies a log filter at the organization level and forwards matching entries from all contained folders, projects, and billing accounts to one destination. The log scoping filter selects relevant log_id values such as compute.googleapis.com/firewall, compute.googleapis.com/nat_flows, dns.googleapis.com/dns_queries, and cloudaudit.googleapis.com/activity, and the filter can be built from the log scoping tool that maps each log to MITRE ATT&CK tactics, CIS Google Cloud Platform violations, and Event Threat Detection rules.

The destination depends on the analytics and retention need:

• Logs routed to a Cloud Logging bucket with Log Analytics enabled actively analyze logs with Logs Explorer and SQL through the linked BigQuery dataset.
• Logs routed to a BigQuery dataset support large-scale analysis optimized for very large log volumes and near real-time export.
• Logs routed to Cloud Storage store logs long-term for compliance, audit, and incident tracking, with optional Bucket Lock to make objects immutable for a defined retention period.
• Logs routed to Pub/Sub export logs to an external SIEM, including third-party Pub/Sub ingestion, streaming to Splunk with Dataflow, or direct ingestion to Google Security Operations.

3.3Querying Structured Network Logs with SQL

SQL analysis in Log Analytics uses BigQuery-compatible syntax against the _AllLogs view, while BigQuery analysis queries exported tables directly. Both approaches filter on timestamp, resource.type, and log_id, aggregate with COUNT(*) and GROUP BY, and sort with ORDER BY to surface anomalous patterns such as spikes, top talkers, or high-severity threats. Community Security Analytics (CSA) provides reusable baseline queries that audit cloud usage and detect threats like malware, phishing, ransomware, and misconfigured assets, and Security Command Center findings can be correlated with these log results for broader context.

The two SQL dialects differ mainly in table reference and field access:

For example, a Cloud IDS hunt for virus activity filters resource.type="ids.googleapis.com/Endpoint" over the last 30 days where alert severity is HIGH or CRITICAL and type is virus, returning alert time, name, details, application, URI or filename, and IP protocol ordered by newest timestamp. A DNS hunt lists the top 10 queried domains over the last 60 days by filtering log_id="dns.googleapis.com/dns_queries", grouping by query name, and ordering by total queries descending. These patterns generalize to flow and firewall analysis by grouping source and destination IPs and ports to estimate inter-region traffic, identify top talkers, and flag unusual protocols or sudden transfer spikes.

3.4Correlating Firewall Rules, Flow Records, and IDS Signals

Correlation links the firewall policy decision to the observed traffic and to the threat verdict for the same connection window. Firewall Rules Logging (opens in a new tab) records TCP and UDP evaluations for a specific rule in either direction and for either allow or deny action, so it shows which rule allowed or blocked traffic. VPC Flow Logs records a sample of network flows sent from and received by VM instances, including GKE nodes, aggregated by connection over an aggregation interval, so it shows who talked to whom, when, and on which port. Cloud IDS inspects mirrored traffic with managed detection for intrusions, malware, spyware, and command-and-control, including VM-to-VM east-west traffic that reveals lateral movement.

Effective correlation joins these sources on time, IP addresses, ports, and protocol to answer whether a deny rule worked as intended or whether allowed traffic later triggered an IDS alert. Firewall Insights strengthens this analysis by using firewall logs to report rule usage, identify shadowed rules overlapped by higher or equal priority rules, and surface packet-drop signals from deny rules to distinguish expected security blocks from misconfiguration. Because flow sampling, metadata reduction, and longer aggregation intervals change volume and detail, long-lived connections often use 15-minute aggregation for simpler monitoring, while forensics removes extra metadata to focus on IP and port relationships and cost control limits logging to in-scope subnets.

3.5Building Dashboards and Exporting Findings for Hunting and Compliance

Operational dashboards turn repeated SQL hunts into continuous monitoring for baselines, anomalies, and compliance evidence. The linked BigQuery dataset behind an upgraded bucket supports saved queries, views, and reports that track normal network behavior, unusual protocols, unexpected connections, and growth for capacity forecasting. Teams combine flow-based traffic analysis, firewall allow and deny counts, DNS query rankings, and IDS severity trends into one view so a spike in outbound flows, a new top DNS domain, and a matching HIGH severity IDS alert appear as a single investigable pattern.

Aggregated findings flow from the analytics destination to enterprise systems for threat hunting and long-term retention. Data flows from the log bucket or BigQuery dataset to Pub/Sub topics for SIEM ingestion, from Cloud Logging through Dataflow to Splunk, or directly into Google Security Operations for YARA-L correlation with asset changes and Security Command Center findings. Data flows from Cloud Logging to Cloud Storage for immutable audit retention when Bucket Lock policies require non-deletable storage from seconds to years. Retention, sink filters, and consistent IAM read access balance security visibility against ingestion and storage cost while preserving evidence for audit and incident response.

Key Facts

• Cloud NGFW Enterprise blocks threats inline through zonal firewall endpoints and threat-prevention security profiles, while Cloud IDS detects threats passively from mirrored VM traffic without sitting in the traffic path.
• Firewall Rules Logging is enabled per rule for allow or deny actions in either direction and records TCP and UDP connections to verify enforcement and detect unapproved use.
• VPC Flow Logs samples packets before egress firewall rules are applied and after ingress rules are applied, so egress denies can appear in logs while ingress denies never appear.
• Packet Mirroring copies all packets with full payloads from VM instances to collector instances for forensic analysis, while VPC Flow Logs stores sampled, aggregated flow records in Cloud Logging.
• An upgraded Cloud Logging bucket with Log Analytics queries the _AllLogs view with JSON_VALUE(json_payload.field), while BigQuery export queries dataset tables with jsonPayload.field.