Analyzing and Correlating Network Telemetry Using Log Analytics and SQL

Enabling SQL Analytics on Network Logs

Log Analytics allows security teams to query network telemetry using standard BigQuery-compatible SQL directly inside Cloud Logging. To use this capability, administrators must upgrade a Cloud Logging log bucket to support Log Analytics features. This configuration lets analysts inspect structured connection records without needing to export raw data to an external database first. Querying log buckets directly accelerates incident response investigations and reduces the operational overhead of managing external data pipelines.

Correlating Telemetry Across Security Sources

Correlating multiple Google Cloud network sources helps identify complex attack patterns across different network layers. Analysts can write SQL queries to join VPC Flow Logs, Cloud Next Generation Firewall (Cloud NGFW) logs, and Cloud Intrusion Detection System (Cloud IDS) threat records. For example, a query can filter Cloud IDS logs for HIGH or CRITICAL severity threat signatures and match those timestamps against firewall rule logs. Analysts can also query DNS telemetry to pinpoint the top queried domains, helping teams isolate command-and-control traffic originating from compromised virtual machines.

Dashboards and SIEM Ingestion

Security teams can turn SQL query results into visual operational dashboards to track real-time security posture and threat trends. For broader enterprise visibility, aggregated security data can be exported to external SIEM platforms such as Google Security Operations or Splunk. Log sinks route security telemetry through Pub/Sub, Cloud Storage, or dedicated datasets to support long-term compliance storage. This pipeline guarantees that real-time alert data reaches both immediate operational dashboards and centralized security teams.

Configuring VPC Flow Logs and Packet Mirroring for Deep Traffic Inspection

Optimizing VPC Flow Log Collection

VPC Flow Logs capture sampled 5-tuple network flows—source IP, destination IP, source port, destination port, and protocol—for VM instances and GKE nodes. Administrators can balance visibility against data storage costs by adjusting the collection parameters:

• The aggregation interval sets how often flow records are consolidated, ranging from 5 seconds up to 15 minutes.
• The secondary sampling rate controls the percentage of total flows exported, ranging from 0.0 (0%) to 1.0 (100%).
• Metadata filters drop unnecessary system labels and annotations to keep log sizes small when only basic IP and port data are required.
• Intranode visibility must be enabled on Google Kubernetes Engine clusters to log pod-to-pod traffic running on the same physical node.

Flow Logging and Firewall Evaluation Order

Understanding where flow logs are generated relative to firewall rules is critical when interpreting connection records. Egress traffic is sampled before outbound firewall rules are processed, meaning blocked outbound attempts are still recorded in the logs. In contrast, ingress traffic is sampled only after inbound firewall rules have allowed the packet through. If an incoming packet is dropped by an ingress firewall rule, no VPC flow log record is created for that connection.

Out-of-Band Inspection with Packet Mirroring

While flow logs summarize connection metadata, Packet Mirroring clones entire raw packets, including headers and payloads, directly from VM virtual interfaces. The mirrored traffic travels out-of-band across an internal passthrough Network Load Balancer to dedicated collector instances or Cloud IDS appliances. Because Packet Mirroring clones traffic directly at the virtual interface, it captures all incoming, outgoing, and lateral intra-subnet communication without consuming VM computing power or adding inline network latency. Teams can analyze these packet captures (PCAP) to reconstruct security incidents, identify zero-day exploits, or inspect unencrypted application payloads.

Implementing and Monitoring Cloud NGFW and Cloud IDS Threat Telemetry

Cloud NGFW Rule Logging and Intrusion Prevention

Cloud NGFW enforces stateful firewall policies across hierarchical folders and individual VPC networks. Enabling rule logging on firewall policies generates structured audit records that track whether a connection was allowed or denied. For deeper packet protection, Cloud NGFW Enterprise uses dedicated firewall endpoints to deliver Layer 7 intrusion detection and prevention (IDPS). Security administrators configure threat-prevention security profiles with specific rules to automatically alert on or deny traffic matching known malware, spyware, and vulnerability exploit signatures.

Managed Threat Detection with Cloud IDS

Cloud IDS provides fully managed network threat detection powered by industry-standard threat inspection engines. The service receives cloned network traffic delivered by Packet Mirroring and inspects east-west and north-south communication for network-based attacks. Cloud IDS identifies evasion techniques, command-and-control callbacks, and lateral movement between workloads in the same subnet. Every detected threat generates a structured alert containing threat severity, source and destination details, and attack identifiers, which are automatically sent to Cloud Logging.

Centralized Security Operations Integration

Enterprise monitoring relies on centralizing network security telemetry into a single analysis workflow. Administrators configure aggregated log sinks at the organization level to automatically route Cloud NGFW logs, Cloud IDS alerts, and flow logs into a central log bucket. Analysts can run Community Security Analytics (CSA) query sets to detect abnormal traffic volumes, verify firewall policy hits, and flag port scans. Security teams can also feed these logs directly into Security Command Center to automatically prioritize findings and trigger automated remediation workflows.