Professional Cloud Security Engineer

Stream and Ingest Log Telemetry into External SIEM Platforms

Configuring Log Export Pipelines

Exporting log telemetry to external security information and event management (SIEM) systems allows organizations to centralize security operations and threat detection across their infrastructure. A pipeline moves log streams from source platforms—including Google Cloud, Google Distributed Cloud (GDC), and external clouds—to centralized SIEM tools such as Google Security Operations or third-party platforms. In standard cloud environments, telemetry routes either through direct ingestion for real-time analysis or via a batch export pattern that stages data in intermediate storage. For air-gapped environments, log forwarding relies on deploying a SIEMOrgForwarder custom resource within the cluster to route event payloads securely.

Establishing a functioning log pipeline requires meeting strict operational prerequisites. Administrators must establish secure network connectivity between the log source and the receiving SIEM platform. The pipeline requires dedicated identity permissions to export logs and access target endpoints without sharing overly broad administrative access. Authentication details, such as API tokens or ingest keys, are stored inside Kubernetes Secret objects that the forwarder references directly. Pipeline connectors also maintain retry limits to handle temporary destination outages, after which undelivered logs may require administrative intervention.

Integrating with Third-Party and Multi-Cloud Sources

Enterprise security architectures frequently ingest telemetry from multiple cloud providers and endpoint detection systems alongside native Google Cloud data. Configuring multi-cloud ingestion typically involves directing external environments to deposit log streams into a central cloud storage staging area. For example, logs generated by AWS services like CloudTrail or VPC Flow Logs write to an Amazon S3 bucket, where an external SIEM pull feed retrieves them. Similarly, Microsoft Azure services route logs to an Azure Storage Blob container for consumption by security tools.

Endpoint detection and response (EDR) platforms also stream endpoint activity into the central telemetry pipeline. Tools such as SentinelOne and CrowdStrike utilize specialized export connectors to push event streams into designated Cloud Storage or S3 storage buckets. Staging telemetry in cloud buckets establishes a resilient buffer between the generating system and the SIEM intake. Once the intermediate bucket receives the data, ingestion connectors ingest the raw logs on an automated, continuous schedule.

AWS / Azure / EDR Logs  -->  Cloud Storage Staging  -->  SIEM Ingestion Feed  -->  Normalized Data

Managing Feeds and Data Processing in the SIEM

Once log data lands in intermediate storage or reaches an active ingestion endpoint, the SIEM manages ingestion through configured data feeds. A feed configuration defines the source storage URI, the target data format, and the authentication credentials needed to pull objects. Administrators often deploy intermediate processing pipelines, such as the Bindplane Agent, to filter out low-value events or mask sensitive fields prior to ingestion. Transforming telemetry before it reaches the core SIEM optimizes ingestion costs and maintains regulatory compliance.

After ingestion, the security platform parses incoming records into a standardized schema such as the Unified Data Model (UDM). Normalizing diverse multi-cloud and application logs into a single schema allows security analysts to run unified detection rules across different cloud providers. Engineers can continuously monitor feed health dashboards to detect data pipeline failures, parsing errors, or expired authentication tokens.

Configure Aggregated Log Sinks and Filter Architectures

Aggregated Log Sinks Across Resource Hierarchies

Aggregated log sinks centralize security telemetry across complex resource hierarchies by collecting events from organizations, folders, or individual projects. Google Cloud Logging captures critical telemetry, including Cloud Audit Logs, VPC Flow Logs, Cloud DNS logs, and findings from tools like Cloud Intrusion Detection System (Cloud IDS). Direct real-time streaming routes these logs straight to analytics engines without added hops, offering immediate threat visibility. Alternatively, routing logs to a Cloud Storage bucket buffers large spikes in volume and supports long-term compliance retention.

Hierarchical aggregation ensures that security teams retain complete visibility without requiring individual project owners to configure their own export rules. Sinks configured at the organization or folder level automatically inherit logs from all child projects, preventing visibility gaps when new resources launch. The sink identifies matched log entries and forwards them to designated destinations like Cloud Pub/Sub, Cloud Storage, or external SIEM endpoints. Choosing the destination depends on whether the analytics platform requires real-time streaming or scheduled batch imports.

Filter Expressions and Exclusion Rules

Managing telemetry volume requires precise filtering to keep security analysis effective and prevent storage bloat. Engineers construct advanced filter expressions using the logging query language to target high-fidelity security events such as administrative privilege changes, firewall drops, and abnormal network flows. At the same time, exclusion rules drop noisy, low-value telemetry like repetitive health-check queries before logs leave the logging router. This targeted filtering strategy reduces data egress and processing costs while maintaining high security visibility.

Cloud Logging  -->  Log Sink Filter  -->  Inclusion: High-Fidelity Logs  -->  Destination (SIEM / Pub/Sub)
                                     -->  Exclusion: Noisy Logs (Dropped)

Selecting the proper log types directly impacts incident response capabilities and operational spend:

• Cloud Audit Logs provide administrative and data access records that confirm who performed specific actions within the cloud environment.
• VPC Flow Logs capture network traffic metadata, which assists in tracing lateral movement, command-and-control communication, and network anomalies.
• Cloud DNS and Firewall Logs reveal name resolution lookups and blocked connection attempts at the network perimeter.

Secure Export Pipelines with Identity, Encryption, and Perimeter Controls

Identity and Access Management for Log Export

Securing the log pipeline relies on strict Identity and Access Management (IAM) controls based on the principle of least privilege. When a log sink is created, Cloud Logging generates a dedicated writer identity service account specific to that sink. Administrators must grant this writer identity only the minimum permissions required at the destination, such as the Storage Object Creator role on a target Cloud Storage bucket or the Pub/Sub Publisher role on a topic. Restricting writer permissions limits the blast radius if an account credential is ever compromised.

Separation of duties should also be enforced across different log categories. For example, organizations often use distinct service accounts and authentication tokens for audit logs compared to regular application or operational logs. This boundary ensures that operational staff cannot modify or disrupt security audit trails. Administrative access to alter sink configurations should be restricted to authorized security administrators through dedicated IAM roles.

Encryption with Customer-Managed Encryption Keys

While Google Cloud encrypts data at rest by default, compliance mandates often require direct control over encryption keys. Administrators can protect exported log data using Customer-Managed Encryption Keys (CMEK) hosted in Cloud Key Management Service (Cloud KMS). When a sink routes telemetry to a Cloud Storage bucket or Pub/Sub topic, the target resource is configured to wrap the stored data using the designated KMS key.

Log Writer Identity  -->  Destination (Cloud Storage / Pub/Sub)  <--  Cloud KMS (CMEK Key)

Managing your own encryption keys introduces specific operational considerations:

• Key access policies must grant the destination service agent permission to encrypt and decrypt using the CMEK key.
• Revoking or disabling the key immediately cuts off access to the underlying log files, preventing unauthorized data inspection.
• Key rotation policies can be scheduled automatically in Cloud KMS to comply with external regulatory frameworks without interrupting active log exports.

Protecting Egress Paths with VPC Service Controls

To prevent data exfiltration, organizations place logging infrastructure inside a VPC Service Controls perimeter. A security perimeter blocks unauthorized communication between managed cloud resources and external networks, ensuring that telemetry only flows along approved network paths. When telemetry must cross a perimeter boundary to reach an external SIEM, administrators configure explicit egress and ingress rules. These perimeter rules restrict data transfers strictly to authorized identity accounts, target projects, and specific API methods.

In high-security and air-gapped environments, perimeter controls are paired with strict forwarding rules. Forwarder resources, such as the SIEMOrgForwarder, must reside in the same namespace as the authentication secrets they reference to maintain isolation. Network administrators also apply strict egress firewall policies to limit outgoing SIEM traffic to approved IP ranges and HTTPS endpoints. Combining identity isolation, CMEK encryption, and perimeter boundaries ensures that sensitive log data remains protected throughout its entire journey.