Prepare and test your skills

Prepare and test your skills

Secure log export is the set of identity, network, encryption, and governance controls that decide who can move logs, where logs can go, and how logs stay protected. In Google Cloud, Identity and Access Management (IAM) (opens in a new tab) controls who can configure exports, BigQuery (opens in a new tab) and Cloud Storage (opens in a new tab) act as governed destinations, and network and encryption controls protect data in transit and at rest. Governance policies add boundaries for regulated data, including requirements to keep sensitive data on-premises or in approved regions.
Identity and Access Management for log exports is the IAM permission model that controls who can create, view, and stop export jobs. IAM enforces permissions at the Google Cloud level, so users receive only the access needed to configure sinks, create export jobs, and read destination datasets. An export job inherits the data visibility of the user who starts it, which prevents a user from exporting logs they are not allowed to read. Users with global access see all data allowed by their roles and are not limited by data role-based access scope restrictions.
Dedicated export permissions control the export lifecycle in projects linked to the logging tenant:
chronicle.dataExports.fetchServiceAccountForDataExport to obtain the service account used for the exportchronicle.dataExports.create to create a new export jobchronicle.dataExports.cancel to stop a queued or running jobchronicle.dataExports.get to view job status and detailsThe workflow depends on two identities working together: the user identity authorizes the request, and the service account identity carries data from the logging system to the user-owned destination bucket.
Fine-grained access control in BigQuery destinations is the layered model that restricts what different readers can see inside the same exported log tables. IAM roles control access to the project, dataset, and tables, while additional controls limit visibility to approved rows or columns. This separation lets operations teams manage exports without giving every analyst full access to raw logs. Views and user-defined functions that read restricted data must also be authorized, so protection travels with the logic.
When exports contain sensitive or mixed-sensitivity fields, BigQuery uses authorized views and policy tags to expose only approved rows or columns to specific users. Access rules from a source system must be mapped to equivalent BigQuery IAM roles and permissions rather than copied directly. This mapping is needed because BigQuery enforces its own resource model for datasets, tables, views, and routines.
Network controls and regional boundaries are the path and location rules that govern how log data moves from Google Cloud to BigQuery, storage, or on-premises systems. Virtual Private Cloud (VPC) (opens in a new tab) and private connections, including Private Service Connect, keep export traffic on private network paths instead of the public internet. These private paths are especially important for hybrid exports where logs flow from Google Cloud to on-premises collectors or third-party analytics platforms. Regional boundaries then control where log data can be stored and queried.
Logging buckets can be regionalized so _Default buckets stay in approved regions, and the default sink can be disabled to stop automatic routing outside those regions. Live tailing does not store log data, but it can query and transmit data across regions, so tailing filters must also respect regional limits. Together, private networking controls the route while regionalization controls the destination location.
Encryption and protection of exported data is the combination of default encryption and customer-controlled options that protect logs at rest and in transit. Cloud Logging and BigQuery encrypt data by default without extra configuration. The export pipeline remains integrated with Cloud Key Management Service (KMS) (opens in a new tab), so data written to a destination bucket stays encrypted at rest and can use customer-controlled keys when configured. Default encryption provides baseline protection, while customer-managed keys add control over key lifecycle and use.
For stronger protection, BigQuery supports customer-managed encryption keys in Cloud Key Management Service and column-level encryption for selected sensitive fields. A key management strategy should be set before migration or large-scale export so the correct keys exist in the destination project. When sensitive values must stay useful for analysis, data masking or tokenization can hide direct identifiers while preserving analytical patterns.
Governance policies that shape data movement are the organizational and compliance rules that decide what log data may leave Google Cloud and what must stay on-premises. A common pattern separates sensitive application data from lower-risk operations data, keeping customer data on-premises while streaming operations telemetry to the cloud. This split helps meet regulatory requirements that limit storage of customer data in public cloud environments.
In controlled environments such as Assured Workloads folders, extra rules restrict how logging features handle customer data. Log sink filters and live tailing filters are stored as configuration, so filters must not contain customer data. SQL-based alerting policies are disabled in those environments, query jobs should be created only inside Assured Workloads folders, and queries that read or join Assured Workloads data should run inside those folders with fully qualified table names.
Exam tip: An export job inherits the visibility of the user who starts it, so a user cannot export logs they are not already allowed to read.
Log sinks and export mechanisms are the routing rules that send copies of incoming log entries from Cloud Logging to storage and analysis destinations. Each sink lives in the Log Router and pairs a filter with a destination, so data flows from the Logging ingestion pipeline to the sink and then to the destination. Common destinations include BigQuery for analysis, Cloud Storage for archive, Logging buckets for retention, and external systems such as on-premises platforms, Elastic, and Splunk.
How log sinks route Cloud Logging data is the ingestion-time process where each sink tests every incoming entry against its filter and forwards matches to its destination. Cloud Logging (opens in a new tab) receives log entries, the Log Router (opens in a new tab) evaluates the sinks in a Google Cloud project, and matching entries flow from the sink to the configured bucket or external destination. The project includes a preconfigured _Default sink that stores most logs in the _Default Logging bucket. In a centralized design, sinks in many projects forward entries to Logging buckets in one central project that aggregates logs across an organization.
Cloud Logging ingestion flows into the Log Router, which fans out to parallel sinks that each test inclusion and exclusion filters and forward matches to its own destination: _Required and _Default logging buckets, BigQuery for SQL analysis, Cloud Storage for bulk export, and Pub/Sub exporting to Elastic and Splunk.
Because routing happens during ingestion, sink configuration sets export scope and volume. A broad sink exports large volumes for central analysis, while a narrow sink exports only entries needed for a specific audit or application. A single entry can match more than one sink, so the same entry can be sent to multiple buckets or external destinations at the same time.
Filters and inclusion and exclusion criteria are the sink queries that select which log entries are exported and which are dropped before routing. An inclusion filter selects matching entries for export, while an exclusion filter drops matching entries to reduce exported volume and cost. Checking the sinks on the Log Router page is important when exported scope looks larger than expected, because another sink may still include entries that one sink excludes.
Log entries excluded by a sink can still appear in Logs Explorer for three peer reasons:
To narrow results in these cases, refine the search scope to the source project or bucket and verify each sink filter on the Log Router page.
Exporting to BigQuery is the sink path that routes Cloud Logging data from the sink to tables in a BigQuery dataset for SQL analysis and long-term retention. Under normal operation, routed entries land in the table named by the sink, where they can be queried alongside other datasets. If a Cloud Logging alert interrupts routing, Logging creates tables with a backfill_ prefix that hold all entries that were supposed to be routed during the alert window. These backfill tables can contain some entries that were already routed successfully, so the recovery pattern is to merge the backfill data into the original tables and then delete the backfill tables to prevent duplicates.
Exporting to on-premises and external systems is the path that moves Logging data across the cloud boundary to an external collection layer. Fluentd (opens in a new tab) and Fluent Bit act as log collectors that read system logs and other inputs directly from virtual machines and containers, then send them through output plugins to destinations that include BigQuery, Elastic, and Splunk. This path suits hybrid environments with both virtual machines and containers because one agent can feed both Google Cloud and external analytics systems. Exported logs sent only to external systems are not available to Google Cloud Support for troubleshooting, so monitoring is configured separately, often with Prometheus as the on-premises collection layer and Grafana as a single pane of glass.
A related pattern keeps sensitive application data on-premises while operations telemetry flows to Google Cloud. In that split design, operations telemetry flows from on-premises collectors such as the OpenTelemetry Collector to Google Cloud, while sensitive application data remains on-premises behind the compliance boundary. Re-importing archived logs reverses the flow through a Cloud Run job that performs an ordered sequence:
LogEntry structures and aggregates multiple entries into batches to reduce API quota consumption, with retry handling for quota errors.Imported entries remain subject to Logging quotas and a 30-day retention period, so entries with timestamps older than 30 days are not stored, and past-timestamp entries are not included in log-based metrics unless a label records the import time.
Exam tip: An exclusion in the _Default sink does not hide logs stored elsewhere — check whether another sink, bucket view, or centralized project still includes the same entries.
Export destinations and their requirements are the storage, schema, and retention differences between keeping logs in Cloud Logging, analyzing them in BigQuery, archiving them in Cloud Storage, or sending them outside Google Cloud. Cloud Logging supports short-term operations, BigQuery supports long-term SQL analysis, Cloud Storage buckets hold bulk exported files in a partitioned layout, and on-premises or third-party systems such as Prometheus, Grafana, Elastic, and Splunk keep or analyze data outside Google Cloud. Each destination has its own setup steps, access model, and retention behavior.
On-premises destinations with Prometheus and Grafana keep collection near on-premises resources while still viewing Google Cloud data in one place. In this architecture, Prometheus (opens in a new tab) acts as the on-premises collection layer for metrics and Grafana acts as a single pane of glass for both Google Cloud and on-premises resources. The pattern suits hybrid environments with both virtual machines and containers, and it lets teams that already use Prometheus and Grafana continue using them. When compliance requires keeping sensitive customer data on-premises, sensitive application data stays on-premises while lower-risk operations telemetry can stream to the cloud.
This separation matters for planning because collection responsibilities are split:
Telemetry collection and routing in that split model can use the OpenTelemetry Collector or observIQ BindPlane.
Log export to external systems with Fluentd and Fluent Bit uses collectors that forward logs directly to external logging or analytics systems. Data flows from system logs and other inputs on virtual machines or containers to the collector, then from the collector through an output plugin to the destination system. The same collector architecture used for Cloud Logging can also send to systems that support Fluentd or Fluent Bit, including BigQuery, Elastic, and Splunk. Fluentd and Fluent Bit fit hybrid environments because they connect many sources to many destinations without custom code.
Fluentd and Fluent Bit share a many-to-many plugin model:
The collectors handle logs only, so monitoring and support must be planned separately. Fluentd and Fluent Bit support logs only, so monitoring has to be configured separately from logging. Fluentd and Fluent Bit are third-party tools and not official Google products, so Google does not offer support for them. Exported logs are not available to Google Cloud Support for troubleshooting, and Google does not offer support for Google Distributed Cloud clusters without Logging enabled.
BigQuery as an analytical export destination is the managed-table destination for long-term storage and SQL-based analysis of log and analytics data. Once logs land in BigQuery, teams can run analytical queries, build notebooks, and connect visualization tools instead of scanning raw files. BigQuery organizes physical storage with explicit partitioning and clustering, which gives control over how data is distributed and can improve cost and query performance. Exports to BigQuery preserve analytics metrics, dimensions, and custom fields so they can be queried after delivery.
Configuration for an export to BigQuery requires enabling the destination and granting the writer identity permission. The setup must enable the BigQuery (opens in a new tab) API in the destination project and grant the service agent roles such as BigQuery Job User and BigQuery Data Editor. The dataset location must also match the source region rule, for example a US multi-region source requires a us multi-region dataset and an EU source requires an eu multi-region dataset. Security depends on Google Cloud controls: access is managed with Identity and Access Management at the resource level for specific datasets and tables, network protection comes from Virtual Private Cloud (VPC) (opens in a new tab) and private connections, and data is encrypted at rest and in transit by default with options for customer-managed keys and column-level encryption.
Cloud Storage buckets for bulk export and re-import are the file-based destination for large, asynchronous exports that may later be imported back into Cloud Logging. A single export job exports a specific time range of logs to a user-owned bucket and is tracked with a unique dataExportId. The job moves through lifecycle states in order, including IN_QUEUE, then PROCESSING, then FINISHED_SUCCESS. Data remains encrypted at rest in the bucket, including support for customer-owned keys, and the job inherits the data visibility scope of the user who triggers it.
Completed exports use a partitioned directory layout that simplifies access and querying:
gs://GCS_BUCKET_NAME/EXPORT_JOB_NAME/LOGTYPE/EVENT_TIME_BUCKET/EPOCH_EXECUTION_TIME/FILE_SHARD_NAME.csv
The bucket name identifies the destination bucket, the export job name identifies the job, and the log type identifies the category such as NIX_SYSTEM or WINDOWS_DNS. The event-time bucket groups entries by hour in UTC, the epoch execution time records when the job began, and each file shard holds raw logs up to 100 MB. Exports contain raw logs only in uncompressed text, not derived events or detections.
Bulk export has strict limits for stability. Each job can request up to 100 TB of data, so larger datasets must be split into multiple jobs with smaller time ranges. Each tenant can run or queue a maximum of three export jobs at the same time, and any new request above that limit is rejected. A single job can take up to 18 hours depending on volume. A Cloud Run (opens in a new tab) job can import previously exported bucket files back into Cloud Logging for investigation or audit, but Cloud Logging quotas and a 30-day retention period apply to the imported entries.
Exam tip: Imported log entries with timestamps older than 30 days from the import time are not stored in Cloud Logging, and past-timestamp imports are not included in log-based metrics unless a label records the import time.
Comparing destinations by storage, schema, and retention shows why teams choose different targets for analysis, archive, and compliance needs:
| Destination | Storage model | Schema and analysis | Retention and access |
|---|---|---|---|
| BigQuery | Managed tables with partitioning and clustering | Structured metrics, dimensions, and custom fields queried with SQL, notebooks, and visualization tools | Long-term analytical retention controlled by dataset settings and IAM |
| Cloud Storage (opens in a new tab) bucket | Partitioned folders and CSV shards up to 100 MB per file | Raw logs in uncompressed text grouped by job, log type, and event hour | File retention in the bucket; re-import to Cloud Logging is limited to the 30-day retention window |
| On-premises with Prometheus and Grafana | On-premises collection layer with Grafana as single pane of glass | Metrics collection only; logging needs a separate path such as Fluentd | Data stays on-premises for compliance; Google Support is not available for the third-party tools |
| Third-party analytics via Fluentd or Fluent Bit | External systems such as Elastic or Splunk fed by output plugins | Many inputs and outputs connected through plugins without custom code | Retention follows the external system; exported logs are not available to Google Cloud Support |
The table makes the tradeoff explicit: BigQuery provides structured analysis, Cloud Storage provides bulk file retention and re-import, and on-premises or third-party systems keep data outside Google Cloud at the cost of separate logging setup and no Google support for those tools.
Log export jobs are controlled by four dedicated permissions: chronicle.dataExports.fetchServiceAccountForDataExport to obtain the service account, chronicle.dataExports.create to create a job, chronicle.dataExports.cancel to stop a queued or running job, and chronicle.dataExports.get to view job status and details. An export job inherits the data visibility of the user who starts it, so a user cannot export logs they are not allowed to read. The workflow uses two identities together, with the user identity authorizing the request and the service account identity carrying data to the user-owned destination bucket.
Each bulk export job to a Cloud Storage bucket can request up to 100 TB of data, each tenant can run or queue a maximum of three export jobs at the same time, and a single job can take up to 18 hours depending on volume. Completed exports contain raw logs only in uncompressed text with each file shard holding up to 100 MB. When files are imported back into Cloud Logging through a Cloud Run job, Logging quotas and a 30-day retention period apply, so entries with timestamps older than 30 days are not stored.
Log entries excluded by one Cloud Logging sink still appear in Logs Explorer when the same entries remain visible through another project scope, sink, or bucket view. This happens when the query runs in the Google Cloud project that generated the logs rather than in the project or view where the exclusion applies, or when entries were sent to multiple log buckets and the visible copy comes from another sink whose filter still includes them. It also happens when the user has access to views on the destination log bucket, which show the stored entries by default.
An organization is configuring a secure telemetry and log collection architecture in Google Cloud to support auditing and compliance requirements. A dedicated automated service account needs to record operational activity and write log data directly into Cloud Logging while adhering to the principle of least privilege.
Which Identity and Access Management (IAM) permission must be granted to the service account to allow it to write log entries?
datamigration.locations.fetchStaticIps
logging.logEntries.create
artifactregistry.files.download
monitoring.timeSeries.create
Last updated