Prepare and test your skills
Prepare and test your skills
Cloud Logging (opens in a new tab) is a fully managed service that stores, searches, analyzes, monitors, and alerts on log data and events. Log data flows from Google Cloud services and Amazon Web Services into Cloud Logging, while data from more than 80 additional applications, on-premises systems, and hybrid cloud systems flows in through Bindplane. The service includes storage in log buckets, a user interface called Logs Explorer (opens in a new tab), and an API to manage logs programmatically.
Cloud Logging supports the full log lifecycle from collection to storage to analysis in one place. Applications and services write log entries through the Ops Agent, the Cloud Logging API, or client libraries, and the entries are stored in log buckets for a retention period and then deleted. The Log Router can forward copies of entries to other destinations, while Logs Explorer and log-based metrics turn stored entries into search, charts, and alerts. This flow lets a data team trace a pipeline failure from symptom back to cause.
Log categories in Cloud Logging describe what kind of information each entry contains, and the categories are not mutually exclusive. Each category points to a different kind of issue detection, from code errors to performance problems to security changes.
Three log types are especially useful for historical troubleshooting of container workloads. Node and runtime logs from kubelet and containerd record Pod lifecycle, container startup, image pulls, Out of Memory events, probe failures, volume mount errors, and node NotReady status. App logs from stdout and stderr record crashes, errors, and unexpected behavior inside containers. Audit logs record API calls to the control plane, such as who deleted a critical Deployment.
Logs Explorer is the Cloud Logging user interface for querying, viewing, and analyzing log entries. A user selects a resource scope, writes a query in the Logging query language in the query pane, runs the query, and then opens an entry to see its full message, JSON payload, metadata, and timestamp. Filtering first by resource, severity, and time narrows millions of entries to the small set related to a failing pipeline step.
Use Logs Explorer to isolate an issue with an ordered workflow:
The following filters are commonly used to isolate relevant data:
| Filter | What it selects | Example value |
|---|---|---|
resource.type | The type of monitored resource | k8s_cluster, k8s_node, k8s_pod, k8s_container |
log_id | The log stream from the resource | stdout, stderr |
resource.labels | The named resource instance | pod_name, namespace_name |
severity | The severity level of the entry | DEFAULT, INFO, WARNING, ERROR, CRITICAL |
jsonPayload.message | Text search inside the message | Regular expression match with =~ |
For example, error logs for one container can be isolated with resource.type="k8s_container" plus the pod name and namespace plus severity=ERROR. If a node shows NotReady, node logs from kubelet and containerd often reveal network problems or resource constraints. If a Pod failed to start and app logs are empty, audit logs and node logs help check for scheduling problems, resource pressure, or image pull errors.
Exam tip: Use severity=ERROR or CRITICAL to find failures, and use audit logs to find who changed configuration when app logs show no code error.
Log storage in Cloud Logging starts automatically for the resource where the data originates. A Google Cloud project, folder, billing account, or organization resource automatically stores the log data that originates in that resource, such as Compute Engine logs stored in the same project. Retention is for a specified length of time, and entries can be configured for which logs are kept, which are discarded, and where they are stored.
Log Router is the component that forwards log entries from their source to a destination. A route can send entries to a destination in a different resource, such as logs from one project stored in a log bucket in another project. Supported destinations are a Google Cloud project, a log bucket, a BigQuery dataset, a Cloud Storage bucket, and a Pub/Sub topic, where Pub/Sub supports third-party integrations like Splunk or Datadog. Stored logs also feed monitoring through log-based metrics, which count error entries, observe trends such as latency, and create charts, while access to log buckets is controlled by Identity and Access Management (IAM) permissions and roles.
Log-based metrics turn specific log entries in Cloud Logging (opens in a new tab) into quantifiable time-series data in Cloud Monitoring (opens in a new tab). Cloud Logging is a fully managed service that stores, searches, analyzes, and monitors logging data through log buckets, the Logs Explorer interface, and an API. Once log data becomes a metric, Cloud Monitoring can chart performance trends over time and trigger alerts when critical pipeline events occur.
Both system-defined and user-defined log-based metrics feed the same workflow, where data flows from matching log entries in Cloud Logging to a metric and then to dashboards and alerting policies in Cloud Monitoring. This translation lets pipeline teams watch text logs as numbers that rise, fall, or spike over time.
Log-based metrics are counters and measurements created from log entries that match a filter you define. The metric extracts data from each matching entry and sends the resulting time-series data from Cloud Logging to Cloud Monitoring for charting and alerting. This translation connects raw log text to measurable signals for pipeline health.
Use log-based metrics when you want to do any of the following:
Cloud Monitoring is the integrated monitoring service that collects and stores performance information for Google Cloud services and applications. Visualization in Cloud Monitoring starts with Google Cloud-managed dashboards, which are created automatically as resources are instantiated and show curated health, log, metric, and open-alert information. When managed dashboards do not show trends or outliers clearly, teams create a custom dashboard to control what metric data is viewed and how it is displayed.
For data pipelines, this means a log-based metric for job failures or task latency appears as a charted line over time on a custom dashboard. The chart makes it easy to see whether error counts are climbing, whether latency is drifting upward, and whether a pipeline stage is behaving differently than before.
An alerting policy in Cloud Monitoring is the rule that watches a metric and sends notifications when the metric meets criteria you define. The policy contains the condition to watch, such as error count exceeding a threshold or latency changing in an unacceptable way, plus the list of people or groups to notify. When the condition is met, Cloud Monitoring sends a notification to each configured channel and stores an alert as a persistent record for troubleshooting.
Cloud Monitoring supports common notification channels, including:
Each notification includes relevant failure information and a link to the alert, and the alert record typically lists the status of the alert, links to logs, a chart of the recorded metric data, labels, and duration. You can create and manage alerting policies by using the Google Cloud console, the Cloud Monitoring API, the Google Cloud CLI, or Terraform.
Exam tip: An alerting policy notifies you when a metric meets defined criteria, while the resulting alert is the persistent record with status, log links, metric chart, labels, and duration.
Cloud Logging (opens in a new tab) is a fully managed service that stores, searches, analyzes, monitors, and alerts on log data and events from Google Cloud and Amazon Web Services. Using Bindplane, Cloud Logging can also collect data from more than 80 additional applications, on-premises systems, and hybrid cloud systems. Logging provides storage through log buckets, a user interface called Logs Explorer, and an API to manage logs programmatically for reading, writing, querying, and routing log entries.
A single data pipeline can produce several log types at the same time, from platform logs from the managed service to security logs from access activity to user-written logs from custom code. Together the categories give a complete picture of service health, user activity, and application behavior for a comprehensive monitoring strategy.
Categories of log data in Cloud Logging describe log information by who wrote it and why, and the categories are not mutually exclusive. Understanding the source of each category shows which team or system owns the data and what kind of question it answers.
| Log category | Source | Purpose |
|---|---|---|
| Platform | Google Cloud services | Debug and troubleshoot issues and understand the managed services in use |
| Component | Google Cloud-provided software running on customer systems, such as GKE components | Provide user support for those components |
| Security | Cloud Audit Logs and Access Transparency logs | Answer who did what, where, and when for auditing and compliance |
| User-written | Custom applications and services sent through the Ops Agent, Cloud Logging API, or client libraries | Record application output, including application and error events |
| Multicloud and Hybrid-cloud | Other cloud providers such as Microsoft Azure and on-premises infrastructure | Bring outside-cloud activity into the same monitoring strategy |
Security logs in Cloud Logging answer who did what, where, and when within Google Cloud resources. Cloud Audit Logs record administrative activities and accesses by customers, such as configuration changes and data access, and they help security, auditing, and compliance teams monitor for vulnerabilities or misuse. Access Transparency logs record actions taken by Google Cloud staff when accessing customer content, which helps organizations track legal and regulatory compliance.
Exam tip: Cloud Audit Logs record customer administrative activity, while Access Transparency logs record Google Cloud staff access to customer content.
User-written logs are logs written by custom applications and services, which is where application logs and application error output live in Cloud Logging. This data is typically written to Cloud Logging by using the Ops Agent, the Cloud Logging API, or the Cloud Logging client libraries. Platform logs from Google Cloud services complement these custom logs by showing errors and behavior inside the managed service itself, while Error Reporting captures specially formatted error logs and displays them in the Error Reporting dashboard.
How log types support monitoring is that different logs in Cloud Logging become measurable signals and alerts in Cloud Monitoring (opens in a new tab) for pipeline reliability. Log data stored in log buckets can be used to monitor the count of a message, such as the number of entries with an error status, and to observe trends such as latency values with notification when values change in an unacceptable way. Numeric data extracted from logs can also be used to create charts, so teams can watch error rates, access patterns, and application performance together.
Log entries are stored automatically in the Google Cloud project, folder, billing account, or organization resource where they originate, such as Compute Engine logs stored in their project, and they are kept for a defined retention period before deletion. Log entries can also be routed to other destinations for longer analysis or integration:
You isolate error logs in Logs Explorer in Cloud Logging by selecting a resource scope, writing a targeted query using resource, log, severity and message filters, running the query, and opening an entry to see its full message, JSON payload, metadata and timestamp. Filtering first by resource, severity and time narrows millions of entries to the small set related to a failing pipeline step, and error logs for one container can be isolated by combining the container resource type with pod name and namespace and severity ERROR.
The difference is that Cloud Audit Logs in Cloud Logging record customer administrative activities and accesses while Access Transparency logs record actions taken by Google Cloud staff when accessing customer content. Cloud Audit Logs cover events such as configuration changes and data access and help monitor for vulnerabilities or misuse for auditing and compliance, while Access Transparency logs help organizations track legal and regulatory compliance.
Log-based metrics in Cloud Logging turn specific matching log entries into quantifiable time-series data in Cloud Monitoring for charting and alerting. An alerting policy in Cloud Monitoring watches the metric and sends notifications to configured channels such as email, Cloud Mobile App, PagerDuty and Slack when the metric meets defined criteria, and stores the resulting alert as a persistent record with status, log links, metric chart, labels and duration.
A data team is operating an automated data processing pipeline deployed on Google Cloud. During a scheduled run, a task fails prematurely because custom transformation code encounters an unexpected schema variation and raises an unhandled exception.
Which type of log in Cloud Logging should the data engineer inspect to examine the stack trace and debugging messages generated by the code?
Last updated