Prepare and test your skills
Prepare and test your skills
Troubleshoot Data Pipeline and Input Failures is the practice of finding why training cannot ingest, parse, or use input data. Data flows from source systems through Dataflow (opens in a new tab) or SQL processing into Cloud Storage (opens in a new tab) and BigQuery (opens in a new tab), then into training. A failure at any boundary can stall training, produce empty batches, or silently bias the model, so diagnosis must combine job logs, throughput metrics, file checks, schema validation, and lineage to separate code errors from data errors and permission errors.
Automated data validation at ingestion and preprocessing checks data types, formats, ranges, and distributions before bad records reach training. TensorFlow Data Validation (TFDV) generates a schema from training data and validates each new batch against that schema, while distribution tracking over time exposes drift that can trigger a retraining pipeline. Validation results and metrics flow into BigQuery for historical analysis, and validation artifacts are archived in Cloud Storage so a failed run can be compared to the last known-good schema and statistics.
Three Google Cloud options divide validation work by data shape and scale:
| Option | Best for | Failure signal it explains |
|---|---|---|
| TensorFlow Data Validation (TFDV) | Feature schema, distribution, and anomaly checks | Schema mismatch, new categorical values, drifted numeric distributions |
| BigQuery | Large-scale SQL data-quality scans | Missing values, out-of-range values, duplicate or invalid rows |
| Dataflow | Streaming and batch programmatic validation | Malformed records, transformation errors, late or dropped events |
Exam tip: Use BigQuery when the check can be expressed as SQL over tables, and use Dataflow when the check needs per-record code over streaming or batch pipelines; use TFDV when the question is whether serving data still matches the training schema.
File and storage validation determines whether training inputs are complete, uncorrupted, and readable by the training service account. Cloud Storage data validation and change detection uses sizes and checksums to detect corruption or incomplete transfers, while format libraries validate that CSV, Parquet, TFRecord, Avro, and image payloads parse as expected. Validation can run inside a Dataflow pipeline for inline filtering or in Cloud Run functions triggered by file-upload events, with pass or fail results written to BigQuery for retrieval during incident review.
Many input stalls are throughput or access failures rather than bad bytes. Data flows from Cloud Storage buckets to training workers, so the training service account depends on the bucket for read permission and the network path for sustained throughput. Store unstructured data such as images, video, and audio as large sharded containers, with each shard at least 100 MB and with 100 to 10,000 shards per dataset, and group shards with buckets and directories to improve read and write throughput. When jobs report permission denied, missing objects, or sudden slow reads, check dedicated least-privilege service accounts, bucket-level Identity and Access Management (IAM) policies, and Cloud Audit Logs streamed to Cloud Logging for denied API calls, unexpected configuration changes, or revoked access.
Pipeline health monitoring tracks whether ingestion and transformation are keeping up with training demand. Dataflow exposes ingestion rates, processed volumes, and transformation latencies, while Agent Platform Pipelines exposes job success and failure rates with built-in logging and tracing integrated with Cloud Monitoring alerts. Detailed application and infrastructure context flows from Cloud Logging and Cloud Trace into dashboards, so an engineer can link a training data-starvation error to a specific Dataflow stage backlog, worker error, or upstream source delay.
Continuous data-quality assessment in Knowledge Catalog (opens in a new tab) separates transient pipeline errors from systematic data decay. Knowledge Catalog centralizes data-quality checks, data profiles, lineage, and discovery across BigQuery, Cloud Storage, and Pub/Sub sources, and BigQuery adds column-level lineage for traceability from raw source to processed feature.
When Cloud Monitoring fires on one of these dimensions, lineage shows which pipeline run and source table produced the bad partition, which shortens root-cause analysis from dataset-wide search to a single job and input.
Schema mismatch, corruption, and training-serving skew are three distinct input failures with different fixes. A schema mismatch occurs when pipeline configurations, arguments, or feature definitions change without updating downstream code, so strict validation of YAML or JSON configs with jsonschema, argument validation with argparse, and component input-type validation in Agent Platform Pipelines prevents runs from starting with wrong types or missing parameters. Store validated, versioned configuration files in Cloud Storage or Artifact Registry so every training run can be reproduced and audited against the exact config that produced its data.
Corruption and skew occur after configs pass but data contents diverge. Corrupted files fail size, checksum, or format checks and must be quarantined, re-exported from the source, or restored from a redundant multi-region or dual-region bucket before retraining. Training-serving skew occurs when serving features drift from training statistics even though files are intact, which appears as falling accuracy, shifted TFDV distributions, or Model Monitoring alerts on prediction inputs. The corrective path is to confirm drift in BigQuery history and Knowledge Catalog profiles, use BigQuery anomaly detection to isolate suspicious partitions, fix the transformation so training and serving share the same Dataflow or Feature Store logic, and then trigger a retraining pipeline on corrected, versioned data.
Analyze and Remediate Model-Specific Training Failures is the process of confirming whether a conversational runtime is emitting expected signals, isolating where the signal path breaks, and escalating with configuration evidence. For a Dialogflow (opens in a new tab) integration with Customer Experience Insights, remediation follows an ordered diagnostic flow from metrics to logs to escalation. The flow depends on notifications moving from the Contact Center AI Runtime source to the Customer Experience Insights Environment destination when conversations transpire.
Verifying runtime notifications uses Metrics Explorer to view metrics specific to the Dialogflow to Customer Experience Insights integration. Select Resource Type Customer Experience Insights Environment and Metric Notifications From Contact Center AI Runtime. This metric shows whether Dialogflow is actually notifying Customer Experience Insights when conversations transpire. If the metric shows no notifications during active conversations, the failure is in the notification path from the runtime to the insights environment, not in downstream analysis.
Isolating the failure point uses Logs Explorer (opens in a new tab) to find where something went wrong after metrics confirm missing or abnormal notifications. Logs Explorer holds detailed execution records from Dialogflow and the integration layer, so the log entries show which component stopped or rejected the notification. Use the logs only after checking the notification metric, because the metric first establishes whether any notification left the runtime. This order prevents investigating the wrong layer when the source never sent data.
Escalating unresolved failures means reaching out to a Google representative for support requests and feedback when metrics and logs do not resolve the issue. A bug report must include the results of three configuration queries that describe the security, agent, and conversation routing state:
securitySettings resource for the project and location, which controls access and data handling for the agent.agents/AGENT-ID resource for the project and location, which identifies the specific agent configuration.conversationProfiles resource for the project and location, which controls how conversations connect to integrations.Each query sends an authenticated request with Content-Type: application/json, Authorization: Bearer $(gcloud auth application-default print-access-token), and x-goog-user-project: PROJECT to LOCATION-dialogflow.googleapis.com. Together the three outputs let support compare the intended security settings, agent definition, and conversation profile against the missing notification behavior observed in metrics and logs.
Diagnose Resource Allocation and System-Level Failures is the practice of using logs and metrics to prove whether a training job failed because compute, memory, disk, or quota was insufficient or misconfigured. In Managed Service for Apache Spark (opens in a new tab), training data flows from storage through executors on worker nodes under YARN, while shuffle data and metadata flow to attached disks, so pressure at any stage can kill executors, containers, or the driver. Cloud Logging carries the failure signatures and Cloud Monitoring carries the resource signals, and the two must be interpreted together to choose between adding memory, enlarging disk, repartitioning, or changing scheduling.
Out-of-Memory (OOM) failures in Managed Service for Apache Spark appear in driver, executor, and YARN logs with distinct signatures that point to heap, overhead, or container limits. Common signatures are Lost executor, java.lang.OutOfMemoryError: GC overhead limit exceeded, and Container killed by YARN for exceeding memory limits, where the first signals a lost executor process, the second signals garbage collection consuming excessive time without reclaiming heap, and the third signals YARN enforcing its container memory limit. Shuffle fetch failures often follow OOM because shuffle data flows from upstream executors to downstream tasks, so a crashed executor leaves no server to answer fetches and downstream tasks report FetchFailedException, Failed to connect to..., or Failed to fetch.
Fixes for OOM change where memory is granted or how much data each task holds at once:
spark.executor.memoryOverhead and lower spark.executor.memory for PySpark, because the Python process consumes off-heap overhead outside the JVM heap.Exam tip: For PySpark OOM, raise spark.executor.memoryOverhead while lowering spark.executor.memory — do not raise both, because total container memory is bounded by YARN.
Disk pressure in Managed Service for Apache Spark causes slow shuffles and unhealthy schedulers because persistent disk throughput scales with disk size and shuffle writes depend on worker disks. Standard persistent disks should be at least 1 terabyte per worker, and disk size can be set significantly larger than the data assigned to each worker in order to raise the throughput cap even when capacity is not needed. YARN reports nodes as UNHEALTHY when local directories for shuffle exceed utilization, typically with 1/1 local-dirs usable space is below configured utilization percentage and used space above threshold of 90.0% in YARN logs, which confirms that insufficient disk space for shuffle data removed the node from scheduling.
Worker disk throughput flows from each worker virtual machine to Cloud Monitoring, and the console exposes it through an ordered check:
When throughput is capped or local disks fill during shuffle, the remedy depends on persistence needs and partition size. Local SSDs are physically attached to the cluster and provide higher throughput than persistent disks, but local SSDs do not persist data after a cluster is shut down and come at a fixed size of 375 gigabytes per device, with multiple devices added to increase performance. SSD persistent disks provide higher throughput for their size than standard persistent disks and persist after shutdown, and SSD persistent disks are also preferred when partition size will be smaller than 8 KB, although small partitions should generally be avoided.
| Disk option | Persistence after shutdown | Throughput and sizing fact |
|---|---|---|
| Standard persistent disk | Persists | Throughput scales with size; use at least 1 TB per worker |
| SSD persistent disk | Persists | Higher throughput for its size than standard disk; preferred for sub-8 KB partitions |
| Local SSD | Does not persist | Higher throughput than persistent disks; fixed 375 GB per device, add devices to scale |
Memory Protection terminations in Managed Service for Apache Spark occur when the system kills a process under memory pressure to avoid a full node hang. When a master node or driver node pool job terminates due to memory pressure, the job fails with Driver received SIGTERM/SIGKILL signal and exited and logs a message that a process is killed due to memory pressure. The driver signal flows from the protection mechanism to the driver process, so filtering Cloud Logging with resource.type="cloud_dataproc_cluster" plus the cluster name, cluster UUID, and jsonPayload.message:"Driver received SIGTERM/SIGKILL signal and exited with" isolates the event, and the google.dataproc.oom-killer log or the dataproc.googleapis.com/node/problem_count metric confirms that Memory Protection terminated the job.
Worker node YARN container terminations follow the same confirmation pattern but name the container and exit code on the affected node. Managed Service for Apache Spark writes container id exited with code EXIT_CODE in the YARN resource manager, and a Cloud Logging filter combining resource.type="cloud_dataproc_cluster", cluster name, cluster UUID, and jsonPayload.message:"container" AND "exited with code" with memory-pressure context isolates candidates. If a container exited with a memory-related code, the google.dataproc.oom-killer log or dataproc.googleapis.com/node/problem_count confirms Memory Protection action, which distinguishes a protection kill from preemption, autoscaler removal, or shuffle-service overload that can produce similar fetch errors.
Scaling and tuning in Managed Service for Apache Spark changes scheduling limits, node size, or parallelism so that the same training logic fits within enforced memory, disk, and quota boundaries. For driver-side kills, the cluster depends on driver memory and concurrency: if the cluster has a driver pool, increase driver-required-memory-mb to actual job memory usage; if the cluster has no driver pool, recreate the cluster while lowering the maximum number of concurrent jobs; or use a master node machine type with increased memory. For worker container kills, the containers depend on YARN scheduling and data distribution: verify that container sizes are configured correctly, consider lowering yarn.nodemanager.resource.memory-mb which controls the amount of memory used for scheduling YARN containers, and if specific containers fail repeatedly, repartition the job or increase worker size because data skew concentrates memory in a few containers.
Preemptible and autoscaled workers add a scheduling dependency that mimics memory failure because shuffle data flows from secondary workers to downstream tasks only while those workers remain present. When preemptible worker virtual machines are reclaimed or non-preemptible workers are removed by the autoscaler while still serving shuffle data, downstream fetches fail even though memory settings are correct, so Enhanced Flexibility Mode makes secondary workers safely preemptible or scalable. When the Spark shuffle service itself is overloaded, decreasing the number of job partitions reduces concurrent fetch load on the service. Master nodes use the earlyoom utility to free memory when available memory is critically low, and thrashing occurs when rapid consumption forces the system to spend most time on memory management before earlyoom or the kernel OOM response can act; advanced tuning edits /etc/default/earlyoom on the master over SSH and raises the -M <kbytes> minimum free-memory threshold from its default of -M 65536 (64 MiB), with the tradeoff that too high a value terminates processes more aggressively even when sufficient memory might remain. Spark 3 supports GPUs through the GPU driver initialization action and the RAPIDS initialization action with the RAPIDS SQL Accelerator, so attaching GPUs is the scaling choice when shuffle, memory, and disk are healthy but compute-bound stages dominate training time.
Use TensorFlow Data Validation (TFDV) when you need to check feature schema, distribution, and anomalies such as schema mismatch, new categorical values, or drifted numeric distributions and whether serving data still matches the training schema. Use BigQuery for large-scale SQL data-quality scans such as missing values, out-of-range values, and duplicate or invalid rows, and use Dataflow for streaming and batch programmatic validation such as malformed records, transformation errors, and late or dropped events.
For PySpark out-of-memory failures in Managed Service for Apache Spark, raise spark.executor.memoryOverhead and lower spark.executor.memory because the Python process consumes off-heap overhead outside the JVM heap and total container memory is bounded by YARN. If executors consistently exceed available RAM even after tuning, move to high-memory machine types and use smaller partitions to reduce the bytes held in memory by any single task and to spread skew.
Diagnose the failure in order by first verifying notification activity in Metrics Explorer using Resource Type Customer Experience Insights Environment and Metric Notifications From Contact Center AI Runtime to confirm whether Dialogflow notified Customer Experience Insights when conversations transpired. If notifications are missing or abnormal, use Logs Explorer to find which component stopped or rejected the notification, then escalate to a Google representative with the securitySettings, agents/AGENT-ID, and conversationProfiles configuration query outputs for the project and location.
Last updated