professional-cloud-data-engineer
This focus area covers how to automatically adjust computing power and provision resources for data processing jobs in Google Cloud Dataflow and Dataproc to meet strict performance deadlines.
Google Cloud Dataflow uses horizontal autoscaling to dynamically change the number of worker instances based on the current workload. It scales compute power by analyzing factors like pipeline parallelism, CPU usage, and the amount of pending data (backlog). This automatic adjustment ensures workloads have enough resources to meet processing deadlines. However, Dataflow enforces strict system limits on the maximum number of instances:
Choosing the right worker setup is vital for complex workloads. Dataflow supports specialized hardware like graphical processing units (GPUs) and tensor processing units (TPUs) to speed up mathematical operations. To use these accelerators, your pipeline must use Runner v2 and a custom container image that includes the necessary GPU drivers. You must also increase the default boot disk size to at least 50 GB to prevent large containers from running out of space.
Unlike serverless platforms, Cloud Dataproc requires you to proactively manage Compute Engine resources and API limits to avoid cluster creation failures. If regional limits are exceeded, Dataproc returns a resource exhausted error (HTTP 429), which causes API requests to fail. To prevent this, administrators should use auto zone placement, a feature that automatically selects zones with the highest available resources. Using this feature alongside flexible machine types helps keep Dataproc pipelines highly available, even when specific zones run out of stock.
This focus area explains how to use BigQuery's reservation system to guarantee dedicated processing power for your most important data jobs, preventing them from competing with other queries.
BigQuery reservations are isolated pools of slots, which are virtual compute units. By creating reservations, you allocate dedicated processing capacity to specific workloads. This is essential for ensuring that business-critical data processes have guaranteed resources and do not compete with ad-hoc or less important jobs. You can create multiple reservations—for example, separate ones for data science, data loading (ELT), and business intelligence—and assign projects, folders, or entire organizations to them. This assignment isolates workloads and prevents resource contention.
Reservations are regional resources and must be created and managed within a specific location. You create them within an administration project, which centralizes billing and management for your slot commitments. Best practice is to use a single, dedicated administration project for your organization, though complex organizations with separate billing needs might require more. A reservation cannot be moved to a different administration project after creation. Within a region, you can mix capacity-based billing (using reservations) and on-demand billing by assigning some projects to a reservation and leaving others unassigned.
Reservations provide enhanced control through Identity and Access Management (IAM) conditions. Administrators can define policies that specify which users, groups, or service accounts are authorized to use a specific reservation. This is done by granting the reservations.use permission with conditions based on reservation attributes, like its name prefix. This ensures sensitive workloads are protected and access is precisely controlled. You can also configure a reservation to ignore idle slots from other pools, guaranteeing it uses only its dedicated capacity.
Choosing the right number of slots involves balancing performance needs with cost. You should experiment with baseline and autoscaling slots and analyze your workload's performance over at least seven days. Tools like the slot estimator can help model the impact of adding or reducing slots. To monitor actual usage, you must use:
StudyCerts
StudyCertsINFORMATION_SCHEMA.JOBS viewsThis monitoring validates your reservation sizing, tracks slot utilization, and ensures critical workloads have the capacity they need without overspending.
This focus area covers how to track and plan for resource constraints across Google Cloud data services to ensure business-critical processes have the capacity they need.
Quotas are limits Google Cloud sets on how many resources you can use in your project. They help protect the community from unexpected usage spikes and ensure fair access. Quotas apply to resources like virtual machines, API calls, storage, and specialized hardware like GPUs and TPUs. It is crucial to understand that having available quota does not guarantee a resource is available in a specific zone or region. Each GCP service, including Dataflow, Dataproc, BigQuery, and Compute Engine, has its own set of quotas and limits that require monitoring.
Cloud Monitoring is the primary tool for tracking resource usage and quotas across GCP. It automatically collects performance data from most services and lets you create dashboards to visualize metrics like CPU utilization and slot consumption. You can set up alerting policies to notify you when you are approaching quota limits or when resource usage exceeds defined thresholds, such as GPU utilization reaching 80%.
Capacity planning involves analyzing historical data to predict future resource needs. You should review metrics like GPU/TPU utilization, throughput rates, and query performance using Cloud Monitoring and Cloud Logging. Tools like BigQuery's slot estimator help estimate processing needs. For large-scale events, you can use Capacity Planner to view historical and forecasted usage of VMs, disks, and GPUs, and to request reservations in advance.
To secure specific compute resources for critical workloads, you can use reservations and commitments. Committed use discounts let you commit to using specific amounts of vCPUs or memory for one or three years at lower prices, ideal for predictable workloads. Reservations provide guaranteed capacity in specific zones for on-demand resources. For high-demand accelerators like GPUs and TPUs, you can use future reservation requests to secure capacity in advance with high assurance.
You can view your current quotas in the Google Cloud console under IAM & Admin > Quotas. To request an increase, select the quota, click Edit quotas, enter your desired value, and submit the request with a justification. The quota adjuster can also automatically monitor usage and submit requests when you approach limits. For large increases, you may need to contact support.
Even with sufficient quota, resources like GPUs may be unavailable if a zone is depleted. Strategies to avoid this include using auto zone placement, distributing workloads across multiple zones, and maintaining flexibility in machine type selection. For high availability, deploy critical workloads across multiple regions using global load balancing and services like GKE regional clusters.
Prepare and test your skills
Prepare and test your skills
BigQuery reservations create isolated pools of slots, which are virtual compute units, to allocate dedicated processing capacity to specific workloads. This guarantees that business-critical data processes have the resources they need and do not compete with ad-hoc or less important jobs.
Cloud Dataflow uses horizontal autoscaling to dynamically change the number of worker instances based on workload factors like pipeline parallelism, CPU usage, and pending data. It has a maximum limit of 2,000 Compute Engine instances for standard jobs and 4,000 instances for Streaming Engine jobs.
To prevent cluster creation failures from resource exhausted errors, administrators should use the auto zone placement feature. This feature automatically selects zones with the highest available resources, helping to keep Dataproc pipelines highly available.
You must monitor actual usage using the INFORMATION_SCHEMA.JOBS views, Cloud Monitoring dashboards, BigQuery Audit logs, and the Jobs API. This monitoring validates reservation sizing, tracks slot utilization, and ensures critical workloads have the capacity they need without overspending.
A data engineering team runs business-critical Apache Spark batch processing workloads on a Google Cloud Dataproc cluster. The team observes two main operational issues:
Which configuration should the data engineering team implement to resolve both issues?