Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Managed Service for Apache Spark
Topic in 25 exam domains
Also called: Cloud Dataproc Serverless, Cloud Managed Service for Apache Spark, Google Managed Service for Apache Spark, Managed Spark, Serverless Spark
A fully managed Google Cloud runtime for processing open table formats like Apache Iceberg.
Where this appears on the exam
18 objectives in 4 of 5 domains
~22% of exam
~25% of exam
Domain 3
Storing the data~20% of exam
~18% of exam
Related topics
FoundationsBuilding blocksThis topicWorks alongsideAppliedApplication Default CredentialsFeaturecustomer-managed encryption keysFeatureVPC Service ControlsServiceDataflowServiceCompute EngineServiceKnowledge CatalogServicePub/SubServiceCloud StorageServiceBigQueryService
Managed Service for Apache SparkService
Lakehouse runtime catalogFeatureCloud ProfilerServiceManaged Service for Apache AirflowServiceCloud LoggingServiceGKEServiceIdentity and Access ManagementServiceCloud SQLServicePrivate Google AccessFeatureCloud Data FusionServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Cloud Data Fusion works with Managed Service for Apache Spark From the official documentation: “Deployed Cloud Data Fusion pipelines run on Managed Service for Apache Spark clusters by default.”
- Managed Service for Apache Spark depends on Private Google Access From the official documentation: “Managed Service for Apache Spark batch workloads and interactive sessions run on VMs with internal IP addresses only and on a regional subnet with [Private Google Access (PGA)](/vpc/docs/configure-private-google-access) enabled automatically on the subnet.”
- BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
- Lakehouse runtime catalog works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark supports only PySpark jobs with borderless Lakehouse Metastore.”
- BigQuery is an alternative to Managed Service for Apache Spark From the official documentation: “Run queries against your federated data using BigQuery or Managed Service for Apache Spark.”
- Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
- Dataflow is an alternative to Managed Service for Apache Spark From the official documentation: “For more complex data loads into your Cloud SQL for PostgreSQL database, consider using Dataflow or Managed Service for Apache Spark, which involves creating an ETL process.”
- Application Default Credentials works with Managed Service for Apache Spark From the official documentation: “To authenticate to Managed Service for Apache Spark, set up Application Default Credentials.”
- Application Default Credentials secures Managed Service for Apache Spark From the official documentation: “To authenticate to Managed Service for Apache Spark, set up Application Default Credentials.”
- Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
- Managed Service for Apache Spark works with Private Google Access From the official documentation: “Managed Service for Apache Spark automatically enables Private Google Access on the regional subnet used by Managed Service for Apache Spark clusters created with image version `2.2` and later.”
- Cloud Profiler works with Managed Service for Apache Spark From the official documentation: “Profiler supports only Managed Service for Apache Spark Hadoop and Spark job types (Spark, PySpark, SparkSql, and SparkR).”
- Compute Engine works with Managed Service for Apache Spark From the official documentation: “Yes, Managed Service for Apache Spark has native and automatic integrations with Compute Engine, Cloud Storage, Bigtable, BigQuery, Logging, and Cloud Monitoring.”
- customer-managed encryption keys secures Managed Service for Apache Spark From the official documentation: “For data encryption at rest, use the [Cloud Key Management Service](/kms/docs/key-management-service) (KMS) or [Customer Managed Encryption Keys](/managed-spark/docs/concepts/configuring-clusters/customer-managed-encryption) (CMEK).”
- Knowledge Catalog works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark integrates with other Google Cloud products, such as BigQuery and Knowledge Catalog.”
- Managed Service for Apache Airflow works with Managed Service for Apache Spark From the official documentation: “# Run Managed Service for Apache Spark workloads with Managed Airflow”
- Managed Service for Apache Spark depends on Application Default Credentials From the official documentation: “To authenticate to Managed Service for Apache Spark, set up Application Default Credentials.”
- Cloud Data Fusion depends on Managed Service for Apache Spark From the official documentation: “Cloud Data Fusion uses Managed Service for Apache Spark as the execution environment for pipelines.”
- Cloud Logging works with Managed Service for Apache Spark From the official documentation: “Use [Cloud Logging](/logging) to view logs from the Managed Service for Apache Spark Autoscaler.”
- GKE works with Managed Service for Apache Spark From the official documentation: “You can create Managed Service for Apache Spark clusters on: - [Managed Service for Apache Spark clusters on Compute Engine](/dataproc/docs/guides/create-cluster) - [Managed Service for Apache Spark clusters on GKE](/dataproc/docs/guides/dpgke/dataproc-gke-overview)”
- Managed Service for Apache Spark depends on Compute Engine From the official documentation: “Managed Service for Apache Spark clusters are built on [Compute Engine](/compute) instances.”
- Managed Service for Apache Spark configures Private Google Access From the official documentation: “Managed Service for Apache Spark enables [Private Google Access (PGA)](/vpc/docs/private-google-access) on the specified subnet.”
- Managed Service for Apache Spark works with Pub/Sub From the official documentation: “You can use Managed Service for Apache Spark to process high-volume streams from a service like Pub/Sub or Apache Kafka and land the data in the bronze tier.”
- VPC Service Controls secures Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark requires <a href="/vpc-service-controls/docs/set-up-private-connectivity">special steps</a> to protect using VPC Service Controls.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Knowledge Catalog depends on Application Default Credentials From the official documentation: “To authenticate to Knowledge Catalog, set up Application Default Credentials.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- BigQuery works with Knowledge Catalog From the official documentation: “Some Knowledge Catalog functionalities trigger job execution using BigQuery.”
- Cloud Logging works with customer-managed encryption keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
- Cloud SQL works with Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
- Cloud SQL works with Compute Engine From the official documentation: “Create a Compute Engine VM Instance to host a sample web app that connects to Cloud SQL.”
- BigQuery works with Lakehouse runtime catalog From the official documentation: “When querying Lakehouse runtime catalog tables from BigQuery, you use a four-part naming structure, often referred to as **P.C.N.T**:”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Application Default Credentials secures Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Logging works with GKE From the official documentation: “Enable Cloud Logging on GKE Clusters”
- Compute Engine is an alternative to GKE From the official documentation: “To use TPU7x, you can use Google Kubernetes Engine (GKE) or Compute Engine.”
- Cloud SQL works with Identity and Access Management From the official documentation: “This page describes how Cloud SQL is integrated with IAM .”
- Cloud SQL works with Knowledge Catalog From the official documentation: “You can use Knowledge Catalog to organize and enrich the metadata of Cloud SQL resources.”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- customer-managed encryption keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Logging works with Cloud SQL From the official documentation: “Cloud SQL uses Cloud Logging.”
How learners score on this topic
Last updated