Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Dataproc Metastore
Topic in 1 exam domain
Also called: Apache Hive metastore, Dataproc Metastore Service, DPMS, HMS, Managed Service for Apache Spark Metastore, metastore service
A Google Cloud service that can be used alongside Knowledge Catalog.
Where this appears on the exam
1 subdomain in 1 of 5 domains
Related topics
FoundationsBuilding blocksBuilding blocksThis topicWorks alongsideAppliedIdentity and Access ManagementServiceVPC Service ControlsServiceCustomer-Managed Encryption KeysFeatureCompute EngineServiceBigQueryServiceCloud StorageServiceManaged Service for Apache AirflowServicePub/SubServiceCloud SQLServiceGKEServiceManaged Service for Apache SparkServiceCloud RunService
Dataproc MetastoreService
Private Service ConnectFeatureDataprocServicetagsFeatureApache IcebergConceptDataflowServiceVPC Network PeeringFeaturePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Dataproc Metastore works with Private Service Connect From the official documentation: “PSC lets you set up a private connection to Dataproc Metastore metadata across VPC networks.”
- Customer-Managed Encryption Keys secures Dataproc Metastore From the official documentation: “MySQL supports Dataproc Metastore encryption, such as using customer-managed encryption keys (CMEK).”
- Dataproc works with Dataproc Metastore From the official documentation: “A Hadoop cluster using Managed Service for Apache Spark must be in the same region as the DPMS instance to which it is attached.”
- Dataproc Metastore depends on Cloud SQL From the official documentation: “When you run a *start migration*, Dataproc Metastore connects to Cloud SQL and uses Cloud SQL as its backend database.”
- Dataproc Metastore works with Managed Service for Apache Spark From the official documentation: “The Managed Service for Apache Spark cluster attached to the Dataproc Metastore service.”
- Dataproc Metastore works with tags From the official documentation: “This page describes how to attach, detach, and list tags on Dataproc Metastore resources.”
- Dataproc Metastore works with VPC Service Controls From the official documentation: “For more information, see [VPC Service Controls with Dataproc Metastore](/dataproc-metastore/docs/vpc-sc).”
- Apache Iceberg works with Dataproc Metastore From the official documentation: “This page explains how to use Apache Iceberg tables with a Dataproc Metastore service attached to a Managed Service for Apache Spark cluster.”
- Cloud Run works with Dataproc Metastore From the official documentation: “These instructions show you how to configure event routing to your Cloud Run service that is triggered by a direct Dataproc Metastore event.”
- Cloud SQL is an alternative to Dataproc Metastore From the official documentation: “If you are using Cloud SQL as a Metastore using the `cloud-sql-proxy.sh` initialization action, refer to its specific troubleshooting in the [Cloud SQL Proxy initialization action…”
- Cloud Storage works with Dataproc Metastore From the official documentation: “[Delete the Cloud Storage bucket](/storage/docs/deleting-buckets) for the Dataproc Metastore service.”
- Customer-Managed Encryption Keys works with Dataproc Metastore From the official documentation: “If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in [Cloud KMS](/kms/docs) with CMEK-integrated services including Dataproc Metastore.”
- Dataproc Metastore depends on Cloud Run From the official documentation: “Dataproc Metastore utilizes Cloud Run for connectivity when using the [gRPC endpoint protocol](/dataproc-metastore/docs/about-endpoint-protocols#grpc).”
- Dataproc Metastore depends on Cloud Storage From the official documentation: “Dataproc Metastore only supports the use of Cloud Storage buckets for the warehouse directory.”
- Dataproc Metastore works with Identity and Access Management From the official documentation: “Dataproc Metastore uses Identity and Access Management (IAM) for authorization.”
- Dataproc Metastore works with VPC Network Peering From the official documentation: “After deleting a Dataproc Metastore, auto-created VPC Network Peering routes used by the service persist for up to 3 hours.”
- Dataproc Metastore configures VPC Network Peering From the official documentation: “After the VPC network is created, Dataproc Metastore also automatically configures [VPC Network Peering](/vpc/docs/vpc-peering) for your service.”
- Identity and Access Management secures Dataproc Metastore From the official documentation: “This page explains the ways you can add new users to your project and how to set access control for your Dataproc Metastore resources.”
- VPC Service Controls secures Dataproc Metastore From the official documentation: “With VPC Service Controls, you create perimeters around the Dataproc Metastore service.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
- Cloud Run depends on Pub/Sub From the official documentation: “Cloud Run function triggered by Pub/Sub to fetch HYPR MFA logs and write to GCS.”
- Customer-Managed Encryption Keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Cloud SQL works with Private Service Connect From the official documentation: “`--psc` Connect to the Cloud SQL instance using Private Service Connect.”
- Customer-Managed Encryption Keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
- Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
- Cloud Run works with Cloud SQL From the official documentation: “Connecting from Cloud Run (fully managed) to Cloud SQL”
- Cloud SQL works with Compute Engine From the official documentation: “Cloud SQL uses Compute Engine virtual machines (VMs) with persistent storage disks.”
- Compute Engine is an alternative to GKE From the official documentation: “You can also deploy a third-party model to either Vertex AI Model Garden or self-host on GKE or Compute Engine.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud SQL works with GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
- Cloud Storage works with Customer-Managed Encryption Keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Storage works with Compute Engine From the official documentation: “For example, you can create a service account that your Compute Engine instances use to access objects stored in Cloud Storage buckets.”
- Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Customer-Managed Encryption Keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
- Customer-Managed Encryption Keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
- Identity and Access Management works with tags From the official documentation: “You can use tags and IAM conditions to conditionally grant role bindings to users in your hierarchy.”
- BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Run works with Cloud Storage From the official documentation: “Receive events from Cloud Storage in a Cloud Run service”
- Cloud Run is an alternative to GKE From the official documentation: “For your containerized websites, you can also deploy them to [Cloud Run](/run) in addition to [using GKE](#kubernetes-engine).”
- Cloud Run works with Identity and Access Management From the official documentation: “For example, a remote backend deployed on Cloud Run can use Identity and Access Management (IAM) to:”
- Compute Engine works with Managed Service for Apache Spark From the official documentation: “Google Compute Engine provides a runtime environment for all features provided by Managed Service for Apache Spark and Gemini Enterprise Agent Platform.”
- Customer-Managed Encryption Keys secures Managed Service for Apache Airflow From the official documentation: “This page describes how to use [Customer Managed Encryption Keys (CMEK)](/kms/docs/cmek) to protect Managed Airflow environments.”
- GKE depends on Compute Engine From the official documentation: “For example, GKE deploys clusters on Compute Engine VMs.”
- Identity and Access Management secures Cloud Storage From the official documentation: “Cloud Storage uses Identity and Access Management (IAM) for authorization.”
- Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
- Private Service Connect is an alternative to VPC Network Peering From the official documentation: “Use VPC Network Peering if you need to insert NVAs or if your application doesn't support Private Service Connect”
- VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
- VPC Service Controls secures Managed Service for Apache Airflow From the official documentation: “Managed Airflow can be selected as a secured service inside a VPC Service Controls perimeter.”
- BigQuery works with Identity and Access Management From the official documentation: “BigQuery uses Identity and Access Management (IAM) for authorization.”
- BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
How learners score on this topic
Last updated