Managed Service for Apache Spark

Related topics

• Cloud Data Fusion works with Managed Service for Apache Spark From the official documentation: “Deployed Cloud Data Fusion pipelines run on Managed Service for Apache Spark clusters by default.”
• Cloud Storage works with Managed Service for Apache Spark From the official documentation: “| Managed Service for Apache Spark | Stored in Cloud Storage, for each data chunk | |”
• Compute Engine works with Managed Service for Apache Spark From the official documentation: “Google Compute Engine provides a runtime environment for all features provided by Managed Service for Apache Spark and Gemini Enterprise Agent Platform.”
• Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
• Managed Service for Apache Spark works with Private Google Access From the official documentation: “Managed Service for Apache Spark automatically enables Private Google Access on the regional subnet used by Managed Service for Apache Spark clusters created with image version `2.2` and later.”
• Managed Service for Apache Spark configures Private Google Access From the official documentation: “Managed Service for Apache Spark enables [Private Google Access (PGA)](/vpc/docs/private-google-access) on the subnet.”
• BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
• Bigtable works with Managed Service for Apache Spark From the official documentation: “With Managed Service for Apache Spark, you can run Hadoop jobs that read from and write to Bigtable.”
• Dataflow is an alternative to Managed Service for Apache Spark From the official documentation: “For more complex data loads into your Cloud SQL for PostgreSQL database, consider using Dataflow or Managed Service for Apache Spark, which involves creating an ETL process.”
• Managed Service for Apache Airflow works with Managed Service for Apache Spark From the official documentation: “This tutorial shows how to use Managed Service for Apache Airflow to create an [Apache Airflow](http://airflow.apache.org) DAG (Directed Acyclic Graph) that runs an Apache Hadoop wordcount job on a Managed Service for Apache Spark cluster.”
• Managed Service for Apache Spark depends on Compute Engine From the official documentation: “Managed Service for Apache Spark clusters are built on [Compute Engine](/compute) instances.”
• Managed Service for Apache Spark depends on Private Google Access From the official documentation: “Run the following command to [enable Private Google Access](/vpc/docs/configure-private-google-access#enabling-pga) on the default subnet in the region where you would like to run Managed Service for Apache Spark to fulfill [networking…”
• BigQuery is an alternative to Managed Service for Apache Spark From the official documentation: “An Analytics Workload refers to the migration of new Data Lake and Data Warehouse environments from on-premises (e.g., Teradata, Cloudera) and cloud platforms (e.g., Snowflake, Redshift, Databricks) to BigQuery or Managed Service for Apache Spark on Google Cloud.”
• Cloud Data Fusion depends on Managed Service for Apache Spark From the official documentation: “Cloud Data Fusion uses Managed Service for Apache Spark as the execution environment for pipelines.”
• Cloud Logging works with Managed Service for Apache Spark From the official documentation: “Enables (`true`) or disables (`false`) Managed Service for Apache Spark job driver logs in [Cloud Logging](/logging/docs).”
• Cloud Monitoring works with Managed Service for Apache Spark From the official documentation: “Monitoring uses workload metadata and metrics to provide insights into the health and performance of Managed Service for Apache Spark workloads.”
• Cloud SQL works with Managed Service for Apache Spark From the official documentation: “Use Datastream to replicate changes from your MySQL instance to either BigQuery or Cloud Storage, and then use Dataflow or Managed Service for Apache Spark to bring the data into your Cloud SQL instance.”
• Customer-Managed Encryption Keys works with Managed Service for Apache Spark From the official documentation: “If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in [Cloud KMS](/kms/docs) with CMEK-integrated services including Managed Service for Apache Spark.”
• Customer-Managed Encryption Keys secures Managed Service for Apache Spark From the official documentation: “You can use customer-managed encryption keys (CMEK) to encrypt the following cluster data: - Data on persistent disks attached to Managed Service for Apache Spark cluster VMs - Job argument data submitted to your cluster, such as a query string submitted with a Spark SQL job -…”
• GKE works with Managed Service for Apache Spark From the official documentation: “You can create Managed Service for Apache Spark clusters on: - [Managed Service for Apache Spark clusters on Compute Engine](/dataproc/docs/guides/create-cluster) - [Managed Service for Apache Spark clusters on GKE](/dataproc/docs/guides/dpgke/dataproc-gke-overview)”
• Google Cloud CLI works with Managed Service for Apache Spark From the official documentation: “When you use the [gcloud CLI](/sdk/gcloud/reference/dataproc) to access Managed Service for Apache Spark, you authenticate to Google Cloud as a principal, which provides the credentials used by the gcloud CLI commands.”
• Google Cloud CLI configures Managed Service for Apache Spark From the official documentation: “You can specify one or more labels to be applied to a Managed Service for Apache Spark cluster or job at creation or submit time using the Google Cloud CLI.”
• Google Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Dataproc job outget gets saved to a GCS bucket allocated to it.”
• Managed Service for Apache Spark depends on Cloud Logging From the official documentation: “Managed Service for Apache Spark clusters require network connectivity to Google Cloud APIs and services, such as the Dataproc API, Cloud Storage, and Cloud Logging, and to user resources, such as data sources in other VPC networks or on-premises environments.”
• Managed Service for Apache Spark works with Pub/Sub From the official documentation: “You can use Managed Service for Apache Spark to process high-volume streams from a service like Pub/Sub or Apache Kafka and land the data in the bronze tier.”
• Managed Service for Apache Spark works with VPC Service Controls From the official documentation: “Consider the following strategies when you use VPC-SC networks with Managed Service for Apache Spark:”
• VPC Service Controls secures Managed Service for Apache Spark From the official documentation: “In this example, a Managed Service for Apache Spark cluster is running in a project protected by VPC Service Controls.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
• Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
• BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
• BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
• Customer-Managed Encryption Keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
• Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
• BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• Cloud Logging works with GKE From the official documentation: “Enable Cloud Logging on GKE Clusters”
• Cloud Monitoring works with GKE From the official documentation: “Enable Cloud Monitoring on GKE Clusters”
• Customer-Managed Encryption Keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
• Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
• BigQuery works with Bigtable From the official documentation: “BigQuery lets you query your Bigtable data from BigQuery.”
• BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
• Cloud SQL works with Compute Engine From the official documentation: “Cloud SQL uses Compute Engine virtual machines (VMs) with persistent storage disks.”
• Compute Engine is an alternative to GKE From the official documentation: “You can also deploy a third-party model to either Vertex AI Model Garden or self-host on GKE or Compute Engine.”
• Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud SQL works with GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
• Cloud Storage works with Customer-Managed Encryption Keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
• Cloud Storage works with Google Cloud CLI From the official documentation: “The gcloud CLI is the preferred command line tool for accessing Cloud Storage.”
• Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
• Cloud Logging works with Cloud SQL From the official documentation: “Database Migration Service and Cloud SQL use Cloud Logging.”
• Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud Storage works with Compute Engine From the official documentation: “For example, you can create a service account that your Compute Engine instances use to access objects stored in Cloud Storage buckets.”
• Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
• Customer-Managed Encryption Keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
• Customer-Managed Encryption Keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
• VPC Service Controls secures Managed Service for Apache Airflow From the official documentation: “Managed Airflow can be selected as a secured service inside a VPC Service Controls perimeter.”
• BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”