Managed Service for Apache Spark

Related topics

• Managed Service for Apache Spark depends on Private Google Access From the official documentation: “Managed Service for Apache Spark batch workloads and interactive sessions run on VMs with internal IP addresses only and on a regional subnet with [Private Google Access (PGA)](/vpc/docs/configure-private-google-access) enabled automatically on the subnet.”
• BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
• BigQuery is an alternative to Managed Service for Apache Spark From the official documentation: “Run queries against your federated data using BigQuery or Managed Service for Apache Spark.”
• Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
• Dataflow is an alternative to Managed Service for Apache Spark From the official documentation: “For more complex data loads into your Cloud SQL for PostgreSQL database, consider using Dataflow or Managed Service for Apache Spark, which involves creating an ETL process.”
• Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
• Managed Service for Apache Spark works with Private Google Access From the official documentation: “Managed Service for Apache Spark automatically enables Private Google Access on the regional subnet used by Managed Service for Apache Spark clusters created with image version `2.2` and later.”
• autoscaling is part of Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark [Autoscaling](/managed-spark/docs/concepts/configuring-clusters/autoscaling) dynamically adds and removes Managed Service for Apache Spark workers from a cluster to help ensure that Spark jobs have the resources needed to complete quickly.”
• Cloud Profiler works with Managed Service for Apache Spark From the official documentation: “Profiler supports only Managed Service for Apache Spark Hadoop and Spark job types (Spark, PySpark, SparkSql, and SparkR).”
• Compute Engine works with Managed Service for Apache Spark From the official documentation: “Yes, Managed Service for Apache Spark has native and automatic integrations with Compute Engine, Cloud Storage, Bigtable, BigQuery, Logging, and Cloud Monitoring.”
• customer-managed encryption keys secures Managed Service for Apache Spark From the official documentation: “For data encryption at rest, use the [Cloud Key Management Service](/kms/docs/key-management-service) (KMS) or [Customer Managed Encryption Keys](/managed-spark/docs/concepts/configuring-clusters/customer-managed-encryption) (CMEK).”
• Knowledge Catalog works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark integrates with other Google Cloud products, such as BigQuery and Knowledge Catalog.”
• Managed Service for Apache Airflow works with Managed Service for Apache Spark From the official documentation: “# Run Managed Service for Apache Spark workloads with Managed Airflow”
• BigQuery ML is an alternative to Managed Service for Apache Spark From the official documentation: “Use BigQuery ML, BigQuery DataFrames, or Managed Service for Apache Spark to perform distributed data processing on large datasets.”
• Cloud Logging works with Managed Service for Apache Spark From the official documentation: “Use [Cloud Logging](/logging) to view logs from the Managed Service for Apache Spark Autoscaler.”
• Cloud Monitoring works with Managed Service for Apache Spark From the official documentation: “Monitoring uses workload metadata and metrics to provide insights into the health and performance of Managed Service for Apache Spark workloads.”
• GKE works with Managed Service for Apache Spark From the official documentation: “You can create Managed Service for Apache Spark clusters on: - [Managed Service for Apache Spark clusters on Compute Engine](/dataproc/docs/guides/create-cluster) - [Managed Service for Apache Spark clusters on GKE](/dataproc/docs/guides/dpgke/dataproc-gke-overview)”
• Managed Service for Apache Spark depends on Compute Engine From the official documentation: “Managed Service for Apache Spark clusters are built on [Compute Engine](/compute) instances.”
• Managed Service for Apache Spark configures Private Google Access From the official documentation: “Managed Service for Apache Spark enables [Private Google Access (PGA)](/vpc/docs/private-google-access) on the specified subnet.”
• Managed Service for Apache Spark works with Pub/Sub From the official documentation: “You can use Managed Service for Apache Spark to process high-volume streams from a service like Pub/Sub or Apache Kafka and land the data in the bronze tier.”
• VPC Service Controls secures Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark requires <a href="/vpc-service-controls/docs/set-up-private-connectivity">special steps</a> to protect using VPC Service Controls.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
• BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
• BigQuery works with Knowledge Catalog From the official documentation: “Some Knowledge Catalog functionalities trigger job execution using BigQuery.”
• Cloud Logging works with customer-managed encryption keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
• BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
• Cloud Run works with Pub/Sub From the official documentation: “Route Cloud Pub/Sub events to Cloud Run”
• Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
• Cloud Run works with Cloud Storage From the official documentation: “Route Cloud Storage events to Cloud Run”
• BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Cloud Logging works with GKE From the official documentation: “Enable Cloud Logging on GKE Clusters”
• Compute Engine is an alternative to GKE From the official documentation: “To use TPU7x, you can use Google Kubernetes Engine (GKE) or Compute Engine.”
• BigQuery ML works with Cloud Storage From the official documentation: “You can export BigQuery ML models to Cloud Storage, and use them for online prediction, or edit them in Python.”
• Cloud Monitoring works with GKE From the official documentation: “Enable Cloud Monitoring on GKE Clusters”
• Cloud Run depends on Pub/Sub From the official documentation: “Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• customer-managed encryption keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
• Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
• BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
• Cloud Logging works with Managed Service for Apache Airflow From the official documentation: “You can use [Cloud Monitoring](/monitoring/docs) and [Cloud Logging](/logging/docs) with Managed Service for Apache Airflow.”
• Cloud Run is an alternative to GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
• Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow associates a Cloud Storage bucket with your environment.”
• customer-managed encryption keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
• customer-managed encryption keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
• BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
• Cloud Monitoring works with Managed Service for Apache Airflow From the official documentation: “You can use [Cloud Monitoring](/monitoring/docs) and [Cloud Logging](/logging/docs) with Managed Service for Apache Airflow.”
• Cloud Storage works with customer-managed encryption keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
• Cloud Storage works with Identity and Access Management From the official documentation: “To learn how to use IAM with Cloud Storage, see [Using IAM Permissions](/storage/docs/access-control/using-iam-permissions).”
• Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
• Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
• Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud Monitoring works with Dataflow From the official documentation: “Dataflow is fully integrated with Cloud Monitoring.”
• customer-managed encryption keys secures Managed Service for Apache Airflow From the official documentation: “This page describes how to use [Customer Managed Encryption Keys (CMEK)](/kms/docs/cmek) to protect Managed Airflow environments.”
• Identity and Access Management secures BigQuery From the official documentation: “Access to BigQuery data from AlloyDB is managed through IAM.”
• Identity and Access Management secures Cloud Storage From the official documentation: “Cloud Storage uses Identity and Access Management (IAM) for authorization.”