Apache Spark

Related topics

• Apache Spark works with Managed Service for Apache Spark From the official documentation: “<li><a href="/bigquery/docs/bqms-use-dataproc">Query data with open source engines such as Apache Spark.</a></li>”
• Apache Spark works with BigQuery From the official documentation: “Built on the *Apache Iceberg REST Catalog API*, this endpoint acts as a single source of truth for open-source or third-party engines, such as Apache Spark, that access data in Cloud Storage and interoperate with BigQuery.”
• Apache Spark works with Cloud Storage From the official documentation: “Built on the *Apache Iceberg REST Catalog API*, this endpoint acts as a single source of truth for open-source or third-party engines, such as Apache Spark, that access data in Cloud Storage and interoperate with BigQuery.”
• Managed Service for Apache Spark depends on Apache Spark From the official documentation: “Compute engines like Managed Service for Apache Spark use open-source Apache Spark with execution optimizations to help ensure workload portability and avoid vendor lock-in.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
• BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
• Customer-Managed Encryption Keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
• Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
• BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
• Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud Storage works with Customer-Managed Encryption Keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
• Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud Storage works with Compute Engine From the official documentation: “For example, you can create a service account that your Compute Engine instances use to access objects stored in Cloud Storage buckets.”
• Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
• Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
• Customer-Managed Encryption Keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
• Customer-Managed Encryption Keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
• BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Cloud Logging works with Compute Engine From the official documentation: “Compute Engine sends you Cloud Logging events for scheduled, started, or completed maintenance.”
• Compute Engine works with Managed Service for Apache Spark From the official documentation: “Google Compute Engine provides a runtime environment for all features provided by Managed Service for Apache Spark and Gemini Enterprise Agent Platform.”
• Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
• VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
• BigQuery depends on Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
• BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
• Cloud Logging is an alternative to Cloud Storage From the official documentation: “Instead, they are routed to either Cloud Storage (recommended) or Cloud Logging, depending on your organization's configuration.”
• Cloud Logging works with Customer-Managed Encryption Keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
• Customer-Managed Encryption Keys depends on Cloud Storage From the official documentation: “If used for storing Customer-managed Encryption Keys (CMEK): Cloud Storage (See [Cloud Storage pricing](/storage/pricing).)”
• Customer-Managed Encryption Keys secures Compute Engine From the official documentation: “In IL5, CMEK is enabled by default for Compute Engine, Google Kubernetes Engine (GKE), Cloud Storage, and Cloud SQL.”
• Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
• Dataflow depends on Compute Engine From the official documentation: “When Dataflow runs a pipeline, the processing is distributed across multiple Compute Engine virtual machines (VMs), also known as workers.”
• Dataflow is an alternative to Managed Service for Apache Spark From the official documentation: “For more complex data loads into your Cloud SQL for PostgreSQL database, consider using Dataflow or Managed Service for Apache Spark, which involves creating an ETL process.”
• Managed Service for Apache Spark depends on Compute Engine From the official documentation: “Managed Service for Apache Spark clusters are built on [Compute Engine](/compute) instances.”
• VPC Service Controls secures Compute Engine From the official documentation: “To further protect your Compute Engine resources, you can protect them using [VPC Service Controls](/vpc-service-controls/docs/overview).”
• BigQuery depends on Cloud Storage From the official documentation: “BigQuery relies on Cloud Storage for staging file data and supports reading from and writing to several other sources and Google Cloud services.”
• BigQuery works with Compute Engine From the official documentation: “Compute Engine integrates with Google Cloud technologies such as Cloud Storage, Google Kubernetes Engine, and BigQuery, to extend beyond the basic computational capability to create more complex and sophisticated applications.”
• BigQuery is an alternative to Managed Service for Apache Spark From the official documentation: “An Analytics Workload refers to the migration of new Data Lake and Data Warehouse environments from on-premises (e.g., Teradata, Cloudera) and cloud platforms (e.g., Snowflake, Redshift, Databricks) to BigQuery or Managed Service for Apache Spark on Google Cloud.”
• BigQuery works with VPC Service Controls From the official documentation: “BigQuery integrates with VPC Service Controls to restrict access to your BigQuery resources from specific networks.”
• Cloud Logging depends on Customer-Managed Encryption Keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
• Cloud Logging works with Dataflow From the official documentation: “You can control the volume of Dataflow logs ingested by Cloud Logging by following the [Filter logs documentation](/dataflow/docs/guides/filter-logs).”
• Cloud Logging works with Managed Service for Apache Spark From the official documentation: “Enables (`true`) or disables (`false`) Managed Service for Apache Spark job driver logs in [Cloud Logging](/logging/docs).”
• Cloud Logging works with VPC Service Controls From the official documentation: “This page describes how you can find VPC Service Controls errors using Cloud Logging.”
• Customer-Managed Encryption Keys is part of BigQuery From the official documentation: “If you require more control over encryption keys, BigQuery supports [customer-managed encryption keys](/bigquery/docs/customer-managed-encryption) in the [Cloud Key Management Service](/kms/docs).”
• Customer-Managed Encryption Keys configures Cloud Logging From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
• Customer-Managed Encryption Keys secures Cloud Logging From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
• Customer-Managed Encryption Keys is part of Cloud Storage From the official documentation: “[Cloud Storage](/storage/docs/introduction) provides features such as customer-managed encryption, lifecycle management, and aging and retention policies.”
• Customer-Managed Encryption Keys works with Dataflow From the official documentation: “This sample shows how to use encryption keys managed by the customer, with a Dataflow pipeline.”
• Customer-Managed Encryption Keys secures Dataflow From the official documentation: “You can either create CMEKs manually to protect your Dataflow resources or use Cloud KMS Autokey.”
• Customer-Managed Encryption Keys works with Managed Service for Apache Spark From the official documentation: “If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in [Cloud KMS](/kms/docs) with CMEK-integrated services including Managed Service for Apache Spark.”
• Customer-Managed Encryption Keys secures Managed Service for Apache Spark From the official documentation: “You can use customer-managed encryption keys (CMEK) to encrypt the following cluster data: - Data on persistent disks attached to Managed Service for Apache Spark cluster VMs - Job argument data submitted to your cluster, such as a query string submitted with a Spark SQL job -…”
• Dataflow works with VPC Service Controls From the official documentation: “To customize DNS resolution when using Dataflow with VPC Service Controls, use Cloud DNS [private zones](/dns/docs/overview#dns-private-zones) instead of using custom BIND servers.”
• Managed Service for Apache Spark depends on Cloud Logging From the official documentation: “Managed Service for Apache Spark clusters require network connectivity to Google Cloud APIs and services, such as the Dataproc API, Cloud Storage, and Cloud Logging, and to user resources, such as data sources in other VPC networks or on-premises environments.”
• Managed Service for Apache Spark works with Pub/Sub From the official documentation: “You can use Managed Service for Apache Spark to process high-volume streams from a service like Pub/Sub or Apache Kafka and land the data in the bronze tier.”
• Managed Service for Apache Spark works with VPC Service Controls From the official documentation: “Consider the following strategies when you use VPC-SC networks with Managed Service for Apache Spark:”
• VPC Service Controls secures Dataflow From the official documentation: “Dataflow VPC Service Controls help protect against accidental or targeted action by external entities or by insider entities, which helps to minimize unwarranted data exfiltration risks.”
• VPC Service Controls secures Managed Service for Apache Spark From the official documentation: “In this example, a Managed Service for Apache Spark cluster is running in a project protected by VPC Service Controls.”