Apache Spark

Related topics

• Apache Spark works with Lakehouse runtime catalog From the official documentation: “Apache Spark connects to *borderless Lakehouse* through the *Lakehouse runtime catalog*.”
• Apache Spark works with Managed Service for Apache Spark From the official documentation: “<li><a href="/bigquery/docs/bqms-use-dataproc">Query data with open source engines such as Apache Spark.</a></li>”
• Apache Spark works with Lakehouse From the official documentation: “Apache Spark connects to *borderless Lakehouse* through the *Lakehouse runtime catalog*.”
• Managed Service for Apache Spark depends on Apache Spark From the official documentation: “Compute engines like Managed Service for Apache Spark use open-source Apache Spark with execution optimizations to help ensure workload portability and avoid vendor lock-in.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• BigQuery works with Lakehouse runtime catalog From the official documentation: “The Lakehouse runtime catalog lets you share data across Apache Spark, Apache Hive, and BigQuery by providing a serverless, unified metastore.”
• Dataflow works with Lakehouse From the official documentation: “Importing Parquet files in cloud-based storage to Lakehouse using Dataflow has the following limitations:”
• BigQuery works with Lakehouse From the official documentation: “BigLake automatically registers the Iceberg table in BigQuery:”
• Cloud Storage works with Lakehouse runtime catalog From the official documentation: “Set up and configure Lakehouse catalog endpoints to unify data silos across AWS S3, Cloud Storage, and AlloyDB.”
• Cloud Storage works with Managed Service for Apache Spark From the official documentation: “| Managed Service for Apache Spark | Stored in Cloud Storage, for each data chunk | |”
• Lakehouse runtime catalog works with Managed Service for Apache Spark From the official documentation: “To get the permissions that you need to use Managed Service for Apache Spark with the Lakehouse runtime catalog as a metadata store, ask your administrator to grant you the following IAM roles:”
• Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
• BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
• Cloud Storage works with Lakehouse From the official documentation: “Using Lakehouse to manage Iceberg tables, you kept the underlying table files secure in Cloud Storage.”
• Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
• Dataflow is an alternative to Managed Service for Apache Spark From the official documentation: “For more complex data loads into your Cloud SQL for PostgreSQL database, consider using Dataflow or Managed Service for Apache Spark, which involves creating an ETL process.”
• Dataproc Metastore works with Managed Service for Apache Spark From the official documentation: “The Managed Service for Apache Spark cluster attached to the Dataproc Metastore service.”
• Lakehouse runtime catalog is part of Lakehouse From the official documentation: “Borderless Lakehouse lets you establish a managed metadata layer in the Lakehouse runtime catalog that connects open source processing engines with Google Cloud.”
• BigQuery depends on Cloud Storage From the official documentation: “BigQuery relies on Cloud Storage for staging file data and supports reading from and writing to several other sources and Google Cloud services.”
• BigQuery depends on Lakehouse runtime catalog From the official documentation: “BigQuery federates queries to these endpoints using the Google Cloud Lakehouse runtime catalog connection.”
• BigQuery is an alternative to Managed Service for Apache Spark From the official documentation: “An Analytics Workload refers to the migration of new Data Lake and Data Warehouse environments from on-premises (e.g., Teradata, Cloudera) and cloud platforms (e.g., Snowflake, Redshift, Databricks) to BigQuery or Managed Service for Apache Spark on Google Cloud.”
• Cloud Storage works with Dataproc Metastore From the official documentation: “[Delete the Cloud Storage bucket](/storage/docs/deleting-buckets) for the Dataproc Metastore service.”
• Dataproc Metastore depends on Cloud Storage From the official documentation: “Dataproc Metastore only supports the use of Cloud Storage buckets for the warehouse directory.”
• Dataproc Metastore is an alternative to Lakehouse From the official documentation: “DPMS is a legacy metastore. Customers are encouraged to use the BigLake Lakehouse REST Catalog (LRC).”
• Dataproc Metastore is an alternative to Lakehouse runtime catalog From the official documentation: “To upgrade and migrate your metadata from Dataproc Metastore to the Lakehouse runtime catalog, see [Migrate metadata from Dataproc Metastore to Lakehouse](/lakehouse/docs/migrate-dataproc-metastore).”