Apache Beam

Related topics

• Apache Beam works with Dataflow From the official documentation: “This page explains how to run an Apache Beam pipeline on Dataflow with GPUs.”
• Apache Beam works with BigQuery From the official documentation: “Apache Beam has many I/O connectors for several Google Cloud services, such as Bigtable, BigQuery, Datastore, Cloud SQL, and others.”
• Apache Beam works with Bigtable From the official documentation: “Write data to Cloud Bigtable with Apache Beam.”
• Dataflow depends on Apache Beam From the official documentation: “Dataflow is based on the open-source Apache Beam project.”
• Apache Beam works with Cloud SQL From the official documentation: “Apache Beam has many I/O connectors for several Google Cloud services, such as Bigtable, BigQuery, Datastore, Cloud SQL, and others.”
• Apache Beam works with Datastore From the official documentation: “Apache Beam has many I/O connectors for several Google Cloud services, such as Bigtable, BigQuery, Datastore, Cloud SQL, and others.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
• BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
• Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
• Customer-Managed Encryption Keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
• BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
• Customer-Managed Encryption Keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
• BigQuery works with Bigtable From the official documentation: “BigQuery lets you query your Bigtable data from BigQuery.”
• Cloud SQL works with Compute Engine From the official documentation: “Cloud SQL uses Compute Engine virtual machines (VMs) with persistent storage disks.”
• Cloud Storage works with Customer-Managed Encryption Keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
• Cloud SQL works with Vertex AI From the official documentation: “This page describes how to integrate Cloud SQL with [Vertex AI](/vertex-ai/docs).”
• Cloud Storage works with Compute Engine From the official documentation: “For example, you can create a service account that your Compute Engine instances use to access objects stored in Cloud Storage buckets.”
• Customer-Managed Encryption Keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
• BigQuery works with Vertex AI From the official documentation: “The page provides guidance for getting predictions from Vertex AI, based on retail [data exported into BigQuery](/retail/docs/export-data-into-bq).”
• Bigtable works with Dataflow From the official documentation: “The Dataflow job then writes the transformed data to your Bigtable table.”
• Customer-Managed Encryption Keys secures Bigtable From the official documentation: “In a CMEK-protected instance, Bigtable uses your CMEK key to protect your data at rest.”
• VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
• Bigtable is an alternative to Cloud Storage From the official documentation: “If the joining data is in the gigabyte or terabyte range, use either Bigtable or Cloud Storage to merge the joining data to the driving data, depending on the nature of the data.”
• Cloud Monitoring works with Cloud SQL From the official documentation: “You can also get more details by viewing the Cloud Monitoring logs for your Cloud SQL destination instance.”
• Cloud Monitoring works with Dataflow From the official documentation: “Dataflow is fully integrated with Cloud Monitoring.”
• Customer-Managed Encryption Keys depends on Cloud Storage From the official documentation: “If used for storing Customer-managed Encryption Keys (CMEK): Cloud Storage (See [Cloud Storage pricing](/storage/pricing).)”
• Customer-Managed Encryption Keys secures Compute Engine From the official documentation: “In IL5, CMEK is enabled by default for Compute Engine, Google Kubernetes Engine (GKE), Cloud Storage, and Cloud SQL.”
• Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
• Dataflow depends on Compute Engine From the official documentation: “When Dataflow runs a pipeline, the processing is distributed across multiple Compute Engine virtual machines (VMs), also known as workers.”
• VPC Service Controls secures Compute Engine From the official documentation: “To further protect your Compute Engine resources, you can protect them using [VPC Service Controls](/vpc-service-controls/docs/overview).”
• BigQuery depends on Cloud Storage From the official documentation: “BigQuery relies on Cloud Storage for staging file data and supports reading from and writing to several other sources and Google Cloud services.”
• BigQuery works with Compute Engine From the official documentation: “Compute Engine integrates with Google Cloud technologies such as Cloud Storage, Google Kubernetes Engine, and BigQuery, to extend beyond the basic computational capability to create more complex and sophisticated applications.”
• BigQuery works with Datastore From the official documentation: “BigQuery lets you load [Datastore](/datastore) entities from a Cloud Storage bucket into a BigQuery table for analysis.”
• BigQuery works with VPC Service Controls From the official documentation: “BigQuery integrates with VPC Service Controls to restrict access to your BigQuery resources from specific networks.”
• Bigtable works with Cloud Monitoring From the official documentation: “Bigtable exports [usage metrics](/monitoring/api/metrics_gcp_a_b#gcp-bigtable) to [Cloud Monitoring](/monitoring).”
• Bigtable is an alternative to Cloud SQL From the official documentation: “If you're interested in evaluating how the performance of your on-premises databases compares to [Cloud SQL](/sql), [Spanner](/spanner), [Firestore](/firestore), or [Bigtable](/bigtable), you could implement a PoC where the same business logic uses different databases.”
• Bigtable works with Cloud Storage From the official documentation: “For example, if you frequently write data to Bigtable from a Cloud Storage bucket in the `us-central1` (Iowa) region, create your Bigtable cluster in a zone within `us-central1`.”
• Bigtable works with Compute Engine From the official documentation: “OAuth scopes let Google Cloud services such as Compute Engine and Dataflow authenticate and connect to Bigtable.”
• Bigtable works with Customer-Managed Encryption Keys From the official documentation: “If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in [Cloud KMS](/kms/docs) with CMEK-integrated services including Bigtable.”
• Bigtable works with Vertex AI From the official documentation: “AssemblyAI builds complex speech-to-text models faster with Bigtable and Vertex AI.”
• Cloud Monitoring works with Compute Engine From the official documentation: “Many Google services like Compute Engine send telemetry to Cloud Monitoring.”
• Cloud Monitoring works with VPC Service Controls From the official documentation: “Cloud Monitoring performs the VPC perimeter check when a project is *added* to a metrics scope:”
• Cloud SQL depends on Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
• Cloud SQL depends on Compute Engine From the official documentation: “Cloud SQL uses Compute Engine virtual machines (VMs) with persistent storage disks.”
• Cloud SQL works with Dataflow From the official documentation: “Use Datastream to replicate changes from your MySQL instance to either BigQuery or Cloud Storage, and then use Dataflow or Managed Service for Apache Spark to bring the data into your Cloud SQL instance.”
• Cloud SQL works with VPC Service Controls From the official documentation: “Make sure that you configure VPC Service Controls so that the project that hosts the CA Service CA pool and CA<br>and the project that hosts Cloud SQL belong to the same service perimeter.”
• Cloud Storage works with Datastore From the official documentation: “Exports a copy of all or a subset of entities from Datastore to another storage system, such as Cloud Storage.”
• Cloud Storage works with Vertex AI From the official documentation: “Access Cloud Storage content through your current project's Vertex AI service agent.”
• Customer-Managed Encryption Keys is part of BigQuery From the official documentation: “If you require more control over encryption keys, BigQuery supports [customer-managed encryption keys](/bigquery/docs/customer-managed-encryption) in the [Cloud Key Management Service](/kms/docs).”
• Customer-Managed Encryption Keys is part of Cloud SQL From the official documentation: “Cloud SQL also lets you add another layer of encryption to data using [customer-managed encryption keys (CMEK)](/sql/docs/postgres/cmek).”
• Customer-Managed Encryption Keys is part of Cloud Storage From the official documentation: “[Cloud Storage](/storage/docs/introduction) provides features such as customer-managed encryption, lifecycle management, and aging and retention policies.”
• Customer-Managed Encryption Keys works with Dataflow From the official documentation: “This sample shows how to use encryption keys managed by the customer, with a Dataflow pipeline.”
• Customer-Managed Encryption Keys secures Dataflow From the official documentation: “You can either create CMEKs manually to protect your Dataflow resources or use Cloud KMS Autokey.”
• Dataflow works with Vertex AI From the official documentation: “Vertex AI models are subject to [Rate Limits and Quotas](https://cloud.google.com/vertex-ai/generative-ai/docs/quotas#view-the-quotas-by-region-and-by-model) and Dataflow automatically retries throttled requests with exponential backoff.”
• Dataflow works with VPC Service Controls From the official documentation: “To customize DNS resolution when using Dataflow with VPC Service Controls, use Cloud DNS [private zones](/dns/docs/overview#dns-private-zones) instead of using custom BIND servers.”
• Datastore depends on Bigtable From the official documentation: “Datastore is built on top of Google's NoSQL database, [Bigtable](https://docs.cloud.google.com/bigtable), and is subject to Bigtable's performance characteristics.”