Managed Service for Apache Airflow

Related topics

• Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
• BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
• Customer-Managed Encryption Keys secures Managed Service for Apache Airflow From the official documentation: “This page describes how to use [Customer Managed Encryption Keys (CMEK)](/kms/docs/cmek) to protect Managed Airflow environments.”
• VPC Service Controls secures Managed Service for Apache Airflow From the official documentation: “Managed Airflow can be selected as a secured service inside a VPC Service Controls perimeter.”
• Cloud Data Fusion works with Managed Service for Apache Airflow From the official documentation: “Recommended: You can also use Composer to orchestrate pipelines in Cloud Data Fusion.”
• Cloud Monitoring works with Managed Service for Apache Airflow From the official documentation: “To learn about Cloud Logging and Cloud Monitoring for your Managed Airflow environment, see [Monitor environments](/composer/docs/composer-2/monitor-environments).”
• Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “When you upload your DAG file to the `dags/` folder in Cloud Storage, Managed Airflow parses the file.”
• GKE works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow runs your workloads on GKE.”
• Logs Explorer works with Managed Service for Apache Airflow From the official documentation: “You can query these logs in the Logs Explorer and check which Managed Service for Apache Airflow API methods generate Cloud Audit Logs.”
• Managed Service for Apache Airflow depends on Cloud SQL From the official documentation: “Managed Airflow uses Cloud SQL to run Airflow database.”
• Managed Service for Apache Airflow depends on Google Cloud Storage From the official documentation: “Managed Airflow uses a [Cloud Storage](/storage) bucket to store DAGs of your Managed Airflow environment.”
• Managed Service for Apache Airflow works with Managed Service for Apache Spark From the official documentation: “This tutorial shows how to use Managed Service for Apache Airflow to create an [Apache Airflow](http://airflow.apache.org) DAG (Directed Acyclic Graph) that runs an Apache Hadoop wordcount job on a Managed Service for Apache Spark cluster.”
• Cloud Logging works with Managed Service for Apache Airflow From the official documentation: “To learn about Cloud Logging and Cloud Monitoring for your Managed Airflow environment, see [Monitor environments](/composer/docs/composer-2/monitor-environments).”
• Cloud SQL works with Managed Service for Apache Airflow From the official documentation: “For Private IP environments, Managed Airflow assigns only private IP ([RFC 1918](https://www.ietf.org/rfc/rfc1918.txt)) addresses to the managed Google Kubernetes Engine and Cloud SQL VMs in your environment.”
• Google Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “When you create an environment, Managed Airflow creates a [Cloud Storage](/storage) bucket and associates the bucket with your environment.”
• Managed Service for Apache Airflow depends on BigQuery From the official documentation: “This happens because Google Kubernetes Engine depends on BigQuery and Managed Service for Apache Airflow inherits this dependency from the Kubernetes Engine API.”
• Managed Service for Apache Airflow depends on GKE From the official documentation: “Managed Airflow runs your workloads on GKE.”
• Managed Service for Apache Airflow works with VPC Service Controls From the official documentation: “By configuring your environment with VPC Service Controls, you can keep sensitive data private while taking advantage of the fully-managed workflow orchestration capabilities of Managed Airflow.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
• BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
• BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
• Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
• Cloud Run depends on Pub/Sub From the official documentation: “Cloud Run function triggered by Pub/Sub to fetch HYPR MFA logs and write to GCS.”
• Customer-Managed Encryption Keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
• Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
• BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• Dataflow works with Spanner From the official documentation: “Use Dataflow to load data into Spanner.”
• VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
• Cloud Data Fusion works with Managed Service for Apache Spark From the official documentation: “Deployed Cloud Data Fusion pipelines run on Managed Service for Apache Spark clusters by default.”
• Cloud Logging works with GKE From the official documentation: “Enable Cloud Logging on GKE Clusters”
• Cloud Monitoring works with GKE From the official documentation: “Enable Cloud Monitoring on GKE Clusters”
• Customer-Managed Encryption Keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
• BigQuery works with Bigtable From the official documentation: “BigQuery lets you query your Bigtable data from BigQuery.”
• BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
• BigQuery works with Spanner From the official documentation: “Federated analytics seamlessly combine data from Spanner and other sources in BigQuery.”
• Cloud Run works with Cloud SQL From the official documentation: “Connecting from Cloud Run (fully managed) to Cloud SQL”
• Cloud SQL works with Compute Engine From the official documentation: “Cloud SQL uses Compute Engine virtual machines (VMs) with persistent storage disks.”
• Compute Engine is an alternative to GKE From the official documentation: “You can also deploy a third-party model to either Vertex AI Model Garden or self-host on GKE or Compute Engine.”
• BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
• Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud SQL works with GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
• Cloud Storage works with Customer-Managed Encryption Keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
• Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Cloud Logging works with Cloud Monitoring From the official documentation: “You can store and view query-response activity logs in Cloud Logging and set up logs-based monitoring by using Cloud Monitoring.”
• Cloud Logging works with Cloud SQL From the official documentation: “Database Migration Service and Cloud SQL use Cloud Logging.”
• Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud Run works with Google Cloud Storage From the official documentation: “The Cloud Run function needs a service account with permissions to write to GCS bucket.”
• Cloud Storage works with Compute Engine From the official documentation: “For example, you can create a service account that your Compute Engine instances use to access objects stored in Cloud Storage buckets.”
• Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
• Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
• Customer-Managed Encryption Keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
• Customer-Managed Encryption Keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
• BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”
• BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Bigtable works with Dataflow From the official documentation: “The Dataflow job then writes the transformed data to your Bigtable table.”
• Cloud Logging works with Compute Engine From the official documentation: “Compute Engine sends you Cloud Logging events for scheduled, started, or completed maintenance.”
• Cloud Monitoring works with Spanner From the official documentation: “Monitor Spanner metrics using Cloud Monitoring.”