Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Concept
Apache Iceberg
Topic in 3 exam domains
- Apache Iceberg · Concept · table format
- Apache Iceberg · Feature · Managed Service for Apache Spark component
Also called: Iceberg
An open-source table format for analytic datasets.
Where this appears on the exam
2 objectives in 2 of 5 domains
Related topics
This topicWorks alongsideBuilds on theseBuilds on theseApplied
Apache IcebergConcept
VPC Service ControlsServiceApplication Default CredentialsFeatureLakehouse runtime catalogFeatureDatastreamServiceCloud StorageServiceGoogle Cloud StorageServiceDataflowServicecustomer-managed encryption keysFeatureManaged Service for Apache SparkServiceBigQueryServicePub/SubServiceManaged Service for Apache AirflowServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Apache Iceberg works with Lakehouse runtime catalog From the official documentation: “Learn how to use the Lakehouse runtime catalog with Apache Spark and Apache Iceberg.”
- Apache Iceberg works with Cloud Storage From the official documentation: “This architecture uses open table formats like Iceberg to add data warehousing capabilities, such as transactions and schema evolution, to data in Cloud Storage.”
- Apache Iceberg works with Dataflow From the official documentation: “Write from Dataflow to Apache Iceberg, using the dynamic destinations feature to route incoming records to different Iceberg tables.”
- Apache Iceberg works with Datastream From the official documentation: “This section describes the destinations to which you can stream data using Datastream. Supported destinations are: - BigQuery - Cloud Storage - Apache Iceberg tables.”
- Apache Iceberg works with Google Cloud Storage From the official documentation: “- BUCKET and FOLDER: The Iceberg catalog location in Cloud Storage.”
- Apache Iceberg works with Managed Service for Apache Spark From the official documentation: “You use Apache Iceberg as the table format, Managed Service for Apache Spark for processing, and the Lakehouse runtime catalog Iceberg REST Catalog for unified metadata management.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Datastream From the official documentation: “Permissions needed for datastream to write to BigQuery.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Cloud Storage works with Datastream From the official documentation: “When you stream to Cloud Storage, Datastream organizes data by object name and timestamp.”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials works with Google Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- BigQuery works with Lakehouse runtime catalog From the official documentation: “When querying Lakehouse runtime catalog tables from BigQuery, you use a four-part naming structure, often referred to as **P.C.N.T**:”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- customer-managed encryption keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
- BigQuery works with Managed Service for Apache Spark From the official documentation: “This means you can quickly read and write BigQuery data to and from Managed Service for Apache Spark.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Storage works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow associates a Cloud Storage bucket with your environment.”
- customer-managed encryption keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
- customer-managed encryption keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
- Cloud Storage works with customer-managed encryption keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Lakehouse runtime catalog works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark supports only PySpark jobs with borderless Lakehouse Metastore.”
- VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
- BigQuery is an alternative to Managed Service for Apache Spark From the official documentation: “Run queries against your federated data using BigQuery or Managed Service for Apache Spark.”
- Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
- customer-managed encryption keys secures Managed Service for Apache Airflow From the official documentation: “This page describes how to use [Customer Managed Encryption Keys (CMEK)](/kms/docs/cmek) to protect Managed Airflow environments.”
- Dataflow is an alternative to Managed Service for Apache Spark From the official documentation: “For more complex data loads into your Cloud SQL for PostgreSQL database, consider using Dataflow or Managed Service for Apache Spark, which involves creating an ETL process.”
- VPC Service Controls secures Managed Service for Apache Airflow From the official documentation: “Managed Airflow can be selected as a secured service inside a VPC Service Controls perimeter.”
- Application Default Credentials works with Managed Service for Apache Spark From the official documentation: “To authenticate to Managed Service for Apache Spark, set up Application Default Credentials.”
- Application Default Credentials secures Managed Service for Apache Spark From the official documentation: “To authenticate to Managed Service for Apache Spark, set up Application Default Credentials.”
- Google Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Managed Service for Apache Airflow works with VPC Service Controls From the official documentation: “When used with Managed Airflow, VPC Service Controls have several [known limitations](/vpc-service-controls/docs/supported-products#composer).”
- Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
- Application Default Credentials works with Managed Service for Apache Airflow From the official documentation: “To authenticate to Managed Airflow, set up Application Default Credentials.”
- Cloud Storage works with Lakehouse runtime catalog From the official documentation: “Set up and configure Lakehouse catalog endpoints to unify data silos across AWS S3, Cloud Storage, and AlloyDB.”
- customer-managed encryption keys is part of Cloud Storage From the official documentation: “[Cloud Storage](/storage/docs/introduction) provides features such as customer-managed encryption, lifecycle management, and aging and retention policies.”
- customer-managed encryption keys secures Managed Service for Apache Spark From the official documentation: “For data encryption at rest, use the [Cloud Key Management Service](/kms/docs/key-management-service) (KMS) or [Customer Managed Encryption Keys](/managed-spark/docs/concepts/configuring-clusters/customer-managed-encryption) (CMEK).”
- Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
- Dataflow works with Datastream From the official documentation: “In addition, Datastream supports writing the change event stream into Cloud Storage, and offers streamlined integration with Dataflow templates to build custom workflows for loading data into a wide range of destinations, such as Cloud SQL and Spanner.”
How learners score on this topic
Last updated