A data analytics team is preparing to ingest historical transaction data stored in Cloud Storage into BigQuery. The team requires an open-source, columnar storage format that is natively supported for automated data loading using the BigQuery Data Transfer Service. Which file format should the team use?

Apache Parquet: ### **What Apache Parquet is:** **Apache Parquet** is an open-source, columnar storage file format designed for efficient data storage and retrieval in big data analytics ecosystems. Unlike row-oriented formats that store entire records sequentially, Parquet organizes data by columns, allowing analytics engines to read only the specific fields necessary to evaluate a query. ### **How it meets the requirements:** - **Native BigQuery ingestion (✓):** BigQuery Data Transfer Service natively supports loading data stored in Cloud Storage when formatted as **Apache Parquet**, alongside Avro, ORC, CSV, and newline-delimited JSON. - **Columnar data structure (✓):** Parquet provides a columnar layout that aligns directly with BigQuery's internal columnar architecture (Capacitor), minimizing conversion overhead during ingestion. - **High performance and compression (✓):** The format includes embedded schema metadata and efficient column-level compression algorithms (such as Snappy or GZIP), optimizing both Cloud Storage footprint and transfer speeds. ### **Key advantages:** - **Column projection:** Analytical queries scan and deserialize only the requested columns, drastically reducing I/O costs and query execution times. - **Self-describing schema:** Parquet files store the schema and data types within the file footer metadata, eliminating schema guessing or mismatches during ingestion. - **Efficient encoding:** Supports advanced encoding schemes such as dictionary encoding, run-length encoding (RLE), and bit packing. - **Broad ecosystem interoperability:** Compatible across GCP analytics services (BigQuery, Dataproc, Dataflow) and third-party big data tools. ### **Why this is optimal:** Parquet is the standard choice for analytical data lakes on Google Cloud because it combines broad ecosystem compatibility, native BigQuery loading support, compact storage size, and high-performance read efficiency for analytics workloads.