A retail enterprise is architecting an enterprise data platform on Google Cloud to ingest both real-time clickstream events and daily batch transactional files from acquired entities. The platform design must satisfy the following technical requirements: Ingestion and Transformation: Support continuous streaming and scheduled batch processing with schema validation, writing invalid records to error tables for forensic analysis without halting the pipeline. Orchestration: Coordinate multi-stage operational extract, transform, and load (ETL) workflows with complex dependencies and automated scheduling. Quality and Governance: Unify distributed assets into logical data lakes and automatically validate data quality rules without moving data into a single consolidated system. Lifecycle Management: Automatically enforce retention policies by purging raw staging records after 90 days and archiving curated analytical tables. Which architectural approach satisfies all of these requirements while adhering to Google Cloud best practices?

Ingest real-time and batch streams using Dataflow with dead-letter branching; orchestrate batch workflows with Cloud Composer; organize distributed assets using Dataplex data lakes and zones with auto data quality enabled; configure partition and table expiration in BigQuery alongside Cloud Storage lifecycle rules.: ### What this solution is This architecture combines **Dataflow** for unified stream/batch data processing, **Cloud Composer** for workflow orchestration, **Dataplex** for decentralized data governance and automated quality validation, and native **BigQuery and Cloud Storage lifecycle management** policies for compliance and data retention. ### How it meets all requirements * **Ingestion and Transformation**: **Dataflow** processes both unbounded streaming clickstream records and daily batch files. Implementing a **dead-letter / branching pattern** allows Dataflow to validate data contracts and route malformed records into dedicated BigQuery failure tables for forensic debugging while valid records flow cleanly to curated layers. * **Orchestration**: **Cloud Composer** (managed Apache Airflow) is the standard Google Cloud tool for orchestrating complex operational workflows with multi-step dependencies, cross-service task triggers, and CI/CD-driven ETL pipelines. * **Quality and Governance**: **Dataplex** creates logical lakes and zones across distributed Cloud Storage buckets and BigQuery datasets, utilizing **auto data quality** scans to enforce standardized validation rules in place without data consolidation. * **Lifecycle Management**: **BigQuery default partition and table expiration** settings combined with **Cloud Storage Object Lifecycle Management** rules automate data deletion and archival after 90 days. ### Key advantages * **Unified stream and batch semantics** using Apache Beam in Dataflow. * **Comprehensive governance** without centralizing physical storage through Dataplex Universal Catalog. * **Automated compliance** by delegating deletion and tiering directly to native storage lifecycle mechanisms. * **Resilience and auditability** by isolating failed records into dedicated forensic tables. ### Why this is optimal This approach aligns with Google Cloud reference architectures for enterprise data platforms, ensuring scalability, low operational overhead, robust data governance, and automated lifecycle compliance.