A media and analytics organization needs to ingest large volumes (over 150 TiB) of raw video assets and recurring batch exports stored across on-premises POSIX file systems into Google Cloud Storage. The organization has an active high-bandwidth Cloud Interconnect connection and requires a solution that: Automates recurring batch synchronization from the on-premises file systems to regional Cloud Storage buckets Scales transfer throughput across available network capacity while minimizing operational maintenance Maintains data integrity by automatically validating checksums between source files and destination objects Which batch transfer solution should the organization implement?

Deploy Storage Transfer Service on-premises agents organized in an agent pool to transfer data directly to Cloud Storage: What Storage Transfer Service with On-Premises Agents Is Storage Transfer Service is a fully managed, scalable Google Cloud service designed for large-scale data movement between on-premises storage systems, other cloud providers, and Cloud Storage. When transferring from local POSIX file systems, it utilizes lightweight on-premises transfer agents packaged as containerized services deployed within your local infrastructure. How It Meets the Requirements Automated recurring batch synchronization: Storage Transfer Service supports scheduled transfer jobs that automatically scan directories, detect modified or newly created files, and synchronize them on a recurring schedule without manual intervention. Scalable throughput over Cloud Interconnect: By deploying multiple agents inside an agent pool, the service automatically balances workloads across instances, fully utilizing the available bandwidth of the dedicated Cloud Interconnect connection. End-to-end data integrity: Storage Transfer Service performs automatic checksum verification (using MD5 or CRC32C) on every transferred file against the destination Cloud Storage object to ensure the data was not corrupted during transit. Key Advantages and Capabilities Dynamic load balancing: Agent pools automatically distribute work units among available agent containers and scale dynamically. Resilience and fault tolerance: Automatically handles network hiccups with built-in retries and exponential backoff. Bandwidth throttling: Enables administrators to set bandwidth limits to prevent data transfers from saturating network pipes during peak business hours. Cost-effective automation: Eliminates the maintenance overhead of managing custom transfer servers, logging systems, or sync scripts. Why This Is Optimal Storage Transfer Service is purpose-built for bulk and batch file synchronization from on-premises file systems to Cloud Storage. When sufficient network bandwidth is present, software-driven parallel agent transfers completely bypass the physical logistical turnaround of hardware appliances while ensuring enterprise-grade verification and orchestration.

Evaluating Batch Transfer Storage Transfer Service | Professional Cloud Data Engineer

AdvancedProfessional Cloud Data Engineer

A financial enterprise is designing a streaming ingestion pipeline to import sensitive customer transactions from an on-premises datacenter into a BigQuery data warehouse. The architecture must satisfy several strict security, encryption, and auditability requirements: Sensitive fields must be protected before leaving the on-premises network and remain encrypted at the column level within BigQuery, requiring dual-access control where analysts need explicit permissions to both the table and the decryption key to view cleartext. All ingestion components (Pub/Sub topics, Dataflow jobs, and BigQuery datasets) must be encrypted at rest using Customer-Managed Encryption Keys (CMEK) backed by Cloud HSM in the matching geographical region. Automated deployment processes and engineers must interact with cloud resources without downloading or storing long-lived service account JSON keys. Ingestion operations and CI/CD pipeline actions must maintain an auditable trail that can be correlated with Cloud Audit Logs. Which architecture and implementation strategy satisfies these requirements?

Data acquisition and import

AdvancedProfessional Cloud Data Engineer

An enterprise architecture team is designing a real-time streaming ingestion pipeline on Google Cloud using Cloud Pub/Sub to process events across multiple microservices. The Pub/Sub topic enforces an Apache Avro schema that evolves over time as upstream services add optional fields, creating multiple schema revisions. The downstream subscriber applications maintain their own reader schema and must reliably deserialize incoming binary-encoded messages that may have been published under different schema revisions. How should the subscriber application be architected to handle messages across evolving schema revisions?

Data acquisition and import

AdvancedProfessional Cloud Data Engineer

Your organization stores customer transaction data in a centralized BigQuery table named sales.transactions. You need to securely share this table with three distinct user groups while enforcing the principle of least privilege: Finance Auditors: Must be able to query all rows and view sensitive payment identifiers (such as credit card numbers) in cleartext. Business Analysts: Must be able to query all rows, but sensitive payment identifiers must be dynamically masked (e.g., using a masking rule or hash) so that existing queries continue to run without exposing cleartext data. Regional Sales Managers: Must be restricted to querying only rows where region = 'EMEA', and payment identifiers must be masked. How should you configure access controls and data policies in Google Cloud to achieve these requirements?

Defining rules to share data

AdvancedProfessional Cloud Data Engineer

An enterprise is migrating a mission-critical transactional database from on-premises to Google Cloud. A thorough analysis of the database workload patterns and performance requirements reveals the following characteristics: Workload profile: Read-intensive transaction processing with frequent complex analytical queries, aggregations, and high concurrency. Throughput and latency targets: Must sustain over 400,000 read transactions per second (TPS) with average query latency below 1 millisecond. Hardware and scaling constraints: Workload requires scaling compute to 64 vCPUs; traditional open-source PostgreSQL instances hit concurrency bottlenecks beyond 16 vCPUs. Compatibility: Full compatibility with PostgreSQL schemas, extensions, and standard SQL query syntax is required without redesigning data models for distributed horizontal sharding. Which Google Cloud database service best satisfies these workload characteristics?

Analyzing data access patterns

AdvancedProfessional Cloud Data Engineer

A financial services organization is building a decentralized data mesh architecture on Google Cloud. Multiple domain teams need to expose curated analytical data products to downstream business analysts across the enterprise. The enterprise architecture board has defined the following technical and regulatory constraints: Downstream business analysts must query aggregated business metrics and sanitized virtual tables without having direct read permissions to the underlying sensitive base datasets. The security and compliance teams require minimal administrative overhead and must avoid managing individual view authorizations as dozens of new analytical views are published. Data domain infrastructure deployments must prevent configuration drift using infrastructure as code (IaC) templates. Which technical requirement and access pattern should the data engineering team implement to satisfy these business objectives?

Mapping current and future business requirements to the architecture

IntermediateProfessional Cloud Architect

Flex-start (Dynamic Workload Scheduler)

Using AI Hypercomputer (e.g., using AI Hypercomputer, Cloud Run functions, and Vertex AI for ML/AI workloads; integrating GPUs and TPUs in ML model training and serving; optimizing for different consumption models; and running large-scale AI model training

Professional Cloud Data Engineer

Evaluating Batch Transfer Storage Transfer Service

Related questions

Flex-start (Dynamic Workload Scheduler)

Professional Cloud Data Engineer

Evaluating Batch Transfer Storage Transfer Service

What Storage Transfer Service with On-Premises Agents Is

How It Meets the Requirements

Key Advantages and Capabilities

Why This Is Optimal

Related questions

Flex-start (Dynamic Workload Scheduler)

What Storage Transfer Service with On-Premises Agents Is

How It Meets the Requirements

Key Advantages and Capabilities

Why This Is Optimal