Mapping Business Requirements to Data Architecture

Assess and Plan for Technological Evolution is the practice of mapping current and expected business goals to a data architecture (opens in a new tab) that can adopt new services without rework or downtime. The assessment weighs scalability, cost, and operational impact on Google Cloud, while the plan defines how new capabilities are added alongside existing pipelines. The goal is an architecture that scales storage and compute independently, integrates new sources, and keeps existing reports and applications working during change.

1.1How Separation of Storage and Compute Enables Evolution

Separation of storage and compute in BigQuery (opens in a new tab) is the foundation that lets a warehouse grow from terabytes to dozens of petabytes without resizing clusters. BigQuery stores data in Colossus, Google's distributed file system, in a compressed columnar format called Capacitor that is queried directly without decompression. Query execution runs on a multilevel tree based on Dremel and is connected by the Jupiter network with 1 petabit per second of bisection bandwidth. Compute is measured in slots, which are units of CPU and RAM managed by Google and allocated by a fair scheduler, with projects limited by default to 2,000 slots.

Because BigQuery allocates storage as data is consumed and scales query resources per query, growth can be absorbed by adjusting slot reservations rather than rebuilding clusters. In contrast, Amazon Redshift ties CPU, memory, and disk together in compute nodes, so growth requires classic resize over hours to days with full data copy or elastic resize in 10 to 15 minutes in read-only mode. Even RA3 nodes with managed storage cap at 64 TB per node. This difference means future growth, new workloads, and cost model changes favor separated storage and compute.

1.2Evaluating Multicloud and Advanced Analytics Services

Evaluating multicloud and advanced analytics services means checking whether a new capability removes a data silo or licensing constraint without forcing data movement. BigQuery Omni (opens in a new tab) provides compute that runs locally to storage on Amazon Web Services (AWS) or Azure and queries data stored in Amazon Simple Storage Service (Amazon S3) or Azure Blob Storage. Teams analyze external data where it rests instead of copying it first, which helps when business initiatives require cross-cloud analytics with data residency limits.

For Oracle migrations, the Oracle Advanced Analytics (opens in a new tab) option for Oracle R Enterprise and Oracle Data Mining requires separate licensing. Google Cloud replaces that pattern with BigQuery ML for in-warehouse linear regression, logistic regression, k-means clustering, and TensorFlow model imports, plus Vertex AI Workbench for development and Vertex AI Training for large-scale training and scoring. For location-aware planning, BigQuery datasets can be regional or multi-region, and resource allocation follows query type and complexity with column statistics gathered at load time and query plan and timing diagnostics exposed for tuning. Choose Omni when data must stay in another cloud, and choose BigQuery ML and Vertex AI when models must run on warehouse data without a separate mining server.

1.3Building Migration and Integration Strategies

Migration and integration strategies define an ordered path from the current warehouse to the future architecture while keeping pipelines running. The recommended initial transfer keeps the schema unchanged so existing feeds, staff training, and automated tools continue to work, and proofs of concept can run in parallel with migration. Transfer options depend on source location and bandwidth, and the choice directly affects timeline and disruption.

The initial load follows an ordered flow from the source system to BigQuery:

1. Extract: Export tables from the source platform to vendor-neutral files in staging storage, preferring Avro, ORC, or Parquet where schema is embedded with data.
2. Upload: Copy files from staging storage to a Cloud Storage (opens in a new tab) bucket, choosing the bucket location closest to the source and ideally matching the BigQuery dataset location.
3. Load: Load schema and data from Cloud Storage into BigQuery using BigQuery Data Transfer Service, Storage Write API, or batch load, with optional transformation during load.
4. Validate: Compare source and target with the Data Validation Tool, which runs aggregations such as count, sum, and average on selected columns.

Data flows from another cloud to Cloud Storage through Storage Transfer Service, from on-premises with good bandwidth through the gcloud CLI with multithreaded parallel uploads that resume after errors and encrypt in transit, and from bandwidth-constrained sites through offline Transfer Appliance. For recurring updates after the first load, scheduled transfers repeat the same path at intervals, while append-only tables partitioned by date can be backed up by copying required partitions before export to handle late updates such as 7-day returns.

1.4Choosing Processing Pipelines That Preserve Flexibility

Choosing processing pipelines that preserve flexibility means selecting fully managed services that can be swapped or scaled as transformation needs change. Data flows from Cloud Storage or Pub/Sub handover queues into the processing service and then into BigQuery, so Pub/Sub queues or Cloud Storage buckets act as the boundary between transactional systems in private environments and analytics in Google Cloud. The handover networking pattern enables ingestion into Google Cloud, and the gated egress pattern is added when analytical results must flow back to transactional systems.

Consistent tooling across environments increases operational efficiency, and long-term high-volume exchange between clouds uses Cross-Cloud Interconnect with VPN, HA VPN over Interconnect, or MACsec encryption at the connectivity layer. Sensitive data is protected before broader use with Sensitive Data Protection de-identification such as masking, encryption, and tokenization for personally identifiable information (PII).

1.5Maintaining Compatibility While Minimizing Disruption

Maintaining compatibility while minimizing disruption means evolving schema, SQL, and downstream consumers in stages rather than cutover. Facade views sit between BigQuery tables and downstream processes, so downstream dashboards first read from views over the source schema, then from a mix of source tables and new tables with nested and repeated fields or UNIONs, and finally only from evolved tables while source tables retire. Changing to nested and repeated fields aligns the model with the Dremel representation, reduces joins compared to star and snowflake schemas, and improves filtering when combined with partitioning and clustering on frequently filtered columns.

SQL compatibility is handled with GoogleSQL, which supports SQL:2011 with extensions for nested data, plus batch SQL translation for bulk script migration and interactive SQL translation for ad hoc queries. Capacity risk is controlled by reserving slots in reservations and assigning projects to reservations, separating slot quotas and billing so autonomous teams query datasets in different projects without affecting each other.

Translating business objectives into technical requirements means converting stakeholder goals, regulatory limits, and key performance indicators (KPIs) into concrete choices for data ingestion, processing, storage, and access on Google Cloud (opens in a new tab). The process starts with an inventory of current pipelines and datasets, then maps each requirement — latency, residency, availability, security, and cost — to specific services and deployment patterns. A flexible architecture keeps the data storage layer separate from the data computation layer while colocating them for performance.

2.1Inventory Business Needs and Technical Dependencies

Inventorying the current state turns vague business goals into measurable technical requirements for migration and growth. The assessment phase determines requirements and dependencies for batch data pipelines in an ordered workflow:

1. Build inventory: Catalog data infrastructure, pipelines scheduled for migration, and datasets read by those pipelines.
2. Catalog properties: Record storage platforms, computation platforms, versions, custom configurations, cluster sizes, and dependencies for each pipeline.
3. Train teams: Educate teams on Google Cloud services and deployment practices.
4. Build proof of concept: Validate data transfer, configuration, and output correctness on Google Cloud.
5. Calculate total cost of ownership (TCO): Compare time, budget, people, and bandwidth for online transfer versus future operating cost.
6. Choose first workloads: Prioritize movable datasets after filtering for regulatory, security, and transformation needs.

Data flows from discovery tools such as Data Catalog groupings into validated migration groups, and the workflow depends on stakeholder input to identify what datasets must move, what datasets can move, and where each movable dataset will land. For each storage layer, the team must verify that the product exists in the target environment or has a compatible replacement, and must practice the transfer process because Cloud Storage uses data migration services while databases may use built-in migration tools.

2.2Separate Storage from Computation for Flexibility and Efficiency

The data platform divides into a data storage layer where data is saved and a data computation layer where processing runs on top of storage. The storage layer manages bytes in files such as Hadoop Distributed File System (HDFS) or Cloud Storage, or manages records through a domain-specific language (DSL) in a database system. The computation layer loads data from the storage layer, processes the data, and then saves results to a target system that can be the source storage layer.

Separation improves flexibility to change computation engines and reduces cost because the computation layer does not run continuously. At the same time, colocation controls performance and cost: data should be stored in the same zone that it is processed in to reduce communication latency, outbound transfer cost, and I/O operations. In practice, this means deploying storage and computation on separate platforms in the same zone and region, for example moving storage from HDFS to Cloud Storage while using a Managed Service for Apache Spark (opens in a new tab) cluster for computation.

2.3Map Storage and Ingestion Needs to GCP Services

Storage and ingestion choices translate durability, residency, access frequency, and sharing needs into specific Google Cloud services. Cloud Storage (opens in a new tab) provides object storage replicated across locations, BigQuery storage supports analytics datasets, and Cloud SQL, Spanner, and Bigtable support transactional, globally distributed, and wide-column workloads. Ingestion often lands first in Cloud Storage or Pub/Sub, then flows to processing and to long-term storage, so access policies and residency must be maintained after migration.

Location and replication define the core tradeoff:

• Data in regional buckets and regional Persistent Disk volumes is replicated synchronously across zones in one region for low-latency resilience to zone failure.
• Data in dual-region or multi-region buckets is stored redundantly in at least two geographic locations, where metadata is written synchronously and data is replicated asynchronously.
• For dual-region buckets, turbo replication provides a recovery point objective (RPO) of 15 minutes across the region pair.

Shared file needs add another option: a Filestore regional instance replicates data synchronously across three zones in a region and can be mounted on multiple virtual machines (VMs) for shared configuration, tools, and logs. For block storage, Google Cloud Hyperdisk allows separate, dynamic provisioning of IOPS and throughput, while regional Persistent Disk provides synchronous replication across two zones compared to zonal disks that lack that protection.

2.4Map Processing Needs to Compute and Access Patterns

Processing requirements for scale, control, and operational effort map to different compute models on Google Cloud. Dataflow handles streaming and batch integration pipelines, Managed Service for Apache Spark handles Hadoop and Spark workloads, and BigQuery combines storage with serverless SQL computation. Custom workloads can run on Google Kubernetes Engine (GKE) (opens in a new tab) or Compute Engine, while Gemini Enterprise Agent Platform (opens in a new tab) or Colaboratory support notebook-based machine learning (ML) development.

The choice between VMs, containers, and serverless involves a tradeoff between configuration flexibility and management effort:

Region selection further constrains the mapping because availability of services and machine types, end-user latency, resource cost, cross-regional transfer cost, regulatory requirements, and sustainability goals may conflict. The most cost-efficient region may not have the lowest carbon footprint or the required machine types, so the architecture must weigh latency and compliance against cost.

2.5Enforce Security, Residency, and Governance Constraints

Security and compliance constraints translate into identity, encryption, and placement controls that bound where data can flow. Identity and Access Management (IAM) is the recommended control for Cloud Storage buckets and objects, granting permissions at bucket and project levels, while access control lists (ACLs) remain for narrow object-level exceptions. Organization design reinforces this boundary: projects group into folders under an organization, roles are set at the project level, and production data can live in a separate project where only a privileged service account grants access from application projects.

Sensitive data and residency add service-specific controls. Data loaded through Cloud Storage may contain sensitive fields, so Sensitive Data Protection discovers, classifies, and de-identifies that data before broader use. Pub/Sub encrypts messages at rest and in transit by default with Google-owned keys and supports customer-managed encryption keys (CMEKs) for application-layer encryption. When residency applies, Cloud Storage stores or replicates data within the specified region, Agent Platform stores source data for retrieval-augmented generation (RAG) in the selected location, and Pub/Sub message storage policies restrict where message data is stored.

2.6Design for Reliability Targets and Cost KPIs

Reliability targets translate business criticality into RTO, meaning time until service resumes, and RPO, meaning how much data loss is tolerable, for zone versus region outages. Tier 1 customer-facing systems may require zero RTO and zero RPO for both scopes, Tier 2 regional systems may allow 15 minutes for a zone outage and 1 hour for a region outage, and Tier 3 departmental systems may allow 1 hour and 12 hours respectively. Managed multi-regional services such as Spanner and Cloud Storage support active-active topologies that reduce manual failover, while zonal or regional resources require defined replication, snapshots stored in multi-region locations, and regular DR testing.

Aggregate availability shows why fewer tiers and parallel paths matter for meeting those targets. Aggregate availability for chained tiers equals tier1_availability x tier2_availability x tierN_availability, so four tiers at 99.9% yield only about 99.6%, and two chained services at 99% yield 98%. Parallelized services use 1 - (1 - A) x (1 - B), so two parallel 99% services yield 99.99% because each runs independently. Increasing nines generally increases operational complexity and cost: 99.9% allows about 86 seconds of downtime per day, while 99% allows 864 seconds, or ten times more.

Cost KPIs translate into matching resource shape to access frequency and demand. Cloud Storage cost depends on choosing the appropriate storage class for retention and access frequency plus lifecycle rules, while Cloud Run cost depends on starting with default CPU and memory limits, tuning only for measured performance, and using committed use discounts when requirements are predictable. Agent Platform cost depends directly on the invoked model and the number of input and output tokens per request, so context caching that minimizes token count lowers operating cost without changing functionality.

Design for scalability and future growth means architecting data solutions that absorb projected increases in data volume, velocity, and variety without redesign. On Google Cloud (opens in a new tab), this is achieved by selecting scaling models that separate storage from compute and by building modular components that can be independently evolved as business requirements change.

3.1BigQuery Separation of Storage and Compute

BigQuery is an enterprise data warehouse that treats compute, memory, and storage as separate resources, so the warehouse can grow without operator intervention. Compute capacity is defined by slots, a unit of computational capacity required to execute queries, and Google manages the entire infrastructure that a slot encapsulates. Teams scale performance by choosing the proper slot quantity, including yearly or three-year commitments through BigQuery reservations, while storage scales independently in the dataset location.

This model contrasts directly with node-coupled architectures where CPU, memory, and disk scale together and growth requires resizing.

Because storage and compute do not contend, historic data can accumulate to terabytes to petabytes while query throughput is tuned separately. For disaster recovery and locality, a BigQuery dataset is created as regional, such as Iowa us-central1, or multi-region, such as US or EU, and data loaded into BigQuery is stored synchronously in two zones within the specified region.

3.2Modular Processing Components That Evolve Independently

Modular processing components are pipeline stages for extract, transform, and load that can be changed or scaled without rebuilding the whole architecture. The extract, load, and transform (ELT) pattern loads data into a BigQuery staging table with an extract and load tool, then transforms the staged data with SQL queries that write into the final production table. Change data capture (CDC) tracks changes from source systems over time so the warehouse can collate history without reprocessing full sources.

When transformation must happen before loading, three modular compute options can be evolved independently:

• Dataflow clones the Java Database Connectivity (JDBC) to BigQuery template and adds Apache Beam transforms to the pipeline.
• Cloud Data Fusion transforms data using CDAP plugins within a managed integration environment.
• Managed Service for Apache Spark transforms data using Spark SQL or custom code in Scala, Java, Python, or R.

Decoupling is extended to storage and serving by Cloud Storage, a low-cost, no-limit object store that holds diverse data types in open formats such as Apache Parquet for access by multiple engines. AlloyDB for PostgreSQL (opens in a new tab) supports hybrid transactional and analytical processing for live inventory or customer profiles, and BigQuery AlloyDB federation sends a query from BigQuery directly to live transactional data in AlloyDB. This federation eliminates the latency and overhead associated with CDC pipelines, while Borderless Lakehouse connects to an external catalog such as Databricks Unity Catalog, verifies credentials in Secret Manager (opens in a new tab), and then establishes a direct connection to lakehouse metadata to query data where it resides.

3.3Handling Growth in Volume, Velocity, and Variety

Handling growth in volume, velocity, and variety requires storage that expands without limits, ingestion that absorbs spikes, and placement controls that preserve availability. Cloud Storage decouples high-throughput data such as clickstream events from specific compute services, and buckets are created in one of three location types. Data in regional buckets is replicated synchronously across multiple zones within a region, while dual-region or multi-region buckets replicate asynchronously across regions for higher availability.

Velocity is managed where data flows from publishers to subscribers through Pub/Sub. To manage transient spikes in message traffic, flow control is configured in the publisher settings to limit the rate of publish requests, and retry-request variables are adjusted to handle failed publishes. For variety and portability, the same Cloud Storage objects and BigQuery tables serve batch Spark jobs, federated queries, and a unified customer profile in BigQuery that consolidates distributed outputs into denormalized types for analysis. When multi-region storage is unavailable or a second region is required, the BigQuery Data Transfer Service copies a dataset to a different region, and Cloud Scheduler or Managed Airflow periodically schedules BigQuery exports to Cloud Storage in formats such as Avro with SNAPPY compression or JSON with GZIP compression. The open lakehouse connection (opens in a new tab) extends this portability by querying external data where it resides instead of copying it first.

Key Facts

• BigQuery separates storage in Colossus and Capacitor from compute in Dremel slots managed by Google, with a default limit of 2,000 slots per project, so storage and query capacity scale independently.
• BigQuery Omni queries data stored in Amazon S3 or Azure Blob Storage where it rests, while BigQuery ML and Vertex AI replace separate Oracle mining servers for in-warehouse modeling.
• Initial warehouse migration follows Extract to vendor-neutral files, Upload to Cloud Storage, Load to BigQuery, and Validate with the Data Validation Tool.
• Regional storage replicates synchronously across zones for high availability, while dual-region and multi-region storage replicate asynchronously, with turbo replication providing a 15-minute RPO for dual-region buckets.
• Aggregate availability for chained services multiplies tier availabilities, while parallel services use 1 - (1 - A) x (1 - B), so fewer tiers and parallel paths improve reliability targets.