Preparing data for AI and ML

Preparing data for feature engineering, training and serving machine learning models (e.g., BigQueryML)

Implement Data Versioning and Reproducibility for ML Experiments

Metadata Management and Lineage Tracking
Reproducible machine learning experiments require tracking where data comes from and how it changes over time. Dataplex Universal Catalog is a managed metadata service that lets practitioners search, tag, and track dataset lineage, which defines the origin and historical processing of data assets. Vertex ML Metadata works alongside the catalog to record metadata associated with developer workflows. It links parameters and metrics to specific model versions stored in the Vertex AI Model Registry. When you register your pipeline artifacts automatically, you create end-to-end traceability that supports both auditing compliance and debugging model iterations.

Freeze Training Data with Table Snapshots
Machine learning models need unchanging datasets to produce reproducible results. When raw data passes validation, create a BigQuery table snapshot to freeze the training inputs. This snapshot captures the data at a specific point in time and cannot be modified afterward. Using regularly updated tables instead of snapshots creates a common problem: as source data changes, tuning, training, and evaluation results shift even when the model code stays the same. Static snapshots prevent these unexpected modifications and guarantee consistent data lineage across the entire ML lifecycle.

Prevent Training-Serving Skew
Training-serving skew describes performance degradation that occurs when production data differs from the data used to train the model. The Vertex AI Feature Store works with BigQuery feature tables to serve both historical and real-time feature values to models in production. To avoid data leakage, incorporate timestamp columns in your feature tables and use point-of-time lookup functions like ML.FEATURES_AT_TIME. This ensures models train on features exactly as they existed at a specific timestamp, preserving experiment integrity by preventing future information from accidentally entering training data.

Monitor Models in Production
BigQuery ML includes built-in monitoring functions that verify model reliability over time. The ML.VALIDATE_DATA_SKEW function detects data skew, which occurs when the distribution of training feature values differs significantly from production serving data. The ML.VALIDATE_DATA_DRIFT function identifies data drift, defined as significant changes in feature data distributions over time. These automated checks let teams proactively identify anomalies and trigger model retraining before performance degrades in production.

Optimize Data Storage and Partitioning for ML Workloads

Choose Efficient Storage Formats
The choice of storage format directly impacts cost and performance for iterative ML tasks. Parquet and Avro are columnar formats that provide compression to reduce storage costs and schema evolution to allow data structures to change without breaking pipelines. These formats suit the large-scale, read-heavy operations common in machine learning. When staging intermediate data in Cloud Storage, using these optimized formats speeds up data movement and processing jobs run by services like Dataflow or Dataproc.

Partition and Cluster BigQuery Tables
BigQuery partitioning splits tables by a date, timestamp, or integer column, allowing the service to prune unnecessary data during queries. When you partition a table, queries scan only the relevant partitions rather than scanning the entire dataset. This dramatically reduces costs and improves performance for training jobs that repeatedly scan large datasets. Clustering goes further by sorting rows within each partition based on specific feature columns. This optimization makes joins and filtering operations in feature engineering faster because related data physically sits together on storage.

Create Immutable Snapshots for Reproducibility
Before running model training or prediction jobs, create immutable BigQuery table snapshots of your prepared datasets. These snapshots provide a reliable point-in-time reference that remains unchanged regardless of what happens to the live data. When preparing data for prediction, always reference these snapshots rather than live, updating tables to prevent unexpected changes from affecting your results.

Connect Pipelines to Optimized Storage
Managed services like Dataflow (for Apache Beam pipelines) and Dataproc (for Spark jobs) efficiently read from and write to optimized storage in Cloud Storage and BigQuery. For batch ML workloads, BigQuery's FILE_LOADS method offers a cost-effective approach. For real-time serving needs, the Storage Write API provides high-performance, exactly-once streaming ingestion directly into BigQuery tables.

Design and Execute Data Transformation Pipelines for Feature Creation

Identify and Ingest Data Sources
Machine learning pipelines start by identifying appropriate data sources based on your requirements. Google Cloud offers structured databases including Cloud SQL, Spanner, AlloyDB, and BigQuery. For data lakes, use Cloud Storage with BigLake. For real-time applications, streaming platforms such as Pub/Sub, Dataflow, or Apache Kafka handle continuous data flow. Ingestion tools include Dataflow for both batch and stream processing with ML integration, Cloud Run functions for event-driven ingestion triggered by data source changes, and BigQuery for classical tabular data with frequent access patterns.

Transform Data into Features
Transformation tools like Dataflow and Dataform automate feature scaling, encode categorical variables, and create new features in batch, streaming, or real-time modes. Vertex AI Feature Store centralizes feature management for reuse across different models and projects, which is essential for maintaining consistency. BigQuery ML provides automatic preprocessing during model training. For numeric columns, standardization centers data at zero. Categorical data receives one-hot encoding. When values are missing, BigQuery ML replaces numeric NULLs with the mean and maps categorical NULLs to an additional category.

Orchestrate Pipeline Workflows
ML pipelines require orchestration to manage complex workflows with multiple steps and dependencies. Vertex AI Pipelines structures pipelines as directed acyclic graphs (DAGs) of containerized tasks, while Cloud Composer provides managed Apache Airflow for more complex operational tasks with heavy ETL requirements. Dataplex Universal Catalog helps ensure data quality and standardization by discovering and managing data assets across your organization. This lets you standardize data definitions and maintain consistency without consolidating datasets in one location.

Enable Reusability and MLOps
A key principle in feature engineering is maximizing reusability. By reusing datasets and features across multiple ML projects, organizations accelerate model development, reduce infrastructure costs, and avoid redundant data engineering efforts. Data products publish and become discoverable through Dataplex Universal Catalog and BigQuery sharing. For ongoing ML development, implement MLOps practices that automate the model-development lifecycle. This includes managed pipeline orchestration, version control for code and models, and continuous monitoring to ensure reliability and performance in production.

Preparing unstructured data for embeddings and retrieval-augmented generation (RAG)

Ingestion, Extraction, and Cleaning of Unstructured Formats

Event-Driven Ingestion Pipeline Architecture
An unstructured data ingestion pipeline automates the ingestion, transformation, and storage of raw source files for retrieval-augmented generation (RAG) systems. The pipeline uses an event-driven design that processes source documents through extraction, intermediate staging, and datastore indexing. Each component operates asynchronously to decouple data producers from downstream consumers:

1. Data ingestion: External applications, databases, or SaaS platforms upload raw files directly into a Cloud Storage bucket.
2. Event notification: Cloud Storage automatically publishes an object creation notification to a Cloud Pub/Sub (Pub/Sub) topic.
3. Metadata extraction: Cloud Run functions consume the Pub/Sub message, extract raw document content, and write structured JSON Lines (JSONL) metadata files to a secondary Cloud Storage bucket.
4. Datastore ingestion: A secondary Pub/Sub message triggers Google Agentspace to ingest the JSONL files, parse the content, split text into chunks, and generate vector embeddings automatically.

This decoupled architecture ensures that high-volume document uploads do not overwhelm downstream processing layers. The system processes records asynchronously while maintaining an immutable audit trail of source documents in Cloud Storage. Automatic embedding generation then prepares the extracted text for downstream vector search without requiring manual model hosting.

Document Extraction and Connector Management
Document extraction extracts machine-readable text and structural elements from complex unstructured formats before embedding generation occurs. Specialized data connectors allow the pipeline to ingest documents directly from enterprise sources, including Google Drive, Slack, Jira, and SharePoint. Within the pipeline, Document AI uses optical character recognition (OCR) and layout parsers to preserve the reading order, tables, and formatting of complex PDFs and scanned images. Pipeline administrators control ingestion costs and prevent API quota exhaustion by setting the max_embedding_requests_per_min parameter on Cloud Storage imports. Furthermore, built-in data deduplication skips unchanged files across recurring synchronization jobs to avoid redundant compute and storage expenses.

Document Chunking Strategies and Metadata Enrichment

Text Chunking Strategies and Selection Criteria
Document chunking is the process of partitioning large, continuous text bodies into discrete segments called chunks to optimize semantic retrieval. Selecting the proper chunking technique determines whether a retrieval-augmented generation pipeline preserves contextual meaning or fragments critical concepts. Engineering teams choose between different splitting algorithms based on the document layout and retrieval precision requirements:

Setting appropriate chunk size and chunk overlap parameters balances context retention with retrieval precision. Chunks that are configured too small lack sufficient surrounding context for large language models to generate accurate answers. Conversely, oversized chunks dilute search precision by matching broad, irrelevant text segments against targeted queries. Proper chunking parameters can be configured during data store creation or adjusted directly via administrative APIs.

Metadata Enrichment for Filtered Retrieval
Metadata enrichment attaches structured attributes to unstructured document chunks to improve search precision and filtering capabilities. Enriched properties typically include document titles, author names, creation timestamps, and section hierarchy headers. Downstream vector databases, such as Cloud Spanner (Spanner) and BigQuery, store these structured metadata fields directly alongside the high-dimensional vector embeddings. During retrieval, search algorithms combine semantic similarity scores with metadata filters, restricting query results to specific authors, dates, or classifications. This multi-layered retrieval strategy prevents generative language models from receiving outdated or irrelevant grounding context.

Embedding Generation and Vertex AI Vector Search Indexing

Vector Embedding Generation and Storage Engines
Vector embedding generation transforms text chunks into high-dimensional numerical arrays that capture semantic meaning. Managed models from the Vertex AI Embedding API, such as text-embedding-004 and text-multilingual-embedding-002, produce consistent dimensional widths across batch and streaming workloads. Alternatively, relational databases like Cloud SQL generate vector embeddings directly within the database engine using the google_ml_integration extension. Organizations store and index these vectors across different Google Cloud datastores based on their query patterns and infrastructure scale. For instance, Cloud SQL employs the pgvector extension to store vector data types, while BigQuery provides native vector search indexes over high-dimensional columns using inverted file (IVF) index structures and Cosine distance metrics.

Retrieval Strategies and Index Performance
Retrieval strategies determine how a vector database scans its indexed embeddings to return the most relevant document chunks for a query. In fully managed environments like RagManagedDb—a Spanner-backed vector datastore in Google Cloud—engineers select between exact search and approximate search algorithms. Choosing between these approaches represents a core architectural trade-off between search recall accuracy and query execution latency:

• k-Nearest Neighbors (KNN): Evaluates the query vector against every indexed vector to provide guaranteed perfect recall, making it ideal for datasets containing fewer than 10,000 files.
• Approximate Nearest Neighbors (ANN): Uses mathematical partitioning to estimate top matches with ultra-low query latency, making it the required choice for large datasets exceeding 10,000 files.

While k-Nearest Neighbors guarantees that no relevant document is missed, its processing overhead scales linearly with dataset size. Approximate Nearest Neighbors uses mathematical indexing techniques to maintain sub-second response times across multi-million row datasets. Selecting the appropriate search strategy ensures the system meets its query latency service level objectives while satisfying accuracy needs.

Key Facts

• Dataplex Universal Catalog tracks dataset lineage, while Vertex ML Metadata links parameters and metrics to specific model versions in Vertex AI Model Registry.
• BigQuery table snapshots freeze training data at a point in time, preventing unexpected dataset modifications from affecting model reproducibility.
• BigQuery ML's ML.VALIDATE_DATA_SKEW detects training-serving skew, and ML.VALIDATE_DATA_DRIFT identifies changes in feature distributions over time.
• Vertex AI Feature Store centralizes feature management for reuse across models, and Vertex AI Pipelines orchestrates ML workflows as DAGs of containerized tasks.
• An event-driven unstructured data pipeline uploads raw files to Cloud Storage, triggers processing in Cloud Run functions via Pub/Sub, and writes JSON Lines (JSONL) metadata for ingestion into Google Agentspace.
• Document AI uses optical character recognition (OCR) and layout parsers to extract structured text from PDFs, HTML, and images, while the max_embedding_requests_per_min setting controls API call rates and costs.
• Sliding-window chunking preserves semantic context across segment boundaries by overlapping text, whereas semantic chunking divides text along natural paragraph or heading boundaries.
• The k-Nearest Neighbors (KNN) retrieval strategy guarantees perfect recall for datasets under 10,000 files, while Approximate Nearest Neighbors (ANN) provides ultra-low latency searches for corpora exceeding 10,000 files.