Preparing Features and Embeddings for Machine Learning

Preparing data for feature engineering, training, and serving is the process of turning raw operational data into clean, reusable features that train models and serve predictions consistently. Raw data flows from operational sources into ingestion tools, then through transformation logic, and finally into training tables or a central feature store. On Google Cloud, BigQuery (opens in a new tab) handles tabular SQL transformation, Dataflow (opens in a new tab) handles batch and streaming transformation, and BigQuery ML applies automatic preprocessing during training.

1.1Transformation Pipelines From Raw Sources to Training Features

Transformation pipelines from raw sources to training features separate where data comes from from how it is moved and reshaped for model use. Sources include operational databases such as Cloud SQL, Spanner, AlloyDB for PostgreSQL, and Firestore, data lakes such as Cloud Storage with BigLake, streaming platforms such as Pub/Sub and Dataflow, and external APIs. For each source, data flows from the source system into an ingestion tool and then into BigQuery or Cloud Storage storage optimized for machine learning training, separate from the operational source.

Tool choice depends on latency, volume, and reuse needs for feature consumption. Dataflow processes data from many sources in batch and stream with ML-component integration, and it combines with Eventarc for event-driven architectures where changes trigger processing. Dataform automates SQL-based transformations for extraction, loading, and transformation (ELT) workflows after raw data lands in BigQuery staging tables. BigQuery itself performs in-warehouse SQL transformation for frequently accessed structured data, where materialized tables provide faster training reads than views or subqueries.

Feature transformation logic in these pipelines covers four core tasks for model consumption. Feature scaling standardizes numeric ranges, categorical encoding converts strings and booleans to numeric vectors, missing-value handling replaces gaps, and derived-feature creation builds new signals such as bucketed ranges or crossed categories. Vertex AI Feature Store (opens in a new tab) centralizes the resulting features in BigQuery tables or views for reuse across models and projects. Because BigQuery stores both online serving data and historical offline data, the same feature definitions support training and serving without duplicate engineering.

1.2Automatic Preprocessing During BigQuery ML Training

Automatic preprocessing during BigQuery ML training is the default transformation that runs during training through the CREATE MODEL (opens in a new tab) statement without custom code. Automatic preprocessing consists of missing-value imputation followed by feature transformation, and the same statistics learned during training are automatically applied during prediction. This behavior ensures that training and serving see consistent feature values even when prediction rows contain nulls or previously unseen categories.

Missing-data handling in BigQuery ML treats NULL values during training as missing data, while prediction treats both NULL values and unseen values as missing. Numeric columns replace nulls with the column mean calculated from the original input data in both training and prediction. One-hot and multi-hot encoded columns map nulls to an added category, while unseen values receive a weight of 0 during prediction. TIMESTAMP columns use a mixed approach where the generated Unix time column uses mean imputation and other extracted fields use a null category, and each field of a STRUCT column is imputed according to its own type.

The following table compares imputation behavior by column type for training and prediction:

Default feature transformations in BigQuery ML depend on input data type and model type. Numeric types including INT64, NUMERIC, BIGNUMERIC, and FLOAT64 are standardized and centered at zero for most models, which regularizes scale differences before training. Non-numerical, non-array columns such as BOOL, STRING, BYTES, DATE, DATETIME, and TIME receive one-hot encoding for most models, which creates a separate feature for each unique value. Non-numerical ARRAY columns receive multi-hot encoding, which creates a separate feature for each unique element in the array.

1.3Custom Preprocessing With the TRANSFORM Clause

Custom preprocessing with the TRANSFORM clause is manual preprocessing defined inside CREATE MODEL that runs in addition to automatic preprocessing. The clause takes raw input columns, applies manual preprocessing functions, and passes only the transformed outputs to training, so the model never sees the untransformed values. The ML.TRANSFORM function then returns those preprocessed rows for inspection, which makes training data and prediction-time serving data transparent and reproducible.

Manual preprocessing functions divide by how they read rows and preserve training-to-serving consistency. Scalar functions operate on a single row, such as bucketizing a value. Table-valued functions operate on all rows and output a table, such as retrieving features at a point in time. Analytic functions operate on all rows but return a result per row based on statistics collected across the full dataset, and they always require an empty OVER() clause so BigQuery ML can capture the statistics during training and reapply the same statistics during prediction.

Common manual functions for normalization, encoding, and cleanup include:

• ML.IMPUTER for data cleanup on string or numerical expressions with custom replacement strategies.
• ML.BUCKETIZE, ML.QUANTILE_BUCKETIZE, and ML.HASH_BUCKETIZE for binning continuous or categorical values into intervals.
• ML.STANDARD_SCALER, ML.MIN_MAX_SCALER, ML.MAX_ABS_SCALER, ML.ROBUST_SCALER, and ML.NORMALIZER for regularizing numeric scale and distribution.
• ML.ONE_HOT_ENCODER, ML.LABEL_ENCODER, and ML.MULTI_HOT_ENCODER for encoding categorical variables into model-ready vectors.
• ML.FEATURE_CROSS for creating derived features by combining two categorical signals, ML.POLYNOMIAL_EXPAND for creating polynomial derived features, and ML.NGRAMS for creating text features from string expressions.

A transform-only model decouples preprocessing from model fitting when teams need to version and reuse transformation logic independently. The transform-only model contains only the TRANSFORM clause and outputs transformed tables without training a predictor. Downstream training jobs then depend on the transform-only model for consistent features, which avoids maintaining multiple custom implementations across projects.

1.4Storage, Partitioning, and Read Paths for ML Workloads

Storage, partitioning, and read paths for ML workloads determine where training data lives, what file format it uses, and how queries reach it during iterative training and batch prediction. BigQuery separates storage, compute, and memory into independently scaled services, while Cloud Storage holds data-lake files that feed training jobs directly or through BigQuery tables. The right combination of columnar formats, Hive-style partitioning, BigQuery native storage, and read APIs determines whether repeated feature queries and scoring jobs stay fast and affordable.

Separated storage, compute, and memory in BigQuery lets each layer scale without forcing growth in the others. Compute is measured in slots, which are units of query-processing capacity managed by Google in reservations assigned to projects, folders, or organizations. Memory comes from a remote distributed service connected to slots by a petabit network, and table data lives in a distributed file system where it is automatically compressed, encrypted, replicated, and distributed without consuming query slots. Because storage growth to dozens of petabytes does not require adding compute, teams can keep large training tables online while sizing slots only for query and BigQuery ML concurrency.

Columnar formats and file layout in Cloud Storage control scan efficiency for feature engineering. Parquet and Optimized Row Columnar (ORC) organize data by column rather than by row, which allows efficient storage and faster access for analytical workloads compared to row-oriented JavaScript Object Notation (JSON) or comma-separated values (CSV) files. A Hive partitioned layout places files in directory prefixes keyed by partitioning columns, so queries that filter on those columns can prune partitions and skip entire directories. File design must also balance the number of files against file size, because too many tiny files increases metadata and listing overhead while very large files can limit parallelism for distributed readers.

The following table compares storage and query behavior for ML scans:

Partitioning, clustering, and BigQuery native storage add controls that external files alone cannot match for repeated training queries. When queries against BigLake tables that read external Parquet, ORC, JSON, or CSV files cannot meet service-level agreement requirements and cannot be tuned further, the supported path is to convert the exposed data to BigQuery storage and redefine authorized views to point at the new BigQuery tables. This migration generally does not disrupt consumers or require query changes, but it unlocks materialized views with different partitioning and clustering than base tables and acceleration with the Business Intelligence (BI) Engine. In practice, data flows from Cloud Storage ingestion buckets through processing into curated BigQuery tables, and the serving layer depends on those tables for consistent, pruned, and clustered access during batch prediction.

Read paths and cost controls reward early filtering for repeated jobs. The BigQuery Storage Read API can read data from both BigQuery tables and BigLake tables, supports filtering and fine-grained access control, and is a good option for stable analytics or ML consumers that repeatedly scan large feature sets. The Cloud Storage JSON or XML APIs read bucket objects directly, which is useful only when consumers cannot use the SQL interface, but direct bucket access does not allow filtering or fine-grained access control and is viable mainly for stable, small-sized datasets measured in gigabytes. For repeated queries, BigQuery builds an execution tree of parallel stages that exchange data through a distributed shuffle tier, and teams control cost with on-demand pricing that charges for bytes scanned versus capacity-based pricing that uses reserved slots, plus cost optimization (opens in a new tab) signals such as query-plan inspection, INFORMATION_SCHEMA.JOBS views, custom quotas, and billing alerts.

1.5Versioning, Lineage, and Reproducibility for Training Runs

Versioning, lineage, and reproducibility for training runs link every training run to the exact code, dataset snapshot, transformations, parameters, and metrics that produced it. On Google Cloud, this linkage is built from Vertex ML Metadata for experiment-level lineage, Knowledge Catalog for organization-wide governance, and Model Registry for model versions, combined with reproducible storage and pipeline practices. Together, Git for code plus metadata plus registry versions mean code changes, data changes, and model changes each have an independent, queryable history.

Lineage tracking across data, code, and artifacts connects data sources, transformations, parameters, metrics, and dataset and model artifacts so any experiment can be audited and rerun. Vertex ML Metadata logs metadata about data sources, transformations, model parameters, and experiment results, and Experiments on Vertex AI integrates with it to build linkage across those objects. Vertex AI TensorBoard visualizes and compares runs by metrics such as loss and accuracy, model graph, and embeddings, while audit logs record who accessed or modified data and models and when. Knowledge Catalog enriches technical metadata with business metadata through aspects, adds BigQuery column-level lineage, and lets teams search Vertex AI Pipelines artifact lineage for debugging and root-cause analysis.

Repeatable training inputs require separating ML training storage from operational sources and freezing the data actually used for training. Data flows from source systems such as logs, devices, or scraped documents into storage optimized for ML training, and for maximum speed and reproducibility the training input should be materialized data rather than views or subqueries. Vertex AI managed datasets create a clear link between that frozen data and custom-trained models, provide descriptive statistics and automatic or manual splitting into train, test, and validation sets, and read data out through the BigQuery Storage API for artifact tracking. Unstructured data follows the same freeze principle in Cloud Storage, where images, video, and audio are combined into large container formats such as sharded TFRecord or Avro files with a target of at least 100 MB per file and between 100 and 10,000 shards.

Training-serving skew happens when features computed for offline training differ from features served for online prediction across model iterations. Vertex AI Feature Store reduces this risk by creating, maintaining, sharing, and serving ML features in a central location optimized for low-latency workloads. The team first creates an online store instance, then defines feature views on top of BigQuery data, so data flows from BigQuery historical storage to offline training jobs and from the online store to serving paths while both paths resolve the same feature logic. Reusable pipelines enforce the same consistency in code: Vertex AI Pipelines structures work as a directed acyclic graph of containerized tasks where outputs flow as inputs to the next task, GoogleSQL procedural language suits less complex SQL pipelines with shared state and IF and WHILE logic, and Dataform suits ELT-heavy feature pipelines that need version control before training.

Preparing unstructured data for embeddings and retrieval-augmented generation is the pipeline that turns raw PDFs, HTML, images, and other files into cleaned chunks, vectors, and indexes that ground generative answers. Data flows from external sources to Cloud Storage (opens in a new tab) buckets, through Pub/Sub triggers, through preparation compute, and then to a managed datastore or vector index where indexed vectors and metadata serve filtered retrieval. The same embedding model must be used for document chunks and user queries so both live in the same vector space.

2.1Event-Driven Ingestion and Cleaning of Unstructured Files

Event-driven ingestion of unstructured files is a pattern where an object landing in Cloud Storage automatically starts downstream preparation without polling. Cloud Storage is a low-cost object store for diverse data types that can be accessed from within and outside Google Cloud and is replicated across locations for redundancy. Pub/Sub is an asynchronous and scalable messaging service that decouples services that produce messages from services that process those messages, so producers and processors can scale and fail independently.

Data flows from external sources through a fixed order in the RAG ingestion subsystem:

1. Land raw objects: Data engineers upload data from applications, databases, or streaming services to a Cloud Storage bucket.
2. Notify arrival: Upon completion, Cloud Storage publishes a message to a Pub/Sub topic.
3. Trigger preparation: The Pub/Sub topic triggers a processing job to run in Cloud Run functions.
4. Create metadata: Cloud Run functions processes the raw data by generating and storing the metadata as JSON Lines (JSONL) files in a separate Cloud Storage bucket.
5. Notify readiness: Upon completion, Cloud Run functions publishes a message to a Pub/Sub topic.
6. Parse and index: The Pub/Sub topic triggers a processing job in the managed datastore within Gemini Enterprise, which pulls the raw data and metadata from the buckets, then parses and chunks the data and automatically generates vector embeddings.

The separation of the raw bucket from the JSONL metadata bucket creates an ownership and lifecycle boundary. The raw bucket preserves source fidelity for reprocessing, while the metadata bucket carries normalized pointers and attributes that downstream parsing depends on. Pub/Sub subscriptions can buffer unacknowledged messages for up to 7 days, so if parsing stops, arrival events wait in the subscription and processing resumes with a low risk of data loss when pipelines restart.

Cleaning is pre-parsing control applied before data is stored and indexed, so only relevant, usable, and safe content moves forward. Cleaning runs in the processing layer between Cloud Storage arrival and managed parsing, where Cloud Run functions or Dataflow can inspect each record and decide to keep, reshape, or mask it. The three cleaning actions divide by intent:

• Filter: Ingest only relevant events to reduce noise and cost, such as dropping empty key-value pairs or reducing high-volume categories before parsing.
• Transform: Modify data formats, parse fields, and enrich logs for better usability, such as normalizing HTML text or attaching source identifiers for multi-instance streams.
• Redact: Protect sensitive information by masking or removing sensitive values before storage, so regulated text never reaches embeddings or analytics.

2.2Metadata Enrichment, Parsing, and Chunking for Retrieval

Metadata enrichment, parsing, and chunking for retrieval prepare unstructured data so each retrievable unit preserves context and supports precise semantic search. A scalable event-driven pipeline divides this work between custom enrichment and managed chunking: Cloud Storage holds raw data, Cloud Run functions applies business logic to create structured metadata, and the managed datastore in Gemini Enterprise parses, chunks, and embeds the combined inputs. Because chunking happens after enrichment, each chunk inherits the structured context needed for filtered semantic search.

Custom metadata enrichment is the step where Cloud Run functions tags each document with structured attributes that preserve context lost in raw text alone. The enrichment job is triggered when data lands in Cloud Storage, applies organization-specific business logic, and writes the result as JSONL files into a separate bucket from the raw data. Typical tags applied by the function include:

• product line that identifies which product or business unit the document belongs to
• author that preserves ownership and source authority for the chunk
• location that preserves regional or site context for the content
• document type that distinguishes guides, reports, policies, or other formats

Parsing, chunking, and embedding in the managed datastore convert raw documents plus metadata into efficient retrieval units with vector representations. The managed datastore job pulls both the raw object and its JSONL metadata, then parses and chunks the data so semantic search can find the most relevant passages during serving. Parsing and chunking (opens in a new tab) preserves more than plain characters by recovering layout elements such as headings, tables, sections, and document order from complex PDFs, HTML, and images. Gemini Enterprise then automatically generates vector embeddings without additional configuration, so ingestion engineers focus on completeness and structure of the input rather than embedding operations.

2.3Embedding Generation and Vector Search in BigQuery

Embedding generation and vector search in BigQuery turn text chunks into high-dimensional vectors and make those vectors searchable with SQL. An embedding model converts text into numerical vectors where semantically similar content sits close together, and similarity can be measured mathematically, for example with cosine similarity. The text-embedding models take a text chunk as input and return an embedding vector, and the same model family must be used for both document chunks and user queries.

BigQuery ML provides scalable SQL paths to Agent Platform embedding models without moving data out of the warehouse. The AI.EMBED function generates a single embedding for text data or image data represented by ObjectRef values, which is useful for a small number of on-demand comparisons. The AI.GENERATE_EMBEDDING function creates a table of embeddings for every row in a source column, working with structured data in standard tables and, for multimodal models, with ObjectRef columns or object tables. Autonomous embedding generation lets BigQuery maintain an embedding column tied to a source text column, so when data is added or modified in the source column, BigQuery automatically generates or updates the embedding column.

BigQuery search functions perform semantic and hybrid search over stored embeddings. Hybrid search combines semantic vector similarity with lexical keyword search to improve relevance when exact terms also matter. VECTOR_SEARCH (opens in a new tab) performs semantic or hybrid search by using SQL, and it is the choice when performance is critical and working with a large number of embeddings. AI.SEARCH performs semantic or hybrid search for results close to a string you provide, and it can be used when the table has autonomous embedding generation enabled. AI.SIMILARITY compares two inputs by computing cosine similarity between their embeddings, which works well for a small number of comparisons where no embeddings have been precomputed. A BigQuery vector index created with CREATE VECTOR INDEX accelerates VECTOR_SEARCH and AI.SEARCH with Approximate Nearest Neighbor search, which improves performance with the tradeoff of reducing recall.

2.4Serving Indexes and Choosing a Vector Store

Serving indexes for RAG is the stage where chunk vectors are deployed to infrastructure that handles high-throughput similarity queries. Vertex AI Vector Search (opens in a new tab) is the specialized and optimized vector database for low-latency, large-scale serving of similarity searches. Data flows from chunked source data to the embedding API to the index, then the index is deployed to serving infrastructure that handles queries. The deployed index depends on the embedding model and chunking upstream, because a change in model or chunk size changes vector dimensionality and meaning and requires re-embedding and re-indexing.

The serving flow for RAG grounding follows an ordered sequence:

1. Embed the query: Gemini Enterprise or the retrieval service creates an embedding of the user query with the same model used for documents.
2. Semantic search: The query vector is searched against the indexed data in the managed datastore or vector index to find the most relevant chunks.
3. Augment the prompt: The original query is combined with the retrieved chunks to create a detailed, contextual prompt.
4. Generate the response: The generative model generates a final response based on the augmented prompt, which is then filtered and returned through the backend to the frontend.

A query flows from a Cloud Run frontend service to a centralized Cloud Run backend service, which performs preprocessing that can include constructing filters for the search query. The backend service sends the prepared request to Gemini Enterprise through the Gemini Enterprise API endpoint to initiate the RAG workflow. In a Cloud SQL (opens in a new tab) pattern, Vertex AI creates embeddings of text fields and stores them as vectors in PostgreSQL, the retrieval service converts each request to an embedding, searches for similar vectors, and sends the matches plus the original prompt to Vertex AI.

Google Cloud vector store options differ by scale, access pattern, and control over embeddings:

Scalable preparation lets teams apply the same extraction and cleaning logic to backlogs and live streams. Dataflow is a fully managed service for running Apache Beam jobs at scale, where Apache Beam provides windowing and session-analysis primitives plus connectors including a BigQuery connector. Engineers submit Dataflow jobs and focus on pipeline code instead of managing clusters, while Cloud Data Fusion offers visual pipeline DAGs executed as Apache Spark jobs with prebuilt plugins. For placement, Pub/Sub topic message storage policy can restrict persistence to named regions with availability tradeoffs, and Dataflow regional endpoint, worker region, and worker zone parameters control where workers deploy and where job metadata is stored.

Key Facts

• BigQuery ML automatic preprocessing applies missing-value imputation followed by feature transformation during CREATE MODEL, and the same training statistics are reapplied during prediction.
• Analytic preprocessing functions in the TRANSFORM clause require an empty OVER() clause so training statistics can be reused at prediction time.
• The BigQuery Storage Read API supports filtering and fine-grained access control for BigQuery and BigLake tables, while direct Cloud Storage API access does not.
• Pub/Sub subscriptions can buffer unacknowledged messages for up to 7 days, allowing ingestion to resume with low risk of data loss after interruption.
• A BigQuery vector index created with CREATE VECTOR INDEX accelerates VECTOR_SEARCH and AI.SEARCH with Approximate Nearest Neighbor search at the tradeoff of reduced recall.