Preparing Unstructured Data for RAG with Vertex AI

Document chunking strategies and metadata enrichment prepare unstructured data so each retrievable unit preserves context and supports precise semantic search in Retrieval-Augmented Generation (RAG). On Google Cloud, a scalable event-driven pipeline divides this work between custom enrichment and managed chunking: Cloud Storage (opens in a new tab) holds raw data, Cloud Run functions (opens in a new tab) applies business logic to create structured metadata, and the managed datastore in Gemini Enterprise (opens in a new tab) parses, chunks, and embeds the combined inputs. Data flows from external sources to Cloud Storage buckets, through Pub/Sub triggers, to the managed datastore where indexed vectors and metadata serve filtered retrieval.

1.1Custom Metadata Enrichment in the Ingestion Pipeline

Custom metadata enrichment is the step where Cloud Run functions tags each document with structured attributes that preserve context lost in raw text alone. The enrichment job is triggered when data engineers upload raw data from applications, databases, or streaming services to a Cloud Storage bucket, which publishes a message to Pub/Sub upon completion and starts the function. The function applies organization-specific business logic and writes the result as JSON Lines (JSONL) files into a separate Cloud Storage bucket from the raw data. This separation keeps the original document intact while the sidecar JSONL carries the filterable context that retrieval will use later.

Enrichment attributes narrow search to the correct slice of the corpus and enable context-aware answers. Typical tags applied by the function include:

• product line that identifies which product or business unit the document belongs to
• author that preserves ownership and source authority for the chunk
• location that preserves regional or site context for the content
• document type that distinguishes guides, reports, policies, or other formats

Upon completion, Cloud Run functions publishes a message to a second Pub/Sub topic, so downstream chunking only starts after both the raw data bucket and the metadata bucket contain matching inputs.

1.2Parsing, Chunking, and Embedding in the Managed Datastore

Parsing, chunking, and embedding in the managed datastore convert raw documents plus metadata into efficient retrieval units with vector representations. The managed datastore processing job is triggered by Pub/Sub, then pulls the raw ingested data and the JSONL metadata from their Cloud Storage buckets. The job parses and chunks the data for efficient retrieval during serving, and Gemini Enterprise automatically generates vector embeddings without any configuration necessary. Because chunking happens after enrichment, each chunk inherits the structured context needed for filtered semantic search.

The division of responsibility between custom code and the managed platform is strict:

This boundary supports real-time data availability with minimal delay between data arrival and use. New or updated documents flow through the same event-driven sequence — Cloud Storage to Pub/Sub to Cloud Run functions to Pub/Sub to managed datastore — so the index stays current without batch rebuilds.

1.3Preserving Context and Maximizing Relevance at Retrieval Time

Context-preserving retrieval is the serving flow where structured metadata filters combine with semantic search to return the most relevant chunks. A query flows from a Cloud Run (opens in a new tab) frontend service to a centralized Cloud Run backend service, which performs preprocessing that can include constructing filters for the search query and keeps that logic transparent to frontends. The backend service sends the prepared request to Gemini Enterprise through the Gemini Enterprise API endpoint to initiate the RAG workflow. Gemini Enterprise then creates an embedding of the query, performs semantic search on the indexed data in the managed datastore, augments the original query with retrieved data to create a detailed contextual prompt, and generates the final response that returns through the backend to the frontend.

Ingestion, extraction, and cleaning of unstructured formats is the end-to-end process that turns raw PDFs, HTML, images, and other source files into high-fidelity text plus structure that a RAG system can retrieve. In Google Cloud, Cloud Storage (opens in a new tab) provides durable landing for diverse object types, Pub/Sub decouples arrival events from processing, Cloud Run functions (opens in a new tab) performs lightweight preparation and metadata creation, and Dataflow (opens in a new tab) provides scalable batch and streaming transformation. The output of this stage is cleaned, parsed, and chunked content with metadata that flows to a managed datastore for embedding and indexing.

2.1Event-Driven Ingestion Through Cloud Storage and Pub/Sub

Event-driven ingestion is a pattern where an object landing in Cloud Storage automatically starts downstream preparation without polling. Cloud Storage is a low-cost, no-limit object store for diverse data types that can be accessed from within and outside Google Cloud and is replicated across locations for redundancy. Pub/Sub (opens in a new tab) is an asynchronous and scalable messaging service that decouples services that produce messages from services that process those messages, so producers and processors can scale and fail independently. This decoupling lets many applications, databases, or streaming services land files concurrently while processing scales behind topics and subscriptions.

Data flows from external sources through a fixed order in the RAG data ingestion subsystem:

1. Land raw objects: Data engineers upload data from external sources such as applications, databases, or streaming services to a Cloud Storage bucket.
2. Notify arrival: Upon completion, Cloud Storage publishes a message to a Pub/Sub topic.
3. Trigger preparation: The Pub/Sub topic triggers a processing job to run in Cloud Run functions.
4. Create metadata: Cloud Run functions processes the raw data by generating and storing the metadata as JSON Lines (JSONL) files in a separate Cloud Storage bucket.
5. Notify readiness: Upon completion, Cloud Run functions publishes a message to a Pub/Sub topic.
6. Parse and index: The Pub/Sub topic triggers a processing job in the managed datastore within Gemini Enterprise (opens in a new tab), which pulls the raw ingested data and metadata from the Cloud Storage buckets, then parses and chunks the data for efficient retrieval and automatically generates vector embeddings.

The separation of the raw bucket from the JSONL metadata bucket creates an ownership and lifecycle boundary: the raw bucket preserves source fidelity for reprocessing, while the metadata bucket carries normalized pointers and attributes that downstream parsing depends on. Pub/Sub subscriptions can buffer unacknowledged messages for up to 7 days, so if parsing stops, arrival events wait in the subscription and processing resumes with a low risk of data loss when pipelines restart.

2.2Extraction of Text, Layout, and Metadata for RAG

Extraction is the step where unstructured bytes become retrievable text plus structural signals and metadata. The managed datastore processing job pulls both the raw object and its JSONL metadata from Cloud Storage, then parses and chunks the data so semantic search can find the most relevant passages during serving. Parsing preserves more than plain characters: it recovers layout elements such as headings, tables, sections, and document order from complex PDFs, HTML, and images, which keeps context intact when a long document is split into chunks. Because chunking happens after parsing, each chunk inherits document structure and metadata rather than becoming an isolated string.

Cloud Run functions handles the pre-parse preparation because it is a serverless compute platform for running single-purpose functions directly in Google Cloud. In this flow the function does not create the final embeddings; it normalizes intake and emits JSONL metadata that describes the raw objects for the later parse-and-chunk job. The downstream managed job then depends on both inputs: the raw bucket supplies content for high-fidelity text extraction, and the JSONL bucket supplies the metadata used for filtering, attribution, and efficient retrieval. Gemini Enterprise automatically generates vector embeddings from the parsed chunks without additional configuration, so ingestion engineers focus on completeness and structure of the input rather than embedding operations.

2.3Cleaning With Filter, Transform, and Redact Controls

Cleaning is pre-parsing control applied before data is stored and indexed, so only relevant, usable, and safe content moves forward. This stage reduces noise that would waste embedding and retrieval capacity, fixes format problems that would break parsing, and removes sensitive values before they reach a vector index. Cleaning runs in the processing layer between Cloud Storage arrival and managed parsing, where Cloud Run functions or Dataflow can inspect each record and decide to keep, reshape, or mask it.

The three cleaning actions divide by intent:

• Filter: Ingest only relevant events to reduce noise and cost, such as dropping empty key-value pairs, filtering by field values, or reducing high-volume categories before parsing.
• Transform: Modify data formats, parse fields, and enrich logs for better usability, such as normalizing HTML text, adding ingestion labels from raw content, or attaching source identifiers for multi-instance streams.
• Redact: Protect sensitive information by masking or removing sensitive values before storage, so regulated text from clinical, legal, or user-behavior sources never reaches embeddings or BigQuery analytics.

2.4Scaling Preparation With Dataflow and Managed Alternatives

Scalable preparation is the choice of compute that applies extraction and cleaning across large backlogs and continuous arrivals with the same logic. Dataflow is a fully managed service for running Apache Beam jobs at scale, where Apache Beam provides windowing and session-analysis primitives plus source and sink connectors, including a connector for BigQuery. Apache Beam lets pipelines transform and enrich data in both stream and batch modes with equal reliability and expressiveness, while the serverless Dataflow service handles performance, scaling, availability, security, and compliance automatically. Engineers submit Dataflow jobs through the command-line interface, Java SDK, or Python SDK and focus on pipeline code instead of managing server clusters.

Locality and placement control where bytes move during scaled processing. If no locality restriction applies, publishers can use the global Pub/Sub endpoint accessible from any Google Cloud location. When messages must stay in specific regions, the Pub/Sub topic message storage policy restricts persistence to named regions, but that restriction carries availability tradeoffs versus global endpoints. For Dataflow, the regional endpoint, worker region, and worker zone parameters control where workers deploy and where job metadata is stored, and sources, sinks, and temporary files must sit in the allowed region or data will cross regions during execution.

Embedding generation and Vertex AI Vector Search indexing is the pipeline that turns unstructured chunks into high-dimensional vectors and makes those vectors searchable for RAG. In this pipeline, an embedding model converts text into numerical vectors where semantically similar content sits close together, and a vector database stores those vectors in an index optimized for similarity search. At query time, the user query is converted to the same vector space, the closest chunk vectors are retrieved, and those chunks ground the generative model response.

3.1Generating High-Dimensional Embeddings with Vertex AI

Vertex AI (opens in a new tab) embedding APIs create dense vector representations used for search, classification, and clustering. The text-embedding models take a text chunk as input and return an embedding vector, and the same model family must be used for both document chunks and user queries so both live in the same embedding space. Because similar words and passages produce nearby vectors, similarity can be measured mathematically, for example with cosine similarity between two embeddings.

BigQuery ML (opens in a new tab) provides scalable SQL paths to the same Agent Platform embedding models without moving data out of the warehouse. The AI.EMBED function generates a single embedding for text data or image data represented by ObjectRef values, which is useful for a small number of on-demand comparisons. The AI.GENERATE_EMBEDDING function creates a table of embeddings for every row in a source column, working with structured data in standard tables and, for multimodal models, with ObjectRef columns or object tables. For remote models, inference runs in Agent Platform and results are stored in BigQuery.

Autonomous embedding generation lets BigQuery maintain embeddings automatically when source data changes regularly. BigQuery maintains an embedding column tied to a source text column, and when data is added or modified in the source column, BigQuery automatically generates or updates the embedding column using an Agent Platform embedding model. This removes manual re-embedding jobs and keeps the retrieval index fresh for frequently updated tables.

3.2Searching Embeddings for Semantic Retrieval

BigQuery search functions perform semantic vector search and hybrid search over stored embeddings. Hybrid search combines semantic vector similarity with lexical keyword search to improve relevance when exact terms also matter.

• VECTOR_SEARCH performs semantic vector search or hybrid search by using SQL, and it is the choice when performance is critical and working with a large number of embeddings.
• AI.SEARCH performs semantic or hybrid search for results close to a string you provide, and it can be used when the table has autonomous embedding generation enabled.
• AI.SIMILARITY compares two inputs by computing cosine similarity between their embeddings, which works well for a small number of comparisons where no embeddings have been precomputed.

A BigQuery vector index created with CREATE VECTOR INDEX accelerates VECTOR_SEARCH and AI.SEARCH with Approximate Nearest Neighbor search. Approximate Nearest Neighbor improves vector search performance with the tradeoff of reducing recall, meaning it returns very close neighbors quickly but may miss some exact nearest neighbors.

3.3Configuring and Serving Indexes with Vertex AI Vector Search

Vertex AI Vector Search (opens in a new tab) is the specialized and optimized vector database for low-latency, large-scale serving of similarity searches. Data flows from chunked source data to the embedding API to the index, then the index is deployed to serving infrastructure that handles high-throughput queries. The deployed index depends on the embedding model and chunking upstream, because a change in model or chunk size changes vector dimensionality and meaning and requires re-embedding and re-indexing.

The serving flow for RAG grounding follows an ordered sequence:

1. Embed the query: Gemini Enterprise (opens in a new tab) or the retrieval service creates an embedding of the user query with the same model used for documents.
2. Semantic search: The query vector is searched against the indexed data in the managed datastore or vector index to find the most relevant chunks.
3. Augment the prompt: The original query is combined with the retrieved chunks to create a detailed, contextual prompt.
4. Generate the response: The generative model generates a final response based on the augmented prompt, which is then filtered and returned through the backend to the frontend.

In a custom pipeline, Cloud Storage holds raw data, Pub/Sub triggers Cloud Run functions to parse data and write metadata as JSON Lines (JSONL), and a managed datastore pulls that data, parses and chunks it, and automatically generates vector embeddings. In a Cloud SQL (opens in a new tab) pattern, Vertex AI creates embeddings of text fields and stores them as vectors in PostgreSQL, the retrieval service converts each request to an embedding, searches for similar vectors, and sends the matches plus the original prompt to Vertex AI.

3.4Choosing a Vector Store on Google Cloud

Google Cloud vector store options differ by scale, access pattern, and control over embeddings. The choice depends on whether the workload needs large-scale semantic serving, fully managed search, transactional access, or analytical SQL.

Key Facts

• Cloud Run functions generates JSONL metadata in a separate Cloud Storage bucket, while parsing, chunking, and automatic embedding generation happen in the managed datastore in Gemini Enterprise.
• Pub/Sub subscriptions can buffer unacknowledged messages for up to 7 days, allowing ingestion to resume with low risk of data loss after a pipeline interruption.
• The same embedding model must be used for document chunks and user queries so both live in the same vector space for cosine similarity search.
• A BigQuery vector index created with CREATE VECTOR INDEX accelerates VECTOR_SEARCH and AI.SEARCH with Approximate Nearest Neighbor search at the tradeoff of reduced recall.
• Vertex AI Vector Search provides low-latency, large-scale similarity serving with direct index control, while Vertex AI Search is fully managed and does not expose underlying embeddings.