StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Preparing unstructured data for embeddings and retrieval-augmented generation (RAG)
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
The foundation for a RAG system is a scalable ingestion pipeline. Data engineers first upload raw, unstructured files—like PDFs, HTML, or images—into a Cloud Storage bucket. This upload triggers an event-driven workflow: Cloud Storage automatically sends a message to a Pub/Sub topic, which then starts a processing job, often in Cloud Run. This automation ensures data flows through the pipeline without manual intervention.
The triggered job begins processing the raw data. A key service here is Document AI, which extracts high-fidelity text and understands the structural layout from complex formats. The processed data and its metadata are saved as JSON Lines (JSONL) files back to Cloud Storage. Another Pub/Sub message then triggers the next stage: the data is pulled into a managed datastore where it is parsed and chunked. Chunking breaks large documents into smaller, optimized segments, which is crucial for retrieving the most relevant pieces of information later.
A major advantage of using Google's managed services like Agentspace is that vector embeddings are generated automatically. These embeddings are numerical representations of the text that capture semantic meaning, enabling similarity-based search. The system handles this complex step transparently, allowing engineers to focus on pipeline design rather than model configuration. The embeddings are created from the parsed and chunked data, readying it for search.
This architecture leverages several core Google Cloud products. Dataflow provides managed Apache Beam jobs for powerful batch and streaming data transformations. Pub/Sub enables the asynchronous, decoupled messaging that drives the event workflow. Cloud Storage acts as the central, low-cost data lake. Together, these services create a reliable and scalable processing pipeline.
The pipeline can ingest data from many sources beyond Cloud Storage, including Google Drive, Slack, Jira, and SharePoint, using specialized connectors. To manage costs and API quotas, engineers can control the rate of embedding generation calls. The system also supports data deduplication, automatically skipping files that haven't changed since the last import to avoid wasteful reprocessing.
Effective preparation requires configuring parsers and chunking strategies. For example, an OCR parser is needed for scanned documents, while a layout parser handles complex formatting. The chunk size and chunk overlap are critical parameters: chunks that are too small may lose important context, while chunks that are too large can reduce the precision of search results. These settings are configured when the datastore is created.
Document chunking is the process of splitting large texts into smaller, manageable pieces called chunks. The goal is to preserve semantic meaning for better retrieval. Simple fixed-size chunking can cut sentences in half. More advanced techniques include semantic chunking, which splits text at logical boundaries like paragraphs, and sliding-window chunking, which creates overlapping chunks to prevent loss of context at the edges of chunks.
Metadata enrichment involves adding structured, descriptive information to each chunk, such as document title, author, or section header. This metadata is stored separately from the text but is crucial for improving search relevance. During retrieval, the system can use this data to filter and rank chunks based on specific criteria, leading to more accurate and contextually appropriate information being sent to the large language model (LLM).
The choice of chunking strategy is a direct trade-off. Semantic chunking maintains logical flow but produces variable-length chunks. Sliding-window ensures context continuity but increases storage and processing due to overlap. In Google Cloud pipelines, services like Vertex AI Search and Document AI can automate these processes. The final enriched chunks and their metadata are stored alongside their vector embeddings in databases like Spanner or BigQuery, creating an optimized knowledge base for RAG.
After data is chunked, it must be converted into numerical vectors. This is done using Vertex AI Embedding APIs. Engineers select specific, versioned models (like text-embedding-004) to ensure consistent embedding generation with precise dimensional widths. Alternatively, databases like Cloud SQL can use integrated extensions to generate embeddings directly.
The generated high-dimensional vectors must be stored in a way that enables fast similarity searches. In Cloud SQL, the pgvector extension manages vector data and creates approximate nearest neighbor (ANN) indexes to speed up queries. BigQuery also supports native vector search by creating indexes on specific columns. Optimizing index parameters, such as the index type and the distance metric (like Cosine), is key to reducing query latency on large datasets.
For fully managed solutions, Google offers RagManagedDb, a vector database powered by Spanner. When configuring retrieval, engineers choose between two core strategies based on their needs. The k-Nearest Neighbors (KNN) strategy compares a query against all data points, guaranteeing perfect recall; this is best for smaller, recall-sensitive datasets (under ~10,000 files). The Approximate Nearest Neighbors (ANN) strategy uses mathematical approximations to enable ultra-low latency searches, which is critical for scaling to massive document corpora (over 10,000 files). This choice represents the fundamental trade-off between search accuracy (recall) and query speed.