professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
An unstructured data ingestion pipeline automates the ingestion, transformation, and storage of raw source files for retrieval-augmented generation (RAG) systems. The pipeline uses an event-driven design that processes source documents through extraction, intermediate staging, and datastore indexing. Each component operates asynchronously to decouple data producers from downstream consumers:
An event-driven Google Cloud pipeline where enterprise sources upload raw documents to a Cloud Storage bucket, which publishes an object notification to Pub/Sub. Cloud Run functions consume the message, use Document AI for OCR and layout parsing, and write JSONL metadata to a second bucket, which triggers Google Agentspace via Pub/Sub to parse, chunk, and generate vector embeddings.
This decoupled architecture ensures that high-volume document uploads do not overwhelm downstream processing layers. The system processes records asynchronously while maintaining an immutable audit trail of source documents in Cloud Storage. Automatic embedding generation then prepares the extracted text for downstream vector search without requiring manual model hosting.
Document extraction extracts machine-readable text and structural elements from complex unstructured formats before embedding generation occurs. Specialized data connectors allow the pipeline to ingest documents directly from enterprise sources, including Google Drive, Slack, Jira, and SharePoint. Within the pipeline, Document AI uses optical character recognition (OCR) and layout parsers to preserve the reading order, tables, and formatting of complex PDFs and scanned images. Pipeline administrators control ingestion costs and prevent API quota exhaustion by setting the max_embedding_requests_per_min parameter on Cloud Storage imports. Furthermore, built-in data deduplication skips unchanged files across recurring synchronization jobs to avoid redundant compute and storage expenses.
Exam tip: The max_embedding_requests_per_min setting controls API call rates to prevent quota exhaustion and reduce embedding generation costs during Cloud Storage imports.
Document chunking is the process of partitioning large, continuous text bodies into discrete segments called chunks to optimize semantic retrieval. Selecting the proper chunking technique determines whether a retrieval-augmented generation pipeline preserves contextual meaning or fragments critical concepts. Engineering teams choose between different splitting algorithms based on the document layout and retrieval precision requirements:
| Chunking Strategy | Splitting Mechanism | Tradeoff and Best Use |
|---|---|---|
| Fixed-size chunking | Splits text at uniform character or token counts | Simple to implement, but risks splitting sentences in half and severing context |
| Semantic chunking | Splits text at logical document boundaries such as paragraphs or topic headers | Preserves thematic integrity and complete thoughts, but produces variable chunk sizes |
| Sliding-window chunking | Splits text into fixed segments that share overlapping tokens with adjacent segments | Maintains contextual continuity across chunk boundaries, but increases storage and compute overhead |
Setting appropriate chunk size and chunk overlap parameters balances context retention with retrieval precision. Chunks that are configured too small lack sufficient surrounding context for large language models to generate accurate answers. Conversely, oversized chunks dilute search precision by matching broad, irrelevant text segments against targeted queries. Proper chunking parameters can be configured during data store creation or adjusted directly via administrative APIs.
Metadata enrichment attaches structured attributes to unstructured document chunks to improve search precision and filtering capabilities. Enriched properties typically include document titles, author names, creation timestamps, and section hierarchy headers. Downstream vector databases, such as Cloud Spanner (Spanner) and BigQuery, store these structured metadata fields directly alongside the high-dimensional vector embeddings. During retrieval, search algorithms combine semantic similarity scores with metadata filters, restricting query results to specific authors, dates, or classifications. This multi-layered retrieval strategy prevents generative language models from receiving outdated or irrelevant grounding context.
Vector embedding generation transforms text chunks into high-dimensional numerical arrays that capture semantic meaning. Managed models from the Vertex AI Embedding API, such as text-embedding-004 and text-multilingual-embedding-002, produce consistent dimensional widths across batch and streaming workloads. Alternatively, relational databases like Cloud SQL generate vector embeddings directly within the database engine using the google_ml_integration extension. Organizations store and index these vectors across different Google Cloud datastores based on their query patterns and infrastructure scale. For instance, Cloud SQL employs the pgvector extension to store vector data types, while BigQuery provides native vector search indexes over high-dimensional columns using inverted file (IVF) index structures and Cosine distance metrics.
Retrieval strategies determine how a vector database scans its indexed embeddings to return the most relevant document chunks for a query. In fully managed environments like RagManagedDb—a Spanner-backed vector datastore in Google Cloud—engineers select between exact search and approximate search algorithms. Choosing between these approaches represents a core architectural trade-off between search recall accuracy and query execution latency:
While k-Nearest Neighbors guarantees that no relevant document is missed, its processing overhead scales linearly with dataset size. Approximate Nearest Neighbors uses mathematical indexing techniques to maintain sub-second response times across multi-million row datasets. Selecting the appropriate search strategy ensures the system meets its query latency service level objectives while satisfying accuracy needs.
Exam tip: Use k-Nearest Neighbors (KNN) for datasets with fewer than 10,000 files where 100% recall accuracy is required, and use Approximate Nearest Neighbors (ANN) for datasets exceeding 10,000 files to achieve low-latency lookups.
StudyCerts
StudyCertsmax_embedding_requests_per_min setting controls API call rates and costs.pgvector extension, in BigQuery using inverted file (IVF) indexes, or in RagManagedDb powered by Cloud Spanner.An organization should choose k-Nearest Neighbors (KNN) for datasets containing fewer than 10,000 files when guaranteed perfect recall is required. Approximate Nearest Neighbors (ANN) should be used for datasets exceeding 10,000 files because it uses mathematical partitioning to deliver ultra-low query latency.
Semantic chunking divides text at logical boundaries like paragraphs or topic headers to preserve thematic integrity, resulting in variable chunk sizes. In contrast, sliding-window chunking splits text into fixed segments that share overlapping tokens across adjacent boundaries to maintain contextual continuity, which introduces additional compute and storage overhead.
Raw files uploaded to Cloud Storage automatically trigger Cloud Pub/Sub (Pub/Sub) notifications that prompt Cloud Run functions to extract content and write JSON Lines (JSONL) metadata to a secondary bucket. A subsequent Pub/Sub message triggers Google Agentspace to ingest the JSONL files, parse content, split text into chunks, and automatically generate vector embeddings. This decoupled design processes records asynchronously while preserving an immutable audit trail in Cloud Storage.
Metadata enrichment attaches structured attributes such as document titles, authors, creation timestamps, and section hierarchy headers directly alongside vector embeddings. During retrieval, search algorithms combine semantic similarity scores with these metadata filters to restrict query results to specific classifications or dates, preventing generative models from receiving outdated or irrelevant context.
A data engineering team is designing an unstructured data ingestion pipeline on Google Cloud for an enterprise Retrieval-Augmented Generation (RAG) application. The source corpus contains lengthy technical manuals and complex policy documents that feature hierarchical section headers, tables, and proprietary part numbers.
During initial testing, the team observes two major retrieval issues:
Which strategy should the team implement in their ingestion and retrieval pipeline to resolve both issues?
Extract text using an OCR parser, split content into isolated individual sentences, generate sparse TF-IDF embeddings exclusively, and index them in an inverted keyword catalog.
Use strict fixed-character chunking with zero overlap, generate dense embeddings using standard text embedding models, and store the output in a basic vector index without metadata fields.
Ingest entire documents without chunking, generate a single dense embedding per document using a digital text parser, and rely on the large language model's maximum context window during generation.
Apply layout-aware parsing with sliding-window chunking, enrich chunks with structured metadata for filtering and serving controls, and implement hybrid search using dense and sparse embeddings.