A data engineering team is designing an unstructured data ingestion pipeline on Google Cloud for an enterprise Retrieval-Augmented Generation (RAG) application. The source corpus contains lengthy technical manuals and complex policy documents that feature hierarchical section headers, tables, and proprietary part numbers. During initial testing, the team observes two major retrieval issues: Queries targeting information split across section boundaries suffer from semantic context fragmentation and lost continuity. Queries containing exact proprietary part numbers and requests filtered by department or document version yield poor retrieval relevance. Which strategy should the team implement in their ingestion and retrieval pipeline to resolve both issues?

Apply layout-aware parsing with sliding-window chunking, enrich chunks with structured metadata for filtering and serving controls, and implement hybrid search using dense and sparse embeddings.: ### What layout parsing, sliding-window chunking, and hybrid search provide This architecture combines advanced **layout-aware parsing**, **sliding-window chunking** (chunking with overlapping context), **structured metadata enrichment**, and **hybrid search** (dense and sparse vector indexing) to ensure optimal retrieval quality for RAG. ### How it meets the requirements - **Preserves boundary context**: Using **sliding-window chunking** (chunks configured with token overlap) combined with a **layout parser** detects structural elements like headers and tables, ensuring that semantic continuity across paragraph boundaries is retained. - **Resolves out-of-domain proprietary part numbers**: Proprietary part numbers and SKUs are often out-of-domain for standard dense embedding models. Employing **hybrid search** couples dense semantic embeddings with token-based sparse embeddings (such as **BM25**, **TF-IDF**, or **SPLADE**) to ensure exact keyword and part number matches succeed. - **Enables granular filtering**: Enriching individual chunks with **structured metadata** (such as department, document hierarchy, and version) allows the retrieval engine to apply precise **field settings**, **serving controls**, and pre/post-retrieval filters. ### Key advantages and capabilities - **Structural awareness**: The **Layout Parser** understands complex document structure, tables, and text flow rather than treating documents as arbitrary raw text strings. - **Sliding-window overlap**: Prevents critical statements spanning chunk boundaries from being split and losing contextual meaning. - **Combined search modalities**: **Vertex AI Vector Search** and **Vertex AI Search** hybrid indexing merge semantic meaning matching with exact token matching in a unified index. - **Metadata-driven governance**: Enables granular attribute filtering and boost/bury rules. ### Why this is optimal This approach directly targets both context loss and keyword mismatch by pairing structurally sound chunking with a dual dense/sparse retrieval index and metadata filtering.

Rag Chunking Metadata Enrichment Hybrid Search | professional-cloud-data-engineer

Rag Chunking Metadata Enrichment Hybrid Search

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team is designing an unstructured data ingestion pipeline on Google Cloud for an enterprise Retrieval-Augmented Generation (RAG) application. The source corpus contains lengthy technical manuals and complex policy documents that feature hierarchical section headers, tables, and proprietary part numbers.

During initial testing, the team observes two major retrieval issues:

Queries targeting information split across section boundaries suffer from semantic context fragmentation and lost continuity.
Queries containing exact proprietary part numbers and requests filtered by department or document version yield poor retrieval relevance.

Which strategy should the team implement in their ingestion and retrieval pipeline to resolve both issues?

Ingest entire documents without chunking, generate a single dense embedding per document using a digital text parser, and rely on the large language model's maximum context window during generation.

Use strict fixed-character chunking with zero overlap, generate dense embeddings using standard text embedding models, and store the output in a basic vector index without metadata fields.

Apply layout-aware parsing with sliding-window chunking, enrich chunks with structured metadata for filtering and serving controls, and implement hybrid search using dense and sparse embeddings.

Extract text using an OCR parser, split content into isolated individual sentences, generate sparse TF-IDF embeddings exclusively, and index them in an inverted keyword catalog.

Professional Cloud Data Engineer

Rag Chunking Metadata Enrichment Hybrid Search

Related questions