StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Preparing data for AI and ML
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
Reproducible machine learning requires tracking data and pipeline steps so experiments can be audited and debugged. Dataplex Universal Catalog serves as a central metadata service where you can search, tag, and view dataset lineage—the record of where data came from and how it changed over time. Vertex ML Metadata links workflow parameters and metrics to specific model versions in the Vertex AI Model Registry, giving you full traceability across the ML lifecycle.
Freezing training data is essential for repeatable results. Create a BigQuery table snapshot as soon as raw data passes validation so training inputs never change; if you use a live, regularly updated table, tuning and evaluation results shift with each run. Snapshots keep data lineage consistent and provide a reliable point-in-time reference for debugging.
Training-serving skew occurs when production data differs from training data, causing performance drops. To prevent this, Vertex AI Feature Store works with BigQuery feature tables that include timestamp columns. Functions like ML.FEATURES_AT_TIME look up feature values exactly as they existed at training time, preventing data leakage—accidentally including future information—and keeping training data temporally aligned with production data.
BigQuery ML includes built-in monitoring to catch drift early. The ML.VALIDATE_DATA_SKEW function checks whether training feature distributions differ from serving data, while ML.VALIDATE_DATA_DRIFT watches for changes in feature distributions over time. These checks let teams spot problems and trigger retraining before model accuracy degrades.
Storage format and partitioning choices directly affect cost and performance for training and batch prediction. BigQuery and Cloud Storage are the primary services for these optimizations. Efficient columnar formats like Parquet or Avro provide compression to reduce storage costs and support schema evolution, allowing data structures to change without breaking pipelines. These formats suit large-scale, read-heavy ML operations; when storing intermediate data in Cloud Storage, they speed up processing jobs run by services like Dataflow or Dataproc.
Within BigQuery, partitioning is a primary optimization technique. Partitioning a table by a date, timestamp, or integer column lets BigQuery prune queries so it only scans relevant partitions, cutting data processed and lowering cost while speeding up training jobs that scan datasets repeatedly. Clustering tables within partitions on specific feature columns improves the performance of joins and filtering during feature engineering.
To keep data reproducible, use immutable BigQuery table snapshots of prepared datasets before training or prediction runs, ensuring data never changes and providing a reliable reference for debugging. Always reference these snapshots instead of live tables so source data updates do not affect ML results.
Data pipelines should take advantage of these optimizations. Managed services like Dataflow (for Apache Beam pipelines) and Dataproc (for Spark jobs) efficiently read and write optimized formats in Cloud Storage and BigQuery. For batch ML workloads, BigQuery's FILE_LOADS method is cost-effective; for real-time serving, the Storage Write API provides exactly-once streaming ingestion directly into BigQuery.
Data transformation pipelines turn raw data into features that ML models can use, handling tasks like normalization, encoding categorical variables, handling missing values, and creating derived features. The result is clean, consistent data ready for training and serving.
Data Sources and Ingestion: Choose data sources based on needs. Google Cloud offers databases like Cloud SQL, Spanner, AlloyDB, and BigQuery for structured data, and data lakes using Cloud Storage with BigLake for less structured data. For real-time applications, use streaming platforms such as Pub/Sub, Dataflow, or Apache Kafka. Dataflow handles both batch and stream processing with ML-component integration, Cloud Run functions work well for event-driven ingestion triggered by data changes, and BigQuery is ideal for frequently accessed tabular data.
Feature Transformation and Engineering: Once data is ingested, tools like Dataflow and Dataform automate feature scaling, encoding categorical variables, and creating new features. Vertex AI Feature Store centralizes feature management so the same features can be reused across different models and projects, maintaining consistency. BigQuery ML provides automatic preprocessing during model training: it replaces missing numeric values with the mean, maps missing categorical values to an extra category, and applies one-hot encoding for categorical data or standardization for numeric columns, simplifying pipeline code.
Pipeline Orchestration and Automation: ML pipelines with many steps and dependencies need orchestration. Vertex AI Pipelines organizes pipelines as directed acyclic graphs (DAGs) of containerized tasks, while Cloud Composer provides managed Apache Airflow for more complex workflows with heavy ETL requirements. Use Dataplex Universal Catalog to discover and manage data assets across your organization, standardizing definitions without moving datasets to one location.
Reusability and MLOps: Reusing datasets and features across multiple ML projects speeds up development and cuts costs. Data products can be published and discovered through Dataplex Universal Catalog and BigQuery sharing. Implement MLOps practices that automate the model-development lifecycle, including version control for code and models, managed pipeline orchestration, and continuous monitoring to ensure production reliability.
Overview of Unstructured Data Pipeline Architecture: The foundation for a RAG system is a scalable ingestion pipeline. Data engineers first upload raw, unstructured files—like PDFs, HTML, or images—into a Cloud Storage bucket. This upload triggers an event-driven workflow: Cloud Storage automatically sends a message to a Pub/Sub topic, which then starts a processing job, often in Cloud Run. This automation ensures data flows through the pipeline without manual intervention.
Data Processing and Transformation: The triggered job processes the raw data. Document AI extracts high-fidelity text and understands structural layout from complex formats. The processed data and its metadata are saved as JSON Lines (JSONL) files back to Cloud Storage. Another Pub/Sub message triggers the next stage: the data is pulled into a managed datastore where it is parsed and chunked. Chunking breaks large documents into smaller, optimized segments, crucial for retrieving the most relevant pieces of information later.
Automatic Embedding Generation: A major advantage of using Google's managed services like Agentspace is that vector embeddings are generated automatically. These embeddings are numerical representations of text that capture semantic meaning, enabling similarity-based search. The system handles this complex step transparently, allowing engineers to focus on pipeline design rather than model configuration. Embeddings are created from parsed and chunked data, readying it for search.
Supporting Products and Services: This architecture leverages several core Google Cloud products. Dataflow provides managed Apache Beam jobs for powerful batch and streaming data transformations. Pub/Sub enables asynchronous, decoupled messaging that drives the event workflow. Cloud Storage acts as the central, low-cost data lake. Together, these services create a reliable and scalable processing pipeline.
Data Sources and Ingestion Methods: The pipeline can ingest data from many sources beyond Cloud Storage, including Google Drive, Slack, Jira, and SharePoint, using specialized connectors. To manage costs and API quotas, engineers can control the rate of embedding generation calls. The system also supports data deduplication, automatically skipping files that haven't changed since the last import to avoid wasteful reprocessing.
Parsing and Chunking Configuration: Effective preparation requires configuring parsers and chunking strategies. An OCR parser is needed for scanned documents, while a layout parser handles complex formatting. The chunk size and chunk overlap are critical parameters: chunks that are too small may lose important context, while chunks that are too large can reduce search result precision. These settings are configured when the datastore is created.
Advanced Chunking Techniques: Document chunking splits large texts into smaller, manageable pieces called chunks, with the goal of preserving semantic meaning for better retrieval. Simple fixed-size chunking can cut sentences in half. More advanced techniques include semantic chunking, which splits text at logical boundaries like paragraphs, and sliding-window chunking, which creates overlapping chunks to prevent loss of context at chunk edges.
The Role of Metadata Enrichment: Metadata enrichment adds structured, descriptive information to each chunk, such as document title, author, or section header. This metadata is stored separately from the text but is crucial for improving search relevance. During retrieval, the system uses this data to filter and rank chunks based on specific criteria, leading to more accurate and contextually appropriate information being sent to the large language model (LLM).
Impact on Retrieval Performance: The choice of chunking strategy is a direct trade-off. Semantic chunking maintains logical flow but produces variable-length chunks. Sliding-window ensures context continuity but increases storage and processing due to overlap. In Google Cloud pipelines, services like Vertex AI Search and Document AI can automate these processes. The final enriched chunks and their metadata are stored alongside their vector embeddings in databases like Spanner or BigQuery, creating an optimized knowledge base for RAG.
Generating Vector Representations: After data is chunked, it must be converted into numerical vectors using Vertex AI Embedding APIs. Engineers select specific, versioned models (like text-embedding-004) to ensure consistent embedding generation with precise dimensional widths. Alternatively, databases like Cloud SQL can use integrated extensions to generate embeddings directly.
Storing and Indexing Vectors for Search: Generated high-dimensional vectors must be stored to enable fast similarity searches. In Cloud SQL, the pgvector extension manages vector data and creates approximate nearest neighbor (ANN) indexes to speed up queries. BigQuery also supports native vector search by creating indexes on specific columns. Optimizing index parameters, such as index type and distance metric (like Cosine), is key to reducing query latency on large datasets.
Managed Retrieval Strategies: For fully managed solutions, Google offers RagManagedDb, a vector database powered by Spanner. When configuring retrieval, engineers choose between two core strategies based on needs. The k-Nearest Neighbors (KNN) strategy compares a query against all data points, guaranteeing perfect recall; this is best for smaller, recall-sensitive datasets (under ~10,000 files). The Approximate Nearest Neighbors (ANN) strategy uses mathematical approximations to enable ultra-low latency searches, critical for scaling to massive document corpora (over 10,000 files). This choice represents the fundamental trade-off between search accuracy (recall) and query speed.
To make machine learning experiments reproducible, you need to track every piece of data and every step of the pipeline. Dataplex Universal Catalog acts as a central metadata service where you can…
To make machine learning experiments reproducible, you need to track every piece of data and every step of the pipeline. Dataplex Universal Catalog acts as a central metadata service where you can…
Ingestion, Extraction, and Cleaning of Unstructured Formats
Ingestion, Extraction, and Cleaning of Unstructured Formats