Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team is building a scalable feature preprocessing pipeline using Apache Beam on Google Cloud Dataflow. The pipeline must preprocess numerical features by scaling them to a fixed range [min_value, max_value] and convert categorical text columns into integer indices for model training.
To prevent training-serving skew, the exact transformation parameters (such as the computed feature boundaries and vocabulary mappings) generated during the initial training run must be deterministically preserved and reused during subsequent model evaluation and batch inference pipelines.
Which approach should the team implement in Apache Beam to meet these requirements?
This approach attempts to build custom feature transformation logic from scratch by using Apache Beam's CombineGlobally to calculate dataset-wide minimum and maximum values and extract unique vocabulary lists. These aggregated values are written out to files and subsequently distributed to worker threads as side inputs in downstream ParDo steps.
You were implementing simple scalar metadata broadcasting across non-ML batch pipelines where managed ML transformation tools were unavailable.
Apache Beam's MLTransform is a specialized framework designed to preprocess and transform datasets for machine learning workflows at scale. It natively integrates feature engineering operations—such as numerical scaling, text tokenization, vocabulary generation, and embedding creation—directly into Google Cloud Dataflow pipelines.
ScaleByMinMax(columns=[...], min_value=min_value, max_value=max_value), MLTransform computes full-dataset minimum and maximum statistics across distributed workers and scales feature values accordingly.MLTransform computes unique token vocabularies across the training dataset and maps them to distinct integer indices.write_artifact_location saves the computed statistics, vocabularies, and transformation metadata to Cloud Storage. Subsequent evaluation or inference pipelines can simply reference read_artifact_location, applying the exact same transformation parameters without re-computing statistics on new batches.ScaleByMinMax, ScaleToZScore), categorical encoding (ComputeAndApplyVocabulary), and text embeddings in a unified interface.MLTransform abstracts the complex multi-pass distributed aggregations required for feature engineering. Reusing artifacts via read_artifact_location ensures that incoming test and inference data are normalized against the exact distribution parameters established during model training, perfectly preventing data leakage and training-serving skew.
This option uses the RunInference API alongside a custom ModelHandler and applies data transformations on-the-fly inside the preprocessing callback (with_preprocess_fn) by calculating min-max boundaries dynamically across each incoming micro-batch.
RunInference is designed for executing model prediction/inference stages rather than preparing large-scale datasets for feature engineering and training.RunInference focuses on model loading, batching, and running predictions.with_preprocess_fn is intended for row-level formatting (such as converting data into PyTorch or TensorFlow tensors), not global statistical feature normalization.You needed to convert structured dictionary rows into PyTorch or TensorFlow tensors immediately before querying a pre-trained model.
This option relies on a standard Google-provided Dataflow template (such as Pub/Sub to BigQuery or Cloud Storage to BigQuery) and injects custom transformation logic using a JavaScript User-Defined Function (UDF) to calculate rolling min-max bounds and apply hash encoding.
You needed lightweight row-level cleansing, JSON key restructuring, or data filtering before loading event streams into BigQuery.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated