Prepare and test your skills
Prepare and test your skills
Configure and Execute Data Collection is the first phase of a standard machine learning (ML) project on Google Cloud. Raw data flows from source systems into central storage, where it is organized, governed, and made reusable for ML tasks. In this phase, Cloud Storage (opens in a new tab) holds unstructured files and BigQuery (opens in a new tab) holds structured tables, so later training and prediction can query data in place.
Storage for ML data collection depends on data shape, with Cloud Storage handling unstructured data and BigQuery handling structured tabular data. Agent Platform Feature Store (Feature Store) provides a centralized repository for organizing, storing, and serving ML features for both online and offline use. Data stays where it is collected so later steps can query it in place instead of copying it to a separate ML environment.
| Storage option | Best for | How it stays governed and reusable |
|---|---|---|
| Cloud Storage | Unstructured data such as images, video, and text files | Governed through BigQuery object tables and signed URLs |
| BigQuery with Feature Store | Tabular and structured datasets and engineered features | Streamlined through BigQuery, with features shared across projects and models |
| Vector Search indexes | Vector embeddings for similarity search | Managed by including metadata directly in the index |
BigQuery further supports readiness by letting teams prepare and use data without movement. BigQuery ML creates and trains models directly inside BigQuery by using SQL queries, while the BigQuery Connector for Apache Spark lets Managed Service for Apache Spark (opens in a new tab) read BigQuery tables into Spark ML workflows.
Centralized governance makes collected datasets discoverable, secure, and reusable without consolidating all data into one place. Knowledge Catalog is an intelligent data fabric that unifies distributed data and automates governance, while BigQuery sharing securely exchanges curated data assets across organizations. Together they define who owns data, who can access it, and how its quality is checked before training begins.
Exam tip: Use Cloud Storage for unstructured files governed by object tables, and use BigQuery with Feature Store for structured tables and reusable features.
Data readiness means collected datasets and features can be reused across multiple ML projects instead of rebuilt for each model. Reuse avoids redundant data engineering, accelerates model development, lowers infrastructure cost, and frees resources for other work. The reuse layer depends on the collection layer, because only governed and discoverable datasets can be safely shared.
Implement Model Training, Evaluation, and Prediction is the ordered lifecycle that turns prepared BigQuery tables into trusted predictions. BigQuery ML (opens in a new tab) lets you create, evaluate, and use machine learning models with SQL instead of Python or Java. Training data flows from BigQuery tables directly into the training query, and the resulting trained model is stored in a BigQuery dataset like a table or view.
CREATE MODEL query that reads training rows from BigQuery tables and writes a trained model into a dataset.ML.EVALUATE with the trained model and held-out test data to measure quality before trusting the model.ML.PREDICT with the trained model and new data to generate predictions for decisions.Each step depends on the one before it. Only after a model is trained can it be evaluated, and only after evaluation shows acceptable quality should the model be used to generate predictions for business use.
Training in BigQuery ML is the process of running a CREATE MODEL statement that learns patterns from rows in BigQuery and saves the result as a model object. You specify the model type in an OPTIONS clause, such as binary logistic regression to predict whether a website visitor will make a transaction from the Google Analytics sample dataset. The model can read training data from multiple BigQuery datasets, and the trained model is then stored in a destination dataset alongside tables and views.
You can run the same training query from several interfaces without changing the SQL logic:
To create a model and run inference, a user needs permission to read and write data and run jobs, typically the BigQuery Data Editor (roles/bigquery.dataEditor) and BigQuery User (roles/bigquery.user) roles. Choose BigQuery ML training when the data already lives in BigQuery and the team wants SQL-based development instead of managing a separate machine learning environment.
Evaluation in BigQuery ML is the process of testing a trained model on data it did not train on by using the ML.EVALUATE function. Evaluation input flows from two sources into the function: the trained model object and a test table with known outcomes. The function returns a result table with quality scores and loss statistics that show how far the model predictions were from the true values.
The correct metric depends on the type of prediction the model makes:
| Metric | Applies to | What it tells you |
|---|---|---|
| Accuracy | Classification, such as logistic regression predicting purchase or no purchase | Share of test rows the model labeled correctly; higher means better classification |
| Mean squared error | Regression, such as predicting a numeric amount | Average squared difference between predicted and actual numbers; lower means closer predictions |
Use accuracy when the answer is a category and mean squared error when the answer is a number. A model that performs well on its training data but poorly on test data is not ready for predictions, so evaluation protects downstream decisions from weak models.
Prediction in BigQuery ML is the process of applying a trained model to new rows by using the ML.PREDICT function. Prediction input flows from two sources into the function: the trained model object and a table of new input rows. The output is a new result table that contains the original columns plus predicted columns, such as predicted label and probability for each visitor or predicted purchases per user.
The same ML.PREDICT pattern also works for imported TensorFlow and XGBoost models that were loaded into BigQuery ML with CREATE MODEL. When predictions are needed outside of BigQuery, the model can be exported to Cloud Storage or registered to the Model Registry in Gemini Enterprise Agent Platform (Agent Platform) for online prediction from an endpoint. You are charged for training compute, for queries against models, and for storage of trained models, while CREATE MODEL statements for imported models do not process bytes and do not incur BigQuery charges.
Exam tip: BigQuery ML is not available in the Standard edition, so training, evaluation, and prediction queries require an edition that supports BigQuery ML.
Utilize GCP for Data Collection and Preparation is the first phase of a standard ML project lifecycle that gathers, cleans, and stores data for training. Raw data is gathered from source systems, cleansed and formatted into a consistent shape, and stored where a model can be trained and evaluated. In this workflow, Cloud Storage (opens in a new tab) holds raw and archival data, Dataflow (opens in a new tab) cleans and transforms data in motion, and BigQuery (opens in a new tab) stores prepared tables ready for training and prediction.
BigQuery is the Google Cloud data warehouse for storing structured, training-ready tables for ML. Data flows from source systems into BigQuery tables, where it can be queried with SQL and used directly for training and forecasting with BigQuery ML. Examples include multivariate and univariate forecasting with TimesFM models using the AI.FORECAST function, time-series forecasting with ARIMA_PLUS models, and text generation with the AI.GENERATE function using a hosted Gemini model.
Continuous collection keeps BigQuery tables fresh for training and re-training. Streaming data flows from Pub/Sub into BigQuery for low-latency use cases, changed data flows continuously from a MySQL database into a BigQuery table, and near real-time analytics flows from Datastream through Dataflow job templates into BigQuery. This pattern makes BigQuery the destination of the collection pipeline, while other services handle movement and transformation before data lands.
Dataflow is the Google Cloud service for processing data with Apache Beam pipelines before it is used for training. A pipeline reads raw records from a source, applies cleansing and shaping logic, and writes clean rows to a destination such as BigQuery. Dataflow handles both batch cleanup of historical files and streaming cleanup of live events, so the same preparation logic can serve initial training and ongoing prediction.
Common preparation work in a Dataflow pipeline includes:
Data flows from the source system to each pipeline step in order, and only after cleansing does clean data flow to the BigQuery destination. Specialized processing is also supported, such as using GPUs on Dataflow to process Landsat 8 satellite images and render them as JPEG files.
Cloud Storage is the Google Cloud object store for archiving original, unmodified copies of data used in an ML project. Raw files, database exports, and backups flow from source systems into Cloud Storage buckets, where they are kept as the durable record even after cleaned copies move forward. One example is database backup and restore, where a SQL Server backup is transferred using Cloud Storage to clone or preserve the source data.
Keeping the raw copy separate from the prepared copy protects the project lifecycle. If training data is later found to be filtered incorrectly or formatted the wrong way, the team can return to the archived raw files in Cloud Storage, fix the Dataflow logic, and rebuild the BigQuery tables. Choose Cloud Storage when the goal is low-cost, durable archival of original data, not interactive querying or direct model training.
How Collection and Preparation Fit Together is the ordered workflow that moves data from archival to training through three services. The workflow preserves the raw record, cleans it in a pipeline, and stores the result where models can use it directly.
A left-to-right process flow with six numbered steps: archive raw files in Cloud Storage, cleanse with Dataflow, store prepared tables in BigQuery, train with CREATE MODEL, evaluate with ML.EVALUATE, and predict business decisions with ML.PREDICT.
| Service | Primary role | Choose when |
|---|---|---|
| BigQuery | Prepared table storage and in-database ML | Data is cleansed and needs SQL access plus training and forecasting |
| Dataflow | Data processing and cleansing | Data needs filtering, mapping, joining, or streaming transformation |
| Cloud Storage | Raw data archival | An original backup or file must be preserved before preparation |
Exam tip: Keep raw files in Cloud Storage and training-ready tables in BigQuery; Dataflow is the pipeline that transforms the first into the second.
Use Cloud Storage for unstructured raw files such as images, video, and text files and for low-cost durable archival of original data, and use BigQuery for structured training-ready tables that can be queried with SQL and used directly for training. Dataflow pipelines transform raw data from Cloud Storage into clean BigQuery tables, and keeping the raw copy separate lets teams fix pipeline logic and rebuild tables if filtering or formatting was incorrect.
Training runs CREATE MODEL that reads training rows from BigQuery tables and writes a trained model into a dataset, evaluation runs ML.EVALUATE with the trained model and held-out test data to measure quality, and prediction runs ML.PREDICT with the trained model and new data to generate predictions. Each step depends on the one before it, so a model can only be evaluated after training and should only be used for predictions after evaluation shows acceptable quality.
Use accuracy for classification such as logistic regression predicting purchase or no purchase, where higher means better classification because it is the share of test rows labeled correctly. Use mean squared error for regression such as predicting a numeric amount, where lower means closer predictions because it is the average squared difference between predicted and actual numbers.
Your team is preparing to build a machine learning model to predict customer churn. You have collected raw historical transaction records and uploaded the CSV files into a Google Cloud Storage bucket. You now plan to ingest and organize this structured data into BigQuery for data exploration and machine learning preparation.
What foundational resource must you create in your Google Cloud project before you can load the data into BigQuery tables?
Last updated