Professional Machine Learning Engineer

Building models in BigQuery ML or Agent Platform AutoML (e.g., classification, regression, forecasting, and clustering) based on the business problem

Building machine learning models in this context means choosing the right type of model for your specific business goal using either BigQuery ML or the Gemini Enterprise Agent Platform. The first step is to match the problem to a model task: use classification for predicting categories (like spam or not spam), regression for predicting a continuous number (like sales revenue), forecasting for predicting future time-series values (like product demand), and clustering for finding natural groupings in your data (like customer segments). You then decide which tool to use: BigQuery ML lets you create and train these models directly using SQL on data already inside BigQuery, while the Agent Platform's AutoML provides a more automated, code-light experience for training models on tabular data. The choice often depends on where your data lives and your team's preferred workflow—SQL experts may prefer BigQuery ML, while others may opt for the guided interface of the Agent Platform.

Performing feature engineering or selection using BigQuery ML

Feature engineering and selection is the process of preparing and choosing the most useful data columns (features) to improve a model's accuracy, and BigQuery ML provides specific SQL functions to do this. You perform feature engineering by creating new, more informative features from your raw data, such as transforming a date into the day of the week or calculating aggregates. BigQuery ML automates feature selection through functions like ML.FEATURE_CROSS to combine features or ML.QUANTILE_BUCKETIZE to bin numerical values, and it can also preprocess data by automatically scaling numeric features. The workflow happens entirely within your SQL pipeline: you write a CREATE MODEL statement that includes these transformations, and BigQuery ML applies them during training, ensuring the same transformations are applied when you generate predictions later.

Generating predictions using BigQuery ML

Generating predictions is the act of using a trained BigQuery ML model to make estimates on new data, and you do this with SQL queries. Once a model is trained and evaluated, you use the ML.PREDICT function in a SQL statement, passing the new data you want predictions for. The function joins your input data with the model and returns a result table that includes the original rows plus new columns for the prediction and often the prediction probability. This process leverages BigQuery's massive parallel processing, allowing you to generate predictions on very large datasets quickly. Because the model and data are both inside BigQuery, there is no need to move data to another system, which keeps the workflow simple and secure.

Training models using Agent Platform AutoML

Training models with the Agent Platform AutoML is a process where the platform automates much of the complex work of building a high-quality machine learning model. You start by providing your labeled dataset, typically in tabular format, and specifying your target column (what you want to predict). The platform then automatically handles tasks like feature engineering, algorithm selection, hyperparameter tuning, and model training across multiple architectures. It runs a series of trials, evaluates the performance of each, and selects the best-performing model for you. This approach is designed for users who want an effective model without needing deep expertise in machine learning theory or writing extensive code, as the platform manages the technical complexity behind a user-friendly interface.

Fine-tuning Gemini models using BigQuery

Fine-tuning Gemini models using BigQuery is the process of customizing a pre-trained, large language model (like Gemini) for a specific task using your own data, all within the BigQuery environment. You start with a foundational Gemini model and provide a dataset of example prompts and desired responses that are relevant to your use case, such as customer service queries. BigQuery ML manages the fine-tuning job, which adjusts the model's internal weights to perform better on your specific examples. The fine-tuned model is then stored as a BigQuery ML model resource, which you can use just like any other BigQuery ML model—including generating predictions with ML.PREDICT—while benefiting from the advanced capabilities of the Gemini architecture tailored to your domain.

Key Facts

• BigQuery ML allows you to build classification, regression, forecasting, and clustering models using SQL on data stored within BigQuery.
• The Gemini Enterprise Agent Platform's AutoML automates model training, including feature engineering and algorithm selection, for users preferring a code-light experience.
• You generate predictions from a BigQuery ML model using the ML.PREDICT SQL function, which runs inside BigQuery on new data.
• Fine-tuning a Gemini model in BigQuery involves customizing a pre-trained large language model with your own example data to create a specialized BigQuery ML model resource.

Frequently Asked Questions

What is the main difference between building a model in BigQuery ML versus the Agent Platform AutoML?

The main difference is the workflow and interface: BigQuery ML uses SQL for model creation and training directly on data in BigQuery, ideal for data analysts. The Agent Platform AutoML provides a more automated, graphical interface that handles the technical details, ideal for users seeking a code-light experience.

When should I use BigQuery ML's feature engineering functions?

You should use BigQuery ML's feature engineering functions (like ML.FEATURE_CROSS) when you are building or training a model within a SQL workflow, as they allow you to define transformations that are automatically applied during both training and prediction, ensuring consistency.

How does fine-tuning a Gemini model in BigQuery work?

Fine-tuning a Gemini model in BigQuery involves providing your own dataset of prompt-response pairs to a base Gemini model. BigQuery ML runs a training job to adapt the model to your specific task, and the resulting fine-tuned model is stored and used within BigQuery just like any other ML model.