Prepare and test your skills
Prepare and test your skills
Implementing data analysis techniques in Jupyter notebooks means cleaning data, transforming it with code, and exploring it to find trends and patterns in one interactive place. Code written in a notebook cell runs on scalable processing resources, and the results return to the notebook for inspection and charting. A notebook environment such as BigQuery Studio or JupyterLab connects to that backend, and conda kernels supply the Python, R, or framework libraries needed for analysis.
Managed Service for Apache Spark (opens in a new tab) (formerly Dataproc on Compute Engine and Google Cloud Serverless for Apache Spark) offers two execution models for data processing tasks. The serverless model runs jobs without provisioning or scaling infrastructure, while the cluster-based model runs jobs on a dedicated set of virtual machines that the analyst fully controls. Data flows from the notebook to the selected backend, which provisions compute, runs the Spark code, and returns results to the notebook.
| Model | Who manages infrastructure | Core units |
|---|---|---|
| Serverless | Managed Service for Apache Spark provisions resources on demand and tears them down after the run | Batches for submitted jobs and interactive sessions for live exploration |
| Cluster-based | Analyst controls cluster size, machine types, and configuration | Clusters that run jobs and workflow templates that orchestrate job dependencies |
Choose the serverless model when the goal is fast exploratory analysis without managing servers. Choose the cluster-based model when the work needs full control over machine types, installed components, or long-running dedicated capacity.
An interactive session is a live, on-demand serverless workspace for exploratory data analysis, typically used from inside a Jupyter notebook. The session stays active while the analyst runs queries and develops code, without provisioning cluster or notebook resources separately. A session template is a reusable configuration that defines the settings for those sessions, including Spark properties and library dependencies.
A template must specify runtime version 2.3 or later and notebook type Spark Connect to work with BigQuery Studio notebooks. When the notebook creates a session from the template, the session inherits those settings and provides the engine that executes DataFrame transformations and explorations.
BigQuery Studio and Workbench are unified analytics and notebook environments where analysts write Jupyter notebook code and run it on a Managed Service for Apache Spark cluster or serverless session. The notebook acts as the frontend where code is written, while the cluster or session acts as the backend engine that processes large datasets. The Managed Service for Apache Spark JupyterLab Plugin adds a control panel inside JupyterLab to browse, create, and manage clusters and submit jobs without leaving the notebook interface.
A workflow template can extend this notebook work into a repeatable pipeline by defining a series of jobs and their dependencies. For example, the template can require a data cleaning job to complete successfully before a machine learning job starts, and it can run on an existing cluster or on a temporary cluster that is created for the workflow and deleted afterward.
A Spark Connect session in a BigQuery Studio notebook is created with the Managed Service for Apache Spark Connect Python Connector, a Python library that links the notebook to a serverless session. The connector builds a DataprocSparkSession from a Session object whose configuration comes from an existing session template. The session depends on four values:
PROJECT_ID: the Google Cloud project ID shown in Project info on the console dashboardLOCATION: the Compute Engine region where the session runsSESSION_TEMPLATE: the name of an existing interactive session templateAPP_NAME: an optional name for the sessionAfter the session is created, data transformation uses PySpark APIs for DataFrame, Functions, and Column operations. The session pre-configures the Spark BigQuery connector for DIRECT writes, which use the BigQuery Storage Write API to write data directly into BigQuery. This differs from INDIRECT writes, the default for batches, which first write data to an intermediate Cloud Storage bucket and then write it to BigQuery. Analysts can also extend the session with Python libraries by calling spark.addArtifacts with version specifiers such as textdistance==4.6.1, which installs the packages on all cluster nodes so user-defined functions can use them on workers.
Exam tip: Spark Connect supports DataFrame, Functions, and Column, but does not support SparkContext and RDD — check for a Supports Spark Connect note in the API reference.
A conda environment in an Agent Platform Workbench instance appears as a kernel in the JupyterLab interface. Analysts add a conda environment to use kernels that are not included by default, such as R and Apache Beam, or to use specific earlier versions of TensorFlow, PyTorch, or Python. The environment isolates its own pip and packages from the system installation, so different analyses can use different library versions.
Libraries in an existing kernel are changed from the JupyterLab terminal by working inside that conda environment:
conda env list to find the environment name, such as pytorch.conda activate pytorch to switch into that environment.python -c "import torch; print(torch.__version__)" to display the current library version.which pip to verify the pip belongs to the environment, such as /opt/conda/envs/pytorch/bin/pip.pip install --upgrade torch to install the new version into that environment.Some conda packages add an extra default python3 kernel that duplicates kernels in the Launcher. That duplicate is removed by deleting its kernel directory, such as /opt/conda/envs/CONDA_ENVIRONMENT_NAME/share/jupyter/kernels/python3, before creating the new named kernel.
A notebook schedule (opens in a new tab) automates repeated execution of a notebook file, and its history is checked from the Google Cloud console or the JupyterLab Notebook Executor. The Schedule details page shows the frequency settings and the last five executions, and opening an execution result launches the executed notebook in a new browser tab. Deleting a schedule stops future runs but does not delete the executions already generated from that schedule.
Unused resources are deleted to avoid charges, with data flowing from the control plane delete action to removal of the underlying virtual machine or project resources. Deleting the Workbench instance removes that instance, while deleting the project removes everything in the project and permanently loses custom project IDs. To preserve URLs that use the project ID, such as an appspot.com URL, analysts delete selected resources inside the project instead of deleting the whole project.
Visualizing data with Python libraries means turning rows and columns in a pandas DataFrame into charts and graphs inside a notebook. The DataFrame holds the cleaned data, Python code in a notebook cell creates the chart, and the chart appears directly below the cell so trends and distributions are easy to see. This workflow in Colab Enterprise (opens in a new tab) lets an analyst explore data and share insights in the same place where the code runs.
Creating a chart from a DataFrame in a notebook follows a simple, ordered flow where data moves from storage to an image on the screen. The notebook kernel runs Python code that reads data into a pandas DataFrame, selects the columns to plot, and passes those columns to a plotting library. The library then renders the visual output in the cell output area, so code, data, and chart stay together.
Because the chart is generated from live DataFrame values, changing the DataFrame and re-running the cell updates the visual immediately.
Matplotlib is the foundational Python plotting library for building basic charts such as line, bar, scatter, and histogram plots. Code using Matplotlib specifies the figure, the x and y data from the DataFrame, and labels for axes and titles. Matplotlib offers detailed control over every part of a figure, which makes it a good choice when a custom layout or fine-grained formatting is needed.
Seaborn is a higher-level library built on Matplotlib that simplifies statistical charts and produces polished styling by default. Seaborn reads DataFrame columns directly and can show distributions, relationships, and categories with short function calls for histograms, box plots, heatmaps, and regression plots. Choose Seaborn when the goal is to reveal patterns in distributions and relationships quickly without styling each element by hand.
Altair is a declarative visualization library that creates interactive charts from DataFrames with concise code. An Altair chart maps DataFrame fields to visual channels such as x-position, y-position, color, and size, and the library handles the rendering. Altair output supports tooltips, zooming, and selection in the notebook, so choose Altair when interactivity helps explore trends across many points or categories.
| Library | Primary style | Choose when |
|---|---|---|
| Matplotlib | Low-level control of figures and axes | The chart needs custom layout or precise formatting |
| Seaborn | Statistical plots with clean default themes | The goal is to show distributions, correlations, or category comparisons quickly |
| Altair | Declarative, interactive charts | The audience benefits from hovering, filtering, or zooming the data |
Choosing the right chart type means matching the shape of the DataFrame data to the question being asked. Time trends flow from an ordered time column to a line chart, category comparisons flow from a name column and a value column to a bar chart, and distributions flow from a single numeric column to a histogram or box plot. Relationships between two numeric columns are best shown with a scatter plot, where each point represents one row.
Common selections from DataFrames include:
Clear titles, axis labels, and consistent scales help the reader connect the visual back to the underlying DataFrame columns.
Exam tip: Use a line chart for trends over time, a bar chart for comparing categories, and a histogram or box plot for showing how values are distributed.
Performing data analysis with BigQuery from a notebook means running SQL directly against cloud datasets and working with the results in Python in the same place. Colab Enterprise (opens in a new tab) notebooks are BigQuery Studio code assets powered by Dataform that support SQL, Python, and common data packages. A notebook connects to a runtime for compute, sends SQL queries to BigQuery for processing, and receives the result set back into the notebook where it can be explored and visualized to find trends and insights.
A Colab Enterprise notebook is a BigQuery Studio code asset that runs analysis code on an attached runtime. The runtime provides the compute that executes cells, while BigQuery (opens in a new tab) stores and processes the table data, so the notebook depends on both services to work. Creating and connecting a notebook requires access to the project, including roles such as BigQuery User (roles/bigquery.user) and Colab Enterprise User (roles/aiplatform.colabEnterpriseUser).
A new notebook is created from BigQuery and then connected to the default runtime in this order:
Only after the notebook shows a connection to the runtime can its cells be run.
A SQL cell (opens in a new tab) is a notebook cell that sends a SQL statement from the notebook to BigQuery for execution. The cell is added through the Insert code cell options menu by selecting Add SQL cell, which places a new SQL cell in the notebook. SQL entered in the cell follows BigQuery SQL syntax and can reference a Python variable by enclosing the variable name in braces, such as {my_threshold} in SELECT * FROM my_dataset.my_table WHERE x > {my_threshold}.
Running the cell sends the query from the notebook runtime to BigQuery, and BigQuery returns the result set to the notebook. Clicking Run cell starts this flow, so data flows from BigQuery tables to the query engine and then back to the notebook as a result. This pattern lets an analyst filter, aggregate, and compare large tables without moving the source data.
Exam tip: Enclose a Python variable name in braces to use it inside a SQL cell, such as WHERE x > {my_threshold}.
Query results in Colab Enterprise are automatically saved as a BigQuery DataFrame with the same name as the title of the SQL cell. The BigQuery DataFrame lives in the notebook runtime and can be used by later Python cells for cleaning, calculation, and charting. Because later cells depend on the SQL cell output, the SQL cell must run successfully first before Python analysis of its DataFrame is meaningful.
Result DataFrames support the core analysis workflow in the notebook:
The Data Science Agent (opens in a new tab) can also generate Python or SQL in the notebook for these tasks, but the generated code only runs in the notebook runtime.
{my_threshold}, and its results are saved as a BigQuery DataFrame named for the SQL cell title.Serverless execution provisions resources on demand for batches and interactive sessions and tears them down after the run, while cluster-based execution runs jobs on analyst-controlled clusters and workflow templates. Choose serverless for fast exploratory analysis without managing servers and choose cluster-based when you need full control over machine types, installed components, or long-running dedicated capacity.
Use Matplotlib when the chart needs custom layout or precise formatting, Seaborn when the goal is to show distributions, correlations, or category comparisons quickly, and Altair when the audience benefits from interactive hovering, filtering, or zooming. Matplotlib provides low-level control of figures and axes, Seaborn simplifies statistical plots with clean default themes, and Altair creates declarative interactive charts from DataFrame fields mapped to visual channels.
The session template must specify runtime version 2.3 or later and notebook type Spark Connect to work with BigQuery Studio notebooks. When the notebook creates a session from the template, the session inherits those settings and provides the engine that executes DataFrame transformations and explorations.
Query results are automatically saved as a BigQuery DataFrame with the same name as the title of the SQL cell. The DataFrame lives in the notebook runtime and can be used by later Python cells for cleaning, calculation, and charting, so the SQL cell must run successfully first.
A data analyst has loaded transaction records from BigQuery into a pandas DataFrame within a Colab Enterprise notebook. The dataset includes continuous numerical transaction amounts across several store departments. The analyst needs to visualize the statistical distribution of transaction amounts for each department to easily compare the median, interquartile ranges, and identify potential outliers.
Which Python visualization approach should the analyst use to meet these requirements?
Use Seaborn's lineplot() function (sns.lineplot(data=df, x='department', y='amount')) to plot trend lines across departments.
Use Seaborn's boxplot() function (sns.boxplot(data=df, x='department', y='amount')) to generate side-by-side box plots for each department.
Use Matplotlib's pie() function (plt.pie(df['amount'])) to show the proportion of every individual transaction value.
Use Altair's mark_bar() to create a stacked bar chart displaying the cumulative sum of transaction amounts per department.
Last updated