Prepare and test your skills

Prepare and test your skills

The Value of Data is the idea that data becomes a strategic asset when it is governed, shared, and reused to guide decisions and create new products. Raw records from sales, operations, and customer interactions gain value only after they are structured, enriched with business context, secured, and made discoverable for many teams. On Google Cloud, that journey centers on governed sharing with Knowledge Catalog (opens in a new tab), large-scale analysis with BigQuery (opens in a new tab), and prediction with machine learning.
Governed data as a foundation means clear policies, procedures, and roles keep data secure, accurate, available, and usable through its lifecycle. A governance framework improves data quality, strengthens security, improves utilization, reduces redundancy, and helps manage artificial intelligence (AI) and machine learning (ML) workloads cost-effectively at scale. Knowledge Catalog automates governance without moving data into one central copy, so data stays where it lives while teams discover and reuse the same trusted assets.
Teams organize distributed data with logical data lakes and zones, control access with Google Groups and Knowledge Catalog roles, improve quality with auto data quality, and add business meaning with aspects. BigQuery sharing extends trusted data across organizations through data exchanges, listings of curated assets, and data clean rooms for sensitive collaboration. Reuse avoids repeated engineering because data products published once can be found through Knowledge Catalog and BigQuery sharing and used in many models, while Feature Store reuses tabular features, Cloud Storage holds unstructured files governed as object tables, and Vector Search indexes manage vector embeddings.
Capturing, storing, and preparing data is the ordered flow that turns raw events into query-ready tables under governance. Capture uses batch loading of Avro, Parquet, ORC, CSV, JSON, Datastore, and Firestore files, continuous streaming through the Storage Write API, Pub/Sub for event streams, BigQuery Data Transfer Service for recurring loads, Dataflow (opens in a new tab) for batch and streaming pipelines, and Dataproc for managed Spark and Hadoop work. Storage keeps data in columnar format built for analytical queries, presents it as tables in datasets, supports ACID transactions, and replicates it automatically for high availability.
Preparation happens in a compute layer that scales independently from storage over a petabit-scale network. Analysts use ANSI-standard SQL with joins, aggregations, nested fields, and spatial functions, plus BigQuery DataFrames for pandas-compatible Python and BigQuery Studio notebooks. Federated queries and external tables let BigQuery query data where it lives in Cloud Storage, Bigtable, Spanner, or Google Sheets. The three services divide insight work by type: BigQuery analyzes large datasets to surface trends, Looker shares one trusted view of metrics across teams, and Vertex AI builds predictive and generative models beyond SQL.
Lakes, databases, and warehouses work together by moving data from raw storage to transactions to analysis. A data lake ingests and stores raw data from many systems for later processing, commonly using Cloud Storage buckets with schema-on-read where structure is applied when data is read. A database runs day-to-day operational transactions on structured current records, while a data warehouse analyzes organized historic data for reporting, commonly using BigQuery on Google Cloud.
Operational databases feed the warehouse one way through extract, transform, and load (ETL) or extract, load, and transform (ELT) pipelines using Pub/Sub, Dataflow, and Spark. A typical batch flow runs in order:
| Paradigm | Primary purpose | Data and schema |
|---|---|---|
| Data lake | Ingest and store data from many systems | Raw and unstructured objects; schema-on-read |
| Database | Run applications and transactions | Structured records; transactions and consistency |
| Data warehouse | Analyze and report on integrated data | Structured tables; defined schemas; columnar storage |
Creating value from all data sources means organizing existing records, capturing new events, and adding outside context in one governed flow. New events flow from producers through Pub/Sub to Dataflow to BigQuery, existing files are organized in BigQuery and Cloud Storage and activated with business intelligence tools, and external datasets are joined through live sharing. Each path ends in dashboards, predictions, and shared insights.
Pub/Sub ingests real-time events where a producer publishes to a topic and subscribers receive independently, which lets new sources be added without rewriting processing. Dataflow then cleans, joins, or enriches the stream with exactly-once processing and built-in ML inference, and BigQuery stores results as warehouse tables with fast streaming writes. For external data, Analytics Hub (opens in a new tab) sharing lets a provider organize an exchange containing listings and lets a consumer subscribe to create a linked dataset, which is a live pointer to the provider asset with zero-copy sharing and no replication.
Governance, quality, and lineage keep the data value chain trustworthy from sources to consumers. The value chain flows in order from transactional sources to temporary staging for cleaning and enrichment, then to warehouse core tables that hold the source of truth, then to BigQuery views that encapsulate business logic, and finally to consumers such as dashboards, notebooks, and operational systems that depend on views and never read core tables directly. ETL transforms data inside the pipeline before loading, while ELT loads raw data first and transforms inside the warehouse with SQL.
Trust controls include Viewer’s Credentials where each viewer uses their own access, Owner’s Credentials where viewers use the owner’s authorization, and Service Account Credentials for automated access. Sensitive Data Protection discovers and de-identifies sensitive fields, Identity and Access Management (IAM) with column-level and row-level security hides data by viewer, and VPC Service Controls place a boundary around shared assets. Knowledge Catalog adds metadata that describes what an asset is, plus data lineage that shows origin, transformations, and destinations at column level.
Exam tip: A data lake stores raw data for later processing, a database runs application transactions, and a data warehouse analyzes modeled data — ETL transforms before loading, while ELT loads raw data first and transforms inside the warehouse.
Continue with this study guide: The Value of Data
Google Cloud Data Management Solutions are the storage, database, and pipeline services chosen by data shape and whether the work is operational or analytical. Cloud Storage (opens in a new tab) holds unstructured objects such as files, images, backups, and static assets in buckets, BigQuery holds large analytical datasets for queries and reporting, and Cloud SQL and Spanner hold operational records for day-to-day relational transactions. Pub/Sub, Dataflow, Dataproc, and Dataplex then move and govern data before analysis.
Core storage and database services divide work by structure, scale, and durability needs. Cloud Storage stores each object with a globally unique ID plus system and user metadata in a flat hierarchy inside buckets, with versioning and soft delete for rollback and replication across regions for durability. BigQuery separates storage from compute so each scales independently, with compressed columnar storage in Colossus and query execution in Dremel using slots that bundle CPU and RAM. Cloud SQL provides regional MySQL, PostgreSQL, and SQL Server for row-level transactions, while Spanner provides global-scale relational transactions that stay available across regions.
| Service | Data structure | Choose when |
|---|---|---|
| Cloud Storage | Objects in buckets | Data is files, media, backups, or static web assets |
| BigQuery | Columnar analytical tables | Work is large-scale queries and reporting |
| Cloud SQL | Regional relational tables | Work is operational transactions with a specific engine |
| Spanner | Global relational tables | Transactions must scale and stay available globally |
Block, shared file, and object storage differ in how application servers attach to data and survive outages. Compute Engine provides virtual machines where teams control the operating system, Google Kubernetes Engine (GKE) runs containers with automated deployment, and Cloud Run runs code serverless. Persistent Disk provides block volumes attached to a single virtual machine, Filestore provides shared files mounted by many virtual machines, and Cloud Storage provides objects accessed from inside and outside Google Cloud.
Replication decides what survives a zone or region outage. Zonal Persistent Disk stores data in one zone, while regional Persistent Disk replicates synchronously across two zones so every write completes in both places. Filestore regional instances replicate synchronously across three zones for shared configuration and logs, while regional Cloud Storage buckets replicate synchronously across zones and dual-region and multi-region buckets store data redundantly in two or more locations with synchronous metadata and asynchronous data.
Relational, non-relational, and analytical models describe how data is structured and queried. A relational database stores highly structured data in tables of rows and columns with links across tables, maintained by a relational database management system with indexes for faster lookups. Structured Query Language (SQL) manages that data with statements such as CREATE, INSERT, and SELECT. Non-relational databases, also called NoSQL databases, store data without forcing every record into fixed tables for flexibility and automatic scale.
Each of the six main services matches a different pattern for transactions, operations, or analytics:
Exam tip: Choose Cloud SQL when a regional managed database is enough; choose Spanner when relational transactions need global consistency and horizontal scale.
BigQuery as a serverless warehouse means teams focus on data instead of servers, scaling, and maintenance with no provisioning or manual scaling. A query flows from the user to the serverless compute layer, the compute pulls data from separate storage over a petabit-scale network, and results flow back without provisioning servers. BigQuery can query terabytes in seconds and petabytes in minutes, supports joins, nested fields, aggregations, spatial functions, and federated queries, and stores BigQuery ML models in datasets like tables so training data never leaves the warehouse.
BigQuery Omni extends this to multicloud by running compute near data stored in Amazon Simple Storage Service (Amazon S3) or Azure Blob Storage without moving data first. The workflow follows a fixed order: first connect BigQuery to the external storage, then create a BigLake table that references the data in place, then run analytics in a BigQuery Omni region near that storage, and optionally join with local tables or move results into BigQuery. Because data stays in place, organizations avoid movement and egress costs while reducing silos.
Storage classes and migration choices balance cost, access frequency, and change effort. Cloud Storage classes match how often objects are read: Standard is the default for frequent access, Nearline and Coldline are for rarely accessed data, and Archive is the cheapest and most durable class for legal and long-term retention. Colder storage costs less to keep but more to read, and Nearline, Coldline, and Archive add minimum durations where the account is charged even if the object is deleted early.
Migration strategies describe how much change happens when moving to Google Cloud, secured with Identity and Access Management (opens in a new tab) and monitored for product availability in supported products and locations (opens in a new tab):
| Strategy | Change required | Choose when |
|---|---|---|
| Lift-and-shift | Move schema and data as-is | Speed and continuity matter most |
| Replatform | Move to managed service | Reliability is needed without rewriting |
| Modernize | Redesign schema | Scale and long-term efficiency justify effort |
Tools move bulk files, live databases, and servers with less downtime. Storage Transfer Service brings data to Cloud Storage over the network, Transfer Appliance ships very large moves over 20 TB up to 1 petabyte, and BigQuery Data Transfer Service automates scheduled moves into BigQuery. Database Migration Service migrates SQL databases to Cloud SQL, while export and import through Cloud Storage moves Bigtable, Firestore, and Spanner data.
Continue with this study guide: Google Cloud Data Management Solutions
Making Data Useful and Accessible is the work of turning stored data into timely decisions, consistent reports, and immediate actions. Smart analytics finds patterns in large datasets, business intelligence shares those patterns as trusted visuals, and streaming analytics acts on events as they happen. On Google Cloud, BigQuery analyzes, Looker presents with shared definitions, and Pub/Sub with Dataflow keeps flows continuous.
Smart analytics on Google Cloud centers on BigQuery as a fully managed, serverless data warehouse with no infrastructure to manage. BigQuery separates storage from compute so each side scales independently: the storage layer keeps data in columnar format with automatic replication, while the compute layer runs distributed queries without slowing ingestion. BigQuery can query terabytes in seconds and petabytes in minutes and accepts batch loads, automated transfers, and continuous streaming inserts.
Analysis happens where the data lives using ANSI-standard SQL, Python with BigQuery DataFrames, geospatial functions, and federated queries to Cloud Storage, Bigtable, Spanner, or Google Sheets. BigQuery ML builds and trains machine learning models with SQL directly inside the warehouse, so data flows from tables into training and the trained model scores new rows without moving data. Governance with IAM, lineage, data quality, and Knowledge Catalog keeps this analysis trusted.
Governed business intelligence turns consolidated data into visuals and metrics that every team can trust. Data flows from sales systems, operations databases, and storage into BigQuery, then from BigQuery into Looker for exploration and sharing. Looker applies a unified data model that defines each metric once, while Connected Sheets brings billions of rows into Google Sheets and Looker Studio (opens in a new tab) builds simple charts and reports. Because everyone uses the same definitions, dashboards, scorecards, and scheduled reports show the same numbers.
Looker democratizes access through LookML, the semantic modeling language that creates a single source of truth. A model controls which tables can be queried together and how joins behave, a view controls how columns become dimensions and measures, and an Explore combines a model and views into a starting point for questions. When a user builds a question, Looker combines join logic with field definitions to generate SQL and sends it to the database, so non-technical users select fields while Looker writes correct SQL against live data.
Streaming analytics captures and processes event data as it arrives instead of waiting for batch jobs. Data flows from apps, websites, servers, and sensors to Pub/Sub for capture, from Pub/Sub to Dataflow for processing, and from Dataflow to BigQuery, Bigtable, and Cloud Storage for immediate use. Pub/Sub is asynchronous messaging with about 100 milliseconds latency where publishers send to a topic and subscribers receive independently, while Dataflow filters, aggregates, enriches, and runs ML inference with exactly-once processing by default.
Immediate insight changes operations because action happens while the event is still relevant:
The event pipeline follows a fixed order: first ingest in Pub/Sub where publishers send JSON events to a topic, then process in Dataflow where a streaming job cleans and enriches and autoscales workers to match volume, then store and analyze in BigQuery where transformed rows land with a schema for alerts and dashboards.
A left-to-right process flow showing event sources publishing to Pub/Sub for ingestion, Dataflow for processing, BigQuery for storage, then dashboards and alerts for use.
Modern pipelines are modernized by splitting ingestion from processing into two managed stages. Pub/Sub ingests and distributes real-time events from many producers, and Dataflow transforms those events and delivers them to storage and analytics. Publishers send without waiting, Dataflow reads as a source and writes to a destination, and dashboards and warehouses receive continuous updates with low operational effort. Ready-made Dataflow templates for Pub/Sub to Cloud Storage and BigQuery let teams reuse pipelines without writing Apache Beam code.
Datastream adds serverless change data capture (CDC) that synchronizes operational databases with minimal latency. Change events flow from sources such as MySQL, Oracle, PostgreSQL, SQL Server, MongoDB, Spanner, Salesforce, and ServiceNow into BigQuery for analysis and Cloud Storage for Avro or JSON streams. A stream depends on connection profiles for source and destination connectivity, and streams, profiles, and private configurations must be in the same region. Starting moves a stream from Not started to Starting to Running, pausing moves it to Draining to Paused, and resuming returns it to Running.
Exam tip: Pub/Sub ingests and distributes messages but does not transform them; Dataflow reads from Pub/Sub and performs the transform and delivery to BigQuery, Bigtable, or Cloud Storage.
Continue with this study guide: Making Data Useful and Accessible
A left-to-right process flow showing the five data value chain stages: transactional sources, temporary staging for cleaning and enrichment, warehouse core tables as the source of truth, BigQuery views encapsulating business logic, and consumers such as dashboards, notebooks, and operational systems that read only views.
A data lake ingests and stores raw data from many systems for later processing, commonly using Cloud Storage buckets with schema-on-read where structure is applied when data is read. A database runs day-to-day operational transactions on structured current records, while a data warehouse analyzes organized historic data for reporting, commonly using BigQuery on Google Cloud.
Choose Cloud SQL when a regional managed database is enough for operational transactions with MySQL, PostgreSQL, and SQL Server. Choose Spanner when relational transactions must scale and stay available globally with global consistency and horizontal scale.
Pub/Sub ingests and distributes real-time events where publishers send to a topic and subscribers receive independently with about 100 milliseconds latency, but it does not transform them. Dataflow then filters, aggregates, enriches, and runs machine learning inference with exactly-once processing, and BigQuery stores the transformed rows as warehouse tables with fast streaming writes for alerts and dashboards.
A provider organizes an exchange containing listings of curated assets and a consumer subscribes to create a linked dataset. The linked dataset is a live pointer to the provider asset with zero-copy sharing and no replication.
Last updated