Prepare and test your skills

Prepare and test your skills

Data as a strategic asset means data is the most valuable resource for changing how an organization operates, decides, and grows. Raw records from sales, operations, and customer interactions become valuable only after they are structured, enriched with business context, governed, and reused. Higher data maturity lets analytics and machine learning (ML) train on complete and current information instead of fragmented copies.
Data as the foundation for digital transformation means governed, high-quality data supports every new process, product, and prediction. A data governance framework manages data through its lifecycle with clear policies, procedures, and roles. This framework improves data quality, strengthens security, improves utilization, and reduces redundancy. It also helps manage artificial intelligence (AI) and ML workloads cost-effectively at scale.
Governing, sharing, and reusing data means organizing distributed data so many teams can discover and reuse the same trusted assets. Knowledge Catalog (opens in a new tab) is a fully managed, scalable metadata management service that automates governance without moving data into one central copy. Because data stays where it lives, distribution and maintenance costs fall while discovery and reuse increase. Teams organize data with logical data lakes and zones, control access with Google Groups and Knowledge Catalog roles, improve quality with auto data quality, and enrich technical metadata with business metadata using aspects.
BigQuery (opens in a new tab) sharing extends that trusted data across organizations through data exchanges, listings of curated assets, and data clean rooms for sensitive collaboration. Reuse avoids repeated data engineering because data products published once can be discovered through Knowledge Catalog and BigQuery sharing and used in many models. Tabular features are reused through Feature Store, unstructured files stay in Cloud Storage and are governed with object tables, and vector embeddings are managed in Vector Search indexes.
Turning governed data into analysis and personalization means using complete datasets to predict trends, tailor experiences, and build new products. BigQuery is a fully managed, scalable data platform with built-in ML for analytic queries over large datasets. Looker provides business intelligence on a trusted, current version of data, while Data Studio communicates results as data stories. Google Cloud Cortex Framework accelerates this work with reference architectures and deployable solutions for Marketing, Sales, Supply Chain, Manufacturing, Finance, and Sustainability.
ML operations (MLOps) keeps those predictions reliable by managing the path from development to deployment and monitoring. Version control with tools such as Git ensures reproducibility, continuous integration and continuous delivery (CI/CD) with Cloud Build automates testing and deployment, Cloud Monitoring and Cloud Logging trigger alerts when performance degrades, and retraining restores quality.
Exam tip: Knowledge Catalog governs and unifies data without moving it, while BigQuery sharing exchanges curated assets across organizations; reuse happens when data products are discovered in both.
From siloed byproduct to strategic asset is the shift from leaving operational records unused to flowing them into governed cloud environments. Transactional workloads run daily operations such as sales and financial processing, while analytics workloads transform and visualize data to aid decision-making. In the hybrid pattern, transactional work stays in the private environment, raw data is extracted and loaded into Google Cloud for analysis, and some results are fed back. Analytics fits the cloud well because bursty workloads scale dynamically without upfront hardware, and inbound data movement into Google Cloud might be free of charge.
The path from raw data to business value is the ordered flow that turns records into decisions on Google Cloud. Raw data is captured from source systems, stored reliably, prepared for analysis, then analyzed and interpreted. BigQuery (opens in a new tab) is the fully managed, AI-ready data platform at the center of this path, with serverless architecture and no infrastructure to manage.
Capturing, storing, and preparing data means moving raw events and files into query-ready tables under governance. Capture uses batch loading of Avro, Parquet, ORC, CSV, JSON, Datastore, and Firestore files, continuous streaming through the Storage Write API, Pub/Sub for event streams, BigQuery Data Transfer Service for recurring loads, Dataflow for batch and streaming pipelines, and Dataproc for managed Spark and Hadoop work. Storage keeps data in a columnar format built for analytical queries, presents it as tables in datasets, supports ACID transactions, and replicates it automatically for high availability.
Preparation cleans and organizes that stored data in the compute layer, which scales independently from storage over Google's petabit-scale network. Analysts use ANSI-standard SQL with joins, aggregations, nested fields, and spatial functions, plus BigQuery DataFrames for pandas-compatible Python and BigQuery Studio notebooks. Federated queries and external tables let BigQuery query data where it lives in Cloud Storage, Bigtable, Spanner, or Google Sheets. Knowledge Catalog adds metadata, data quality, lineage, and discovery, while Identity and Access Management (IAM) secures access.
How BigQuery, Looker, and Vertex AI guide decisions is explained by three connected layers for analysis, sharing, and prediction. BigQuery surfaces trends and patterns at scale by querying terabytes in seconds and petabytes in minutes, with built-in ML, search, geospatial analysis, and business intelligence. Looker delivers trusted dashboards by defining key metrics once in a unified data model, so every team sees the same revenue, cost, or performance definition. Vertex AI adds forecasts by training models on BigQuery tables and returning predictions to dashboards and applications.
The three services divide work by insight type:
| Service | Primary unit | Choose when |
|---|---|---|
| BigQuery | SQL or Python query and ML model | The work is analyzing large datasets to surface trends and patterns |
| Looker | Governed dashboard and visualization | The work is sharing one trusted view of metrics across teams |
| Vertex AI | Generative AI and ML development | The work is building predictive or generative models beyond SQL |
BigQuery ML builds models directly with SQL inside BigQuery, while Vertex AI supports fuller generative AI development, Vertex AI Workbench notebooks, and Dataflow ML pipelines.
How analytics creates new business value is the link from insight outputs to better operations, strategy, and measurement. Operational improvements use real-time insights to reveal bottlenecks and demand shifts. Strategic planning combines historical trends with forecasts to guide investment and cost choices. Performance measurement tracks governed Looker metrics over time to confirm whether a change worked. Together, BigQuery provides scale and speed, Looker provides governance and clarity, and Vertex AI provides foresight.
The analytics hybrid and multicloud pattern supports this value by keeping transactional systems separate from analytical systems:
BigQuery Omni extends the pattern by running compute near data stored in Amazon Simple Storage Service (Amazon S3) or Azure Blob Storage, so teams query other-cloud data without copying it first.
A data lake is a scalable central platform that ingests and stores raw data from many systems for later processing. Cloud Storage is commonly used as the storage layer for a data lake on Google Cloud. A data lake differs from a database, which runs day-to-day transactions, and from a data warehouse, which analyzes organized data for reporting.
Side-by-side comparison of data lake on Cloud Storage, database on Cloud SQL, and warehouse on BigQuery across purpose, service, and data and schema.
Data lakes on Cloud Storage store data as objects in buckets with a flat hierarchy. Each object has a globally unique ID and can carry system-assigned or user-defined metadata for organization and access. Applications reach those objects through REST APIs or client libraries from inside or outside Google Cloud. Cloud Storage is low-cost, highly durable, and highly scalable, with optional redundancy across regions.
Storage classes match access frequency, with Standard for frequent access including data lakes, Nearline for at least 30 days, Coldline for at least 90 days, Archive for at least 365 days, and Autoclass for automatic movement. Ingestion can use console uploads, gcloud CLI, APIs, Storage Transfer Service for online transfer, or Transfer Appliance for offline transfer. Processed data is often saved in columnar formats such as Parquet or ORC for efficient storage and faster reads. This approach is called schema-on-read because structure is applied when data is read rather than enforced at write time.
Databases for operational transactions keep structured current data correct and quickly accessible for applications. A SQL database, also called a relational database, stores highly structured data in tables of rows and columns and links related facts across tables without duplication. A relational database management system (RDBMS) maintains those links and uses indexes for faster lookups. Structured Query Language (SQL) uses CREATE to define tables, INSERT to add rows, and SELECT to read matching data.
Common choices differ by ownership and focus, including open-source PostgreSQL for enterprise performance, MySQL for flexible SQL deployment, and Microsoft-owned SQL Server for transaction processing and analytics. On Google Cloud, Cloud SQL is the managed service for MySQL, PostgreSQL, and SQL Server and removes most maintenance work. Self-managed databases on Compute Engine virtual machines require the team to handle updates, availability, backups, and recovery but offer more operating-system control. Spanner supports globally distributed applications, while Firestore data can flow into BigQuery for later analysis.
Data warehouses for analysis are enterprise systems for reporting on historic structured data that usually comes from operational databases. BigQuery (opens in a new tab) is the commonly used warehouse on Google Cloud, optimized to query terabytes in seconds and petabytes in minutes. Datasets are the top-level containers for tables and views, while schemas, labels, partitioning, and clustering reduce the amount of data scanned. Storage resources allocate as data is consumed, so no pre-provisioning is required.
BigQuery decouples storage from compute for speed and isolation. Data is compressed in columnar Capacitor format in Colossus, so queries read only needed columns without decompressing everything. The Dremel execution engine uses a multilevel tree with in-memory execution and autoscaling, and each query uses slots that bundle CPU and RAM. Different projects and users can query without affecting each other because slot quotas and billing are separated.
Exam tip: A data lake stores raw data for later processing, a database runs application transactions, and a data warehouse analyzes modeled data.
How lakes, databases, and warehouses work together is explained by pipelines that move data from raw storage to analysis. Operational databases feed the warehouse one way through extract, transform, and load (ETL) or extract, load, and transform (ELT) pipelines using Pub/Sub, Dataflow, and Spark. A typical batch flow runs in order:
| Paradigm | Primary purpose | Typical service | Data and schema |
|---|---|---|---|
| Data lake | Ingest and store data from many systems | Cloud Storage | Raw and unstructured objects; schema-on-read |
| Database | Run applications and transactions | Cloud SQL, Datastore, Bigtable | Structured records; transactions and consistency |
| Data warehouse | Analyze and report on integrated data | BigQuery | Structured tables; defined schemas; columnar storage |
Creating value from all data sources means organizing existing records, capturing new events, and adding outside context in one governed flow. New events flow through Pub/Sub to Dataflow to BigQuery, existing files are organized in BigQuery and Cloud Storage and activated with business intelligence tools, and external datasets are joined through live sharing. Each path ends in dashboards, predictions, and shared insights.
Collecting new data captures information the organization does not already have, from game logs and clickstreams to device signals, transactions, and geospatial events. Pub/Sub is the messaging service that ingests these events in real time. A producer publishes a message to a topic, Pub/Sub holds and delivers it to subscriptions, and a subscriber such as a Dataflow pipeline receives it downstream. This decoupling lets new sources be added without rewriting processing.
Dataflow is the serverless service for batch and streaming pipelines written with the Apache Beam model. A pipeline reads from Pub/Sub, cleans, joins, or enriches the data, and writes to BigQuery. Templates allow reuse, exactly-once processing avoids double-counting replayed events, and built-in ML inference supports real-time anomaly detection. BigQuery (opens in a new tab) then stores the results as both warehouse tables and lake-scale files, with streaming writes through the Storage Write API making events queryable quickly. IAM controls access, while VPC Service Controls place a boundary around shared assets to reduce exfiltration risk.
Using existing data organizes operational and historical records so they can be queried, visualized, and shared. BigQuery separates storage from compute over a petabit-scale network, so each layer scales without competing for resources as legacy databases do. Data is organized into datasets containing tables and views, stored in columnar format with ACID semantics and automatic replication. Existing data enters through BigQuery Data Transfer Service, streaming APIs, or batch loads of Avro, Parquet, ORC, CSV, JSON, Datastore, and Firestore files.
Cloud Storage holds files that BigQuery can batch load for optimized storage or query in place through external tables and federated queries. Looker then activates the data for advanced analytics and custom applications, Data Studio builds simple reports and dashboards, and Connected Sheets exposes billions of rows in Google Sheets without exports. Governance includes Knowledge Catalog for lineage and discovery, row-level and column-level controls, column masking, and Sensitive Data Protection scans for personally identifiable, financial, and health information.
Adding external data combines internal tables with public datasets, partner data, and commercial products for broader insight. External signals add context that internal data alone cannot provide, improving forecasts and ML models. Examples include Search Trends, WeatherNext models, Maps Platform data, and Earth Engine data joined with internal tables in BigQuery.
BigQuery sharing, built on Analytics Hub, discovers and consumes this data without custom integration. A provider organizes sharing through an exchange containing listings, and a consumer subscribes to a listing to create a linked dataset. The linked dataset is a live pointer to the provider asset, so queries always run against current data with zero-copy sharing and no replication. Listings can share tables, views, Pub/Sub topics, stored procedures, and BigQuery ML models. Owners retain control with access rules and VPC Service Controls, monitor usage by jobs and bytes scanned, and use data clean rooms for sensitive joint analysis without revealing underlying data.
Exam tip: A linked dataset is a live pointer to the provider data, not a copy — zero-copy sharing needs no replication and always sees the latest version.
An integrated data ecosystem is a connected set of pipelines, analytics services, and shared datasets that moves data from creation to decision. Operational databases feed unified pipelines, BigQuery provides large-scale analysis, and sharing services distribute trusted results. This connection shortens the path from raw information to action and lets teams work from the same evidence.
Unified pipelines and shared datasets connect operational systems to analytics without manual copying. Datastream is a serverless change data capture (CDC) and replication service that synchronizes data with minimal latency and no provisioning. It supports MySQL, Oracle, PostgreSQL including AlloyDB, SQL Server, MongoDB, Spanner, Salesforce, Microsoft Dataverse, and ServiceNow. Data flows in order from capture to analysis to distribution:
Shared datasets reuse those fresh tables through Analytics Hub exchanges and listings, Knowledge Catalog data products with contracts and access groups, and data clean rooms for joint analysis without revealing underlying data. Data lineage in Knowledge Catalog tracks origin, transformations, and destinations at column level, which improves trust, troubleshooting, change management, and compliance.
Unlocking unstructured data with AI turns documents, images, audio, video, and free-form text into searchable signals. Structured data already lives in rows and tables, while unstructured content such as contracts, invoices, emails, photos, and call recordings has no fixed schema. Cloud storage and analytics hold both types at scale, and AI services convert unstructured content into structured outputs for automation and prediction.
Vertex AI is the fully managed, unified platform for predictive ML and generative AI, with pre-built models, custom training on business data, and MLOps for deployment and monitoring. Extraction services feed it by converting inputs to text: Optical Character Recognition (OCR) extracts text from images and documents, Speech-to-Text converts spoken audio into transcripts, and Translation makes content usable across regions. Generative AI then creates summaries, classifications, and answers from that text in Vertex AI Workbench notebooks. Online Prediction serves live scores to applications, while BigQuery ML lets analysts train models directly with SQL where the data lives.
| Service | Input it reads | Output it creates |
|---|---|---|
| Optical Character Recognition (OCR) | Images and scanned documents | Machine-readable text from visual content |
| Speech-to-Text | Spoken audio such as calls | Written transcripts for search and analysis |
| Generative AI | Text and combined context | Summaries, classifications, and answers |
Serverless storage and processing at scale removes capacity limits and maintenance work that restrict on-premises databases. BigQuery provides a uniform way to work with structured and unstructured data, supports open formats such as Apache Iceberg, Delta, and Apache Hudi, and queries terabytes in seconds and petabytes in minutes with zero infrastructure management. Storage ingests, stores, and optimizes data while compute provides analytics, and the two layers communicate over a petabit-scale network so each scales independently.
Ingestion follows three patterns that differ by movement:
| Ingestion pattern | How data moves | Sources supported |
|---|---|---|
| Streaming ingestion | Data flows continuously through the Storage Write API | Real-time events and application data |
| Batch loading | Data flows in bulk from files into tables | Local files or Cloud Storage objects |
| Federated query | Data stays in place while compute queries it | Cloud Storage, Bigtable, Spanner, Google Sheets |
Cloud Data Fusion (opens in a new tab) and Dataflow build batch and streaming pipelines before analysis, Pub/Sub ingests event streams at any scale, and BigQuery Data Transfer Service automates scheduled loads. IAM secures resources with a consistent access model, while Knowledge Catalog adds semantic search, lineage, and quality controls so diverse datasets stay discoverable and trusted.
Value realization is the stage of the data value chain where processed data becomes business value through analysis, visualization, and prediction. Raw records have already been staged, cleaned, and stored as trusted tables, and analysis turns those tables into answers. In Google Cloud, this work centers on BigQuery for large-scale analysis, Looker for business intelligence, and ML services for prediction.
Stages of the data value chain form an ordered flow that protects consistency while allowing change. Data flows from transactional sources through staging into long-lived core tables, then through a business-logic layer to consumers who rely on stable schemas:
A five-step left-to-right flow showing data moving from sources through staging, core tables, and BigQuery views to consumers like dashboards and notebooks.
Consumers use the same trusted logic in different ways, including BI dashboards for repeatable metrics, notebooks for ad hoc exploration, operational systems for daily actions, and ad hoc queries for new questions. Looker supports this stage by providing exploration on an up-to-date, trusted version of data. BigQuery ML builds predictions directly with SQL where data lives, while Vertex AI supports broader AI and Dataflow manages full ML pipelines.
Data types, lakes, warehouses, and metadata describe what data exists and where it should live. Structured data is organized into tables with schemas, partitions, and column statistics. Semi-structured data has tags or keys but no rigid schema, while unstructured data has no predefined model. Metadata describes what an asset contains, who owns it, and how it should be used.
A data warehouse is a central repository for structured data used for reporting, dashboards, and analysis, with data marts as subsets for specific business units. A data lake stores underlying files and uses a metastore to assign meaning to those files. Dataproc Metastore (opens in a new tab) is a fully managed Apache Hive metastore that manages schemas, partitions, and column statistics and provides interoperability between Hive, Spark, and Presto. Managed internal tables store data in the Hive warehouse directory, while unmanaged external tables point to files elsewhere.
| Dimension | Data lake | Data warehouse |
|---|---|---|
| Primary data | Underlying data files | Tables with schemas |
| Metadata | Managed by Dataproc Metastore | Table definitions in the warehouse |
| Processing engines | Hive, Spark, Presto | SQL, BigQuery |
| Main use | Assign meaning to files and enable interoperability | Reporting, dashboards, analysis, data marts |
Ingestion and transformation move raw data into the warehouse and prepare it for dashboards and reports. Streaming ingestion writes continuously through the Storage Write API for near-real-time dashboards where latency is critical. Batch ingestion collects and processes on a schedule, such as nightly regulatory reports that must complete correctly by a fixed deadline.
Extract, transform, and load (ETL) extracts data, transforms it inside the pipeline, then loads the prepared result. The transform step cleans errors, applies business rules, checks integrity, and creates aggregates. Extract, load, and transform (ELT) loads raw data first, then transforms inside the warehouse with SQL, often at night to avoid competing with concurrent queries. Change data capture (CDC) tracks source changes over time with Datastream or Debezium, flows records through Dataflow into BigQuery, and uses SQL to find the latest version.
| Workflow | Order of operations | Where transformation happens |
|---|---|---|
| ETL | Extract, then transform, then load | Inside the pipeline before entering the warehouse |
| ELT | Extract, then load, then transform | Inside the warehouse using SQL after loading |
Exam tip: ETL transforms data in the pipeline before loading it, while ELT loads raw data first and then transforms it inside the warehouse with SQL.
Continue with this study guide: Discuss the main data value chain concepts and terms.
Data governance is the set of policies, procedures, and technology that keeps data secure, private, accurate, available, and usable. Governance builds trust by protecting sensitive information, ensuring integrity, meeting compliance requirements, and maintaining stakeholder confidence. In reporting tools such as Data Studio, governance controls who can access, edit, and share reports while supporting data democratization.
Building trust with quality and access control means giving more people data access while still protecting sensitive fields. Data credentials determine whose authorization shows data: Owner's Credentials lets viewers use the owner's authorization without their own dataset access, Viewer's Credentials requires each viewer to have dataset access, and Service Account Credentials uses a non-human account for automated access. For maximum control, use Viewer's or Service Account Credentials. Reusable data sources created from the home page restrict editing to users with explicit access, while embedded sources created inside a report can be edited by report editors and copied with the report.
| Credential type | How data access works | Best for |
|---|---|---|
| Owner's Credentials | Viewers use the owner's authorization | Convenience when viewers lack direct access |
| Viewer's Credentials | Each viewer uses their own credentials | Maximum control over who sees data |
| Service Account Credentials | Non-human account accesses data | Automated or application access |
Hiding fields removes them from report field lists, row-level security limits viewers to permitted rows, and sharing limits prevent downloading, printing, copying, or changing access. For AI, a governed environment with model explainability, fairness metrics, and alignment to the Secure AI Framework supports responsible development and deployment.
Protecting data with security controls and compliance builds legal and technical safeguards into every lifecycle stage. A governance strategy aligns with frameworks such as ISO 27001, NIST 800-53 and Cybersecurity Framework, PCI DSS, and SOC 2. Security Command Center assesses benchmark compliance, with data profile findings flowing into it so evidence reflects actual sensitive data.
Sensitive Data Protection (opens in a new tab) provides discovery, inspection, and de-identification, while IAM with column-level and row-level security hides or shows data by viewer. For data in motion, the DLP API inspects and masks input before storage:
Choose in-motion de-identification when sensitive data might arrive through user input, and choose IAM or column and row controls when sensitive data must stay stored but visible only to authorized viewers. Assured Workloads adds data boundaries for regulated workloads including residency and audit requirements.
Finding and understanding data with metadata and lineage means describing what exists and showing its history. Knowledge Catalog provides metadata management, search, and lineage so users find trusted assets and judge fitness for use. Its metamodel uses entry types to define kinds of assets, aspect types to define structured attributes, and entry groups to organize related entries.
Data lineage shows origin and movement end to end, with column-level visualization of how fields flowed and changed from sources through transformations to tables, reports, and dashboards. Asset discovery with enriched descriptions, classifications, and glossary terms reduces duplication and promotes reuse. If a source column changes, lineage shows which downstream reports and models depend on it before the change is made.
Exam tip: Metadata describes what a data asset is, while data lineage shows where the asset came from, how it was transformed, and where it goes next.
A data lake ingests and stores raw data from many systems for later processing and commonly uses Cloud Storage with schema-on-read, a database runs day-to-day application transactions with structured current data such as Cloud SQL for MySQL, PostgreSQL, and SQL Server, and a data warehouse analyzes organized historic structured data for reporting and commonly uses BigQuery to query terabytes in seconds and petabytes in minutes. Operational databases feed the warehouse through extract, transform, and load (ETL) or extract, load, and transform (ELT) pipelines using Pub/Sub, Dataflow, and Spark.
Use BigQuery when the work is analyzing large datasets to surface trends and patterns with SQL or Python queries and machine learning (ML) models, use Looker when the work is sharing one trusted view of metrics across teams with governed dashboards defined once in a unified data model, and use Vertex AI when the work is building predictive or generative models beyond SQL. Together BigQuery provides scale and speed, Looker provides governance and clarity, and Vertex AI provides foresight.
Extract, transform, and load (ETL) extracts data, transforms it inside the pipeline, then loads the prepared result, while extract, load, and transform (ELT) loads raw data first, then transforms it inside the warehouse with SQL. In ETL transformation happens inside the pipeline before entering the warehouse, and in ELT transformation happens inside the warehouse using SQL after loading.
Last updated