Prepare and test your skills

Prepare and test your skills

Evaluating strategies for collecting and managing new data means deciding how to capture information an organization does not already have and move it into systems where it can be processed, stored, and governed. New data comes from applications, connected devices, and user interactions, and the strategy depends on whether answers are needed in real time or can wait for batch results. On Google Cloud, the common path is that Pub/Sub ingests events, Dataflow (opens in a new tab) processes them, and BigQuery stores the results for analysis.
New data sources are the applications, devices, and interactions that continuously create fresh events. Examples include game usage logs, clickstream and tag-manager events, smart home device signals, credit card transactions used for fraud detection, fan interactions in a stadium app, shared mobile phone data, and geospatial crop data. These sources differ in shape and speed, so the collection strategy must match the source. Steady streams of small events need a service that accepts events as they happen, while large groups of files that arrive together can be ingested in batches into a data lake.
Pub/Sub is the Google Cloud messaging service that receives events from producers and delivers them to consumers. A producer such as an application, device, or user interaction publishes a message to a topic, and subscribers receive those messages without the producer needing to know who is listening. This decoupling lets new sources be added without changing the processing pipeline, and lets the pipeline scale without changing the sources.
The ingestion flow follows a fixed order:
A left-to-right process flow from consolidated new data sources through Pub/Sub ingestion to Dataflow transformation to BigQuery storage and governance.
Dataflow is a serverless, fully managed service for running data processing pipelines, and Apache Beam is the programming model those pipelines are written in. Apache Beam uses the same model for batch and streaming data, so a team can process a live stream and a stored batch with similar pipeline code. Dataflow runs the pipeline without the team managing servers or clusters. A pipeline reads from a source such as Pub/Sub, applies transformations to clean, join, or enrich the data, and writes the result to a destination such as BigQuery.
Dataflow includes capabilities that matter for newly collected data, which is often messy, late, or duplicated:
Handling these conditions inside the pipeline keeps bad data from reaching the storage layer.
Newly collected data storage is centered on BigQuery, which can act as both a warehouse for structured tables and a data lake for large volumes of files. Real-time events can be streamed into BigQuery through the BigQuery Storage Write API using gRPC, and streaming JSON messages can be written directly into a BigQuery JSON-type column. This makes events queryable shortly after they are produced. Governance then controls who can see and share that stored data. Identity and Access Management (IAM) controls access to resources, and VPC Service Controls reduce the risk of data exfiltration by placing a boundary around shared assets.
Exam tip: Pub/Sub moves events, Dataflow transforms them, and BigQuery stores them — a pipeline usually chains all three in that order rather than using one service for the whole job.
Analyzing the use of external and third-party data sources means understanding how datasets created outside an organization can be combined with internal data for broader insight. External sources include public datasets, partner-provided data, and commercial data products from many providers. In Google Cloud, data analytics (opens in a new tab) sharing in BigQuery lets organizations discover, access, and consume this external library without building custom integrations.
External data improves analytics because it adds context that internal data alone cannot provide. An organization can join its own sales or operations data with outside signals to see larger trends, improve forecasts, and train better machine learning models. Google datasets provide ready-to-use examples, including Search Trends, DeepMind WeatherNext models, Google Maps Platform data, and Google Earth Engine data. By joining these external datasets with internal tables in BigQuery, teams can strengthen analytics and machine learning without collecting all the data themselves.
BigQuery sharing, built on Analytics Hub, is the platform for discovering and consuming shared data and AI assets within and across organizational boundaries. A data provider organizes sharing through an exchange, which is a container that holds listings, and each listing describes one specific shared asset. A data consumer discovers the listing in the exchange and subscribes to it, which creates a linked dataset in the consumer's project. The linked dataset is a live pointer to the provider's shared asset, so queries from the consumer run against the latest provider data.
A listing can share more than raw tables:
This range means a partner can share live event data, reusable logic, or a trained model in the same way it shares a table.
Secure and governed data sharing in BigQuery keeps control with the data owner while removing the need to copy data. Sharing is live and zero-copy, which means data is shared in place with no integration, data movement, or replication needed. Because the subscriber queries through a live pointer, analysis is always based on the latest information and neither side must manage duplicate copies. Data owners retain control and can define rules or policies to restrict access and usage.
Governance and monitoring travel with the shared asset. Access controls, including built-in support for VPC Service Controls (VPC SC), reduce overprovisioning and help prevent data exfiltration around shared assets. Providers can monitor usage for each shared asset, including jobs executed, bytes scanned, and subscriber counts. For highly sensitive collaboration, data clean rooms provide a security-enhanced environment where multiple parties can share, join, and analyze data assets without moving or revealing the underlying data.
Exam tip: A linked dataset is a live pointer to the provider's data, not a copy — zero-copy sharing means no replication is needed and queries always see the latest version.
Assessing the value of existing organizational data means determining how operational and historical records can be organized, processed, and activated to improve decisions. Google Cloud supports this work through BigQuery for analytics, Cloud Storage for holding files, and Looker for business intelligence and visualization. Existing data supports descriptive and prescriptive uses such as business intelligence, ad hoc analysis, geospatial analytics, and machine learning.
BigQuery (opens in a new tab) is a fully managed, AI-ready data platform with built-in machine learning, search, geospatial analysis, and business intelligence. Its serverless architecture lets users work with SQL and Python without managing infrastructure, and its distributed engine can query terabytes in seconds and petabytes in minutes. BigQuery works with structured and unstructured data and supports open table formats such as Apache Iceberg, Delta, and Apache Hudi.
BigQuery separates storage from compute over Google's petabit-scale network. The storage layer ingests, stores, and optimizes data, while the compute layer provides analytics, and each layer can allocate resources without affecting the other. This differs from legacy databases that share resources between read and write operations and analytical operations, which can cause conflicts and slow queries. Data is organized into datasets, which are the top-level containers of tables and views. Storage uses a columnar format optimized for analytical queries, supports ACID transaction semantics, and is automatically replicated across multiple locations for high availability.
Existing data can enter BigQuery through the BigQuery Data Transfer Service, which automates ingestion, through streaming with the Storage Write API using gRPC, or through batch loads from local files or Cloud Storage in formats such as Avro, Parquet, ORC, CSV, JSON, Datastore, and Firestore. BigQuery can also query data where it lives using external tables or federated queries, including data in Cloud Storage, Bigtable, Spanner, or Google Sheets in Google Drive. For analysis, users can run ANSI-standard SQL with joins, nested and repeated fields, aggregation and analytic functions, multi-statement queries, and spatial functions. They can also use the Pandas-compatible BigQuery DataFrames Python API, create views to share analysis, use BigQuery ML for predictions, and use BigQuery Studio for Python notebooks with version control.
BigQuery governance is powered by Knowledge Catalog, which helps discover and curate data and manage metadata, data quality, semantic search, and lineage. Identity and Access Management (IAM) secures resources with the access model used throughout Google Cloud. Deeper controls include:
Cloud Storage is the object store that holds existing organizational files that BigQuery uses for analysis. BigQuery can batch load data from Cloud Storage into its storage layer when data should be stored and optimized inside BigQuery. BigQuery can also query Cloud Storage data where it lives through external tables or federated queries when the organization wants to analyze files without first loading them. The choice depends on whether the goal is long-term optimized storage or fast analysis of files in place.
Looker is Google's enterprise platform for data visualization and business intelligence. Looker is useful for advanced analytics, predictive analysis, and custom data applications, and it can connect to transactional databases in other public clouds. Looker helps users access, analyze, and act on an up-to-date, trusted version of their data.
Other tools activate BigQuery data for different audiences:
| Tool | Best for | Connection detail |
|---|---|---|
| Looker | Advanced analytics, predictive analysis, custom data applications | Connects to BigQuery and transactional databases in other public clouds |
| Data Studio | Simple graphics, reports, and dashboards | Connects to BigQuery directly from the Google Cloud console |
| Connected Sheets | Collaboration with non-technical stakeholders | Accesses billions of rows of BigQuery data from Google Sheets with no additional exports |
Additional activation options include ODBC and JDBC drivers for existing tooling, programmatic tools such as Jupyter Notebooks, Apache Zeppelin, and Vertex AI Workbench, and third-party tools such as Tableau and Microsoft Power BI. Gemini in BigQuery can also assist exploration with data insights and conversational analytics.
The path from existing data to value is an ordered flow from organization to activation to secure sharing:
Exam tip: BigQuery can query data where it lives with external tables or federated queries, so existing data does not always need to be loaded into BigQuery before it can be analyzed.
New data typically flows from Pub/Sub for ingestion to Dataflow for transformation to BigQuery for storage and analysis. A source publishes an event to a Pub/Sub topic, Pub/Sub delivers it to each subscription, and a Dataflow pipeline running an Apache Beam pipeline reads, transforms, and writes the result to BigQuery.
A linked dataset is a live pointer in the consumer project to the provider shared asset discovered through an exchange listing. Because sharing is live and zero-copy with no replication needed, queries from the consumer run against the latest provider data.
External tables or federated queries should be used when an organization wants to analyze files where they live without first loading them. BigQuery can query data in Cloud Storage, Bigtable, Spanner, or Google Sheets in Google Drive in place, while batch loading from Cloud Storage is used when data should be stored and optimized inside BigQuery.
A retail organization wants to improve its demand forecasting models by enriching its internal sales data with external information. The analytics team has the following requirements:
Which Google Cloud capability should the organization use to meet these requirements?
BigQuery Omni
BigQuery sharing (Analytics Hub)
Private Google Access
Sensitive Data Protection
Last updated