Associate Data Practitioner

Section 1: Data Preparation and Ingestion

1.1 Prepare and Process Data

Data preparation involves getting raw data ready for analysis or storage. Before data can be useful, it often needs cleaning, transforming, or restructuring. Dataflow is a service that processes data in both batch and streaming modes, meaning it can handle data that arrives in regular batches or continuously in real time. Dataproc handles batch processing jobs using open-source tools like Apache Spark and Hadoop, which are useful when teams already have scripts written in those frameworks.

When preparing data, teams must consider the format they are working with. Structured data fits neatly into tables with rows and columns, while unstructured data like images or text files requires different handling. Semi-structured data such as JSON or Avro files falls in between, offering some organization but not the rigid structure of a database. The choice of format affects how the data can be processed and what tools work best.

Data quality matters because poor-quality data leads to incorrect results. Common quality tasks include removing duplicates, filling in missing values, and standardizing formats so that information from different sources matches. Teams often validate data against rules or schemas to catch errors early, before bad data flows into storage systems where it causes problems.

1.2 Extract and Load Data into Appropriate Google Cloud Storage Systems

Extracting data means pulling it from source systems such as databases, APIs, or files. Loading means placing that data into a destination within Google Cloud. The right destination depends on how the data will be used later. Cloud Storage works well for storing files like CSV, JSON, or Parquet that will be processed later or served to applications. BigQuery serves as a data warehouse where analytical queries run quickly across large datasets.

Data can move into Cloud Storage through batch uploads or streaming uploads, depending on how quickly it needs to be available. Batch uploads happen on a schedule, while streaming uploads insert records one at a time as they arrive. BigQuery accepts data through batch loads from Cloud Storage or from other sources, and it also supports streaming inserts for real-time data.

When choosing a storage system, consider the access pattern. If the data will be analyzed with SQL queries and needs fast performance on large volumes, BigQuery is the right fit. If the data will be processed by code running on virtual machines or containers, Cloud Storage provides flexible file storage that those tools can read. For transactional workloads that require strong consistency and frequent updates, Cloud SQL or Firestore offer traditional database capabilities. The key is matching the storage service to the intended use case rather than forcing all data into one system.