Associate Data Practitioner

Distinguish the format of the data (e.g., CSV, JSON, Apache Parquet, Apache Avro, structured database tables)

Data comes in different formats, and the format you choose affects how you store and analyze it. Structured data, like the rows and columns in a CSV file or a structured database table, is very organized and predictable. Semi-structured data, like JSON documents, is more flexible, grouping related information together even if the details vary from one record to another.

For large-scale analytics, specialized formats like Apache Parquet and Apache Avro are often used. Parquet is highly efficient for reading specific columns of data, which saves time and cost in analytical systems. Avro is good for serializing data for movement between systems, as it includes the data's schema right alongside the data itself. Choosing the right format is a tradeoff between how easy the data is to move, how efficiently it can be stored, and how quickly it can be queried later.

Choose the appropriate extraction tool (e.g., Dataflow, BigQuery Data Transfer Service, Database Migration Service, Cloud Data Fusion)

Extracting data means getting it from its original source and preparing it to be loaded into Google Cloud. The right tool depends on where the data is coming from and what needs to happen to it. For scheduled, managed transfers from sources like Google Marketing Platform or SaaS applications, the BigQuery Data Transfer Service handles the work automatically.

When you need to move databases with minimal downtime, the Database Migration Service is designed for continuous replication from sources like MySQL or PostgreSQL into Cloud SQL or other targets. For building complex data pipelines that involve both extraction and transformation, Cloud Data Fusion provides a visual interface. If you need to process streaming data or perform custom transformations during extraction, Dataflow is a powerful, code-based option that can handle both batch and real-time data.

Select the appropriate storage solution (e.g., Cloud Storage, BigQuery, Cloud SQL, Firestore, Bigtable, Spanner)

Google Cloud offers different storage systems, each built for a specific type of workload. For storing raw files of any format—like your source CSV, JSON, or Parquet files—Cloud Storage is the universal object store. When your primary goal is running fast analytical queries on massive datasets, BigQuery is the data warehouse designed for that purpose.

For transactional applications that need a relational database, you choose between Cloud SQL for regional MySQL or PostgreSQL databases and Spanner for globally scalable, strongly consistent relational data. Firestore is a flexible NoSQL document database ideal for mobile and web apps. Bigtable is a high-performance, low-latency NoSQL database perfect for analytical workloads like time-series data or machine learning feature stores. The decision hinges on whether you need transactions, analytics, global scale, or simple file storage.

Load data into Google Cloud storage systems using the appropriate tool (e.g., gcloud and BQ CLI, Storage Transfer Service, BigQuery Data Transfer Service, client libraries)

Loading is the final step of moving data into your chosen Cloud storage system. You can use command-line tools like gcloud and the bq CLI for scripting and one-time operations, such as uploading a file to a Cloud Storage bucket or loading a CSV into a BigQuery table. For automated, scheduled data ingestion into BigQuery from specific sources, you again use the BigQuery Data Transfer Service.

When you need to move large volumes of files from other cloud storage providers or on-premises systems into Cloud Storage, the Storage Transfer Service manages the copy job efficiently. For applications that need to load data programmatically, Google provides client libraries in various programming languages, allowing your code to interact directly with Cloud Storage, BigQuery, or any other service. The flow is straightforward: you select a tool that matches your destination, your source, and your preferred method of operation—manual, scheduled, or programmatic.