Newline-Delimited JSON for Nested and Repeated Fields

What Newline-delimited JSON is

Newline-delimited JSON (often referred to as ndjson or JSON Lines) is a semi-structured, text-based data interchange format where each individual line in a file contains a complete, valid JSON object. In data analytics and ingestion platforms like BigQuery, newline-delimited JSON is the standard JSON format used for batch data loads and streaming ingestion because each record is self-contained and separated by a newline character (\n).

How it meets the requirements:

• Nested records support (✓): JSON objects natively support key-value pairs where the value can be another JSON object. In BigQuery, these map directly to STRUCT (RECORD) data types, allowing complex sub-entities like customer contact profiles to remain grouped within the parent record.
• Repeated fields support (✓): JSON natively supports arrays (e.g., "items": [...]), which BigQuery represents as REPEATED (ARRAY) fields. This allows multiple purchased items to be stored directly inside a single transaction record without requiring denormalization or separate child tables.
• Direct ingestion from Cloud Storage (✓): BigQuery natively supports loading newline-delimited JSON files directly from Cloud Storage buckets into managed analytical tables.

Key advantages:

• Flexible and self-describing: JSON encodes attribute names alongside values, making it easier to represent evolving data structures and variable schemas.
• Hierarchical data representation: Eliminates the immediate need for complex relational multi-table database joins when querying parent-child relationships.
• Standard cloud interchange format: Highly compatible with web APIs, logging services, and distributed stream processing engines across Google Cloud.
• Parallel processing friendly: Because each record occupies exactly one line, distributed ingestion systems can split large files efficiently along line boundaries.

Why this is optimal:

Unlike flat, tabular formats such as CSV that only represent two-dimensional rows and columns, newline-delimited JSON allows complex, multi-tiered business objects to be ingested into BigQuery in their natural structure. This avoids the overhead of flattening arrays or maintaining multiple related tables, making it the ideal choice for hierarchical transaction logs.