Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Resolving Pub/Sub Avro Schemas with DatumReader
Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise architecture team is designing a real-time streaming ingestion pipeline on Google Cloud using Cloud Pub/Sub to process events across multiple microservices. The Pub/Sub topic enforces an Apache Avro schema that evolves over time as upstream services add optional fields, creating multiple schema revisions.
The downstream subscriber applications maintain their own reader schema and must reliably deserialize incoming binary-encoded messages that may have been published under different schema revisions.
How should the subscriber application be architected to handle messages across evolving schema revisions?
Explanations
Configure a dead-letter topic on the subscription to discard messages whose revision ID does not match the latest registered schema version.
Configure a dead-letter topic on the subscription to discard messages whose revision ID does not match the latest registered schema version.
What this option does
This approach uses a dead-letter topic (DLT) to route and discard any messages that were published using older or differing schema revisions that do not match the latest version.
Why it doesn't meet the requirement
- Data loss: Schema evolution is explicitly intended to support backward- and forward-compatible changes without discarding valid business events.
- Defeats schema evolution: Dropping messages published under valid prior schema revisions breaks continuous real-time data ingestion for downstream analytics.
Key limitations
- Dead-letter topics are designed for handling unprocessable poison-pill messages after maximum delivery attempts, not as a mechanism for schema version branching.
- Diverts valid transactions away from primary processing pipelines.
This answer would be correct if
The organization enforced strict schema immutability and treated any message not matching the exact active revision as invalid, non-conforming data to be quarantined.
Deserialize incoming messages using only the compiled local Avro reader schema directly, without retrieving or referencing the specific writer schema revision.
Deserialize incoming messages using only the compiled local Avro reader schema directly, without retrieving or referencing the specific writer schema revision.
What this option does
This approach attempts to decode binary-encoded Avro records using only the consumer application's local reader schema without supplying the corresponding writer schema.
Why it doesn't meet the requirement
- Binary decoding failure: Binary Avro serialization does not embed field tags, names, or types inside the serialized byte payload. It only contains raw values in order.
- Schema resolution incompatibility: If the publisher added, reordered, or removed fields in a new revision, the reader schema alone cannot determine which fields in the binary stream correspond to which attributes, resulting in data corruption or deserialization exceptions.
Key limitations
- Fails to leverage the
googclient_schemarevisionidattribute provided by Pub/Sub. - Breaks whenever schema changes alter the byte layout between writer and reader versions.
- Violates standard Apache Avro binary deserialization requirements.
This answer would be correct if
The messages were serialized with self-describing formats (such as JSON) where field names are explicitly present in every payload, or if schemas were completely immutable and guaranteed never to change.
Inspect the googclient_schemarevisionid attribute on each message, fetch the corresponding writer schema revision definition from the Pub/Sub Schema service, and deserialize using an Avro DatumReader configured with both the writer schema and the reader schema.
Inspect the googclient_schemarevisionid attribute on each message, fetch the corresponding writer schema revision definition from the Pub/Sub Schema service, and deserialize using an Avro DatumReader configured with both the writer schema and the reader schema.
What Schema Revision Handling in Pub/Sub Is
When Cloud Pub/Sub enforces schemas on topics, it attaches system metadata attributes to published messages—such as googclient_schemaname, googclient_schemarevisionid, and googclient_schemaencoding. These attributes identify the exact schema and revision used by the publisher to serialize the message payload.
How it meets the requirements
- Dynamic Schema Evolution: By reading the
googclient_schemarevisionidattribute, the subscriber detects the exact writer schema revision used for encoding. - Accurate Binary Deserialization: In Apache Avro, binary deserialization requires resolving the writer schema (how the binary was structured at publication) against the reader schema (how the consumer expects the data). Configuring an
Avro DatumReaderwith both schemas allows Avro to apply schema resolution rules (e.g., handling added optional fields or default values). - Caching Efficiency: The subscriber can cache fetched writer schemas in memory by
schema_revision_idto avoid repeated remote API lookups to the Pub/Sub Schema service on every message.
Key advantages
- Seamless schema evolution: Supports backward and forward compatibility across independent publisher and subscriber release cycles.
- High performance: Binary Avro encoding minimizes message payload size and network serialization overhead.
- Reliable schema governance: Eliminates runtime deserialization errors caused by mismatched field layouts between schema versions.
Why this is optimal
This pattern conforms to standard Avro binary schema resolution while utilizing Pub/Sub's native schema revision metadata, enabling decoupled, robust streaming ingestion pipelines.
Configure Cloud Pub/Sub to automatically rewrite and translate all message payloads to the subscriber's expected reader schema version before message delivery.
Configure Cloud Pub/Sub to automatically rewrite and translate all message payloads to the subscriber's expected reader schema version before message delivery.
What this option does
This option presumes that Cloud Pub/Sub provides a managed server-side payload transformation layer to transcode and rewrite message bodies between different Avro schema revisions on the fly.
Why it doesn't meet the requirement
- Non-existent Pub/Sub feature: Pub/Sub validates messages against topic schemas at publication time, but it does not alter, mutate, or transcode message payloads to match subscriber schema revisions during message delivery.
- Consumer responsibility: Schema resolution in binary formats must occur on the client side during deserialization using the appropriate writer and reader schemas.
Key limitations
- Pub/Sub delivers the exact raw byte payload published by the producer along with metadata attributes.
- Server-side automatic format translation between arbitrary schema revisions is not supported by Pub/Sub subscriptions.
This answer would be correct if
Pub/Sub offered an inline schema transformation service at the subscription layer, which is not part of the Pub/Sub messaging architecture.
Try 10 interactive questions like this one, free
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
- 1,353 practice questions across all 5 exam domains
- Every answer explained: why it is right, and why the others are not
- Scout, an AI tutor that answers your follow-up questions
- Readiness tracking by domain, so you know when to book the exam
Related questions
Keep the momentum going with these hand-picked practice scenarios
AdvancedProfessional Data Engineer
Storage Transfer Service On-Premises Agents
Data acquisition and import
AdvancedProfessional Data Engineer
Tink Envelope Encryption with Cloud HSM CMEK
Data acquisition and import
AdvancedProfessional Data Engineer
Cloud Storage Lifecycle BigQuery Partitioning Clustering Materialized Views
Building a data platform based on requirements by using Google Cloud tools (e.g., Dataplex, Dataplex Catalog, BigQuery, Cloud Storage)
AdvancedProfessional Data Engineer
Data Sovereignty With CMEK And VPC Service Controls
Regional considerations (data sovereignty) for data access and storage
AdvancedProfessional Data Engineer
Cloud Monitoring BigQuery Query Execution Alerts
Monitoring the data lake
AdvancedProfessional Cloud DevOps Engineer
Cloud Trace Critical Path Analysis
Analyzing trace waterfalls and spans
Last updated Validated