Format Handlers

Summary

This chapter covers the five format handler implementations that adapt different data sources to the SchemaHandler protocol. You will learn about the JSON Schema Handler with dataclass field introspection, the XML Schema Handler with xsdata integration (including the XML streaming handler, parser, and serializer), the Protobuf Schema Handler and parser, the Pydantic Schema Handler, and the Avro Schema Handler with Avro codegen.

Concepts Covered

• JSON Schema Handler
• Dataclass Field Introspection
• XML Schema Handler
• XML Streaming Handler
• XML Parser
• XML Serializer
• Protobuf Schema Handler
• Protobuf Parser
• Pydantic Schema Handler
• Avro Schema Handler
• Avro Codegen

Prerequisites

• Chapter 1: Foundation Concepts (Hierarchical Data, Python Dataclasses)
• Chapter 4: Type System (Xsdata Integration, UniversalType Enum)
• Chapter 5: Schema Handling (SchemaHandler Protocol)
• Chapter 2: Spec Model (StreamingConfig)

Learning Objectives

After completing this chapter, you will be able to:

• Use the JSON Schema Handler to introspect Python dataclasses
• Configure the XML Schema Handler with xsdata-generated classes
• Stream, parse, and serialize XML documents
• Set up the Protobuf Schema Handler for protocol buffer messages
• Integrate Pydantic models via the Pydantic Schema Handler
• Use Avro codegen to generate classes for the Avro Schema Handler

---

Introduction

The format handlers are the concrete implementations of the SchemaHandler protocol introduced in Chapter 5. Each handler adapts a specific data format's type system, introspection mechanisms, and construction patterns to the uniform interface that the flatten and build engines expect.

Mountainash-treespec ships with five format handlers, ranging from the always-available DataclassSchemaHandler to optional handlers that require additional dependencies. The handlers form a layered system: every subsequent handler builds on the DataclassSchemaHandler's foundation while adding format-specific capabilities.

This chapter examines each handler's implementation, its introspection strategy, its limitations, and the additional I/O utilities (parsers, serializers, codegen) that accompany some formats.

Dataclass Field Introspection

Before examining the handlers themselves, it is important to understand the shared field introspection utilities that most handlers use. The field_info module provides helper functions for unwrapping Python type annotations -- the complex wrappers like Optional[List[Annotated[str, ...]]] that real-world code produces.

The unwrapping pipeline processes type annotations in three stages:

1. _unwrap_annotated -- Peels typing.Annotated wrappers, extracting the underlying type from Annotated[T, metadata]
2. _unwrap_optional -- Detects Optional[T] (or T | None in PEP 604 syntax) and returns the inner type plus an is_optional flag
3. _unwrap_list -- Detects list[T] and returns the element type plus an is_list flag

These functions compose to handle arbitrarily nested wrappers. For Optional[list[Annotated[str, MaxLen(100)]]], the pipeline produces: inner type str, is_optional=True, is_list=True.

Two additional helpers support the handlers:

• _is_dataclass_type -- Returns True if a type object is a dataclass (used to detect nested children)
• _resolve_type_hints -- Resolves type hints with support for from __future__ import annotations (PEP 563 string annotations)

These utilities are shared across the DataclassSchemaHandler, XsdataSchemaHandler, and PydanticSchemaHandler, ensuring consistent type unwrapping behavior.

Diagram: Type Annotation Unwrapping Pipeline

Type Annotation Unwrapping Pipeline

Type: workflow sim-id: type-unwrapping-pipeline
Library: p5.js
Status: Specified

Purpose: Visualize how complex type annotations are progressively unwrapped through the three-stage pipeline (Annotated, Optional, List), showing the intermediate type and flags at each stage.

Components:

• Input: complex type annotation (e.g., Optional[list[Annotated[str, MaxLen(100)]]])
• Three processing stages with before/after type display
• Output: inner type + is_optional + is_list + is_nested flags
• Multiple example annotations to cycle through

Interactions: Click to cycle through example type annotations of increasing complexity. Hover over each stage to see the exact transformation applied. Toggle "verbose mode" to see the Python code for each unwrapping function.

Learning Objective: (Apply, Bloom Level 3) Students can predict the result of unwrapping any Python type annotation through the three-stage pipeline.

JSON Schema Handler

The DataclassSchemaHandler (confusingly named "JSON" because it handles the JSON/dataclass format family) is the default handler for plain Python dataclasses. It is the only handler that is always available regardless of installed dependencies, and it serves as the fallback when no more specific handler matches.

The handler's can_handle method checks isinstance(cls, type) and dataclasses.is_dataclass(cls). This broad check matches all dataclasses, including xsdata-generated ones. The registry ordering ensures that more specific handlers (Xsdata, Pydantic) get priority when applicable.

The get_fields implementation iterates over dataclasses.fields(), resolves type hints, and runs each annotation through the unwrapping pipeline:

def get_fields(self, cls: type) -> list[_FieldInfo]:
    result = []
    hints = _resolve_type_hints(cls)
    for f in dc_fields(cls):
        raw_type = hints.get(f.name, f.type)
        raw_type = _unwrap_annotated(raw_type)
        inner, is_optional = _unwrap_optional(raw_type)
        inner = _unwrap_annotated(inner)
        inner, is_list = _unwrap_list(inner)
        inner = _unwrap_annotated(inner)
        is_nested = _is_dataclass_type(inner)
        universal = resolve_universal_type(inner) if not is_nested else UniversalType.OBJECT
        result.append(_FieldInfo(...))
    return result

The JSON interop methods use dacite for dict-to-dataclass conversion (from_json_dict) and a recursive _tree_to_dict function for dataclass-to-dict conversion (to_json_dict). The tree-to-dict conversion handles xsdata date types by calling str() on them, ensuring JSON serialization works even for xsdata-generated objects processed through the dataclass handler.

XML Schema Handler

The XsdataSchemaHandler handles xsdata-generated dataclasses, which carry rich XML Schema metadata on their fields. It takes priority over the DataclassSchemaHandler for classes that have a Meta inner class and xsdata-style metadata on their fields.

The key differentiator is metadata awareness. Where the DataclassSchemaHandler can only determine type, optionality, and nesting from Python annotations, the XsdataSchemaHandler reads additional metadata:

• Field type -- Whether the field represents an XML element, attribute, text content, or wildcard
• Constraints -- min/max length, patterns, numeric bounds extracted from XML Schema restrictions
• Cardinality -- min_occurs and max_occurs from the schema definition
• Choice groups -- Mutually exclusive elements from <xs:choice> constructs

The can_handle check is deliberately specific:

def _has_xsdata_metadata(cls: type) -> bool:
    if not (isinstance(cls, type) and dataclasses.is_dataclass(cls)):
        return False
    if not hasattr(cls, "Meta"):
        return False
    for f in dc_fields(cls):
        if f.metadata and "type" in f.metadata:
            return True
    return False

This three-part check ensures that only genuine xsdata classes are handled, preventing false positives on dataclasses that happen to have a Meta class for other reasons.

The constraint extraction logic translates xsdata metadata keys into FieldConstraints objects:

xsdata metadata key	FieldConstraints field
min_length	min_length
max_length	max_length
length	Both min_length and max_length
pattern	pattern
min_inclusive	minimum
max_inclusive	maximum

XML Parser

The XML parser (parse_xml function in formats/xml/parser.py) uses xsdata's XmlParser to parse XML files into typed dataclass instances. It serves as the default parser for the BatchFlattener when no custom parser is provided.

The parser accepts a file path and a target class (the root element's xsdata-generated dataclass), and returns a typed instance ready for flattening. Under the hood, xsdata handles namespace resolution, element-to-field mapping, and type coercion from XML string values to Python types.

The parser is used in the standard batch processing workflow:

from mountainash_treespec.formats.xml.parser import parse_xml

tree = parse_xml(Path("patient.xml"), Patient)
# tree is a Patient dataclass instance with all children populated

XML Serializer

The XML serializer complements the parser by converting typed dataclass instances back into XML documents. This enables the full round trip: XML -> dataclass tree -> flat tables -> dataclass tree -> XML.

The serializer uses xsdata's XmlSerializer with configurable output formatting. It handles namespace declarations, element ordering, and type serialization (converting Python types back to XML string representations).

Together, the parser and serializer provide the I/O layer for XML-based workflows. The flatten and build engines handle the tree-to-table transformation, while the parser and serializer handle the XML-to-tree conversion.

XML Streaming Handler

The StreamingFlattener handles XML files that are too large to parse entirely into memory. It combines lxml's iterparse with xsdata's XmlParser to process elements one at a time.

The streaming architecture works as follows:

1. lxml's iterparse reads the XML file element by element, yielding each depth-1 child element as it closes
2. The streaming handler feeds each element to xsdata's XmlParser to produce a typed dataclass instance
3. The typed instance is passed to a standard TreeFlattener for decomposition into tables
4. Results are accumulated in memory or flushed to Parquet files based on the StreamingConfig.sink setting

class StreamingFlattener:
    def __init__(
        self,
        spec: TreeSpec,
        hooks: Optional[HookRegistry] = None,
        class_registry: Optional[dict[str, type]] = None,
        child_tags: Optional[dict[str, tuple[str, type]]] = None,
        element_filter: Optional[Callable] = None,
    ):

The child_tags dictionary maps XML tag names to (node_name, class) tuples, telling the streaming handler which xsdata class to parse each element into. The optional element_filter callback enables selective processing -- skipping elements based on tag name or attributes without parsing them.

Streaming Limitations

The streaming handler processes only depth-1 child elements. Root-level attributes are extracted, but the root element itself is not treated as a complete tree. Inline children (which require parent and child to share a row) are not supported in streaming mode. Use the standard TreeFlattener for specs with inline children.

Diagram: Format Handler Capabilities Comparison

Format Handler Capabilities Comparison

Type: chart sim-id: handler-capabilities-matrix
Library: Chart.js
Status: Specified

Purpose: Interactive comparison matrix showing what capabilities each format handler provides: field introspection depth, constraint extraction, choice group support, JSON interop, streaming support, and codegen availability.

Components:

• Five handler columns (Dataclass, Xsdata, Protobuf, Pydantic, Avro)
• Capability rows with filled/empty indicators
• Radar chart overlay showing capability coverage per handler
• Dependency information (which Python packages each handler requires)

Interactions: Click a handler column to see its full capability details. Toggle between matrix view and radar chart view. Hover over capability cells to see implementation details.

Learning Objective: (Evaluate, Bloom Level 5) Students can select the appropriate format handler for a project based on its data format and capability requirements.

Protobuf Schema Handler

The ProtobufSchemaHandler adapts Protocol Buffer message classes (generated by protoc with Python output) to the SchemaHandler protocol. It requires the protobuf package (version 5.0+) and detects messages through issubclass(cls, pb_message.Message).

Protobuf introspection differs fundamentally from dataclass introspection. Instead of Python type annotations, protobuf messages carry a DESCRIPTOR attribute that describes all fields through FieldDescriptor objects. The handler reads this descriptor to determine field names, types, repetition rules, and nesting.

The type mapping translates protobuf field types to UniversalType:

PROTO_TYPE_TO_UNIVERSAL = {
    FD.TYPE_STRING: UniversalType.STRING,
    FD.TYPE_BOOL: UniversalType.BOOLEAN,
    FD.TYPE_INT32: UniversalType.INTEGER,
    FD.TYPE_INT64: UniversalType.INTEGER,
    FD.TYPE_FLOAT: UniversalType.NUMBER,
    FD.TYPE_DOUBLE: UniversalType.NUMBER,
    FD.TYPE_ENUM: UniversalType.STRING,
    # ... additional integer types
}

Well-known protobuf types receive special handling. google.protobuf.Timestamp maps to DATETIME, google.protobuf.Duration maps to DURATION, and wrapper types (Int32Value, StringValue, etc.) map to their underlying scalar types.

Protobuf Parser

The Protobuf parser handles the conversion of serialized protobuf bytes or JSON representations into message instances. Unlike XML parsing (which has a standardized xsdata workflow), protobuf parsing depends on the specific generated message class.

The parser provides utilities for:

• Parsing from binary protobuf format using the message class's ParseFromString method
• Parsing from JSON using google.protobuf.json_format.Parse
• Converting between protobuf messages and Python dictionaries

The handler's construct method deserves special attention. Protobuf messages do not accept keyword arguments in their constructors the way dataclasses do. Instead, the handler creates an empty message and sets fields individually, handling the distinction between singular fields, repeated fields, and nested messages.

Pydantic Schema Handler

The PydanticSchemaHandler adapts Pydantic v2 BaseModel subclasses to the SchemaHandler protocol. It requires the pydantic package and detects models through issubclass(cls, BaseModel).

Pydantic models provide their own introspection API through cls.model_fields, which returns a dictionary of FieldInfo objects. The handler reads these to determine field types, optionality, and validation constraints.

The constraint extraction translates Pydantic's metadata annotations (from Annotated[int, Ge(0), Le(100)]) into FieldConstraints:

def _extract_pydantic_constraints(fi: FieldInfo) -> Optional[FieldConstraints]:
    kwargs = {}
    for m in fi.metadata:
        if hasattr(m, "ge"):
            kwargs["minimum"] = m.ge
        if hasattr(m, "le"):
            kwargs["maximum"] = m.le
        if hasattr(m, "min_length"):
            kwargs["min_length"] = m.min_length
        if hasattr(m, "pattern"):
            kwargs["pattern"] = m.pattern
    return FieldConstraints(**kwargs) if kwargs else None

The nesting detection checks for issubclass(tp, BaseModel) instead of dataclasses.is_dataclass(tp), since Pydantic models are not dataclasses. This means a tree can mix Pydantic models and dataclasses at different levels, as long as the appropriate handlers are registered.

The construct method uses Pydantic's standard model construction with validation, ensuring that field validators run during the build phase. This provides an additional layer of data integrity checking during tree reconstruction.

Avro Schema Handler

The AvroSchemaHandler adapts Apache Avro records to the SchemaHandler protocol. Avro schemas are defined in JSON-format .avsc files rather than in Python code, which creates a bootstrapping challenge: the handler needs Python classes to introspect, but Avro schemas define records in JSON.

The solution is the Avro codegen pipeline (covered in the next section), which generates Python dataclasses from Avro schemas. Once generated, these dataclasses are handled by the AvroSchemaHandler, which reads Avro-specific metadata to provide richer type information than the generic DataclassSchemaHandler would.

The Avro type system includes logical types that map to UniversalType through a dedicated mapping:

Avro Type	Logical Type	UniversalType
string	(none)	STRING
int	(none)	INTEGER
long	(none)	INTEGER
float	(none)	NUMBER
double	(none)	NUMBER
boolean	(none)	BOOLEAN
int	date	DATE
long	timestamp-millis	DATETIME
string	uuid	STRING
bytes	decimal	NUMBER

Avro Codegen

The Avro codegen utility (avro_to_dataclasses function) generates Python dataclass source code from Avro schema files. This bridges the gap between Avro's JSON-based schema definition and the Python-class-based world that mountainash-treespec operates in.

The codegen process:

1. Reads the .avsc JSON schema file
2. Recursively collects all record definitions (including nested records)
3. Maps Avro primitive types and logical types to Python type annotations
4. Generates dataclass definitions with appropriate field types, defaults, and imports
5. Writes the generated code to a Python file in the specified output directory

def avro_to_dataclasses(
    schema_path: str | Path,
    package: str,
    output_dir: str | Path,
) -> Path:

The primitive type mapping translates Avro types to Python annotations:

• "null" becomes None
• "boolean" becomes bool
• "int" / "long" becomes int
• "float" / "double" becomes float
• "string" becomes str
• "bytes" becomes bytes

Logical types add precision: "date" becomes datetime.date, "timestamp-millis" becomes datetime.datetime, and "decimal" becomes decimal.Decimal. Union types (Avro's way of expressing optionality) are converted to Optional[T] when one branch is "null".

The generated classes are standard Python dataclasses, which means they are automatically handled by the DataclassSchemaHandler. The Avro-specific handler adds awareness of Avro logical types and constraints that the generic handler would miss.

Diagram: Format Handler Architecture

Format Handler Architecture

Type: diagram sim-id: format-handler-architecture
Library: vis-network
Status: Specified

Purpose: Show the layered architecture of format handlers: the SchemaHandler protocol at the top, the five concrete handlers implementing it, and the supporting utilities (parsers, serializers, codegen) attached to each handler.

Components:

• Top layer: SchemaHandler protocol box
• Middle layer: five handler implementation boxes (Dataclass, Xsdata, Protobuf, Pydantic, Avro)
• Bottom layer: supporting utilities (XML Parser, XML Serializer, Streaming, Protobuf Parser, Avro Codegen)
• Arrows showing implementation relationships and utility dependencies
• Optional dependency indicators (dashed borders for handlers requiring extra packages)

Interactions: Click a handler to see its supported operations and required dependencies. Click a utility to see its relationship to the handler and usage examples. Toggle "dependency view" to see which pip packages each handler requires.

Learning Objective: (Understand, Bloom Level 2) Students can identify the architectural relationship between handlers, their supporting utilities, and the SchemaHandler protocol.

Choosing the Right Handler

The handler selection is largely automatic -- the registry dispatches to the most specific handler that can handle each class. However, understanding the capabilities of each handler helps when designing your data model.

• Use plain dataclasses (DataclassSchemaHandler) for JSON-native data or when you want the simplest possible setup with no extra dependencies
• Use xsdata classes (XsdataSchemaHandler) for XML data where you want rich constraint extraction and choice group support
• Use Pydantic models (PydanticSchemaHandler) when you need built-in validation during the build phase or are integrating with a Pydantic-based API
• Use Protobuf messages (ProtobufSchemaHandler) for gRPC service data or binary-efficient serialization
• Use Avro-generated classes (AvroSchemaHandler) for Avro-native data pipelines, especially in Kafka/streaming contexts

All five handlers produce identical SchemaFieldInfo output, so the choice of handler does not affect the flatten/build engine behavior. The differences lie in constraint richness, construction behavior, and the I/O utilities available.

Key Takeaways

• Dataclass field introspection utilities (unwrap Annotated, Optional, List) are shared across handlers for consistent type analysis.
• The JSON Schema Handler (DataclassSchemaHandler) is the always-available fallback that handles all Python dataclasses.
• The XML Schema Handler reads xsdata metadata for rich constraints, cardinality, and choice group information.
• The XML Streaming Handler combines lxml iterparse with xsdata parsing for constant-memory processing of large XML files.
• XML Parser and XML Serializer provide the I/O layer for XML round-trip workflows.
• The Protobuf Schema Handler reads protobuf descriptors and handles well-known types with special mappings.
• The Pydantic Schema Handler reads model_fields and extracts Pydantic v2 validation constraints.
• The Avro Schema Handler works with Avro-codegen-generated dataclasses, and Avro Codegen bridges .avsc schemas to Python code.