Skip to content

Type System

Summary

This chapter covers the type system that bridges Python types, xsdata XML types, and PyArrow types through a unified abstraction. You will learn about the UniversalType enum, the Python-to-Universal and xsdata-to-Universal type maps, the resolve function that selects the correct mapping, and the Universal-to- PyArrow map. The chapter also covers the Arrow Schema Builder, FieldSpec Bridge, ResolvedField dataclass, SchemaFieldInfo protocol, and the Xsdata integration that underpins XML type handling.

Concepts Covered

  • Xsdata Integration
  • UniversalType Enum
  • Python Type to Universal Map
  • Xsdata Type Names Map
  • Resolve Universal Type
  • Universal to PyArrow Map
  • Arrow Schema Builder
  • FieldSpec Bridge
  • ResolvedField Dataclass
  • SchemaFieldInfo Protocol

Prerequisites

  • Chapter 1: Foundation Concepts (Hierarchical Data, Relational Tables, Python Dataclasses, Python Protocols, PyArrow Tables)

Learning Objectives

After completing this chapter, you will be able to:

  • List the values in the UniversalType enum and their purpose
  • Map Python types and xsdata type names to UniversalType
  • Resolve a field's type through the universal type system
  • Convert UniversalType values to PyArrow types
  • Build Arrow schemas from SchemaFieldInfo using the Arrow Schema Builder
  • Explain the role of FieldSpec Bridge and ResolvedField in type resolution

Introduction

Mountainash-treespec transforms data across multiple format boundaries. A single pipeline might read XML parsed by xsdata (with types like XmlDate and XmlDateTime), flatten it through Python dataclasses (with types like str, int, datetime.date), and write the output to Parquet via PyArrow (with types like pa.date32() and pa.timestamp("us")). Each of these systems has its own type vocabulary, and a direct mapping from every source type to every target type would produce a combinatorial explosion.

The type system solves this problem with a hub-and-spoke architecture. Every source type maps to a UniversalType value, and every UniversalType value maps to a target type. Adding a new source format requires only mapping its types to UniversalType. Adding a new output format requires only mapping UniversalType to the target's type vocabulary.

This chapter walks through each layer of the type system, from the foundation (UniversalType enum and Xsdata integration) through the mapping layers (Python-to-Universal, Xsdata-to-Universal, Universal-to-PyArrow) to the schema construction utilities (Arrow Schema Builder, FieldSpec Bridge, ResolvedField).

UniversalType Enum

The UniversalType enum is the central abstraction of the type system. It defines a closed set of type categories that are expressive enough to represent the types found in XML schemas, JSON Schema, Protocol Buffers, Avro, and Python type annotations, yet simple enough that every target format can map from them.

The UniversalType values are:

UniversalType Description Example source types
STRING Text data str, bytes, XmlPeriod
INTEGER Whole numbers int, proto INT32/INT64
NUMBER Floating-point numbers float, Decimal, proto DOUBLE
BOOLEAN True/false values bool
DATE Calendar date without time datetime.date, XmlDate
TIME Time of day without date datetime.time, XmlTime
DATETIME Date and time combined datetime.datetime, XmlDateTime
DURATION Time interval datetime.timedelta, XmlDuration
OBJECT Nested complex type Dataclass, BaseModel, proto Message
ARRAY List/repeated field list[T]
ANY Unknown or unresolvable type Fallback for unrecognized types

The enum is intentionally conservative. Rather than trying to model every possible type constraint (length limits, regex patterns, value ranges), it captures only the fundamental data category. Constraints are handled separately through the FieldConstraints dataclass in the FieldSpec system.

Xsdata Integration

Xsdata is a Python library that generates dataclasses from XML Schema (XSD) definitions. When mountainash-treespec processes XML data, xsdata-generated classes serve as the typed representation of XML elements. These classes look like standard Python dataclasses but carry additional metadata in their field definitions and use xsdata-specific date/time types.

The xsdata integration surfaces in three places in the type system:

  1. Xsdata date types -- Classes like XmlDate, XmlDateTime, XmlTime, and XmlDuration replace Python's standard datetime types. These need their own type map entries.

  2. Field metadata -- Xsdata fields carry metadata dictionaries with keys like "type" (element vs. attribute), "min_occurs", "max_occurs", and constraint information. The XML Schema Handler (Chapter 7) reads this metadata.

  3. Schema handler -- The XsdataSchemaHandler uses the Meta inner class and field metadata to provide richer type information than plain dataclass introspection can offer.

Understanding xsdata integration is important because many real-world mountainash-treespec deployments process XML data (healthcare records, financial data, government filings) where the schema is defined in XSD and the Python types are xsdata-generated.

Python Type to Universal Map

The Python-to-Universal type map is a dictionary that translates standard Python type annotations into UniversalType values. It handles the most common scalar types that appear in Python dataclass field annotations.

PYTHON_TYPE_TO_UNIVERSAL: dict[type, str] = {
    str: UniversalType.STRING,
    int: UniversalType.INTEGER,
    float: UniversalType.NUMBER,
    bool: UniversalType.BOOLEAN,
    Decimal: UniversalType.NUMBER,
    bytes: UniversalType.STRING,
    datetime.date: UniversalType.DATE,
    datetime.time: UniversalType.TIME,
    datetime.datetime: UniversalType.DATETIME,
    datetime.timedelta: UniversalType.DURATION,
}

The mapping makes pragmatic choices. Decimal maps to NUMBER rather than a separate decimal type because most output formats (PyArrow, Parquet) represent high-precision numbers as float64. bytes maps to STRING because binary data in relational tables is typically base64-encoded.

This map is the first lookup tried by the resolve_universal_type function. If a Python type appears here, the resolution is immediate. If not, the resolver falls through to the xsdata type map.

Xsdata Type Names Map

The Xsdata-to-Universal type map handles xsdata-specific date and time types that do not appear in the Python type map. Rather than mapping by type object (which would require importing xsdata as a hard dependency), it maps by type name string. This allows the map to work even when xsdata is not installed -- the type name lookup simply returns no match, and the resolver falls through to the ANY default.

XSDATA_TYPE_NAMES: dict[str, str] = {
    "XmlDate": UniversalType.DATE,
    "XmlDateTime": UniversalType.DATETIME,
    "XmlTime": UniversalType.TIME,
    "XmlDuration": UniversalType.DURATION,
    "XmlPeriod": UniversalType.STRING,
}

Notice that XmlPeriod maps to STRING rather than a date-related type. XML periods represent recurrence patterns (like "P1Y2M" for one year and two months) that have no direct equivalent in columnar data formats, so they are serialized as strings.

Diagram: Type System Hub-and-Spoke Architecture

Type System Hub-and-Spoke Architecture

Type: diagram sim-id: type-system-hub-spoke
Library: vis-network
Status: Specified

Purpose: Visualize the hub-and-spoke type mapping architecture where UniversalType sits at the center, source types (Python, xsdata, Protobuf, Avro) connect on the left, and target types (PyArrow, Parquet) connect on the right.

Components:

  • Central hub: UniversalType values arranged in a circle
  • Left spokes: Python types, xsdata types, Protobuf types, Avro types
  • Right spokes: PyArrow types, Parquet logical types
  • Directed edges from source to UniversalType, then UniversalType to target
  • Color-coded by source format family

Interactions: Click any source type to trace its resolution path through UniversalType to the target type. Click a UniversalType value to see all sources that map to it and all targets it maps to. Filter by format family.

Learning Objective: (Understand, Bloom Level 2) Students can trace the type resolution path from any source type through UniversalType to the corresponding PyArrow type.

Resolve Universal Type

The resolve_universal_type function is the entry point for type resolution. Given a Python type annotation (which might be a standard Python type, an xsdata type, or something else entirely), it returns the corresponding UniversalType string value.

The resolution follows a two-step priority chain:

  1. Direct lookup in PYTHON_TYPE_TO_UNIVERSAL -- if the type object is a key in the map, return immediately
  2. Name-based lookup in XSDATA_TYPE_NAMES -- extract the type's __name__ attribute and check the xsdata map
  3. Fallback -- return UniversalType.ANY if neither map has a match
def resolve_universal_type(python_type: Any) -> str:
    if python_type in PYTHON_TYPE_TO_UNIVERSAL:
        return PYTHON_TYPE_TO_UNIVERSAL[python_type]
    type_name = getattr(python_type, "__name__", str(python_type))
    if type_name in XSDATA_TYPE_NAMES:
        return XSDATA_TYPE_NAMES[type_name]
    return UniversalType.ANY

The ANY fallback ensures the type system never crashes on an unrecognized type. Instead, the field is treated as a generic value that will be serialized as a string in the output. This is a deliberate "fail open" design: missing type information produces a suboptimal but functional result rather than an error.

Universal to PyArrow Map

The Universal-to-PyArrow map completes the type resolution chain by translating UniversalType values into PyArrow type identifiers. This map is used by the Arrow Schema Builder to construct typed schemas for Parquet output.

UNIVERSAL_TO_PYARROW: dict[str, str] = {
    "string": "string",
    "integer": "int64",
    "number": "float64",
    "boolean": "bool_",
    "date": "date32",
    "datetime": "timestamp_us",
    "time": "time64_us",
    "duration": "int64",
    "year": "int32",
    "yearmonth": "string",
    "object": "string",
    "array": "string",
    "any": "string",
}

The map uses string keys (matching the UniversalType string values) that map to PyArrow type name strings. The resolve_pyarrow_type function then translates these name strings into actual pyarrow.DataType objects using a secondary mapping that calls PyArrow constructors.

Several design choices are worth noting:

  • duration maps to int64 rather than a native duration type, because not all query engines handle Arrow duration types consistently
  • object and array map to string, expecting JSON serialization for nested types that appear in scalar columns
  • any maps to string as the safest fallback -- any value can be string-serialized

SchemaFieldInfo Protocol

The SchemaFieldInfo protocol defines the information contract between format handlers and the rest of the type system. Every format handler (JSON, XML, Protobuf, Pydantic, Avro) must produce field information objects that satisfy this protocol.

The protocol requires these attributes:

  • name -- The field's Python attribute name
  • source_name -- The field's name in the source format (may differ from name for XML attributes vs. elements)
  • universal_type -- The resolved UniversalType (string or enum value)
  • is_optional -- Whether the field can be absent or None
  • is_list -- Whether the field is a collection (list, repeated, etc.)
  • is_nested -- Whether the field contains a child dataclass/message (not a scalar)
  • constraints -- Optional FieldConstraints with validation rules
  • nested_type -- The Python type of the nested child (if is_nested is True)
  • metadata -- Format-specific metadata dictionary

The is_nested flag is critical for the flattener. Nested fields are not extracted as scalar columns; instead, they trigger child dispatch. The flattener uses this flag to distinguish between "this field is a string that happens to be complex" and "this field is a child object that needs its own table or inline treatment."

Diagram: SchemaFieldInfo Data Flow

SchemaFieldInfo Data Flow

Type: workflow sim-id: schema-field-info-flow
Library: p5.js
Status: Specified

Purpose: Show how SchemaFieldInfo objects flow from format handlers through type resolution to Arrow schema construction, with each attribute highlighted at its point of use.

Components:

  • Three stages: Handler produces SchemaFieldInfo, Type resolver reads universal_type, Arrow Schema Builder reads all attributes
  • Sample SchemaFieldInfo objects with populated attributes
  • Arrows showing which consumer reads which attribute
  • Output: PyArrow field with name, type, and nullable flag

Interactions: Click any SchemaFieldInfo attribute to highlight everywhere it is consumed. Toggle between different source formats (dataclass, xsdata, Protobuf) to see how the same protocol is populated differently.

Learning Objective: (Analyze, Bloom Level 4) Students can identify which SchemaFieldInfo attributes influence each stage of the type resolution pipeline.

ResolvedField Dataclass

The ResolvedField dataclass is a runtime-only metadata object that stores pre-computed field information for use during flatten and build operations. Unlike SchemaFieldInfo (which is a protocol satisfied by different handler implementations), ResolvedField is a concrete dataclass with a fixed set of fields.

@dataclass
class ResolvedField:
    name: str          # Python attribute name
    column: str        # Target column name (after naming convention)
    universal_type: str
    kind: Literal["scalar", "nested", "list", "map"] = "scalar"
    source_name: str = ""
    metadata: dict[str, Any] = field(default_factory=dict)

ResolvedField instances are populated by the derive_treespec function, which iterates over each node's fields (via the schema handler registry) and pre-computes the column name, type, and nesting status. These pre-computed values are stored in NodeSpec.resolved_fields and used by the flattener to avoid repeated introspection during traversal.

The kind field uses a string literal union to classify the field:

  • "scalar" -- A simple value that becomes a column
  • "nested" -- A child dataclass that triggers child dispatch
  • "list" -- A repeated field (also triggers child dispatch)
  • "map" -- A dictionary field (rare, typically serialized as string)

Two convenience properties, is_nested and is_list, provide boolean checks. The is_nested property returns True for both "nested" and "list" kinds, because both represent child relationships rather than scalar columns.

FieldSpec Bridge

The FieldSpec Bridge connects mountainash-treespec's type system to the broader mountainash ecosystem's FieldSpec type (from mountainash.typespec). Each format handler implements a to_fieldspec method that converts a SchemaFieldInfo object into a FieldSpec instance.

The bridge serves two purposes:

  1. Standardization -- Different handlers produce SchemaFieldInfo objects with varying levels of detail. The to_fieldspec conversion normalizes this into a standard representation with a UniversalType enum value and optional FieldConstraints.

  2. Ecosystem integration -- Other mountainash packages that need field metadata (API clients, validation libraries, code generators) consume FieldSpec objects. The bridge ensures that metadata derived from any supported format (XML, JSON, Protobuf, Pydantic, Avro) is available in this common format.

The conversion process is straightforward:

def to_fieldspec(self, field_info: _FieldInfo) -> FieldSpec:
    ut = field_info.universal_type
    if isinstance(ut, str):
        try:
            ut = UniversalType(ut)
        except ValueError:
            ut = UniversalType.ANY
    constraints = None
    if not field_info.is_optional:
        constraints = FieldConstraints(required=True)
    return FieldSpec(name=field_info.name, type=ut, constraints=constraints)

Arrow Schema Builder

The Arrow Schema Builder (derive_arrow_schemas function) is the culmination of the type system. Given a TreeSpec and a class registry, it constructs a PyArrow schema for every table that the flattener will produce. These schemas enable strongly-typed Parquet output where column types are known before any data is processed.

The builder discovers columns from six distinct sources:

  1. Dataclass fields -- Scalar fields from each node's registered class, typed via the handler's get_fields method
  2. Foreign key columns -- From join_key declarations and parent primary_key fallbacks
  3. Propagated keys -- From meta.propagate_keys and node-level propagate_keys
  4. Synthetic keys -- Generated {prefix}{table}_id columns for children without explicit keys
  5. Pivot columns -- From RoutingConfig.pivot entries that add columns to the parent table
  6. Source columns -- From BatchingConfig.source_columns that add file-metadata columns

This six-source discovery mirrors the column-production logic of the flattener itself, ensuring the schema exactly matches the columns that will appear in the output.

Column Source Example PyArrow Type
Dataclass field patient_name: str pa.string()
Foreign key join_key: patient_id Inherits from parent PK type
Propagated key propagate_keys: [patient_id] pa.string() (default)
Synthetic key _encounters_id pa.int64()
Pivot column height_cm pa.string()
Source column source_file (filename_stem) pa.string()

Diagram: Arrow Schema Builder Column Discovery

Arrow Schema Builder Column Discovery

Type: microsim sim-id: arrow-schema-discovery
Library: Chart.js
Status: Specified

Purpose: Interactive visualization showing how the Arrow Schema Builder assembles a complete schema by discovering columns from all six sources, with a sample TreeSpec as input.

Components:

  • Input panel: TreeSpec configuration (editable YAML snippet)
  • Discovery panel: six source categories, each contributing columns
  • Output panel: the resulting PyArrow schema with field names, types, and nullable flags
  • Stacked bar chart showing column count by source for each table

Interactions: Toggle source categories on/off to see how the schema changes. Edit the YAML snippet to add/remove nodes, children, pivot entries, and source columns. Hover over schema fields to see which source contributed them.

Learning Objective: (Apply, Bloom Level 3) Students can predict the Arrow schema that will result from a given TreeSpec configuration.

Type Resolution End-to-End

To solidify the complete type resolution pipeline, consider a field date_of_birth: Optional[XmlDate] on an xsdata-generated dataclass:

  1. The XML Schema Handler introspects the field and produces a SchemaFieldInfo with universal_type="date", is_optional=True, is_nested=False
  2. The derive_treespec function creates a ResolvedField with column="date_of_birth", universal_type="date", kind="scalar"
  3. The Arrow Schema Builder calls resolve_pyarrow_type("date"), which looks up "date" in UNIVERSAL_TO_PYARROW to get "date32", then translates that to pa.date32()
  4. The final schema contains pa.field("date_of_birth", pa.date32(), nullable=True)

At runtime, the flattener uses the ResolvedField to extract the value, and the output Parquet file contains a properly typed date column rather than a string.

Key Takeaways

  • UniversalType is the hub of a hub-and-spoke architecture that prevents combinatorial type mapping between source and target formats.
  • The Python Type to Universal Map handles standard Python types; the Xsdata Type Names Map handles xsdata-specific date/time types via name-based lookup.
  • resolve_universal_type chains through both maps with an ANY fallback, ensuring the system never crashes on unrecognized types.
  • The Universal to PyArrow Map completes the chain, translating abstract types into concrete columnar storage types.
  • SchemaFieldInfo is the protocol contract between format handlers and the type system; ResolvedField is the pre-computed runtime cache.
  • The FieldSpec Bridge connects to the broader mountainash type ecosystem.
  • The Arrow Schema Builder discovers columns from six sources (fields, FK, propagated keys, synthetic keys, pivots, source columns) to produce complete typed schemas.
  • Xsdata integration underpins XML type handling with specialized date types and rich field metadata.