Spec Model

Summary

This chapter covers the declarative specification layer of mountainash-treespec. You will learn about the TreeSpec, NodeSpec, and ChildSpec dataclasses that define how tree nodes map to table rows, along with MetaConfig, SettingsConfig, RoutingConfig, RoutingRule, PivotEntry, BatchingConfig, DedupConfig, DedupTableConfig, SourceColumn, and StreamingConfig. The chapter concludes with the Spec Loader and Spec Validator that parse and validate YAML/JSON spec files.

Concepts Covered

• TreeSpec Dataclass
• MetaConfig
• SettingsConfig
• NodeSpec Dataclass
• ChildSpec Dataclass
• RoutingConfig
• RoutingRule
• PivotEntry
• BatchingConfig
• DedupConfig
• DedupTableConfig
• SourceColumn
• StreamingConfig
• Spec Loader
• Spec Validator

Prerequisites

• Chapter 1: Foundation Concepts (Python Dataclasses, Primary Keys, Foreign Keys, YAML Configuration)

Learning Objectives

After completing this chapter, you will be able to:

• Describe the role of TreeSpec, NodeSpec, and ChildSpec in defining tree-table mappings
• Configure MetaConfig and SettingsConfig for a TreeSpec
• Define routing rules and pivot entries for polymorphic child dispatch
• Set up batching, dedup, and streaming configurations
• Load a TreeSpec from YAML/JSON and validate its structure

---

Introduction

The spec model is the declarative heart of mountainash-treespec. Rather than writing imperative code to traverse trees and populate tables, you write a specification that declares the mapping between tree structure and relational schema. The engines (flatten in Chapter 3, build in Chapter 6) interpret this specification at runtime to perform the actual transformation.

This separation between specification and execution brings several benefits. Specs are serializable to YAML, making them versionable, shareable, and auditable. A single spec drives both directions of the transformation, ensuring consistency. And because the spec is data rather than code, it can be validated, visualized, and generated programmatically.

The spec model is implemented as a hierarchy of Python dataclasses. The top-level TreeSpec contains metadata, settings, node definitions, and optional batching configuration. Each node definition (NodeSpec) describes one entity type in the tree and its mapping to a relational table. Child relationships (ChildSpec) describe how parent nodes reference their children and how those children should be decomposed.

MetaConfig

The MetaConfig dataclass holds top-level metadata about the specification -- information that identifies the spec and configures global behavior that applies across all nodes.

MetaConfig has the following fields:

• name -- A human-readable identifier for the spec (e.g., "patient-spec")
• root_element -- The name of the root node in the nodes dictionary; this is where tree traversal begins
• version -- An optional version string for tracking spec revisions
• description -- An optional human-readable description
• source_format -- The expected input format: "xml", "json", or "protobuf" (defaults to "xml")
• propagate_keys -- A list of column names that should flow from parent to all descendant tables

The propagate_keys field deserves special attention. When you declare a key in propagate_keys, the flattener copies that column value from the parent row into every child and grandchild row, regardless of table depth. This enables analytical queries that join any child table directly back to the root without intermediate joins.

meta:
  name: clinical-data
  root_element: Patient
  version: "2.1"
  source_format: xml
  propagate_keys: [patient_id]

In this example, every table produced by the flattener (encounters, diagnoses, medications, procedures) will contain a patient_id column with the root patient's identifier.

SettingsConfig

The SettingsConfig dataclass controls engine-wide behaviors that affect how the flatten and build engines process data. It provides sensible defaults that most specs can leave unchanged.

Field	Default	Description
column_naming	"snake_case"	Strategy for converting field names to column names
null_handling	"skip_all_none"	What to do with rows where all values are None
synthetic_key_prefix	"_"	Prefix for auto-generated key columns
streaming	None	Optional StreamingConfig for large-file processing

The column_naming setting determines how Python-style field names (often CamelCase from xsdata-generated classes) translate to column names. The "snake_case" option converts PatientName to patient_name. This is the only supported mode in the current implementation; future versions may add alternatives.

The null_handling setting controls row emission. With "skip_all_none", the flattener suppresses rows where every column value is None. This prevents empty rows from accumulating in child tables when optional children are absent in the source tree.

The synthetic_key_prefix determines the naming pattern for auto-generated primary keys. When a child spec has no explicit join_key and the parent has no primary_key, the flattener creates a column named {prefix}{table_name}_id (e.g., _encounters_id) with incrementing integer values.

StreamingConfig

The StreamingConfig dataclass configures large-file streaming behavior, enabling the processing of XML files that exceed available memory. It is embedded optionally within SettingsConfig.

StreamingConfig has three fields:

• sink -- Where to write streaming output: "memory" (accumulate in RAM) or "parquet" (flush to disk periodically)
• flush_interval -- How many records to accumulate before flushing to disk (default: 10,000)
• output_dir -- Directory path for Parquet output files (default: "./output/")

When streaming is configured, the StreamingFlattener uses lxml's iterparse to process XML elements one at a time, feeding each parsed element through the standard TreeFlattener machinery. This produces identical output to batch processing while maintaining constant memory usage regardless of input file size.

settings:
  column_naming: snake_case
  streaming:
    sink: parquet
    flush_interval: 5000
    output_dir: ./output/tables/

TreeSpec Dataclass

The TreeSpec dataclass is the top-level container that holds the entire specification. It composes the metadata, settings, node definitions, hook references, and batching configuration into a single object that the engines consume.

The TreeSpec fields are:

• meta -- A MetaConfig instance with identification and global settings
• nodes -- A dictionary mapping node names to NodeSpec instances
• settings -- A SettingsConfig instance (defaults to sensible values if omitted)
• hooks -- A dictionary mapping hook categories to hook name/reference pairs
• batching -- An optional BatchingConfig for multi-file processing

TreeSpec provides three convenience methods that the engines use internally. The root_node property returns the NodeSpec for the root element. The all_tables method returns a list of (table_name, NodeSpec) tuples for every node that emits a table. The get_node method performs a safe lookup with a helpful error message when a node name is not found.

@dataclass
class TreeSpec:
    meta: MetaConfig
    nodes: dict[str, NodeSpec]
    settings: SettingsConfig = field(default_factory=SettingsConfig)
    hooks: dict[str, dict[str, str]] = field(default_factory=dict)
    batching: Optional[BatchingConfig] = None

The nodes dictionary is the structural core of the spec. Each key is a node name (matching a dataclass name in the class registry), and each value describes how that node maps to a table and what children it contains.

Diagram: TreeSpec Dataclass Composition

TreeSpec Dataclass Composition

Type: diagram sim-id: treespec-composition
Library: vis-network
Status: Specified

Purpose: Show the composition hierarchy of TreeSpec -- how MetaConfig, SettingsConfig, NodeSpec, ChildSpec, RoutingConfig, BatchingConfig, and DedupConfig nest within the top-level TreeSpec container.

Components:

• Central TreeSpec node
• MetaConfig, SettingsConfig, BatchingConfig as direct children
• NodeSpec array branching into multiple NodeSpec instances
• Each NodeSpec containing ChildSpec instances
• ChildSpec optionally containing RoutingConfig or DedupConfig
• Color-coded by dataclass type

Interactions: Click any dataclass node to expand/collapse its fields. Hover to see field types and defaults in a tooltip panel. Toggle between "structure view" (composition) and "YAML view" (how it serializes).

Learning Objective: (Understand, Bloom Level 2) Students can identify the composition relationships between spec model dataclasses.

NodeSpec Dataclass

The NodeSpec dataclass describes how a single entity type in the tree maps to a relational table. Every entry in the TreeSpec.nodes dictionary is a NodeSpec.

NodeSpec fields:

• table -- The output table name (or None if this node does not emit its own table, e.g., an inline-only node)
• class_name -- Optional override for the dataclass class name (when the node name differs from the Python class)
• join_key -- Deprecated; use primary_key instead
• propagate_keys -- Additional keys to propagate from this node downward (supplements meta.propagate_keys)
• primary_key -- List of column names that uniquely identify rows in this node's table
• field_mapping -- Either "auto" (introspect the dataclass) or a dict mapping source field names to target column names
• children -- Dictionary mapping attribute names to ChildSpec instances
• resolved_fields -- Runtime-only list of ResolvedField objects populated by derive_treespec

The field_mapping field provides two modes of operation. In "auto" mode (the default), the flattener introspects the dataclass to discover all scalar fields and maps them using the column naming convention. In explicit dict mode, you specify exactly which fields appear in the output and what they are called:

nodes:
  Patient:
    table: patients
    primary_key: [patient_id]
    field_mapping:
      patientId: patient_id
      givenName: given_name
      familyName: family_name

The resolved_fields attribute is never set in YAML. Instead, the derive_treespec function populates it at runtime by introspecting the registered class through the schema handler registry. This gives the flattener pre-computed field metadata (column names, types, nesting status) that eliminates repeated introspection during traversal.

ChildSpec Dataclass

The ChildSpec dataclass is the richest configuration object in the spec model. It describes the relationship between a parent node and one of its children, controlling how that child is decomposed during flattening and reassembled during building.

The most important fields on ChildSpec determine the dispatch mode:

• node -- The name of the child's node definition in TreeSpec.nodes
• inline -- Boolean; if True and table is None, the child's fields merge into the parent row
• table -- If set, the child emits rows to this separate table
• routing -- If set, a RoutingConfig that dispatches items to different tables based on a discriminator attribute

These three options (inline, separate table, routing) are mutually exclusive dispatch modes. The is_inline and is_separate_table properties provide clean boolean checks:

@property
def is_inline(self) -> bool:
    return self.inline and self.table is None

@property
def is_separate_table(self) -> bool:
    return self.table is not None

Additional ChildSpec fields control the decomposition details:

• cardinality -- "one" or "many" (affects how the builder reconstructs: single object vs. list)
• prefix -- Column name prefix for inline children (e.g., "addr" produces addr_street, addr_city)
• variant -- Alternative to prefix, used for discriminated inline unions
• join_key -- Foreign key column(s) linking this child's table to the parent
• wrapper / wrapper_attr / child_node -- For wrapper patterns where an intermediate container holds the actual children
• condition -- Name of a registered condition hook that controls whether this child is processed
• positional_fields -- Maps list/tuple positional values to named fields
• lossy -- Boolean indicating the transformation intentionally loses information

Diagram: ChildSpec Dispatch Modes

ChildSpec Dispatch Modes

Type: infographic sim-id: childspec-dispatch-modes
Library: p5.js
Status: Specified

Purpose: Interactive decision tree showing how ChildSpec configuration determines which dispatch mode is used (inline, separate table, or routed), with examples of the resulting table structures for each mode.

Components:

• Decision tree flowchart with three branches
• Each branch shows: the ChildSpec YAML config, the resulting table structure, and when to use it
• Animated highlighting as user selects a mode
• Sample data showing before (tree) and after (tables) for each mode

Interactions: Click on a dispatch mode to see its full configuration example. Toggle between "decision view" (when to use each) and "output view" (what tables result). Slider adjusts example cardinality to show one-to-one vs. one-to-many behavior.

Learning Objective: (Apply, Bloom Level 3) Students can select the appropriate ChildSpec dispatch mode for a given data modeling scenario.

RoutingConfig

The RoutingConfig dataclass enables polymorphic child dispatch -- routing different items from the same collection into different target tables based on a discriminator attribute value. This is essential when a parent has a heterogeneous child list (e.g., a list of "observations" where some are vital signs, some are lab results, and some are clinical notes).

RoutingConfig fields:

• attribute -- The field name on the child object to inspect for routing decisions
• rules -- List of RoutingRule entries defining match patterns and target tables
• pivot -- List of PivotEntry entries for values that should be inlined into the parent row
• overflow_table -- Optional fallback table for items that match no rule

The has_default property checks whether any rule is marked as the default fallback. The spec validator requires either a default rule or an overflow_table to ensure no items are silently dropped.

children:
  observations:
    node: Observation
    cardinality: many
    join_key: patient_id
    routing:
      attribute: category
      rules:
        - match: vital_sign
          table: vital_signs
        - match: lab_result
          table: lab_results
        - default: other_observations

RoutingRule

A RoutingRule is a single entry in a routing configuration's rules list. Each rule maps a discriminator value to a target table.

RoutingRule fields:

• table -- The target table name where matching items go
• match -- The attribute value that triggers this rule (None for the default rule)
• is_default -- Boolean marking this as the fallback for unmatched values

Rules are evaluated in order. The first rule whose match value equals the item's attribute value wins. If no rule matches and no default exists, the item goes to the overflow_table (if configured) or raises a validation error at spec validation time.

PivotEntry

A PivotEntry transforms a routed child item into a single column on the parent row rather than a row in a separate table. This is the spec model's mechanism for denormalization -- taking a value that would normally go into a child table and pulling it up to the parent.

PivotEntry fields:

• key -- The discriminator attribute value that triggers this pivot
• column -- The column name to create on the parent table
• type -- The column's data type (default: "string")
• value_field -- Which field on the child to use as the pivoted value (if None, uses the first non-discriminator field)

For example, if a patient has observations with category values of "height" and "weight", a pivot entry can extract these directly onto the patient row:

routing:
  attribute: category
  pivot:
    - key: height
      column: patient_height
    - key: weight
      column: patient_weight
  rules:
    - default: other_observations

The flattener processes pivot entries before routing rules. When an item's attribute value matches a pivot entry, the item's value is placed directly into the parent row's pivot column, and the item does not go to any child table.

BatchingConfig

The BatchingConfig dataclass configures multi-file batch processing. It sits at the top level of TreeSpec and controls how the BatchFlattener processes multiple input files and how the BatchBuilder reconstructs trees from combined tables.

Field	Default	Description
mode	"single_file"	Processing mode (currently only single_file supported)
source	None	Glob pattern for input files (e.g., "./data/*.xml")
concat	"vertical"	How to combine per-file tables: "vertical" or "horizontal"
source_columns	{}	Dict of column names to SourceColumn specs
inject_into	"all"	Which tables receive source columns: "all", "root_only", or a list
partition_key	None	Column used to split combined tables back into per-file partitions
dedup	None	Optional DedupConfig for removing duplicates after concatenation

The typical batch workflow is: glob for files, flatten each file independently, inject source columns to track provenance, concatenate all results vertically, then optionally deduplicate. The partition_key enables the reverse: the BatchBuilder uses it to split combined tables back into per-file partitions before rebuilding individual trees.

DedupConfig

The DedupConfig dataclass controls row deduplication after batch concatenation. When multiple files contain overlapping records (same patient appearing in multiple data exports), dedup removes duplicates based on configurable key columns and strategy.

DedupConfig fields:

• key -- List of column names that define row uniqueness
• strategy -- One of "first_wins", "last_wins", or "error"
• table -- Optional target table (if None, dedup applies to the root table)
• cascade -- Boolean; if True, dedup cascades through child tables maintaining referential integrity
• tables -- Optional per-table override dictionary mapping table names to DedupTableConfig

The cascade mode is the most sophisticated dedup behavior. When enabled, the dedup engine first deduplicates the root table, then for each child table, removes orphaned rows (whose FK no longer matches a surviving parent) before deduplicating the child itself. This process recurses through grandchildren and beyond.

DedupTableConfig

The DedupTableConfig provides per-table dedup overrides within a cascade dedup configuration. It allows different tables to use different dedup keys and strategies.

batching:
  dedup:
    key: [patient_id]
    strategy: first_wins
    cascade: true
    tables:
      encounters:
        key: [patient_id, encounter_date]
        strategy: last_wins
      medications:
        key: [patient_id, medication_code]
        strategy: first_wins

Each DedupTableConfig has two fields: key (list of column names) and strategy (dedup strategy string). When a table has no override in the tables dict, cascade dedup falls back to the table's declared primary_key, then to all-columns as a last resort.

SourceColumn

The SourceColumn dataclass specifies how to extract per-file metadata columns that get injected into the flattened output. These columns track the provenance of each row -- which file it came from, when that file was modified, or what value a root attribute had.

SourceColumn fields:

• from_ -- The extraction method: "filename_stem", "filename", "path", "file_stat", or "attribute"
• path -- Dot-delimited attribute path on the parsed tree root (used with from_: "attribute")
• stat -- Which file stat to extract: "mtime", "size", or "ctime" (used with from_: "file_stat")

batching:
  source_columns:
    source_file:
      from: filename_stem
    export_date:
      from: attribute
      path: meta.export_date
    file_modified:
      from: file_stat
      stat: mtime

Source columns are injected into table rows during the batch flatten phase. The inject_into setting on BatchingConfig controls which tables receive these columns. Setting it to "root_only" keeps source columns off child tables, which is useful when the partition key (often the source filename) should only exist on root rows.

Spec Loader

The Spec Loader is the parsing layer that transforms YAML (or raw Python dictionaries) into typed TreeSpec instances. It provides two entry points: load_spec(path) for reading from a file, and load_spec_from_dict(raw) for constructing from an already-parsed dictionary.

The loading process walks the raw dictionary recursively, instantiating the appropriate dataclass for each section. Private helper functions (_parse_meta, _parse_settings, _parse_node, _parse_child, _parse_routing, _parse_batching) handle the details of each configuration block, applying defaults for missing fields and performing basic type coercion.

One notable behavior: the loader handles the deprecated join_key field on NodeSpec. If a node declares join_key without primary_key, the loader emits a deprecation warning and treats it as primary_key. This backward compatibility support ensures existing specs continue to work while encouraging migration to the preferred field name.

def load_spec(path: Union[str, Path]) -> TreeSpec:
    """Load a TreeSpec from a YAML file."""
    with open(path) as f:
        raw = yaml.safe_load(f)
    return load_spec_from_dict(raw)

The loader does not perform structural validation beyond what the dataclass constructors enforce. For comprehensive validation against a class registry, use the Spec Validator after loading.

Diagram: Spec Loading and Validation Pipeline

Spec Loading and Validation Pipeline

Type: workflow sim-id: spec-loading-pipeline
Library: p5.js
Status: Specified

Purpose: Visualize the pipeline from YAML text to validated TreeSpec, showing the intermediate stages (raw dict, parsed dataclasses, validated spec) and what errors can occur at each stage.

Components:

• Pipeline stages: YAML file, yaml.safe_load, parse* functions, TreeSpec object, validate_spec
• Error callouts at each stage (YAML syntax errors, missing required fields, validation failures)
• Sample YAML flowing through the pipeline with highlighted sections

Interactions: Click each stage to see its input/output types. Toggle "error mode" to inject common mistakes and see where they are caught. Hover over error callouts to see example error messages.

Learning Objective: (Apply, Bloom Level 3) Students can diagnose at which pipeline stage a spec configuration error will be caught.

Spec Validator

The Spec Validator (validate_spec function) performs structural and semantic validation of a loaded TreeSpec against a class registry and optional hook registry. It catches configuration errors that the loader cannot detect because they depend on the actual Python classes being used.

The validator performs two categories of checks:

Hard errors (raise exceptions immediately):

• Unknown node names not found in the class registry (UnknownClassError)
• Condition hooks referenced but not registered (UnregisteredHookError)
• Routing configurations without a default or overflow table (MissingDefaultRouteError)
• Routing attribute not present on the child class (InvalidRoutingAttributeError)
• Join key columns not present on the parent class (InvalidJoinKeyError)
• Wrapper attribute not present on the wrapper class (InvalidWrapperError)

Soft errors (collected and returned as a list of strings):

• Deprecated join_key conflicting with primary_key on the same NodeSpec
• Diamond paths where the same child table is reachable from multiple parents without explicit join keys
• Incompatible dedup configurations (cascade with strategy="error")

The validator also emits UncoveredFieldsWarning when a dataclass has fields that are neither mapped nor declared as children. This helps catch accidental omissions where new fields added to a dataclass are silently dropped during flattening.

errors = validate_spec(spec, class_registry, hooks=hook_registry)
if errors:
    for err in errors:
        print(f"Warning: {err}")

Putting It All Together

A complete specification connects all these pieces into a coherent declaration of how your data model decomposes into tables. Here is a realistic example that uses most of the spec model features:

meta:
  name: clinical-record
  root_element: Patient
  version: "3.0"
  propagate_keys: [patient_id]

settings:
  column_naming: snake_case
  null_handling: skip_all_none

nodes:
  Patient:
    table: patients
    primary_key: [patient_id]
    children:
      address:
        node: Address
        inline: true
        prefix: addr
      observations:
        node: Observation
        cardinality: many
        join_key: patient_id
        routing:
          attribute: category
          pivot:
            - key: height
              column: height_cm
          rules:
            - match: vital_sign
              table: vital_signs
            - default: other_obs

  Address:
    children: {}

  Observation:
    children: {}

batching:
  source: "./exports/*.xml"
  source_columns:
    source_file:
      from: filename_stem
  partition_key: source_file
  dedup:
    key: [patient_id]
    strategy: first_wins
    cascade: true

This spec declares that Patient is the root, with an inline address (prefixed columns) and routed observations (vital signs go to one table, heights get pivoted onto the patient row, everything else goes to a default table). Batch processing globs XML files, injects the source filename, and deduplicates patients by ID with cascade cleanup.

Diagram: Complete Spec Model Relationships

Complete Spec Model Relationships

Type: graph-model sim-id: spec-model-relationships
Library: vis-network
Status: Specified

Purpose: Interactive graph showing all 15 spec model concepts and their relationships (composition, reference, dependency). Users can explore how configurations connect.

Components:

• 15 nodes representing spec model concepts
• Composition edges (solid) showing dataclass containment
• Reference edges (dashed) showing cross-references (e.g., RoutingRule references a table name that matches a NodeSpec.table)
• Node tooltips showing the dataclass field list
• Color-coded by functional area (core in blue, routing in purple, batching in gold)

Interactions: Click a node to highlight all related nodes. Filter by functional area. Toggle between "composition view" and "data flow view" (how data moves through the spec during flatten/build).

Learning Objective: (Analyze, Bloom Level 4) Students can trace how changes to one spec model component affect the behavior of related components.

Key Takeaways

• TreeSpec is the top-level container composing MetaConfig, SettingsConfig, node definitions, hooks, and optional batching.
• MetaConfig identifies the spec and declares global propagate_keys that flow to all descendant tables.
• SettingsConfig controls column naming, null handling, synthetic key prefixes, and optional streaming.
• NodeSpec maps a single entity type to a table, with auto or explicit field mapping and a children dictionary.
• ChildSpec determines dispatch mode (inline, separate table, or routed) and carries cardinality, join keys, and prefixes.
• RoutingConfig enables polymorphic dispatch where a discriminator attribute routes items to different tables, with PivotEntry for denormalization.
• BatchingConfig controls multi-file processing with source columns, concatenation, partitioning, and dedup.
• The Spec Loader parses YAML into typed dataclasses; the Spec Validator checks structural correctness against actual Python classes.