Skip to content

Transformers

The Transformer step. A transformer's methodology is a location, which it reads rows through and content-addresses them. It is where data enters a plan.

matchlab.transformers.transform

Transform, a plan node that reshapes one record step with a Transformer.

Transform is to Transformer what Model is to a Deduper/Linker, the lazy plan node that wraps a serialisable methodology, folds its configuration into a cache key, and runs it on collect. Its single input is a RecordStep, so transforms chain, each its own cached artifact.

Classes:

  • Transform –

    A record step reshaped by one transformer.

Functions:

Transform 

Transform(parent: RecordStep, transformer: Transformer | type[Transformer] | str, transformer_settings: dict | None = None, transformer_resources: dict[str, Any] | None = None)

Bases: RecordStep


              flowchart TD
              matchlab.transformers.transform.Transform[Transform]
              matchlab.recordstep.RecordStep[RecordStep]
              matchlab.steps.Step[Step]

                              matchlab.recordstep.RecordStep --> matchlab.transformers.transform.Transform
                                matchlab.steps.Step --> matchlab.recordstep.RecordStep
                



              click matchlab.transformers.transform.Transform href "" "matchlab.transformers.transform.Transform"
              click matchlab.recordstep.RecordStep href "" "matchlab.recordstep.RecordStep"
              click matchlab.steps.Step href "" "matchlab.steps.Step"

A record step reshaped by one transformer.

Define a transform.

Parameters:

  • parent

    (RecordStep) –

    The record step to reshape.

  • transformer

    (Transformer | type[Transformer] | str) –

    A Transformer instance, or a subclass / its registered name to build from transformer_settings. The instance form is what select, clean and group use.

  • transformer_settings

    (dict | None, default: None ) –

    The configuration dict, when transformer is a class or a name. Ignored when it is already an instance.

  • transformer_resources

    (dict[str, Any] | None, default: None ) –

    Resources the transformer needs that cannot be serialised, keyed by field name. See matchlab.resources.

Raises:

  • ValueError –

    If resources are given alongside an already-built instance, which has nowhere left to put them.

  • ResourceError –

    If a field was passed in the wrong one of transformer_settings and transformer_resources, or if this transformer's resources conflict with one already in the plan.

Methods:

  • collect –

    Materialise this step and everything it depends on.

  • lineage –

    Return this step and all its inputs, upstream-first.

  • draw –

    Render this step's sub-plan as a tree.

  • fingerprints –

    Address every artifact this plan is made of, its own and its inputs'.

  • identifiers –

    Return (id, source, key, leaf) for every record this record step reads.

  • data –

    Return this record step's records, collecting the plan first if needed.

  • transform –

    Reshape this record step with a transformer.

  • select –

    Keep only the named columns, plus id.

  • clean –

    Derive columns with DuckDB SQL, keeping the rest.

  • group –

    Collapse each id to one row using aggregate SQL.

  • dedupe –

    Deduplicate this record step.

  • link –

    Link this record step to another. A Source is one, needing no wrapping.

Attributes:

kind class-attribute 

kind: StepKind = TRANSFORM

transformer instance-attribute 

transformer: Transformer

transformer_settings instance-attribute 

transformer_settings: dict[str, Any]

transformer_resources instance-attribute 

transformer_resources: dict[str, Resource]

parents property 

parents: tuple[RecordStep, ...]

The record step this transform reshapes. See Model.parents on the type.

transformer_class property 

transformer_class: type[Transformer]

The class implementing this transform.

spec property 

spec: TransformSpec

The serialisable spec for this transform.

is_collected property 

is_collected: bool

Whether this step has been materialised.

collect 

collect(store: Store | None = None, interactive: bool | None = None) -> Self

Materialise this step and everything it depends on.

Steps whose artifact is already stored are skipped without being run, so re-collecting after adding a downstream step only does the new work.

Reports as it goes: the plan, a record per step, and a closing summary of what ran, what was cached, how long it took and what the store now holds. No logging setup is needed for any of that — a collection lends the matchlab logger a console handler where the application hasn't configured one, and leaves an application that has entirely alone. See matchlab.core.logging.audible.

Parameters:

  • store
    (Store | None, default: None ) –

    Where to read and write artifacts. Defaults to the module-level store (a DuckDB store in the user cache directory).

  • interactive
    (bool | None, default: None ) –

    Whether someone is watching. None, the default, takes a terminal or a notebook as a yes. When they are, the plan is drawn as a live tree redrawn in place, and not logged. The tree on screen is the key those [step N] records need, and it stays there. When they are not, the plan is logged instead. See matchlab.progress.

Returns:

  • Self –

    This step, now collected.

lineage 

lineage() -> list[Step]

Return this step and all its inputs, upstream-first.

draw 

draw() -> str

Render this step's sub-plan as a tree.

fingerprints 

fingerprints() -> set[Fingerprint]

Address every artifact this plan is made of, its own and its inputs'.

Which artifacts belong to a plan is the plan's own business, so this is where a store gets told: store.prune(keep=plan.fingerprints()) hands storage a set of addresses it already understands, rather than a graph it would have to learn to walk.

Returns:

  • set[Fingerprint] –

    One fingerprint per step in lineage(). A set, because two steps in one

  • set[Fingerprint] –

    plan can address the same artifact. Identical specs over identical

  • set[Fingerprint] –

    inputs is the same bytes, and it is stored once.

Raises:

  • RuntimeError –

    If any step has not been collected. An uncollected plan names no artifacts at all, so answering with a smaller set would quietly tell a caller that less is worth keeping than they think.

identifiers 

identifiers(store: Store) -> DataFrame

Return (id, source, key, leaf) for every record this record step reads.

id is the resolver's entity root when reading through one, otherwise the source leaf. This is the upstream resolver output a downstream resolver needs to carry every reachable leaf forward, including records no model matched.

data 

data(return_type: DataFrameType = POLARS) -> DataFrameClass

Return this record step's records, collecting the plan first if needed.

transform 

transform(transformer: Transformer | type[Transformer] | str, transformer_settings: dict | None = None, transformer_resources: dict | None = None) -> Transform

Reshape this record step with a transformer.

select 

select(*columns: str) -> Transform

Keep only the named columns, plus id.

clean 

clean(cleaning: dict[str, SQLExpression]) -> Transform

Derive columns with DuckDB SQL, keeping the rest.

group 

group(aggregates: dict[str, SQLExpression]) -> Transform

Collapse each id to one row using aggregate SQL.

dedupe 

dedupe(model_class: type[Deduper] | str, model_settings: dict | None = None, model_resources: dict | None = None) -> Model

Deduplicate this record step.

link(other: RecordStep, model_class: type[Linker] | str, model_settings: dict | None = None, model_resources: dict | None = None) -> Model

Link this record step to another. A Source is one, needing no wrapping.

add_transformer_class 

add_transformer_class(transformer_class: type[Transformer]) -> None

Register a custom transformer so it can be named in a plan (and a document).

matchlab.transformers.base

Base class for transformer methodologies, and the DuckDB query runner some share.

A Transformer is a pure function of a record step, called by a Transform step's apply() on every collect. Both input and output must carry id, the grouping every downstream model and resolver reads.

Transformers are declarative and serialisable. Their fields are the configuration a Transform folds into its spec and cache key, so keep them to plain data, with no callables, exactly as a Deduper's settings are.

Classes:

  • Transformer –

    Base contract every transformer implements.

Functions:

  • reject_id_output –

    Refuse id as a column a transformer writes.

  • run_sql –

    Execute one DuckDB query against data, registered as data.

Transformer

Bases: BaseModel, ABC


              flowchart TD
              matchlab.transformers.base.Transformer[Transformer]

              

              click matchlab.transformers.base.Transformer href "" "matchlab.transformers.base.Transformer"

Base contract every transformer implements.

Concrete transformers (Select, Clean, Group, Explode) carry their configuration as flat fields, so MyTransformer(...) reads naturally, and model_dump(mode="json") is the whole of its serialisation.

Frozen, and extra="forbid" so a mistyped setting is refused rather than silently ignored — which would leave the transform running a default under a fingerprint that never mentioned the field.

Every field is a setting unless marked matchlab.resources.FromResources. A fingerprint ignores a resource, so a marked field must not change the output.

Methods:

  • apply –

    Return data reshaped. Both the input and the output carry id.

Attributes:

model_config class-attribute instance-attribute 

model_config = ConfigDict(extra='forbid', frozen=True)

version class-attribute 

version: int | None = None

apply abstractmethod 

apply(data: DataFrame) -> DataFrame

Return data reshaped. Both the input and the output carry id.

reject_id_output 

reject_id_output(names: Iterable[str]) -> None

Refuse id as a column a transformer writes.

id is the grouping every model matches on, derived by matchlab from record content. A transformer that assigns to it changes which records a model treats as the same, and nothing downstream can tell that happened: the fingerprint covers the expression, not what the expression displaced.

Raises:

run_sql 

run_sql(query: SQLQuery, data: DataFrame) -> DataFrame

Execute one DuckDB query against data, registered as data.