Skip to content

Recordstep

matchlab.recordstep

RecordStep, the records a model matches over, and the verbs that build a plan.

A RecordStep is a step whose artifact is a table of records carrying an id. Source, Resolver and Transform are the three kinds. On a Source, id is the leaf. On a Resolver, id is the entity root. Every RecordStep chains the same verbs (select, clean, group, transform, dedupe, link). A Model reads a RecordStep and yields edges, not records.

RecordStep is not user-facing. Users hold a Source, a Resolver or a Transform and call verbs on it. Each concrete kind supplies only how it materialises (_read_cache) and which source rows it stands for (_identifier_reads).

A record step does not store identifiers(), the (id, source, key, leaf) mapping a downstream resolver needs. Reshaping cannot change it, so it is read back from the source leaves and the upstream resolver output instead.

Classes:

  • RecordStep –

    A step whose artifact is a table of records a model matches over.

Functions:

  • build_record_step –

    Assemble the id + qualified-columns table from stored extracts and identifiers.

RecordStep 

RecordStep()

Bases: Step


              flowchart TD
              matchlab.recordstep.RecordStep[RecordStep]
              matchlab.steps.Step[Step]

                              matchlab.steps.Step --> matchlab.recordstep.RecordStep
                


              click matchlab.recordstep.RecordStep href "" "matchlab.recordstep.RecordStep"
              click matchlab.steps.Step href "" "matchlab.steps.Step"

A step whose artifact is a table of records a model matches over.

Initialise a plan node.

Steps have no names. They are identified by position, where they fall in lineage.walk, which is the order collect runs them in and the order PlanDocument lists them in. step 7 in a log, [7] in draw(), and steps[7] in a document are therefore the same node.

A position is not stored here, because it is not a property of the step. It belongs to the walk it came from, and the same step numbers differently in walk(deduped) and walk(companies). Whoever does the walking passes it to whoever needs it, collect to its reporter, draw to its own renderer.

Finding a result later is a separate matter, and a separate act. Resolver.publish points a label at a resolver's output. Source is the one step with a name, and it means something else again. A source's name is part of its output, prefixing every column it contributes and tagging its rows.

Methods:

  • identifiers –

    Return (id, source, key, leaf) for every record this record step reads.

  • data –

    Return this record step's records, collecting the plan first if needed.

  • transform –

    Reshape this record step with a transformer.

  • select –

    Keep only the named columns, plus id.

  • clean –

    Derive columns with DuckDB SQL, keeping the rest.

  • group –

    Collapse each id to one row using aggregate SQL.

  • dedupe –

    Deduplicate this record step.

  • link –

    Link this record step to another. A Source is one, needing no wrapping.

  • __init_subclass__ –

    Derive _METHODOLOGY from the *_resources parameter __init__ declares.

  • __setattr__ –

    Refuse writes to what a spec, or the plan's shape, is built from.

  • __repr__ –

    Return a short representation showing kind and collection state.

  • __str__ –

    How this step appears in a drawing.

  • collect –

    Materialise this step and everything it depends on.

  • lineage –

    Return this step and all its inputs, upstream-first.

  • draw –

    Render this step's sub-plan as a tree.

  • fingerprints –

    Address every artifact this plan is made of, its own and its inputs'.

Attributes:

  • parents (tuple[Step, ...]) –

    This step's direct inputs, in the order they fold into its fingerprint.

  • is_collected (bool) –

    Whether this step has been materialised.

  • spec (BaseModel) –

    This step's settings, as a serialisable model.

parents abstractmethod property 

parents: tuple[Step, ...]

This step's direct inputs, in the order they fold into its fingerprint.

The kind-agnostic view of a step's edges.

is_collected property 

is_collected: bool

Whether this step has been materialised.

spec abstractmethod property 

spec: BaseModel

This step's settings, as a serialisable model.

One model per step kind, in matchlab.specs. It must carry everything this step's output depends on and nothing else. That is the invariant _spec_key rests on, and the one to check when adding a setting. Omit something that changes the output and collect will hand back a stale artifact without re-running (see _fingerprint).

Specs describe a step's own settings, not its inputs'. Edges live on parents, and _fingerprint already folds in their fingerprints.

identifiers 

identifiers(store: Store) -> DataFrame

Return (id, source, key, leaf) for every record this record step reads.

id is the resolver's entity root when reading through one, otherwise the source leaf. This is the upstream resolver output a downstream resolver needs to carry every reachable leaf forward, including records no model matched.

data 

data(return_type: DataFrameType = POLARS) -> DataFrameClass

Return this record step's records, collecting the plan first if needed.

transform 

transform(transformer: Transformer | type[Transformer] | str, transformer_settings: dict | None = None, transformer_resources: dict | None = None) -> Transform

Reshape this record step with a transformer.

select 

select(*columns: str) -> Transform

Keep only the named columns, plus id.

clean 

clean(cleaning: dict[str, SQLExpression]) -> Transform

Derive columns with DuckDB SQL, keeping the rest.

group 

group(aggregates: dict[str, SQLExpression]) -> Transform

Collapse each id to one row using aggregate SQL.

dedupe 

dedupe(model_class: type[Deduper] | str, model_settings: dict | None = None, model_resources: dict | None = None) -> Model

Deduplicate this record step.

link(other: RecordStep, model_class: type[Linker] | str, model_settings: dict | None = None, model_resources: dict | None = None) -> Model

Link this record step to another. A Source is one, needing no wrapping.

__init_subclass__ 

__init_subclass__(**kwargs: object) -> None

Derive _METHODOLOGY from the *_resources parameter __init__ declares.

A step that takes no such parameter builds no methodology, so it keeps whatever it inherits. That covers RecordStep and any test double subclassing Step directly.

Raises:

  • TypeError –

    If __init__ declares more than one *_resources parameter, which leaves no single prefix to quote back at a caller.

__setattr__ 

__setattr__(name: str, value: object) -> None

Refuse writes to what a spec, or the plan's shape, is built from.

__repr__ 

__repr__() -> str

Return a short representation showing kind and collection state.

__str__ 

__str__() -> str

How this step appears in a drawing.

collect 

collect(store: Store | None = None, interactive: bool | None = None) -> Self

Materialise this step and everything it depends on.

Steps whose artifact is already stored are skipped without being run, so re-collecting after adding a downstream step only does the new work.

Reports as it goes: the plan, a record per step, and a closing summary of what ran, what was cached, how long it took and what the store now holds. No logging setup is needed for any of that — a collection lends the matchlab logger a console handler where the application hasn't configured one, and leaves an application that has entirely alone. See matchlab.core.logging.audible.

Parameters:

  • store
    (Store | None, default: None ) –

    Where to read and write artifacts. Defaults to the module-level store (a DuckDB store in the user cache directory).

  • interactive
    (bool | None, default: None ) –

    Whether someone is watching. None, the default, takes a terminal or a notebook as a yes. When they are, the plan is drawn as a live tree redrawn in place, and not logged. The tree on screen is the key those [step N] records need, and it stays there. When they are not, the plan is logged instead. See matchlab.progress.

Returns:

  • Self –

    This step, now collected.

lineage 

lineage() -> list[Step]

Return this step and all its inputs, upstream-first.

draw 

draw() -> str

Render this step's sub-plan as a tree.

fingerprints 

fingerprints() -> set[Fingerprint]

Address every artifact this plan is made of, its own and its inputs'.

Which artifacts belong to a plan is the plan's own business, so this is where a store gets told: store.prune(keep=plan.fingerprints()) hands storage a set of addresses it already understands, rather than a graph it would have to learn to walk.

Returns:

  • set[Fingerprint] –

    One fingerprint per step in lineage(). A set, because two steps in one

  • set[Fingerprint] –

    plan can address the same artifact. Identical specs over identical

  • set[Fingerprint] –

    inputs is the same bytes, and it is stored once.

Raises:

  • RuntimeError –

    If any step has not been collected. An uncollected plan names no artifacts at all, so answering with a smaller set would quietly tell a caller that less is worth keeping than they think.

build_record_step 

build_record_step(store: Store, sources: tuple[Source, ...], resolver: Resolver | None) -> DataFrame

Assemble the id + qualified-columns table from stored extracts and identifiers.

Every source's extract is prefixed with the source name (company → crn_company) and joined to its identifiers, so each row gains the id it belongs to. That is the record's leaf when read directly, or the entity root when read through resolver. Several sources are concatenated diagonally, each row carrying its own source's columns and nulls for the rest. This is what both Source and Resolver read.