Steps¶
The basic building block of a matchlab plan.
The lazy plan node.
A Step holds references to its inputs only (parents). There is no registry,
no parent pointer, and no downstream list. "The DAG" is whatever is reachable
upstream from the node you hold, and lineage operations are pure functions of a root
node (matchlab.lineage).
Nothing is computed until collect(). Collection walks the plan upstream-first and
runs only the steps whose artifact is not already stored.
Fingerprints identify artifacts. A step's fingerprint combines its kind, its
spec, and its inputs' fingerprints. Everything downstream of a source can therefore
derive it from the plan alone, before any work happens, and collect
can skip a cached step without running it.
Two kinds of step are keyed by more than their plan. A source is where raw data
enters, so its spec key includes a content hash of the data it read. Constructing a
fresh Source therefore re-reads the warehouse (the documented way to refresh), while
an existing Source object memoises its read. A step whose methodology declares no
version is keyed by a fresh nonce on every collect, because matchlab has been told
nothing about that code and will not cache what it cannot identify. See
matchlab.core.versioning.
A step is settled once built. Everything its spec is derived from refuses assignment, so a plan is changed by rebuilding it, not by editing a node in place.
Classes:
-
Step–A node in a lazy plan.
Bases: ABC
flowchart TD
matchlab.steps.Step[Step]
click matchlab.steps.Step href "" "matchlab.steps.Step"
A node in a lazy plan.
Initialise a plan node.
Steps have no names. They are identified by position, where they fall in
lineage.walk, which is the order collect runs them in and the order
PlanDocument lists them in. step 7 in a log, [7] in draw(), and
steps[7] in a document are therefore the same node.
A position is not stored here, because it is not a property of the step. It
belongs to the walk it came from, and the same step numbers differently in
walk(deduped) and walk(companies). Whoever does the walking passes it to
whoever needs it, collect to its reporter, draw to its own renderer.
Finding a result later is a separate matter, and a separate act.
Resolver.publish points a label at a resolver's output. Source is the
one step with a name, and it means something else again. A source's name is
part of its output, prefixing every column it contributes and tagging its rows.
Methods:
-
collect–Materialise this step and everything it depends on.
-
lineage–Return this step and all its inputs, upstream-first.
-
draw–Render this step's sub-plan as a tree.
-
fingerprints–Address every artifact this plan is made of, its own and its inputs'.
Attributes:
-
kind(StepKind) – -
parents(tuple[Step, ...]) –This step's direct inputs, in the order they fold into its fingerprint.
-
is_collected(bool) –Whether this step has been materialised.
-
spec(BaseModel) –This step's settings, as a serialisable model.
abstractmethod
property
¶
This step's direct inputs, in the order they fold into its fingerprint.
The kind-agnostic view of a step's edges.
abstractmethod
property
¶
This step's settings, as a serialisable model.
One model per step kind, in matchlab.specs. It must carry everything this
step's output depends on and nothing else. That is the invariant _spec_key
rests on, and the one to check when adding a setting. Omit something that
changes the output and collect will hand back a stale artifact without
re-running (see _fingerprint).
Specs describe a step's own settings, not its inputs'. Edges live on
parents, and _fingerprint already folds in their fingerprints.
collect(store: Store | None = None, interactive: bool | None = None) -> Self
Materialise this step and everything it depends on.
Steps whose artifact is already stored are skipped without being run, so re-collecting after adding a downstream step only does the new work.
Reports as it goes: the plan, a record per step, and a closing summary of what
ran, what was cached, how long it took and what the store now holds. No logging
setup is needed for any of that — a collection lends the matchlab logger a
console handler where the application hasn't configured one, and leaves an
application that has entirely alone. See matchlab.core.logging.audible.
Parameters:
-
(store¶Store | None, default:None) –Where to read and write artifacts. Defaults to the module-level store (a DuckDB store in the user cache directory).
-
(interactive¶bool | None, default:None) –Whether someone is watching.
None, the default, takes a terminal or a notebook as a yes. When they are, the plan is drawn as a live tree redrawn in place, and not logged. The tree on screen is the key those[step N]records need, and it stays there. When they are not, the plan is logged instead. Seematchlab.progress.
Returns:
-
Self–This step, now collected.
fingerprints() -> set[Fingerprint]
Address every artifact this plan is made of, its own and its inputs'.
Which artifacts belong to a plan is the plan's own business, so this is where
a store gets told: store.prune(keep=plan.fingerprints()) hands storage a set
of addresses it already understands, rather than a graph it would have to learn
to walk.
Returns:
-
set[Fingerprint]–One fingerprint per step in
lineage(). A set, because two steps in one -
set[Fingerprint]–plan can address the same artifact. Identical specs over identical
-
set[Fingerprint]–inputs is the same bytes, and it is stored once.
Raises:
-
RuntimeError–If any step has not been collected. An uncollected plan names no artifacts at all, so answering with a smaller set would quietly tell a caller that less is worth keeping than they think.