Skip to content

Models

The Model step, and the methodologies it runs. Methodologies are either dedupers or linkers.

matchlab.models.models

Model — a deduper or linker producing scored candidate edges.

Classes:

  • Model –

    A step that runs one methodology (a Deduper or a Linker) over record steps.

Functions:

  • add_model_class –

    Register a custom deduper or linker so it can be named in a plan.

  • normalise_model_scores –

    Validate a methodology's raw output and cast it to SCHEMA_MODEL_EDGES.

Model 

Model(left: RecordStep, model_class: type[Deduper] | type[Linker] | str, model_settings: dict[str, Any] | None = None, right: RecordStep | None = None, model_resources: dict[str, Any] | None = None)

Bases: Step


              flowchart TD
              matchlab.models.models.Model[Model]
              matchlab.steps.Step[Step]

                              matchlab.steps.Step --> matchlab.models.models.Model
                


              click matchlab.models.models.Model href "" "matchlab.models.models.Model"
              click matchlab.steps.Step href "" "matchlab.steps.Step"

A step that runs one methodology (a Deduper or a Linker) over record steps.

Define a model.

Parameters:

  • left

    (RecordStep) –

    The record step to deduplicate, or the left side of a link.

  • model_class

    (type[Deduper] | type[Linker] | str) –

    A Deduper/Linker subclass, or its registered name.

  • model_settings

    (dict[str, Any] | None, default: None ) –

    That class's configuration, as a dict.

  • right

    (RecordStep | None, default: None ) –

    The right side of a link. Omit for a deduper.

  • model_resources

    (dict[str, Any] | None, default: None ) –

    Resources the methodology needs that cannot be serialised, keyed by field name. See matchlab.resources.

Raises:

  • ValueError –

    If a linker was given no right input or a deduper was given one, or if the two inputs bring two different sources under one name.

  • ResourceError –

    If a field was passed in the wrong one of model_settings and model_resources, or if the two inputs bring one resource name over two different objects.

Methods:

  • edges –

    Return this model's scored edges. Collects the plan first if needed.

  • resolve –

    Resolve this model (and any others) into clusters.

  • collect –

    Materialise this step and everything it depends on.

  • lineage –

    Return this step and all its inputs, upstream-first.

  • draw –

    Render this step's sub-plan as a tree.

  • fingerprints –

    Address every artifact this plan is made of, its own and its inputs'.

Attributes:

kind class-attribute 

kind: StepKind = MODEL

left instance-attribute 

left: RecordStep

right instance-attribute 

right: RecordStep | None

model_class instance-attribute 

model_class: type[Deduper] | type[Linker]

model_settings instance-attribute 

model_settings: dict[str, Any]

model_resources instance-attribute 

model_resources: dict[str, Resource]

model_instance instance-attribute 

model_instance: Deduper | Linker

parents property 

parents: tuple[RecordStep, ...]

The record steps this model reads: one for a deduper, two for a linker.

model_type property 

model_type: ModelType

Whether this model dedupes or links.

Derived from model_class, so it cannot drift from it.

spec property 

spec: ModelSpec

The serialisable spec for this model.

is_collected property 

is_collected: bool

Whether this step has been materialised.

edges 

edges() -> DataFrame

Return this model's scored edges. Collects the plan first if needed.

resolve 

resolve(*other_models: Model, resolver_class: type[ResolverMethod] | str = 'Components', resolver_settings: dict[str, Any] | None = None, resolver_resources: dict[str, Any] | None = None) -> Resolver

Resolve this model (and any others) into clusters.

collect 

collect(store: Store | None = None, interactive: bool | None = None) -> Self

Materialise this step and everything it depends on.

Steps whose artifact is already stored are skipped without being run, so re-collecting after adding a downstream step only does the new work.

Reports as it goes: the plan, a record per step, and a closing summary of what ran, what was cached, how long it took and what the store now holds. No logging setup is needed for any of that — a collection lends the matchlab logger a console handler where the application hasn't configured one, and leaves an application that has entirely alone. See matchlab.core.logging.audible.

Parameters:

  • store
    (Store | None, default: None ) –

    Where to read and write artifacts. Defaults to the module-level store (a DuckDB store in the user cache directory).

  • interactive
    (bool | None, default: None ) –

    Whether someone is watching. None, the default, takes a terminal or a notebook as a yes. When they are, the plan is drawn as a live tree redrawn in place, and not logged. The tree on screen is the key those [step N] records need, and it stays there. When they are not, the plan is logged instead. See matchlab.progress.

Returns:

  • Self –

    This step, now collected.

lineage 

lineage() -> list[Step]

Return this step and all its inputs, upstream-first.

draw 

draw() -> str

Render this step's sub-plan as a tree.

fingerprints 

fingerprints() -> set[Fingerprint]

Address every artifact this plan is made of, its own and its inputs'.

Which artifacts belong to a plan is the plan's own business, so this is where a store gets told: store.prune(keep=plan.fingerprints()) hands storage a set of addresses it already understands, rather than a graph it would have to learn to walk.

Returns:

  • set[Fingerprint] –

    One fingerprint per step in lineage(). A set, because two steps in one

  • set[Fingerprint] –

    plan can address the same artifact. Identical specs over identical

  • set[Fingerprint] –

    inputs is the same bytes, and it is stored once.

Raises:

  • RuntimeError –

    If any step has not been collected. An uncollected plan names no artifacts at all, so answering with a smaller set would quietly tell a caller that less is worth keeping than they think.

add_model_class 

add_model_class(model_class: type[Linker] | type[Deduper]) -> None

Register a custom deduper or linker so it can be named in a plan.

normalise_model_scores 

normalise_model_scores(scores: DataFrame) -> DataFrame

Validate a methodology's raw output and cast it to SCHEMA_MODEL_EDGES.

Raises ValueError if scores isn't a DataFrame with exactly left_id, right_id, and a numeric score in [0.0, 1.0]. An unordered pair can appear twice, once as (a, b) and once as (b, a). When that happens, this keeps only the highest-scoring row and logs a warning, since a resolver expects one edge per pair.

matchlab.models.comparison

Functions to compare fields in different sources.

Functions:

  • comparison –

    Validate a SQL WHERE-clause condition and recompile it for dialect.

comparison 

comparison(sql_condition: SQLCondition, dialect: str = 'duckdb') -> SQLCondition

Validate a SQL WHERE-clause condition and recompile it for dialect.

Every column must be qualified with the alias l or r, the two sides a Model compares. Raises ParseError if a column isn't qualified this way, if the condition isn't a valid boolean expression, or if it doesn't reference both sides.

dialect only changes how the validated condition renders back to SQL, not how it's parsed, so sql_condition must already be syntax sqlglot can parse without help.

Logs a warning for OR conditions, since DuckDB can't use an index to prune them.