Skip to content

Dedupers

Model methodologies that deduplicate a RecordStep.

matchlab.models.dedupers.base

Base class for deduplication methodologies.

Classes:

  • Deduper –

    A methodology that finds candidate duplicate pairs within one record step.

Deduper

Bases: BaseModel, ABC


              flowchart TD
              matchlab.models.dedupers.base.Deduper[Deduper]

              

              click matchlab.models.dedupers.base.Deduper href "" "matchlab.models.dedupers.base.Deduper"

A methodology that finds candidate duplicate pairs within one record step.

A Model step calls prepare() once, then dedupe(), each time it collects. Put one-off setup in prepare() instead, for example fitting a model over the whole dataset, so it doesn't repeat on every call to dedupe(). dedupe() must return a table with left_id, right_id, and score columns. normalise_model_scores casts that table to SCHEMA_MODEL_EDGES.

Every field is a setting unless marked matchlab.resources.FromResources. A fingerprint ignores a resource, so a marked field must not change what this scores.

Methods:

  • prepare –

    Run once before dedupe(), for setup that shouldn't repeat per call.

  • dedupe –

    Score candidate duplicate pairs within data.

Attributes:

model_config class-attribute instance-attribute 

model_config = ConfigDict(extra='forbid', frozen=True)

version class-attribute 

version: int | None = None

id class-attribute instance-attribute 

id: Literal['id'] = Field(default='id', description='The unique ID field in the data to dedupe')

prepare abstractmethod 

prepare(data: DataFrame) -> None

Run once before dedupe(), for setup that shouldn't repeat per call.

dedupe abstractmethod 

dedupe(data: DataFrame) -> DataFrame

Score candidate duplicate pairs within data.