Skip to content

Stores

Entity resolution is iterative, long-running, and can easily explode in compute and memory. matchlab uses caching for stability and sharing. Stores are how you configure it.

matchlab.stores.default

Functions and variables for the default matchlib store.

Functions:

  • default_store –

    Return the default store, creating a DuckDB store in the cache dir if unset.

  • set_default_store –

    Set the store used by collect() when none is passed. None resets it.

Attributes:

CACHE_DIR module-attribute 

CACHE_DIR = user_cache_path('matchlab')

default_store 

default_store() -> Store

Return the default store, creating a DuckDB store in the cache dir if unset.

set_default_store 

set_default_store(store: Store | None) -> None

Set the store used by collect() when none is passed. None resets it.

matchlab.stores.base

The storage contract for matchlab.

Storage persists the artifacts each collected DAG step produces, keyed by that step's content fingerprint, and reads them back. It does not resolve anything on demand. Resolvers materialise their complete, merge-forward output at collect time and hand the store a finished table.

Artifacts, by step kind (schemas in matchlab.core.schemas, which holds exactly the shapes that cross this boundary):

  • Source → warehouse extract (arbitrary schema) + leaf assignment (key, leaf).
  • Transform → the reshaped record step it materialises (arbitrary schema).
  • Model → edge list, SCHEMA_MODEL_EDGES (left_id, right_id, score).
  • Resolver → complete flat output, SCHEMA_RESOLVER_OUTPUT (root, leaf, key, src). This is the merge-forward guarantee. It computes merge(upstream complete output, own clusters), not just the resolver's own clusters.

A Source and a Resolver produce a record step too, but neither stores one. A source is fully materialised as its extract and derives its record step on read. A resolver read is a re-derivable join over the source's extract and the resolver's own output. Only a Transform stores its materialisation.

Plus evaluation storage (judgements + cluster expansion), publication (publish points a label at a resolver's output) and close.

Nothing here deletes an artifact on the store's own initiative. A store keeps what it is given until the owner disposes of it. prune is that disposal. It deletes only what the caller has said it may, and never a published resolver output. See the guide's "Reclaiming storage".

Classes:

  • StoreStats –

    What a store holds, and what it costs.

  • PruneResult –

    What a prune removed, and what that actually recovered.

  • Store –

    Fingerprint-keyed storage for collected DAG-step artifacts.

Functions:

  • format_bytes –

    Render a byte count the way someone reading a disk would.

Attributes:

Fingerprint module-attribute 

Fingerprint = bytes

StoreStats

Bases: BaseModel


              flowchart TD
              matchlab.stores.base.StoreStats[StoreStats]

              

              click matchlab.stores.base.StoreStats href "" "matchlab.stores.base.StoreStats"

What a store holds, and what it costs.

A store keeps everything collected into it, so it grows with every plan and every edit to one. Stores subclass this to report metrics that are relevant to them.

Attributes:

  • location (str) –

    Where the store is, as a reader would name it, such as a path, a URI, or ":memory:". Always present, because a store you cannot point at is one you cannot go and delete.

  • bytes (int) –

    The store's size. For anything file-backed, this is a high-water mark. A store that briefly held 4 GB reports 4 GB after the rows are gone, because deleting inside a database file does not return space to the OS. That is the right number for a disk filling up, and the wrong one for "how much data do I have".

  • artifacts (dict[StepKind, int]) –

    How many artifacts of each step kind are stored.

  • labels (int) –

    How many published labels point into the store.

Methods:

  • describe –

    One clause saying what the store costs, for a collect to print.

model_config class-attribute instance-attribute 

model_config = ConfigDict(frozen=True)

location instance-attribute 

location: str

bytes instance-attribute 

bytes: int

artifacts class-attribute instance-attribute 

artifacts: dict[StepKind, int] = Field(default_factory=dict)

labels class-attribute instance-attribute 

labels: int = 0

size property 

size: str

This store's size, as a phrase.

A hook, because a bare figure can mislead. Bytes held in memory and bytes written to disk are not the same claim, and only the backend knows which one it just reported.

describe 

describe(since: StoreStats | None = None) -> str

One clause saying what the store costs, for a collect to print.

Parameters:

  • since
    (StoreStats | None, default: None ) –

    An earlier reading, so the growth between them can be attributed to whatever happened in between. Without it the figure is the store's size and nothing more, which names no cause.

PruneResult

Bases: BaseModel


              flowchart TD
              matchlab.stores.base.PruneResult[PruneResult]

              

              click matchlab.stores.base.PruneResult href "" "matchlab.stores.base.PruneResult"

What a prune removed, and what that actually recovered.

Stores subclass this where they have more to say.

Attributes:

  • removed (int) –

    How many artifacts were deleted.

  • kept (int) –

    How many survived. This includes the ones named, the ones their lineage needed, and everything published.

  • reclaimed (int) –

    Bytes genuinely returned. Measured as the store's size before minus after, not totted up from what was deleted. The two are not the same number. Deleting inside a database file usually frees nothing at all until the file is rewritten, and a reclaim that reported the bytes it deleted would claim to have freed space while the disk sat unchanged.

Methods:

  • describe –

    One line saying what happened, for a caller to print.

model_config class-attribute instance-attribute 

model_config = ConfigDict(frozen=True)

removed instance-attribute 

removed: int

kept instance-attribute 

kept: int

reclaimed instance-attribute 

reclaimed: int

describe 

describe() -> str

One line saying what happened, for a caller to print.

Store

Bases: ABC


              flowchart TD
              matchlab.stores.base.Store[Store]

              

              click matchlab.stores.base.Store href "" "matchlab.stores.base.Store"

Fingerprint-keyed storage for collected DAG-step artifacts.

Implementations are single-user and local. DuckDBStore is the reference.

Methods:

  • has –

    Return whether an artifact for this fingerprint is already stored.

  • store_source –

    Store a collected source.

  • read_source_extract –

    Return the cached warehouse extract for a stored source.

  • read_source_leaves –

    Return the (key, leaf) assignment for a stored source.

  • read_identifiers –

    Return (id, source, key, leaf) for one source's records.

  • store_model –

    Store a model's edge list (SCHEMA_MODEL_EDGES).

  • read_model –

    Return a stored model's edge list.

  • store_transform –

    Store a transform's materialised record step (arbitrary schema).

  • read_transform –

    Return a stored transform's record step.

  • store_resolver –

    Store a resolver's complete flat output.

  • publish –

    Point label at fp, replacing whatever it pointed at before.

  • find –

    Return the fingerprint a label points at, if any.

  • labels –

    Return every label in this store, sorted.

  • source_key_field –

    Return which column of a stored source's extract holds the key.

  • read_source_records –

    Return a source's stored rows for keys, every column qualified by name.

  • resolver_output_sources –

    Return source name to fingerprint for a stored resolver output.

  • read_resolver –

    Return a resolver's stored output (root, leaf, key, source).

  • store_judgement –

    Persist a user judgement, expanding its clusters to leaves for scoring.

  • read_eval_data –

    Return (judgements, expansion) tables for matchlab.core.eval.

  • sample –

    Sample up to n clusters from a stored resolver output for evaluation.

  • stats –

    Report the store's size and contents.

  • prune –

    Delete every artifact except the ones named, and reclaim what that frees.

  • close –

    Release any underlying resources. Override if needed.

has abstractmethod 

has(fp: Fingerprint) -> bool

Return whether an artifact for this fingerprint is already stored.

The client uses this to skip re-running a step whose plan is unchanged.

store_source abstractmethod 

store_source(fp: Fingerprint, key_field: str, extract: DataFrame, leaves: DataFrame) -> None

Store a collected source.

Parameters:

  • fp
    (Fingerprint) –

    The source step's fingerprint.

  • key_field
    (str) –

    Which column of extract holds the key. Stored so the extract can be read back and joined to a resolver's output without the plan.

  • extract
    (DataFrame) –

    The warehouse extract (arbitrary schema) to cache.

  • leaves
    (DataFrame) –

    The leaf assignment, columns (key: str, leaf: uint64).

read_source_extract abstractmethod 

read_source_extract(fp: Fingerprint) -> DataFrame

Return the cached warehouse extract for a stored source.

read_source_leaves abstractmethod 

read_source_leaves(fp: Fingerprint) -> DataFrame

Return the (key, leaf) assignment for a stored source.

read_identifiers abstractmethod 

read_identifiers(source_fp: Fingerprint, source_name: str, resolver_fp: Fingerprint | None = None) -> DataFrame

Return (id, source, key, leaf) for one source's records.

id is resolver_fp's root cluster when reading through a resolver, otherwise the source's own leaf. This is the upstream output a downstream resolver needs in order to carry every reachable leaf forward, including records no model matched.

A query rather than an artifact. Nothing here is computed. Both readings are projections of tables this store already holds, source_leaves and resolver_output. There is nothing to cache, and caching it under a record step's fingerprint would be wrong anyway. The result depends only on the source and resolver read, not on how a record step reshapes the data.

Parameters:

  • source_fp
    (Fingerprint) –

    Fingerprint of the stored source whose records are wanted.

  • source_name
    (str) –

    The source's name, which is returned in the source column and tags each row for the resolver output below.

  • resolver_fp
    (Fingerprint | None, default: None ) –

    Fingerprint of the resolver to read through, or None to read the source's own leaves.

store_model abstractmethod 

store_model(fp: Fingerprint, edges: DataFrame) -> None

Store a model's edge list (SCHEMA_MODEL_EDGES).

read_model abstractmethod 

read_model(fp: Fingerprint) -> DataFrame

Return a stored model's edge list.

store_transform abstractmethod 

store_transform(fp: Fingerprint, table: DataFrame) -> None

Store a transform's materialised record step (arbitrary schema).

Called on every collect, so a transform feeding several models is computed once and read back by each.

read_transform abstractmethod 

read_transform(fp: Fingerprint) -> DataFrame

Return a stored transform's record step.

store_resolver abstractmethod 

store_resolver(fp: Fingerprint, resolver_output: DataFrame, sources: Mapping[str, Fingerprint] | None = None) -> None

Store a resolver's complete flat output.

Parameters:

  • fp
    (Fingerprint) –

    The resolver step's fingerprint.

  • resolver_output
    (DataFrame) –

    SCHEMA_RESOLVER_OUTPUT columns (root, leaf, key, source), already merged forward over all upstream leaves. The store does not verify the merge. That is the client's contract. The store does validate the schema.

  • sources
    (Mapping[str, Fingerprint] | None, default: None ) –

    Source name to fingerprint, for every source this output covers. A resolver's output names its sources but one store can hold several generations of a name, so this records which were actually used.

publish abstractmethod 

publish(label: str, fp: Fingerprint) -> None

Point label at fp, replacing whatever it pointed at before.

The store moves the pointer without arguing. The caller decides whether overwriting is allowed, since it knows what the user asked for.

find abstractmethod 

find(label: str) -> Fingerprint | None

Return the fingerprint a label points at, if any.

labels abstractmethod 

labels() -> list[str]

Return every label in this store, sorted.

source_key_field abstractmethod 

source_key_field(fp: Fingerprint) -> str

Return which column of a stored source's extract holds the key.

read_source_records 

read_source_records(source_fp: Fingerprint, source_name: str, keys: Series) -> tuple[DataFrame, str]

Return a source's stored rows for keys, every column qualified by name.

A query rather than an artifact, like read_identifiers, and concrete rather than abstract because it is nothing but source_key_field and read_source_extract composed.

The extract cached when the source was collected is the data the matching saw. That is what you want in front of you when judging a resolver's output, and it means both reading an entity and reviewing one work with no warehouse connection.

Parameters:

  • source_fp
    (Fingerprint) –

    Fingerprint of the stored source whose rows are wanted.

  • source_name
    (str) –

    The source's name, which every column is prefixed with.

  • keys
    (Series) –

    The keys to return rows for.

Returns:

  • tuple[DataFrame, str] –

    (rows, qualified key column).

resolver_output_sources abstractmethod 

resolver_output_sources(fp: Fingerprint) -> dict[str, Fingerprint]

Return source name to fingerprint for a stored resolver output.

read_resolver abstractmethod 

read_resolver(fp: Fingerprint) -> DataFrame

Return a resolver's stored output (root, leaf, key, source).

store_judgement abstractmethod 

store_judgement(judgement: Judgement, user_name: str = 'local') -> None

Persist a user judgement, expanding its clusters to leaves for scoring.

read_eval_data abstractmethod 

read_eval_data(tag: str | None = None) -> tuple[DataFrame, DataFrame]

Return (judgements, expansion) tables for matchlab.core.eval.

judgements follows SCHEMA_JUDGEMENTS. expansion follows SCHEMA_CLUSTER_EXPANSION. Both filter to tag when it is given.

sample abstractmethod 

sample(resolver_fp: Fingerprint, n: int, seed: int | None = None) -> DataFrame

Sample up to n clusters from a stored resolver output for evaluation.

Returns SCHEMA_RESOLVER_OUTPUT rows (root, leaf, key, source) for the sampled roots.

stats abstractmethod 

stats() -> StoreStats

Report the store's size and contents.

Abstract rather than an optional hook like close, because there is no honest default. A store that reported zero would be wrong in exactly the place a user is being asked to trust a number, and a store that cannot say how much room it is taking is not one to hand a growing cache to.

Called on every collect, so it must be cheap. It may also settle pending writes to report a size that has stopped moving, since DuckDBStore checkpoints. This is not guaranteed to be a pure read.

prune abstractmethod 

prune(keep: Iterable[Fingerprint] = ()) -> PruneResult

Delete every artifact except the ones named, and reclaim what that frees.

You say what to keep. Nothing is inferred. This is deliberately not the gc() that used to live here. That one worked out what was still alive from which Python objects happened to be reachable. A fresh interpreter has nothing reachable, so it considered the whole store garbage and duly emptied one that had a published resolver output in it. An artifact's worth has nothing to do with whether some process is holding the variable that produced it. The root set therefore arrives as an argument, from a caller who has just named it.

The caller names those roots as fingerprints. Step.fingerprints() gives the fingerprints relevant to a plan, so storage does not have to learn to walk a graph in order to know what it may delete.

Published labels are kept whether or not they are named, along with the sources their resolver output needs in order to stay readable. Publishing is the strongest "keep this" the system has, and losing one because a caller forgot to list it would be the old mistake in new clothes.

Implementations must:

  • never touch stored judgements, which are human work and cannot be recomputed from anything
  • report reclaimed as space genuinely returned, measured rather than assumed. Deleting is not always the same as reclaiming, and a store that says it freed something it did not is worse than one that frees nothing.

Parameters:

  • keep
    (Iterable[Fingerprint], default: () ) –

    Fingerprints to preserve. Published labels are kept whether or not their fingerprints are listed.

Returns:

  • PruneResult –

    What was removed and what that recovered.

Raises:

  • ValueError –

    If nothing would survive the prune, because keep was empty and nothing is published. Deleting everything is the same as deleting the file, and should not be what an accidentally-empty list does.

close 

close() -> None

Release any underlying resources. Override if needed.

format_bytes 

format_bytes(count: int, *, signed: bool = False) -> str

Render a byte count the way someone reading a disk would.

Binary units, because that is what a file browser and du -h report, and a size that disagrees with the one the user can check is worse than no size at all.