Skip to content

Duckdb

matchlab.stores.duckdb

The DuckDB store for matchlab.

A single DuckDB database (a file, or :memory:) holds every collected artifact, keyed by step fingerprint. There is no engine here that resolves on demand. Resolvers arrive already materialised (merge-forward), so reads are plain table scans. Analysts can point their own SQL at the resolver_output table. That is the whole point.

Classes:

  • DuckDBStoreStats –

    What a DuckDB store holds, plus what only a file of blocks can report.

  • DuckDBStore –

    Store collected artifacts in a DuckDB database, keyed by fingerprint.

DuckDBStoreStats

Bases: StoreStats


              flowchart TD
              matchlab.stores.duckdb.DuckDBStoreStats[DuckDBStoreStats]
              matchlab.stores.base.StoreStats[StoreStats]

                              matchlab.stores.base.StoreStats --> matchlab.stores.duckdb.DuckDBStoreStats
                


              click matchlab.stores.duckdb.DuckDBStoreStats href "" "matchlab.stores.duckdb.DuckDBStoreStats"
              click matchlab.stores.base.StoreStats href "" "matchlab.stores.base.StoreStats"

What a DuckDB store holds, plus what only a file of blocks can report.

Attributes:

  • path (Path | None) –

    The database file, or None for :memory:. location already names the store. This is the file itself, for code that wants to stat or delete it.

  • free_bytes (int) –

    Space already freed inside the file. DuckDB reuses those blocks for later writes, but never returns them to the OS. This is the gap between what the store weighs and what it holds. It is the only figure that says what a reclaim could recover, without first deciding what to delete. It has no meaning for a backend that is not a file of reusable blocks. That is why it lives here rather than on StoreStats.

Methods:

  • describe –

    One clause saying what the store costs, for a collect to print.

path class-attribute instance-attribute 

path: Path | None = None

free_bytes class-attribute instance-attribute 

free_bytes: int = 0

size property 

size: str

Say when the bytes are resident rather than written.

4.6 MB reads as a disk size, but for :memory: it isn't one. The store vanishes with the process. The distinction matters most exactly when a user is least likely to be thinking about it.

model_config class-attribute instance-attribute 

model_config = ConfigDict(frozen=True)

location instance-attribute 

location: str

bytes instance-attribute 

bytes: int

artifacts class-attribute instance-attribute 

artifacts: dict[StepKind, int] = Field(default_factory=dict)

labels class-attribute instance-attribute 

labels: int = 0

describe 

describe(since: StoreStats | None = None) -> str

One clause saying what the store costs, for a collect to print.

Parameters:

  • since
    (StoreStats | None, default: None ) –

    An earlier reading, so the growth between them can be attributed to whatever happened in between. Without it the figure is the store's size and nothing more, which names no cause.

DuckDBStore 

DuckDBStore(path: str | Path = ':memory:')

Bases: Store


              flowchart TD
              matchlab.stores.duckdb.DuckDBStore[DuckDBStore]
              matchlab.stores.base.Store[Store]

                              matchlab.stores.base.Store --> matchlab.stores.duckdb.DuckDBStore
                


              click matchlab.stores.duckdb.DuckDBStore href "" "matchlab.stores.duckdb.DuckDBStore"
              click matchlab.stores.base.Store href "" "matchlab.stores.base.Store"

Store collected artifacts in a DuckDB database, keyed by fingerprint.

Open (or create) the store at path. Use :memory: for ephemeral stores.

Methods:

  • has –

    Return whether an artifact for this fingerprint is already stored.

  • stats –

    Report the store's size and contents.

  • store_source –

    Store a collected source.

  • read_source_extract –

    Return the cached warehouse extract for a stored source.

  • read_source_leaves –

    Return the (key, leaf) assignment for a stored source.

  • read_identifiers –

    Return (id, source, key, leaf) for one source's records.

  • prune –

    Delete every artifact except the ones named, and reclaim what that frees.

  • store_model –

    Store a model's edge list (SCHEMA_MODEL_EDGES).

  • read_model –

    Return a stored model's edge list.

  • store_transform –

    Store a transform's materialised record step (arbitrary schema).

  • read_transform –

    Return a stored transform's record step.

  • store_resolver –

    Store a resolver's complete flat output.

  • read_resolver –

    Return a resolver's stored output (root, leaf, key, source).

  • store_judgement –

    Persist a user judgement, expanding its clusters to leaves for scoring.

  • read_eval_data –

    Return (judgements, expansion) tables for matchlab.core.eval.

  • sample –

    Sample up to n clusters from a stored resolver output for evaluation.

  • publish –

    Point label at fp, replacing whatever it pointed at before.

  • find –

    Return the fingerprint a label points at, if any.

  • labels –

    Return every label in this store, sorted.

  • source_key_field –

    Return which column of a stored source's extract holds the key.

  • resolver_output_sources –

    Return source name to fingerprint for a stored resolver output.

  • close –

    Release any underlying resources. Override if needed.

  • read_source_records –

    Return a source's stored rows for keys, every column qualified by name.

Attributes:

path instance-attribute 

path = str(path)

conn instance-attribute 

conn = connect(path)

has 

has(fp: Fingerprint) -> bool

Return whether an artifact for this fingerprint is already stored.

The client uses this to skip re-running a step whose plan is unchanged.

stats 

stats() -> DuckDBStoreStats

Report the store's size and contents.

Checkpoints a file store before measuring it. That makes this the one method here that writes without being asked to. Without it, the figure is not just imprecise. It is the wrong order of magnitude. Recent writes sit in the write-ahead log as a compact journal, and settling them into 256 KB blocks can turn 21 KB of log into 5.5 MB of file. Measured on examples/companies: 33 KB reported against a store that became 5.5 MB the moment it was closed. Reporting a size that a user's next du contradicts by 170x is worse than reporting none.

DuckDB's own block count is no help before that point. It reads zero until a checkpoint has happened. Afterwards the two agree to within the file header, so stat is used instead. It counts the .wal sibling too, and it is the number a user can actually check.

The checkpoint is cheap, because DuckDB has usually already done most of it: 1.4 ms after writing 10M rows, 0.08 ms when there is nothing pending.

free_blocks still comes from DuckDB. Nothing outside the file can see how much of it is reusable, and an in-memory store's size is no different: it allocates no blocks and has no file to measure.

store_source 

store_source(fp: Fingerprint, key_field: str, extract: DataFrame, leaves: DataFrame) -> None

Store a collected source.

Parameters:

  • fp
    (Fingerprint) –

    The source step's fingerprint.

  • key_field
    (str) –

    Which column of extract holds the key. Stored so the extract can be read back and joined to a resolver's output without the plan.

  • extract
    (DataFrame) –

    The warehouse extract (arbitrary schema) to cache.

  • leaves
    (DataFrame) –

    The leaf assignment, columns (key: str, leaf: uint64).

read_source_extract 

read_source_extract(fp: Fingerprint) -> DataFrame

Return the cached warehouse extract for a stored source.

read_source_leaves 

read_source_leaves(fp: Fingerprint) -> DataFrame

Return the (key, leaf) assignment for a stored source.

read_identifiers 

read_identifiers(source_fp: Fingerprint, source_name: str, resolver_fp: Fingerprint | None = None) -> DataFrame

Return (id, source, key, leaf) for one source's records.

id is resolver_fp's root cluster when reading through a resolver, otherwise the source's own leaf. This is the upstream output a downstream resolver needs in order to carry every reachable leaf forward, including records no model matched.

A query rather than an artifact. Nothing here is computed. Both readings are projections of tables this store already holds, source_leaves and resolver_output. There is nothing to cache, and caching it under a record step's fingerprint would be wrong anyway. The result depends only on the source and resolver read, not on how a record step reshapes the data.

Parameters:

  • source_fp
    (Fingerprint) –

    Fingerprint of the stored source whose records are wanted.

  • source_name
    (str) –

    The source's name, which is returned in the source column and tags each row for the resolver output below.

  • resolver_fp
    (Fingerprint | None, default: None ) –

    Fingerprint of the resolver to read through, or None to read the source's own leaves.

prune 

prune(keep: Iterable[Fingerprint] = ()) -> PruneResult

Delete every artifact except the ones named, and reclaim what that frees.

Deleting is only half of it. DuckDB marks freed blocks for reuse but never hands them back to the OS, so purging alone does not shrink the file at all. Measured on a real 575 MB store, deleting 77% of its artifacts freed 0 bytes. The space comes back only by rewriting the database, copying what is left into a fresh file and swapping it in. Purge and rewrite together recovered 437 MB of that store, in half a second.

The swap is a rename over the original. A failure anywhere leaves the store exactly as it was, with the half-written copy orphaned beside it.

This reopens the connection. That is the one internal detail anything outside this class needs to know about. Session settings applied through store.conn (memory_limit and temp_directory, as the guide suggests) do not survive. The store itself stays valid, so anything holding it, rather than its connection, is unaffected.

An in-memory store is purged but not rewritten. It has no file, and reopening one would hand back an empty database rather than a smaller one. Its freed blocks return to the allocator anyway, so the reclaim is real regardless.

store_model 

store_model(fp: Fingerprint, edges: DataFrame) -> None

Store a model's edge list (SCHEMA_MODEL_EDGES).

read_model 

read_model(fp: Fingerprint) -> DataFrame

Return a stored model's edge list.

store_transform 

store_transform(fp: Fingerprint, table: DataFrame) -> None

Store a transform's materialised record step (arbitrary schema).

Called on every collect, so a transform feeding several models is computed once and read back by each.

read_transform 

read_transform(fp: Fingerprint) -> DataFrame

Return a stored transform's record step.

store_resolver 

store_resolver(fp: Fingerprint, resolver_output: DataFrame, sources: Mapping[str, Fingerprint] | None = None) -> None

Store a resolver's complete flat output.

Parameters:

  • fp
    (Fingerprint) –

    The resolver step's fingerprint.

  • resolver_output
    (DataFrame) –

    SCHEMA_RESOLVER_OUTPUT columns (root, leaf, key, source), already merged forward over all upstream leaves. The store does not verify the merge. That is the client's contract. The store does validate the schema.

  • sources
    (Mapping[str, Fingerprint] | None, default: None ) –

    Source name to fingerprint, for every source this output covers. A resolver's output names its sources but one store can hold several generations of a name, so this records which were actually used.

read_resolver 

read_resolver(fp: Fingerprint) -> DataFrame

Return a resolver's stored output (root, leaf, key, source).

store_judgement 

store_judgement(judgement: Judgement, user_name: str = 'local') -> None

Persist a user judgement, expanding its clusters to leaves for scoring.

read_eval_data 

read_eval_data(tag: str | None = None) -> tuple[DataFrame, DataFrame]

Return (judgements, expansion) tables for matchlab.core.eval.

judgements follows SCHEMA_JUDGEMENTS. expansion follows SCHEMA_CLUSTER_EXPANSION. Both filter to tag when it is given.

sample 

sample(resolver_fp: Fingerprint, n: int, seed: int | None = None) -> DataFrame

Sample up to n clusters from a stored resolver output for evaluation.

Returns SCHEMA_RESOLVER_OUTPUT rows (root, leaf, key, source) for the sampled roots.

publish 

publish(label: str, fp: Fingerprint) -> None

Point label at fp, replacing whatever it pointed at before.

The store moves the pointer without arguing. The caller decides whether overwriting is allowed, since it knows what the user asked for.

find 

find(label: str) -> Fingerprint | None

Return the fingerprint a label points at, if any.

labels 

labels() -> list[str]

Return every label in this store, sorted.

source_key_field 

source_key_field(fp: Fingerprint) -> str

Return which column of a stored source's extract holds the key.

resolver_output_sources 

resolver_output_sources(fp: Fingerprint) -> dict[str, Fingerprint]

Return source name to fingerprint for a stored resolver output.

close 

close() -> None

Release any underlying resources. Override if needed.

read_source_records 

read_source_records(source_fp: Fingerprint, source_name: str, keys: Series) -> tuple[DataFrame, str]

Return a source's stored rows for keys, every column qualified by name.

A query rather than an artifact, like read_identifiers, and concrete rather than abstract because it is nothing but source_key_field and read_source_extract composed.

The extract cached when the source was collected is the data the matching saw. That is what you want in front of you when judging a resolver's output, and it means both reading an entity and reviewing one work with no warehouse connection.

Parameters:

  • source_fp
    (Fingerprint) –

    Fingerprint of the stored source whose rows are wanted.

  • source_name
    (str) –

    The source's name, which every column is prefixed with.

  • keys
    (Series) –

    The keys to return rows for.

Returns:

  • tuple[DataFrame, str] –

    (rows, qualified key column).