Skip to content

Resolver output

matchlab.core.resolver_output

Each resolver materialises its complete, merge-forward output once.

Every resolver, when it runs, computes that output and stores it. Everything downstream (entities, lookup_key, models querying through it) just reads the stored table.

This module holds the pure functions that produce that output:

  • leaf_id / root_id — content-addressed cluster IDs. A leaf is a hash of a record's content. That is the correct anchor for evaluation, not merely a reproducibility trick — see leaf_id for the full reasoning.
  • materialise_resolver_output — turns a resolver's clusters and upstream output into the complete result table. See its own docstring for the shape of each.

Merge-forward means a leaf grouped upstream but untouched by this resolver inherits its upstream cluster rather than collapsing to a singleton. materialise_resolver_output implements this by giving every untouched reachable ID its own component ("fall-through"), so the upstream grouping survives.

This module assumes a resolver's inputs share a consistent lineage, so each reachable leaf maps to exactly one upstream ID. DAG construction enforces that, so nothing here needs to reconcile a leaf that maps to more than one.

Functions:

  • leaf_id –

    Map a column of row content hashes to stable 64-bit leaf cluster IDs.

  • root_id –

    Deterministic 64-bit root cluster ID for a column of leaf lists.

  • root_id_of –

    root_id for a single leaf set. For tests and one-off checks, not for loops.

  • materialise_resolver_output –

    Build a resolver's complete, merge-forward output.

leaf_id 

leaf_id(row_hash: Expr) -> Expr

Map a column of row content hashes to stable 64-bit leaf cluster IDs.

A record's identity is a hash of its content, not of its key. This is the load-bearing decision, and it is deliberate, not an accident of implementation. The reason is evaluation.

A judgement is a person or a model saying "looking at this, these records are the same entity." The only input to that decision is the content shown — a name, a postcode. The key plays no part. The judge never sees it. A judgement must therefore be anchored to the content it was actually made against:

  • If you anchor to the key, a judgement outlives a content change. Say a judgement was made when the evidence read "acme / london = acme / london". If that evidence later becomes "acme / manchester", the judgement still stands — attributed to evidence the judge never saw. It looks inexplicable later, or quietly validates a match nobody made.
  • If you anchor to the content instead (this module's choice), the leaf ID changes whenever the content does, so the old judgement stops applying. That is correct. Nobody has judged the new content, so there should be no judgement about it. Judgements decay exactly when their evidence does.

The same logic explains why identical rows share a leaf. To any judge they are indistinguishable, so there is no decision to make. Differing keys don't separate them, because a key is not evidence.

What gets hashed must match what gets shown. If a column feeds the hash but the sampler doesn't display it, a judgement decays for a reason the judge never saw. If a column appears in the sampler but doesn't feed the hash, a judgement survives a change the judge did see, which is wrong.

The two stay in sync today because both read the same definition. leaf_id hashes every non-key column (see Source._read_warehouse), and get_samples displays every non-key column. Selecting a column in the extract is what makes it both hashed and shown — there is no second list to keep in step by hand (see Source's "the extract is the whole declaration").

Content-addressing also makes runs reproducible and IDs stable across re-collect, but that is a consequence, not the reason.

This is vectorised deliberately. Calling a Python function once per row, as map_elements would, dominates collection time.

root_id 

root_id(leaves: Expr) -> Expr

Deterministic 64-bit root cluster ID for a column of leaf lists.

This is invariant to leaf order (callers sort) and to the arbitrary component label, so two runs that produce the same clustering produce the same root IDs.

This is vectorised for the same reason as leaf_id. Calling this once per cluster, in Python, would mean as many calls as there are entities.

root_id_of 

root_id_of(leaves: Iterable[int]) -> int

root_id for a single leaf set. For tests and one-off checks, not for loops.

materialise_resolver_output 

materialise_resolver_output(clusters: DataFrame, upstream: DataFrame) -> DataFrame

Build a resolver's complete, merge-forward output.

Parameters:

  • clusters

    (DataFrame) –

    SCHEMA_CLUSTERS (parent_id, child_id) from the resolver's methodology. child_ids are query-space IDs the models formed edges over. IDs not present here are untouched by this resolver.

  • upstream

    (DataFrame) –

    The union of the resolver's input queries as (id, source, key, leaf) rows, covering every leaf reachable by the resolver, whether or not an edge formed over it. id is the query-space ID the model referenced.

Returns:

  • DataFrame –

    One row per reachable source record, shaped as SCHEMA_RESOLVER_OUTPUT

  • DataFrame –

    (root, leaf, key, source), with root the cluster it resolves to. Complete

  • DataFrame –

    and merge-forward.