Skip to content

Sampling

matchlab.eval.samples

Client-side helpers for retrieving and preparing evaluation samples.

Everything here takes a Resolver: one you are holding, the label one was published under, or a sequence of either. The sequence form is what makes two methodologies comparable. Sampling across several resolvers unions their components, so one round of judging covers all of them and the scores answer the same question.

Classes:

  • EvaluationFieldMetadata –

    Metadata describing one field shown to a reviewer during evaluation.

  • EvaluationItem –

    One cluster shown to a reviewer, with its leaves' source data alongside it.

  • EvalData –

    Caches a store's judgements, and scores resolvers against them.

Functions:

  • create_judgement –

    Build the Judgement for one reviewed item from its column assignments.

  • create_evaluation_item –

    Build an EvaluationItem, grouping each cluster's columns by field across sources.

  • get_samples –

    Retrieve sampled clusters enriched with source data, as EvaluationItems.

Attributes:

  • ResolverRef (TypeAlias) –

    One resolver to read: a live Resolver, or the label one was published under.

  • Reading (TypeAlias) –

    A located resolver: the store holding it, and its fingerprint.

ResolverRef module-attribute 

ResolverRef: TypeAlias = 'Resolver | str'

One resolver to read: a live Resolver, or the label one was published under.

Reading module-attribute 

Reading: TypeAlias = 'tuple[Store, Fingerprint]'

A located resolver: the store holding it, and its fingerprint.

EvaluationFieldMetadata

Bases: BaseModel


              flowchart TD
              matchlab.eval.samples.EvaluationFieldMetadata[EvaluationFieldMetadata]

              

              click matchlab.eval.samples.EvaluationFieldMetadata href "" "matchlab.eval.samples.EvaluationFieldMetadata"

Metadata describing one field shown to a reviewer during evaluation.

Attributes:

display_name instance-attribute 

display_name: str

source_columns instance-attribute 

source_columns: list[str]

EvaluationItem

Bases: BaseModel


              flowchart TD
              matchlab.eval.samples.EvaluationItem[EvaluationItem]

              

              click matchlab.eval.samples.EvaluationItem href "" "matchlab.eval.samples.EvaluationItem"

One cluster shown to a reviewer, with its leaves' source data alongside it.

records holds the leaf IDs and their source-qualified data columns. For example:

leaf src_a_first src_a_last src_b_first src_b_last
1 Thomas Bayes
2 Tommy B
12 Tom Bayes

fields maps each display name to the source-qualified columns holding it, so a reviewer sees one column per field rather than one per source. For example:

EvaluationFieldMetadata(
    display_name="first", source_columns=["src_a_first", "src_b_first"]
)

Methods:

Attributes:

model_config class-attribute instance-attribute 

model_config = {'arbitrary_types_allowed': True}

leaves instance-attribute 

leaves: list[int]

records instance-attribute 

records: DataFrame

fields instance-attribute 

fields: list[EvaluationFieldMetadata]

get_unique_record_groups 

get_unique_record_groups() -> list[list[int]]

Group identical records by leaf ID.

Returns:

  • list[list[int]] –

    List of groups, where each group is a list of leaf IDs

  • list[list[int]] –

    that have identical values across all data fields.

  • Example ( list[list[int]] ) –

    [[1, 3], [2], [4, 5, 6]] means records 1 & 3 are identical.

EvalData 

EvalData(store: Store, tag: str | None = None)

Caches a store's judgements, and scores resolvers against them.

Load judgement and expansion data used to compute evaluation metrics.

Parameters:

  • store

    (Store) –

    The store holding judgements, for example the one a resolver was collected into.

  • tag

    (str | None, default: None ) –

    Optional tag to filter judgements by.

Methods:

Attributes:

store instance-attribute 

store = store

tag instance-attribute 

tag = tag

precision_recall 

precision_recall(resolver: ResolverRef | Sequence[ResolverRef]) -> PrecisionRecall | list[PrecisionRecall]

Score one or more resolvers against these judgements.

Only pairs present in every resolver's output and in the judgements are compared, so scoring several at once is the fair way to rank them. Each is measured over the same records, and none is flattered by clusters the others never saw. Scoring them one at a time gives each its own comparison set, and those numbers do not line up.

Parameters:

Returns:

create_judgement 

create_judgement(item: EvaluationItem, assignments: dict[int, str], tag: str | None = None) -> Judgement

Build the Judgement for one reviewed item from its column assignments.

Parameters:

  • item

    (EvaluationItem) –

    The reviewed cluster.

  • assignments

    (dict[int, str]) –

    Group letter chosen for each unique record group, keyed by that group's index in item.get_unique_record_groups().

  • tag

    (str | None, default: None ) –

    Tag to record on the judgement.

Returns:

  • Judgement –

    A Judgement with one endorsed group per distinct letter used in

  • Judgement –

    assignments. A leaf whose group was never assigned is missing from

  • Judgement –

    endorsed, which Judgement's own validation rejects, so an incomplete

  • Judgement –

    assignments raises rather than producing a partial judgement.

create_evaluation_item 

create_evaluation_item(df: DataFrame, source_fields: list[tuple[str, list[str]]], leaves: list[int]) -> EvaluationItem

Build an EvaluationItem, grouping each cluster's columns by field across sources.

Parameters:

  • df

    (DataFrame) –

    The cluster's rows, with source-qualified data columns.

  • source_fields

    (list[tuple[str, list[str]]]) –

    (prefix, qualified columns) per source. The columns come from the fetched data rather than from the sources, which would have to re-read the warehouse just to list their names.

  • leaves

    (list[int]) –

    The leaf IDs in this cluster.

get_samples 

get_samples(n: int, resolver: ResolverRef | Sequence[ResolverRef], store: Store | None = None, seed: int | None = None) -> dict[int, EvaluationItem]

Retrieve sampled clusters enriched with source data, as EvaluationItems.

Record values come from the extract stored when each source was collected, not from a fresh warehouse read. This works offline, and shows the data the matching actually saw.

Parameters:

  • n

    (int) –

    Number of clusters to sample.

  • resolver

    (ResolverRef | Sequence[ResolverRef]) –

    The resolver to sample from, collected first if it isn't already, or the label one was published under. The label form needs no plan, because a stored resolver's output records which source artifacts it covers. Pass several and the sample is drawn from their merged components, so one round of judging scores all of them against the same clusters.

  • store

    (Store | None, default: None ) –

    Where to read from. Defaults to the resolver's, else the module default.

  • seed

    (int | None, default: None ) –

    Fixes which clusters come back. The same store, n and seed give the same sample, which is how two people review the same clusters.

Returns:

Raises:

  • SourceTableError –

    If nothing is published under resolver, if a source the resolver's output covers isn't in the store, or if several resolvers disagree about a source.

  • ValueError –

    If resolver is an empty sequence.