Metrics

Compute precision and recall for each model's clusters against judgements.

The function turns both model clusters and judgements into pairs of leaves, then compares them:

• Convert each model's clusters and the judgements into pairwise connections between leaves. For judgements, this includes pairs the user was shown but rejected, not just the ones they endorsed. Each pair's net score is the number of times it was endorsed, minus the number of times it was rejected.
• Keep only pairs whose leaves appear in every model and in the judgements, so each model is compared over the same data.
• Drop a pair from both model and validation pairs if it was rejected as often as it was endorsed. If it was rejected more often than endorsed, drop it from validation pairs only, and keep it in model pairs.
• Compute precision and recall for each model against the remaining validation pairs.

This ignores user IDs, so judgements are not yet weighted by reviewer.

Parameters:

• models_root_leaf ¶

  (list[DataFrame]) –

  One (root, leaf) table per model. Each must be merge-forward, including every leaf reachable from the model's inputs, even ones no model touched, which appear as their own root.

• judgements ¶

  (DataFrame) –

  Table following SCHEMA_JUDGEMENTS.

• expansion ¶

  (DataFrame) –

  Table following SCHEMA_CLUSTER_EXPANSION.

Returns:

• list[PrecisionRecall] –

  One (precision, recall) pair per model, in the order given.

Turn judgements into leaf pairs, a net score per pair, and the leaves shown.

Expanding a cluster's leaves into every sorted pair is not quite enough on its own. When a user is shown (123) and endorses (12), they also reject pairs (1,3) and (2,3), not just endorse (1,2). Each pair's net score sums +1 for every endorsement and -1 for every rejection implied this way.

This function assumes well-formed input:

• Every shown cluster ID has a matching row in expansion.
• Every endorsed cluster ID has a matching row in expansion, unless it is a single leaf.
• No cluster is split across judgement rows unless every other part is expanded too. For example, if (123)->(12) appears, (123)->(3) must appear too.

Parameters:

• judgements ¶

  (DataFrame) –

  Table following SCHEMA_JUDGEMENTS.

• expansion ¶

  (DataFrame) –

  Table following SCHEMA_CLUSTER_EXPANSION.

Returns:

• Pairs –

  A tuple of:

• dict[Pair, float] –
  • Every pair implied by an endorsement or a rejection.
• set[int] –
  • Each pair's net score, where positive means endorsed more than rejected.
• tuple[Pairs, dict[Pair, float], set[int]] –
  • Every leaf ID shown to a user.