Schemas

Arrow schemas for the artifacts a store persists.

These are the shapes that cross the boundary between a step and its storage: what a Model or Resolver hands the store, and what the store hands back. They live here rather than with their producers because both sides need them, and core is the only layer both a producer and a store can depend on without inverting.

The membership rule is exactly that boundary — the store persists it or returns it. A shape that never leaves the process belongs with the code that produces it: SCHEMA_CLUSTERS is a resolver methodology's output, read by the Resolver that called it and by nothing else, so it lives in matchlab.resolvers.base. Keeping those out is what stops this module from becoming a catch-all for anything that happens to be an Arrow schema.

Functions:

• check_schema_subset –

  Check presence of Arrow fields, ignoring field order, extras and metadata.

Attributes:

• SCHEMA_MODEL_EDGES (Final[Schema]) –

  A deduper's or linker's output: a scored edge list.

• SCHEMA_RESOLVER_OUTPUT (Final[Schema]) –

  A resolver's complete flat output: one row per reachable source record.

• SCHEMA_JUDGEMENTS (Final[Schema]) –

  Stored user judgements, as returned by Store.read_eval_data.

• SCHEMA_CLUSTER_EXPANSION (Final[Schema]) –

  A cluster ID mapped to all its source cluster IDs.

SCHEMA_MODEL_EDGES: Final[Schema] = schema([('left_id', uint64()), ('right_id', uint64()), ('score', float32())])

SCHEMA_RESOLVER_OUTPUT: Final[Schema] = schema([('root', uint64()), ('leaf', uint64()), ('key', large_string()), ('source', large_string())])

SCHEMA_JUDGEMENTS: Final[Schema] = schema([('user_name', large_string()), ('endorsed', uint64()), ('shown', uint64())])

SCHEMA_CLUSTER_EXPANSION: Final[Schema] = schema([('root', uint64()), ('leaves', list_(uint64()))])

check_schema_subset(expected: Schema, actual: Schema) -> None