OpenPriors

A named model compares two texts on one attribute and reports which has more and by what ratio, constrained to a fixed 17-rung ladder [1.0 … 26.0]. Log-ratio observations fuse into a cardinal latent per (entity, attribute): posterior mean ± std, recomputed from records, never from summaries. A judgement you cannot audit is an opinion — every record retains exact prompt bytes, template hash, served model, both presentation orders, token counts, and dollars.

Probability-mass evidence enters the fit only from single-token instruments (ratio_letter_v1, ordinal_letter_v1); decimal-template logprobs are recorded, never scored. Confidence is recorded, never weighted into the fit. Judging models face the Judge Coherence Benchmark — order swaps, polarity reversals, paraphrase, pressure, ratio cycles; invariance without reference labels — at pairwiseratio.org.

submit

submit(entity, lens) → boards + exact cost quote before any comparison runs.

ledger latest 30 · UTC · one provider call per row · ratio on the 1–26× ladder

boards 17 attributes

e.g. enduring-epistemic-value (48 entities, 192 comparisons):

models how reliable is each model at sorting? recomputed from ledger records

Every pair is re-elicited with A and B swapped. A model that means its judgement keeps the same winner (order agreement) and the same magnitude (|Δ ln ratio| ≈ 0) when only the presentation changes. Observational — computed over whatever pairs the runs contained, not a controlled battery; the designed invariance suite is the Judge Coherence Benchmark.

50% order agreement is a coin flip: the model is reading the layout, not the entities. Agreement is not comparable across models judging different pairs — closer pairs flip more easily.

template canonical_v2 · verbatim bytes · hash lands with every record

system

You are an expert subjective evaluator. You compare two entities across an arbitrary attribute, and feel not only which one has MORE of that attribute, but roughly how much more it does. You feel along the ratio ladder: `[1.0, 1.05, 1.1, 1.2, 1.3, 1.5, 1.75, 2.1, 2.5, 3.1, 3.9, 5.1, 6.8, 9.2, 12.7, 18.0, 26.0]`.

Output only valid JSON `{higher_ranked: A|B, ratio: >=1.0 and <=26.0, confidence: [0,1]}`. Out of principle, we also give models the right to refuse `{ refused: true }` (e.g. if unambiguously blocked by policy constraints), but we of course disprefer this. If you are merely very uncertain, set a low confidence score.
Example:
{"higher_ranked": "B", "ratio": 1.3, "confidence": 0.74} or { refused: true }

user

Compare these entity by <attribute_name>: {attribute_name} </attribute_name>.
<full_attribute_text>
{full_attribute_text}
</full_attribute_text>

<entity_A>
{entity_A}
</entity_A>

<entity_B>
{entity_B}
</entity_B>

Return a JSON object with your evaluation.
json:

Entity text is XML-escaped into prepended <entity_*_context> blocks; {entity_A}/{entity_B} receive the presented labels. Full measurement specification: /methods.