How a number gets onto a board

## 1 · The instrument

One comparison shows a model two entity texts and one attribute — a natural-language axis prompt — and asks which entity has more of the attribute and by roughly what ratio. The system prompt fixes a reference ladder of ratios [1.0, 1.05, 1.1, 1.2, 1.3, 1.5, 1.75, 2.1, 2.5, 3.1, 3.9, 5.1, 6.8, 9.2, 12.7, 18.0, 26.0] (advisory — any value in [1, 26] parses) and requires a JSON verdict:

{ "higher_ranked": "A" | "B", "ratio": r ∈ [1.0, 26.0], "confidence": c ∈ [0, 1] }

Models hold an explicit right to refuse ({"refused": true}); refusals are recorded, contribute nothing to the fit, and permanently blacklist that pair from re-proposal. A malformed response is treated as a refusal, not silently repaired. Sampling temperature is 0. Entity texts are XML-escaped and wrapped in delimited context blocks, so entity content cannot break the prompt structure.

Counterbalancing. Every planned pair is asked in both presentation orders — two independent provider calls, the second with the entities' A/B labels swapped and the verdict re-mapped afterward. Position preference is a first-order bias of judge models (we measure order-disagreement rates around 20% on hard pairs); paying 2× per pair converts that bias from a confound into data. Per run we land the count of pairs whose two orders disagreed, and a signed order-residual whose expectation is zero for an unbiased judge.

## 2 · Which pairs get asked

Comparisons are not sampled uniformly. Each batch is chosen by an adaptive top-k design that spends budget where the ranking is still uncertain:

• Confidence bounds lcb/ucb = μ ∓ β·σ (β = 1.64 at the deployed error tolerance) define an uncertainty band around the rank-k boundary; candidate pairs are frontier incumbents × challengers, rank-neighbors inside the band, and the critical pair (weakest incumbent lower bound vs strongest outsider upper bound).
• Each candidate is scored by expected information gain Δ = ½·ln(1 + λ·R_eff) — R_eff the effective resistance between the two entities in the current comparison graph — blended with expected rank-risk reduction and weighted by the probability the pair could actually invert the board.
• Entities with fewer than 2 comparisons get forced-exploration pairs against anchors chosen by a golden-ratio stride across the ranking — deliberately not hub-and-spoke, which our regret benchmark measured as worse than uniform random pairing.

The budget is fixed at 8·n provider calls for n entities (≈ 4·n distinct pairs under dual-order counterbalancing). A run stops early only by certification: the critical top-k pair must separate under inflated bounds (β×1.25) computed with exact effective-resistance variance, with both endpoints well-connected, on two consecutive checks, and never from a numerically degraded solve. Otherwise it runs to budget and says so — stop_reason is landed on every score row.

## 3 · The estimator

Each accepted comparison becomes a signed log-ratio observation μ_e = ±ln(ratio) on the edge between its two entities. Repeated observations of the same ordered pair are precision-fused into a single edge before fitting. Scores are the solution of a robust weighted least-squares problem on the comparison graph:

s* = argmin_s  Σ_e  λ_e · ρ_huber( μ_e − (s_i − s_j) )

solved by iteratively reweighted least squares on the graph Laplacian: Cholesky solve, residuals scaled by 1.4826·MAD, Huber reweighting at k = 1.5, iterate to a 10⁻⁸ relative objective tolerance (max 12 iterations). A 10⁻⁹ ridge guards the solve; if factorization still fails the ridge escalates and the run is flagged degraded, which blocks certified stopping. One node per connected component is pinned and each component is shifted to min-zero — the gauge is explicit: scores are relative positions within a run, not portable absolutes.

The Huber loss matters: a single wild judgement (a misread pair, a provider glitch) enters with bounded influence instead of dragging the whole board. Each solve also lands consistency diagnostics — the fraction of judgement energy that is structurally intransitive (the cyclic component of a Hodge decomposition of the flow), and leave-one-out studentized residuals flagging individual comparisons the fit had to override.

## 4 · Uncertainty

latent_std is the posterior marginal standard deviation from the inverse reduced Laplacian at the converged robust weights — computed exactly (per-column Cholesky solves) for boards up to 256 entities, and by Hutchinson trace estimation (12 ±1 probes, deterministically seeded from the run seed and edge multiset) above that. Boards display 95% intervals as μ ± 1.96σ. z_score is a robust standardization within the run (median/MAD); percentile is the mid-rank percentile; topk_error is the expected number of top-k boundary inversions summed over the monitored frontier — an expected count, not a probability.

## 5 · What we disclose that most eval harnesses don't

• Stated confidence is recorded but does not weight the fit. Every comparison's confidence lands in the ledger for audit, but point judgements enter the likelihood with unit precision: we have not validated model self-reported confidence as calibrated, so we refuse to launder it into the posterior.
• Scores are run-relative. The gauge (min-zero per component), the z-scores, and the percentiles are all defined within a run's entity set. Cross-run comparability is an empirical question — which is why we measure replication (below) instead of asserting it.
• Provider nondeterminism exists at temperature 0. Replays of the same comparison can differ. What is deterministic: given the same set of accepted observations, the solve is byte-identical (ordered edge fusion, seeded probes), and the full trace makes every run re-fittable offline.
• Ratio elicitation is capped at 26×. The instrument cannot express larger single-pair ratios; extreme spreads accumulate across chains of pairs instead.
• Failure is loud. A run whose comparisons all failed lands as a failure naming the dominant provider error — never as a quietly flat board. Zero-cost cached comparisons are labeled (cached); in current production every comparison is a live call.

## 6 · Replication, measured

Same attribute, same entities, independent runs (fresh seed → different comparison graphs) — how stable is the ranking? Every duplicate campaign pair currently in the public ledger:

k = 9 duplicate pairs · median ρ = 0.805 · range [0.51, 0.95] · all pairs same model and budget, independent seeds. This is run-to-run rank stability, honestly bounded — not a claim that any single number is a portable absolute. Axes with lower ρ are themselves informative: they are the attributes the judge model holds least stably.

Exact reproducibility, when the provider permits it. Rerunning the full landmark-abstracts board end to end with claude-haiku-4.5 (24 entities, 192 calls) reproduced all 93 comparison pairs and the exact rank order (ρ = 1.000): the planner is deterministic given identical judgements, and haiku at temperature 0 returned identical judgements. The sonnet duplicates above vary because that provider path does not — which is exactly why we report both layers separately instead of averaging them into one "reliability" number.

Cross-model agreement is attribute-dependent — and that is the point. The same 24 landmark ML paper abstracts, judged on two attributes by three model families (192 calls each, zero refusals, zero errors):

Claim density is a construct three model families hold nearly identically (ρ 0.86–0.98). Self-containedness is construal-divergent: haiku and gemini agree with each other and gpt-5.4-nano ranks by a different reading of the same words. A leaderboard that averaged over judges would bury this; a ledger indexed by (model, attribute, entity) measures it. This is why every OpenPriors row names its model.

## 7 · Lineage and what's different

The estimator stands in the Thurstone / Bradley–Terry tradition of paired comparison, and the arena-style Elo boards popularized for chatbots are the nearest neighbors in spirit. OpenPriors differs in four deliberate ways: the attribute is an arbitrary natural-language axis, not a fixed preference; the elicitation is a cardinal ratio, not a binary win — so magnitudes, not just orderings, constrain the fit; every fitted number carries a posterior interval from the comparison graph rather than a point rating; and the entire evidentiary chain — prompt, template hash, model, seed, each raw comparison with its tokens and dollar cost — is a public, per-row ledger, not a leaderboard summary. LLM-as-judge pipelines typically hide exactly the things this system is built to expose.

## 8 · Read it yourself

Boards are free to read, as JSON, no key required:

curl 'https://openpriors.com/v1/judgements?lens={lens}&axis_key={axis}'
curl 'https://openpriors.com/v1/judgements/attributes'

Each row carries the entity (id, exact judged text, content hash), the attribute (axis prompt and hash), the estimate (point, 95% interval, distribution), context (rank, percentile, z, cohort size), and provenance (run, model, harness version, temperature, seed, comparison counts, stop reason, cost in nanodollars). Runs are submitted from the run form or the same API with prepaid credits or your own OpenRouter key.