How we check our numbers

Agentic onchain finance runs on estimates. An agent is about to sign a swap on your behalf, and something upstream has judged the route. On what basis, checked how? In a system where the transaction is irreversible the moment it settles, an unchecked estimate is a real risk. So this post is narrow and specific: it describes exactly what Routescore publishes on its calibration surface, exactly what that surface does and does not measure, and how you can hold it to account.

Routescore is a read-only, pre-sign evidence element for that moment. It produces a modeled route score with caveats and records the decision so it can be reviewed later. This post is about the accountability layer underneath: the public /calibration surface.

What is calibrated — and what is not

An important scope statement first, because it is easy to over-read.

The calibration surface does not evaluate the composite 0–100 Routescore (heuristic.route_score.v1). It evaluates three published, source-specific baseline forecasting tasks — each a probability forecast for its own measured quantity, built from that source's own prior observations:

• settlement_cost_baseline.v1
• onchain_quote_dispersion.v1
• routing_leak_baseline.v1

These baselines are simpler, checkable reference forecasts — not the full heuristic that produces the number you see in the app. Calibrating them tells you how well those baseline forecasts track their measured quantities. It does not, on its own, tell you the composite score is calibrated, ranks routes well, or would have saved money on any particular trade. We are careful about that boundary because collapsing it would be exactly the kind of over-claim this surface exists to avoid.

What "calibrated" means, in plain language

A probability forecast is calibrated when its stated probabilities match outcomes over the long run: when a baseline says "about 20%," that class of cases should resolve the bad way roughly one time in five. We report two standard numbers:

• Brier score — the mean squared gap between predicted probability and outcome, 0 (perfect) to 1 (worst).
• Expected calibration error (ECE) — group predictions into confidence bands and measure how far each band's realized rate sits from its claimed rate. Low ECE means "when we say 30%, it really is about 30%."

Both numbers test probability calibration only. They do not measure discrimination (telling good routes from bad), route-ranking skill, or causal savings — a naive base-rate forecaster can score well on calibration while being useless for ranking. Calibration is a floor to clear, not proof of usefulness. The full caveats live on the methodology page and limitations.

Modeled predictions versus measured outcomes

Every pre-sign risk estimate Routescore produces is modeled: a point-in-time forecast built from public signals before the trade exists. The calibration surface holds each baseline's modeled prediction up against a measured quantity recorded afterward. (Not everything on the surface is modeled — observed counts, error figures, and content hashes are measured or computed, not forecast.)

"Measured," though, does not mean all three sources are executed trades. Each measures a different quantity in a different way, and the differences matter:

• Settled execution cost (onchain_settlement_reconciler) — real executed swaps: the cost at settlement versus a reconstructed pre-trade pool mid. This is what the chain actually recorded happening.
• Reconstructed historical on-chain quote dispersion (onchain_quote_dispersion_reconciler) — cross-venue dispersion rebuilt from historical on-chain pool state via QuoterV2 reconstruction. This is a real-but-hypothetical quote reconstructed from chain state at the time — not a trade that executed.
• Subsequently-observed cross-aggregator quote dispersion (dex_quote_diff_reconciler) — the best-versus-worst spread across independent aggregators (1inch, 0x, CoW) for the same intent, observed after the fact — a second-opinion market view, again not an execution.

Two of these paths are on-chain and one reads aggregators; each measures a distinct quantity.

Three acquisition paths — and the honest limit of "independent"

The value here is not that one part predicts while another grades — each reconciler produces both its prediction and its grade within the same pipeline. The independence is across acquisition paths and measured quantities: three different ways of obtaining a realized-quantity signal, so the calibration record does not lean on any single source or single measurement.

That reduces single-source dependence. It does not eliminate common-mode failure, market manipulation, or model error — two of the three paths read the same Ethereum state and the same pools, so a fault in that shared substrate can move more than one of them together. Three lenses are better than one; they are not three fully independent worlds.

The public gate — the exact bar

A cohort earns a public claim only when it clears a strict, per-source gate. The published thresholds are:

• Each qualifying source must have ≥ 30 confirmed labels and ≥ 30 probability-eligible pairs on its own outcomes.
• A label counts only at label confidence ≥ 0.6.
• On its own pairs the source must reach Brier ≤ 0.2 and ECE ≤ 0.1.
• A "bad outcome" is the source's own measured quantity reaching ≥ 30 bps (for settled execution cost that is an incurred loss; for the two dispersion sources it is a ≥ 30 bps dispersion, not an incurred loss).
• The source must measure a known quantity — a source whose realized quantity is not in the published mapping never counts.
• At least two such qualifying sources must clear the gate independently for the cohort to be public-eligible.

The per-source gate is deliberately not a blended average: a pooled Brier across three different measured quantities would hide a weak source behind a strong one, so each source is judged on its own outcomes.

What "no public claim" actually does

If a cohort does not clear the gate, it does not disappear. Its diagnostic Brier, ECE, and counts stay visible for transparency; it simply carries a "No public claim" badge and is not designated public-eligible. (When the live snapshot is unavailable, the surface can render a clearly labeled fixture whose metrics are illustrative, marked as such.) The number is never hidden, and illustrative figures are never presented as live — each is shown with an honest label about whether it is a live measurement and whether it has cleared the bar.

A note on scope: cohorts are one-dimensional breakdowns — by chain, by venue, by token pair, by size bucket, by freshness, or by source. They are not joint chain×pair×size×freshness slices. Each dimension is grouped and reported on its own.

Drift — how a claim is withdrawn

The gate also runs forward. A drift monitor watches live outcomes for degradation, in particular high-confidence "low-risk" predictions that resolved to a bad outcome. In the current version this suppression is overall: when the monitor flags a downgrade, all public claims in the snapshot are withdrawn together back to "No public claim" until they recover. Per-cohort drift — suppressing only the cohorts that actually drifted — is a future enhancement, not what runs today. The proof self-suppresses on drift, at the whole-snapshot level.

Provenance — bound to a version and a hash

A snapshot must be bound to a recorded source-set version and content hash before any public-eligible claim in it is shown. An unbound snapshot is treated as "no public claim" by construction — so every public-eligible claim carries that binding. This means the provenance of a published number can be challenged: you can see the recorded source-set version and hash it is bound to, and hold that fixed reference to account.

To be precise about what exists today: the surface exposes the version and hash, not a downloadable observation bundle or a one-command re-derivation. The claim is not "reproduce it offline right now" — it is "the number is pinned to a specific, recorded source set that can be cited and disputed." The machine-readable snapshot is available at /api/calibration.

Why the honest states are the point

In agentic finance the expensive failure is false confidence — a clean-looking number an agent leans on and a user pays for. A surface that only ever showed green would be less trustworthy, not more; it would be hiding the places it does not yet know. So "no public claim" is a designed feature: a cohort either clears the published gate on measured outcomes, or it plainly says it has not. An illustrative figure is never presented as a live measurement.

A gentle invitation

If any of this is useful to how you or your agents make onchain decisions, you are welcome to look. Read a route's modeled score with its caveats attached. Open /calibration, find a cohort that has cleared the gate, and check the source-set version and hash it is bound to. The methodology (/docs/methodology), the known limits (/limitations), and the raw snapshot (/api/calibration) are all public.

Routescore is one composable, read-only evidence element in the agentic finance stack — it models exposure and records the decision; it does not execute, route, sign, or guarantee anything. The part we ask you to weigh is the part you can inspect for yourself.