Calibration scoreboard
A prediction is well-calibrated when its stated probabilities match observed frequencies. Lower Brier and lower expected calibration error (ECE) are better. Measured on a pooled walk-forward holdout, 124,731 predictions across 49 investors. Data as of Jun 2026.
Base rate (new buys)
12.46%
Brier (raw → calibrated)
0.0950 → 0.0947
ECE (raw → calibrated)
0.0066 → 0.0068
Top decile hit rate
35.29% (2.9x base)
Method: isotonic (pooled walk-forward holdout). Calibration cuts the expected calibration error from 0.0066 to 0.0068, but the underlying signal is weak, new-buy events are rare (12.46% base rate), so a small Brier improvement (0.4%, 0.0950 → 0.0947) is all that is honestly on the table.
Two things this page used to get wrong, corrected 2026-08-13 by
scripts/gates/q23_calibration_crossfit.py. The ECE above is now measured on equal-frequency bins and on out-of-fold predictions. The earlier figure used equal-width bins and scored the calibration map on the same rows it was fitted on, and on this data that combination cannot fail: predictions cluster near the base rate, so a constant predictor that ignores every feature scores an ECE of exactly zero. Second, the isotonic map tops out at 43.56%, so no number on this site can honestly read like a coin flip or better. Rare events stay rare. The claim being made is lift, and it is that the top decile fires at 35.29% against a 12.46% base rate.Leak-free walk-forward backtest: top-10 predicted new buys ranked from a broad ~2,000-name universe known at the rebalance date, equal-weight, 1-quarter holds, complete quarters only, gross (before costs). Small samples, per-investor edges are noisy and not statistically significant.