eagL / calibration and brier scores
Brier scores and calibration curves for prediction markets
A forecast of 70% is not right or wrong when the event happens. It is right if events called 70% happen about 70% of the time. That property is calibration, and grading it takes more care than counting wins.
Why accuracy is the wrong grade
Score a probability model by whether its favored side won and you reward cowardice. A model that calls everything 51% and a model that calls everything 99% earn the same accuracy on the same outcomes, yet one of them is lying to you about risk. Binary contracts pay out on probability, so the grade has to be a proper scoring rule, one that a forecaster maximizes only by reporting what they actually believe.
Brier score
The Brier score is the mean squared error between the forecast and the outcome: (p − o)² averaged over forecasts, where o is 1 if the event happened and 0 if not. Forecast 0.70 on an event that happens and you contribute 0.09; on one that misses, 0.49. Lower is better, 0 is perfect, and always answering 0.5 scores 0.25, which makes 0.25 the number to beat. Its useful property is symmetry: it punishes overconfidence and underconfidence alike, in proportion to how wrong the stated probability was.
Log loss
Log loss is −log(p) when the event happens and −log(1 − p) when it does not. It agrees with Brier most of the time but diverges hard at the extremes: forecast 0.99 on an event that misses and Brier charges you about 0.98 while log loss charges 4.6. If your forecasts feed position sizing, that severity is a feature. Ruin lives in the tails, and log loss is the score that refuses to let a model be casually certain.
Calibration curves
A single score hides where the model fails. A calibration curve bins forecasts, all the 60 to 70% calls in one bucket, and compares each bin's average forecast to its realized frequency. Ten bins is the conventional resolution. A well-calibrated model traces the diagonal. A model that bends below it in the high bins is overconfident exactly where confidence is expensive. The catch is sample size: ten bins over 40 forecasts is four outcomes per bin, and four coin flips prove nothing.
How eagL applies this
eagL scores every resolved crypto forecast with Brier, log loss and a ten-bin calibration curve, and refuses to publish any of it below 100 settled outcomes. The model being graded is described on the crypto markets page; the point of grading it at all is the argument of the prediction market quant page.
All of this ships inside eagL.
$499 once, for Apple Silicon Macs. No subscription.