Gridiron
A forecasting and calibration testbed. Models are scored against public closing lines, which is the sternest benchmark available because a closing line already contains everyone else's information.
- Status
- In development — not in anyone's hands
- Benchmark
- Public closing lines
- Measures
- Calibration, not only accuracy
- Public URL
- None
Accuracy is the easy half. Calibration is the hard one.
A model that is right sixty per cent of the time tells you very little on its own. The question that matters is whether it is right sixty per cent of the time when it says it is sixty per cent confident — and wrong the rest. A model whose confidence does not mean what it says is worse than a coin, because it invites you to trust it exactly when you should not.
Most published forecasting work reports accuracy and stops. This exists to report the other number.
Feature engineering, a backtest harness, calibration scoring.
Features are engineered from public data and versioned alongside the models that consume them. The backtest harness replays them forward in time, so a model is never evaluated on information that did not exist when the forecast would have been made.
Scoring reports calibration explicitly — how closely stated confidence tracks observed frequency — rather than collapsing performance into a single accuracy figure.
Scored against public closing lines.
A closing line is a consensus forecast that has already absorbed everyone else's information, right up to the moment the event begins. That makes it an unusually honest benchmark: beating it is hard, and failing to beat it is informative rather than embarrassing.
The testbed is built to make that comparison routine and unflattering by default. A model that cannot outperform the public consensus is not promoted.
In development, and not in anyone's hands.
There is no public URL and no user other than its author. The status says so rather than implying momentum that does not exist.