1. Define the question and horizon
Decide whether you are evaluating game probabilities, a future FSPI level, or a longer-horizon distribution. A good result on one target is not validation of the others. Record the target and evaluation end before examining outcomes.
2. Freeze what was knowable
Choose a forecast origin and limit the model to information that was available by then. A database downloaded today may contain corrections made later, even when the event itself happened before the cutoff. Historical tests also need the roster known at the time rather than today’s roster.
3. Compare on equal terms
Use the same games, horizon, information policy, and evaluation metric for the reference and challenger. Change one assumption at a time where possible. More simulation paths reduce simulation noise; they do not repair a weak model or leaked inputs.
An example using a 30-day forecast
Suppose the question is, “Does adding this layer improve the predicted index at the end of a 30-day window?” We freeze a forecast origin, set the evaluation end, and run the reference and challenger with the same permitted information. The 30-day target stays fixed even if a seven-day result happens to look better.
In a fictional three-team example, predicted minus actual errors for the reference are +20, −20, and +60 points. The challenger has +10, −10, and +50. Their mean absolute errors are 33.3 and 23.3 points; their root mean squared errors are about 38.3 and 30.0. That is an improvement on those three cases, not proof of superiority across seasons.
| Metric | Reference | Challenger |
|---|---|---|
| Mean absolute error | (20+20+60)/3 = 33.3 | (10+10+50)/3 = 23.3 |
| Root mean squared error | √((400+400+3,600)/3) = 38.3 | √((100+100+2,500)/3) = 30.0 |
| Mean error (bias) | (+20−20+60)/3 = +20.0 | (+10−10+50)/3 = +16.7 |
4. Evaluate uncertainty as well as average error
For an index target, inspect error, bias, and interval coverage across held-out windows. For probabilities, inspect calibration and proper scores such as Brier score or log loss. Include recent seasons and difficult tails, not just an aggregate headline. If data or validation rules changed, document the changed comparison.
5. Use the workspace as a research record
Open Historical Tests, choose a supported origin and end date, then run an eligible configuration. Custom historical-test execution requires Pro access (including legacy Premium and admin access). Record the returned source, profile, and settings. Historical actuals evaluate the run; they are not training inputs. A temporary test does not update the published reference.
Looking beyond the best backtest
If we try 30 variants and publish only the best one, part of the apparent gain may simply come from having tried so many. We keep the experiment list, compare results across several historical periods, and reserve a fresh period that did not influence the choices. We also account for the fact that team outcomes within a season are related rather than treating every row as an independent experiment.
Look for where the result fails. A small average improvement with much worse recent-season error or unreliable tails may be a poor tradeoff. Report both the size of the difference and its uncertainty, and be explicit when an information-policy or scoring change makes an older comparison inapplicable.
6. Separate exploration from promotion
Choose success criteria before selecting the best result. If many variants are tried, preserve a fresh holdout or prospective evaluation. Save the negative findings too. A promising experiment is a reason for further testing, not automatic authority to publish a new champion.
Workflow aligned with the site’s historical-test and immutable-reference contracts; examples explain evaluation practice rather than disclose fitted model settings.
Independent educational research. Examples are hypothetical unless stated otherwise. Read current exchange rules and fund documents for product terms.
Open historical tests →