Forecast accuracy without score-chasing
A practical guide to measuring forecast error, spotting bias and evaluating with time-ordered holdouts.
A forecast score is useful only when it helps someone choose a better action. Start by defining the item, time bucket, forecast horizon and decision. Then use a small set of measures that explain both the size and direction of misses.
1. Compare forecasts with outcomes they did not train on
Keep the test period later in time than the training period. For a planning horizon of four weeks, assess four-week-ahead forecasts; a one-step score may not describe the decision you actually make. Rolling-origin evaluation repeats the exercise at several historical cutoffs, always using only information that would have been available at each cutoff.
Do not treat training residuals as a substitute for real forecast errors. Residuals help diagnose a fitted model; holdout forecast errors show how it performed on unseen periods.
2. Read the scorecard in units and direction
Suppose four periods had actual demand of 100, 110, 90 and 105 units. Forecasts were 95, 108, 100 and 100. The signed errors (forecast minus actual) are −5, −2, +10 and −5.
MAE answers “how many units off, on average?” Bias answers “which direction does the plan lean?” A model can have a small average signed error because positive and negative misses cancel, while individual misses remain large.
3. Pick measures that fit the data
- MAE: simple to explain in units, but it cannot compare series with different scales by itself.
- RMSE: also in units, and it penalizes a few large misses more strongly.
- WAPE: weights absolute misses by total actual volume. It can make a portfolio summary easier to interpret, but large-volume products can dominate it.
- MAPE: percentage errors can be undefined or extreme when actuals are zero or close to zero.
- MASE: scales error against a simple naïve benchmark and can help compare series, but its scale denominator also needs to be defined and nonzero.
No one metric describes customer service, inventory cost, forecast bias, and forecast value at the same time. Review the units and segments behind the total.
4. Make the score actionable
Break results out by product family, horizon, channel and lifecycle stage. Separate baseline performance from overrides where possible. When a segment repeatedly misses, inspect stockouts, promotions, product changes, calendar alignment and data latency before replacing the model.
Use the scorecard as a diagnostic. A forecast that is slightly less accurate in aggregate may still support a better decision if it identifies risk earlier or performs better on the products that matter most.
What the cited sources report
Hyndman and Koehler compare forecast-accuracy measures and show that several common measures can become undefined or misleading for ordinary data patterns, including zeros and repeated values. Their paper proposes mean absolute scaled error (MASE) for comparisons across series. Forecasting: Principles and Practice describes genuine holdout forecasts and rolling-origin evaluation, where each test forecast uses only earlier observations.
How to use this in a planning conversation
For an operations review, keep a compact trio: an error measure in units, an aggregate relative measure where its denominator is appropriate, and signed bias. Match the horizon to the decision and discuss service or inventory outcomes separately. These are practical recommendations, not a claim that one metric works for every product.
Sources and publication details
Source findings are summarized in our own words. Follow the links for the original work and context.
- Rob J. Hyndman & Anne B. Koehler, “Another look at measures of forecast accuracy,” International Journal of Forecasting, 22(4), 679–688.Peer-reviewed research article · October–December 2006
- Rob J. Hyndman & George Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., “The forecaster’s toolbox” and “Time series cross-validation.”Open textbook · 3rd edition · accessed 28 October 2026
- Forecasting: Principles and Practice, “Time series cross-validation.”Open textbook chapter · accessed 28 October 2026