Accuracy and honest validation

Here error equals actual demand minus forecast. Positive means underforecasting; negative means overforecasting. Other businesses may reverse the sign, so publish the formula. Mean error describes signed bias over the selected rows. Near-zero bias does not imply small misses, and a short window does not prove long-run systematic behavior. For actuals forty, sixty, zero, twenty and forecasts fifty, fifty, ten, ten, errors are minus ten, ten, minus ten, ten. Their mean is zero, while mean absolute error is ten bottles. Both are correct. Inspect signed row errors and magnitude together rather than treating one balanced total as evidence that the planning process needs no improvement. RMSE is the square root of mean squared error and penalizes large misses more heavily. WAPE here is summed absolute error divided by summed actual demand. The example gives forty divided by one hundred twenty, or thirty three point three three percent. Ordinary MAPE divides each row by its actual and is undefined at the zero-demand row; silently dropping that row changes the evaluation. WAPE is undefined when total actual demand is zero. Use absolute errors and explain the missing denominator. Portfolio WAPE emphasizes high-volume demand and can hide weak performance on small critical items. Show selected product metrics as well as the aggregate, and state whether you pool units or average item-level percentages. Compare methods on identical targets, origins, and horizons. A forecast updated one day before shipment has more information than one frozen four weeks earlier. Preserve both versions if both matter, but do not confuse the information advantage with model quality. Use demand actuals consistent with the target, not shipments capped by capacity. Business costs can be asymmetric: a shortage may cost more than an excess unit. Keep cost scenarios separate from statistical error metrics and state the rates used. Preserve baseline and adjusted forecasts to test whether judgment helped. A favorable business result does not retroactively make an incorrect forecast correct. At an origin after week four, only information actually available then is permitted. Even a week-four observation may arrive late. List fields with availability dates, not just event dates. Reconstruct those historical information sets when possible and disclose when a modern corrected extract cannot fully reproduce what was known at the time. Use early data for initialization, later validation for selection, and a final chronological holdout for checking the selected method. Once the holdout influences changes, it is development data. You may still learn from it, but you cannot continue calling it untouched evidence. Record candidate methods and selection metrics before inspecting outcomes. For actuals ten, twelve, fourteen, sixteen, eighteen, twenty, an origin after week three predicts week four using only ten, twelve, fourteen. Naive gives fourteen and a three-week mean gives twelve. After week four becomes known, advance the origin and repeat. This resembles a real weekly planning process and creates comparable target rows. A two-week-ahead forecast for week six issued after week four cannot use week-five actual. Keep horizons separate before pooling scores, and use common origin-target pairs for every method. Missing forecasts need explanations; otherwise a method can appear stronger merely because it skipped the hardest targets or lacked enough history. Leakage enters before model fitting. Full-history scaling, outlier decisions informed by later outcomes, and interpolation using future neighbors can all leak. A promotion schedule known at the origin may be valid; final promotion sales are not. Distinguish planned predictors from realized values and repeat data preparation within each training cutoff. Small validation advantages may be unstable. Inspect the number of origins, changes in demand, and consequences of errors. A maintainable simple method may be sensible when performance is similar. Publish origin policy, horizon, candidate set, metric, and monitoring rule so the decision can be reproduced rather than accepted on authority. In the lab, put a bold line at the end of what was known. Each forecast must stay on its own side of that line until the next fact arrives. Check one row by hand before you fill the rest. Then show both the size and the sign of each miss. Ask if two large misses have cancelled in the total. Ask if a zero has been dropped just to make a score work. The best score is of no use if the test was not fair. Your job is to make the score honest and its use clear.

1. Declare the sign

   Original flat diagram with separate labeled input and output boxes, using Northstar bottle icons.

   • Declare the sign
   • Zero bias coexists with ten-unit MAE and RMSE.
   • Check units and timing
2. Make denominators visible

   Original calculation card with aligned quantities and an arrow showing one change.

   • Make denominators visible
   • Zero bias coexists with ten-unit MAE and RMSE.
   • Check units and timing
3. Score the decision fairly

   Original comparison chart with clearly labeled cases and a small assumption note.

   • Score the decision fairly
   • Zero bias coexists with ten-unit MAE and RMSE.
   • Keep assumptions visible
4. Freeze the information set

   Original flat diagram with separate labeled input and output boxes, using Northstar bottle icons.

   • Freeze the information set
   • Build chronological rolling-origin tests and preserve an untouched final holdout.
   • Check units and timing
5. Walk the origin forward

   Original calculation card with aligned quantities and an arrow showing one change.

   • Walk the origin forward
   • Build chronological rolling-origin tests and preserve an untouched final holdout.
   • Check units and timing
6. Audit preprocessing too

   Original comparison chart with clearly labeled cases and a small assumption note.

   • Audit preprocessing too
   • Build chronological rolling-origin tests and preserve an untouched final holdout.
   • Keep assumptions visible