Start with something familiar
Practicing an exam with the answer sheet open can create confidence without testing knowledge. A forecast can fail the same way when it uses a corrected future record, final event results, or an average spanning later weeks. The resulting score answers an easier question than the planner faced. A fair rehearsal reveals each observation only when it would have been known, and it preserves the first answer before allowing a revision.
Words to know: Forecast origin, Leakage, MAE.
Freeze the information set
At an origin after week four, only information actually available then is permitted. Even a week-four observation may arrive late. List fields with availability dates, not just event dates. Reconstruct those historical information sets when possible and disclose when a modern corrected extract cannot fully reproduce what was known at the time.
Use early data for initialization, later validation for selection, and a final chronological holdout for checking the selected method. Once the holdout influences changes, it is development data. You may still learn from it, but you cannot continue calling it untouched evidence. Record candidate methods and selection metrics before inspecting outcomes.
Walk the origin forward
For actuals 10,12,14,16,18,20, an origin after week three predicts week four using only 10,12,14. Naive gives fourteen and a three-week mean gives twelve. After week four becomes known, advance the origin and repeat. This resembles a real weekly planning process and creates comparable target rows.
A two-week-ahead forecast for week six issued after week four cannot use week-five actual. Keep horizons separate before pooling scores, and use common origin-target pairs for every method. Missing forecasts need explanations; otherwise a method can appear stronger merely because it skipped the hardest targets or lacked enough history.
Audit preprocessing too
Leakage enters before model fitting. Full-history scaling, outlier decisions informed by later outcomes, and interpolation using future neighbors can all leak. A promotion schedule known at the origin may be valid; final promotion sales are not. Distinguish planned predictors from realized values and repeat data preparation within each training cutoff.
Small validation advantages may be unstable. Inspect the number of origins, changes in demand, and consequences of errors. A maintainable simple method may be sensible when performance is similar. Publish origin policy, horizon, candidate set, metric, and monitoring rule so the decision can be reproduced rather than accepted on authority.
- Define target
- Check evidence
- Calculate
- Challenge assumptions
- Record decision
Test with the future hidden — worked case
All inputs below describe fictional Northstar bottles. Keep the stated units and timing when reproducing the calculation.
| Target week | Actual | Naive | Three-week mean |
|---|---|---|---|
| 4 | 16 | 14 | 12 |
| 5 | 18 | 16 | 14 |
| 6 | 20 | 18 | 16 |
- 1. Origin three uses actuals 10,12,14 only.
- 2. Advance the cutoff one week for each target.
- 3. Naive absolute errors [2,2,2]; mean errors [4,4,4].
Naive MAE two; three-week mean MAE four for these one-step targets.
Assumptions
- Periods and quantities use the definitions stated in this example.
- Only explicitly supplied events or adjustments are included.
Where this can fail: The arithmetic checks the stated assumptions; it does not establish that uncertain future events will occur.
Build, test and explain a plan
Twelve actuals [10,12,14,16,18,20,22,24,26,28,30,32]. Methods naive and three-week mean. Select on targets four to eight; final holdout targets nine to twelve. Completed weeks are immediately available. Evaluate two-week horizons separately for targets five to twelve from origins three to ten. All units are bottles unless stated otherwise. Use a spreadsheet or ruled paper. Eight hours includes construction, comparison, writing, and teach-back; this is a planning estimate, not an experimentally validated completion time.
Use paper or a spreadsheet. The estimates below include the study and worked example already on this page.
| Estimated hours | Activity |
|---|---|
| 0.5 | Study the lesson and write a data dictionary defining every field, time bucket, unit, and cutoff. Separate observed facts from assumptions. Identify two ways a colleague could misread the target and rewrite its definition to prevent those errors. |
| 0.5 | Reproduce the worked example independently, showing formulas and intermediate values. Compare against the answer and keep a correction log explaining each difference. Check one result a second way using a total, balance, or direct manual calculation. |
| 3 | Create an origin-target grid with horizon, permitted training rows, forecast, outcome, and error. Select on validation before inspecting holdout. Report separate one-step and two-step scores. Keep inputs separate from formulas, label every origin and target, and add independent row and total checks. Produce an annotated chart with at least two observations tied to specific rows; include a brief explanation of why each check matters. |
| 2 | Add a one-week reporting lag: origin t knows only through t-1. Rebuild eligible rows. Demonstrate an invalid centered mean using target-1,target,target+1 for targets four to eleven and quantify its deceptive score. Preserve each variant separately. Create an output comparison table and explain which changed assumption caused each movement. Identify a scenario that affects timing differently from totals and state what the planner should do with that distinction. |
| 1 | Write a 400-word decision memo using your own numerical outputs. State the proposed action, evidence, two limitations, an owner for the next check, and a measurable condition that would change your recommendation. Attach the model so a reader can audit every cited value. |
| 1 | Prepare a five-minute teach-back using one worked row and one scenario comparison. Explain it aloud to a colleague or recording. Write three skeptical questions, answer them with evidence, and revise the memo or model where your explanation exposes an unsupported assumption. |
Your deliverables
- Auditable input and formula model with checks
- Scenario comparison and annotated chart
- Decision memo, correction log, and teach-back notes
Check your practice work
- No-lag one-step naive MAE two; moving mean four, in both validation and holdout.
- No-lag two-step naive MAE four; moving mean six.
- One-week reporting lag raises one-step naive MAE to four and mean to six where three training rows exist.
- Centered mean equals every target on this linear series: invalid MAE zero.
- Choose naive before opening holdout; later changes require a new future check.
Keep these three ideas
- Build chronological rolling-origin tests and preserve an untouched final holdout.
- Keep original evidence distinct from modeled values and assumptions.
- A precise output remains conditional on definitions and timing.
Common mistakes
- Overwriting the original evidence after a revision.
- Reporting a number without its unit, period, cutoff, and assumptions.
Optional video study
Build chronological rolling-origin tests and preserve an untouched final holdout.
- Freeze the information set
- Walk the origin forward
- Audit preprocessing too
One half of the module narration, with a distinct Northstar calculation and a bridge to the paired lesson.
Open the module media & quizPut this idea to use
Carry your assumptions into a practical tool, then compare your result with a short guide.
Open a related planning toolRead the related planning guidePractice with the planning gameThis saves a completion marker in this browser only. It does not verify time spent.