STUDENT WYCKOFFPro
Blog and practice / Risk and results

Risk and results · Next step

HistoryEvaluation

What a backtest leaves unknown: samples, fitting and changing conditions

A historical table only answers the question actually built into it. If rules were chosen after seeing the outcome, fills were assumed perfect and new conditions were not examined, an attractive total still leaves important unknowns.

All numerical examples use fictional educational data. No real orders were submitted.

Write the rule and experiment boundaries first

This invented sequence of opens and closes is neither market data nor a Student Wyckoff engine test. The toy rule is: if the last completed close is above the preceding close, buy one hypothetical unit at the next open and exit at that bar’s close. Otherwise do nothing.

Exact open and close executions are assumptions. Commission and slippage are zero; taxes and financing are excluded. This does not establish that such prices could actually have been traded. Results are hypothetical money units, not account-return percentages.

The signal and its result belong to different times

An entry on bar t can compare only C[t−1] and C[t−2]. C[t] is not yet known and is used later to calculate the exit. The first two bars lack enough previous closes. For example, entry on bar 3 uses the already known 102 and 100, not the future 104.

Using bar 3’s close to decide a purchase at its open would introduce future knowledge. The table keeps the sequence intact: past observations, assumed execution, then the outcome once the bar closes.

12 fictional bars: one unit, ideal fills, costs 0
BarSegmentOpenCloseAction from previous closesHypothetical result
1Development100.00100.00Insufficient previous data—
2Development100.00102.00Insufficient previous data—
3Development102.00104.00Buy at open → exit at close2.00
4Development104.00103.00Buy at open → exit at close-1.00
5Development103.00105.00No transaction—
6Development105.00107.00Buy at open → exit at close2.00
7Evaluation107.00103.00Buy at open → exit at close-4.00
8Evaluation103.00102.00No transaction—
9Evaluation102.00104.00No transaction—
10Evaluation104.00100.00Buy at open → exit at close-4.00
11Evaluation100.0099.00No transaction—
12Evaluation99.00101.00No transaction—

Separate development and evaluation in time

Imagine that only the first six bars were available when the rule was chosen. Freeze it after bar 6 closes. Bars 1–6 produce three toy transactions: +2, −1 and +2, totalling +3. This describes the development segment rather than proving quality.

Treat bars 7–12 separately. The signal for bar 7 is already known at bar 6’s close, which is valid. The second segment produces −4 and −4, totalling −8. The positive first segment does not carry into the next even under idealised assumptions.

Two time segments, not actual trading
SegmentBarsAttemptsHypothetical gross total
Development; freeze after bar 61–633.00
Separate evaluation without retuning7–122-8.00

Where fitting begins

If you alter the rule or select another parameter after seeing the second segment, it has become development data too. A new independent evaluation needs unused material. Trying many alternatives increases the chance of selecting a fortunate coincidence; the list of attempts matters alongside the winning parameter.

This rule was not selected as an optimum from hundreds of strategies. The small table illustrates timing and splitting, not statistical evidence. Its negative evaluation does not prove that every other possible rule must fail.

Questions the result still cannot answer

Real fills, market depth, costs, data quality, a different market regime and human behaviour during losses remain untested. Twelve invented bars cannot establish reliable future returns or large-loss risk. Published historical indicator events are likewise not a complete trading-system test.

For further research retain a fixed rule version, source sample, parameter-freeze time and a separate unused period. Negative, unfinished and skipped cases belong in the journal too; do not remove them to improve the appearance of a report.

  • What was known before the decision, and what arrived later?
  • How many alternatives were inspected before choosing the rule?
  • Which execution prices and costs were assumed?
  • Which data have not been used for tuning?

Primary sources

Sources explain the mechanics and limitations; the editorial examples use invented numbers rather than market quotations.

Your next practical step

Student Wyckoff Pro helps you study closed bars and save your analysis. It does not execute trades; access and the Windows connection are explained in the guide.

Explore saving your analysis ↗