Count the complete search
A trial includes more than a named strategy. Changing lookback length, entry threshold, universe, date range or exit filter creates additional opportunities to select a favourable outcome. Combining the best components after examining their results also belongs in the research search history.
Keep a registry of every candidate and its selection rule. Reserve data for evaluation before choosing the winner, and avoid revising the method after inspecting that evaluation. Statistical adjustments can help, but they require defensible assumptions about the tests and do not establish that trading costs or data quality are realistic.
Worked example: twenty chances at 5%
Assume, purely for illustration, that 20 independent tests each have a 5% false-positive probability under a world with no genuine effect. The probability of at least one false positive is 1 − 0.95²⁰ ≈ 64.15%. With 100 independent tests, it is approximately 99.41%.
The 5% input is a statistical test threshold, not a strategy win rate or a promised probability of profit. Real strategy variants are usually correlated, so these numbers cannot be applied mechanically. They demonstrate the mechanism: a small per-test error can become a large search-wide error when many chances are offered.
A reused holdout stops being independent
An untouched holdout helps only while it remains untouched. Repeatedly checking it and changing the strategy turns it into development data. A smooth parameter neighbourhood, an economic explanation and results across several conditions are useful supporting evidence, but none erases an undisclosed search through thousands of alternatives.
Predeclare one chart-review rule
For a chart exercise on available closed daily, weekly or monthly bars, prewrite one setup definition and the period to inspect. Record every qualifying example, including failures, before trying a revised filter. Student Wyckoff Pro can support this price-structure review; it does not by itself measure statistical significance or validate a strategy search.
Primary sources
Sources explain the mechanics and limitations; the editorial examples use invented numbers rather than market quotations.
Your next practical step
Explore the available closed daily, weekly and monthly bars in the analysis workspace.
Open the first-chart instructions ↗