Good article, well articulated:
http://www.philosophicaleconomics.com/2 ... cktesting/
Financial Backtesting
Moderator: Global Moderator
-
Jack Jones
- Executive Member
- Posts: 755
- Joined: Mon Aug 24, 2015 3:12 pm
Re: Financial Backtesting
He makes a lot of good points about interpreting data. One should never put all their eggs in the backtesting basket, especially when they do not understand the assumptions involved.
He loses me at testing the system in independent international markets after arguing that one must fully understand why an investment worked. International markets fundamentally act very differently than the US, so trying to directly translate results is silly. But then I realized that he's approaching this from a theoretical quant mindset where outperformance is achieved from a trading system independent of the markets themselves. Notice his example is based solely on automated buy/sell signals and not on any kind of market diversification or economic theory. It's also why he prioritizes recent data -- he's looking for a near-term edge, not long-term risk mitigation.
Fundamentally, he just has very different goals than I do:
So while I agree that backtesting results must be properly framed and that the assumptions are always way more important than many people realize, I think it's more valuable for certain types of investors than he concludes. Just because it's problematic for his personal goals does not mean it's similarly unhelpful for others.
He loses me at testing the system in independent international markets after arguing that one must fully understand why an investment worked. International markets fundamentally act very differently than the US, so trying to directly translate results is silly. But then I realized that he's approaching this from a theoretical quant mindset where outperformance is achieved from a trading system independent of the markets themselves. Notice his example is based solely on automated buy/sell signals and not on any kind of market diversification or economic theory. It's also why he prioritizes recent data -- he's looking for a near-term edge, not long-term risk mitigation.
Fundamentally, he just has very different goals than I do:
His critique of backtesting is framed in the mindset of identifying a trading system that guarantees near-term outperformance superior to his own active trading instincts. Using the above quoted criteria, he'll abandon any investment system that does not "deliver success" (however he chooses to define that) in the short term. That's not a particularly healthy investing mindset, either.If a strategy can’t deliver success on a near-term basis, in the out-of-sample test that reality is putting it through, then I’d rather just abandon the systematic approach altogether and invest on my own concrete analysis of the situation, my own gut feel for where it is likely headed, given the present facts. If I don’t have confidence in my own analysis, if I can’t trust my gut, then I shouldn’t be actively investing.
So while I agree that backtesting results must be properly framed and that the assumptions are always way more important than many people realize, I think it's more valuable for certain types of investors than he concludes. Just because it's problematic for his personal goals does not mean it's similarly unhelpful for others.