Mfjohnso44 posted a message, on both the Foolish Workshop board and the Foolish Four board on Dec. 30, 2000, analyzing 266 daily Foolish Four and BSP portfolios. The author obviously invested considerable effort in collecting data meticulously for over a year. He put together a very detailed and readable presentation, covering almost every possible angle in the data. And it was all presented with the utmost objectivity � nothing to prove, no hidden agenda, just the facts ma'am. What a professional job! To my surprise there was absolutely no response on the Foolish Four board.
The data mfjohnso44 presented was collected starting December 10, 1999. Of course, it wouldn't be enough to validate or invalidate the strategies he was tracking. The time period studied is too short and the portfolios overlap quite a bit. That's not the point, though.
I dug into the spreadsheets (links are provided near the end of the message) to observe the returns of the 12 daily portfolios that had completed their full annual cycle. Here are the returns of those 12 Foolish Four portfolios:
Started F4-RP12/10/99 -23.3%
12/13/99 -22.3%
12/14/99 -23.5%
12/15/99 -7.8%
12/16/99 -8.5%
12/17/99 -6.9%
12/20/99 -5.4%
12/21/99 -3.8%
12/22/99 -3.0%
12/23/99 -5.5%
12/27/99 -14.1%
12/28/99 -15.5%
There is a striking difference in annual returns, divided into three groups that reflect different stock selections by the Foolish Four. All it took was one stock substitution between 12/14 and 12/15, and another one between 12/23 and 12/27 (over the Christmas weekend) to significantly change the strategy's return.
This change in returns, which I believe is par for the course, is in my view the most important observation from mfjohnso44's work. Anyone who invested in the Foolish Four last year, anyone contemplating such an investment, should pay close attention to the variation in returns. Not only are your returns unlikely to match the backtest, they are also likely to be different from those of someone who started out one day earlier or one day later than you. This is true of the Foolish Four, and it may be even truer of other, more volatile, mechanical strategies.
Does this kind of variability mean that mechanical strategies are useless? Well, certainly they would be if one expected to come close to the average return of the backtest in one's first year. Theoretically, these variations should smooth out over time. But how many people will have the patience to wait for that, especially if their first experience with a strategy was in 2000?
The other thought rattling in my brain after reading mfjohnso44's message was, what if we had such daily results for all 30+ years of backtesting? The danger of data mining, in my opinion, is not so much in the possibility that we picked the best strategy from among some unknown set of random strategies. That, to me, is a theoretical and largely unresolvable issue. The more tangible data mining problem is that we may have selected a starting date for the Foolish Four that was exceptionally lucky. Indeed, we seemed to do so with the January start. When we looked at 11 other monthly starts the returns were lower; in fact, portfolios renewed in the summer months barely beat the indices and would not have been statistically significant. But here we have the potential of looking at all 250 possible starting days in the year and figuring out for sure whether our current results were just a statistical fluke.
I expect that these ideas may raise howls from some. It's just a thought about another, seemingly overlooked dimension of data mining. I doubt that the energy remains in anyone to pursue this idea. Perhaps it will come about some day when our computers become smart enough that we can say � computer, tell me what the Foolish Four would have done at every possible starting day, and it would be done. Until then, I'll be left with my musings.
Foolishly Yours,
Elan
