Last Sunday's New York Times carried a short article pointing out the dangers of data mining, but none of its advantages. That's OK. We don't mind. The biggest danger to a properly designed mechanical strategy is too much attention from the Wise.
By
I mentioned the article yesterday, and a couple of readers wrote in asking for a response to Mark Hulbert's comments on the closing of the Beating the Dow newsletter. He used the occasion as an object lesson in the dangers of data mining "in which advisers continually run investment strategies through databases until they find one that has beaten the market."
That is certainly a very bad practice if it isn't accompanied by appropriate safeguards. The problem is that nothing suggests that the Beating the Dow strategy was developed that way. Maybe it was a slow news day.
If you want to read the entire article (it's not very long), click here. You will have to register, but it's quick and free. (If you are as tired of junk e-mail as I am, watch out for the boxes you have to click or unclick to make sure you don't go on endless e-mail lists.)
The part that people wanted me to respond to was in the last half of the article where Hulbert said:
"UNFORTUNATELY, a number of other advisers have yet to learn such lessons. One of the best-known proponents of these strategies is the Motley Fool, the online investment site. It recommends the Foolish Four strategy, whose portfolio construction rules are constantly being tweaked by backtesting. Initially, the strategy worked like this: Start with the Five-Stock System, but eliminate the lowest-priced stock and give double weight to the so-called Penultimate Profit Prospect. In the face of subsequent backtesting, this rule was tweaked to eliminate the double weighting and to add a few other features.
"In response to its disappointing performance, Motley Fool is now searching for yet more construction rules. Ann Coleman, a Motley Fool writer, said the approach was to 'test and tweak and brainstorm and test some more.'
"Sounds like data mining to me."
I guess people expected me to be upset. Several of our staff writers asked how I was taking it... as though a pet had died. When I realized I was supposed to be upset, I searched and searched through my soul for some feeling of having been dissed -- couldn't find it.
The quote he took is from the September 21 Foolish Four report on The Dangers of Mechanical Investing. It's not like he called me up to ask for an opinion. It would have been nice to have an opportunity to explain how some of those "tests" I was talking about are designed to tell if the Foolish Four was just a statistical fluke. We know it was data mined -- the question is whether the mining process found a useful correlation or a random association.
It would also have been nice to add that we consider the data-mining issue very carefully in the Workshop, where new strategies are being developed. Oh, well. Apparently Mark doesn't call people up and ask them for comments. He skewered the Fool a couple years ago, too, without ever talking to us.
I really hate being on Mark Hulbert's bad side, though. He has built a career out of demanding accountability from newsletter publishers -- guys that charge a couple hundred to a couple thousand dollars to send you an eight-page advisory every month telling you what the macroeconomic climate is like and which stocks will prosper in it. He publishes a scorecard for them, and that's a noble undertaking. Almost... Foolish.
You know what my very first reaction to the article was? It was: Good. The last thing we need is for mutual funds and brokerages to start noticing the Workshop strategies. That's one of the reasons mechanical strategies often stop working after discovery. You get billions of dollars chasing a few small stocks at 3:00 p.m. every Friday, and the natural process gets distorted.
So thanks, Mark, for pooh-poohing the idea of data mining. We'll keep digging.
Fool on and prosper!