One of the most useful and astounding sites on the Whole World Wide Web (that's the wwww) is Jamie Gritton's Backtest Engines. Jamie is a Mechanical Investing community member who is a genius with data. He has managed to do something that I would have sworn was impossible -- he's made backtesting both easy and fun! (For the rest of us, anyway.)
Lately, I've been quoting data from the backtester fairly frequently, and I expect it to be a vital part of our strategy selection process for next year's real-money portfolio, so I think it's time that you all learned a bit about how it works. Remember, this is no longer the 15-minutes-a-year portfolio. You're expected to work now!
(If you just tuned in, next year's Foolish Four Portfolio will be expanded to include some of the best strategies from the Foolish Workshop along with the Foolish Four. We will be discussing how to build a diversified portfolio of mechanical strategies over the next several weeks.)
The backtest engine is easier to demonstrate than to explain. So, class, please get out a pad and a nice, sharp number two mouse. I recommend opening a new browser window for Jamie Gritton's TMF Workshop Page. If you arrange your screen so that you can read this article and see the backtest site at the same time, this lesson will be much easier to follow. I promise, it's worth it. You're gonna love it.
First, we are going to use the simplest tester. Click on Standard. That brings up the basic backtest window. Here you can select various time periods, specify which month you would like to start a strategy in, and pick a holding period (from 1 to 24 months for most strategies). For our first example, let's use the Keystone 100, a growth strategy that we've been discussing off and on since February.
For your first run, leave all the defaults in place and simply select Key100 from the drop-down list under "screen." Now, go back up to the top and click Run.
Almost instantly you will get a calculation of the average annual return (CAGR) for Key100 portfolios started each January along with the geometric standard deviation (GSD, a measure of volatility), GSD(M) (monthly volatility), and the Sharpe Ratio (a measure of risk). All of the Key100 numbers are compared to the S&P 500.
Now, scroll down to the yearly returns. Hmmm, we see that a Key100 January portfolio had a loss of 13% in 1990 but was up 24% in the year of the Last Great Crash, 1987. And it lost money in 1997 when the market was going like gangbusters. But, wait! We are looking at a strategy where the stocks were changed every month. Let's compare it with the exact same strategy except that this time we will hold the stocks for 12 months. Make that one change and hit Run again.
Well, that's much better. Last year's shockingly high returns are gone, but so are most of the losing years and the average annual return went from 34% to 40% while the volatility dropped from 46% to 29%. How very interesting. Is it safe to infer that the Key100 strategy is best suited for annual trading?
I wouldn't say yes immediately, because these are just January portfolios. We would want to see how 12-month strategies starting in other months did first. But that's easy as pie. Go ahead and try a few other months. Whoa! Huge differences! In some months the GSD is much higher than the return. Try August. CAGR is 25% and GSD is 51%. Well, that's something I didn't expect, but I'm really glad to have found this out. Let's think about that Key100 some more.
This backtester is a pretty useful tool. If you've been following along, you should be getting a feel for just how useful it can be, but let's look at the returns more closely. Where do those numbers come from? You want to see? Just click the box "List picked stocks," and run it again. There you go. The tester lists each stock selected by the strategy for each year (or month if you are trading monthly, etc.) If you are wondering about the validity of that return of 76% in 1988, you can see exactly which stocks the formula picked and check out how they did for yourself.
Have you been wondering about all this volatility we've been experiencing lately? Is it "normal"? Click the "Show all Months" box and see monthly gains and losses across all those years.
I'm not going to go through every single feature of this amazing tool, because it is very self-explanatory, and you will have fun figuring out some of its more interesting features, but I do want to talk about one other tester. That's the blend tester. Go back to the main page and click Blend.
The Blender lets you design and test a portfolio of strategies. That's what we are going to be doing here between now and the end of the year as we put together our new real-money portfolio. Using the Blend tester, you can see how any given mix of strategies would have performed over the past 14 years.
Now, I don't expect that a great blend over the last 14 years will necessarily be a great blend next year. In fact I would be shocked if its return was close to the tester s return. The value of this tool isn't in predicting the optimum strategy, but in helping you avoid some really bad mixes.
For example, if we ran two different blends and one had an average return of 35% and a standard deviation of 20% and another had an average return of 40% and a standard deviation of 22%, I don't think that tells us much about which one will perform better in the future. You could toss a coin to pick between them.
But if you have two strategies with CAGRs around 30% and one has a GSD of 20% and one has a GSD of 40%, it would certainly be reasonable to go with the blend that has shown lower volatility. Past performance does not predict future performance and all that, but it might give us a better guide than drawing straws.
Let's look at two different blends: The first is composed of five-stock relative strength strategies, 25% each in RS13, RS26, RS4, and RSCAP, all traded every three months (quarterly). The second is more diversified. It's an equal blend of Key100 traded annually, PEG26 traded semiannually, Low Price Book Value (LowPBV) traded annually and RSCAP traded quarterly -- all five-stock strategies.
RS Blend S&P500 Diversified Blend
CAGR 49% 18% 41%
GSD 41% 12% 24%
Sharpe Ratio 0.74 1.05 1.13
First, let's take those CAGRs with a grain of salt. The higher CAGR for the RS blend is going to be seriously impacted by its more frequent trading costs and even the diversified blend's return will be degraded somewhat relative to the S&P 500 since these returns don't include commission or spreads. (Want to see how much difference it makes? Click on Trading Simulator. The answer will be different for portfolios of different sizes and other factors, so we aren't getting into that today.)
Given that we aren't totally comparing apples to apples here, we can still see that the diversified blend is a better deal. Look at the GSD numbers. The relative strength blend is far more volatile, and has a far lower Sharpe Ratio than the more-diversified blend. But the diversified blend has a higher Sharpe Ratio than the S&P 500. That's what we want to look for, a strategy with high returns but acceptable risk. In this case, the Sharpe Ratio is telling us that the diversified blend would provide a better risk-adjusted return than the S&P 500. That's hard to beat.
So here's your assignment, class. Play with the backtester some over the next few days and set up a trading simulator to match your individual situation. Next Thursday we will run some more blends and run the trading simulator for some potential Workshop portfolios. I don't expect we will nail down the Workshop portfolio into its final form, but I do hope that we can come up with some ideas worth talking about on the Workshop discussion board.
Oh, and if you come across any well-diversified blends with CAGRs above 50% and GSDs below 20%, send them to [email protected]. Right away.
Fool on and prosper!
