The best models are not the models that fit past data the best, they are the models that predict new data the best. This seems obvious, but a surprising number of business and financial decisions are based on best-fit of past data, with no idea of how well they are expected to correctly model future data.
Instant Profit, or Too Good to be True?
For instance, a stock analyst reports to you that they have a secret recipe to make 70% annualized returns by simply trading KO (The Coca-Cola Company). The analyst’s model tells what FOK limit price, y, to buy KO stock at each market open. The stock is then always sold with a market order at the end of each trading day.
The analyst tells you that her model is based on three years of trading data for KO, PEP, the S&P 500 index, aluminum and corn spot prices. Specifically, the analyst’s model uses closing data for the two preceding days, thus the model has 10 inputs. Back testing of the model shows that it would have produced 70% annualized returns over the past three years, or a whooping 391% total return over that time period. Moreover, the analyst points out that over 756 trading days 217 trades would have been executed, resulting in profit a 73% of the time (that the stock is bought).
The analyst, Debra, says that the trading algorithm is already coded, and U.S. markets open in 20 minutes. Instant profit is only moments away with a simple “yes.” What do you do with this information?
Choices, Chances, Risks and Rewards
You know this analyst and she has made your firm’s clients and proprietary trading desks a lot of money. However you also know that, while she is thorough and meticulous; she is also bold and aggressive. You decide that caution is called for, and allocate a modest $500,000 to the KO trading experiment. If after three months, the KO experiment nets at least 7% profit, you’ll raise the risk pool to $2,000,000. If, after another three months, the KO-experiment generates at least 7% again; you’ll raise the risk pool to $10,000,000 as well as letting your firms best clients in on the action.
Three months pass, and the KO-experiment produces good results: 17 trades, 13 winners, and a 10.3% net profit. You OK raising the risk pool to $2,000,000. After only 2 months the KO-experiment has executed 13 trades, with 10 winners, and a 11.4% net profit. There is a buzz around the office about the “knock-out cola trade”, and brokers are itching to get in on it with client funds. You are considering giving the green light to the “Full Monty,” when Stan the Statistician walks into your office.
Stan’s title is “Risk Manager”, but people around the office call him Stan the Statistician, or Stan the Stats Man, or worse (e.g. “Who is the SS going to s*** on today?”) He’s actually a nice guy, but most folks consider him an interloper. And Stan seems to have clout with corporate, and he has been known to use it to shut down trades. You actually like Stan, but you already know why he is stopping by.
Stan begins probing about the KO-trade. He asks what you know. You respond that Debra told you that the model has an R-squared of 0.92 based on 756 days of back-tested data. “And now?” asks Stan. You answer, “a 76% success rate, and profits of around 21% in 5 months.” And then Stan asks, “What is the probability that that profit is essentially due to pure chance?”
You know that the S&P 500 historically has over 53% “up” days, call it 54% to be conservative. So stocks should follow suit. To get exactly 23 wins on KO out of 30 tries is C(30, 23)*0.54^23*(0.46)^7 = 0.62%. To get at least 23 (23 or more wins) brings the percentage up to about 0.91%. So you say 1/0.091 or about one in 110.
Stan says, “Your math is right, but your conclusion is wrong. For one thing, KO is up 28% over the period, and has had 69% up days over that time.” You interject, “Okay, wait one second… so my math now says about 23%, or about a 1 in 4.3 chance.”
Stan smiles, “You are getting much closer to the heart of the matter. I’ve gone over Debra’s original analysis, and have made some adjustments. My revised analysis shows that there is a reasonable chance that her model captures some predictive insight that provides positive alpha.” Stan’s expression turns more neutral, “However, the confidence intervals against the simple null hypothesis are not as high as I’d like to see for a big risk allocation.”
Getting all Mathy? Feedback Requested!
Do you want to hear more from “Stan”? He is ready to talk about adjusted R-squared, block-wise cross-validation, and data over-fitting. And why Debra’s analysis, while correct, was also incomplete. Please let me know if you are interested in hearing more on this topic.
Please let me know if I have made any math errors yet (other than the overtly deliberate ones). I love to be corrected, because I want to make Sigma1 content as useful and accurate as possible.
Balhiser Financial 