Two mathematical equations have transformed the world of modern finance. The first was CAPM, the second Black-Scholes. CAPM gave a new perspective on portfolio construction. Black-Scholes gave insight into pricing options and other derivatives. There have been many other advancements in the field of financial optimization, such as Fama-French — but CAPM and Black-Scholes-Merton stand out as perhaps the two most influential.

### Enter Semi-Variance

When CAPM (and MPT) were invented, computers existed, but were very limited. Though the father of CAPM, Harry Markowitz, wanted to use semi-variance, the computers of 1959 were simply inadequate. So Markowitz used variance in his ground breaking book “Portfolio Selection — Efficient Diversification of Investments”.

Choosing variance over semi-variance made the computations orders of magnitude easier, but the were still very taxing to the computers of 1959. Classic covariance-based optimizations are still reasonably compute-intensive when a large number of assets are considered. Classic optimization of a 2000 asset portfolio starts by creating a 2,002,000-entry (technically 2,002,000 unique entries which, when mirrored about the shared diagonal, number 4,000,000) covariance matrix; that is the easy part. The hard part involves optimizing (minimizing) portfolio variance for a range of expected returns. This is often referred to as computing the efficient frontier.

The concept of semi-variance (SV) is very similar to variance used in CAPM. The difference is in the computation. A quick internet search reveals very little data about computing a “semi-covariance matrix”. Such a matrix,** if it existed in the right form,** could possibly allow quick and precise computation of portfolio semi-variance in the same way that a covariance matrix does for computing portfolio variance. Semi-covariance matrices (SMVs) exist, but none “in the right form.” Each form of SVM has strengths and weaknesses. Thus, one of the many problems with semi-covariance matrices is that there is no unique canonical form for a given data set. SVMs of different types only capture an incomplete portion of the information needed for semi-variance optimization.

The beauty of SV is that it measures “downside risk”, exclusively. Variance includes the odd concept of “upside risk”* and penalizes investments for it*. While not going to the extreme of rewarding upside “risk”, the modified semi-variance formula presented in this blog post simply disregards it.

I’m sure most of the readers of this blog understand this modified semi-variance formula. Please indulge me while I touch on some of the finer points. First, the 2 may look a bit out of place. The 2 simply normalizes the value of SV relative to variance (V). Second, the “question mark, colon” notation simply means if the first statement is true use the squared value in summation, else use zero. Third, notice I use r_{i} rather than r_{i} – r_{avg}.

The last point above is intentional and another difference from “mean variance”, or rather “mean semi-variance”. If R is monotonically increasing during for all samples (n intervals, n+1 data points), then SV is zero. I have many reasons for this choice. The primary reason is that with r_{avg} the SV for a straight descending R would be zero. I don’t want a formula that rewards such a performance with 0, the best possible SV score. [Others would substitute T, a usually positive number, as *target return*, sometimes called *minimal acceptable return*.]

Finally, a word about r_{i }— r_{i} is the total return over the interval i. Intervals should be as uniform as possible. I tend to avoid daily intervals due to the non-uniformity introduced by weekends and holidays. Weekly (last closing price of the trading week), monthly (last closing price of the month), and quarterly are significantly more uniform in duration.

### Big Data and Heuristic Algorithms

Innovations in computing and algorithms are how semi-variance equations will change the world of finance. Common sense is why. I’ll explain why heuristic algorithms like Sigma1’s HALO can quickly find near-optimal SV solutions on a common desktop workstation, and even better solutions when leveraging a data center’s resources. And I’ll explain why SV is vastly superior to variance.

Computing SV for a single portfolio of 100 securities is easy on a modern desktop computer. For example 3-year monthly semi-variance requires 3700 multiply-accumulate operations to compute portfolio return, R_{p}, followed by a mere 37 subtractions, 36 multiplies (for squaring), and 36 additions (plus multiplying by 2/n). Any modern computer can perform this computation in the blink of an eye.

Now consider building a 100-security portfolio from scratch. Assume the portfolio is long-only and that any of these securities can have a weight between 0.1% and 90% in steps of 0.1%. Each security has 900 possible weightings. I’ll spare you the math — there are 6.385*10^{138} permutations. Needless to say, this problem cannot be solved by brute force. Further note that if the portfolio is turned into a long-short portfolio, where negative values down to -50% are allowed, the search space explodes to close to 10^{2000}.

I don’t care how big your data center is, a brute force solution is never going to work. This is where heuristic algorithms come into play. Heuristic algorithms are a subset of metaheuristics. In essence heuristic algorithms are algorithms that guide heuristics (or vise versa) to find approximate solution(s) to a complex problem. I prefer the term heuristic algorithm to describe HALO, because in some cases it is hard to say whether a particular line of code is “algorithmic” or “heuristic”, because sometimes the answer is both. For example, semi-variance is computed by an algorithm but is fundamentally a heuristic.

Heuristic Algorithms, HAs, find practical solutions for problems that are too difficult to brute force. They can be configured to look deeper or run faster as desired by the user. Smarter HAs can take advantage of modern computer infrastructure by utilizing multiple threads, multiple cores, and multiple compute servers in parallel. Many, such as HAL0, can provide intermediate solutions as they run far and deep into the solution space.

Let me be blunt — If you’re using Microsoft Excel Solver for portfolio optimization, you’re missing out. Fly me out and let me bring my laptop loaded with HAL0 to crunch your data set — You’ll be glad you did.

### Now For the Fun Part: Why switch to Semi-Variance?

Thanks for reading this far! Would you buy insurance that paid you if your house didn’t burn down? Say you pay $500/year and after 10 years, if your house is still standing, you get $6000. Otherwise you get $0. Ludicrous, right? Or insurance that only “protects” your house from appreciation? Say it pays 50 cents for every dollar make when you resell your house, but if you lose money on the resale you get nothing?

In essence that is what you are doing when you buy (or create) a portfolio optimized for variance. Sure, variance analysis seeks to reduce the downs, but it also penalizes the ups (if they are too rapid). Run the numbers on any portfolio and you’ll see that SV ≠ V. All things equal, the portfolios with SV < V are the better bet. (Note that * classic*_SV ≤ V, because it has a subset of positive numbers added together compared to V).

Let me close with a real-world example. SPLV is an ETF I own. It is based on owning the 100 stocks out of the S&P 500 with the lowest 12-month volatility. It has performed well, and been received well by the ETF marketplace, accumulating over $1.5 billion in AUM. A simple variant of SPLV (which could be called PLSV for PowerShares Low Semi-Variance) would contain the 100 stocks with the least SV. An even better variant would contain the 100 stocks that *in aggregate* produced the lowest SV portfolio over the proceeding 12 months.

HALO has the power to construct such a portfolio. It could solve preserving the relative market-cap ratios of the 100 stocks, picking which 100 stocks are collectively optimal. Or it could produce a re-weighted portfolio that further reduced overall semi-variance.

[Even more information on semi-variance (in its many related forms) can be found here.]