Semi-variance: Choosing the Best Formula

Unlike variance, there a several different formulas for semivariance (SV).  If you are a college student looking to get the “right” answer on test or quiz, the formula you are looking for is most likely:

Classic Semi-Variance Formula

The question-mark-colon syntax simply means if the expression before the “?” is true then the term before the “:” is used, otherwise the term after the “:” is used.  So a?b:c simply means chose b if a is true, else chose c.  This syntax is widely used in computer science, but less often in the math department.  However, I find it more concise than other formulations.

Another common semivariance formula involves comparing returns to a required minimum threshold r_t.  This is simply:

Min Return Threshold SV

Classic mean-return semivariance should not be directly compared to mean-return variance.  However a slight modification makes direct comparison more meaningful.  In general approximately half of mean-adjusted returns are positive and half are negative (exactly zero is a relatively rare event and has no impact to either formula).  While mean-variance always has n terms, semi-variance only uses a subset which is typically of size n/2.  Thus including a factor of 2 in the formula makes intuitive sense:

Modified Semi-Variance

Finally, another useful formulation is one I call “Modified Drawdown Only” (MDO) semivariance.  The name is self-explanatory… only drawdown events are counted.  SV_mdo does not require r_avg (r bar) nor r_t.  It produces nearly identical values to SV_mod for rapid sampling (say for anything more frequent than daily data).  For high-speed trading it also has the advantage of not requiring all of the return data a priori, meaning it can be computed as each return data point becomes available, rather than retrospectively.

Modified Drawdown-Only Semi-variance

Why might  SV_mdo be useful in high-speed trading?  One use may be in put/call option pricing arbitrage strategies.  Black–Scholes, to my knowledge, makes no distinction between “up-side” and “down-side” variance, and simply uses plain variance. [Please shout a comment at me if I am mistaken!]    However if put and call options are “correctly” priced according to Black–Scholes, but the data shows a pattern of, say, greater downside variance than normal variance on the underlying security, put options may be undervalued.  This is just an off-the-cuff example, but it illustrates a potential situation for which SV_mdo is best suited.

Pick Your Favorite Risk Measure

Personally, I slightly favor SV_mdo over SV_mod for computational reasons. They are often quite similar in practice, especially when used to rank risk profiles of a set of candidate portfolios. (The fact that both are anagrams of each other is deliberate.)

I realize that the inclusion of the factor 2 is really just a semantic choice.  Since V and (classic) SV, amortized over many data sets, are expected to differ by a factor of 2, standard deviation, σ,  and semideviation, σ_d, can be expected to differ by the square root of 2.  I consider this mathematically untidy.  Conversely, I consider SV_mod to be the most elegant formulation.

Advertisement

The Best Financial Models for Insight and Prediction?

The best models are not the models that fit past data the best, they are the models that predict new data the best. This seems obvious, but a surprising number of business and financial decisions are based on best-fit of past data, with no idea of how well they are expected to correctly model future data.

Instant Profit, or Too Good to be True?

For instance, a stock analyst reports to you that they have a secret recipe to make 70% annualized returns by simply trading KO (The Coca-Cola Company).  The analyst’s model tells what FOK limit price, y, to buy KO stock at each market open.  The stock is then always sold with a market order at the end of each trading day.

The analyst tells you that her model is based on three years of trading data for KO, PEP, the S&P 500 index, aluminum and corn spot prices.  Specifically, the analyst’s model uses closing data for the two preceding days, thus the model has 10 inputs.  Back testing of the model shows that it would have produced 70% annualized returns over the past three years, or a whooping 391% total return over that time period.  Moreover, the analyst points out that over 756 trading days 217 trades would have been executed, resulting in profit a 73% of the time (that the stock is bought).

The analyst, Debra, says that the trading algorithm is already coded, and U.S. markets open in 20 minutes. Instant profit is only moments away with a simple “yes.” What do you do with this information?

Choices, Chances, Risks and Rewards

You know this analyst and she has made your firm’s clients and proprietary trading desks a lot of money. However you also know that, while she is thorough and meticulous; she is also bold and aggressive. You decide that caution is called for, and allocate a modest $500,000 to the KO trading experiment.  If after three months, the KO experiment nets at least 7% profit, you’ll raise the risk pool to $2,000,000.  If, after another three months, the KO-experiment generates at least 7% again; you’ll raise the risk pool to $10,000,000 as well as letting your firms best clients in on the action.

Three months pass, and the KO-experiment produces good results: 17 trades, 13 winners, and a 10.3% net profit. You OK raising the risk pool to $2,000,000.  After only 2 months the KO-experiment has executed 13 trades, with 10 winners, and a 11.4% net profit.  There is a buzz around the office about the “knock-out cola trade”, and brokers are itching to get in on it with client funds. You are considering giving the green light to the “Full Monty,” when Stan the Statistician walks into your office.

Stan’s title is “Risk Manager”, but people around the office call him Stan the Statistician, or Stan the Stats Man, or worse (e.g. “Who is the SS going to s*** on today?”)  He’s actually a nice guy, but most folks consider him an interloper.  And Stan seems to have clout with corporate, and he has been known to use it to shut down trades. You actually like Stan, but you already know why he is stopping by.

Stan begins probing about the KO-trade.  He asks what you know.  You respond that Debra told you that the model has an R-squared of 0.92 based on 756 days of back-tested data.  “And now?” asks Stan.  You answer, “a 76% success rate, and profits of around 21% in 5 months.”  And then Stan asks, “What is the probability that that profit is essentially due to pure chance?”

You know that the S&P 500 historically has over 53% “up” days, call it 54% to be conservative. So stocks should follow suit.  To get exactly 23 wins on KO out of 30 tries is C(30, 23)*0.54^23*(0.46)^7 = 0.62%. To get at least 23 (23 or more wins) brings the percentage up to about 0.91%.  So you say 1/0.091 or about one in 110.

Stan says, “Your math is right, but your conclusion is wrong.  For one thing, KO is up 28% over the period, and has had 69% up days over that time.”  You interject, “Okay, wait one second… so my math now says about 23%, or about a 1 in 4.3 chance.”

Stan smiles, “You are getting much closer to the heart of the matter. I’ve gone over Debra’s original analysis, and have made some adjustments. My revised analysis shows that  there is a reasonable chance that her model captures some predictive insight that provides positive alpha.”  Stan’s expression turns more neutral, “However, the confidence intervals against the simple null hypothesis are not as high as I’d like to see for a big risk allocation.”

Getting all Mathy? Feedback Requested!

Do you want to hear more from “Stan”? He is ready to talk about adjusted R-squared, block-wise cross-validation, and data over-fitting. And why Debra’s analysis, while correct, was also incomplete. Please let me know if you are interested in hearing more on this topic.

Please let me know if I have made any math errors yet (other than the overtly deliberate ones).  I love to be corrected, because I want to make Sigma1 content as useful and accurate as possible.

The Future of Investing is Automation

A significant and growing portion of today’s individual investors have never placed a trade using a human stock broker.

Developing an Automation Mindset for Investing

In 2010, I bought the domain name Sigma1.com with the idea of creating an hedge fund that I would manage.  In order to measure and manage my investment strategies objectively, I began thinking about benchmarks and financial analysis software.  And as I ran scenarios through Excel and some light-weight analysis software I created, I began to realize that analysis, by itself was very limited.  I could only back-test one portfolio at a time, and I had to construct each portfolio’s asset weights manually.  It soon became obvious that I needed portfolio optimization software.

I learned that portfolio optimization software with the capabilities I wanted was extremely expensive. Further, I realized that even if, say, I negotiated a deal with MSCI where they provided Sigma1 Financial with their Barra Portfolio Manager for free, it would not differentiate a Sigma1 hedge fund from other hedge funds using the same software.

I was beginning to interact with several technology entrepreneurs and angel investors.  I quickly learned that legal costs and barriers to entry for a new hedge were intractable.  If Sigma1 attracted $10M in assets from accredited investors in 12 months, and charged 2 and 20, it would be a money loosing enterprise.  Cursory research revealed that critical mass for a profitable (for the hedge fund managers) hedge fund could be as high as $500M.  Luckily, I had learned about the concept of the “entrepreneurial pivot“.

The specific pivots Sigma1 used were a market segment pivot followed by a technology pivot. I realized that while the high cost of good portfolio optimization software is bad for a hedge fund startup, it was great for a financial software startup.  Suddenly, the Sigma1 Financial target market switched from accredited investors to financial professionals (investment managers, fund managers, proprietary traders, etc).  This was a key market segment pivot.

Just creating a cheaper portfolio optimizer seemed unlikely to provide sufficient incentive to displace entrenched portfolio optimizers. Sigma1 needed a technology pivot — finding a solution using a completely different technology.  Most prior portfolio optimizers use some variant of linear programming (LP) [or QP or NLP] to help find optimal portfolios. Moreover they also create an asset covariance matrix as a starting point for the optimization.

One stormy day, I realized that some algorithms I created to solve statistical electrical engineering problems in grad school could be adapted to optimize investment portfolios. The method I devised not only avoided LP, QP, or NLP methods; it also dispensed with the need for a covariance matrix.  Over then next several days I realized that by eliminating dependence on a covariance matrix, the algorithm I later named HALO, could use both traditional and alternate risk measures ranging from variance-based (eg. standard-deviation of return) to covariance-based ones (e.g. beta) to semivariance to max draw down.  By developing a vastly different technology, HALO could optimize for risks such as semivariance and Sortino ratios, or max drawdown, or even custom risk measures devised by the client.

Algorithms Everywhere

Long before Sigma1 began developing HALO, the financial industry has been increasingly reliant on digital systems and various financial algorithms. As digital communication networks and electronic stock exchanges gained trading volume, various forms of program trading began to flourish.  This includes the often maligned high-frequency trading variant of automated trading.

Concurrently, more and more trading volume has gone online.  A significant portion of today’s individual investors have never placed a trade using a human stock broker.

Automated Investment Advice, Analysis, and Trading

There are now numerous automated investment analysis tools, many of which come free with a brokerage account, while others are free or low-cost stand-alone online tools.  Examples of the former include the Fidelity’s nascent GPS (Guided Portfolio Summary) to more seasoned offerings such as Financial Engines.  Online portfolio analysis offering range from Morningstar’s Instant X-Ray, to sites like ETFreplay.

However these software offerings are just the beginning. A company call FutureAdvisor has partnered with Fidelity and TD Ameritrade to allow its automate portfolio software to make trades on its users behalf. Companies like Future Advisor have the potential to help small investors benefit from custom-tailored investment advice utilizing proven academic research (e.g. Fama French) at a very low cost — costs so low that they would not be profitable for human investment advisers to provide.

If successful (and I believe some automated investment companies will be), why should they stop at small-time investors, with less than $500,000 in investable assets?  Why not $1,000,000 or more?  Nothing should stop them!

I could easily imagine Mark Zuckerberg, Sergey Brin, or Larry Page utilizing an automated investment company’s software to manage a large part of their portfolios.  If we, as a society, are considering allowing automated systems to drive our cars for us, surely they can also manage our investment portfolios.

The Future Roll of the Human Financial Adviser

There will always be some percentage of investors who want a personal relationship with a financial adviser. Human investment advisers can excel at explaining investment concepts and putting investors at ease during market corrections.  In some ways human investment advisers even function as personal financial counselors, listening to their clients emotional financial stories.  And, of course, there are some people who want to be able to pick up the phone and yell at a real person for letting them suffer market losses.  Finally, there are people with Luddite tenancies who want as little to do with technology as possible.  For all these reasons human investment advisers will have a place in the future world of finance.

Investment Automation will Accelerate

There are some clear trends in the investing world.  Index investing will continue to grow, as will total ETF assets under management (AUM). Alternative investments from rental property to master limited partnerships (MLPs) to private equity are also likely to become part of the portfolios of more sophisticated and affluent investors.

With the exception of high-frequency trading, which has probably saturated arbitrage and front-running opportunities, I expect algorithmic (algo) management to increase as an overall percentage of US and global AUM. Some algorithmic trading and investing will be of the “hardwired” variety where the algo directly connects to the exchanges and makes trades, while the rest of the algo umbrella will comprise trading and investing decisions made by financial software and entered manually by humans with minimal revision.  There will also be hybrid methods where investment decisions are a synthesis of “automated” and “manual” processes.  I expect the scope of these “flavors” of automated investing to not only increase, but to accelerate in the near term.

It is important to note, however, that for the foreseeable future, the ultimate arbiters of algorithmic investing and portfolio optimization will be human.  The software architects and developers will exercise significant influence on the methodology behind the fund and portfolio optimization software.  Furthermore, the users of the software will have supreme control over what parameters go into the optimization process such as including or excluding or bounding certain assets and asset classes (amongst many other factors under their direct control).

That being said, the future of investing will be increasingly the domain of financial engineers, software developers and testers, and people with skills in financial mathematics, statistics, algorithms, data structures, GUIs, web interfaces and usability. Additionally, the financial software automation revolution will have profound impacts on legal professionals and marketers in the financial domain, as well as more modest impacts on accountants and IT professionals.

Some financial professionals will take the initiative and find a place on the leading edge of the financial automation revolution. It is likely to be a wild but lucrative ride. Others will seek the short-term comfort of tradition. They may be able to retain many of their current clients through sheer charisma and inertia, but may find it increasingly difficult the appeal to younger affluent clients steeped in a culture of technology.

HALO Portfolio Software Availability

We’ve received several requests from individuals about obtaining access to HAL0 Portfolio-Optimization software. We’re very appreciative of the interest!

In the long-term Sigma1 Financial might consider a SaaS (software as a service) model for individuals. Currently, however, Sigma1 Financial is only pursuing select relationships with financial professionals.

If you are an financial professional interested in HALO software, please use this Contact Form to contact us.  Also if you are a web software developer or business manager with experience in sales to financial companies who wishes to learn more about opportunities with Sigma1 Financial, please contact us as well.

Regardless of your financial background, we welcome comments and questions from our readers.  We use a lot of math in this blog and naturally make a few mistakes on occasion.  We love to get feedback on any mathematical typos, oversights, or just plain errors; and we strive to correct them as quickly as possible.

Pursuing Alpha with Antivariance

A simple and marginally-effective strategy to reduce portfolio variance is by constructing an asset correlation matrix, selecting assets with low (preferably negative) correlations, and building a portfolio of low-correlation assets.  This basic strategy involves creating a set of assets whose cross-correlations (covariances) are minimized.

One reason this basic  strategy is only somewhat effective is that a correlation matrix (or covariance matrix) only provides a partial picture of the chosen investment landscape.  Some fundamental limitations include non-normal distributions, skewness, and kurtosis to name a few.  To most readers these are fancy words with varying degrees of meaning.

Personally, I often find the mathematics of the work I do seductive like a Siren’s song.  I endeavor to strike a balance between exploring tangential mathematical constructs, and keeping most of my math applied. One mental antidote to the Siren’s song of pure mathematics is to think more conceptually than mathematically by asking questions like:

What are the goals of portfolio optimization?  What elements of the investing landscape allow these goals to be achieved?

I then attempt to answer these questions with explanations that a person with a college degree but without a mathematically background beyond algebra could understand.  This approach lets me define the concept first, and develop the math later.  In essence I can temporarily free my mind of the slow, system 2 thinking generally required for math.

Recently, I came up with the concept of antivariance.  I’m sure others have had similar ideas and a cursory web search reveals that as profession poker player’s nickname.  I will layout my concept of antivariance as it relates to porfolio theory in particular and the broader concept in general.

By convention, one of the key objective of modern portfolio theory is the reduction of portfolio return variance.  The mathematical concept is the idea that by combining assets with correlations of less than 1.0, the return variance is less than the weighted sums of each asset’s individual variance.

Antivariance assumes that there are underlying patterns explain why two or more assets should be somewhat less correlated (independent), but at times negatively correlated.  Consider the affects of major hurricanes like Andrew or Katrina.  Their effects were negative for insurance companies with large exposures, but were arguably positive for companies that manufactured and supplied building materials used in the subsequent rebuilds.  I mention Andrew because there was much more and more rapid rebuilding following Andrew than Katrina.  The disparate groups of stocks of (regional) insurance versus construction companies can be considered to exhibit paired antivariance to devastating weather events.

Nicholas Nassim Taleb coined the the term antifragile, because terms such as robust simply don’t convey the exact mental connections.  I am beginning to use the term antivariance because it conveys concepts not well captured by terms like “negatively correlated”, “less correlated”, “semi-independent”, etc.   In many respects antifragile systems should exhibit antivariance characteristics, and vice versa.

The concept of antivariance can be extended to related concepts such as anticovariance and anticorrelation.

 

 

Software Development Choices for Portfolio Optimization

The first phase of developing the HALO (Heuristic Algorithm Optimizer) Portfolio Optimizer was testing mathematical and heuristic concepts.  The second phase was teaming up with beta partners in the financial industry to exchange optimization work for feedback on the optimizer features and results.

For the first phase, my primary tool for software development was the Ruby language.  Because Ruby is a “high-level” extensible language I was able to quickly prototype and test many diverse and complex concepts.  This software development process is sometimes referred to as software prototyping.

For the second, beta phase of software development I kept most of the software in Ruby, but began re-implementing selected portions of the code in C/C++. The goal was to keep the high-change-rate code in Ruby, while coding the more stable portions in C/C++ for run-time improvement.  While a good idea in theory, it turned out that my ability to foresee beta-partner changes was mixed at best.  While many changes hit the the Ruby code, and were easily implemented, a significant fraction hit deep into the C/C++ code, requiring significant development and debugging effort.  In some cases, the C/C++ effort was so high, I switched back portions of the code to Ruby for rapid development and ease of debugging.

Now that the limited-beta period is nearly complete, software development has entered a third phase: run-time-performance optimization.  This process involves converting the vast majority of Ruby code to C.  Notice, I specifically say C, not C/C++.   In phase 2, I was surprised at the vast increase in executable code size with C++ (and STL and Boost).  As an experiment I pruned test sections of code down to pure C and saw the binary (and in-memory) machine code size decrease by 10X and more.

By carefully coding in pure C, smaller binaries were produced, allowing more of the key code to reside in the L1 and L2 caches.  Moreover, because C allows very precise control over memory allocation, reallocation, and de-allocation, I was able to more-or-less ensure than key data resided primarily in the L1 and/or L2 caches as well.  When both data and instructions live close to the CPU in cache memory, performance skyrockets.

HALO code is very modular, meaning that it is carefully partitioned into independent functional pieces.  It is very difficult, and not worth the effort, to convert part of a module from Ruby to C — it is more of an all-or-nothing process.  So when I finished converting another entire module to C today, I was eager to see the result.  I was blown away.  The speed-up was 188X.  That’s right, almost 200 times faster.

A purely C implementation has its advantages.  C is extremely close to the hardware without being tied directly to any particular hardware implementation.   This enables C code (with the help of a good compiler) to benefit from specific hardware advantages on any particular platform.  Pure C code, if written carefully, is also very portable — meaning it can be ported to a variety of different OS and hardware platforms with relative ease.

A pure C implementation has disadvantages.  Some include susceptibility to pointer errors, buffer-overflow errors, and memory leaks as a few examples.  Many of these drawbacks can be mitigated by software regression testing, particularly to a “golden” reference spec coded in a different software language.  In the case of HALO Portfolio-Optimization Software, the golden reference spec is the Ruby implementation.  Furthermore unit testing can be combined with regression testing to provide even better software test coverage and “bug” isolation.  The latest 188X speedup was tested against a Ruby unit test regression suite and proven to be identical (within five or more significant digits of precision) to the Ruby implementation.  Since the Ruby and C implementations were coded months apart, in different software languages, it is very unlikely that the same software “bug” was independently implemented in each.  Thus the C helps validate the “golden” Ruby spec, and vice versa.

I have written before about how faster software is greener software.  At the time HALO was primarily a Ruby implementation, and I expected about a 10X speed up for converting from Ruby to C/C++.  Now I am increasingly confident that an overall 100X speedup for an all C implementation is quite achievable.  For the SaaS (software as a service) implementation, I plan to continue to use Ruby (and possibly some PHP and/or Python) for the web-interface code.  However, I am hopeful I can create a pure C implementation of the entire number-crunch software stack.  The current plan is to use the right tool for the right job:  C for pure speed, Ruby for prototyping and as a golden regression reference, and Ruby/PHP/Python/etc for their web-integration capabilities.

 

Principles of Portfolio Optimization Software

Explaining technical investment concepts in a non-technical way is critical to having a meaningful dialog with individual investors.  Most individual investors (also called “retail investors”, or “small investors”) do not have the time nor the desire to learn the jargon and concepts behind building a solid investment portfolio.  This is generally true for most individual investors regardless of the size of their investment portfolios.  Individual investors expect investment professionals (also called “institutional investors”) to help manage their portfolios and explain the major investment decisions behind the management of their individual portfolios.

In the same way that a good doctor helps her patient make informed medical decisions, a good investment adviser helps her clients make informed investment decisions.

I get routinely asked how the HALO Portfolio Optimizer works.  Every time I answer that question, I face two risks: 1) that I don’t provide enough information to convince the investment profession or their clients that HALO optimization provides significant value and risk-mitigation capability and 2) I risk sharing key intellectual property (IP) unique to the Sigma1 Financial HALO optimizer.

This post is my best effort to provide both investment advisers and their clients with enough information to evaluate and understand HALO optimization, while avoiding sharing key Sigma1 trade secrets and intelectual property.  I would very much appreciate feedback, both positive and negative, as to whether I have achieved these goals.

First Principle of Portfolio Optimization Software

Once when J.P. Morgan was asked what the market would do, he answered “It will fluctuate.”  While some might find this answer rather flippant, I find it extremely insightful.  It turns out that so-called modern portfolio theory (MPT) is based understanding (or quantifying) market fluctuations. MPT labels these fluctuations as “risk” and identifies “return” as the reward that a rational investor is willing to accept for a given amount of risk.  MPT assumes that a rational investor, or his/her investment adviser will diversify away most or all “diversifiable risk” by creating a suitable investment portfolio tailored to the investor’s current “risk tolerance.”

In other words, the primary job of the investment adviser (in a “fiduciary” role), is to maximize investment portfolio return for a client’s acceptable risk.  Said yet another way, the job is to maximize the risk/reward ratio for the client, without incurring excess risk.

Now for the first principle: past asset “risk” tends to indicate future asset “risk”.  In general an asset that has been previously more volatile will tend to remain more volatile, and and asset that has been less volatile will tend to remain less volatile.  Commonly, both academia and professional investors have equated volatility with risk.

Second Principle of Portfolio Optimization Software

The Second Principle is closely related to the first.  The idea is that the past portfolio volatility tends to indicate future portfolio volatility. This thesis is so prevalent that it is almost inherently assumed.  This is evidenced by search results that reaches beyond volatility and looks at the hysteresis of return-versus-volatility ratios, papers such at this.

Past Performance is Not Necessarily Indicative of Future Results.

Third Principle of Portfolio Optimization Software

The benefits of diversification are manifest in risk mitigation.  If two assets are imperfectly correlated, then their combined volatility (risk) will be less than the weighted averages of their individual volatilities.  An in-depth mathematical description two-asset portfolio volatilities can be found on William Sharpe’s web page.  Two-asset mean-variance optimization is relatively simple, and can be performed with relatively few floating-point operations on a computer.  This process creates the two-asset efficient frontier*.  As more assets are added to the mix, the computational demand to find the optimal efficient frontier grows geometrically, if you don’t immediately see why look at page 8 of this paper.

A much simpler explanation of the the third principle is as follows.  If asset A has annual standard deviation of 10%, and asset B an annual standard deviation of 20%, and A and B are not perfectly correlated, then the portfolio of one half invested in A and the other half invested in B will have a annual standard deviation of less than 15%.  (Non-perfectly correlated means a correlation of less than 1.0).  Some example correlations of assets can be found here.

In so-called plain English, the Third Principle of Portfolio Optimization can be stated: “For a given level of expected return, portfolio optimization software can reduce portfolio risk by utilizing the fact that different assets move somewhat independently from each other.”

Forth Principle of Portfolio Optimization Software

The Forth Principle of Portfolio Optimization establishes a relationship between risk and return.  The classic assumption of modern portfolio theory (MPT) is that so-called systematic risk is rewarded (over a long-enough time horizon) with increased returns.  Portfolio-optimization software seeks to reduce or eliminate unsystematic risk when creating an optimized set of portfolios.  The portfolio manager can thus select one of these optimized portfolios from the “best-in-breed” list created by the optimization software that is best suited to his/her client’s needs.

Fifth Principle of Portfolio Optimization Software

The 5th Principle is that the portfolio manager and his team adds value to the portfolio composition process by 1) selecting a robust mix of assets, 2) applying constraints to the weights of said assets and asset-groups, and 3) assigning expected returns to each asset.  The 5th Principle focuses on the assignment of expected returns.  This  process can be grouped under the category of investment analysis or investment research.  Investment firms pay good money for either in-house or contracted investment analysis of selected securities.

Applying the Portfolio Optimization Principles Together

Sigma1 Financial HALO Software applies these five principles together to help portfolio managers improve or fine-tune their proprietary-trading and/or client investment portfolios.  HALO Portfolio Optimization software utilizes the assets, constraints, and expected returns from the 5th Principal as a starting point.  It then uses the 4th Principal by optimizing away systematic risk from a set of portfolios by taking maximum advantage of varying degrees of non-correlation of the portfolio assets.  The 3rd Principle alludes to the computational difficulty of solving the multi-asset optimization problem.  Principles 1 and 2 form the bedrock of the concepts behind the use of historical correlation data to predict and estimate future correlations.

The Fine Print

Past asset volatility of most assets and most portfolios is historically well correlated with future volatility. However, not only are assets increasingly correlated, there is some evidence that asset correlations tend to increase during times of financial crisis. Even if assets are more correlated, there remains significant value in exploiting partial-discorrelation.
(*) The two-asset model can be represented as two parametric functions of a single variable, “t”, ER(t), and var(t).  t simply represents the investment proportion invested in asset 0 (aka asset A).  For three variables, expected return becomes ER(t0,t1) as does var(t0,t1).  And so on for increasing numbers of assets.  The computational effort required to compute ER(t0…tn) scales linearly with number of assets, but var(t0…tn) scales geometrically.
Optimizing efficiently within this complex space benefits from creative algorithms and heuristics.

Performance

The basic heuristics and algorithms I envisioned a year and a half ago have stood up well to testing, both internal testing and using external beta tester and client data.  Lessons learned have largely been associated with learning what is important to institutional investors.  Some of those lessons:

1. To date, every client has avoided long-short portfolios.  All of my initial clients have asked for strictly long-only portfolio optimization. Based on their directions, I have temporarily incorporated a “zero-floor” for all securities, which modestly speeds up the optimization process.  (Note: this constraint can easily be reversed.)
2. The initial portfolio-optimization code ran open-loop; that is to say that any asset could be assigned a weighting between 0 and 100%.  Generally, extreme asset-weightings were mathematically avoided for the majority of the “optimization surface.”   However, all Sigma1 clients have requested individual asset minimum and maximum asset-weighting constraints.  While these constraints somewhat reduce the enormous search space, in practice, they tend to slow down the heuristic algorithms.  Much of my optimization effort has been focused on efficiently enabling individual asset range constraints.
3. The third major lesson was that some clients want layered asset-class constraints.  This capability has been incorporated into the base code.

The primary thesis behind HALO Portfolio Optimization is that compute technology and algorithms have sufficiently progressed to optimize portfolios beyond simple mean-variance optimization (MVO).  Moreover, creating a set of three(an efficient surface) of portfolios optimized for multiple objectives (three: risk1, risk2, and expected return) is performed, rather than a simpler 2-D optimization.

Much more run-time optimization is on the Sigma1 road map.  The primary speed up is via conversion of increasing conversion of key parts of the HALO Ruby code to C/C++.  In the meantime, upgrading to arguably the fastest processor on the planet, the Intel i7-4700K, has shown a 2.95X speed up over benchmarks running the i5-2647M CPU running Ruby code that is currently the HALO Portfolio Optimization run-time bottleneck.  The primary operations are (billions of) double-precision floating-point arithmetic computations.

The HALO portfolio-optimization algorithms/heuristics have already “fast enough” for every single institutional investor we have worked with to date.  That does not measurably dampen my personal desire to push optimization speed to its limit.  I intend to crush previous performance benchmarks, again and again.

Does a hard-working professional investment advisory team need optimization to faster than 30 minutes?  I’d argue “No.”  But do they want faster, of course “Yes!”  If the crunch time is reduced to 5 minutes — the same logic applies — they want faster.  I understand.

It could easily be argued that it would be better to apply my efforts to developing the web UI.  I am, in parallel, at my own pace.  Currently, however, my passion is speed.  Having achieved some speed, I crave more.  When I follow my passion, my productivity is dramatically improved.  Moreover, the skill set I am trying to master has intrinsic value beyond the field of portfolio optimization.  Fun and profit, in a start-up, is often more important than maximum profit (or maximum revenue).

The HAL0 algorithms and heuristics are intrinsically fast and scalable.  Since I am not planning on sharing their inner architecture (except for millions of dollars), the proof of their power is measured in raw performance.  If that effort results in temporary loss of revenue for enhanced future revenue, then so be it.

 

.

 

Using HALO Portfolio Optimization Software

Setting up a basic HALO optimization requires a list of asset tickers, their min and max constraints, and expected returns.  Also at least one user specified category designation is required. Below is a short example:

SPY 7% 40% 9.11% Equities
PBP 0% 15% 8.53% Equities
USMV 5% 15% 9.05% Equities
VTI 5% 25% 9.35% Equities
IJH 5% 20% 9.55% Equities
VB 3% 15% 9.81% Equities
VEA 5% 12% 8.69% International
VEU 5% 20% 9.21% International
EEM 1% 11% 10.07% International
JNK 3% 15% 6.03% Bonds
BKLN 0% 8% 3.61% Bonds
AGG 1% 9% 2.32% Bonds

Generally, it is advisable to keep the sum of the individual asset minimums below 50%, and the sum of maximums above 200%. This provides the HALO optimizer the freedom to create a wide range of optimized portfolios with different risk/reward trade offs.

The above example is a very basic configuration. In order for asset managers to specify asset-class constraints, it is necessary to tell the optimizer that the “string” is a user-defined category.  Currently this is done with a leading gastritis (*):

*Equities       25% 85%
*International  10% 30%
*Bonds          15% 45%

The above config specifies that Equities must comprise a minimum of 25% of the investment portfolio and a maximum of 85%.  As with the individual asset constraints, it is advised to provide reasonably wide latitude to the optimization algorithms to produce a diverse set of optimized portfolios.

By default, the HALO Optimizer will produce a set of portfolios optimized to:

1) minimize:
a) semi-variance, σ_d (the default)
b) –OR– annualized standard deviation of total return, σ

2) maximize expected return, E(R)

The default time series used for computing σ and σ_d is end-of-month total-return deltas for the previous 36 months.  (This requires 37 months of total-return data for each security.)  The time period can be customized to use, say 60 months worth of data in the analysis.  HALO also supports using weekly closing data or even daily closing data — however I generally recommend using monthly data for a variety of reasons.  First, it speeds the computation.  Second, monthly data captures multi-day and multi-week trends, correlations, and specifically low-correlation asset optimization.  Third, monthly data is closer to the sampling period of a “typical” high-net-worth retail investor.  [That said, a case could be made for using quarterly data — which is also supported.]

Frequently HALO clients want to model newer securities that do not have 37 months of historical data.  For example, min-volatility ETFs such as SPLV, USMV, and EEMV are popular ETFs that are less than 3 years old. The HALO software suite has utilities that can statistically back fill the missing data.  The configuration of the statistical back-fill process is beyond the scope of this blog post, however it is an important and popular HALO Optimization Suite capability that so far has been used by all of Sigma1’s clients and beta testers.

Occasionally, Sigma1 clients and beta testers have had in-house funds that do not externally report their price or total return data.  For in-house funds, HALO can read client-supplied total-return data.  Naturally, HALO can include stocks, bonds, commodities, futures, and other assets with historical data into the portfolio optimization mix.

 

 

The Technology Behind the Tech

I started developing HALO portfolio-optimization software just shy of 1.5 years ago.  I owned a Linux workstation with a simple RAID-1 file system running Debian Linux, and I was on a 8-week sabbatical from work.  I started development using Ruby 1.8.7.  Later I switched to Ruby 1.9.1, then 1.9.3.  I have chosen, for now, to remain “frozen” on the 1.9.3 Ruby implementation.

I chose Ruby as my primary prototyping language because it allows rapid development, intuitive and highly-readable code, and easy of debugging.  Moreover, the Ruby HALO code is highly portable: The code developed Linux ran identically on Window with little to no modification.  After the initial development push that produced 3-D optimization graphs, I began developing on Windows under Cygwin.  I did this for three reasons.  1)  I wanted to run and test on Windows, 2) I had a Windows laptop and wanted to be able to develop anywhere: at the park, at the coffee shop, on an airplane, etc.  3)  I wanted to bring and demo my software to potential investors and potential clients.  I periodically sync the code between my Windows laptop and my Linux desktop.

Eventually, I began to wonder how much run time performance I was giving up by using Ruby instead of C/C++.  I profiled the Ruby code and began re-implemented the most compute-time-intensive parts of the code in C/C++.  Finding C++ a bit lacking by itself, I began using C++11 with Boost.  I am currently using Boost 1.53.0.

I was expecting perhaps a 10X speed up, but was happily surprised that I saw speed ups of between 20X and 50X when converting from Ruby to C++.  The C/C++ implementation is not a line-by-line parroting of the Ruby code, but the algorithms remained the same.  Coding in C/C++ caused me to focus more on memory allocation and, yes, garbage collection, so I attribute part of the massive speed up to that.

(I also tried JRuby — compiling the Ruby code to Java bytecode and running on the JVM.  I did see a speedup, but only about 15%.)

To date I have converted less than half of the Ruby to C++.  Currently the complete HAL0 Suite is 70% Ruby, 26% C++, and 4% other (Perl, Shell, etc).  By converting the most critical algorithms to C++, overall run time has improved approximately 3.5X.  Less than 10% of the end-to-end run time is consumed by the C++ portion.  That leads me to believe that an additional 10X performance increase is likely by the time I convert to a 100% C++ implementation.  [All of my run-time benchmark data are based on runs on an Intel(R) Core(TM) i5-2467M CPU @ 1.60GHz.]

My current C++ compiler is gcc 4.5.3.  All the C++ benchmarks above were performed with no compiler optimization (-O0).  Simply switching to “-O2” optimization resulted in a 15.6% performance boost.  I plan on experimenting with other flags starting with some suggestions from this GCC Intel x86 performance webpage. I will admit I have a bias toward Intel hardware.  That being said, it likely be a long time before I choose to plunk down $699 x 2 of Sigma1 Financial’s capital to buy even the basic Intel C++ compilers for Windows and Linux.

I ordered an Intel Core i7-4770K (desktop) system for Sigma1 Financial within a day of the chip’s official release.  It is scheduled to arrive next week.  Naturally, I am excited to see how much performance gain results from jumping two CPU generations and over 2X in core frequency.