Semivariance-Based and Hybrid Financial Ratios

When evaluating investment portfolios, a higher Sharpe ratio is typically better than a lower Sharpe ratio.  Similarly, a higher Sortino ratio is better than a lower one.  Finally, for portfolios with identical expected returns, the one with a lower standard deviation (sigma, σ) is better.

My best estimate is that less than 2% of portfolio assets are currently optimized for semivariance of any kind (mean-return semivariance, return semivariance, semivariance with respect to minimum acceptable return, etc.).   Meanwhile at least 60% of portfolio assets take variance into account.  According to the laws of supply and demand, the variance contribution (often related to beta) of an asset is strongly priced into its market value.  Conversely, the asset’s semivariance is essentially not priced in.

If, as I believe,  semivariance-based models continue to play an increasing roll in portfolio optimization strategies then asset-semivariance contributions will increasingly price in.   If this prediction holds true, then early adopters of semivariance optimization will front-run asset pricing changes by buying assets with superior price-to-semivariance properties and shunning assets that detract from, say, Sortino ratios.

Sharpe and Sortino ratios differ only in the denominator (σp versus σd) .  For portfolio-optimization, I prefer to modify σp and σd such that they would be identical for a normal distribution.  In such a scenario, each sigma is the square root of its corresponding (return) variance.  Because downside variance (σd ), in a normal distribution, is half of total variance I multiply downside variance by 2.   For the same reason, I tend to avoid anything mathematically arbitrary such as minimum acceptable return (MAR), and use zero or mean-return instead.  Empirically, I have found that zero works quite well.

My preliminary data shows is that for virtually any portfolio σp ≠ σd.  While σp and σd are positively correlated, their ex-post correlation is as low as 0.5.   In some cases, for optimized portfolios of the same return,  increasing  σp  by one unit (say 0.01) reduces σd by a similar amount  and vice versa.   This is an astounding trade off — it means variance and semivariance compete head to head.

I don’t believe that the investment industry will switch from variance to semivariance models overnight.  Similarly individual investment houses desire to shift gradually (each on their own timescale).  Thus I propose the use of hybrid variance/semivariance ratios.  With hybrid ratios, financial professionals can choose just how much they want semivariance to factor into their optimization… from zero to 100%, and everything in between.

The shifting portfolio-optimization market is large enough large enough for several players to make healthy profits.  For those players that can and do make a shift to semivariance, I recommend a contiguous hybrid model.




Beta Software, First Month

This marks the first month (30 days) of engagement with beta financial partners.  The goal is to test Sigma1 HAL0 portfolio-optimization software on real investment portfolios and get feedback from financial professionals.  The beta period is free.  Beta users provide tickers and expected-returns estimates via email, and Sigma1 provides portfolio results back with the best Sharpe, Sortino, or Sharpe/Sortino hybrid ratio results.

HAL0 portfolio-optimization software provides a set of optimized portfolios, often 40 to 100 “optimal” portfolios, optimized for expected return, return-variance and return-semivariance.   “Generic” portfolios containing a sufficiently-diverse set of ETFs produce similar-looking graphs.  A portfolio set containing SPY, VTI, BND, EFA, and BWX is sufficient to produce a prototypical graph.  The contour lines on the graph clearly show a tradeoff between semi-variance and variance.


Portfolio Optimization, Variance, Semi-Variance, and Total Return

Portfolio Optimization Graph


Once the set of optimized portfolios has been generated the user can select the “best” portfolio based on their selection criteria.

So far I have learned that many financial advisers and fund managers are aware of post-modern portfolio theory (PMPT) measures such as semivariance, but also a bit wary of them.  At the same time, some I have spoken with acknowledge that semivariance and parts of PMPT are the likely future of investing.  Portfolio managers want to be equipped for the day when one of their big investors asks, “What is the Sortino ratio of my portfolio? Can you reduce the semi-variance of my portfolio?”

I was surprised to hear that all of Sigma1 beta partners are interested exclusively in a web-based interface. This preliminary finding is encouraging because it aligns with a business model that protects Sigma1 IP from unsanctioned copying and reverse-engineering.

Another surprise has been the sizes of the asset sets supplied, ranging from 30 to 50 assets. Prior to software beta, I put significant effort into ensuring that HAL0 optimization could handle 500+ asset portfolios. My goal, which I achieved, was high-quality optimization of 500 assets in one hour and overnight deep-dive optimization (adding 8-10 basis points of additional expected-return for a given variance/semi-variance). On the portfolio assets provided to-date, deep-dive runtimes have all been under 5 minutes.

The best-testing phase has provided me with a prioritized list of software improvements. #1 is per-asset weighting limits. #2 is an easy-to-use web interface. #3 is focused optimization, such as the ability to set max variance.  There have also been company-specific requests that I will strive to implement as time permits.

Financial professionals (financial advisers, wealth managers, fund managers, proprietary trade managers, risk managers, etc.) seem inclined to want to optimize and analyze risk in both old ways (mean-return variance) and new (historic worst-year loss, VAR measures, tail risk, portfolio stress tests, semivariance, etc.).

Some Sigma1 beta partners have been hesitant to provide proprietary risk measure algorithms.  These partners prefer to use built-in Sigma1 optimizations, receive the resulting portfolios, and perform their own in-house analysis of risk.  The downside of this is that I cannot optimize directly to proprietary risk measures.  The upside is that I can further refine the HAL0 algos to solve more universal portfolio-optimization problems.  Even indirect feedback is helpful.

Portfolio and fund managers are generally happy with mean-return variance optimization, but are concerned that semivariance-return measures are reasonably likely to change the financial industry in the coming years.   Luckily the Sharpe ratio and Sortino ratio differ by only the denominator (σp versus σd) .  By normalizing the definitions of volatility (currently called modified-return variance and modified-return semivariance) HAL0 software optimizes simultaneously for both (modified) Sharpe and Sortino ratios, or any Sharpe/Sortino hybrid ratios in-between.  A variance-focused investor can use a 100% variance-optimized portfolio.  An investor wanting to dabble with semi-variance can explore portfolios with, say, a 70%/30% Sharpe/Sortino ratio.   And an investor, fairly bullish on semivariance minimization, could use a 20%/80% Sharpe/Sortino hybrid ratio.

I am very thankful to investment managers and other financial pros who are taking the time to explore the capabilities of HAL0 portfolio-optimization software.  I am hopeful that, over time, I can persuade some beta partners to become clients as HAL0 software evolves and improves.  In other cases I hope to provide Sigma1 partners with new ideas and perspectives on portfolio optimization and risk analysis.  Even in one short month, every partner has helped HAL0 software become better in a variety of ways.

Sigma1 is interested in taking on 1 or 2 additional investment professionals as beta partners.  If interested please submit a brief request for info on our contact page.


The #1 Question about Sigma1 Financial Software

The question that potential investors and clients inevitably ask me is, “With thousands of very smart people working in finance, what makes you think Sigma1 Portfolio-Optimization Software is better?”   Others are more blunt, asking, “Do you think you are smarter than all the quants working at, say, Goldman Sachs?”

Let me start by saying that there are people in the industry that could understand Sigma1’s HAL0 algorithms.  In order to do so, it would be very helpful to have an electrical engineering background, a computer science background and understanding of CAPM.  Because of this fact, it is very important that Sigma1 keep these proprietary algorithms and trade secrets secret.  Sigma1 has also started evaluating patent attorneys and exploring patent protection.

The seeds of HAL0 methods and algos were born out of my early experiences in electrical engineering, specifically the area of VLSI design.  In the late 90’s and early 00’s it was becoming apparent to leading-edge VLSI design companies that silicon physical design could not continue to keep up with Moore’s Law unless there was a paradigm shift in our industry.  You see, VLSI design had already been revolutionized by a technology called logic synthesis.  But synthesis was breaking down for large designs due to it’s reliance on statistical wireload models (WLMs).    It turned out that actual wireloads did not follow normal probability distributions.  They had long, fat tails on the side of bad variance.

In 2001, my college and I wrote an article that became the July cover story for an industry magazine.  One of the two main points of the article was that the statistical models for wire delay were sub-optimal to the point of being broken.   My co-author and I were suggesting that companies that continued to follow the statistical methods of the past were going to either adopt new methods or become obsolete.  More than ten years later, the VLSI industry has shown this to be a true prediction.  Today, statistical WLMs are virtually obsolete.

A couple years later, while still working in VLSI design, I started grad school, taking graduate-level classes in finance and electrical engineering.  I worked with some really smart folks working on algorithms to solve some difficult circuit optimization problems.  Meanwhile I was learning about CAPM and MPT, as well as Fama French factors.  As I applied myself to both disciplines, little did I suspect that, years later, I would connect them both together.  As I learned more about finance, I saw how the prices of fixed income investments made mathematical sense.  Meanwhile I was stuck with the nagging idea that the theories of stock portfolio optimization were incomplete, especially because variance was a sub-optimal risk model.

I was on sabbatical, driving across New Mexico, thinking about my favorite theoretical problems to pass the time.  I was mentally alternating between Fourier transforms, semivariance portfolio-optimization, and heuristic optimization algorithms.   I was thinking back to when my undergrad colleagues and I predicted that class-D audio amplifiers would replace class A, A/B, and B amplifiers, especially for sub-woofers.  [I now own a superb class-D subwoofer that I bought on the web.]   In a flash it occurred to me that the same principles of superposition that make Fourier transforms and class-D amplifiers work, also apply to investment portfolios.  I suddenly knew how to solve semivariance and variance simultaneously and efficiently.  In the next 5 minutes I know what I was going to do for the rest of my sabbatical:  develop and test software to optimize for 3 variables — variance, semivariance, and total return.

A bit like Alexander Fleming, who “accidentally” discovered penicillin from observing a discarded Petri dish, I discovered something new my “accident” while driving on that New Mexico highway.  The Petri dish was a thought experiment in my mind.  Bits of knowledge of VLSI and computer science fell into the Petri dish of semivariance, and I saw it flourish.  All the previous semivariance dishes had languished, but suddenly I had found a glimmer of success.

In short, I have been thinking about semivariance for almost 10 years.  I have seen the limitations of variance-based models fail in my line of work in VLSI, and seen how improved models transformed the industry, despite early doubters.  The financial industry has been  happy to hire Ph.D.s in physics, a field in which variance-based models have proven extremely successful, from quantum physics, to thermodynamics, to PV=nRT.  Where I saw the breakdown of variance-based models, physicists have seen near universal success in applications tied to physics.

When your only tool is a hammer, nails appear everywhere.   I happened to have the right tools at the right time, and a spark of innovation on a stormy day driving across New Mexico.  I was lucky and sufficiently smart.  That is why I believe Sigma1’s portfolio-optimization solution is perhaps the best solution currently on the planet.  It is possible others have found similarly-effective solutions, but I my reasonably rigorous search I have found no evidence of such yet.

There are two primary factors that set Sigma1 portfolio-optimization software apart:  1)  Efficient solutions to semivariance-based portfolio optimization that scale to 1000+ assets, 2) 3-objective (3-D) models that concurrently optimize for, say, variance, semi-variance and expected return simultaneously.  I have seen a handful of software offerings that support semivariance and expected return in addition to variance and expected return, but independently (in a 2-D objective space).  Superimposing two 2-D models on the same chart is not the same as HAL0’s 3-D optimization.  For exampled, HAL0’s 3-D surface models (of the efficient frontier in 3 dimensions) allow exploration of the the variance/semivariance trade offs over any desired common expected return contour.  In the same way that topo maps graph the topology of the Earth’s terrain, Sigma1 technology can map the objective space of portfolio optimization in 3-D.

In summary, I was in the right place at the right time with the right knowledge to create revolutionary portfolio-optimization software.





Capital Allocation

Let’s start with the idea that CAPM (Capital Asset Pricing Model) is incomplete.   Let me prove it in a  few sentences.  Everyone knows that, for investors, “risk-free” rates are always less than borrowing (margin) rates.  Thus the concept of CAL (the capital asset line) is incomplete.  If I had a sketch-pad I’d supply a drawing showing that there are really three parts of the “CAL” curve…

  1. The traditional CAL that extends from Rf to the tangent intercept with the efficient-frontier curve.
  2. CAC (capital-asset curve)
  3. CAML (capital-asset margin line, pronounced “camel”)

Why?  Because the CAML has it’s own tangent point based on the borrower’s marginal rate.  Because the efficient frontier is monotonically-increasing the CAL and CAML points will be separated by a section of the EF curve I call the CAC.

All of this is so obvious, it almost goes without saying.  It is strange, then, that I haven’t seen it pointed out in graduate finance textbooks, or online.  [If you know of a reference, please comment on this post!]  In reality, the CAL only works for an unleveraged portfolio.

CAPM is Incomplete; Warren Buffett Shows How

Higher risk, higher return, right?  Maybe not… at least on a risk-adjusted basis.  Empirical data suggests that high-beta stock and portfolios do not receive commensurate return.  Quite to the contrary, low-beta stocks and portfolios have received greater returns than CAPM predicts.   In other words, low-beta portfolios (value portfolios in many cases) have had higher historical alphas.  Add leverage, and folks like Warren Buffett have produced high long-term returns.

Black Swans and Grey Swans

On the fringe of modern-portfolio theory (MPT) and post-modern portfolio theory (PMPT), live black swans.   Black swans are essentially the most potent of unknown unknowns, also known as “fat tails”.

At the heart of PMPT is what I call “grey swans.”  This is also called “breakdown of covariance estimates” or, in some contexts, financial contagion.  Grey-swan events are much more common, and somewhat more predictable… That is if one is NOT fixated on variance.

Variance is close, semivariance is closer.  I put forth the idea that PMPT overstates its own potential.  Black swans exists, are underestimated, and essentially impossible to predict.  “Grey swans” are, however, within the realm of PMPT.   They can be measured in retrospect and anticipated in part.

Assets are Incorrectly Priced

CAPM showed a better way to price assets and allocate capital.  The principles of semivariance, commingled with CAPM form a better model for asset valuation.  Simply replacing variance with semivariance changes fifty years of stagnant theory.

Mean-return variance is positively correlated with semivariance (mean semi-variance of asset return), but the correlation is far less than 1.   Further, mean variance is most correlated when it matters most; when asset prices drop.  The primary function of diversification and of hedging is to efficiently reduce variance.  Investors and pragmatists note that this principle matters more when assets crash together — when declines are correlated.

The first step in breaking this mold of contagion is examining what matter more: semivariance.   Simply put, investors care much less about compressed upward variance than they do about compressed downward variance.   They care more about semivariance.  And, eventually, they vote with their remaining assets.

A factor in retaining and growing an AUM base is content clients.  The old rules say that the correct answer the a key Wall Street interview question is win big or lose all (of the client’s money).  The new rules say that clients demand a value-add from their adviser/broker/hybrid.  This value add can be supplied, in part, via using the best parts of PMPT.  Namely semivariance.

That is the the end result of the of the success of semivariance.  The invisible hand of Sigma1, and other forward-looking investment companies, is to guide investors to invest money in the way that best meets their needs.  The eventual result is more efficient allocation of capital.  In the beginning these investors win.  In the end, both investors and the economy wins.  This win/win situation is the end goal of Sigma1.



Financial Software Tech

In order to create software that is appealing to the enterprise market today, Sigma1 must create software for five years from now.   In this post I will answer the questions of why and how Sigma1 software intends to achieve this goal.

The goal of Sigma1 HAL0 software is to solve financial asset allocation problems quickly and efficiently.  HALo is portfolio-optimization software that makes use of a variety of proprietary algorithms.  HALo’s algorithms solve difficult portfolio problems quickly on a single-core computer, and much more rapidly with multi-core systems.

Savvy enterprise software buyers want to buy software that runs well on today’s hardware, but will also run on future generations of compute hardware.   I cannot predict all the trends for future hardware advanced, but I can predict one:  more cores.  Cores per “socket” are increasing on a variety of architectures:  Intel x86, AMD x86, ARM, Itanium, and IBM Power7 to name a few.  Even if this trend slows, as some predict, the “many cores” concept is here to stay and progress.

Simply put — Big Iron applications like portfolio-optimization and portfolio-risk management and modelling are archaic and virtually DOA if they cannot benefit from multi-core compute solutions.   This is why HAL0 is designed from day 1 to utilize multi-core (as well as multi-socket) computing hardware.  Multiprocessing is not a bolt-on retrofit, but an intrinsic part of HAL0 portfolio-optimization software.

That’s the why, now the how.  Google likes to use the phrase “map reduce”  while others like the phase embarrassingly parallel.   I like both terms because it can be embarrassing when a programmer discovers that the problems his software was slogging through in series were being solved in parallel by another programmer who mapped them to parallel sub-problems.

The “how” for HAL0’s core algorithm is multi-layered.   Some of these layers are trade secrets, I can disclose one.  Portfolio optimization involves creating an “efficient frontier” comprised of various portfolios along the frontier.  Each of these portfolios can be farmed out in parallel to evaluate its risk and reward values.   Depending on the parameters of a particular portfolio-optimization problem this first-order parallelism can provide roughly a 2-10x speedup — parallel, but not massively parallel.

HALo was developed under a paradigm I call CAP (congruent and parallel).  Congruent in this context means that given the same starting configuration, HAL0 will always produce the same result.  This is generally easy for single-threaded programs to accomplish, but often more difficult for programs running multiple threads on multiple cores.    Maintaining congruence is extremely helpful in debugging parallel software, and is thus very important to Sigma1 software.  [Coherent or Deterministic could be used in lieu of Congruent.]

As HAL0 development continued, I expanded the CAP acronym to CHIRP (Congruent, Heterogeneous, Intrinsically Recursively Parallel).   Not only does CHIRP have a more open, happier connotation that CAP, it adds two additional tenets:  heterogeneity and recursion.

Heterogeneity, in the context of CHIRP, means being able to run, in parallel, on a variety of machines will different computing capabilities.  On on end of the spectrum, rather than requiring all machines in the cloud or compute queue having the exact same specs (CPU frequency, amount of RAM, etc), the machines can be different.  On the other end of the spectrum, heterogeneity means running in parallel on multiple machines with different architectures (say x86 and ARM, or x86 and GPGPUs).  This is not to say that HAL0 has complete heterogeneous support; it does not.  HALo is, however, architected with modest support for heterogeneous solutions and extensibility for future enhancements.

The recursive part of CHIRP is very important.  Recursively parallel means that the same code can be run (forked) to solve sub-problems in parallel, and those sub-problems can be divided into sub-sub problems, etc.   This means that the same tuned, tight, and tested code can leveraged in a massively parallel fashion.

By far the most performance-enhancing piece of HAL0 portfolio-optimization CHIRP is RP.  The RP optimizations are projected to produce speedups of 50 to 100X over single-threaded performance (in a compute environment with, for example, 20 servers with 10 cores each).  Moreover, the RP parts of HAL0 only require moderate bandwidth and are tolerant of relatively high latency (say, 100 ms).

Bottom line:  HAL0 portfolio-optimization software is designed to be scalable and massively parallel.




Building a Financial Software Company


Saturday, I had a fascinating morning starting a 10-week entrepreneurship program.  I sat face-to-face with a couple of people who could likely write 7-figure checks that would clear the bank.

Tuesday was day 2 of the entrepreneurship class.  I can sum up the general mood of the class with two words:  energy and optimism.  Speakers, coaches, and students were increasingly energized as the 3.5 hour class progressed.  There are some talented students in my class who have already have revenue, employees, and venture capital.  Others have a Ph.D. thesis and novel ideas from their post-doc work.  I am presently somewhere in between:  I am beyond the idea phase and well into the software product development phase.  I have demo-able software and am self-funded to the tune of approx. $50,000.

Apparently my ideas and software are starting to get traction within the group.  It is possible that some college students will write the Sigma1 Financial business plan as a for-credit exercise.  If this pans out, I get a free first-draft business plan, and they get college credit.  That spells “win-win” to me.

I have also learned that my “elevator pitch” needs work.  My off-the-cuff pitch on Day 1 was my best.  Successive effort have been worse.  Feedback examples include:  I felt buried; too much financial jargon; too long.  This is great feedback!  Better ideas from the group include gems such as:

  • Financial Risk:  If you can see it, you can avoid it.  Understand risk, visualize risk, and using proprietary software decrease risk by X%.
  • Nobody likes risk, but almost nobody truly understands it.  Sigma1 software does.
  • Helping financial advisers model and explain risk to their clients.
  • Visualizing risk… Really!

Here’s the revised elevator pitch based off of their feedback.   “Hi, I founded Sigma1 Software.   If you own an investment portfolio there is one think you should know:  it was probably constructed with using flawed risk models.   Sigma1 Software can analyze your existing portfolio using better risk models.  It will also build alternative portfolios and display the results visually.  Our software not only reduces risk; it helps you visualize risk.

Few people like financial risk, and fewer still truly understand risk.  Sigma1 Software does, and it uses advanced risk models and proprietary algorithms to help you build a better portfolio.

That’s the pitch:  Smarter risk models, risk reduction, and powerful visualization tools.  It’s what we do at Sigma1 Software.”

Building a Solid Team

To take Sigma1 Software to the next level, I need a great team.  At a minimum I need a salesperson.  This person needs both sales experience and experience in the financial industry.  Having a wide network of connections to financial professionals is a plus.  I’m looking for someone will no only close deals, but work with me and our attorneys to construct standard contracts and pricing models.  A person who will build and maintain long-term relationships with our customers while finding new customers.  Finally a salesperson who listens to customer needs in order to help us improve our software and software support.

Next on my hiring list is a second software developer specializing in user interface development for Windows, Linux, and Web applications. I’m looking for a developer who also has the interpersonal skills to 1) contribute to a highly collaborative environment,  2)  conduct customer training, 3) join sales visits to demo Sigma1 software and answer technical questions.  Pluses would be Windows and/or Linux sysadmin and IT skills.  The software developer position would also require occasional work performing unit, regression, and beta testing of the portfolio-optimization suite.

It is crucial that any employees, partners, or employee/partners believe in Sigma1’s mission:  To build and sell revolutionary portfolio-optimization technologies which will revitalize and reinvent the financial industry.  To that end, equity in the form of voting, non-voting, dilutable, and possibly non-dilutable stock are likely to be an important part of any hiring agreement.

Getting Technical

Software Scalability

Without IPC (Inter-process communication) scalability is virtually impossible.  With IPC comes numerous choices and tradeoffs:  pipes, named pipes, messages, semaphore-managed file-sharing, memory-sharing, sockets, socket pairs… to name some.   The tradeoffs involve platform support, job granularity, and performance.

After much thought, I have temporarily committed to Linux/UNIX named pipes as the primary IPC mechanism.  The pros of this choice include robustness, reasonable speed, and a wide range of support for multi-thread, multi-core, and multi-server parallelism.  The primary downside: lack of Windows compatibility.

The HAL0 portfolio-optimization suite can still run on Windows, but for now Windows parallelism is limited to thread-level scalability on a single machine.


Linux and UNIX programs often communicate very well together with streaming data.  [For this post, I’ll used the term Linux to generally refer to Linux and/or UNIX].  Streaming data has its limitations, however.  Perhaps the biggest limitation is “large-grain granularity”.  By this I mean that an interaction involves loading the program (high-latency), processing the stream, closing the program.  Streaming is expensive for fine-grain ad hoc requests because of the open/close overhead.

Named pipes (especially when properly paired) are an elegant way around the open/close overhead issue.

It is often a simple task to modify a Linux program to support named pipes.  It does not matter if said program is written in C++, Python, Java, Ruby, Perl, shell, etc.  If a program can be modified to accept input from a named pipe and write results to another named pipe, it can be integrated with other program that supports a common name, pipe-based API.  HAL0 financial software does just that.

Scalability Squared

HAL0 software forms the core engine or kernel of the Sigma1 portfolio software.  In parallel mode, HAL0 spawns worker jobs that compute portfolio metrics.  HAL0 gathers, organizes, and evaluates the results.  Then, based on the past and current wave of results, HAL0 identifies the most fruitful next wave of results to explore and “farms out” the next wave of computing.  This is the primary means of achieving scalability:  the worker jobs are distributed to other cores and other other servers.  These “other” compute resources perform the massive heavy-lifting in parallel, speeding up computation immensely.

When there are enough worker jobs, there comes a point when the worker jobs cease to be the primary compute-speed limiter.  This is where Amdahl’s law really kicks in.  At some point maximum speedup is achieved, limited by the “core” processes ability to send, wait for, receive and process worker-job data.

If the “core” (or master or “boss”) process itself can be split into parallel processes, an additional level of scalability kicks in.  This core HAL0 algorithm is designed to do just that.

Based on preliminary estimates, it can scale efficiently to 4 cores, delivering up to a 3.5X core speedup.   Additionally the current HAL0 periphery for each HAL0 instance scales efficiently to up to 10-ways, providing about a 6X additional speedup per instance (depending on IPC latency). Ideally, Sigma1 portfolio-optimization software can provide up to a 21X speedup (3.5 times 6) operating in parallel mode versus single-CPU mode.

There are many caveats in the preceding paragraph.  Right now I am focused on implementing and testing scalability, less so than on optimizing it.  For example I am currently implementing single-kernel instance scalability in a manner than is deterministic, repeatable, and producing results identical to single-CPU operation.  This limits the speedup, but makes regression testing practical.  Regression testing in turn helps keep the code robust and reliable.

Portfolio-Optimization Software for the Enterprise

So far, ensuring that Sigma1 portfolio software is capable of massive scalability has roughly tripled the software development effort. This is obviously slowing time-to-market, but I continue to believe it is worth the effort and schedule impact. First, scalability is a key product differentiator for enterprise-level customers. Second, supporting scalability from day 1 is much easier and more reliable that trying to retrofit scalability into an intrenched software architecture.


Signficance of the Name Sigma 1 (Σ1)

Σ1 is shorthand for Σwi = 1.   This means that any given instant a portfolio can be expressed as a series of investment weights which always sum to 1.  This principle even applies to long-short funds with positive and negative investment weightings and leveraged funds with negative cash weightings.  This is the primary meaning of Sigma1.

σ1, with a lower-case sigma, is shorthand for one standard deviation (σ times 1).  This is the second thing I associate with Sigma1 Software  — a classic measure of financial risk.

While a portfolio manager may have many responsibilities, his or her “job one” is deciding which investments in what proportion.   It is surprising how challenging this job can be, given that even a 100-asset portfolio can have many more than trillions of trillions of possible weightings [actually a gross understatement].

Wrapping up;  Sigma1 is a startup creating financial software focused on investment portfolio optimization.  In this context “investment portfolio” can mean a proprietary-trading fund, a mutual fund, an ETF, a financial endowment, “index plus”,  or other managed investment.  At the end the day, Sigma1 means business.


Portfolio-Optimization Software: A Financial Software Suite for Power Users

When I’m not coding Sigma1 financial software, I’m often away from the keyboard thinking about it.  Lately I’ve been thinking about how to subdivide the HAL0 portfolio-optimization code into autonomous pieces.  There are many reasons to do this, but I will start by focusing on just one:  power users.

For this blog post, I’m going to consider Jessica, a successful 32 year-old proprietary trader.  Jessica is responsible for managing just over $500 million in company assets.  She has access to in-house research, and can call on the company’s analysts, quants, and researchers as needed and available.  Jessica also has a dedicated $500,000 annual technology budget, and she’s in the process of deciding how to spend it most effectively. She is evaluating financial software from several vendors.

Jessica is my target market for B2B sales.

In my electrical engineering career I have been responsible for evaluating engineering software.  Often I would start my search with 3 to 5 software products.  Due to time constraints, I would quickly narrow my evaluation to just two products.   Key factors in the early software vetting process were 1) ease of integration into our existing infrastructure and 2) ease of turn-on.  After narrowing the search to two, the criteria switched to performance, performance, performance, and flexibility… essentially a bake-off between the two products.

Ease of use and integration was initially critical because I (or others on my team) needed a product we could test and evaluate in-house.  I refused to make a final selection or recommendation based on vendor-provided data.  We needed software that we could get up and running quickly… solving our problems on our systems.  We’d start with small problems to become familiar with the products and then ramp up to our most challenging problems to stress and evaluate the software.

Assuming Jessica (and others like her) follow a similar approach to software purchases, HAL0 portfolio software has to get through both phases of product evaluation.

HAL0 software is optimized to “solve” simultaneously for 3 goals, normally:

  1. A risk metric
  2. A total-return metric
  3. An “x-factor” metric (often an orthogonal risk metric)

While being great for power users, the x-factor metric is also non-standard.  To help get through the “ease of start up” phase of product evaluation, I will likely default the x-factor to off.

Once the preliminary vetting is complete and final evaluation begins in earnest, performance and flexibility will be put to the test.  If, for instance, Jessica’s team has access to a proprietary risk-analysis widget utilizing GPGPUs that speeds up risk computation 100X over CPUs, HAL0 software can be configured to support it.  Because HAL0 is increasingly modular and componentized, Jessica’s resident quants can plug in proprietary components using relatively simple APIs.  Should competing products lack this plug-in capability, HAL0 software will have a massive advantage out of the starting gate.

When it comes to flexibility, component-based design wins hands down.  Proprietary risk metrics can be readily plugged in to HAL0 optimization software.  Such risk models can replace the default semi-variance model, or be incorporated as an adjunct risk metric in “x-space.”  Users can seed portfolio optimization with existing real and/or hypothetical portfolios, and HAL0 software will explore alternatives using user-selected risk and performance metrics.

HAL0 Portfolio-Optimization Basic Features

Out-of-the-box HAL0 software comes pre-loaded with semi-variance and variance risk metrics.  Historic or expected-return metrics are utilized for each security.  By default, the total return of any portfolio is computed as the weighted average of expected (or historic) total return of the securities that portfolio.  Naturally, leveraged and short-position weightings are supported as desired.

These basic features are provided for convenience and ease of setup.  While robust and and “battle-ready”, I do not consider them major value-add components.  The key value-proposition for HAL0 portfolio-optimization is its efficient, multi-objective engine.  HAL0 software is like a race car delivered ready to compete with a best-in-class engine.  All the components are race-ready, but some world-class race teams will buy the car simply to acquire the engine and chassis, and retrofit the rest.

Because HAL0 software is designed from the ground up to efficiently optimize large-asset-count portfolios using 3 concurrent objectives, switching to conventional 2-D optimization is child’s play.  Part of the basic line up of “x-space” metrics includes:

  • Diversification as measured against root-mean-square deviation from market or benchmark in terms of sector-allocation, market-cap allocation, or a weighted hybrid of both.
  • Quarterly, 1-year, 3-year, or 5-year worst loss.
  • Semi-variance (SV) or variance (V) — meaning concurrent optimization for both V and SV is possible.

Don’t “Think Different”, Be Different!

My primary target audience is professional investors who simply refuse to run with the herd.  They don’t seek difference for its own sake, but because they wish to achieve more.  HAL0 is designed to help ease its own learning curve by enabling users to quickly achieve the portfolio-optimization equivalent of “Hello, World!”, while empowering the power user to configure, tweak, and augment to virtually any desired extreme.

The Equation that Will Change Finance

Two mathematical equations have transformed the world of modern finance.  The first was CAPM, the second Black-Scholes.  CAPM gave a new perspective on portfolio construction.  Black-Scholes gave insight into pricing options and other derivatives.  There have been many other advancements in the field of financial optimization, such as Fama-French — but CAPM and Black-Scholes-Merton stand out as perhaps the two most influential.

Enter Semi-Variance

Modified Semi-Variance Equation

Modified Semi-Variance Equation, A Financial Game Changer

When CAPM (and MPT) were invented, computers existed, but were very limited.  Though the father of CAPM, Harry Markowitz, wanted to use semi-variance, the computers of 1959 were simply inadequate.  So Markowitz used variance in his ground breaking book “Portfolio Selection — Efficient Diversification of Investments”.

Choosing variance over semi-variance made the computations orders of magnitude easier, but the were still very taxing to the computers of 1959.  Classic covariance-based optimizations are still reasonably compute-intensive when a large number of assets are considered.  Classic optimization of a 2000 asset portfolio starts by creating a 2,002,000-entry (technically 2,002,000 unique entries which, when mirrored about the shared diagonal, number 4,000,000) covariance matrix; that is the easy part.  The hard part involves optimizing (minimizing) portfolio variance for a range of expected returns.  This is often referred to as computing the efficient frontier.

The concept of semi-variance (SV) is very similar to variance used in CAPM.  The difference is in the computation.  A quick internet search reveals very little data about computing a “semi-covariance matrix”.  Such a matrix, if it existed in the right form, could possibly allow quick and precise computation of portfolio semi-variance in the same way that a covariance matrix does for computing portfolio variance.  Semi-covariance matrices (SMVs) exist, but none “in the right form.” Each form of SVM has strengths and weaknesses. Thus, one of the many problems with semi-covariance matrices is that there is no unique canonical form for a given data set.  SVMs of different types only capture an incomplete portion of the information needed for semi-variance optimization.

The beauty of SV is that it measures “downside risk”, exclusively.  Variance includes the odd concept of “upside risk” and penalizes investments for it.  While not  going to the extreme of rewarding upside “risk”, the modified semi-variance formula presented in this blog post simply disregards it.

I’m sure most of the readers of this blog understand this modified semi-variance formula.  Please indulge me while I touch on some of the finer points.   First, the 2 may look a bit out of place.  The 2 simply normalizes the value of SV relative to variance (V).  Second, the “question mark, colon” notation simply means if the first statement is true use the squared value in summation, else use zero.  Third, notice I use ri rather than ri – ravg.

The last point above is intentional and another difference from “mean variance”, or rather “mean semi-variance”.  If R is monotonically increasing during for all samples (n intervals, n+1 data points), then SV is zero.  I have many reasons for this choice.  The primary reason is that with  ravg the SV for a straight descending R would be zero.  I don’t want a formula that rewards such a performance with 0, the best possible SV score.  [Others would substitute T, a usually positive number, as target return, sometimes called minimal acceptable return.]

Finally, a word about r— ri is the total return over the interval i.  Intervals should be as uniform as possible.  I tend to avoid daily intervals due to the non-uniformity introduced by weekends and holidays.  Weekly (last closing price of the trading week), monthly (last closing price of the month), and quarterly are significantly more uniform in duration.

Big Data and Heuristic Algorithms

Innovations in computing and algorithms are how semi-variance equations will change the world of finance.  Common sense is why. I’ll explain why heuristic algorithms like Sigma1’s HALO can quickly find near-optimal SV solutions on a common desktop workstation, and even better solutions when leveraging a data center’s resources.  And I’ll explain why SV is vastly superior to variance.

Computing SV for a single portfolio of 100 securities is easy on a modern desktop computer.  For example 3-year monthly semi-variance requires 3700 multiply-accumulate operations to compute portfolio return, Rp, followed by a mere 37 subtractions, 36 multiplies (for squaring), and 36 additions (plus multiplying by 2/n).  Any modern computer can perform this computation in the blink of an eye.

Now consider building a 100-security portfolio from scratch.  Assume the portfolio is long-only and that any of these securities can have a weight between 0.1% and 90% in steps of 0.1%.  Each security has 900 possible weightings.  I’ll spare you the math — there are 6.385*10138 permutations. Needless to say, this problem cannot be solved by brute force.  Further note that if the portfolio is turned into a long-short portfolio, where negative values down to -50% are allowed, the search space explodes to close to 102000.

I don’t care how big your data center is, a brute force solution is never going to work.  This is where heuristic algorithms come into play.  Heuristic algorithms are a subset of metaheuristics.  In essence heuristic algorithms are algorithms that guide heuristics (or vise versa) to find approximate solution(s) to a complex problem.   I prefer the term heuristic algorithm to describe HALO, because in some cases it is hard to say whether a particular line of code is “algorithmic” or “heuristic”, because sometimes the answer is both.  For example, semi-variance is computed by an algorithm but is fundamentally a heuristic.

Heuristic Algorithms, HAs, find practical solutions for problems that are too difficult to brute force.  They can be configured to look deeper or run faster as desired by the user.  Smarter HAs can take advantage of modern computer infrastructure by utilizing multiple threads, multiple cores, and multiple compute servers in parallel.  Many, such as HAL0, can provide intermediate solutions as they run far and deep into the solution space.

Let me be blunt — If you’re using Microsoft Excel Solver for portfolio optimization, you’re missing out.  Fly me out and let me bring my laptop loaded with HAL0 to crunch your data set — You’ll be glad you did.

Now For the Fun Part:  Why switch to Semi-Variance?

Thanks for reading this far!  Would you buy insurance that paid you if your house didn’t burn down?   Say you pay $500/year and after 10 years, if your house is still standing, you get $6000. Otherwise you get $0. Ludicrous, right?  Or insurance that only “protects” your house from appreciation?  Say it pays 50 cents for every dollar make when you resell your house, but if you lose money on the resale you get nothing?

In essence that is what you are doing when you buy (or create) a portfolio optimized for variance.   Sure, variance analysis seeks to reduce the downs, but it also penalizes the ups (if they are too rapid).  Run the numbers on any portfolio and you’ll see that SV ≠ V.  All things equal, the portfolios with SV < V are the better bet. (Note that classic_SV ≤ V, because it has a subset of positive numbers added together compared to V).

Let me close with a real-world example.  SPLV is an ETF I own.  It is based on owning the 100 stocks out of the S&P 500 with the lowest 12-month volatility.  It has performed well, and been received well by the ETF marketplace, accumulating over $1.5 billion in AUM.  A simple variant of SPLV (which could be called PLSV for PowerShares Low Semi-Variance) would contain the 100 stocks with the least SV.  An even better variant would contain the 100 stocks that in aggregate produced the lowest SV portfolio over the proceeding 12 months.

HALO has the power to construct such a portfolio. It could solve preserving the relative market-cap ratios of the 100 stocks, picking which 100 stocks are collectively optimal.  Or it could produce a re-weighted portfolio that further reduced overall semi-variance.

[Even more information on semi-variance (in its many related forms) can be found here.]