Choices, Opportunities, and Solutions

To date I’ve invested approximately 800 hours developing and testing the heuristics and algorithms behind HALO. Finding exact solutions (with respect to expected-return assumptions) to certain real-world portfolio-optimization problems can be solved. Finding approximate solutions to other real-world portfolio-optimization problems is relatively easy, but finding provably optimal solutions is currently “impossible”. The current advanced science and art of portfolio optimization involves developing methods to efficiently find nearly optimal solutions.

I believe that HALO represents a significant step forward in finding nearly-optimal solutions to generalized risk models for investment portfolios. The primary strengths of HALO are in flexibility and dimensionality of financial risk modeling. While HALO currently finds solutions that are almost identical to exact solutions for convex optimization problems; the true advantage of HALO is in the quality of solutions for non-convex portfolio-optimization problems

Do you know if your particular optimization metric can be articulated in canonical convex notation? I argue that HALO does not care.  If it can be, HALO will find a near-optimal solution virtually identical to the ideal convex optimization solution.  If it cannot be, and is indeed non-convex, HALO will find solutions competitive with other non-convex optimization methods.

It could be argued that “over-fitting” is a potential danger of optimal and near-optimal solutions. However, I argue that given a sufficiently diverse and under-constrained optimization task, over-fitting is less worrisome.   In other words, the quality of the inputs greatly influences the quality of the outputs.  One secret is to supply high-quality (e.g. asset expected return) estimates to the optimization problem.

Advertisement

The Best Financial Models for Insight and Prediction?

The best models are not the models that fit past data the best, they are the models that predict new data the best. This seems obvious, but a surprising number of business and financial decisions are based on best-fit of past data, with no idea of how well they are expected to correctly model future data.

Instant Profit, or Too Good to be True?

For instance, a stock analyst reports to you that they have a secret recipe to make 70% annualized returns by simply trading KO (The Coca-Cola Company).  The analyst’s model tells what FOK limit price, y, to buy KO stock at each market open.  The stock is then always sold with a market order at the end of each trading day.

The analyst tells you that her model is based on three years of trading data for KO, PEP, the S&P 500 index, aluminum and corn spot prices.  Specifically, the analyst’s model uses closing data for the two preceding days, thus the model has 10 inputs.  Back testing of the model shows that it would have produced 70% annualized returns over the past three years, or a whooping 391% total return over that time period.  Moreover, the analyst points out that over 756 trading days 217 trades would have been executed, resulting in profit a 73% of the time (that the stock is bought).

The analyst, Debra, says that the trading algorithm is already coded, and U.S. markets open in 20 minutes. Instant profit is only moments away with a simple “yes.” What do you do with this information?

Choices, Chances, Risks and Rewards

You know this analyst and she has made your firm’s clients and proprietary trading desks a lot of money. However you also know that, while she is thorough and meticulous; she is also bold and aggressive. You decide that caution is called for, and allocate a modest $500,000 to the KO trading experiment.  If after three months, the KO experiment nets at least 7% profit, you’ll raise the risk pool to $2,000,000.  If, after another three months, the KO-experiment generates at least 7% again; you’ll raise the risk pool to $10,000,000 as well as letting your firms best clients in on the action.

Three months pass, and the KO-experiment produces good results: 17 trades, 13 winners, and a 10.3% net profit. You OK raising the risk pool to $2,000,000.  After only 2 months the KO-experiment has executed 13 trades, with 10 winners, and a 11.4% net profit.  There is a buzz around the office about the “knock-out cola trade”, and brokers are itching to get in on it with client funds. You are considering giving the green light to the “Full Monty,” when Stan the Statistician walks into your office.

Stan’s title is “Risk Manager”, but people around the office call him Stan the Statistician, or Stan the Stats Man, or worse (e.g. “Who is the SS going to s*** on today?”)  He’s actually a nice guy, but most folks consider him an interloper.  And Stan seems to have clout with corporate, and he has been known to use it to shut down trades. You actually like Stan, but you already know why he is stopping by.

Stan begins probing about the KO-trade.  He asks what you know.  You respond that Debra told you that the model has an R-squared of 0.92 based on 756 days of back-tested data.  “And now?” asks Stan.  You answer, “a 76% success rate, and profits of around 21% in 5 months.”  And then Stan asks, “What is the probability that that profit is essentially due to pure chance?”

You know that the S&P 500 historically has over 53% “up” days, call it 54% to be conservative. So stocks should follow suit.  To get exactly 23 wins on KO out of 30 tries is C(30, 23)*0.54^23*(0.46)^7 = 0.62%. To get at least 23 (23 or more wins) brings the percentage up to about 0.91%.  So you say 1/0.091 or about one in 110.

Stan says, “Your math is right, but your conclusion is wrong.  For one thing, KO is up 28% over the period, and has had 69% up days over that time.”  You interject, “Okay, wait one second… so my math now says about 23%, or about a 1 in 4.3 chance.”

Stan smiles, “You are getting much closer to the heart of the matter. I’ve gone over Debra’s original analysis, and have made some adjustments. My revised analysis shows that  there is a reasonable chance that her model captures some predictive insight that provides positive alpha.”  Stan’s expression turns more neutral, “However, the confidence intervals against the simple null hypothesis are not as high as I’d like to see for a big risk allocation.”

Getting all Mathy? Feedback Requested!

Do you want to hear more from “Stan”? He is ready to talk about adjusted R-squared, block-wise cross-validation, and data over-fitting. And why Debra’s analysis, while correct, was also incomplete. Please let me know if you are interested in hearing more on this topic.

Please let me know if I have made any math errors yet (other than the overtly deliberate ones).  I love to be corrected, because I want to make Sigma1 content as useful and accurate as possible.

The Future of Investing is Automation

A significant and growing portion of today’s individual investors have never placed a trade using a human stock broker.

Developing an Automation Mindset for Investing

In 2010, I bought the domain name Sigma1.com with the idea of creating an hedge fund that I would manage.  In order to measure and manage my investment strategies objectively, I began thinking about benchmarks and financial analysis software.  And as I ran scenarios through Excel and some light-weight analysis software I created, I began to realize that analysis, by itself was very limited.  I could only back-test one portfolio at a time, and I had to construct each portfolio’s asset weights manually.  It soon became obvious that I needed portfolio optimization software.

I learned that portfolio optimization software with the capabilities I wanted was extremely expensive. Further, I realized that even if, say, I negotiated a deal with MSCI where they provided Sigma1 Financial with their Barra Portfolio Manager for free, it would not differentiate a Sigma1 hedge fund from other hedge funds using the same software.

I was beginning to interact with several technology entrepreneurs and angel investors.  I quickly learned that legal costs and barriers to entry for a new hedge were intractable.  If Sigma1 attracted $10M in assets from accredited investors in 12 months, and charged 2 and 20, it would be a money loosing enterprise.  Cursory research revealed that critical mass for a profitable (for the hedge fund managers) hedge fund could be as high as $500M.  Luckily, I had learned about the concept of the “entrepreneurial pivot“.

The specific pivots Sigma1 used were a market segment pivot followed by a technology pivot. I realized that while the high cost of good portfolio optimization software is bad for a hedge fund startup, it was great for a financial software startup.  Suddenly, the Sigma1 Financial target market switched from accredited investors to financial professionals (investment managers, fund managers, proprietary traders, etc).  This was a key market segment pivot.

Just creating a cheaper portfolio optimizer seemed unlikely to provide sufficient incentive to displace entrenched portfolio optimizers. Sigma1 needed a technology pivot — finding a solution using a completely different technology.  Most prior portfolio optimizers use some variant of linear programming (LP) [or QP or NLP] to help find optimal portfolios. Moreover they also create an asset covariance matrix as a starting point for the optimization.

One stormy day, I realized that some algorithms I created to solve statistical electrical engineering problems in grad school could be adapted to optimize investment portfolios. The method I devised not only avoided LP, QP, or NLP methods; it also dispensed with the need for a covariance matrix.  Over then next several days I realized that by eliminating dependence on a covariance matrix, the algorithm I later named HALO, could use both traditional and alternate risk measures ranging from variance-based (eg. standard-deviation of return) to covariance-based ones (e.g. beta) to semivariance to max draw down.  By developing a vastly different technology, HALO could optimize for risks such as semivariance and Sortino ratios, or max drawdown, or even custom risk measures devised by the client.

Algorithms Everywhere

Long before Sigma1 began developing HALO, the financial industry has been increasingly reliant on digital systems and various financial algorithms. As digital communication networks and electronic stock exchanges gained trading volume, various forms of program trading began to flourish.  This includes the often maligned high-frequency trading variant of automated trading.

Concurrently, more and more trading volume has gone online.  A significant portion of today’s individual investors have never placed a trade using a human stock broker.

Automated Investment Advice, Analysis, and Trading

There are now numerous automated investment analysis tools, many of which come free with a brokerage account, while others are free or low-cost stand-alone online tools.  Examples of the former include the Fidelity’s nascent GPS (Guided Portfolio Summary) to more seasoned offerings such as Financial Engines.  Online portfolio analysis offering range from Morningstar’s Instant X-Ray, to sites like ETFreplay.

However these software offerings are just the beginning. A company call FutureAdvisor has partnered with Fidelity and TD Ameritrade to allow its automate portfolio software to make trades on its users behalf. Companies like Future Advisor have the potential to help small investors benefit from custom-tailored investment advice utilizing proven academic research (e.g. Fama French) at a very low cost — costs so low that they would not be profitable for human investment advisers to provide.

If successful (and I believe some automated investment companies will be), why should they stop at small-time investors, with less than $500,000 in investable assets?  Why not $1,000,000 or more?  Nothing should stop them!

I could easily imagine Mark Zuckerberg, Sergey Brin, or Larry Page utilizing an automated investment company’s software to manage a large part of their portfolios.  If we, as a society, are considering allowing automated systems to drive our cars for us, surely they can also manage our investment portfolios.

The Future Roll of the Human Financial Adviser

There will always be some percentage of investors who want a personal relationship with a financial adviser. Human investment advisers can excel at explaining investment concepts and putting investors at ease during market corrections.  In some ways human investment advisers even function as personal financial counselors, listening to their clients emotional financial stories.  And, of course, there are some people who want to be able to pick up the phone and yell at a real person for letting them suffer market losses.  Finally, there are people with Luddite tenancies who want as little to do with technology as possible.  For all these reasons human investment advisers will have a place in the future world of finance.

Investment Automation will Accelerate

There are some clear trends in the investing world.  Index investing will continue to grow, as will total ETF assets under management (AUM). Alternative investments from rental property to master limited partnerships (MLPs) to private equity are also likely to become part of the portfolios of more sophisticated and affluent investors.

With the exception of high-frequency trading, which has probably saturated arbitrage and front-running opportunities, I expect algorithmic (algo) management to increase as an overall percentage of US and global AUM. Some algorithmic trading and investing will be of the “hardwired” variety where the algo directly connects to the exchanges and makes trades, while the rest of the algo umbrella will comprise trading and investing decisions made by financial software and entered manually by humans with minimal revision.  There will also be hybrid methods where investment decisions are a synthesis of “automated” and “manual” processes.  I expect the scope of these “flavors” of automated investing to not only increase, but to accelerate in the near term.

It is important to note, however, that for the foreseeable future, the ultimate arbiters of algorithmic investing and portfolio optimization will be human.  The software architects and developers will exercise significant influence on the methodology behind the fund and portfolio optimization software.  Furthermore, the users of the software will have supreme control over what parameters go into the optimization process such as including or excluding or bounding certain assets and asset classes (amongst many other factors under their direct control).

That being said, the future of investing will be increasingly the domain of financial engineers, software developers and testers, and people with skills in financial mathematics, statistics, algorithms, data structures, GUIs, web interfaces and usability. Additionally, the financial software automation revolution will have profound impacts on legal professionals and marketers in the financial domain, as well as more modest impacts on accountants and IT professionals.

Some financial professionals will take the initiative and find a place on the leading edge of the financial automation revolution. It is likely to be a wild but lucrative ride. Others will seek the short-term comfort of tradition. They may be able to retain many of their current clients through sheer charisma and inertia, but may find it increasingly difficult the appeal to younger affluent clients steeped in a culture of technology.

HALO Portfolio Software Availability

We’ve received several requests from individuals about obtaining access to HAL0 Portfolio-Optimization software. We’re very appreciative of the interest!

In the long-term Sigma1 Financial might consider a SaaS (software as a service) model for individuals. Currently, however, Sigma1 Financial is only pursuing select relationships with financial professionals.

If you are an financial professional interested in HALO software, please use this Contact Form to contact us.  Also if you are a web software developer or business manager with experience in sales to financial companies who wishes to learn more about opportunities with Sigma1 Financial, please contact us as well.

Regardless of your financial background, we welcome comments and questions from our readers.  We use a lot of math in this blog and naturally make a few mistakes on occasion.  We love to get feedback on any mathematical typos, oversights, or just plain errors; and we strive to correct them as quickly as possible.

Portfolio Software Development: Day 3

Portfolio Software: Plain English

Yesterday I wrote an early version of financial software to help users improve their investing portfolios.  This software has the ability to solve financial problems in a very different way than taught in graduate-level finance classes.   Rather than relying solely on a type of mathematics called statistical analysis, Sigma1 software uses techniques from computer science called artificial intelligence or AI.  (I prefer the term machine intelligence because there is nothing artificial about the intelligence results produced by a solid AI algorithm.  If you doubt this, I challenge you to beat Chessmaster 11 running on your PC… on max difficulty.)

My idea has been to develop a sophisticated program that would allow institutional investors such as fund managers to  “plug in” their proprietary valuation models and come up with solid portfolios in minutes or hours, rather than days or weeks using brute-force techniques.    As I was working, I realized that smaller “normal” investors could also benefit from a simplified version of Sigma1 software.

Rather than sell this lite version of portfolio-opt software I may provide a free version on a website.  The free version would have limitations on both the number of securities and the “depth” of analysis and reporting.  For example the user may only be able to enter a maximum of 20 securities in their current (or proposed) starting portfolio.  The free web version would quickly suggest an asset-allocation mix of those securities that is (potentially) safer with the same expected return or (potentially) equally safe with a higher expected return.

If the free web version is popular enough, Sigma1 may introduce a paid web subscription service that allows a larger portfolio, a wider selection of securities, more detailed reports and even sample portfolios to “blend” with the investor’s favorite tickers.

Even after the free web version is released, I plan to refine the advanced institutional version of the software.  I plan to use it to improve the composition of the Sigma1 proprietary trading fund.  I also intend to develop a world-class product that institutional investors will want to have access to… for a very reasonable price.

At this time I have zero interest in sharing the source code or specific concepts underlying the current and future Sigma1 software.  Many of these ideas stem from work in my undergrad engineering and computer science studies.  They have developed in my graduate work in finance and engineering.   The realization that the techniques I have developed for engineering, game-theory, poker and number theory apply most directly to portfolio construction and optimization hints at the possibility that I have hit upon one of those rare ideas that strike gold.  Not academic gold; real “gold” with real financial value.

I love academic research and open-source software.  I don’t intend to keep the concepts and code that Sigma1 is developing locked up forever.  If the Sigma1 financial software is financially successful enough, I hope to release pieces of it to the open-source community over time.  (Conversely, if the software does  not ultimately find a lucrative market, I will eventually release it too 🙂 )

Portfolio Optimization Software: Tech Speak

Yesterday I wrote the key pieces of an algorithm to build and optimize securities portfolios.   The remaining pieces: heuristics and selection should be relatively easy to code.   The coding and testing was very quick  1) because I’ve written similar optimizers many times before, 2) because I had 2 days to think about it as I was driving and 3) because I wrote it in Ruby.

Based on previous experience (and depending on the complexity of the heuristics), run-times should be swift for portfolios of 500 securities or less. In previous research I’ve been able to use distributed computing when the heuristics/analysis dominated run-time.  Generally the optimizer has not been the limiting factor for speed.

I plan to start with relatively simple heuristics to test the portfolio-optimization software.  Likely the first test will merely compute the (near-optimal) efficient frontier for a basket of securities, plotting 3-year standard deviation of various portfolios on the frontier versus expected return.  If I wish I may even compare the results to efficient frontiers constructed with classic methods using covariance matrices.

Once I create a Ruby prototype I plan to re-code the software in C/C++, both for execution speed and for the relative IP-protection provided by releasing only compiled binary executables.