Sunday, January 07, 2007

Universal Portfolios

Let me describe a portfolio optimization scheme that, over the long run, is supposedly guaranteed to outperform the best stock in the portfolio.

Before we begin, let’s agree that we will rebalance our portfolio every day so that each stock has a fixed percent allocation of capital, just as your favorite financial consultant would have advised you. What this means is that if you own IBM and MSFT, and IBM went up after one day whereas MSFT went down, you should sell some IBM and use the capital to buy some more MSFT. There is a technical term for such portfolios: they are called “constant rebalanced portfolios”. Notice also the similarity with the Kelly criterion which I wrote about before: Kelly criterion asks you to maintain a constant leverage, which is like maintaining a fixed percent allocation between cash (debt) and stock.

But what should the fixed percent allocation be? Here is where the scheme gets interesting. Suppose we start with an equal capital allocation, for lack of any better choice. At the end of the day, your portfolio has a certain net worth. But then you can calculate what the net worth would have turned out if you had started with a different allocation. Indeed, we can run this simulation: try all possible initial allocations, and calculate the hypothetical net worth of the resulting portfolio. Use these hypothetical net worth as weights (after normalizing them by the sum of all net worth), and compute a weighted-average percent allocation. Finally, adopt this weighted average allocation as the new desired allocation and rebalance the portfolio accordingly. So actually the “fixed” percent allocation is not fixed after-all: it gets adjusted daily, but probably not by much. Repeat this process everyday, always calculating a new weighted allocation by simulating various initial allocations since day 1.

This scheme of portfolio optimization can be proven to produce a net worth greater than just holding the best stock, given long enough time. If this sounds like a miracle, it is partly because this is in fact an ingenious result of information theory, and partly because there are various caveats that actually limit its practical application. The proof that it works (at least in theory) is rather technical and I will let the interested reader peruse the original paper published by Prof. Thomas Cover, a noted information theorist from Stanford University. He coined the term “Universal Portfolios” for portfolios rebalanced/optimized with this scheme. Without understanding the mathematical intuition, this scheme may appeal to those who believe in long-term trending behavior of stocks, because if a stock performs very well in the past, we will end up allocating more capital to it in the long run. It may also appeal to those who believe in short-term mean reversal behavior, since in the short-term, we are performing daily rebalancing of the stock positions based on an approximately constant allocation. However, this seeming confirmation of either trending or mean-reverting characteristics of stock prices is illusory – this scheme is supposed to work even if the stock prices are totally random! How can we manage to squeeze out a gain even with random price series? Remember that we have done the opposite before (see my earlier articles): we manage to lose money even when a price series exhibits a geometric random walk. So it is not too surprising that we can also make money using similar information theoretic juggling.

Now for the caveats. Every time an information theorist start saying “In the long run, …”, you will be well-advised to ask: How long? In my geometric random walk example where the volatility (standard deviation) of returns every period is 1%, we find that the compounded rate of return is an agonizingly small -0.005% per period. In the case of the universal portfolio scheme, the out-performance over the best stock in the portfolio is similarly dependent on the volatilities of the stocks: the higher the volatility, the faster the out-performance. Let me run a simulation with a portfolio consisting of two ETF’s RTH and OIH. If we were to run the Universal Portfolio scheme from 2001/5/17 – 2006/12/29, I find that the cumulative return is 32% (without transaction cost). Contrast that with just buying-and-holding the best ETF (namely OIH here): the cumulative return is 54%. The Universal Portfolio loses. Does this mean the theory is wrong? Not really: RTH and OIH may just have too low volatility. Herein lies the first practical caveat with the Universal Portfolio scheme: it can take too long to realize its benefit if the volatility is low.

How do we find ETF’s that have high enough volatility to realize the out-performance of Universal Portfolio? Actually, we can simply boost the volatility of RTH and OIH artificially by increasing their leverage. So let’s say we leverage both of them 2x. This means their daily returns and volatilities are both doubled. Now the best ETF (which is still OIH here) has a return of 23% (why is it lower than the un-leveraged case? Remember the formula m-s2/2 in my previous article.) , but the Universal Portfolio has a return of 45%. So now the Universal Portfolio wins. But this is a Pyrrhic victory: if you factor in a transaction cost of 10 basis points, the Universal Portfolio scheme actually returns only 4%. This is the second caveat of Universal Portfolios: because of the frequent rebalancing required, transaction costs tend to eat up all the out-performance.

Now there is a final caveat. The reader may ask why I don’t just pick two stocks instead of two ETF’s to illustrate this scheme. Aren’t most stocks more volatile than ETF’s and therefore much better suited for this scheme? Indeed, most academic papers, including Prof. Cover’s original paper, use a pair of stocks for illustration. But if we do that, we run the risk of introducing survivorship bias. Naturally, if you know ahead of time that none of these two stocks will go bankrupt, the Universal Portfolio scheme may look great. But if you run a simulation where one of the stocks suddenly went bankrupt one day (which tend to be a fairly mathematically discontinuous affair), the Universal Portfolio scheme will most likely not beat holding just the non-bankrupt stock in the beginning. Using ETF’s eliminated this problem. But then ETF’s are far less volatile.

So given all these caveats, is Universal Portfolio really practical? Prof. Cover seems to think so. That’s why he has started a hedge fund to prove it.


Anonymous said...

interesting, I'm getting a 404 error when trying to view the paper :(

Ernie Chan said...

It seems that either Prof. Cover has removed his reprint from public access, or the Stanford server is down. You can read other similar papers by googling Universal Portfolios. -Ernie

Anonymous said...


Why did you use ETFs as opposed to futures on indexes, commodities and interest rates, which would eliminate issues with suvivorship, allow for more volatility with higher leverage and lower transaction costs? Also, did you try different time periods for portfolio rebalancing such as weekly or monthly? Would rebalancing using these time periods provide some benefits?


Ernie Chan said...

Dear Anonymous,
Certainly using futures will reduce transaction cost, which is the main reason why this strategy didn't work well for stocks or ETF's. I will post the results on futures here when I get around to doing the calculations.

I have tried rebalancing at different time period -- the results are not significantly different.

Thanks for your suggestions!

Anonymous said...

Hi Ernie,
Dokuchaev and Savkin have published an improved version of Cover's Ultimate Portfolio theory (in the journal: Insurance:Mathematics and Economics 34 (2004) 409 - 419). Are you able to decode from the heavy maths what their modified version entails?I could not follow the maths..

Ernie Chan said...

Hi Anonymous,
Thank you for the heads-up on the new article on Universal Portfolio. I will see if I have time to read such heavily mathematical article. My past experience has been that the more mathematical an article is, the less likely it will actually be a profitable strategy.

Anonymous said...

Mr. Chan -

Have you tried this concept with the 3x (Direxion or other) funds? Also, if we start with the following:

ETF A - $5000
ETF B - $5000
Cash - $10,000

and both ETF A and ETF B lose value over time, say both drop to:

ETF A - $3000
ETF B - $2000
Cash - $10,000

How should one rebalance?

Ernie Chan said...

If one is to use UP for reallocation, one has to know the history of returns of the various instruments. The latest returns alone are not sufficient.

If one, however, assumes that the allocation is fairly constant, then one should make sure that their percentages in the portfolio are constant. In your example, you need to use cash to buy ETF A and B so that they return to 25% of equity.


Unknown said...

Prof. Cover's papers on portfolio theory are posted at

Unknown said...

sorry... that was one particular paper. here's the larger list:

Kirill said...


the paper that your are referring to is available at the Stanford's web site. The hyperlink should start with "www" instead of "itg":


Ernie Chan said...

Hi Kirill,
Yes, your link works... thanks!

Anonymous said...

> this scheme is supposed to work even if the stock prices are totally random

Well, that is definitely to good to be true..

Unknown said...

Dear Ernest,

Having read your blog a little more and discovered that it's actually quite easy to add comments, I thought I would!

As part of my philosophy of `learning by doing', I coded up a simple example of volatility pumping based on a paper from This of course is a prelude to Cover's Universal Portfolios. The MATLAB code is available from the MATHWORKS website under
example-of-volatility-pumping and contains references.

I am happy to receive comments / bug suggestions.



Ran said...

In a 2004 paper "Can We Learn to Beat the Best Stock" it showed that although the optimal re-balance vector hindsight beats the best stock, when running the Universal portfolio algo by cover it really don't get near the best stock. Surely not in the short term of 3-4 years as their dataset contains. The dataset used in cover original paper is also very very biased.

Ernie Chan said...

Hi M,
Thanks for the link.

Yes, there are various authors who claim they can construct optimal trading strategies that can overcome transaction costs. Unfortunately, I have never been able to verified their results out-of-sample using a representative universe of instruments.

IHateToBurstYourBubble said...

This seems contradictory to me. At first it sounds like a mean reversion strategy:

"if you own IBM and MSFT, and IBM went up after one day whereas MSFT went down, you should sell some IBM and use the capital to buy some more MSFT"

eg: if something goes down (relative to the avg stock in the port), buy more. if it goes up relatively, sell it.

But later, there is this:

"try all possible initial allocations, and calculate the hypothetical net worth of the resulting portfolio. Use these hypothetical net worth as weights (after normalizing them by the sum of all net worth), and compute a weighted-average percent allocation. Finally, adopt this weighted average allocation as the new desired allocation and rebalance the portfolio accordingly."

The easiest hypothetical portfolios to calculate are where each respective stock, except for 1, is 0% and then 1 stock is all that is held. eg: 100%

If this is the criteria for evaluating and re-balancing, the strategy is simple: Buy the stock that has had the largest percent gain since the beginning date of the evaluation as your only holding. Simply hold 100% of your portfolio in the stock that has gone up the most. Re-balancing only happens when another stock has a larger percent gain in the period.

I am confused by the seeming contradictory nature of Kelly-criterion betting, which maximizes geometric returns, which seems congruent with the above scheme (eg: "trend following" and "averaging up"), and the seeming opposite idea of mean reversion, or averaging down, which seems parallel to your IBM/MSFT example, as well as the "Shannons Demon" example on page 203 of "Fortune's Formula", where Shannon seems to indicate a geometric random walk can be exploited in a mean reversion scheme of money management.

Hoping you might clarify..

Ernie Chan said...

The daily rebalancing indeed has a mean reversion character, as the IBM/MSFT example shows.
However, the question is "rebalance to what?" It is rebalancing to an allocation scheme that is constantly changing, and giving more weights to the most profitable allocation in the past. So the allocation scheme itself has a momentum/trending character, but that happens on a slower time scale.

This is actually quite generic in the financial markets: mean reversion can happen in a very short time scale, even milliseconds, but momentum can take weeks. Both can happen simultaneously, but at different time scales.

IHateToBurstYourBubble said...

the question is "rebalance to what?"

Right, and it seems in some simplified way that if you consider the "universe" of investable securities to be be 2 "stocks", A & B, that the rebalancing question immediately boils down to a very simple answer: Own ALL of the highest performing stock since inception. Owning any positive percentage of the lesser performing stock be definition results in lower total wealth.

Abstracting this idea "outward" to include all/any securities, and any time period of arbitrary length, it seems this argument holds of owning 100% of the best performing security, all other allocations result in lower total wealth. eg: Cover's Universal Portfolio theory seems to be a sort of ultimate pathological trend following system.

The only question seems to be "when" to choose as your start date. It seems at different points such as the 90's you would have owned nothing but AOL, and perhaps Priceline or Amazon recently.

To me, this has a whiff of Kelly "overbetting" which inevitably leads to ruin, especially if you have chosen to leverage something like the 12.5X (full Kelly bet) as stated in another post. (Overbetting is something that I believe that a competitive society compels its institutions to engage in, and leads to virtually all financial catastrophes, such as Tulip-mania, 2008 housing collapse, etc. Personally believe the 2008 bailouts have "institutionalized" overbetting in the US financial system.)

I guess I am having a hard time rationalizing Cover's Universal Portfolios, which seems to have a pathological 100% single security solution, and Kelly criterion, which NEVER has a 100% allocation when there is ANY positive chance of loss.

Ernie Chan said...

Universal portfolio scheme by Cover would *not* recommend that the allocation be set to just trading the "best" stock at 100%. It just gives the highest weight to this allocation scheme, but all other less profitable schemes also get their weights based on how profitable they were.