Aug 8, 2007: towards an optimal portfolio mix
Why should you diversify your portfolio? How to quantify the diversity? How do we know when enough diversity has been achieved? Highlights:Brief Recap A tribute to Harry Markowitz and MPT Diversity misunderstood Mean variance optimization When less is more Current ETF rankings & model mixed portfolios Next column
Brief recap
In the last column we compared a large number of mini-strategies using back-testing. If you missed that column, here's a link to it.The question we left open was how should we select the components in a portfolio in a way that maximizes our forward-looking risk-adjusted returns.
A tribute to Harry Markowitz and MPT
Over 50 years ago, in 1952, a young finance student in the University of Chicago wrote a 15 page paper with the unassuming name "portfolio selection." The paper, which was largely overlooked at the time, laid the foundation to what came to be known as "Modern Portfolio Theory" (MPT). It took almost 40 more years for that student, Harry Markowitz to be awarded the Sveriges Riksbank Nobel Memorial Prize in Economic Sciences [footnote #1] for this and his other pioneering work on MPT, and making economic decisions under uncertainty.
Harry Markowitz analyzed, quantified, and answered the question "Given possible components, how should an 'optimal' portfolio be constructed from these components?" An optimal portfolio is one that provides the highest expected-return for a given amount of assumed-risk, or conversely that minimizes the assumed risk for a given expected return.
For example: assume there are many different portfolios which are expected to return 10%/year, based on their long term historical averages, but some of them may fluctuate a lot along the way (i.e. be more risky than others). The best of these 10%/year portfolios is one that returns the 10%/year in the straightest line possible, i.e. one that is the most consistent in shorter periods as well.
There's more to the theory, and there were other contributors (William Sharpe, Merton Miller) so I won't try to recite all of their work here. Those interested can read more on Modern Portfolio Theory, and the "efficient frontier" on Wikipedia.
Diversity misunderstood
When I ask people "why should you diversify your portfolio?" It seems that everyone intuitively grasps the first reason: reduction in risk. Yet many, or even most, misunderstand the second effect of diversification: increasing long-term returns.To most people these two aspects seem contradictory: since they know that, as a rule, risk is inversely correlated with return.
The reasons why diversification actually increases long term returns are:
- The uncertainty factor; not all risks increase returns, and one can always be wrong when picking bad risks - gambling and the lottery are good examples.
- The fact that returns (and losses) are compounded
An example should make it clear. Suppose you have to chose between two scenarios:
Which one would you pick?
- One year the market goes up 30%, the other it goes down 10%
- Every year the market goes up exactly 10%
At a first look, the two might look the same over the long term, since the simple average of every two years is 10% in both scenarios:
(30-10) / 2 = (10+10) / 2 = 10But try and compound the two instead of taking simple averages and you'll see how the stable 10% wins:
For 2 years we actually have:
Scenario 1: 1.10 * 1.10 = 1.21 (21% return in 2 years) Scenario 2: 1.30 * 0.90 = 1.17 (17% return in 2 years)Not convinced? See [footnote #2]
Since in investing, nothing is ever guaranteed, diversifying is a simple probabilistic way of making random unpredictable moves of components within the portfolio cancel each other on average and achieve a uniform as possible return per unit of time. Keeping periodic returns as uniform as possible, as we saw above, maximizes long term returns by compounding the small returns.
Mean variance optimization
If you look at the chart of a "top N" ETF portfolio like the one in the July 6th article you may notice a problem. The 9 ETFs charted in this portfolio tend to move together: their 1-year lines rarely cross each other. When they dip, they all tend to dip together. In other words, the correlation between their movements is high. How high? Here is the correlation-matrix of returns between the components, by week, in the 20 weeks prior to picking them:
Backward correlation matrix for the top-9 (as ranked on June 29, 2006) ETF portfolio
[Feb 2006 .. Jun 2006: 20-weeks by week] VOX EWI EWK EWG ADRD EWP EWQ EZU IEV VOX - .64 .70 .73 .71 .74 .73 .74 .75 EWI .64 - .90 .88 .91 .94 .92 .93 .93 EWK .70 .90 - .94 .93 .96 .96 .96 .95 EWG .73 .88 .94 - .95 .95 .98 .98 .97 ADRD .71 .91 .93 .95 - .94 .97 .97 .99 EWP .74 .94 .96 .95 .94 - .96 .98 .97 EWQ .73 .92 .96 .98 .97 .96 - .99 .98 EZU .74 .93 .96 .98 .97 .98 .99 - .99 IEV .75 .93 .95 .97 .99 .97 .98 .99 - Mean correlation: 0.900748As you can see, ETFs like EZU and IEV have been almost identical in their movements (correlation = 0.99). This begs the question: why not pick just one of them for our portfolio. Going from 8 ETFs to 9 ETFs here, contributes almost nothing to diversification.
Harry Markowitz had observed that mixing less correlated assets leads to reduced risk, and better long term returns. He formalized the problem like this: given a set of M components, each with a history of returns, and variations in these returns, plus the correlation-matrix of returns between these M components, can we pick N out of M (where N <= M) such that the portfolio will be close to optimal, as we defined it above, on a risk adjusted basis?
Markowitz then went on to prove that the lowest risk portfolio is the one where the NxN correlation sub-matrix has the smallest mean correlation between components. Thus the term "Mean Variance Optimization" (MVO) was born.
Maximizing the variance within a portfolio is equivalent to minimizing the portfolio matrix mean correlation [footnote #3].
When less is more
For convenience, the correlation matrix above is sorted from left-to-right (and top to bottom) from the least correlated component (relative to all others) to the most correlated one.
This makes it easy to look at strict subsets of the whole portfolio. We could drop any right-end subset of the components and be left with a less redundant portfolio. For example: drop the leftmost 3 components: (EWQ, IEV, EZU), and remain with only 6 components (instead of 9). Net result: a more diversified portfolio! Yes, sometimes less is more.
The top-9 portfolio was a very good choice as a buy-and-hold for 1 year since it was picked: in the period [2006-06-30 .. 2007-06-29] its mean correlation dropped a little bit to 0.859, and it returned 36.70% with a pretty good risk profile [footnote #4]:
Actual 1-year performance stats & correlations for the (June 2006 to June 2007) top-9 ETF portfolio
$ etf-cor -c -l 52w 0w 1w 9 `etf-rank -T -D 52w mmvr 9` Using mmvr ranking method on 20060630 VOX EWP EWI EWK ADRD EWG EWQ EZU IEV VOX - .69 .62 .66 .73 .71 .71 .72 .73 EWP .69 - .83 .85 .85 .86 .84 .89 .89 EWI .62 .83 - .88 .84 .89 .91 .92 .93 EWK .66 .85 .88 - .89 .88 .91 .93 .94 ADRD .73 .85 .84 .89 - .91 .92 .92 .95 EWG .71 .86 .89 .88 .91 - .94 .94 .94 EWQ .71 .84 .91 .91 .92 .94 - .96 .96 EZU .72 .89 .92 .93 .92 .94 .96 - .97 IEV .73 .89 .93 .94 .95 .94 .96 .97 - Mean correlation: 0.859272 Portfolio[9]: VOX EWI EWK ADRD EWG EWP EWQ IEV EZU ROCs[52]: 0.36 3.11 0.88 -3.70 0.74 5.00 0.41 -1.24 3.60 -0.86 2.00 -2.35 1.59 1.30 0.79 0.76 1.22 0.96 0.63 1.52 0.80 0.28 0.10 2.39 1.03 0.60 -1.15 -0.86 1.07 -0.21 0.63 1.33 -0.08 2.68 -4.56 -1.03 -0.61 3.53 2.15 2.01 1.06 2.20 1.09 -0.21 0.70 0.21 1.18 0.64 -4.18 2.49 0.85 -0.66 52w-0w/1w 9 36.70% avg 0.62% std 1.77% SH 0.349 SO 0.455 DD -4.56% MeanCorrelation: 0.859272 Sharpe/Correlation: 0.406071 Sortino/Correlation: 0.529910Thank the long bull market for this. Despite all the components moving together in unison, it managed to do just fine. The question is what would have happened if a downturn had hit.
How did the smaller (yet more diversified) less correlated 6 out of top-9 portfolio do? Don't miss [footnote #5]
Actual 1-year performance stats & correlations for the low-correlation 6 of 9 (June 2006) ETF portfolio
$ etf-cor -c -l 52w 0w 1w 6 `etf-rank -T -D 52w mmvr 9` Using mmvr ranking method on 20060630 VOX EWP EWI EWK ADRD EWG VOX - .69 .62 .66 .73 .71 EWP .69 - .83 .85 .85 .86 EWI .62 .83 - .88 .84 .89 EWK .66 .85 .88 - .89 .88 ADRD .73 .85 .84 .89 - .91 EWG .71 .86 .89 .88 .91 - Mean correlation: 0.806032 Portfolio[6]: VOX EWP EWI EWK ADRD EWG ROCs[52]: 0.27 2.96 0.97 -3.72 0.66 5.05 0.47 -1.21 3.45 -0.77 1.97 -2.08 1.58 1.47 0.73 0.98 1.29 0.83 0.74 1.58 0.60 0.42 -0.08 2.57 1.16 0.45 -0.97 -0.78 1.07 -0.12 0.65 1.34 0.09 2.56 -4.39 -1.36 -0.46 3.41 2.20 1.92 0.99 1.99 0.73 -0.18 0.70 0.33 1.16 0.72 -4.06 2.33 0.85 -0.68 52w-0w/1w 6 37.04% avg 0.62% std 1.73% SH 0.360 SO 0.470 DD -4.39% MeanCorrelation: 0.806032 Sharpe/Correlation: 0.446079 Sortino/Correlation: 0.582569Let's compare the two side by side, since sometimes bigger is better (e.g. returns) and sometimes smaller is better (risk, standard-deviation, mean correlation) I added an improvement column to make the advantage clearer:
Metric Portfolio Improvement Top-9 6-of-9 Mean weekly return: 0.62% 0.62% 0.00% Standard Deviation of weekly return: 1.77% 1.73% 2.25% Total 1-year return: 36.70% 37.04% 0.93% Mean weekly correlation: 0.859 0.806 6.17% Weekly Sharpe ratio: 0.349 0.360 3.15% Weekly Sortino ratio: 0.455 0.504 3.30% Draw-Down (worst week loss) -4.56% -4.39% 3.73% What this shows is that sometimes more components can only hurt performance (even when ignoring increased commissions, and complexity) and that a lower-correlation is good for a portfolio.
Obligatory-notes:
- This is just one data point, with a larger search, I am able to find all kinds of examples, some better, some worse than the top-N portfolio. I hope to be able to collect more data in the future that will help me to better quantify the advantages of good mixing.
- Please don't buy this portfolio today. The present optimal picks are very different. One of the present best mixes is published below.
Current ETF rankings & a diversified ETF portfolio
The market has been sharply down in the past few weeks in a pattern that looks similar to the February correction. Stocks are down 5%-10% with a very high correlation (everything is down except the short ETFs and bonds/fixed-income.) The 10 best performing (not the highest ranking) ETFs in the past 4 weeks of trading have been these:
1 21.10% SJH ProShares UltraShort Russell 2000 Value 2 20.64% SKF ProShares UltraShort Financials 3 19.32% SRS ProShares UltraShort Real Estate 4 16.09% TWM ProShares UltraShort Russell 2000 5 13.56% SKK ProShares UltraShort Russell 2000 Growth 6 13.19% SJL ProShares UltraShort Russell MidCap Value 7 11.81% SDD ProShares UltraShort SmallCap 600 8 10.73% SJF ProShares UltraShort Russell 1000 Value 9 10.50% MZZ ProShares UltraShort Mid 400 10 10.12% SCC ProShares UltraShort Consumer ServicesPlease don't rush and buy these, buying leveraged ETFs after extreme moves up is an invitation for disaster.
The correction has been overdue after several strings of new highs in the broad indexes. Since earnings for the most part have been good, and I see no global signs of slow down yet, I feel it would be premature to call this a new bear market. It had all the markings of a sharp correction: panic, big drops, very high correlation between most ETFs. If this is a correction, this may be a good opportunuty to buy high quality ETFs which you may have wanted to buy anyway, at a discount.
With that in mind, here are last Friday's (Aug 3, 2007) rankings (top 40 ETFs among about 500).
Note: both ranking and portfolio updated late Friday night with more up-to-date results. The suggested optimal portfolio was optimized targetting maximum alpha/beta vs SPY. A target function which seems to produce the best portfolios of all those I tried so far.
Using mmvr ranking method on 20070803 1 2.8472 EWY iShares MSCI South Korea Index 2 2.3886 TTH Telecom HOLDRs 3 2.3146 EWG iShares MSCI Germany Index 4 2.2428 VWO Vanguard Emerging Markets Stock VIPERs 5 2.0398 WMH Wireless HOLDRs 6 1.9702 PPA PowerShares Aerospace & Defense 7 1.9170 ITA iShares Dow Jones US Aerospace & Defense 8 1.8610 EWT iShares MSCI Taiwan Index 9 1.8239 VIS Vanguard Industrials VIPERs 10 1.8165 IAH Internet Architecture HOLDRs 11 1.7172 IXP iShares S&P Global Telecommunications 12 1.6907 EWC iShares MSCI Canada Index 13 1.6884 EWZ iShares MSCI Brazil Index 14 1.6556 EWP iShares MSCI Spain Index 15 1.5022 VOX Vanguard Telecom Services VIPERs 16 1.4494 DND WisdomTree Pacific ex-Japan Total Dividend 17 1.4314 PID PowerShares Intl Dividend Achievers 18 1.4156 XLI SPDR Industrial Select Sector 19 1.3945 PXJ PowerShares Dynamic Oil & Gas Services 20 1.3774 DLS WisdomTree Intl SmallCap Dividend 21 1.3559 EWN iShares MSCI Netherlands Index 22 1.3134 PHO PowerShares Water Resources 23 1.3106 SLX Market Vectors Steel 24 1.3053 EEM iShares MSCI Emerging Markets Index 25 1.2963 EWD iShares MSCI Sweden Index 26 1.2935 VAW Vanguard Materials VIPERs 27 1.2774 BHH B2B Internet HOLDRs 28 1.2661 FEZ streetTRACKS Dow Jones Euro STOXX 50 29 1.2511 PHW PowerShares Dynamic Hardware & Consumer Electronics 30 1.2204 ILF iShares S&P Latin America 40 Index 31 1.2153 PWJ PowerShares Dynamic Mid Cap Growth 32 1.2135 DNH WisdomTree Pacific ex-Japan Hi-Yld Eq 33 1.1859 PKB PowerShares Dynamic Building & Construction 34 1.1802 EZU iShares MSCI EMU Index 35 1.1722 EPP iShares MSCI Pacific ex-Japan 36 1.1601 FXI iShares FTSE/Xinhua China 25 Index 37 1.1491 IYJ iShares Dow Jones US Industrial 38 1.1486 EWS iShares MSCI Singapore Index 39 1.1316 MTK streetTRACKS Morgan Stanley Technology 40 1.1143 PTE PowerShares Dynamic Telecom & WirelessAssuming this is a temporary set-back, here's a good forward-looking very minimal portfolio (only 7 components) picked on 2007-08-03 by combining MMVR ranking with a "maximum alpha/beta_squared vs the SPY" optimization. The output includes a 24-week back-testing of the portfolio. Based on recent performance, including the recent correction (remember: past performance is no guarantee for future performance) this portfolio looks less risky than the SPY (smaller standard-deviation, 19% smaller worst week, 5% less beta, & only 0.64 mean internal correlation) and has a much higher weekly mean return, a 36% alpha, and better risk adjusted return characteristics.
=== Portfolio summary: 10 of 45 (+prunning) RF=mmvr r=0.88 24w-0w/1w Portfolio[7]: PXJ TTH IAH DNH EWY ITA EWC %Change[24]: 2.19 -3.48 -0.23 0.06 3.03 1.74 1.76 0.69 2.06 1.50 0.60 %2.18 0.94 0.96 1.56 -1.61 2.69 0.89 -1.14 3.95 2.09 0.47 -4.58 -1.79 Correlation matrix [24w..0w/1w]: PXJ TTH IAH DNH EWY ITA EWC PXJ - .42 .54 .68 .61 .44 .73 TTH .42 - .57 .40 .49 .74 .66 IAH .54 .57 - .60 .73 .83 .68 DNH .68 .40 .60 - .71 .60 .82 EWY .61 .49 .73 .71 - .77 .74 ITA .44 .74 .83 .60 .77 - .79 EWC .73 .66 .68 .82 .74 .79 - Mean correlation: 0.645444 [24w-0w/1w 7] Portfolio SPY Portfolio-vs-SPY (> 1.0 is better) ------------- --------- ------ ---------------- %Total_Return: 17.36 1.11 15.63 %Annualized: 39.96 2.35 17.03 %Return_Mean: 0.67 0.05 14.54 %Return_StdDev: 2.00 2.01 1.00 %Max_DrawDown: -4.58 -5.47 1.19 %Alpha(annual): 36.99 0.00 - Beta: 0.95 1.00 1.05 R(correlation): 0.95 1.00 1.06 %R2: 89.53 100.00 1.12 Sharpe_ratio: 0.3345 0.0229 14.58 Sortino_ratio: 0.6586 0.0453 14.55Next column: readers ask
In the next column, we'll take a little break from our usual course, and try and answer some very good reader questions. If you have one, please email me and it might make it to the next article.
As always, I hope you found this column useful. Feedback, good or bad, is always more than welcome.
-- ariel
Footnotes
footnote #1: Nobel Memorial prize in Economic Sciences (1990):
The 1990 prize was joint awarded to three MPT pioneers: Harry M. Markowitz, Merton H. Miller, William F. Sharpe
footnote #2: steady wins the race:
An even more obvious example would be two investors, one who completely shuns risk and just wants to preserve his capital (i.e. one who targets a stable "no pain, no gain": 0% per year return) vs. a gambler who loves taking extreme risks. The latter just loves "excitement" and is thrilled to either double his money or lose it all in any given year with equal probability. Both investors have the same simple "average" return of 0%/year since (100% - 100%) / 2 = 0%.
After a few years the gambler may have doubled, or quadrupled his money, but eventually, inevitably, would hit the bad luck draw of -100% and lose all his capital. At this point no doubling would ever get him out of the hole of total ruin since 0 * N is always 0, regardless of N. Obviously, given enough years the risk-averse 0%/year investor, would be way ahead.
footnote #3: On the complexity of the MVO problem, and practical solutions to it:
Optimally selecting N components out of M has a deceptively simple sound to it. Yet, for moderately large numbers N, M it turns out to be a pretty hard selection problem. For the sake of brevity I left the details out of this article.
Interested readers may read this separate more technical article in which I describe two algorithms I have implemented to construct forward-looking, near-optimal portfolios.
The theory behind these 2 algorithms is sound, and the back-testing evidence of both of them selecting good portfolios is strong. My current favorite is to optimize for minimum correlation using simulated annealing.
footnote #4: Dates and methodology of experiments:
Note: all the experiments described above were run at the end of June 2007. All the portfolio picks: i.e. both the top-9 portfolio selection and the 6-out-of-9 low-correlated selection were done based on data available on June 30, 2006 (i.e. a year earlier).
In other words: the selections did not rely in any way on future (at selection time) data. The actual results are the actual statistics on the year that followed both selections (June 30, 2006 to June 29, 2007) i.e. what we know in hindsight.
footnote #5: How did the smaller, less correlated portfolio do?
If you guessed that the 6-component portfolio did better than its 9-component superset, right you are!