I once was a subscriber to Bob Prechter’s Elliott Wave Theorist, though not for a couple of years. One of his themes at the time (and probably still) was “All the same market”. He commented on the way in which more and more assets were moving with greater correlation than in the past. In this post we perform a reality check on futures market correlation.

I have been exploring portfolio-level correlation filters for my trading system. This is not for risk-control per se, it is aimed at reducing portfolio volatility so I can increase cumulative average growth rate (see Ralph Vince’s “Leverage Space Trading Model”, pp50-54).

Correlations change over time, so I wanted to take a look to see how much and how fast so I can get a handle on how frequently I need to check correlations. This is some interesting data I found …

## Correlation Coefficient Heatmaps

Each of the following 3 charts shows a heatmap of the Pearson correlation coefficient matrix of 40 futures contracts (although it includes both ES and SP) over a 1,000 bar period. The colors range from dark red (-1) to white (+1) with zero being orange. The histogram in the top right indicates the distribution of coefficients. The first chart covers 1999 – 2002, the second 2003 – 2006 and the third is 2007 – 2010. Each chart is roughly 3 years.

### Baseline

In the first chart we see a lot of orange. The histogram is nicely centered and symmetrical about zero. There are some correlations you would expect (energies; meats; grains; Euro, Swiss and Pound; USD vs other currencies; etc), but generally these contracts show the low correlation for which futures are renowned.

### Building

Now we are beginning to see stronger sector correlations, only the dollar is showing any real -ve correlations. The metals now stand out, and the currencies have formed an anti-dollar block. The dollar now shows significant negative correlation to almost half the portfolio. Only the softs remain as an uncorrelated group. The histogram at top right has become skewed to the right, though it is still centered on zero.

### All the Same Market?

The final chart sees the evolution continue. It looks like the world of futures vs US$, Yen, the Note and the Bond! The histogram now has a left skew as the mode is solidly in positive territory. This seems like the perfect illustration of Prechter’s point.

### Details

These data were derived from the “free” portfolio of futures provided by CSI (it still includes Bellies, which I hope they will soon replace). I constructed a ratio-adjusted continuous contract. I constructed a date-synchronized matrix of closing prices then removed any rows that are not “full” of data. I then calculated the log difference for each period (simple division of the matrix by a version of itself shifted by a row). Then I used R’s “cor” function which produces the 40 x 40 correlation matrix in two shakes of a lamb’s tail. I submitted the results to the “heatmap.2” function in the “gplots” package.

I also spot-checked the significance of a handful of the correlation coefficients (about 100 of them) bootstrapping the series to find a confidence interval and permutating the series to estimate p-values.

In the bootstrap approach I created 500 new series by randomly selecting paired measurements (with replacement) from the subject series. I then calculated the correlation coefficient of each synthetic series. I used the 5th and 95th percentiles of the correlation coefficients to create a confidence interval.

In the permutation approach I created 500 new series by taking one of the series and randomly re-ordering it then I paired it with an original series to create randomly paired data. I then calculated the correlation coefficients of each new series. I compared the distribution of the coefficients from the random series with the coefficient from the actual data to estimate the p-value of the original coefficient.

The majority of correlation coefficients with a magnitude of greater than .4 or so were significant.

Jez,

Figured out an answer re: colors.

Set breaks=seq(-1, 1, length=21) to fix the color range into 20 buckets from -1 to 1. Also set symkey=T (it supposedly defaults to TRUE) to force the color key to display the entire palette from -1 to 1.

Here's a chat board message from October 2010 which also cites the exact same pp. 50-54 of Ralph Vince's book "Leverage Space Trading Model", for the exact same reason: to reduce volatility, allowing the trader to increase CAGR. What a coincidence! The URL is http://bit.ly/fB8Pls (shortened for ease of copy/paste)

No worries.

Glad to see that you get inspiration from my post then ðŸ˜‰

I can only recommend a CSI subscription. It's very goodd if you want to test different methods for roll-overs, have access to all the contracts (ie different months, etc.)

The R code would definitely be interesting to see for a "R newbie" like me… I'll check that Project Euler. By the way, do you know if there is a way to force the range of colors to go from -1 to +1 (even though the data in the matrix might only go from -.3 to +1, is so that -.3 is not assigned the bright red color but closer to orange)? Could not find an option for this even in the heatmap.2 documentation.

ps: a subscribe to comments functionality would be great (not sure if possible on blogspopt though?…)

Hi Jez, thanks for commenting.

Of course, I 'alf-inched the heat map idea from your recent post! I left the defaults for rowv=T and colv=T, but switched the dendrogram display off (dendrogram="none"). This is what makes for the clear visual impact.

RE: Ratio Adjusted Contracts. Yes I reverse engineered them from the Panama closes compared to the Raw Closes. I know you are a CSI subscriber – I need to make that leap, it would be so much easier and I could construct a much better (larger) portfolio.

For this exercise, I just blocked the data into 3 consecutive 1000-bar chunks. I will post the R code for you.

As an aside, I really began to make progress with R when I discovered "Project Euler". It's a collection of math problems that lend themselves to programmatic solutions.

Cool use of the heatmap.2 function!

It's very interesting how you can see visually (therefore quickly) the evolution from a more uniform shade of orange (less correlation) to a higher contrast red/white image (higher absolute correlations), as you mention.

PS: did you reverse engineer the point-based adjusted CSI futures to create a ratio-adjusted set?

Curious at to what you used for the rest of the data manipulation (rolling algorithm, synchronising and cleaning the data with missing complete rows, running the significance tests). All using R?

I'm quickly "falling in love" with it but still feel low on the learning curve…