Paul Teetor, who guest-blogged here about
seasonal spreads, recently wrote an
article about how to test for cointegration using R. Readers who don't want to pay for a copy of Matlab should find this free alternative with similar syntax quite interesting.
I use R in conjunction with other tools (AmiBroker, Perl) to test econ/market hypothesis all the time. Nice to see it get some love! :-)
ReplyDeleteThanks for this! Loved reading your book, it is very inspiring and practical. Being financial quant professional also interested in trading from home. In fact have IB account. At the moment is very keen on trading commodity futures semi-fundamentally. Would you suggest anything on this? maybe any books or papers.
ReplyDeleteBTW, in book Commodities Investing there is interesting strategy mentioned which takes into account moving averages, steepnesszescore of futures curve and cost of business (say crack spread for refineries for oil)
Thanks, Anonymous.
ReplyDeleteAside from my writings (in my book and blog) on seasonal commodities futures trading, I am not familiar with semi-fundamental commodities strategies. Maybe other readers can suggest some good books?
Ernie
Hi:
ReplyDeleteFrom your blog and book, you seldom mention Java, C++, I am wondering if MATLAB is enough to do pairs trading?
Great post, thanks. Out of curiosity, why do you force the the regression through the zero intercept.
ReplyDeleteDW: Thanks for your comment. I forced the zero intercept because of the economic logic of the spread: If one leg of the spread is priced at zero, the other leg should be zero, too. Or, in common sense terms, if one stock is worth nothing, the related stock should be worth nothing, too. If I did not force a zero intercept, the regression might find a non-zero intercept (just by chance), and the intercept would be meaningless and difficult to interpret; but more seriously it would distort the model.
ReplyDeleteA technical point: The zero-intercept requirement assumes we are modeling asset prices. Modeling interest rate spreads, for example, might very well require a non-zero intercept. I would not model rates using this model, however.
Matt: Thanks for your positive feedback on R. You might be interested in the R Special Interest Group (SIG) for Finance. Their mailing list carries daily discussions on using R for investing and trading. Details at https://stat.ethz.ch/mailman/listinfo/r-sig-finance .
ReplyDeleteAnonymous: I can suggest two books which describe fundamental models useful for futures trading:
ReplyDelete"The Futures Game" by Teweles and Jones contains an example of a model for grain prices.
"A Complete Guide to the Futures Market" by Jack Schwager devotes several chapters to models for livestock, grains, and financials.
BTW, in my humble opinion, no one should trade futures without reading the Teweles and Jones book cover to cover. It is excellent.
Ernie, Paul,
ReplyDeleteI really like the information that both of you have provided over the last few months. I especially like the R code as part of the article. It allows me to answer many, but not all, of my own questions.
From your experience, what is a reasonable "percentage wins" for mean-reverting methods? I realize that it depends on your "take profits", "dump this thang", and other triggers, but do you have a typical "% wins" for a reasonable risk-balanced mean-reverting process?
Bill S
Paul,
ReplyDeleteNot sure if this is the right format but it is related to your article, In your article you mentioned the book by Bernhard Pfaff. I coded up your example and ran on a series I suspected would be cointegrated (IYR and VNQ) and got
Date range is 2004-10-01 to 2009-08-21
Assumed hedge ratio is 1.096113
ADF p-value is 0.01
The spread is likely mean-reverting
No surprise.
Using sprd, I decided to load library (urca) and use ur.df(sprd,type="drift") which seems to give a conflicting result. Curious if you've observed similar or if I am doing something wrong. Thx again.
Anonymous,
ReplyDeleteYou can use any language you like to implement your strategies. I choose Matlab because it is a very high level language. Many institutional traders use Matlab. It is sufficient for any strategies except the truly high frequency end.
Ernie
Bill,
ReplyDeleteI do not calculate % wins for any of my strategies. I rely on Sharpe ratio. I would only trade a strategy with a backtested Sharpe ratio of over 2. Capital allocation to a strategy depends on Kelly formula, which I explained in my book.
Ernie
This has to be the most polite "RTFM" ever...
ReplyDelete"If you have any questions (...), use the help facility to learn more about the function"
---
Also, Ernie, when you say a sharpe ratio of 2, what time frame are you referring to? since the longer the time frame, the easier it becomes to increase the sharpe ratio,[mu*t / sd*(t)^1/2]
Anonymous,
ReplyDeleteIn calculating Sharpe ratio in backtest, I typically sample the returns daily and annualize the Sharpe ratio obtained.
(In performance reporting for a fund, people typically sample returns monthly, which will result in a slightly higher Sharpe.)
Ernie
DW:
ReplyDeleteThanks for mentioning the difference between the adf.test and ur.df functions. The discrepancy has two sources.
First, in ur.df you must set type="trend" rather than type="drift". The adf.test removes both the trend and intercept, which is what type="trend" does. Setting type="drift" only removes the intercept.
Second, you must use the same lag order for both functions. This subject is too complicated to cover easily in a blog posting, so I'll refer you to the documentation.
Bottom line: After loading the IYR and VNQ data, I got identical results when I execute these two function calls:
> adf.test(sprd, alternative="stationary", k=1)
> ur.df(sprd, type="trend")
I hope that helps.
Paul
Thanks for this post. I am an R user, and have attempted to use it for co-integration testing.
ReplyDeleteMy comments are:
1) As Paul suggests, definitely check out the R SIG for finance. Earlier this year, there was an informative exchange of comments on different test methods for co-integration. Basically, the experts warn: "When dealing with real world time series, conflicting results are the norm, not the exception."
2) Regarding co-integration between IYR and VNQ. Their prices may indeed be co-integrated. They are both real estate oriented ETF's. A quick look on Yahoo shows they have many holdings in common. I don't know how this would affect any trading results.
3) R is free, and has an impressive amount of packages developed by users. Nice to see plug for it here!
Ernie,
ReplyDeleteFor a position that I'm currently studying, the annualized Sharpe Ratio is 1.9 (which is OK for me), but the %wins is only 55%.
Since this is a futures position (an integer number of contracts), I use the Kelly number to make sure my total account level is high enough rather than as a bet sizer.
Anyway, it is the 55% wins number that bothers me. The Sharpe number is coming from BOTH the %wins and the fact that the average win dollar amount is statistically greater than the average loss dollar amount (a situation which I have no way to justify).
In the past, I haven't accepted anything less than 60% wins, with roughly equal win/loss dollar amounts. That allows enough bleeding room for unknowns in my model. Until I can justify the win/loss levels or bump the 55% number higher, I think I'll hold off on this position.
Thanks,
Bill S
Bill,
ReplyDeleteYes, futures trading with few trades is quite different from equities trading (my specialty) which typically have hundreds of open positions at a time. Hence we equity traders are not so concerned with % wins, since it is the whole portfolio that matters.
On the other hand, if you do high frequency futures trading with lots of orders, then I also won't be too concerned about winning ratio, since the wins may, as in your case, have much larger profits than the loss per losing trade.
Ernie
Paul said less than 0.05 is mean reverting (more or less), but when i use the adf.test example code:
ReplyDelete> x <- rnorm(1000) # no unit-root
> adf.test(x)
Augmented Dickey-Fuller Test
data: x
Dickey-Fuller = -10.9453, Lag order = 9, p-value = 0.01
alternative hypothesis: stationary
Warning message:
In adf.test(x) : p-value smaller than printed p-value
>
> y <- diffinv(x) # contains a unit-root
> adf.test(y)
Augmented Dickey-Fuller Test
data: y
Dickey-Fuller = -2.1641, Lag order = 9, p-value = 0.5089
alternative hypothesis: stationary
>
how come the one with no unit root has small p-value ?? Thanks
Hi All
ReplyDeleteOn a related point I'm in Australia so we are limited in stocks we can short.
Hence I'm interested in how these mean reversion/cointegration type strategies perform from the long side only.
From your experience and/or practice can a viable stategy be constructed from the long only side based around mean reversion? I guess just take the long side of the pair trade?
Any thoughts?
Thanks
Anonymous,
ReplyDeleteMean reversion can certainly work from the long side only. That is the time-tested technique of buying low and selling high! It works very well in practice.
Cointegration of course works only when there are both long and short sides. I don't think taking just the long side will work.
Ernie
lawrence124:
ReplyDeleteThanks for posting your note.
First, your y variable is a pure random walk, the sum of normally distributed values. A random walk cannot be mean-reverting because it "wanders around" randomly and does not necessarily return to its mean. So the ADF test reported a p-value of 0.5089: the chances that y is mean-reverting are 50/50, and we cannot conclude anything.
Second, the I worded my article incorrectly regarding unit roots, and I apologize. It should say, "If the spread has a root inside the unit circle, the underlying securities are cointegrated." I will correct the article immediately. (The R code is correct, even if I bungled my English.)
I hope that clears any confusion I may have created. Again, thanks for pointing that out.
Paul
Ernie,Paul
ReplyDeleteFrom the angle of trading, do you think R and can anything that Matlab can do?
And besides pairs trading, Ernie, do you trade stocks using techinical indicators like MACD, RSI etc, and do you use any idea like divergence to trade?
Thanks
Anonymous:
ReplyDeleteFrom a theoretical angle, yes, R and MATLAB are identical tools in the sense that any given algorithm can be implemented in either language.
As a practical matter, I think the choice comes down to whether the packages you need are available. The core languages are really secondary compared to the available packages (or "add-ons" or "libraries"), because the packages provide huge leverage on your productivity.
For example: Suppose I want to do real-time trading. If a language has an interface to my broker, I'll consider it using the language, but without that interface it would be too much programming.
Paul:
ReplyDeleteHow abt x variable ?? it is just some random generated numbers within normal distribution. Then, how come adf test says p=0.01 and the null hypothesis is rejected, hence, it is stationary??
Anonymous,
ReplyDeleteThough I have not personally used R, I know that you can use R to automate your trading system by submitting orders to Interactive Brokers. See my previous post here:
http://epchan.blogspot.com/2009/01/algorithmic-trading-technology-update.html
I certainly trade stocks with strategies besides pair-trading, and certainly some of these strategies involve technical indicators.
Ernie
lawrence124:
ReplyDeleteActually, yes, x is stationary. A time series is "stationary" if it has the same distribution, no matter where (in time) we sample from the data. The x = rnorm(1000) data is definitely stationary: No matter where we sample from x, the data is normally distributed with a mean of 0 and a standard deviation of 1.
The ADF test is returning 0.01, indicating that the probability of being non-stationary is only 1%. So we reject the possibility of being non-stationary and conclude the data is very likely stationary.
I hope that helps.
Paul
Ernie and Paul, particularly in light of some of Ernie's comments on stop losses do you agree that a portfolio of pairs trades - based on a cointegration method - should be rebalanced daily?
ReplyDeleteMaybe run the Kelly formula from Ernie's book over the portfolio daily.
Or do you think its best to put on your ten or so pairs and leave them alone till they get back to 1.0 stddev and then exit?
Thanks
David
David,
ReplyDeleteAs a risk management measure, I agree that you should rebalance your portfolio daily based on Kelly's formula. But if your leverage is far below the optimal f, you may not need to.
Ernie
Good article. I too am doing the same thing with R (coint using the adf). A couple of questions:
ReplyDelete-Did you have any method to choosing your lag param? Too much and you will overfit and reduce the power of the test stats and too little and you'll run risk of bogus results due to serial correlation. Some suggest starting with a high lag and reducing until the test stats indicate it is costly to do so.
-Time period of the cointegration must be important - I am calculating rolling 14 or 30 day cointegration and surely this would be more important given the expected duration of convergence trades... are you cointegrating the entire data set b/c this would be less useful - Cheers FZ
Fazza,
ReplyDeleteBy "lag" I think you mean the lookback period used for computing adf or hedge ratio? If so, I generally choose 1 year, since anything longer will represent a different market condition.
However, cointegration is a concept that applies to a longer period of time. It certainly needs to be many times the expected half life for mean-reversion. So if the expected half life is a few months, then you need at least one year to validate cointegration.
Ernie
Thanks for your response.
ReplyDeleteNo, I wasn;t referring to the actual lookback period (although this point of yours is well taken and I agree... testing over a years worth of daily data makes sense). By lag I mean the adf lag parameter. I understand lag to fix the problem with the original dickey fuller which only tests for 1st order autocorrelation ie. y(t-1). If the order is higher ie. where series is responsive to its remembered values at t-2 (2nd order), t-3 (3rd order), then D-F suffers from "residual correlation" b/c the 1st order difference will be a function of the 2nd order, which is a function of the 3rd order etc etc
To fix above, Augmented D-F adds lagged values of change in y to eliminate this residual trend/autocorrelation. As I understand this is the power of the test. Some use model specification tests like Akaike IC or Bayes IC to decide on how many lags to use to add signal to the test without adding noise, ie. being sure the we don;t reduce significance of the test stats for our test for presence of a unit root
Hope I'm being clear - the thrust of my question is that it may not be wise to choose lag in a arbitrary way for the ADF
Fazza:
ReplyDeleteThanks for your comment on the ADF lag parameter. I agree with your logic and with importance of the parameter. I have studied it and experimented with various settings, but I don't have enough insight (yet) to give a good public answer to your questions. The article suggests a small setting, which is what I use in my trading. As with all tests, I use the ADF as an aid to my judgment, not an absolute answer.
Paul
Fazza,
ReplyDeleteI generally choose the lag to be 1, quite arbitrarily. In building trading models, we try to use as few free parameters as possible. If by using a Bayesian method to estimate lag will reduce the number of arbitrary parameters, I would say that is a great idea.
Ernie
I know you've commented on a pairs regression before, but I'm still trying to determine if there is any bias in some of the choices. For example, I'm looking at one year of daily data for motorola-cisco. The ADF test (I use R) gives a nice low p-value, and the graph of the spread looks sufficiently mean-reverting. However, the price for motorola is in the range 3-9 in the last year, and the price of cisco is in the range 13-24. The last 12 months contains both a down and an up market, so that bias shouldn't be a problem. If I do the regression using log prices, I get a cointegration ratio of 1.8 or so, the pair would be +100 shares motorola, -180 cisco. If I do the regression using just prices, I get a ratio of 0.5 or so. Then the pair would be +100 shares motorola, -50 shares cisco. That's a pretty big difference, and while it's probably possible to make profits with either one, it would be nice to know which one makes more sense. One book in particular (pairs trading) seems to go out of its way to say use logs for the regression.
ReplyDeleteThis also leads to whether pairs profits are path-dependent. The spread for mot-csco has recently been near its highs, and so the strategy (let's use the log regression - ratio is 1.8) would say, go long 180 shares of cisco, and short 100 shares of motorola. now there are 9 different possibilities for the price movement. each either goes up, down, or stays the same (roughly). if either one stays roughly the same, and the other moves enough, then you have some profits. you get more profits if the market moves in the direction of the larger position, ie. cisco. does that mean there is a long bias in this pair? another possible movement is that they both move down, and let's say motorola drops 4%, and cisco 2%. using today's closing prices, the cisco position would lose 180*23.40*.02 = 84.24. the motorola position would gain 100*8.97*.04 = 35.88, for a net loss of 48.36 plus commissions. the value of the spread (log p_mot - 1.8*log p_csco) has actually moved slightly towards the mean, yet the position has a loss. is there a flaw in this puzzle?
jh,
ReplyDeleteIf you regress the log prices, then the hedge ratio represent the relative weights in dollar capital, not number of shares. Hence the ratio of shares of CSCO vs MOT will not be 1.8.
Hope this helps.
Ernie
Hi Ernie
ReplyDeleteI ran a series of cointegration tests based on the code in your book and calculated the half-lifes.
I found some high t-stat pairs but with very large half-lifes (I ran the tests over 252 days of data)
Is there a point where you say even though the t-stat is good (say -4), because the half life is too large (say 112 days) you'd pass on entering the trade?
Should we focus only on smaller half life trades (say less than 30 days)?
Curious
Thanks
Bottom line: After loading the IYR and VNQ data, I got identical results when I execute these two function calls:
ReplyDelete> adf.test(sprd, alternative="stationary", k=1)
> ur.df(sprd, type="trend")
I hope that helps.
Paul
----------------------------------
In relation to the above comment, I tried the FXC vs FXA ETFs from 6/26/2006 to 9/11/09. The adf.test gives me a p-value of 0.381 indicating they are not mean-reverting. The ur.df(sprd, type="trend") shows a p-value of 0.003996, which indicates they are mean-reverting. There appears to be a conflict between these two tests.
Thanks, that makes sense. Then whether you use prices or log prices, your positions are beta-neutral.
ReplyDeleteHi Curious,
ReplyDeleteYes, I don't like to trade pairs with long half-life. 30 days is about the maximum.
But that's just because I dislike holding long term positions. If you are like Warren Buffet, you wouldn't mind a pair with half-year half-life.
Ernie
Forgive my not having the benefit of having read the definition in Ernie's book - but is the concept of "half life" a measure of the frequency with which the spread crosses the axis of its long term trend? This crossing frequency was mentioned in Vidyapurthys book, and I understand why you would use this in trade planning and helping parameterise your correlation and cointegration measures, but how (quantitatively) are you calculating it?
ReplyDeleteCheers
fazza,
ReplyDeleteYes, half-life can be considered the average time the spread reverts to the mean. The calculations of the half-life is based on a regression analysis -- the details are in my book.
Ernie
Paul/Ernie,
ReplyDeleteThere was some discussion about parameterising the adf test to eliminate trends (as well as constant) in the cointegrating ratio. But is there a good reason to eliminate trend? The ideal instrument for pair trading should fluctuate around a non-trending mean, where deviations are only caused by error. Anyone who has traded pairs knows that the pair ALWAYS reverts to its mean, but if the mean is changing over time you may end up with losses! My concern is that by eliminating/neutralising trend in test results we will end up identifying pairs with very low p-values (ie. suggesting stationarity) but with strongly trending ratios...
Thanks! Fazza
Fazza,
ReplyDeleteI take care of trend by updating means and stds everyday using a moving lookback window. So yes, there is an argument to be made to allow trend in the cointegration test as well. Please let us know if that works better in your backtest or actual trading!
Ernie
Ernie,
ReplyDeleteThanks for your reply - I will include both cases in my backtesting and let you know which prove a better predictor of a pairs profitability
Regards - FZ
Paul,
ReplyDeleteI looked at your website. I have questions about some of your other methods (and R routines), however there's no simple way to get in touch with you. Could you add a "Comments Section" to your website posts so questions could be asked? Or, send me an e-mail at:
xls5929 "at" hotmail "dot" com
If you're trying to avoid one-to-one e-mails, I understand.
Thanks,
Bill S
Fazza:
ReplyDeleteLike Ernie, I detrend my spread data before testing for a unit root (via the ADF test). This gives a more meaningful result, and it creates more trading opportunities. I must be careful, of course, because the spread is not reverting to a static mean now, but rather to the linear fit through the data. In other words, I am trading the reversion to a target which is moving along a line, so target changes daily, albeit slowly. I check my trades and targets daily.
Maybe that sounds complicated, but it's not.
Paul
This comment has been removed by the author.
ReplyDeletePaul,
ReplyDeleteThanks for your explanation, which makes perfect sense. Its understandable that detrending creates more trading opportunities but we need to be aware that a pair with a trending ratio is not cointegrated in the strictest sense; we are just recalibrating the test so it treats reversion to the trend the same as reversion a stable mean.
Perhaps its more useful to say: when identifying tradeable pairs from detrended data, take care to avoid volatile/strong trends and for "drifting" trends don;t trade against the direction of this trend.
Hi Paul,
ReplyDeleteI am not completely familiar with R, and I have a question with your code... You have sprd = t$gdx - beta * t$gld
Why do you use the 't$' prefix?
Thanks
Steve
Steve:
ReplyDeleteThe historical data is held in a variable called t, which is a data frame. A data frame consists of several columns, and the individual columns are accessed by name using the dollar sign ($). So t$gdx is the column which contains the GDX data, and t$gld is the column which contains the GLD data.
I use the contents of t because it contains the merged data, unlike the raw data in the simple gdx and gld variables.
Paul
That's very helpful. Great stuff! Thanks!
ReplyDeleteHi Ernie,
ReplyDeleteI have a question on how to set up a cointegration test. Should the regression use the actual price or the log of price? Would it be better just to test the price ratio? Thanks
Anonymous,
ReplyDeleteYou can use either price or log price for your cointegration test.
I don't know what you mean by using price ratios though. If I understand you correctly, price ratios no different from prices.
Ernie
By price ratio, I mean price of security A divided by B... versus A less B
ReplyDeleteAnonymous,
ReplyDeleteUsing price ratios is not as accurate as using linear regression on prices (i.e. least square fit).
Ernie
Hi Ernie,
ReplyDeleteThanks for your book! Very interesting.
I tried to run the script about cointegration (chapter 7). I first downloaded the pakage at the spatial econometrics website.
Here is the error I receive:
??? Undefined function or method 'lag' for input arguments of type
'double'.
Error in ==> cadf at 51
z = trimr(lag(r,1),1,0) ;
Error in ==> test at 54
res=cadf(adjcls(:, 1), adjcls(:, 2), 0, 1); % run cointegration check
using augmented Dickey-Fuller test
Can you help please?
Thanks
Hi Stephane,
ReplyDeleteYou need to add all the subfolders of the package to your Matlab path. Furthermore, you may need to add my util folder which I posted on my website to your path as well.
Ernie
Hi Ernie,
ReplyDeleteThanks, it works now.
I have a question about the basket arbitrage on XLE you describe in the premium area.
how do you test a basket of 10 stocks against the index and how do you find the hedge ratio?
Thanks
Stephane
Hi Stephane,
ReplyDeleteThat test is done using multivariate linear regression fit of the index against the prices of the 10 stocks. The hedge ratios are just the regression coefficients that result.
Ernie
Jez,
ReplyDeleteAny trading strategy, reversal or trending, has the problem that it works until it doesn't! In fact, from my experience, trending systems have generally shorter shelf life than mean-reverting strategies, for reasons I detailed in chapter 7 of my book.
Ernie
Thanks Ernie,
ReplyDeleteWhich tyest do you use for the multi regression?
Thanks
Stephane
Stephane,
ReplyDeleteAfter multiple regression is done and hedge ratios found, I simply use adf test to determine if the combination is stationary.
Ernie
Ernie,
ReplyDeleteWhich fonction do you use to do a multiregression and find HR in Matlab?
Thanks
Ernie,
ReplyDeleteDo you still trade GLD/GDX?
It seems it is no more cointegrated since it fails the cadf test.
Stephane,
ReplyDeleteIf you use the spatial-econometrics.com toolbox, it is ols.
For Matlab statistics toolbox, it is regress.
I no longer trade GLD-GDX for the reason you mentioned.
Ernie
Ernie/Paul,
ReplyDeleteWith your effectively detrended approach, is your mean something like the average spread over the last x days? And you use spread stdev over last x days to determine trading opportunities?
where spread = Stk1 - hedgeratio*Stk2
If so, what do you think is reasonable for x? How far back would you go when testing for cointegration?
(All of this is assuming you look at daily data)
Thank you for any suggestions.
-Pete
Pete:
ReplyDeleteYes, I do use daily data.
I detrend over the entire dataset, not merely the most recent N days. I test for cointegration (unit root) over the entire history, too.
It's funny you ask this because I was just speaking with Mohamed Amezziane of DePaul University, and he pointed out that testing over multi-year datasets might be misleading because market dynamics are constantly changing. Let me know if you try testing shorter-horizon datasets.
BTW, detrending a time series in R is pretty simple. Just extract the residuals from a linear model of the data where the dependent variable is time. If y is your time series data, this should do it:
t <- seq_along(y)
detrend <- resid(lm(y ~ t))
Paul
Paul,
ReplyDeleteThanks very much for your response. After re-reading it, I realize that I wasn't very clear with my comment. In fact, with my limited understanding, my question was far more elementary.
My reference to your 'detrending' data when testing for cointegration was in regards to the way the DF test works in your "Using R..." article. If I understand correctly, the DF test (not ADF) considers spread changes relative to the previous spread level when testing for a unit root:
delta_spd(t) = X * spd(t-1) + e
-if X < 0, process is stationary (i.e. on balance, changes in the spread are more positive for lower absolute spread levels and vice versa)
Other than the differencing done with delta_spd(t), do you (or the DF test) do any other kind of detrending prior to running the test?
Also, with a moving target mean, how do you determine the length of the lookback period to use for calculating the moving average and standard deviation of the spread level when creating trading rules?
Thanks again for your contributions.
-Pete
Paul,
ReplyDeleteThanks for this great article. I have been trying to replicate Ernie's results (using the same data set for GLD and GDX) in R following your article but I do not obtain the same results and I would have liked to get some help.
In R I get:
ht <- adf.test(sprd, alternative="stationary", k=1)
test statistics: -4.2917
p-value: 0.01
note: I get the same results with CADFtest and adf.urca so everything seems fine.
In Matlab using the dfTSTest function:
[H,pValue,TestStat,CriticalValue] = dfTSTest(sprd,1,0.01)
test statistics: -4.2917
critical value: -3.4232
p-value:0.0042
In Matlab using the cadf function:
res = cadf(adjcls(:, 1), adjcls(:, 2), 0, 1);
test statistics -3.5176
AR(1) estimate: -0.0653
In Matlab, using the adf function:
res = adf(sprd, 0, 1);
test statistics: -3.8984
AR(1) estimate: 0.9179
I assume the explanation for this is very simple but I could not find it from the help pages of cadf and adf, so if you knew the reason for this discrepancy that would help me a lot.
Also, how can I obtain the critical values from adf.test? I only managed to get them using adf.urca.
Thanks a lot for your contribution,
A.
Hi Ernie,
ReplyDeleteI am reading your book and trying to working through example3_6.m and example3_6_1.m matlab codes, however it seems that there are no "ols" and "cadf" function for Matlab. Could you let me know how to find these functions so that I can test out the codes? Btw, what is "ols" and "cadf" supposed to output? Many thanks.
Hi Anon,
ReplyDeleteThose functions are from spatial-econometrics.com, free for download.
Ernie
Thanks Ernie!
ReplyDelete@Paul Teetor
ReplyDeleteThank you very much for your guidance on this. I ran your exact script in R, but I got that the hedge ratio was 1.932818 and the p-value was 0.2609425. Is there a setting in R that I need to change to duplicate your results?
I wanted to know because I will be working on automating these calculations. :)
@arnaudb
ReplyDeleteIf I understand your question, you are wondering about the discrepancies between Matlab's results and R's results (yes?). Unfortunately, I cannot easily comment on differences between the systems, mostly because I am not familiar with Matlab, but also because there is no "universal" implementation of the ADF test. Even within R, the multiple implementations of ADF can given varying results. I am not surprised that the two systems give different answers.
As for critical values, the object returned by adf.test does not contain the critical value. You could display the source for adf.test to see the table of critical values. You could even copy the source and modify it to include the CV in the returned object -- one of the cool benefits of using an open source system.
Paul
@RogerYang
ReplyDeleteThe script results depend upon the input data, of course. I ran the script for data from 2006-05-23 through 2009-08-10. If you run the script for a different range of dates, you'll get different results.
Try running for the same date range as the original article. If you still get different results, let me know.
BTW: The GLD/GDX pair was known to be mean-reverting a while ago. Then it stopped, and my software reports it is no longer mean-reverting. Interestingly, however, if I detrend the spread, the detrended data are mean-reverting. I leave the interpretation of this curious development to the reader.
Paul
@Paul
ReplyDeleteThanks a lot for your answer. Indeed, learning more about adf tests I realized there were several implementations and that results were not directly comparable. Still, even if I cannot replicate the results, I would like to have the same testing procedure. In matlab I start by testing whether both series are I(1) with a plain adf test and then I use the cadf function (cointegrated adf test) to test that a linear combination of them is I(0). adf.test, ur.df and others can perform a plain adf test but I do not know what could be the equivalent of the cadf function... Therefore until now I tested whether the spread I derived myself is I(0)... I would be interested in knowing whether this is the good provedure and how you personally proceed to test that series are I(1) and that a linear combination of them is I(0).
For the CVs I simply use ur.df, it works perfectly for this purpose.
Thanks,
Arnaud
Question for Paul:
ReplyDeleteI noticed in your code that you read the data in using the read.csv method instead of the read.zoo method. Is there a reason for your preference not to use read.zoo.
I'm sure you know, but with read.zoo you can take care of the as.date issue in the same line thusly:
GOOG <- read.zoo ("C:/R/TESTDATA/GOOG.csv", format="%Y-%m-%d", header = TRUE, sep =",")
Also for Ernie. I'm reading your book on my Kindle. I haven't gotten to chapter 7 yet, but that seems to be where all the fun is.
Hi Ernie,
ReplyDeleteWhat is the ideal period over which 2 stocks should be cointegrated. I have checked different periods and found that different pairs of stocks are cointegrated over different periods of time For eg. stk A and stk B may be cointegrated based on 1 year price data but not based on last 6 months' price data.
Also when spread = residual/ std dev of residuals, then over what period should such std dev be calculated.
Thnx
Hi Anon,
ReplyDeleteI would measure cointegration over at least 1 year worth of data, if not longer.
For computing standard deviation, the lookback depends on the halflife of mean reversion. It should be approximately the same as the halflife.
Ernie
Hi Ernie,
ReplyDeleteThe half life sometimes comes as negative. What could be the reason?
Thnx
Hi Anon,
ReplyDeleteNegative half-life means that your time series is not mean-reverting.
Ernie
Question for Paul:
ReplyDeleteHow does one calculate half-life using r? I have read Ernie's book but am using R (thanks to your code) to learn about and implement the models and am struggling with half-life calculations. Thanks
Regarding my question about calculating half-life: I figured it out. I think my data is slightly different than what is in the book since I get a half-life of 9.7902
ReplyDelete@Anonymous: I apologize for not replying promptly. I did not notice your question until today.
ReplyDeleteI'm glad the half-life calculation worked for you (eventually). Perhaps Ernie still uses that particular half-life calculation, but I've given up. The underlying model (the Ornstein-Uhlenbeck process) does not fit my spread data well, so now I use a purely empirical estimation of the half-life. That approach is working better for me. I'll document it when I get a chance. The idea grew out of some work I did with Mohamed Amezziane at DePaul University, studying mean-reverting processes.
Hi Paul,
ReplyDeleteI had a question. I have read through your guide and compared your method of testing for cointegration with that used in papers:
http://www.ljmu.ac.uk/AFE/AFE_docs/ARTCDRH_01051.PDF
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=315619
I am trying to build a basket of components that cointegrates with an ETF. Based on your method I would do the following:
ETF, X1, X2, X3 are time series prices
lm(ETF ~ X1 + X2 + X3 +0, data=t)
sprd = ETF - Beta1*X1 - Beta2*X2 - Beta3*X3
Test for stationarity of the spread (or the residuals of the original regression).
But the papers above establish regressions of Log prices and they incorporate a constant in their regression which you have left out in yours and they run a ADF on that regression. In your opinion is there a benefit in using their approach over yours or vice versa?
Also I was wondering why you suppress the augmented dicker-fuller and just do the basic test.
I would greatly appreciate your insight. Thank you.
Hi Paul -
ReplyDeleteAs for your comment on detrending. I have read through your guide on cointegration on pairs located here:
http://quanttrader.info/public/testForCoint.html
In your comments in this blog you indicate that you detrend your time series spread before running a ADF. In your guide you don't explicitly mention anything on detrending. Is it because the ADF test in R automatically detrends?
I'm just trying to better understand. Thank you very much. It would help me a great deal.
Pete
Dear all,
ReplyDeleteGreat discussion here. Just want your help in understanding better here:
Are pairs with 1.) high chance of cointegration (very negative statistics below critical value in the Matlab CADF test), 2.) high correlation(measure by return correlation) and 3.) small half-life (less than 20 days for 1yr time series daily data) more favorable than pairs with only item 1&3? Alternatively, how does "Correlation" come to help a pair selection?
Q2: Someone mentioned that in calculating half life, the figure could be negative which means no-mean reversion. But is it possible one could get a high negative statistics yet combined with NEGATIVE half-life? What does that mean?
Sony:
ReplyDeleteThanks for your questions.
It seems your first question is essentially asking, does a shorter half-live indicate a pair is more favorable for trading? It indicates the pair could generate profits more quickly. It does not show the pair is more likely to be mean reverting.
As for correlation, it is not useful for trading mean-reverting pairs.
I am not exactly clear what you mean by "negative statistics". In any event, I would never execute a mean-reversion trade on a pair whose half-life was negative. That strikes me as illogical.
hi guys,
ReplyDeleteare there any java libraries that does the cointegration test using ADF?
i have found one that (Suanshu) that uses Johanson test; but not the above.
regards.
issy.
Hi Ernie,
ReplyDeleteI have been following your blog regularly, great work! I don't have Matlab but can use R. I want to use Paul T's R code to test co-integration, but somehow his website seems no longer available.
Do you or anyone else happen to have saved his code/note?
Thanks a lot,
Pete
Pete,
ReplyDeleteMy web site has become unstable and occasionally becomes unavailable. Obviously, that needs to be fixed. In the meantime, that code is still available at the site (when it's up), http://quanttrader.info/public/ .
Paul
@Paul Teetor/@Ernie Chan:
ReplyDeleteThanks for the great post, I like your sites and blog. I'm a new fun of R and quant trading. I understand that adf.test function removes intercept AND trends before unit root test. that means also that if adf.test says it's cointegrated, the mean of spread reverts to a linear equation (Y = a*X + b). In practice, we need to reconstruct this relation by lm(Y ~ X) and update mean everyday to fit this equation. Is my understanding correct ? and just one question stupid maybe, why also to update stds everyday (is it because the trading threshold changed everyday because of trends of spread) ?.
thx in advance
@Paul Teetor/@Ernie Chan:
ReplyDeleteThanks for the great post. I like your sites and blog. I'm a new fun of R and quant trading. I understand that adf.test function removes intercept AND trends before unit root test. that means also that if adf.test says it's cointegrated, the mean of spread reverts to a linear equation (Y = a*X + b). In practice, we need to reconstruct this relation by lm(Y ~ X) and update mean everyday to fit this equation. Is my understanding correct ? and just one question stupid maybe, why also to update stds everyday (is it because the trading threshold changed everyday because of trends of spread) ?.
thx in advance
Ive read your algo trading book and its great! I have recommended it as a great overview of the area. The difficult thing is to get an overview. After reading it you can choose yourself what specialty you want to study more.
ReplyDeleteI got a fundamental question from a trader: why do I look for a signal on something, and trade on a totally another thing (why is the trade signal differently weighted than the spread)??
If I find a cointegrated pair, I also get the hedge ratio. Thus, I can form the
spread = s1 - (hedgeratio * s2)
So I listen to trade signals, calculated where stock1 and s2 has equal weight (typically the signal is s1/s2 and then we check the std dev).
But I trade the spread, where s1 and s2 has different weight (s2 has a weight of Hedgeratio).
Should not the trade signal be calculated from s1 and (hedgeratio * s2)? Now the signal is calculated from s1 and s2. I look for a signal on s1/s2, and trade on a different thing: s1 - hedgeratio * s2???
Hi Anon,
ReplyDeleteYes, you can trade the exact same spread that you use to generate signals.
In my book, I trade a spread with same dollar amount on both sides for simplicity only. Strictly speaking, the shares on the two sides should be weighted by the hedgeRatio.
Ernie
Hi Ernie (and Paul):
ReplyDeleteI am new at this but really enjoying your blog and two books. I too am trying to use R for my analysis, and I found this code for calculating half-lifes:
http://pcweicfa.blogspot.com/2010/08/r-implementation-of-ornstein-uhlenbeck.html
This (after running the fixes he mentions) seems to produce very different results for half-lifes than what you have on your subscriptions/spreads page. Am I missing something or is the code incorrect? Is there anywhere I can find working Ornstein-Uhlenbeck code?
Many, many thanks. Now back to reading your new book!
David
Hi David,
ReplyDeleteThe halflife calculation on my spreads page is based on very old data - only the spread itself is updated live, while the average, stddev, and halflife are all computed using 2007 data. This is deliberate, because we want to see how stationary the spread really is.
I have Matlab code for halflife calculation in my first book example 7.5.
Ernie
Ernie
Thanks Ernie,
ReplyDeleteDo you know of any similar R code?
I tried to re-create the code from your book in R but to no avail.
Thank you in advance.
David
@David:
ReplyDeleteThere are serious problems with the code in the 2010 blog post that you cite; and the blog comments confirm that it was problematic.
Rather than cutting and pasting that experimental code, I suggest using the O-U functions of a proven R package, such as:
- sde package
- ouch package
Good luck with your modeling.
Paul
Dr Ernie,
ReplyDeleteSharpe ratio of 2 is before or after transaction cost for backtesting?
Ernie: Sharpe ratio of 2 is before or after transaction cost?
ReplyDeleteHi cheerful,
ReplyDeleteWe should only be concerned with Sharpe ratio after transaction costs.
Ernie
Hi Ernie (and Paul),
ReplyDeleteI'm trying to reproduce the EWC-hedgeRatio*EWA spread chart in Ernie's book (Figure 2.6) in R using Paul's recommended method of computing the hedgeRatio: lm( EWC ~ EWA + 0 ) and my spread plot looks different from Figure 2.6. Mine goes from low points of -3 to highs of +3.5 . I'm using Adjusted Close prices from Yahoo between "2006-4-4" and "2012-4-9" as used in the book. Perhaps the prices used to generate the plots in the book used in the book were the non-Adjusted prices and thus resulted in a different spread?
Ernie, can you confirm? As well, can you confirm the method Paul recommends for computing the hedge ratio in R is analogous to ols() method from your Matlab implementation?
Thanks!!
Hi Ernie (and Paul),
ReplyDeleteI'm trying to reproduce the EWC-hedgeRatio*EWA spread chart in Ernie's book (Figure 2.6) in R using Paul's recommended method of computing the hedgeRatio: lm( EWC ~ EWA + 0 ) and my spread plot looks different from Figure 2.6. Mine goes from low points of -3 to highs of +3.5 . I'm using Adjusted Close prices from Yahoo between "2006-4-4" and "2012-4-9" as used in the book. Perhaps the prices used to generate the plots in the book used in the book were the non-Adjusted prices and thus resulted in a different spread?
Ernie, can you confirm? As well, can you confirm the method Paul recommends for computing the hedge ratio in R is analogous to ols() method from your Matlab implementation?
Thanks!!
Hi Justin,
ReplyDeleteI definitely used adjusted closes for that backtest. However, please note that Yahoo's adjustment is based on a multiplier, and so if you take spreads (differences) using the current data, the spread will depend on when you download the data and how many adjustments have been made since my backtest.
Yes, lm is the R equivalent of ols of spatial-econometrics.com.
Ernie
Thanks Ernie. One more clarification if you don't mind.
ReplyDeleteI notice in your ols() expression when computing the hedgeRatio, you include a vector of ones along with the 'x' variable vector, specifically the code in your book below Figure 2.5: ols(y, [ x ones(size(x)) ] )
Is this how you tell ols() to *not* use an intercept with the regression, and effectively fit the model of just:
y = beta * x
I ask, because the way to accomplish this in R (from Paul's code) is:
lm( y ~ x + 0 )
and I'm getting confused with how in R i'm effectively using '0''s but in Matlab its apparently '1''s ! ;)
Hi Justin,
ReplyDeleteWhen you wrote "my book", I am assuming you are referring to my second book. In Chapter 2 of that book, I don't think I specified that the y-intercept of the regression between EWA and EWC should be 0. Indeed, as you have noticed, I have included a column of ones in the independent variable of the ols function, indicating that we expect a non-zero intercept. This differs from Paul's R lm fit, which assumes zero intercept.
Ernie
Justin: You could always plug into Matlab and compare the results =)
ReplyDeleteErnie,
ReplyDeleteThanks for that clarification. Yes I was referring to your 2nd book. I'm working through implementing everything in it in R as a way to learn your methods via coding everything as much from scratch as possible! I'm only using R since its basically the programming language I'm most familiar with and feel most comfortable with doing "data wrangling" with, which will be helpful once i attempt applying these pairs trading methods to other datasets which might not be as clean as stock data,etc.
Many thanks!
-Justin
Ernie,
ReplyDeleteThanks for that clarification. Yes I was referring to your 2nd book. I'm working through implementing everything in it in R as a way to learn your methods via coding everything as much from scratch as possible! I'm only using R since its basically the programming language I'm most familiar with and feel most comfortable with doing "data wrangling" with, which will be helpful once i attempt applying these pairs trading methods to other datasets which might not be as clean as stock data,etc.
Many thanks!
-Justin