In my recent book, I highlighted a difference between cointegration (pair) trading of price spreads and log price spreads. Suppose the price spread hA*yA-hB*yB of two stocks A and B is stationary. We should just keep the number of shares of stocks A and B fixed, in the ratio hA:hB, and short this spread when it is much higher than average, and long this spread when it is much lower. On the other hand, for a stationary log price spread hA*log(yA)-hB*log(yB), we need to keep the market values of stocks A and B fixed, in the ratio hA:hB, which means that at the end of every bar, we need to rebalance the shares of A and B due to price changes.
For most cointegrating pairs that I have studied, both the price spreads and the log price spreads are stationary, so it doesn't matter which one we use for our trading strategy. However, for an unusual pair where its log price spread cointegrates but price spread does not (Hat tip: Adam G. for drawing my attention to one such example), the implication is quite significant. A stationary price spread means that prices differences are mean-reverting, a stationary log price spread means that returns differences are mean-reverting. For example, if stock A typically grows 2 times as fast as B, but has been growing 2.5 times as fast recently, we can expect the growth rate differential to decrease going forward. We would still short A and long B, but we would exit this position when the growth rates of A vs B return to a 2:1 ratio, and not when the price spread of A vs B returns to a historical mean. In fact, the price spread of A vs B should continue to increase over the long term.
This much is easy to understand. But thanks to a reader Ferenc F. who referred me to a paper by Fernholz and Maguire, I realize there is a simple mathematical relationship between stock A and B in order for their log prices to cointegrate.
Let us start with a formula derived by these authors for the change in log market value P of a portfolio of 2 stocks: d(logP) = hA*d(log(yA))+hB*d(log(yB))+gamma*dt.
The gamma in this equation is
gamma=1/2*(hA*varA + hB*varB), where varA is the variance of stock A minus the variance of the portfolio market value, and ditto for varB.
Note that this formula holds for a portfolio of any two stocks, not just when they are cointegrating. But if they are in fact cointegrating, and if hA and hB are the weights which create the stationary portfolio P, we know that d(logP) cannot have a non-zero long term drift term represented by gamma*dt. So gamma must be zero. Now in order for gamma to be zero, the covariance of the two stocks must be positive (no surprise here) and equal to the average of the variances of the two stocks. I invite the reader to verify this conclusion by expressing the variance of the portfolio market value in terms of the variances of the individual stocks and their covariance, and also to extend it to a portfolio with N stocks. This cointegration test for log prices is certainly simpler than the usual CADF or Johansen tests! (The price to pay for this simplicity? We must assume normal distributions of returns.)
===
My online Quantitative Momentum Strategies workshop will be offered on December 2-4. Please visit epchan.com/my-workshops for registration details.
Ernie,
ReplyDeleteYou wrote,
"For example, if stock A typically grows 2 times as fast as B, but has been growing 2.5 times as fast recently, we can expect the growth rate differential to decrease going forward. We would still short A and long B, but we would exit this position when the growth rates of A vs B return to a 2:1 ratio, and not when the price spread of A vs B returns to a historical mean. In fact, the price spread of A vs B should continue to increase over the long term."
I believe this explanation is incorrect (no offense!).
Let's assume A and B have fixed long-term growth rates, but that each has an instantaneous growth rate that fluctates randomly around the mean. If stock A typically grows twice as fast as stock B, then the log(A) price series will grow, on average, at twice the linear rate as the log(B) price series. So, viewed in log space, A and B will both tend to rise linearly with some fluctuations around the best fit line, but log(A)'s best fit line will have twice the slope as log(B)'s. Therefore, log(A) will diverge from log(B) over time, and therefore they cannot be cointegrated.
In order for log(A) and log(B) to be cointegrated, A and B must have the same long-term growth rate. Consider the example of two classes of common stock for a single company that trade on different exchanges in different countries. After accounting for forex effects these two stocks must grow at the same rate since they fundamentally represent the same company. Therefore their log-price series will grow at the same rate and their log-prices will be cointegrated. But over a very long period of time both price series should grow exponentially, so their raw price series will diverge because even a small percentage difference between the two will correspond to a large absolute difference compared to their inital values.
- aagold (Adam G.)
Adam,
ReplyDeleteFor 2 stocks with growth rates in the ratio of 2:1, we merely have to keep the ratio of their market values to be 1:2 so that their positions will have the same long-term growth rate.
As I wrote, if log prices are cointegrating, we need to constantly rebalance these positions so that their market values are always in 1:2 ratio.
Ernie
Ernie,
ReplyDeleteLet's separate the discussion of a trading strategy from the discussion of cointegration definition. Let's see if we can agree on the following definitions.
1) If stocks A and B are cointegrated in raw price space with hedge ratio h, then the difference A - h*B will fluctuate randomly around 0 with no drift.
2) If stocks A and B are cointegrated in log price space with hedge ratio h, then the difference log(A) - log(h*B) will fluctuate randomly around 0 with no drift.
My claim is that any two stocks A & B which satisfy definition #2 must have the same long-term growth rate. Consider this example: A=exp(alpha0*t) and B=exp(alpha1*t). The difference in their logs is (alpha0 - alpha1)*t, which has no drift only when alpha0 = alpha1 (i.e., same growth rate).
- Adam
Adam,
ReplyDeleteI believe your definition of cointegration of log prices is incorrect.
The spread in this case is defined as log(A)-h*log(B), not log(A)-log(h*B) as you wrote.
Ernie
Ernie,
ReplyDeleteOk, I see your point. Defined this way, stocks with different growth rates can be cointegrated in log space. The hedge ratio h compensates for the different growth rates.
However, I think the most interesting real life examples where log prices are cointegrated, but raw prices are not, occur when h=1 (i.e., same growth rate). At least that's the case for any real-world examples I can think of.
Regards,
Adam
Ernie,
ReplyDeleteSorry if I'm beating this to death, but in the example you cited with A and B the hedge ratio would be 2. So the log spread would be log(A) - 2*log(B).
You wrote we would exit this position when the growth rate of A reverts to 2 times the growth rate of B, but I think this is incorrect. We should exit the position when the *ratio* A/B^2 reverts to its historical mean (which is equivalent to the log spread returning to its historical mean).
The analagous statement for raw prices is, we would exit the position when the *difference* A - 2*B reverts to its historical mean.
- Adam
Adam,
ReplyDeleteYou are right that I was being imprecise when I said entry/exit signals should be based on differential "growth rates". By growth rates, I don't mean the instantaneous growth rates d(log(P))/dt, but the average growth rate log(P)/t where t is the time since some distant past at the beginning of our backtest period. Since t is the same for both stocks, difference in average growth rates are essentially the same as the difference in log prices.
Ernie
Ernie,
ReplyDeleteON the topic of Kelly leverage. I'm following the example in your book (Quantitative Trading pp. 99) and for SPY I calculate a leverage of 21.08 usinfg the last 252 returns. Is that also what you get?
Anon,
ReplyDeleteYes, that's about what I get too.
Ernie
Anon,
ReplyDeleteI certainly hope you're not planning on investing on SPY with a leverage factor of 21.08! You know that would be absolutely insane, right? Even half-Kelly at 10.5 would be insane.
It might be ok to estimate future variance using the past 252 daily returns, but it's certainly not correct to estimate future expected returns that way.
I use a half-Kelly model to determine how much stock market exposure I should have with my real-life portfolio, and right now it's saying I should be 70% in the US stock market and 30% in cash. My estimate of the market's future daily mean return is 6.05% per year, annualized standard deviation is 15.4%, and risk-free interest rate is 2.71%.
- Adan
Hi Ernie,
ReplyDeleteIt seems Yahoo Finance starts to provide real-time tick data (no delay).
Do you hear any good news or comments about that?
Hi Anon,
ReplyDeleteIndeed Yahoo Finance now offers real-time data, but only from Nasdaq. So if a stock like IBM is primarily traded on NYSE, the Nasdaq price may be slightly different.
Ernie
Hi Ernie,
ReplyDeleteIs that ok to use IB real-time data stream to do pairs trading in US markets?
Hi Ernie
ReplyDeleteAfter reading your books, I find that a pair of futures contracts traded on Shnaghai Futures exchange that is great to extract roll return form. the question is if one is in backwardation and the other is in contango, how do you determine the hedge ratio between them. Are we suppose to belance their spot return fluctuation? Right now I optimize the backtesting sharp ratio,to determine the ratio, any advice?
Ruan Xun
Hi Anon,
ReplyDeleteI find the real-time feed of IB too noisy for pair trading stocks. I recommend IQFeed or even Yahoo Realtime instead.
Ernie
Hi Ruan,
ReplyDeleteYou can use linear regression on their prices (or log prices) to determine the optimal hedge ratio.
You will always be long the one in backwardation, and short the one in contango if you want to extract roll return.
Ernie
Hi Ernie
ReplyDeleteSo should I use continuious futures prices of these 2 contracts,or should I use the 2 underlying spot prices instead?
Hi Ruan,
You can use linear regression on their prices (or log prices) to determine the optimal hedge ratio.
You will always be long the one in backwardation, and short the one in contango if you want to extract roll return.
Ernie
Ruan,
ReplyDeleteYou should not use continuous futures, nor spot prices.
You should be using individual futures contracts to test for roll returns.
Ernie
Ernie,
ReplyDeleteI tested the price spread you talked about in your book A - h*B for stationarity and I find that for a lot of pairs , the stationarity keeps fluctuating from true to false , and vice versa if I retest the pair for stationarity every month (verying lookback windows). Even a stable pair like GLD-GDX that you mentioned isn't very stationary month to month. How does one interpret this result ?
Anon,
ReplyDeleteStationarity tests should involve at least 1 year of daily data. How are saying that even with 1 year of lookback, the test statistic changes greatly from month-to-month?
Ernie
Yes. Using a 1 year lookback & testing for stationarity after every month, I notice that stationary flips from True to False and vice versa on majority of the pairs.
ReplyDeleteIs this something you notice too ?
Anon,
ReplyDeleteUsually stationarity test is more stable than that. Perhaps increasing your lookback to 3 years would help. If not, it simply indicates those pairs are not really stationary.
Ernie
Hi Ernie
ReplyDeletedo you use fundamental data such as PE and ROE etc? Any good service/api recommendation? thanks
Paul
Hi Paul,
ReplyDeleteI don't currently use fundamental data. But I believe you can scrape the Yahoo Finance website for such info. Also, IB's API also provides such data.
Ernie
Have you tried Quantum Mechanics trading?!
ReplyDeletehttp://arxiv.org/abs/1307.6727
Hi Anon,
ReplyDeleteThanks for the article. I find this article lacking in empirical support.
Ernie
Hi Erine
ReplyDeleteI just read the paper of
"Optimal Pairs Trading: A Stochastic Control Approach"
http://www.nt.ntnu.no/users/skoge/prost/proceedings/acc08/data/papers/0479.pdf
As I m not familiar of Ornstein–Uhlenbeck process and its application on pairs trading, so I would like to seek your opinion,
Isn't it true that the OU process can model the spread and the mean reverting behaviour in continuous time and dynamic way but the cointegration approach cannot ,but the weakness of the OU process is it does not tell us what is the weightage of each stock in a pair. Thus,We have to use the stochastic control approach to get this weightage, but we will have to set a final period T when we close the position. Whereas for the cointegration approach, it explicitly shows the weightage of each stock in a pair.
Hi Anon,
ReplyDeleteIndeed the OU process does not tell you the optimal hedge ratio. It is a model of one mean-reverting time series, not the cointegration of several series. The only use for an OU model for me is to extract the halflife of mean reversion from the regression coefficient.
Ernie
Hi Erine
ReplyDeleteWhen you do simulation or backtesting for the pairs, you use tick data or just daily High, Low, Open and Close Price
Thx
James
Hi James,
ReplyDeleteThat depends on whether the mean reverting strategy is daily, or intraday. In either case, you need bid-ask quotes: trade prices will inflate results.
Ernie
Hi Ernie,
ReplyDeleteI do not quite understand the concept of cointegration trading with log prices.
Assuming we have a cointegration relationship
log(A) - 0.5 log(B) = 0
If log(A) - 0.5 log(B) > 0, we will short A and long B.
I do not understand why we have to short A. Our assumption here is the growth rate of A may decrease, and this does not mean that the price of A will drop. Hence, we will make a loss if we short A, as the growth rate of A is still increasing.
In pair trading, whether using raw prices or log prices, we should expect one side to lose while the other side profit.
ReplyDeleteIn your example, hopefully the gain in B will be more than enough to offset the loss in A.
Ernie
Dr Ernie,
ReplyDeleteFor stationary test on USDCAD, you do a logarithmn on it. I read that if we use prices like USDCAD, we should use the difference and hence use a random walk model. If we use returns, we should log it and an exponential random walk model is used. Does it matter if we log or dont log it?
Thank you
Leo
Dr Ernie,
ReplyDeleteI am trying to see the variance ratio test on your data of USDCAD (1min) for different time frame. If I want to look at 60minute time-frame, how do I choose the number of periods for the numerator and denominator based on the below formula? Is it sampling at 60 points for the top and 1 point for the bottom?
VR(k) =Variance(rk t)/k Variance(rt)
Thank you
Leo
Hi Leo,
ReplyDeleteNo, you should just input the 60-min bars into the vratiotest function. This is a hypothesis test, so you can't just compute the ratio and see if it is 1.
Ernie
Dr Ernie,
ReplyDeleteThis is because we can see the variance ratio <1 , =1, >1 and so know the "state" if we use the formula. But if we use vratio from matlab, it can only as you said reject RW with a probability. No way to form the equation with that 1min data for 60min time frame?
Thank you
Leo
maybe I'm missing something in this discussion, but for me a pair is cointegrated if there is a linear combo that is stationary. Stationarity is not the same as driftless. driftless can be non stationary and stationary processes can have drift, i.e., that which is implied by the OU process.
ReplyDeletetechnically, if log(X) and log(Y) cointegrated, one gets a fairly nonlinear expression for the innovation of prices, owing to the fact that aLog(X)-bLog(Y)=Log(X^a/Y^b), then do OU process and express innovation of X in terms of Y. I get that it isn't as simple as what Ernie says in his opening comments.
Marc,
ReplyDeleteYou are right that stationary processes can have polynomial dependence on t. We are only concerned with whether the residual is a bounded function in time.
And you are also right that a driftless process, such as a random walk, is not stationary, since the variance can increase without bound.
When we test for cointegration of log prices, what we are really testing for is whether the returns per period have a linear relationship to each other.
Ernie
Oh, yes, interesting. thanks Ernie Logs is looking nice for a number of reasons now....
ReplyDeleteHi Leo,
ReplyDeleteThe variance ratio is a t-statistic. You need critical values of the t-statistic to test if it is significant. vratiotest provides that (hypothesis) test.
Ernie
Dr Ernie,
ReplyDeleteIn your book, you need holding period and look back period to check for trending. Any statistics to test for trending using lookback period? like the way variance ratio test?
Thank you very much
Leo
Hi Ernie,
ReplyDeleteI have read the post in detail and find it very interesting.
I just wanted to clarify something... can you please define exactly what you mean by "growth rate" of a stock. And also "growth rate differential".
Thanks.
We assume that stock prices grow exponentially. Thus the growth rate is the exponent. This is also ordinarily known as the CAGR (compounded annual growth rate) in financial literature.
ReplyDeleteGrowth rate differentials is the difference of this exponent for 2 different stocks.
Ernie
Hello Ernie, I'm building a mean reversion algorithm based on H1 data and depends which dates I use to check the cointegration I have good or bad results. So, in live trading I have to check each new candle if the pairs are still cointegrated or not? I'm a little bit confused with this topic because maybe you get 1000 candles and the pairs is cointegrated but then I check 1500 and it is not.
ReplyDeleteI think that in GLD-GDX case it happens something like this, that the cointegration was broken and then continue.
Another question is that if I have to recalculate my parameters each candle or maybe it is better once a day/week?
Thanks.
Greetings
Hi Carlos,
ReplyDeleteTypically, cointegration need to be determined only using daily data. It doesn't help to use intraday data, unless you want to liquidate your positions at the close each day.
Even if you were to use daily data, there is no need to update your cointegration test daily. Monthly update is good enough. You can run that on the most recent 3 years of data.
Ernie
Hi Ernie,
ReplyDeleteIn Pairs Trading by Vidyamurthy on page 83, the author describes an elementary example of trading with log prices. However, he seems to use the cointegration coefficient to indicate the ratio of shares to hold rather than to indicate the relative market value of positions (as you state above). As I'm sure you have read this book, can you reconcile this discrepancy?
From the book, with a cointegration coefficient of 1.5, he states "At time t, buy shares of A and short shares of B in the ratio 1:1.5" on page 83. Would very much appreciate your input.
Best,
Flapjacks
Hi Flapjacks,
ReplyDeleteI have not read that book. But based on your description, I can only say that I disagree with his interpretation. For a mathematical justification of my interpretation, please read p. 65 of my book Algorithmic Trading.
Ernie
Hi Dr Ernie:
ReplyDeleteI am your reader. however, I found the allocation ratio for pair trade under log price is difficult to comprehend.
Why we want to make the market value of asset A and B in fixed ratio Ha:Hb?
shall their ratio be always proportional to Ha*Log(Pa)/Hb*Log(Pb)?
Hi Chen,
ReplyDeleteI don't necessarily recommend a fixed market cap ratio. You can choose to keep the ratio of shares to be fixed instead - in this case you won't have to perform rebalancing each day. However, this runs into the danger that your portfolio may have a net exposure over time.
If you do want fixed market cap ratio, the hedge ratio b is determined by a linear regression of their log prices. The ratio you displayed does not seem right.
Ernie
Dr Ernie,
ReplyDeleteI am trying to develop a mean reversion strategy with FX intraday data (1H) between NZDUSD - CHFUSD using log(rt/rt-1) and Johansen. If one log price grows to any standard deviation above the mean, should I enter my positions according to the signs of Johansen test?. As I read, you expect to get profit in one pair and loss in another? Should I use only daily price? Did not cointegration make sense with intraday data as 15m, 30m, 1h?
Hi Camilo,
ReplyDeleteThe Johansen test determines the hedge ratio between your 2 instruments. So yes, you should enter into a position based on the hedge ratio (which is signed).
Both sides may win, lose, or one-side win and the other lose.
The optimal frequency of your data is something you need to backest.
Cointegration is not particularly useful for short time frame, but you can still use it to obtain your hedge ratios (and the half-life of mean reversion.)
For the best way to trade mean reversion, see Cartea, 2015, which I referenced in a recent Tweet.
Ernie
Hi Ernie,
ReplyDeleteI am reading your "Algorithmic Trading", and in page 65 chapter 3, you mentioned that when h1=-h2 in y = h1y1+h2y2, then log(y1/y2) and y1/y2 are indeed stationary...
I can't see why, could you please elaborate more?
By the way the "stationary" you mentioned here means the Hurst exponent does not equal to 0.5 right? or you mean the mean and auto-covariance independent with time?
Thank you very much!
Hi Tianyi,
ReplyDeleteActually there is a typo: I was referring to Equation 3.3, not 3.1. I.e. if the log price of y1 and y2 are cointegrated with hedge ratio = -1, then y1/y2 are stationary.
The stationarity definition I used is that the mean and covariance is independent of time.
Ernie
Dr. Chan,
ReplyDeleteThank you for putting together such an excellent book, and providing such diligent blog responses. They've been a tremendous help!
A variant of this question seems to have been posted earlier, but here goes:
When forming a stationary time series using log prices, as you outline in your book, you advocate holding market values as opposed to ratio of shares. However, while researching this subject, I came across two authors, Professor Ruey Tsay (Booth) and Ganapathy Vidyamurthy (author of Pairs Trading 2004), who seemingly advocate contradictory advice. Mainly, that one should hold a ratio of shares irregardless of the logged price levels. As I'm just beginning my exploration on this topic, I'm hoping that you might add a bit more color to the topic and perhaps specify why the market value method is superior.
I've provided a link to Professor Tsay's online lecture slides (Page 10 and 11), and linked the below unanswered stackexhange thread referencing Vidyamurthy.
http://faculty.chicagobooth.edu/ruey.tsay/teaching/bs41202/sp2012/lec10-12.pdf
http://quant.stackexchange.com/questions/19340/what-does-the-cointegration-coefficient-represent-in-pairs-trading-when-cointegr?newreg=52e62b15fddf4c7ea1c826fa106d3401
Best,
Gyllenhaal
Hi Gyllenhaal,
ReplyDeleteThank you for your kind words on my books and blog!
Using ratio for trading pairs has the virtue that one does not have to adapt the hedge ratio constantly. It is also equivalent to fixing the hedge ratio for log prices at 1. But that also means that one must adjust the market value of the two legs regularly. Also, if the growth rates of the two legs aren't the same, a hedge ratio of 1 won't be optimal.
Ernie
Hi Ernie,
ReplyDeleteFirst of all, thanks for writing such a great blog and those books. Your books and blog opened the quantitative trading world door to me.
I'm currently using linear regression to calculate the optimal hedge ratio between stocks with raw price. As stocks have earning season, the prices processes could be more volatile during those period. As to eliminate the noises during those periods, I wanted to remove the price data during those period in the optimal hedge ratio calculations. However, due to the autoregressive component of the time series data, I cannot simply remove the data in between and join the data. What should be the best method of doing this? Or, Should I even be doing this? My reason of doing this is that I don't want the optimal hedge ratios are fitted with data with too much noises.
I'm considering convert the time series into return space or log return space. Remove the data that fall into the time period , run regression on return or log return. However, I'm not sure the result of the regression on return/log return will give me the correct optimal hedge ratio.
Thanks again for your work and effort.
Sincerely,
Simon
Hi Simon,
ReplyDeleteThanks for your kind words on my writings.
I don't see what's wrong with simply removing those days within the earnings season from your regression fit of prices. Linear regression does not assume the lack or presence of autocorrelations between different data points.
Ernie
Hi Ernie,
ReplyDeleteThanks for the prompt reply. I see your point. Linear regression does not require that assumption. However, in the case of a Johansen Test, should I be worried about the data being cut off during the earning seasons? Because the change from the the last price before the cut off to the first price after the cut off could be misleading, just like the rolling over of the future contracts.
Thanks again for your work.
Sincerely,
Simon Z
Hi Simon,
ReplyDeleteIndeed, you can't just remove those prices from a Johansen test.
If you want to perform Johansen test on a continuous price series while removing those that are in earnings season, you have to adjust for the price gap like the way people piece together futures contracts to form continuous contract. Please see http://epchan.blogspot.com/2015/07/time-series-analysis-and-data-gaps.html
Ernie
Hi Ernie,
ReplyDeleteThanks for the reply again. I shall do that instead.
Another question that I wanted to ask is that why don't we run a linear regression on returns/log returns to determine the optimal hedge ratio? I noticed in your post that you are comparing price vs log price. I have tried to run linear regression using price,log price, return, log return for one pair using 1-min bar. After that I used a ADF test on the price residuals of the regressions( obtain the optimal hedge ratios and calculate residual in terms of price), I realized that although all of the residuals past the ADF with p-value <.01, regression using return or log return gave me much more negative t- statistic than using price/log price. The optimal hedge ratios calculated from these regressions are slightly different.
Well, let me try to answer that question and see if I'm on the right track.
if we convert a price series to a return series, it lost some information about its current level.The regression on return data will not know the current prices of the stocks. The regression is trying to calculate the instantaneous relationships between two stocks returns within a fixed interval. For example, if the optimal hedge ratio between stock A and B is 2:1 using daily data, it suggests that if stock A has a x% price increase in one day, at the same day, we should expect stock B to have a x/2% price increase . If we noticed that stock B over performed or under performed, we might put up a trade to make advantage of it. Am I wrong here?
Thanks for your time reading and answering.
Sincerely,
Simon
Hi Simon,
ReplyDeleteWhen we pair trade stocks, we are not interested in having the returns cancel each other as well as possible. If we find a hedge ratio based on returns, we are doing exactly that. In pair trading, we are interested in mean reversion of the spread. The spread is made of 2 price series, not returns series. We are trading the deviation from a straight line fitted through the scatter plot of the prices. Hence the best way to construct the spread is to use the slope of that straight line as the hedge ratio.
Ernie
Hi Ernie,
ReplyDeleteThanks a lot! This definitely clears my confusion. Thanks again for the great work you put up together.
Hi Ernie,
ReplyDeleteIn your blog above you wrote that for hA*yA-hB*yB, "We should just keep the number of shares of stocks A and B fixed, in the ratio hA:hB, and short this spread..."
I believe that this is only true if you are using a stationary hedge ratio. When you start using dynamically hedged ratios (e.g. with lookback n or kalman filter) and have an open position, you must rebalance your portfolio periodically to best match the new synthetic spread by holding hA in stock A and short hB in stock B?
Can you let me know if my understanding is correct.
Hi Vincent,
ReplyDeleteYes, if you dynamically adapt the hedge ratio, you do need constant rebalancing.
Ernie
Hi Ernie,
ReplyDeleteI love your books and blog comments.
I see for pair trading people often use ADF instead of DF.
What is the rationale?
Ryo
Hi Ryo,
ReplyDeleteThe Dickey-Fuller test assumes the underlying time series model is AR(p). But the Augmented Dickey Fuller test assumes a more general error correction model with both lagged prices and lagged differences.
So in general we should use ADF because of its generality.
Ernie
Ernie,
ReplyDeleteThank you very much for your reply. Please allow me to ask you more questions.
For the two underlying securities, do we need to do ADF test each?
If we do ADF for each, do they have to have the same integrated of order?
Or we need just compute ADF for the pair?
Thank you,
Ryo
Thank you very much for your reply.
ReplyDeletePlease allow me for a few more questions.
When identifying a pair, do we need to do ADF test for each security that they have same integrated of order? Or do we just need to do the test for the pair?
I am guessing that we need same order otherwise we have spurious results.
Or is it not necessary so?
Regards,
Ryo
Ryo,
ReplyDeleteIt is necessary to first run ADF on each price series to ensure that are NOT I(0).
If it is I(0), then you don't need to trade it within a pair!
If it isn't I(0), then you should pair it with another price series and run cadf (not adf) test on the pair.
Ernie
Thank you very much!
ReplyDeleteRyo
Hey Ernie,
ReplyDeleteTrading pairs using ask-bid price, one of the complications that would arise would be whether to use the average of the ask1-bid1 of contract1 and ask2-bid2 of contract 2 price to perform the OLS.
With of these methods or if any other methods exists would be best to perform the OLS? One issue with using the average is that the ask-bid price spread would not be accounted for when a signal is generated to trade. What I mean by this is that the signal would trigger one to short and long a contract and we will be using the bid and ask price respectively to place a limit order. The average price may not accurately account for spread differences of the contract spreads.
Thanks
Hi James,
ReplyDeleteTo find the hedge ratio using OLS, midprices can be used.
When deciding whether we should execute at the current spread (based on your limit price), the spread should be formed with market prices (i.e. bid price for sell order, and ask price for buy).
Best,
Ernie
Hey Ernie,
ReplyDeleteThis question does not pertain to pairs trading, just a general question. I often see strategies mention their capacity constraints. I've been wondering how does one tell what the capacity constraint is for a specific strategy?
Thanks
Hi,
ReplyDeleteGenerally, one cannot trade at more than 1% of average daily volume of a particular instrument. So given the ADV, you can work backward to find out how big your order can be, and therefore how big a position you can have based on how frequently you have to rebalance your position and what % of the position you have to rebalance. The size of the position is your capacity.
Ernie
Hi Ernie,
ReplyDeleteI find CADF hedge ratio often shows values of long and short are quite different, e.g. long $1m stock A and short $2m stock B, resulting in $-1m net delta. Furthermore, when constructing portfolio with long/short pairs, net delta of portfolio is far from zero, putting the portfolio in market risk. Do you find it is often so? My pairs port is currently net short and am long futures to hedge the delta risk. It is bull market and it is very hard to win with net short exposure. I would like to know what your thought is.
Regards,
Ryo
Hi Ryo,
ReplyDeleteTo clarify: CADF does not generate a hedge ratio. For that, you have to use either linear regression coefficent between the stock (log) prices, or the eigenvectors of the Johansen test. If you use either methods to find the hedge ratio, the portfolio should have a beta close to zero. After all, beta is the regression coefficient of returns, and hedge ratio is regression coefficient of prices or log prices. In the latter case, they should be identical.
Ernie
Hi Ernie,
ReplyDeleteI looked up their 3 year beta (3 years is the time span I used to calculate CADF) and the net beta is much closer to zero (5% net short, though).
Thank you,
Ryo
Hi all. this a correspondence i had with Ernie about this issue. I said I would post this in a comment so here it is:
ReplyDeleteQ:
==
I was wondering if there is a difference between using the CADF test and the Johansen test if I want to test two time series for cointegration.
A:
==
cadf test can only be used on a pair of log price series. Johansen test can be used up to a panel of 14 or so, and furthermore output the hedge ratio appropriate for each.
Followup:
=========
So there is no advantage for using CADF over Johansen for two time series? the
output is the same?
A:
==
Output won't be identical since they use different methods, but there is no
advantage in using cadf.
Q:
==
And why only log prices?
A:
==
You can use prices too, but log prices have more normal distributions.
Q:
==
Also, I am wondering on how to go about trading logs of prices. As I understood it, if I run a cointegration test on log prices, the hedge ratios will be the actual capital allocated to each asset, because of the first order approximation, so we are basically trading on the returns. On the other hand, if the series itself is cointegrating, hedge ratios represent the number of units to hold. So as I see it, the difference between the two is the normalization by the asset's price inverse in each time step (for the non log series). Is my understanding correct?
If so, what if both the log series and the time series itself are cointegrating? which hedge ratio should I use? How often do both cointegrate? How often does one cointegrate while the other random walks?
A:
==
When price series cointegrate, log price series almost always (a.a., a mathematical term) cointegrate. So you can just use price series if you want to specify the shares as hedge ratio. Otherwise use log price series, if you want hedge ratio to represents dollars.
Q:
==
In our previous correspondence, and in the answer to the first question, you said that log prices are preferable... Following this answer I'm thinking to test for log series cointegration, and trade on the hedge ratio given by the original so as to not normalize the number of shares in each timestep. is that a correct assessment?
A:
==
If you use log prices and allocate based on dollars, all you need to do is to find out the sum of abs(hedgeRatio), and normalize the dollar amount so that this sums to 1.
followups:
=========
But I need to do this at every time step don't I? as opposed to the non log series..
Also, the way i see it, Johansen on the log series will give the half life time for returns whereas running the test on the non log series will give the time for mean reversion of the series. Is that right?
Thanks again:)
Hi all.
ReplyDeleteI'm trying to rate cointegrating portfolios by their Johansen test results. I have two factors by which to rate: The shortest half life time, and the test's p-value. I was thinking either to score hadge ratios accordingly, or just use the p-value and exclude times that are longer than the duration of the data set fed to the test (or maybe half the data set?).
How would you do it?
love and gratitude
Hi again:) sorry if I'm asking too much, please tell me if I am
ReplyDeleteHow many data ticks would you recommend on using to test for mean reversion?
and how many for strategy optimization?
thanks:)
Hi Ernie and all.
ReplyDeleteI was wondering on how to interpret the Johansen statistics:
Say I'm using three time series, and that the rank of the matrix is significantly 0 (say above 95%), rank = 1 is non- significant, and rank = 3 again highly significant. how should I interpret that? is the rank 3 or is it zero? do I even care about the rank if its higher than 1? Do I need a rank of at least 1?
Thanks:)
One thing i don't understand is why use the log series if it is only approximated to the returns, meaning, why not use the returns series as an indicator? is it only for the ease of writing or am I not getting something?
ReplyDeleteHi Danny,
ReplyDeleteLog returns actually have better statistical properties than net returns. Log returns follow an approximate normal distribution, while net returns can never be normal due to non-negativity. Many statistical techniques work better with normal assumption.
Ernie
Hi Ernie,
ReplyDeleteThanks so much for your blog post, I just starting to experiment with returns (i.e. log of prices) using the EWA-EWC kalman filter example from chapter 3 of your book. When using prices, the ve was set to 0.001 and delta =0.0001 book. Were these covariance determined by CovB via the following:
x=(cl(:, idxA));
y=(cl(:, idxC));
x=[x ones(size(x))];
[betareg,Sigma,E,CovB,logL] = mvregress(x,y);
CovB=[0.0001,-0.0011], [-0.0011,0.0240]
Am I correct in when using returns (log prices) that will vary between 0 and 1, we need to estimate ve and delta accordingly to be ve= 0.00000003 delta=0.0001 , as below:
xprices=(cl(:, idxA));
yprices=(cl(:, idxC));
x=diff(log(xprices))
y=diff(log(yprices));
x=[x ones(size(x))];
[betareg,Sigma,E,CovB,logL] = mvregress(x,y);
CovB=[0.00011236,-0.00000003],[-0.00000003,0.00000009]
Hi Ernie,
ReplyDeleteThanks so much for your blog post, I just starting to experiment with returns (i.e. log of prices) using the EWA-EWC kalman filter example from chapter 3 of your book. When using prices, the ve was set to 0.001 and delta =0.0001 book. Were these covariance determined by CovB via the following:
x=(cl(:, idxA));
y=(cl(:, idxC));
x=[x ones(size(x))];
[betareg,Sigma,E,CovB,logL] = mvregress(x,y);
CovB=[0.0001,-0.0011], [-0.0011,0.0240]
Am I correct in when using returns (log prices) that will vary between 0 and 1, we need to estimate ve and delta accordingly to be ve= 0.00000003 delta=0.0001 , as below:
xprices=(cl(:, idxA));
yprices=(cl(:, idxC));
x=diff(log(xprices))
y=diff(log(yprices));
x=[x ones(size(x))];
[betareg,Sigma,E,CovB,logL] = mvregress(x,y);
CovB=[0.00011236,-0.00000003],[-0.00000003,0.00000009]
Hi Douggie,
ReplyDeleteYours is a good question! ("How do we set hyperparameters in a KF model?")
In the second book, those are set by manual optimization of the backtest results, which isn't really kosher.
However, in my third book (p.71) I have used Matlab's ssm (state space models) class for their optimization, which uses maximum likelihood estimation.
Ernie
Hi Ernie,
ReplyDeleteThanks so much for the quick and helpful response, I had not quite progressed onto book 3, although I have it! In the pg 71 example B is calculated to be
B=[-0.01015 0.02114; 0.40606 -0.32381];
D=-0.07687/D^2=0.0059;
with the cov(B) Cov([-0.01015 0.02114; 0.40606 -0.32381])=[0.0866 -0.0717;-0.07178 0.0594]
however in the example, the covariance is [-0.00055 -0.011; -0.011 0.27], I just wondered how this covariance was calculated and am I correct in assuming that Ve=0.0059 and Vw=0.00055, implying a delta of 0.000549698 when applying the kalman filter from chapter 3 of your second book?
Hi Douggie,
ReplyDeleteAll initial covariances and other parameters for Kalman Filter in the 3rd book are based on MLE.
All of those in the 2nd book are based on manual trial-and-error optimization.
Ernie
Hi Ernie,
ReplyDeleteWow, so fast, I was just trying to work out to it possible map the output of the MLE to the free parameters ve and vw/delta used in the a regular kalman filter programmed using linear algebra rather than using a linear state model, i.e. does ve map to the D2 value from MLE and the Vw to the 1st entry in the cov(B) matrix?
Hi Douggie,
ReplyDeleteSure we can map Ve to D^2. However, the covariancea of the state noise is more complicated. They now have off-diagonal elements. Besides, the two diagonal elements (the variances) are different. So there is no straightful mapping of Vw to the w (omega) matrix. If the diagonal elements of w turned out to be the same, then they can be mapped to Vw.
Ernie
Hi Ernie,
ReplyDeleteThanks so much for all the replies especially given it is a sunday, clears up quite a lot of things!
Just one last thing, when dealing with logs, we are interested in the $value, rather than number of shares, the current PnL calculations are as follows,
Is it just a case of investing y2 to get the correct positions and PnL when using log(prices)?
y2=[cl(:, idxA) y];
% For log prices we are interested in the dollar value rather than shares.
y2(:, 1)=1./(y2(:,1));
y2(:,2)=1 ./(y2(:,2));
%remove first entry to match up with beta's calculated from returns
y2(1,:)=[]
longsEntry=e < -sqrt(ymse); % a long position means we should buy EWC
longsExit=e > -sqrt(ymse);
shortsEntry=e > sqrt(ymse);
shortsExit=e < sqrt(ymse);
numUnitsLong=NaN(length(y2), 1);
numUnitsShort=NaN(length(y2), 1);
numUnitsLong(1)=0;
numUnitsLong(longsEntry)=1;
numUnitsLong(longsExit)=0;
numUnitsLong=fillMissingData(numUnitsLong); % fillMissingData can be downloaded from epchan.com/book2. It simply carry forward an existing position from previous day if today's positio is an indeterminate NaN.
numUnitsShort(1)=0;
numUnitsShort(shortsEntry)=-1;
numUnitsShort(shortsExit)=0;
numUnitsShort=fillMissingData(numUnitsShort);
numUnits=numUnitsLong+numUnitsShort;
positions=repmat(numUnits, [1 size(y2, 2)]).*[-beta(:, 1) ones(size(beta(:, 1)))].*y2; % [hedgeRatio -ones(size(hedgeRatio))] is the shares allocation, [hedgeRatio -ones(size(hedgeRatio))].*y2 is the dollar capital allocation, while positions is the dollar capital in each ETF.
pnl=sum(lagmatrix(positions, 1).*(y2-lagmatrix(y2, 1))./lagmatrix(y2, 1), 2); % daily P&L of the strategy
ret=pnl./sum(abs(lagmatrix(positions, 1)), 2); % return is P&L divided by gross market value of portfolio
ret(isnan(ret))=0;
Hi Douggie,
ReplyDeleteIn this code fragment, y2 is the number of shares per dollar position. I don't see log prices used anywhere.
I discussed using log prices in the book in connection with defining a spread. It isn't clear from this fragment that it was used that way.
Ernie
HI Ernie,
ReplyDeleteBelow, is where i am defining the log prices, just I would have expected the sharpe ratios to be similar when using prices and log prices, for prices I get Sharpe ratio=1.345389 , however when using log prices I get Sharpe ratio=-0.630704 , which suggests may be the pnl calc is not correct, i will probably figure it just been a while since using matlab, any pointers on where I might have made an error would be greatly appreciated, once again thanks fo all you insights so far!
lclear;
% Daily data on EWA-EWC
load('inputData_ETF', 'tday', 'syms', 'cl');
idxA=find(strcmp('EWA', syms));
idxC=find(strcmp('EWC', syms));
xprices=(cl(:, idxA));
yprices=(cl(:, idxC));
x=diff(log(xprices));
y=diff(log(yprices)); % EWC price is measurement (observation)
lim=round(size(xprices, 1)*0.75);
trainset=1:lim;
testset=trainset(end)+1:(size(x, 1));
C=[x ones(size(x, 1), 1)]; % EWA prices augmented with constant offset as time-varying measurement 1x2 matrix.
A=eye(2); % State transition matrix
% B=num2cell(NaN(2, 2, size(cl, 1)), [1 2]); % state-disturbance-loading matrix. Cell array of 2x2 matrices, undetermined values.
B=NaN(2); % state-disturbance-loading matrix. Time invariant, undetermined values.
C=mat2cell(C, ones(size(x, 1), 1)); % Time-varying measurement matrix
% D=mat2cell(NaN(size(cl, 1), 1), ones(size(cl, 1), 1)); % measurement-innovation matrix, time varying variance. Cell array of 1x1 scalar, undetermined values.
D=NaN; % measurement-innovation matrix, time-invariant variance, undetermined values.
model=ssm(A, B, C(trainset, :), D);
rng('default'); % Fix random number generator seed to get repeatable results
rng(1);
param0=randn(5, 1); % 5 unknown parameters per bar.
model=estimate(model, y(trainset), param0);
B=model.B;
D=model.D;
covB=cov(B);
model=ssm(A, B, C, D); % Now fully specified
[beta, logL, output]=filter(model, y);
...
%model when here but had to truncate it to get into the char limit for a comment
y2=[xprices yprices];
y2(:, 1)=1000./(y2(:,1));
y2(:,2)=1000 ./(y2(:,2));
y2(1,:)=[];
longsEntry=e < -sqrt(ymse); % a long position means we should buy EWC
longsExit=e > -sqrt(ymse);
shortsEntry=e > sqrt(ymse);
shortsExit=e < sqrt(ymse);
numUnitsLong=NaN(length(y2), 1);
numUnitsShort=NaN(length(y2), 1);
numUnitsLong(1)=0;
numUnitsLong(longsEntry)=1;
numUnitsLong(longsExit)=0;
numUnitsLong=fillMissingData(numUnitsLong); % fillMissingData can be downloaded from epchan.com/book2. It simply carry forward an existing position from previous day if today's positio is an indeterminate NaN.
numUnitsShort(1)=0;
numUnitsShort(shortsEntry)=-1;
numUnitsShort(shortsExit)=0;
numUnitsShort=fillMissingData(numUnitsShort);
numUnits=numUnitsLong+numUnitsShort;
positions=repmat(numUnits, [1 size(y2, 2)]).*[-beta(:, 1) ones(size(beta(:, 1)))].*y2; % [hedgeRatio -ones(size(hedgeRatio))] is the shares allocation, [hedgeRatio -ones(size(hedgeRatio))].*y2 is the dollar capital allocation, while positions is the dollar capital in each ETF.
pnl=sum(lagmatrix(positions, 1).*(y2-lagmatrix(y2, 1))./lagmatrix(y2, 1), 2); % daily P&L of the strategy
ret=pnl./sum(abs(lagmatrix(positions, 1)), 2); % return is P&L divided by gross market value of portfolio
ret(isnan(ret))=0;
cumret=cumprod(1+ret(trainset))-1;
cumret=cumprod(1+ret(testset))-1;
Hi Douggie,
ReplyDeleteTo compute P&L of a strategy using log prices z, you can simply write
pnl=lagmatrix(positions, 1).*(z-logmatrix(z, 1)).
There is no need to divide this by logmatrix(z).
Ernie
Hi ernie,
ReplyDeletethanks so much, this looks a little bit more inline with the returns from prices, I assume this is kinda what you meant?
logxprices=diff(log(xprices));
logyprices=diff(log(yprices));
y3=[logxprices logyprices];
longsEntry=e < -sqrt(ymse); % a long position means we should buy EWC
longsExit=e > -sqrt(ymse);
shortsEntry=e > sqrt(ymse);
shortsExit=e < sqrt(ymse);
numUnitsLong=NaN(length(y2), 1);
numUnitsShort=NaN(length(y2), 1);
numUnitsLong(1)=0;
numUnitsLong(longsEntry)=1;
numUnitsLong(longsExit)=0;
numUnitsLong=fillMissingData(numUnitsLong); % fillMissingData can be downloaded from epchan.com/book2. It simply carry forward an existing position from previous day if today's positio is an indeterminate NaN.
numUnitsShort(1)=0;
numUnitsShort(shortsEntry)=-1;
numUnitsShort(shortsExit)=0;
numUnitsShort=fillMissingData(numUnitsShort);
numUnits=numUnitsLong+numUnitsShort;
positions=repmat(numUnits, [1 size(y2, 2)]).*[-beta(:, 1) ones(size(beta(:, 1)))].*y2; % [hedgeRatio -ones(size(hedgeRatio))] is the shares allocation, [hedgeRatio -ones(size(hedgeRatio))].*y2 is the dollar capital allocation, while positions is the dollar capital in each ETF.
%pnl=lagmatrix(positions, 1).*(z-logmatrix(z, 1)).
%pnl=sum(lagmatrix(positions, 1).*(y2-log(lagmatrix(y2, 1)))./lagmatrix(y2, 1), 2); % daily P&L of the strategy
pnl=sum(lagmatrix(positions, 1).*(y3-lagmatrix(y3, 1)), 2); % daily P&L of the strategy
%pnl=sum(lagmatrix(positions, 1).*(y2-logm(y2, 1))./lagmatrix(y2, 1), 2); % daily P&L of the strategy
ret=pnl./sum(abs(lagmatrix(positions, 1)), 2); % return is P&L divided by gross market value of portfolio
ret(isnan(ret))=0;
Hi Douggie,
ReplyDeleteIt looks right (though obviously I haven't carefully scrutinized each line of code!)
Ernie
Ernie, your looks ok is good enough for me, thanks for the lesson 101!
ReplyDeleteHappy trading!
Hi Ernie,
ReplyDeleteThanks once again for the post, one thing I noticed, that when dealing with log price, the beta slope appears to be monotonically increasing with time, whereas the intercept does not. Intuitively is this because the difference between the returns is around zero, where as over time the returns tend to move together, hence the slope approaches 1.
Given the slope is monotonically increasing with time, how can it be used as a reliable hedge ratio? as I would expect too low hedge ratio near the start of trading sequence that would lead to an insufficiently hedged pair position relative to the same trade taken much later when the beta slope/hedge ratio is closer to 1?
Hi Douggie,
ReplyDeleteIf you are plotting the log price of stock A against log price of stock B, the slope will be one only if they are the same rates of returns. That is generally not the case. You can trade a cointegrating pair of stocks even if they have different returns - you just need to use a hedge ratio different from one to adjust for that.
Indeed if the hedge ratio varies too quickly, the pair is not really cointegrating, and mean reversion strategy will fail. However, this strategy will still work if hedge ratio varies much slower than the half life of mean reversion.
(A truly unchanging numerical relationship almost never exist in finance. It is the relative time scales of variations that matters.)
Ernie
Hi Ernie,
ReplyDeleteIndeed! I should have been more explicit, using the kalman filter example of EWC/EWA pair from chapter 3, with delta and ve set to 0.0001, but using x and y as the difference in the log of price i.e. (diff(log(ewa)). I would have expected the slope to be similar to the ewma of the ratio of the returns of EWC/returns of EWA, and hence can be used to weight the value of EWC and EWA in the pair, simliar to how the slope stabilises quickly when using EWA/EWC prices to around 1.4 i.e. (approx price EWC/price EWA). However it does not seems to stabilise to the ratio returns of EWC/returns of EWA and just wondered the intuition for why this might be and if there is another way to determine weights for the values of EWC and EWA such as just taking the return of EWC/return of EWA at the given point in time when the pair is entered?
snippet of prices
ewa ewc EWMA (ewc/ewa) slope intercept
16.1 22.95 1.425465839 0 0
15.98 22.78 1.425498877 1.419417579 0.088824629
16.1 22.94 1.425280825 1.41932804 0.08881911
16.31 23.05 1.422271471 1.407844084 0.088133361
16.29 23.32 1.424127797 1.426067038 0.089256123
16.32 23.12 1.422884275 1.41125829 0.088355404
16.27 23.01 1.421652146 1.408838994 0.08820443
16.56 23.33 1.420047681 1.403509371 0.087916979
16.63 23.23 1.41747273 1.391632438 0.087223941
snippet of log prices
ewa ewc log ewa log ewc EWMA (ewc/ewa) slope intercept
16.1 22.95
15.98 22.78 -0.007481332 -0.007434978 0.993804156 0 0
16.1 22.94 0.007481332 0.006999154 0.964676667 4.76029E-06 0.000636289
16.31 23.05 0.012959145 0.004783658 0.766162364 1.35492E-05 0.001301109
16.29 23.32 -0.001226994 0.011645592 -1.798167308 6.64899E-06 0.003437702
16.32 23.12 0.001839927 -0.008613318 -2.374801214 2.61813E-06 0.000610267
16.27 23.01 -0.003068428 -0.00476914 -1.71995744 1.04202E-05 -0.000738481
16.56 23.33 0.017667228 0.013811182 -1.362572064 0.000125489 0.003038882
16.63 23.23 0.004218144 -0.004295539 -1.319544071 0.000124074 0.001098631
Hi Douggie,
ReplyDeleteIf you are using Kalman filter to find the hedge ratio, then there is no expectation that the stocks A and B are cointegrating at all. Kalman filter is typically used in a mean reversion strategy where there is no true cointegration - otherwise the hedge ratio will be constant and there is no need for dynamic update. So I am not sure that the slope will converge to anything at all. It certainly need not converge to the slope of static linear regression of the past log prices of the stocks.
Ernie
Hi Ernie,
ReplyDeleteI was not expecting it to converge, but was kinda expecting it to be in line with the ratio of returns, for example if EWC had a daily return of 2% and EWA of 1% I would have expected a hedge ratio of the region of a 2, as I would have to invest twice as much in EWA relative to EWC to make a profit,, assuming they are co-integrated and returns tend to zero. In a simliar manner the hedge ratio is around 1.4 when using prices and reflects the rough ratio of price EWC to price of EWA. If I multiple the returns by 100 i.e. (x=diff(log(xprices)).*100; and y=diff(log(yprices)).*100;), the hedge ratio's seems to be more inline with the ratio of returns.
Is my expectation reasonable?
Hi Douggie,
ReplyDeleteYou lost me there. Why would multiplying the returns on both stocks by 100 give a different hedge ratio than not multiplying them? After the 100 should drop out as a common factor when you take their ratio.
Ernie
exactly! it seems odd.
ReplyDeleteI'll step through the may be something changed in the code.
hi ernie
ReplyDeleteThanks for your nice book, I'm a beginner in this area and having a great help from your book
I have a simple question. What do you mean by "setting the market value of each asset to h" in log price spread according to your book page 65? I have trouble understanding this part, and I would appreciate it if you give a more elaborate description.
Sorry if my English isn't fluent(since first language is not English...)
Hi,
ReplyDeleteh is derived from the regression fit in equation 3.3. These are dimensionaless numbers, which you can interpret as the capital allocation weights (e.g. 10% to MSFT, 15% to IBM, etc.)
So if you have $1M invested in the portfolio, you would invest $100K to MSFT, and $150K to IBM, etc.
Ernie
Ernie
ReplyDeleteFirst of all, thanks for your reply
The thing I was confused was the difference btw using price and log price(which is eq (3.1) and (3.3)).
Just in case if I have understood correctly, does it mean the weight "h" in eq(3.1) corresponds to "number" of stocks to hold, not "share(%)" of stocks to hold, and that's the only difference btw the meaning of "h's" in eq(3.1) and eq(3.3) respectively?
Enjoy your day, thanks a lot
Hi,
ReplyDeleteYour understanding is correct!
Ernie
Dear Ernie,
ReplyDeletefirst of all, congrats on your EXCELLENT books. Just started reading book 2.
I am used to read about pair trading assets where one would sell A and Buy B (or the other way around, with a direct correlation between the assets).
But I have seen a strategy where the idea is either go long or short BOTH assets (when the correlation between them is inverted, I mean, negative beta). Can you help me understand this? What is the main concept to have in mind when the assets have an inverted correlation (negative beta), and does it really mean one always go either long or short both assets under this situation or is there other things to take into consideration? May be you can point to where I can read more about it.
Thanks for your kind words!
ReplyDeleteYes, when the correlation of the returns of two assets are negative, you should long or short both. I gave an example of that in my 2nd book Algorithmic Trading of VX vs ES in a momentum trade. Because VX is negatively correlated with ES, we normally short both (using ES to hedge VX).
Ernie
Dear Ernie,
ReplyDeletethanks for the quick answer for my question above.
I am almost done with your first book, but I need your assistance with the following question.
Suppose a pair A,B (independent, dependent), with the following Johansen results:
Eigen Values
[0.00654948 0.00060003]
Eigen vectors (aka #of shares to hold or hedge ratio)
[[ 1.25796607e+00 6.53907500e-04]
[-7.25841553e-01 1.97461874e-01]]
When using CADF, II understand that I use the beta from the regression equation A,B and B,A to calculate allocation on dependent variable. But I am unsure when using Johansen
I know I have to use the Eigen vectors to calculate unit portfolio (price), but not sure on how to read the numbers above. I understand the first line is associated with the best pair and I understand the best pair is A,B because first eigen value in vector is > second eigen vector (0.00654948 > 0.00060003). Is that correct?
Besides this, supposing it is correct, I understand I should use eigenv x price of stock. But I am not sure if I did it right. I proceeded like:
For the best portfolio (A,B), mktval = Pa x 1.25796607e+00 + Pb x 6.53907500e-04
And for B,A, mktval = Pb x -7.25841553e-01 + Pa x 1.97461874e-01
Am I correct? If so, I can have negative portfolio mkt values and use it to calculate half-life.
Can you assist?
Kind regards
Hi,
ReplyDeleteYes, we should always use the first eigenvector because they are sorted in decreasing order of eigenvalues.
The eigenvectors are arranged in columns, not rows. So the first eigenvector is [1.25... -7.2...e-1]', contrary to what you wrote.
Ernie
Thank you again, and sorry for my ignorance. Just to confirm my understanding of your explanation
ReplyDelete1- Is my understanding that the pair AB is the best one because Eigen values [0.00654948 > 0.00060003] correct?
2- Are you confirming that the best pair is AB and that:
mkt val AB = 1.25 * PA + (-7.26e-1) * PB
3- Reversed pair is BA:
mkt val BA = 6.54e-4 * Pb + 1.98e-1
Sorry and thank you again
Rhaufed
1) Yes
ReplyDelete2) Yes
3) The second pair is 6.54e-4 * Pa + 1.98e-1 Pb. It isn't "reversed", but a different linear combination.
Ernie
Thank you once again.
ReplyDeleteI am having a hard time to understand the Matlab code. I'm translating into python but I am not versed in Matlab, and could not understand the code:
positions = repmat(numunits, [1 size(y3,2)]).*repmat(results(evec(:,1)', size[(y3,1) 1])*y3
May be you can translate it into words and therefore I can implement it in python
I know what results(evec.. mean.
y3 I understood it is the matrix with the 3 stocks EWC,EWA and ICE prices
But I simply could not understand what you doing here.
As I am interested basically in 2 assets only, maybe you can simplify the answer, if you prefer
Sorry ince again
Python codes are available for Quantitative Trading and Algorithmic Trading. See epchan.com/book1 and epchan.com/book2 resp.
ReplyDeleteThis will be Great,
ReplyDeletebut I get: This page doesn't seem to exist.
It looks like the link pointing here was faulty. Maybe try searching?
for book1 and a screen requesting UN and PW when using it for book 2.
I purchased the 3 books at amazon in Kindle versions
Sorry, should be epchan.com/book
ReplyDeleteUN/PW at the end of Example 3.1 using Python
ReplyDeleteFor both books: Quantitative trading and Algorithmic Trading?
ReplyDeleteI am studying Algo Trading now.
For Algorithmic Trading, PW on on page 199 (About the Website page)
ReplyDeleteDear Chan,
ReplyDeleteI guess we are not on the same page.
going to epchan.com/book it asks for UN and PW.
According to your instructions above, I should get the UN/PW on ABOUT THE WEBSITE chapter of your book Algorithmic Trading. And it is indeed there in a phrase like ...Once Redirected to the website ... - use "k***ly" for both.
But this UN/PW is not working. It does nothing when I use it and keeps waiting for UN/PW
I guess I am missing something
Codes for Quant Trading is on epchan.com/book
ReplyDeleteFor Algo Trading is epchan.com/book2, PW on on page 199 (About the Website page)
Hi Ernie,
ReplyDeletecan you elaborate on my understanding/questions below and check if it is correct?
1- We trade prices spread signals when prices is cointegrating
2- we trade logs of prices spread signals when log of prices is cointegrating
3 - Ratios is a question: Can we trade log of prices ratios or only prices ratios?
Thank you
1) Yes
ReplyDelete2) Yes
3) Ratio is similar to log prices. But in general, ratio only good for short term mean reversion, and not for long term cointegration.
Ernie
Than you,
ReplyDeleteand I hope as a last question:
When I apply Kalman over log prices the results look weird. Do I need to modify the Kalman routine when using Kalman over log prices?
Sorry again, I am working hard to understand your book. Thank you
I don't think KF would change whether you use log or raw prices.
ReplyDeleteErnie
Hi,
ReplyDeleteregarding Kalman you wrote:
#we will use a 2 x 1 vector Beta to denote both the intercept mi and the
#slope of the linear relation x and y
Should I understand that intercept is in [0] and slope (beta) is in [1]?
Thanks
Hi Ernie,
ReplyDeletealmost there, I guess.
If I may, can you comment on the following situation?
Previously I asked you about going either long/long or short/short on a pair that is negatively correlated (beta -).
But if that is the usual thing to do, what if the eigen vectors are like -/+, suggesting going short and long on same pair with - beta. Would I use eigen to decide what position to take and use beta to compute the hedge?
(I could use eigen itself to compute position, but I cannot calculate eigen in my trading platform easily (in python it is easy, but the trading platform language would complicate things, if not make it impossible to use Johansen), therefore it is easier to use regression and compute beta)
Thanks
If the returns are anti correlated, the eigenvector components will have same sign, so correct to use them to determine their positions.
DeleteAlways?
ReplyDeleteBecause I get negative beta for A/B and B/A, with B/A cointegrated
and Eigen Vector is:
[[ 2.32108973 -15.00937015]
[-20.3579613 -13.7165537 ]]
Therefore, first linear combination is not short/short, second is. According to my understanding
Thanks
Try Deming regression instead of OLS.
ReplyDeleteHi Ernie, may be you can help me to undestand this:
ReplyDeleteSomewhere above you wrote:
When we pair trade stocks, we are not interested in having the returns cancel each other as well as possible. If we find a hedge ratio based on returns, we are doing exactly that. In pair trading, we are interested in mean reversion of the spread. The spread is made of 2 price series, not returns series. We are trading the deviation from a straight line fitted through the scatter plot of the prices. Hence the best way to construct the spread is to use the slope of that straight line as the hedge ratio.
Ernie
But I came to a situation where the log return is cointegrated (or even the return), but the log prices or the prices are not!
So now I am lost on how to:
1- Should I calculate the straight line on log prices and use the slope (beta) to trade?
2- Eigen as well would be based on log prices?
3 - What about half life?
4- For the trading signal (linear, log or ratio) I would use the log prices instead off return values as well?
In other words, if 1,2,3 is correct, return would be used only for checking cointegration??
Kind regards, Rhaufed
Hi Rhaufed,
ReplyDeleteCointegration does not apply to returns (or log returns) series, because they are always stationary. If the price or log price of two price series are not cointegrated, you can still use them for short term mean reversion. Apply Deming regression to find their hedge ratio and apply Bollinger bands on the spread.
Ernie
Once again, thank you
ReplyDeletemay you help me understand another one?
Eigen vector is always # of shares to allocate in x,y or it may be $ to allocate like when using beta from regression?
If your input is price series, then eigenvector is number of shares. If input is log price series, then eigenvector is $.
ReplyDeleteHi Chan,
ReplyDeleteI would like to hear from you about the following:
I have been presented to a trading strategy based on mean reversion, where they propose trading two FUTURE contracts (Brazilian mkt): Dolar index x Brazilian Stock index.
They always trade the next expiring contract, either shorting or longing both (they argued that the dolar and index are inversely correlated). I decided to study this, based on your text books, to check it before I commit any capital (if) to the strategy.
But I could not find the linear inverse correlation they argued there exists between the pairs and I asked them about it. They replied that the correlation is not in dy/dx, but instead in d(dy)/d(dx) = dret(y)/dret(x). So to say, not on "speed" of prices, but on "acceleration" of them.
Indeed, if I plot dret(x)/dret(y) = d(dy)/d(dx) there exist a linear relation. Can you comment on this and if it is possible (and how) to trade using this beta derived from the "acceleration"?
If that is the case, what do I need to have in mind?
Fundamentally, we can only invest in priced. We buy at x and hope to see at >x. So if you predict ret(x) >0, great! But if you say ret(x) may be <0, but dret(x) >0, I am still going to lose money if I bought at x. I am not sure second derivatives will help to decide on direction of trade.
ReplyDeleteTo find beta when assets have different point values I need to scale quotes.
ReplyDeleteDo I need to have the prices scaled to run ADF, CADF or Johansen tests as well, or I can run these without scaling?
Tks
Scale them so quotes are in dollars.
ReplyDeleteHi Chan,
ReplyDeleteFirst of all, thank you very much for all your answers. I got to implement everything from book 2. Your assistance is second to none.
Can you check my understanding below and confirm if it is ok?
-I want to earn, say y=6000k trading x , y log spread
-I calculate a beta of -0.25
My equivalent $ value x= 6000/abs(-0.25) = 24000
y contracts = y/y price price and x contracts = x/xprice
Am I right?
One point I have after giving a second thought to my solution:
ReplyDeleteI found beta =-0.25 when calculating linear regression (x,y). Therefore if I have x I can estimate y.
But I understand that the lin(y,x) is not the inverse of it lin(x,y)
With that in mind, the correct answer would not be to calculate the beta1 from lin(y,x) and use 6000 * beta1 to calculate my x value?
Thanks
Use Deming regression instead of ols
ReplyDeleteBut with what coefficient. Deming regression is great, but it is tricky to guess the correct values for δ, which is a major drawback because the result of the regression relies heavily on δ.
ReplyDeleteBesides, if using it d I still need to do x,y and y,x to find beta in both directions or not?
Regards
You can estimate δ by using the volatility of the two individual series. Otherwise, assuming it to be 1 it is still OK.
ReplyDeleteYes, you still need to find the beta in either direction, but in this case one beta is just the reciprocal of the other.
Ernie
Thank you. Up and running
ReplyDeleteWhen using prices or log prices to calculate the number of stocks to trade, I noticed the numbers do not match, and sometimes they deviate by a lot. Can you comment on this? I expected they to deviate, but not that much
Thanks
Why would you expect the number of shares to match based on prices vs log prices?
ReplyDeleteThe latter assumes that you want to rebalance your capital regularly by changing number of shares, whereas the former fixes the number of shares.
Hi Ernie,
ReplyDeleteIn your great book "algorithmic trading', you introduce the notion of using the kalman fitler to create a VWAP/TWAP combined price ("kalman filter as a market making model").
In the case of two cointegrated assets, we set the Kalman filter up as follows, where X and Y are the prices or log returns of assets x and y.
for t=1:length(y)
if (t > 1)
beta(:, t)=beta(:, t-1); % state prediction. Equation 3.7
R=P+Vw; % state covariance prediction. Equation 3.8
end
yhat(t)=x(t, :)*beta(:, t); % measurement prediction. Equation 3.9
Q(t)=x(t, :)*R*x(t, :)'+Ve; % measurement variance prediction. Equation 3.10
% Observe y(t)
e(t)=y(t)-yhat(t); % measurement prediction error
K=R*x(t, :)'/Q(t); % Kalman gain
beta(:, t)=beta(:, t)+K*e(t); % State update. Equation 3.11
P=R-K*x(t, :)*R; % State covariance update. Euqation 3.12
end
However it was not clear to me how to set it up similarly when just using 1 asset price and volume to use the kalman filter as a market making model.
Hi Douggie,
ReplyDeleteThe equations for the market making model are displayed on p.82 of Algorithmic Trading. The main difference is that their is no beta (or you can set beta to 1), and K and R are scalars. Ve(t) needs to be made a function of the trade size T.
Best,
Ernie
Hi Ernie,
ReplyDeletefor POSITIVE linear regression coefficient between two assets X and Y (beta), one should sell Y and buy X when z is positive and buy Y and sell X when z is negative.
But what is the reasoning behind selling both or buying both when beta is NEGATIVE.
Can you help me understand it?
Hi rhaufed,
ReplyDeleteIf X and Y are anticorrelated (e.g. ES vs VX future), and buying both or selling both means that you own a market-neutral portfolio. Anticorrelated instruments result in negative beta.
Ernie
Hello
ReplyDeleteThanks for this wonderful discussion. It cleared up a couple of things. Can anyone comment on the case whereby the model parameters (i.e. hedge ratio or variance) change significantly within a single (average) mean-reversion period yet cointegration still holds. The change is so significant that what was bought may now be appropriate if it was sold rather.
I ran a simple Dickey-Fuller test with EURUSD vs USDCHF (and also EURUSD vs GBPUSD) and came across such cases where p-Value is less or equal 5% yet the model parameters are changing significantly (thus invalidating some opened positions). Note: The model was updated on every new candlestick (in receeding horizon approach).
Is the pair still cointegrating with the old hedge ratio, or with the new one only? If the latter, it can't be considered cointegrating, since that assumes we are speaking of a constant hedge ratio.
ReplyDeleteErnie
Hi Ernie
ReplyDeleteThe pair is still cointegrating with both the old and new parameters (hedge ratio, intercept, variance, and p-value). I keep both the old (long-term) and new (instantaneous) parameters for comparison purposes. It happened a few times that the p-values would still be less or equal to 5% yet the old parameters would have said "go long on spread" and the instantaneous parameters are suggesting the opposite position. This happens a lot when either the hedge ratios or intercepts (or both) change yet the cointegration test is still valid. I have also been trying to find some information on the stability measure for hedge ratio and intercept; that is, how much (in percentage) they can change with respect to mean-reversion period to consider the model stable enough to execute pairs trading or statistical arbitrage in general.
Here is what I usually do;
1. On the very first iteration I evaluate a linear regression with N historical data points (raw price values, not log prices). Then perform the Dickey-Fuller test and keep track of the (i) hedge ratio, (ii) the intercept, (iii) variance/std deviation, and (iv) p-value.
2. On the next (or k-th) iteration (dictated by new candlestick arrival), I shift the horizon/window forward (by one instance) and repeat the same process again. This time I keep both the first (or old or long-term) iteration parameters and the new (or k-th or instantaneous) iteration parameters.
3. As I keep updating the model on every iteration, I monitor the mean-crossing period (and its average) of the spread (i.e. iterations/candles count from mean-crossing to at least one std deviation and back to mean-crossing).
4. At every mean-reversion instance, I reset the long-term model parameters (such that they are similar to the instantaneous model parameters). I do this in order to have some kind of a measure of how much the instantaneous model parameters deviate from the long-term model parameters at least within one (average) mean-reversion period. I think of it as evaluating the stability of the model in some sense.
5. I also play around with the horizon/window length N to see if N affects the model stability.
Hi Luh,
ReplyDeleteCointegration by itself doesn't determine a trading strategy. Their are various choices of strategies, such as the Bollinger Bands that you are apparently using. But even with BB, there are many choices of lookback and other parameters. Any such choice can result in a different trading signals. To determine which particular trading strategy or what parameters are optimal, some sort of optimization method is needed. Of course, optimizing on the backtest period is the most obvious, but optimizing for the past does not guarantee optimal outcome in the future. You can try https://epchan.blogspot.com/2021/04/conditional-parameter-optimization.html or https://epchan.blogspot.com/2017/11/optimizing-trading-strategies-without.html
Ernie
Hi Ernie
ReplyDeleteThank you for the links, I will learn more about optimization. I already found something interesting about optimizing for training set length using F1 score plot. Let me read through and see if I can make some improvements to my BB strategy. Thanks a lot.
Lord Luh
Hi Ernie!
ReplyDeleteI wonder how should we interpret the physical meaning of the cointegrated log prices. For example, if we have two assets A and B, and Johansen test gives us a cointegration vector of [1, -1.5] between log(A) and log(B). That is log(A) - 1.5log(B) is mean-reverting. However, if we try to interpret this relationship's physical meaning by transforming it back to real price, we will get A/B^1.5 is mean-reverting, which is difficult to understand: how is this relationship possible between two assets!? This is even worse if more assets are involved, giving us relationships like (A^2 * B^3)/C^4 = c. I know how cointegration works with log prices, but I am still struggling with understanding it in real world. Could you please enlighten me more on how to understand the log cointegration relationship? Thank you so much!
Hi! I actually discussed that in my book Algorithmic Trading (p.64). In your example, the hedge ratio of 1.5 means that you need invest 1.5x more dollars in B than A to maintain a cointegrated spread, and you have to rebalance at every period to maintain this dollar ratio. In a spread based on prices, the hedge ratio is that of the number of shares, which can remain constant and no rebalancing each period is needed.
ReplyDeleteHi Ernie,
ReplyDeleteI have just finished the edition 2 of your Quantitative Trading book.
There is something I still can't comprehend about long and short position sizing.
Assuming:
1) my target is a trade volume of $10k (long + short position)
2) hedge ratio of 2
3) Long A + Short B is triggered at a given moment
My conclusion based on your methodology is that my long position would be $3.33k while my short position $6.66k.
However, I get very strange results when there is a big nominal difference in price between A and B, such as the following case:
In the Brazilian exchange, SPX-equivalent ETF trades at ca. R$338 while QQQ-equivalent trades at ca. R$15 (quite big nominal difference). This gives me a hedge ratio of about 22.
Going back to my $10k capital allocation target, this would mean that I go long only $400 SPX-equivalent ETF and short $9,600 of QQQ-equivalent ETF, which I struggle to accept.
What am I overlooking here?
Hi,
DeleteWhen you said hedge ratio of 2, did you regress prices or log prices? If the former, then it does not mean the trade size should be 1:2. It means the number of shares should be 1:2.
Ernie
I regress prices indeed. Thanks, this is clear.
DeleteHi Ernie.
ReplyDeleteWhat would be the difference between:
1) pre-compute half life of (stationary) spread series and close the trade after half life has passed (taught in your book); vs
2) close the trade after my normalized spread has converged back to 0
Hi,
ReplyDeleteBoth 1) + 2) should be exit rules, whichever one comes first.
Ernie
Hi Ernie
ReplyDeleteGiven a linear model y = mx + c, I used to update my model parameters (m, c) at every candlestick (as a way to monitor when I lose/regain cointegration) until the time when a signal allows me to open a trade. When a trade is running I kept the model parameters that were valid when the trade was opened but alongside I kept updating the model at every candlestick just to compare the new model parameters with the ones for the opened trade. It so happens that sometimes the two model parameter settings would conflict in such a way that one would be saying mean is above the current spread value while the other would be saying its below it, yet cointegration had not been lost along the way.
This is one of the observations that made me doubt that repeating similar data points during training is the best way to go about getting reliable and stable model parameters (m, c).
I saw that in your earlier comments you mentioned that there is no need to update the model frequently and I agree that there is no need as long as you have some justification that the model parameters change rather slowly (or not at all) compared to the average mean reversion period of the spread.
Hi Luh,
ReplyDeleteIf you use Kalman Filter, it is essentially doing an exponentially weighted linear regression constantly and it forces your to change model parameters continuously. Some people find that superior to fixed regression parameters.
Ernie