Friday, July 30, 2010

Pair trading technologies update

Pair trading was invented two decades ago, but automating its implementation has only recently become fashionable with independent traders. But once the spotlight is on, innovations come fast and furious. Here are a number of recent developments that I find interesting:


1. I mentioned previously the software called quant2ib. It is an API which allows us to get market data and send orders from a Matlab program to Interactive Brokers (IB). I have used it extensively for our trading, and it is as reliable as IB's native API. Their latest version now includes functions for constructing a "combo" security. This combo security can be pairs of stocks, ETF's, futures, etc. (with the notable exception of currencies), and the API allows you to get market data as well as to submit orders on a combo. This is a huge improvement because you can now automatically trade a pair of securities as one unit by submitting limit orders on the combo. (Previously, you would have had to submit market order on at least one side of the pair, and this would have required your program to continuously monitor the market prices and send orders when appropriate. Or else you had to give up using the API and manually enter a "generic combo" limit order in IB's TWS.)

2. Alphacet Discovery also has the ability to send limit orders on pairs, due to its partnership with Knight Trading. Besides, based on a demo that I have recently seen, they also now have great pairs portfolio and execution reporting functionality. (Full disclosure: I used to consult for them.)

3. IB itself has released a "Scale Trader" algorithm that can be applied to combos (see 1. above. Hat tip: Mohamed.) I can't explain this better than their press release: "... ScaleTrader algorithm allows clients to create conditions under which a long position in one stock is built while simultaneously creating an offsetting short position in the other. The ScaleTrader is named because investors can 'scale-in' to market weakness by setting orders to buy as the market moves lower. Similarly, sell orders can be 'scaled' into when a market is rising. The ScaleTrader algorithm can be programmed to buy the spread and subsequently take profit by selling the spread if the difference reaches predetermined levels set by the user." In other words, it allows us to automatically implement the "parameterless trading" or the "averaging-in" strategy that I blogged about previously without any programming on our part!

Speaking of pair trading, I will be teaching my first New York workshop in October.  (My editor inevitably picks touristy locations for these workshops. My London workshop takes place across the street from the Tower of London, my New York workshop is across from the new World Trade Center, and my Hong Kong workshop is in the "Golden Mile" shopping district of Tsim Sha Tsui.)

60 comments:

Anonymous said...

Is it possible to record realtime bid-ask data and to backtests a pairtrading strategy with this data in Mathlab?
I am currently using Openquant - but I am not so happy with it...

Ernie Chan said...

Anon,
It is always possible to save the real-time bid-ask data to file and use it for backtesting.

Alternatively, you can request historical bid-ask data (for free) using the API for each leg of the combo, and construct the bid-ask for the combo yourself with the appropriate hedge ratio.
Ernie

Ernie Chan said...

Anon,
It is always possible to save the real-time bid-ask data to file and use it for backtesting.

Alternatively, you can request historical bid-ask data (for free) using the API for each leg of the combo, and construct the bid-ask for the combo yourself with the appropriate hedge ratio.
Ernie

Unknown said...

Hello Ernie,

I really Appreciate both your blog and book, both are very insightful.

I'm currently using Matlab for backtesting and it's very smooth, and I'm a beginner on C++ also.

I have an impression that it might be tedious to backtest using C++ instead of Matlab, I wonder if you have an insight on this?

Or if there are any libraries currently used in the market specifically catered for such work?

Mohamad

Ernie Chan said...

Hi Mohamad,
Indeed it is quite tedious to backtest using C++, and I am not aware of any libraries developed for this purpose since I don't use C++.
Best,
Ernie

Stewart said...

Hey Ernie,

Really enjoy your blog and book. I love the idea of pairs trading. However when divergence occurs between two similar (possibly conintegrated) instruments the question is whether this is because (1) although the fundamentals remain the same the market has moved to an inconsistent state which will revert with time, or (2) the fundamentals have changed and are unlikely to revert with time. It would be really interesting to try and create some type of model to suggest which of these two is occurring. Perhaps this might involve looking for recent news on either company, looking at volume, price entropy or changes in the order book. Any thoughts on this?

Windsurfing Stew

Anonymous said...

I recently read your book, and, paired with another matlab reference book, have found it enormously helpful in getting my feet off the ground!

Do you have any plans to come to Chicago?

rara avis said...

Matlab is a poorly designed language, and it is overpriced. However, I must admit that it has fantastic integration with Java: robust, complete, with plenty of syntactic sugar. It's like a poor man's Eclipse, really, but actually interactive. For these reasons, I am puzzled that anyone may buy quant2ib. Unless quant2ib changes radically the design of of the IB API (I hope not), using the IB API from Matlab is more flexible, robust and faster.

I assume there will always be a market for quant2ib for the same reason there will be a market for bottled purified water.

Ernie Chan said...

Hi Stewart,
Yes, news and fundamental changes do affect pairs trading, and it would be wise not to enter into a position due to price movement as a result of news. It may even be prudent to exit positions prior to news.
Ernie

Ernie Chan said...

Hi Anon,
Thanks for your kind words. I don't yet have plans to come to Chicago in the next few months, but will surely go there before too long.
Ernie

Ernie Chan said...

Hi Rara,
I am unaware that Mathworks sells an IB API, and in fact, a search on its website yields no results. Can you please provide a link to such a product?
Ernie

Unknown said...

Hello Rara,

I agree with Ernie's comment above.
Also what would you suggest as a back testing platform? since Matlab is not your favorite software. And what would you use for executing orders?

Best,
Mohamad

Peter Harrington said...

I think what Rara meant was that he can use the Java IB API. That is similar to what I do: backtesting in Matlab, generate a text file or database then do the actual trading with another language.
Actually I've been using Python, and someone ported the Java IB apis to Python (search for IbPy.) Python has allowed me to do A LOT of automation and offload the heavy number crunching to C-compiled functions.

David said...

Hi Ernie,

I use Matlab2IB very successfully. I am subscribed to the software, yet see no details of this new functionality on their website. How do you come to hear of the changes and get upgrades?

Thanks.

David

Ernie Chan said...

Hi David,
It is not on their website, but you can ask customer support to send you an example of how to construct a combo using existing functions.
Ernie

Jimmie Goode said...

Ernie,

Going back to your original blog post, on an IB combo order consisting of a stock or ETF on both legs (ie. IYE/XLE), I don't think you're guaranteed execution of both legs simultaneously since the order is "non-guaranteed" (see http://ibkb.interactivebrokers.com/taxonomy/term/128).

I've never had one of these partial execution happen in practice, but I use Quant2IB as well and am curious how the getOrderStatus(oid) function handles a partially filled combo.

Ernie Chan said...

Jimmie,
It is true that combo orders are not guaranteed execution, and indeed I have seen one leg not fully filled while the leg was in practice. But this is not such a big difference with limit orders: they can be partially filled, or not at all filled. There is just a bit more risk with combo orders because each leg is really a separate order for the exchange.

Your question about the getOrderStatus is a good one. I will forward it to quant2ib customer support and see what they say.

Ernie

henn.de said...

Hey Ernie
I have a question concerning a difference in beta when finding pairs. Other traders have recommended having pairs with a different in beta no greater than 0.2 while from experience one of our most valueable pair, at least in my opinion, GDX / GLD has a much larger difference in beta. Do you have a beta cut off criteria and how large is it ?

rara avis said...

Ernie, what I mean is that you can actually use java within MATLAB. MATLAB comes with a JVM and seamless integration with Java. This means that you can instantiate objects, assign them to MATLAB variables, and pass objects to/from Java seamlessly. In practice, this allows you to submit an order from within MATLAB to IB without the need for any 3rd party software.

For the record, I am doing most of my analytics in R, Python and C. My turnover is low, so I don't have an automatic execution issue. If I had it, I'd probably use IBPy.

Ernie Chan said...

Hi Henn,
I don't use a beta cutoff for my pairs. However, I have not backtested whether this cutoff is beneficial. I don't believe that this cutoff is particularly useful for ETF pairs though.
Ernie

Ernie Chan said...

Hi Rara,
Thanks for the clarification. Yes, I believe you are right: if you are already somewhat familiar with Java, you can certainly invoke it within Matlab to use IB's Java API. But what about the rest of us who has forgotten most everything we have learned about Java?
Ernie

henn.de said...

Thanks for your answer. I have a question concerning your book. Does it include a password for premium content on your blog ?
It would be interesting to run a back test taking into account a difference in beta.
Another beginner question concerning back testing ? I think you said you trade pairs where you find cointegration on no less than a 3 year basis. Do you think longer backtesting delivers better trading results for current pairs ? I have the feeling the last crash has been a real gamechanger for many pairs and that the extreme period from late 2008 and early 2009 can often kill results.

Anonymous said...

Hi Ernie,

I was hoping you could clarify some confusion I have regarding the training set in Example 3.6 in your book.

My question is if you move the training set as time moves forward. Meaning, for day 253, would you compare it against the training set of days 2:253?

Hoping this makes sense....thx

Ernie Chan said...

Hi Henn,
The password is on page 34.

Indeed, the last 2 years had not been good for pairs trading in general. So yes, backtesting last 5 years may give you a more balanced picture of its profitability.

Ernie

Ernie Chan said...

Hi Anon,
Yes, if you want to update your hedge ratio, you should move the training set one day forward every day.
Ernie

henn.de said...

I am a beginner to these type of investments searching for as much of a market neutral strategy as possible. Right now I am doing some rudimentary paper trades to test whether i have evrything straight before i start putting real money into it. Does your book include a method of testing for maximum drawdown and does it include explenations of how to use the different tools (matlab). Going to be college freshmen in engineering but have little experience with using these tools yet. Is IB the best broker for this type of business ?

Ernie Chan said...

Henn,
Yes, my book does explain how to calculate drawdown, and Matlab as well. I would rank Interactive Brokers as one of the best for pair trading tools.
Ernie

Anonymous said...

does matlab have the ability to connect to a broker via FIX connection for automated trading?

Ernie Chan said...

Anon,
Yes, see the quant2tt product of exchangeapi.com.
Ernie

Anonymous said...

Hi Ernie,

In pairs trading, you have to choose a window size for doing the regression.

In fact, this is probably the only parameter that's needed, besides the number of standard deviation away from the mean.

But the performance of pairs trading is very sensitive to this parameter, and in fact, the PNL can vary from very negative to very positive, depending on the value of this parameter.

The sensitivity curve can be very peaked, with good Sharpe ratios on only few window size values, and with mediocre (around 0) Sharpe ratios on a lot window size values and with negative Sharpe ratios on also a lot window size values.

Any thoughts? Thanks a lot!

Ernie Chan said...

Hi Anon,
The number of standard deviations to enter need not be a free parameter. You can scale the market value of your position to the zscore. Please see many discussions on parameterless trading in this blog (and discussed extensively in my workshop as well).

The lookback period also need not be a free parameter. It can be set equal to the half-life of mean-reversion computed using the Ornstein-Uhlenbeck formula, discussed in this blog, in my book, and in my workshop as well.

Best,
Ernie

Hank said...
This comment has been removed by the author.
Hank said...

Hi Ernie -
Thanks for putting together a great blog and a great book. I am enjoying both, and learning a great deal. I just made a post but saw an error in the code, so yanked it and here is the correct question:

In the pairs trading example (Example 3_6.m), for the appropriate calculation of P&L:

pnl=sum(lag1(positions).*dailyret, 2);,

should we calculate the return using the hedge ratio we used for our position sizing, as in

weightedPositions = repmat([1 hedgeRatio], [length(positions) 1]).*positions;
pnl=sum(lag1(weightedPositions).*dailyret, 2);


I definitely may be missing something here. Thanks again for sharing your knowledge.

- Hank

Ernie Chan said...

Hi Hank,
I made a simplification in the pair trading strategy in assuming that we have equal long-short capital. But you are right... strictly speaking we should use 1 share of A and hedgeRatio number of shares of B in forming the portfolio.
Best,
Ernie

Unknown said...

Ernie,
I know it is a little bit "out there", but you should come to Brazil to have your workshop!!
Just think about it, ok!?

Ernie Chan said...

Hi Bia,
Sure, I will come to Brazil if there is enough local interest.
Ernie

Hank said...

Just a follow-up to my earlier post, if anyone implements the use of the hedge ratio as I suggested: to accurately capture your performance, you either need to keep track of this value as your 'traded' hedge ratio throughout the life of that position, or you need to expect to make daily adjustments to maintain the latest hedge ratio.

Just shows you that Ernie is right about using equal long-short capital being a simplification -- it's definitely more difficult to deal with an always-changing hedge ratio!

Julia Moyes said...

Dr. Chan,

Thank you so much for providing both your book and this blog as an excellent reference for those of us wishing to learn quantitative trading. I've studied both extensively and have used your techniques as the basis for my pair-trading model for ETFs.

As I started putting real trades on, I began to quickly realize the benefit of having a model that uses dynamic parameters, and I read in an earlier post that one could use the result from the Ornstein-Uhlenbeck formula to generate the lookback period when entering a pair-trade.

I'm hoping you can clarify a bit. Example 7.5 of your book calculates the half-life of GLD/GDX at approximately 10 days. For example 3.6 in your book, does that mean you could generate the Z-score by reducing the lookback period from 252 days to only 10 days?

Thank you so much.
Julia

Ernie Chan said...

Julia,
That is right. If the OU formula tells you that the half-life is 10 days, you can use that as your lookback for calculating hedge ratio, moving avg and std.
Best,
Ernie

Hank said...

So, do you do this in an iterative way, Ernie? From what I understand from your book (as in Example 7.2 plus Example 7.5), you
1. calculate beta (the hedge ratio) using a particular lookback period and then
2. use that beta to find the residual that is then
3. used by the OU formula to tell you the half life, which you're then
4. using to find the lookback period to calculate the hedge ratio.

Perhaps you use all available data for the CADF and OU calculations, and then work the hedge ratio, moving avg, and std from there? That seems attractively "conservative", but also means that you might not be responsive to the latest data regarding cointegration.

Also, for many pairs I am getting half-life values in their lifespan that 'blow up' to very large numbers and sometimes negative ones as well, apparently when the theta value is near zero and even when the cointegration tests show high confidence levels. Do you have any suggestions for handling that, so that one can use dynamic parameters as Julia is trying to do?

Your comments from your experiences are very helpful.

Ernie Chan said...

Hank,
Yes, I agree with the iterative way you used to calculate half-life and lookback. You can certainly start with lookback=all data, but once OU suggests a half-life, you should then use lookback=half-life for your next OU calculation.

If the calculated half-life is negative or too large, I would not trade that pair even if it cointegrates.

Ernie

DanR said...

Regarding using OU to recompute a lookback window width, you say in your book there need to be N x 252 data points where N is the number of free parameters. If N=3, and I am trading daily (I require some 3 years of data), but for a given pair OU reports HL=10 days, isn't that a contradiction given the number of free parameters?

Ernie Chan said...

DanR,
Halflife is not a free parameter. So it is not subject to the N*3years rule.
Ernie

Unknown said...

When is your next Hong Kong seminar? Couldn't find it from your link.

Thanks.

Ernie Chan said...

Hi RT,
Thanks for your interest in my workshops. The Hong Kong pairs workshop scheduled for January has been converted to an e-learning course, with my narration and demo embedded, together with Q&A with me. Please check that out.
Ernie

Julia Moyes said...

Dr. Chan,

Do you know of any way to see if a stock was borrowable when backtesting?

I'm running in to this issue and how to factor it in when backtesting my ETF pair-trading strategy. I'm not sure if the short side of the trade could have been made or not.

Thanks so much,
Julia

Ernie Chan said...

Julia,
Yours is a very good question, but unfortunately whether you can borrow a stock is dependent on your brokerage. So you would have to ask your broker to see if they have such data available.
Ernie

Anonymous said...

I have to disagree about using IB for trading stock and/or ETF pairs. It is extremely cumbersome for the following reasons:

1) You have to manually specify the static ratio and then trade the spread (price difference). In reality, most people would rather trade the ratio -- e.g., buy at 2.30, sell at 2.37.

2) To use the pair trader, you have to use scale trader. There is no reason one should have to specify scale in levels simply to trade a pair.

3) If you try to make a combo instead of using the scale trader, you have to manually adjust the ratio each day to make sure each side has the proper notional. The PnL on the TWS line is never even remotely close to accurate, so you have no idea if you are making money or not.

In an ideal world, IB would make a very simple interface where you specify the pair, the buy entry ratio, buy exit ratio, sell entry ratio and sell exit ratio. You would also specify the ratio of the notional desired for each side (which has nothing to do with the price ratio of the stocks used for entry/exit). You would specify whether to be passive or aggressive on entry orders and hedge orders. You would specify what assumptions to use to display current ratio (spread crossing or not, and net or gross of est. comm), and triggering entries.

That's just a start. It's really not that hard. Stunned that IB can;t seem to put this together.

Ernie Chan said...

Anon,
Thanks for your detailed critique of IB pair trading facility.

1) I disagree that pair trading using ratio of prices is better than using difference of prices.

By trading on ratio, you are assuming R=price1/price2 is stationary, or equivalently, log(price1)-log(price2) is stationary. But maybe in reality log(price1)-hedgeRatio*log(price2) is stationary -- in that case, your ratio trading would not work well.

In my book, however, my Zscore is defined on differences, not ratios. I assume D=price1-hedgeRatio*price2 is stationary. I believe this covers a more general situation and therefore more often correct than assuming stationarity of log(price1)-log(price2).

2) I never needed to use the scale trader before to trade pairs. As you said, you can just as well use their Combo orders.

3) There is no reason one must keep the capital on each side constant or the same. I typically keep the number of shares on each side constant, and backtest shows that this method has very similar performance as keeping capital the same which requires daily rebalancing.

I am not a salesman for IB, so I am not defending IB's technology. I just want to point out that the limitations you pointed out are unique to your method of pair trading, but not to other methods.

Ernie

Anonymous said...

Hi, Ernie:

I had a hard time trying to figure out how to put multiple legs arbitrage (>3 stocks) order into Interactive Brokers. Does IB only support pairs (on their web they say they support multiple legs)? Even with 2 legs trade, they are not guaranteed to be executed simultaneously. Could you please give some hints as to how to deal with this issue?

Thanks a lot.

Ernie Chan said...

Hi Anon,
I don't believe IB allows you to create a generic combo of more than 2 instruments.

If one short leg is a stock that is hard to borrow, then it won't be executed.

I think you should ask their customer service rep to clarify such issues further for you.

Ernie

Anonymous said...

Hi, Ernie:

Thanks a lot! Could you please recommend any flatform that supports multi-leg arb trade? Or we are confined to trading pairs?

Ernie Chan said...

Anon,
Sorry, I am not aware of any platforms that allow multi-leg trades. When I want to implement such trades, I just write a Matlab program that send out orders to my broker via their API. I think this is the most universal and flexible way.
Ernie

Anonymous said...

Thanks Ernie.

How would you avoid the situation when one or some of the legs are executed while others are not? Would an autospreader help?

Ernie Chan said...

Anon,
You should always make sure that the short leg is executed first. Some auto-spreader may do that for you.
Ernie

Alpha said...

hi Ernie,

great blog; and great book.

you mentioned in one of the comments above that we can use half-life as a lookback period for moving average, stds, and ratios.

in case we trade Mean reversion on cross currency pairs; such as AUDCAD; how do we calculate the lookback of the moving average; since half-life itself is calculated based on the relaionship between the price and its moving average?

regards. ismail.

Ernie Chan said...

Hi Ismail,
In all cases, half-life is calculated on a mean-reverting time series, whether the time series is the raw price itself, or the spread between 2 instruments, or in your case, the spread between the raw price and its moving average.
Ernie

Alleres said...

Please has anyone use ArbMaker before ? is a cointegration with autotrading .i want to know if it's good.i am new in this quantitative trading.
arb-maker.com please check and advise.

Anonymous said...

Ernie, read somewhere that tws is allow to trade the ratio rather diff.

The combo currently only have diff for order

Ernie Chan said...

Hi Anon,
Actually, I haven't used ratio for trading in IB, and I am not aware that they have this feature. It is best to ask their customer service.
Ernie