I have given a 2-part interview (here and here) on the various nuances of backtesting on tradingmarkets.com. Most of the ideas have been covered in my book, but it does serve as a summary of what I consider to be the most important issues.
For those of you who are interested, I may be giving a workshop on general techniques in backtesting in London as well, in addition to my pairs trading workshop. Additional details will be available on epchan.com at a later date.
Dr.Chang,
ReplyDeletejust out of curiosity. Why are you making all these methods available for public consumption if indeed they are usefull?
I'm wondering why Jim Simon doesnt have or didnt have a blog or wrote a book when he started?
Ernie,
ReplyDeleteon p 58 of your book, what you call positions actually look more like trades. Did you mean trades? If thats the case then pl should be lag1(cumsum(positions)).*dailyRet,
right?
To #1 :
ReplyDeleteDr. Chang is not publishing(or selling) a magic formula - just the ingredients that might be useful. All ingredients have a shelf life and expiry. Some ideas will be useful, others will be irrelevant.
It is up to you to come up with what works and what doesn't.
All major works of science and innovation have been built up the work and ideas of others and predecessors.
Even Jim Simmons' Chern-Simmons quantum field theory is based on ideas and work of predecessors.
The positions on p58 are actually positions. The "longs" and "shorts" are trades.
ReplyDeleteErnie
By publishing the general techniques of algorithmic trading and sometimes specific strategies, I benefit from feedback, critiques and corrections which enable me to improve my own strategies and techniques.
ReplyDeleteFurthermore, as I stated in my book, I don't believe that these techniques and strategies are all that proprietary. Only the details and subtleties of implementation are really proprietary.
Ernie
Isnt position supposed to be cumsum of trades?
ReplyDeleteI dont see anywhere where you actually cumsum trades.
What am I missing?
There is a line
ReplyDeletepositions=fillMissingData(positions) on pg. 58.
It unfortunately was concatenated to a previous comment, so it wasn't too clear on the printed page. This line will carry forward any trades until their exits.
Positions are not just cumsum of trades as I defined the trades. It is not easy to distinguish exits and shorts if you just do cumsum.
Ernie
I have read with interest your book and the part on back testing. I am looking for your opinion on using a rolling Sharpe-ratio and especially the volatility of the rolling sharpe ratio as an indicator of model stbility (making out of sample tests an unnecessary step)
ReplyDeletenikkus,
ReplyDeleteRolling Sharpe ratio is a good idea, but I don't think that obviates the need for out-of-sample testing. True out-of-sample testing is done on a time period when you have ceased any optimization or improvement on the model. It is often the case that when the rolling Sharpe ratio doesn't look good, one would change the model to make it look better. In that case, the rolling Sharpe ratio will typically be inflated relative to a true out-of-sample test.
Ernie
thanks for your prompt reply, but I am still not convinced. Assuming a strategy using daily data for the last 10years, with a relativily good rolling sharpe ratio (over a period of one year) showing value between 2 and 3 for the last 10 years. It would mean that if we would have chosen the first 5 years and the last 5 years as my 2 sample set, both would have shown value between 2 and 3, the out of sample test is confirmed. no need to approve my comment if you still feel you should have the same reply.
ReplyDeleteIs it correct to say that you will have a lookahead bias in backtesting if trades are done on the basis of the signals based on the low or high of the day. How do you get rid of it? if you trade a. next day at the same price or b.at next day open with some random noise? or c. trade the same day which is the industry practice(?)
ReplyDeleteYou cannot know the low and high of the day until you are at the close. If your signal is at the close, or next day's open, then there won't be look-ahead bias.
ReplyDeleteErnie
Even close cannot be known,until after the close. If I am implementing the cross-sectional strategy on close. I can close existing positions as MOC. But how do I place MOC for new positions with new weights at close when I don't know the closing prices exactly? Surely there is slight look ahead bias if I have to estimate the weights?
ReplyDeleteEven close cannot be known,until after the close. If I am implementing the cross-sectional strategy on close. I can close existing positions as MOC. But how do I place MOC for new positions with new weights at close when I don't know the closing prices exactly? Surely there is slight look ahead bias if I have to estimate the weights?
ReplyDeleteYes, the close cannot be known, but we can come very close to a good estimate if you submit an order 1 second before close.
ReplyDeleteAll automated strategies incur slippage cost relative to backtest, so this slight difference between actual close and "signal-triggering" close is not unique.
Ernie
ref MOC above, just noticed you already answered it on pg 105, though I termed it as slight lookahead bias.
ReplyDelete