A reader (Hat tip: Ken) told me that my new book Algorithmic Trading: Winning Strategies and Their Rationale is now available for purchase at Amazon.com. The difference with my previous book? A lot more sample strategies with an emphasis on their "rationale", and more advanced techniques. It covers stocks, futures, and FX. A big thank-you to my editors, reviewers, and you, the reader, for your on-going support.
And when you are done with it, please post a review on Amazon whether you like it or hate it!
Also, I am now offering a live online course on Backtesting in June. It covers in excruciating details the various nuances of conducting a correct backtest and the numerous pitfalls one can encounter when backtesting different types of strategies and asset classes. For syllabus and registration details, please visit my website.
260 comments:
1 – 200 of 260 Newer› Newest»Hi Ernie,
The book looks very interesting - I hope I can get a copy and take a look at it this summer (although I work in R and only know very little Matlab).
I was wondering if I may ask you a question or two if you didn't mind? I wanted to know how you would deal with the initial starting date as a nuisance parameter in backtesting? Would I just vary the starting date a certain number of times and see if the results are consistently strong across starting dates?
I also wanted to know if you knew of any resources that describe the life of daily work in quantitative finance? I know "quantitative finance" is a somewhat vague description but the reason I ask is because I've read some pessimistic accounts of what the work is like but I know that can be the case in any demanding field (the grass is always greener..); so I wanted to get some accounts of the work before I dive too deeply into computational finance as a concentration.
Thanks again!
RM
Hi Ernie,
Congrats!
-HK
I am a practitioner. Based on the summary on Amazon, at least, this appears to be one of the most comprehensive books on the subject available.
I'm curious, though, as to why you choose to disclose so much. As I said, it's very thorough -- it's different in that sense from the articles you post here. This sort of thing makes the market more efficient, particularly as the barriers to entry with respect to market access continue to diminish.
From a business standpoint, do you gain enough in course fees, book royalties, and reputation for your management company to justify giving away so much strategic information to existing and prospective competitors? And if not, then why?
That said, I'm entirely familiar with the Red Queen principle as applied to this industry. I ordered a copy immediately.
Regards,
TH
Hi RM,
To determine the best period for backtesting, one needs to have some understanding of market regimes. We should make sure to include both bear and bull periods, but avoid older data when the market regulatory structure is different. In practical terms, I usually start with data in 2007 for equities strategies.
You will find various anecdotes about what quant finance work is like in my first book Quantitative Trading. And your experience will vary greatly from one firm to another. But as you can see from my work history, I prefer to work for myself.
Ernie
Hi TH,
There are multiple motivations.
But a recent best-seller called "Give and Take" by Adam Grant explains one of them.
Also, all of the auxiliary benefits of publishing my understanding of strategies that you pointed out also apply.
As I mentioned in the Preface, many (but not all) of the strategies I discussed trace their origins to published academic work, and I believe institutional traders are already familiar with most of them. These may benefit smaller hedge funds or individual traders, but what I gain in return is valuable criticisms, error-corrections, new lines of thoughts, and constant stimulating discussions with a diverse group of traders and researchers. I worked in the secretive hedge fund industry for years, and never have I been as productive in coming up with new strategies as I do now because of such public ideas sharing.
Ernie
Thanks, HK!
Ernie
This is interesting conversation between TH and Ernie. There are tons of books and forums on trading strategies, many of which seem to make sense. But not many people would have the patience to test, implement, and most importantly, constantly improve these strategies, as it takes tremendous devotions. Also,any strategies have draw-downs, and you have to be very confident about yourself when you bet real money on it. Those who simply copy and paste the strategies may easily quit as any draw-down would appear as a sign that the strategy has stopped working, especially after it has gone public. In this sense, strategies appear in published books may even be less applied continuously. :-)
I happen to be a reader of Ernie's first book and a constant visitor on the blog here, it seems to me that many of the strategies in the book have already been covered in previous blogs. Some of the strategies (such as open gap trading) are actually "famous" over the internet (e.g., elite trader). One more thing to note: strategies (if valuable) shown in academic papers had usually been previously known among practitioners. For example, it was said that the famous BS formula has been derived well before academics (whom ironically received Nobel price for it).
Hi Ernie,
You are welcome. May I know how can I calculate the theoretical price of an index future and how does it become higher/lower then the theoretical price? Seem like this is very important to build an arbitrage strategy.
-HK
Hi Ernest,
Congratulations on the new book. I will be putting in an order tonight.
One quick question though. Did you postpone your Singapore workshop to November this year? Or are you holding this in addition to the July one which is no longer listed anywhere?
Regards,
SP.
Hi HK,
See http://www.cmegroup.com/trading/equity-index/fairvalue.html
for the formula and example of calculating fair value of index futures.
In practice, such index arbitrage strategies can be profitable only for the largest of banks, and the highest frequency traders. For a variation of the conventional index arb strategy that may still offer some profit opportunities, see the section on Arbitrage between an ETF and its Component Stocks in Chapter of my new book.
Ernie
Hi SP,
Thanks!
Yes, NTU told me that due to their scheduling difficulties, they have postponed my course to November. There will not be one in July.
Ernie
Hi
Does you book available in PDF format?
Hi Anon,
My book is available as Kindle ebook. Kindle reader is now available for free download on many tablets and mobile devices.
Ernie
Hi Ernie,
Congratulations on your new book. I have just purchased my copy and look forward to reading it. I have glanced through it and found the FX One platform you have mentioned for algo development. What is your opinion on the viability of this platform for a starting quant trader and do you see it viablefor trading using cointegrated pairs of FX pairs.
Thank you in advance.
Hi Cristian,
The FXOne platform is ideal for building execution strategies for FX strategies, especially for beginning quant traders.
You can use it for cointegrated pairs strategies.
It is not a backtesting platform.
Ernie
Hi Ernie,
Thanks for your info. I have finished reading your first book and will read your second book very soon.
-HK
It seems Kindle edition of your book is not available in Asia
Anon,
Besides Kindle, I have been told that the ebook is available on Google Books and iTunes App Store as well.
Ernie
Hi Ernie
I tried to search your book from google book but it states " No ebook available" . Also tried to searc hit from ibook store but it cannot be found too. Did they tell you when it will available roughly? Thx
Hi Anon,
That is odd - it is available on my own Google Books store. However, the official publication date is June 4, so please check back later.
Ernie
Hi Ernie,
Just started working through the book. It would appear some of the datasets are missing. Specifically, inputData_USDCAD for stationarityTests. Would it be possible to upload these data please?
Thank you,
Tony
Hi Tony,
Actually, the reason I did not upload some data is due either to licensing issue, and in this particular case, size issue (web host restriction).
However, you can easily obtain USD.CAD data for free from http://ratedata.gaincapital.com/
Ernie
I see some reviews have started to appear.
See the Amazon sales page and http://readingthemarkets.blogspot.com.au/2013/05/chan-algorithmic-trading.html
Hi Ernie,
Just got the new book today. I'm a junior quant and found your books give the best high level/practical ideas for the subjects. Thanks a lot!
Can I ask you why the new book seems to focus on stocks, ETFs, futures and FX but not vanilla fixed-income products? e.g. gov bond, bond futures, interest rate futures?
Thanks!
JQ
Hi Anon,
Thanks for the heads-up about the reviews!
Ernie
Hi JQ,
There are a few examples on trading interest rate futures, especially in the momentum chapters.
As for the underlying fixed income instruments, I have not traded them myself.
Ernie
Checked-out! Thanks Earnie, been a big fan of your website and work.
Thanks, Estinox!
Ernie
I normally don't buy books (usually go to libraries, read academic papers, or just prod friends from whom I can borrow), but the sample chapter had me hooked so I had to get this one.
Just flipping through real quick has me pretty impressed, at least as a quant newbie. I've been trading since late '08, and this has LOADS more information than you will find in 99% of practitioner books today. And it doesn't hurt that I am a fan of MATLAB, so what you've written is just my kind of reading.
Even though I haven't really dived into the book yet, I do have to say that I hope for a sequel. The mention of FXOne also perked my interest as I had never heard of it before, and after playing with it I think it is a great tool for new algorithmic traders.
By the way, although FXOne is not a backtesting platform, you CAN backtest strategies (there is even a sample workbook). Overall the platform offers quicker "plug and play" abilities to implement new strategies. Also, there is still no direct API between MATLAB and my broker(s) currently available (outside of custom dll), whereas FXOne will connect to FXCM but not my primary broker. For backtesting I still like MATLAB the best.
Anyway, cheers for the new book.
Thanks, Anon, for the kudos!
Ernie
hi Ernie
At which page of your new book, I can find the user name and password to download the source code ?
Thx
Hi Anon,
You can find that either on page 199, or for Kindle reader, the About The Website page.
Ernie
Hi Ernie,
I'm reading the first pages of your book available from the amazon preview.
I see you mention forward-looking Monte-Carlo paths.
The process for the underlying would look something like
dS/S = mu(t)dt + sig(S,t)dW, where W is a brownian motion ( ok the sig process could have more parameters but I'm looking at a simple diffusion here for the sake of being practical )
When you create stock paths like this, the question would be what do you use for sig ?
i) for intraday strategies there is no options market to look at to infer a process at all.
ii) for longer term strategies, where you have an options market, it would only make sense to use the vol surface to infer a sig process only in the context of pricing options on a strategy.
is your sigma a fixed number based on based realised vol ? is it a process ?
If it is a number then you have a random walk type of process, where no strategy would produce an alpha.
I realise that you could do a mean reverting model for the vol too, calibrate mean reversion parameters according to past data and then stick to that the diffusion process but then this begs the question of what is the timeframe upon which the parameters should be calibrated.
Not saying it's not good to have a look at these things, just dont see as much need to do them for a buy side strategy - but again I dont work on the buy side -
I'll buy your book & read it for details but I'm completely under the water with other stuff at the moment. Enjoyed your first book.
Cheers,
K
Hi K,
I did mention using Monte Carlo simulations to estimate the statistical significance of a backtest, and to estimate the tail risk of a strategy, but I don't believe I use the term "forward looking Monte Carlo path".
In the Monte Carlo simulations that I ran, the noise is not a Gaussian process, so I don't see how your argument applies here.
Ernie
Hi Ernie,
Thanks for your reply - seeing Monte Carlo, I assumed it was a forward looking thing, naturally without having the book yet I dont (yet) know how you use Monte Carlo simulations.
So essentially you are disturbing historical fixings and see how the backtest is affected by it ? this makes sense.
Cheers,
K
Dear Ernie, Thank you for writing the second book. I read your first one and liked it very much. I do not currently have Matlab since it is very expensive for me, and wanted to find out if i can use anything else instead. How about excel? Would that work? If I buy matlab, what toolkits should I buy with the basic package? Thanks veena.
Hi Ernie,
Thanks for the book, really good work! In Chapter 2 you use the routine genhurst() to calculate the Hurst exponent. I tested the routine and for genhurst((randn(1000,1))) it give 0.01 which is expected for a strongly mean reverting series. For genhurst(cumsum(randn(10000,1))) it gives ~0.5 which is also expected.
However, for genhurst(cumsum(randn(10000,1)+0.1)) it gives number around 0.53, which does not seem right for a strongly trending series. Am I missing something here?
Thanks,
Tom
Hi Veena,
Excel would work for some of the simpler strategies, but won't work for things like Kalman filter.
You can buy Octave, a Matlab clone which is just a few hundred dollars.
Most of my strategies require no additional toolboxes. Just a few require the statistics toolbox.
Ernie
Hi Tom,
Good that you are putting genhurst through some due diligence!
Have you looked at the vratiotest and see if 0.53 is statistically significantly different from 0.5? If so, then it indicates momentum.
Ernie
Hi Ernie,
Thanks for the suggestion on genhurst. I'm actually running Octave because normally I use Python and don't have Matlab. I could not find a function for the vratiotest in Python so for. Might have to write one myself, it doesn't look too difficult. I keep you posted.
Tom
Hi Ernie,
Due to the lack of available vratio tests in Python I wrote a little script in Octave that tests vratio and hurst, script below. It's a bit simplistic but it seems to give the correct results now. Not sure if this is the correct implementation of the vratio though. Any thoughts?
#a = log((randn(100000,1)+0)); #Mean reverting: hurst very small
#a = log(cumsum(randn(100000,1)+0)); #random walk: hurst ~ 0.5
a = log(cumsum(randn(100000,1)+1)); #trending: hurst > 0.5; vratio ~ 1
hurst = genhurst(a)
lag = 19;
t = (std((a(lag:end)) - (a(1:end-lag+1)) ))^2;
b = (std((a(2:end)) - (a(1:end-1)) ))^2;
vratio = t/(lag*b)
Hi Tom,
Yes, that's the correct formula for Variance Ratio Test. However, to test for the probability that this ratio is equal to 1 (null hypothesis), you need to compute a few more things.
You can take a look at Andrew Lo's book, or google online to find out the details (which I myself have ignored).
Ernie
Hi Ernie,
I just tried to search your book from google book but it still states " No ebook available". Can you please confirm with them when it will be available?
Thank you,
WL
Hi WL,
You should be able to buy it from Amazon Kindle. Kindle books can now be viewed on any computer or most mobile devices.
Ernie
Hi Ernie,
After I optimize a system with around 20 years of daily data, I get a result with 3.23 sharpe ratio, can I consider this is not curve fitting? What is the range of sharpe ratio for an extremely good system?
I did a walk forward test with one year in-sample for optimizing then the next year out-of-sample for 20 years, I get similar optimize variables result and most of the out-of-sample sharpe ratio are around 2 something to 3 something. Do you consider walk forward test is helpful?
Other than sharpe ratio, any other measurement should I use to determine the credibility of the system? Thanks.
-HK
Hi HK,
As long as your optimization is done on in-sample data, while testing is on out-of-sample data, it isn't curve-fitting.
I believe Sharpe ratio is the best indicator of statistical significance of your strategy.
Ernie
Hi Ernie,
Thanks for a great book.
Do you have any book to recommend on Econometrics that could be useful for us algo traders. I have a math background but not that familiar with Econometrics.
Thanks,
Robert
Hi Robert,
You can check out Market Models: http://tinyurl.com/mbytptb
Also, search by books by Andrew Lo.
Ernie
Hi Ernie,
I repeated the adf test for EWA-EWC from chapter 2. I used the Python adf test from statsmodels. The results are similar but the p-value and the critical values are quite different from the ones you given. Could it be that your data series uses a different tick frequency? I used adjusted close data from Yahoo.
Again, thanks for your great book.
Tom
Ernie,
Some advice...why do you not start a proper quant forum off the back of this? You've already got tonnes of followers! I find more of my questions are answered here than places like Wilmott, plus you give most the answers unlike Paul!
Just a thought
Regards
James
Hi Tom,
I also use adjusted data closing prices for my test. But if you use a different period for the data, you will no doubt get different values. Did you try it using my provided data?
Ernie
Hi James,
Thanks for your suggestion about starting a quant forum.
Well, I think the comments section already serve as a forum: readers can ask any questions there whether relevant to the topic in my post or not. And if the question is significant and points to a new direction, I will create a new post to highlight that and continue the discussions.
There are already other quant forums such as nuclearphynance.com that are very active and popular, why not use them for topics that are not in my own area of expertise?
Ernie
Hi Ernie,
I had a closer look at the calculation of the Johansen test with 3 different stocks. When you show the results you say that they all should be cointegrated with 95% certainty. However, for r <= 0 in the Eigen statistic we find a value of 16.9 which is less that the critical value for 90%. Wouldn't that mean that we cannot reject the null-hypotesis?
Thanks, Tom
Hi Tom,
Thanks for pointing that our in Example 2.7. I was focusing on the r<=2 results, and found that both Trace and Eigen statistic are larger than the 95% critical value, and therefore rejecting the null hypothesis. It is very unusual that we cannot reject the r<=0 null hypothesis but can reject the r<=2 null hypothesis, but that seems to be what the Johansen test show us in this case.
Ernie
For those who want to run Variance Ratio Test but does not have Matlab Econometrics toolbox and its vratiotest function, here is a free download:
http://ideas.repec.org/c/boc/bocode/t850804.html
(Hat tip: Antonio, our workshop participant.)
Ernie
Hi Ernie,
I have read your book and have one question about the currency pair strategy (AUD/USD and CAD/USD pairs). While the co-integration tests offer an answer about the pairs being mean-reverting, the formula also gives a hedge ratio. How do you trade the currency pairs given the fixed lot sizes. if you have a hedge ratio of 1.892 for example, do you just round up to 2? or do you trade more lots so as to come as close as possible?
Hi Anon,
If the hedge ratio is 1.892 between X and Y, why can't you buy 1892 units of one, and short 1000 units of the other? In many FX brokers, there is no minimum order size.
Ernie
Hi Ernie,
For proprietary traders in financial institutions, how many different kind of time fame and team side are there? For time fame, I know the two extreme would be HFT and holding for years kind. For team side, I heard there is single proprietary trader but most are team base.
For individual who tries to be systemic full-time trader at home, would that be very similar as a kind proprietary trader in financial institution who also has total responsibility to make trading decision by himself?
-HK
Hi Ernie,
I am using FXCM, and as far as I know they do not use odd lots, so trading 1000 and 1892 would not be possible. If they do allow odd lots, i can do that.
Thanks
Hi Ernie
I just got your new book and start trying the example codes.
Under the first example "stationarityTests.m", it loads file "inputData_USDCAD". May I know where I can download this historical data, I couldn't find it under "http://epchan.com/book2/" .
I m new to matlab, can you share with me what is the difference in between the adf() from the jplv7 package and the adftest()from Matlab library?
Thx
Hi HK,
Yes, prop traders in banks trade strategies from microseconds to months. Team sizes range from 1 person to dozens.
Ernie
Hi HK,
Yes, prop traders in banks trade strategies from microseconds to months. Team sizes range from 1 person to dozens.
Ernie
Hi Ernie,
Then what is the most common time fame and team size? Is one person team size very rarely? Why would a very large investment bank have one person team size prop trader?
Thanks.
-HK
HK,
One or two persons prop trading teams are quite common. As long as the tech/programming support is good, there is no reason why 1 person cannot manage half a billion quite profitably.
Ernie
Hi Ernie,
Thanks for your answers. For a big fund, 30% to 50% annual return is extremely outstanding. They manage too much money then there is some limitation comparing to an individual with small fund. What would be a very outstanding Sharpe ratio or annual return for an individual trading system?
-HK
Hi Anon,
There is little difference in the adf function from spatial-econometrics.com and adftest from Matlab; I have not tried the latter though.
The USDCAD data set is too large to upload to my web host. However, you can download any FX data freely from ratedata.gaincapital.com/. If you really need my data, please email me privately.
Ernie
HK,
For an individual trader, if you can get a Sharpe of 2 over several years, it would be outstanding. I refrain from saying returns because that's leverage-dependent.
Ernie
Hi Ernie,
Would leverage create a better Sharpe ratio? I thought it would...
-HK
Hi HK,
No, Sharpe ratio is not dependent on leverage. Check out its definition and you will see why.
Ernie
Hi Ernie,
I mean:
If I have a strategy that works with an index, let's say I get a Sharpe ratio 1.5 with real asset ETF. If I optimize the variables for the same index future, very likely I will get a better Sharpe ratio like 2 with better "expected value of the excess of the asset return over the benchmark return" in the Shapre ratio definition.
-HK
Hi HK,
Sharpe ratio is not leverage-dependent if you are applying a strategy to the same instrument, but with different leverages. If you are applying a strategy to an ETF vs a future, we are not just changing the leverage, so the Sharpe ratios may well be different. (For e.g. we incur financing costs for ETF, but not futures.)
Ernie
Hi Ernie,
Do you trade any momentum strategies and at what time frame in that case? I hope you don't mind me asking.
Thanks,
Bobby
Hi Bobby,
I do trade momentum strategies. Some of the ones I considered are discussed in the second half of my new book.
Ernie
Hi Ernie,
Yes, Sharpe ratio stays the same with leverage because gain and lose both increase in the same ratio.
Is that true that even trying to keep parameters numbers within 5 to 6 to prevent curve-fitting, I can have a very complicated math/stats model to deal with these 5 to 6 parameters?
-HK
Hi HK,
Complexity of a model refers to both number of parameters and rules. With many trading rules, you can still overfit your model.
Ernie
Hi Ernie,
If the columns of data increase like including open interest, volume, some other daily/weekly data from government/stock exchange are used with price instead of just price itself, then can the ideal number of parameters be much more than suggested maximum 5 in total?
-HK
Hi Ernie,
I bought your book on Kindle the first week and finished it almost immediately. Thanks for writing that. I have failed with strategies that up to well into millions HMM/NN/GA optimized parameters before, and I became a believer of simple modeled strategies after your first book. Compare to 7 years ago when you first started independent, how easy do you find exploring/tweeking simple strategies now? I haven't had many tradable ones besides tweeking your book's. Thanks
IC
Hi HK,
If you have more complex input data, you can judiciously add more parameters.
5 is not some magic number ... choosing the right model to start is more of an art rather than science. It is validating this model that is a science.
Ernie
Hi IC,
I find it is easy to come up with models that work, at least for some period of time, and for low capacity. It is much more difficult to find models that continue to work for a long time and have high capacity (> $10M, say).
Ernie
Hi Ernie,
I have two questions:
1. When working for a firm, is there any way in which I can protect the intellectual property rights of my strategies? At least minimizing the risk of being sued, should I quit and start elsewhere.
What's your experience in terms of property rights? Are legal fights common or how is this in general dealt with when changing firms?
2. Very different question: I have a strategy with a Sharpe of 2.3 from 1997, assuming 30 bps roundtrip costs, for Russell 1000 stocks. It is a daily strategy backtested on end of day quotes.
Do i need to backtest it also on intraday day data since I cant really trade on end of day quotes?
In practice, I would trade 1555-1600.
Are the last minute prices in general close to the closing price so in the long run, the difference washes out?
Thanks a lot
Hi Anon,
1) The best way to take your IP away with you when you quit is to negotiate that BEFORE you start the job. I know some hedge funds allow that. Once you signed an employment contract, you have given up your rights to your employer. So you are at the mercy of their IP enforcement. Some firms won't hesitate to sue. (Of course, you can also negotiate this just before you quit.)
2) You should focus on results only since 2007. Market structure is so different before that year that the backtest is not only useless, but misleading.
You should either use midpoints or primary exchange closes for this backtest. The "last price" reported in most daily historical prices is often far from the price you can be filled.
Ernie
Hi Erine,
I read a article mentioned of using TLS (total Least Square) to find the spread ratio instead of using OLS.
The reason is TLS
minimizes the distance from each (xi, yi) point to the regression line. The distances to be minimized are vectors which
are orthogonal to the regression line, hence the alternative name "orthogonal regression." That distance is unchanged if we exchange X and Y , so the resulting fit is unchanged, too. That gives the needed symmetry.
I would like to hear your opinion, thx
Hi Ernie,
I was going through your book and I think you have a bug in example 7.1 on page 156. The last line is (op-cl)./op but it should be (cl-op)./op
The strategy as given in the code (and as simulated) is a mean-reversal rather than momentum strategy.
Hi Anon,
TLS sounds like a very reasonable scheme - thanks for introducing it to us. It should be a useful substitute for Johansen to find hedge ratios in a symmetric way.
Ernie
Hi Thomas,
You are right ... the return should be (cl-op)./op.
Thanks for pointing it out!
Ernie
Anyone know what happened to earnings.com? Seems to have integrated into Reuters and lost a lot of its functionality!
Hi Anon,
I still use earnings.com very recently for dividends and splits info. Which functions have disappeared?
Ernie
Hi Ernie,
Thanks for you reply about IP rights above.
One should be able to include a clause in the contract saying that both parties don't have the right to sue and that both the employee and his former employer have the right to use whatever algos the employee developed. What do you think of this?
I've also heard that as long as you don't take "physical media" when you leave, e.g. printouts etc, you are fine. You are allowed to reconstruct your algos from memory at your new employer. Have you also heard that?
Hi Anon,
Sure, if you have included a clause in your contract stipulating that the IP can be used by you in perpetuity, that would be very nice.
With regard to physical media or not, it is a murky area. I am not a lawyer, and have no idea how this will play out in the court room if they do sue.
Ernie
Hi Ernie,
What is your input on implementing the strategies you present on CFDs? they offer higher leverage, without owning the actual stock.
What do you think about them as an asset class?
Regards
Hi anon,
Some of the strategies I discussed will no doubt work on CFDs as well, as long as the bid-ask spread is narrow enough. Feel free to backtest them on this asset class!
Ernie
Hi Ernie,
got the book last week and have been going through it in detail. I'm using R with Bloomberg data and have some trouble recreating some of the results. Do you think you could add the data (EWA, EWC, IGE, GLD, ...) you use on your website in a non-MATLAB -format (eg. .csv, .txt)? Thanks.
Regards,
TM
Hi Ernie,
Have you ever used IB's NYSE Closing Auction D-Quote Orders? Are they reliable?
Those are order sent to floorbrokers and will get you the closing print. You can send them up until 3:59:40, which is very nice compared to 3:45:00.
Hi TM,
Actually, some readers are glad that I used .mat format for the data, since it is a bit cumbersome for Matlab to read in text files.
As there are a large number of input files involved, I am afraid I won't be uploading the text files, as they occupy much more disk space and my web host has a strict limit on file upload size.
Ernie
Hi Anon,
Thanks for the tip about D-Quote Orders. I have never heard of them, let alone used them!
Ernie
Hi Ernie,
thanks for your answer. Luckily I found a workaround: the R package "R.matlab" works great for reading .mat files into R.
Regards,
TM
Hi TM,
Thanks for telling us about R.matlab!
Ernie
Hi Ernie,
Do you use a database to store historical data and then connect matlab to it when you need to backtest or live trade? What database do you use?
-HK
Hi Ernie,
Do you have any trick to come up with good estimates for variances for the Kalman filter? Do you use the ALS method? I've been using using trial and error but I kind of what to verify the that my results makes sense.
- BM
Hi HK,
I do not use any databases to connect to Matlab. Every trader that I met, whether institutional or individual, has used plain text files or Excel files for data storage.
Ernie
Hi BM,
I think you meant the initial estimates of the variances? If so, I simply used the estimates quoted in the Montana paper quoted in my book. See the last paragraph of Box 3.1 for details.
Ernie
Hi Ernie,
Thanks. For IB API activex vs java socket topic, is it safer or "better" to use java socket? I guess you are using activex, do you have any problem with it especially with your millisecond time frame system?
-HK
Hi Ernie,
IB mentions "ActiveX can lose events" which sounds scary:
http://www.interactivebrokers.com/en/?f=%2Fen%2Fsoftware%2Fhighlights%2FapiHighlights.php
That should mean...a buy/sell event can be lost...
-HK
Hi Ernie,
Yes, I did mean initial estimates, thanks. I've been running your example KF_beta_EWA_EWC but on intraday data. I find I have to reduce delta and Ve quite a bit or Q will be to big and not make sense as compared to e(t). Or am I missing something here?
Btw at the top of KF_beta_EWA_EWC.m you have:
% 1 minute data on EWA-EWC
Do you mean daily? Otherwise you have to adjust Sharpe calculation I think.
-BM
Hi HK,
We use a Java API to connect with IB. The packet dropping I think mainly refers to market data, not order or position updates.
Ernie
Hi BM,
Yes, as I wrote, delta and Ve have to be quite small to get good results, but I don't see why small values are necessarily bad.
Yes, I meant daily data.
I will change that comment, thanks!
Ernie
Hi Ernie,
I will use Java API to connect with IB as well since I am not good at C++ and Java API socket is better than activex.
For IB, what is the fastest data feed that they can provide? How much money would it be? How many ms time unit does it take from them to your server? Do you rent a server near exchange or you can host it at home? For the time to receive data feed, computer does analysis, then send back possible buy/sell signal, how much time does it take for you? IB is the only choice that has api in HK, while they have local server in HK so it should be less than 10ms from my home to their server.
-HK
Hi Ernie,
I just find out one of the three computer centers of IB is in HK
http://www.interactivebrokers.com/en/?f=%2Fen%2Fsoftware%2FsystemStatus.php%3Fib_entity%3Dllc
This could be very helpful for me if I can find a good strategy works well with ms time frame. I expect ms time frame is a different world from second/minute time frame.
-HK
Hi HK,
Don't expect IB's data feed to be better than 250ms, no matter how much money you pay them.
Better to use IQFeed if you want true tick feed.
I locate my trading program at a 3rd party data center close to IB's CT data/order server. If you can do that in Hong Kong, that's great.
The network latency is 10-20ms with this arrangement.
Ernie
Hi Ernie,
Do you use IB data feed? Can you choose which IB server do you get data feed? I find a old web page having 3 IB data feed IP but the HK one is not working. Their data feed and order have different ip. If I need to get data feed from IB usa server, that is already at least 200ms delay already.
-HK
Hi Ernie,
You just mentioned IB's data feed to be better than 250ms, do you mean even you rent a server close to ib data center in US, the data feed is still over 259ms delay? Or you mean their data feed is a snapshot and the refresh frequency is not better than 250ms, so they can send out around 4 snapshot every second.
I use a tool online to ping IB data feed server from a USA location, the delay is around 5ms.
-HK
Hi HK,
I don't think you can choose which region's IB data server to use, but you should direct that question to their customer support.
The 250ms latency is the update frequency (snapshots of data) of the data feed.
Ernie
Dear Ernie,
Thank you for another great book again.
I have a question regarding using Co-integration to trade basket. In your book, you use "Price" to identify pair which cointegrate with Index and then set up an "EQUAL WEIGHT" basket to test Cointegration between Basket Value and Index ETF using "Log Price".
It still bugs me that you suggest "Equal Weight". I wouldn't doubt that the statistical test suggest that. But in some of these ETFs, the top 3-5 names is like 50% of the basket with some of the top names sometimes as big as 20-30%. But assuming an equal weight, do we end up running a lot of trading error? I know the whole point of trading imperfect basket using Cointegration is to come up with a mean-reverting "Tradeable" tracking error.
Any better suggestion so we don't run too big a risk. I know most indexes in the US like S&P has diversified weight. But if you take things like the Hang Seng in Hong Kong, HSBC alone is like 15%. If I take 10 cointegration stocks and assume 10% each, it seems I am running quite a big tracking risk here.
Hi Ernie,
We know there is Forex Economic Calendar.
I wonder if there is software or website we can use to download long history economic event data for backtesting and real-time feed for live trading.
Many thanks.
Hi All,
Any comment on Kyte Group as a broker ?
And as a prop trading group ?
Thanks
Hi Ernie,
I know you use Mathlab for statistical R&D. Is it fast enough to backtest tick data ?
I believe you dont use Matlab to make your algorihmic trading, how do you enforce that your mathlab backtest is equivalent to your alogrithmic trading ?
I am currently amking my R&D and the algorithmic trading will be done in C++. I am afraid of the risk of not doing the same thing on the two sides.
Thanks
I loved your second book !
Hi Anon,
Whether the tracking error is big or not is not really important. The more important question is whether the tracking error mean-reverts, and how much tail risk there is.
If you feel the tail risk is too big, you can certainly weight your stocks differently from equal-weights to resemble the ETF composition better.
Ernie
Hi Anon,
Each type of economic event has its own website with its own historical data. For example, http://www.federalreserve.gov/releases/h15/ gives you access to historical interest rate changes.
You would have to go to each organisation's website for similar data since I don't know of a place that consolidate them and is free.
Ernie
Hi Ken,
I am afraid I have not heard of Kyte Group.
I do use Matlab to backtest tick data, but usually one has to break up the data into many chunks to avoid memory problems.
Indeed there is a software risk in backtesting in Matlab and executing in C++, but the gain in productivity in using Matlab for backtesting is very considerable. We always backtest the same strategy again using a compiled language that can be used execution also, and make sure that the backtest is still good.
Ernie
Thanks Ernie,
So you have your trading rules both inside mathlab and the compiled language solution ?
Do you copy/paste these trading rules between mathlab and the execution language or do you have any kind of direct connection between the two ?
Hi Ken,
No, there is no direct translation between Matlab and C# which is what we use.
In fact, we like the independent verification of the soundness of the strategy.
Ernie
Hi Ernie,
I bought the book and
I would like to download your book's examples, but I don't know where are located the password and username. I was looking pag 19 and 42. Could you help me?
Hi Ernie,
If I understand correctly, you are no longer using matlab for execution.
Are you using Lime Strategy Studio as your EMS now?
Hi Ernie,
I have been reading your blog and new book, and I have found the material very helpful... thanks a bunch.
I have a very specific question about an equation you have mentioned in your book. On page 68 (Example 3.1) of your Algorithmic Trading book, the hedge ratio between two assets is calculated using CADF method. It is done assuming that y is the dependent asset and x is the independent asset. Later in the example when you calculate the positions based on applying a simple mean reversion algo on the price spread, you multiply numUnits (MR outcome) by the weighting matrix [hedgeRatio -ones(size(hedgeRatio))] to retrieve positions on assets x and y. Based on your formulation shouldn't the weighting matrix be [-hedgeRatio ones(size(hedgeRatio))] since spread=y-hedgeRatio*x?
Thanks,
HE
Hi Jose,
The userid/password are on page 199.
Ernie
Hi Ernie,
Are you not using matlab for execution anymore?
Are you using Lime Strategy Studio as your current EMS? If not, then what do you recommend?
Hi HE,
From regression, we find y=hedgeRatio*x. So the market value of the hedged portfolio is -hedgeRatio*x+y, or sum([x y].*[-hedgeRatio ones(...)], 2)=sum([-hedgeRatio ones(...)].*y2, 2) as I wrote in Example 3.1.
Ernie
Hi Anon,
I use Matlab for executing low frequency strategies, and C# for higher frequency ones. I do not currently use Lime Strategy Studio.
Ernie
Then what EMS do you recommend or what are you using at the moment or are you just coding everything manually?
Anon,
We code our execution codes in C# ourselves.
Ernie
Are proprietary execution management systems, worth it, in your view?
Also, is there a reason for choosing c# over java and c++?
Is c# considerably easier, for example?
Thank You
Hi Ernie,
Thanks for your response. I agree with you on how you calculate the market value of the hedged portfolio. However when you move on to calculate the "positions" in example 3.1, you used [hedgeRatio -ones(...))] * repmat(numUnits, ...) whereas i believe it should be [-hedgeRatio ones(...))] * repmat(numUnits, ...). Do you agree?
Hi baran,
Yes, I agree with you.
Actually, the Matlab code on epchan.com/book2 is correct ... I just forgot to update the text in the book.
Thanks for pointing out.
Ernie
Hi Anon,
I have not used any EMS myself, as I prefer the maximal flexibility of building our own.
C# is easier to program than C++, and the IDE is better than Java. But I learn this second-hand, as my partner does all the C# development.
Ernie
Hi Ernie, your book is very interesting. I tried to replicate your results. I downloaded daily FX data (USD/CAD) from ducascopy.
Half-Life of Mean Reversion (p.48) = 115 days
I get (after applying OLS with a constant) a beta of -0.004026 which results in -0.3010 / -0.004026 = 74.77 days.
If I calculate Half-Life with an AR(1) Process, I get the parameter 0.995444 which results in 151 days. (log0.5 / log(abs(parameter)
Could you also calculate Half-Life with an AR(1) process?
Which method do you prefer in general?
Thank you very very much for your answer!
Hi Ernie
I'm trying to run bog.m, pag: 94. But I can not find in http://epchan.com/book2/ , funcions: smartMovAvg, SmartStd. Do i have to look in another place?
Thanks
Hi Half-Life,
Yes, you can also calculate half-life using AR(1). AR(1) and OU are fundamentally the same: it is just that OU derives from a continuous equation. I have no preference amongst the 2, though it is more common to use OU.
Ernie
Hi Jose,
Thanks for the heads-up: I have uploaded the 2 missing functions to book2.
Ernie
Hi Ernie,
I find your work (blog, books) very interesting and your message that retail traders can succeed with algorithmic trading is inspiring.
Personally, however, I struggle with the mathematics since my background is in the soft sciences. I was wondering if you could make a comment about the mathematics involved with quantitative trading, and how one might best re-educate themselves for this field (if that’s a viable option in your view).
Have a great weekend,
JM
Hi JM,
For the type of trading that I wrote about, college level linear algebra and statistics would be more than sufficient.
Ernie
Hi Ernie,
For monthly release data like federal interest rate change, there would be just 12 "bars" per year to backtest, and very likely it just has main change like once in years. Would that make it not statistically significant enough to consider there would be predictive power?
-HK
Hi HK,
Statistical significance is not just a function of the number of trades, but of Sharpe ratio. (See my new book for elaboration.)
Also, if you have strong fundamental reasons for believing the trade should work, that adds to your confidence too.
Ernie
Dear Ernie,
I have a question regarding the Basket Arbitrage in your Example 4.2. After you run Johansen test on individual stocks against the ETF and figure out the specific numbers of stocks to be included in the "Sub-Basket", you said that you would create portfolio with EACH CAPITAL WEIGHT and rerun another Johansen test on this Sub-Basket against the ETF. For this "Equal Weight" basket, which stock price do you use to calculate the WEIGHT for the second Johansen test? Do you use "last price" of each chosen stocks or "beginning price" of the data series?
Thank you
SM
Hi SM,
After you have formed the equal-weighted portfolio, you will use the log of the closing prices of those stocks to find the log market value of the portfolio to test for cointegration. We should not use the raw prices between the number shares may be rebalanced each day.
Ernie
Hi Ernie,
Another great book! Many thanks. Both of your books have been very helpful for me to get started with my own algo trading.
Do you think the type of strategies you talk about is suitable for intraday trading of fx? I haven't done any tests on this asset class myself yet. But was concerned that spreads might be to wide for intraday trading on this fairly efficient asset class. Any suggestions on an asset class I should look at to get started with intraday trading.
Many thanks
John
Hi John,
FX is certainly suitable for intraday spread trading. I don't believe the spread between any 2 currencies is necessarily narrow.
Ernie
Hi Ernie
I get this error: kentdaniel.m
idx=setdiff(idx, nodata, 'stable');
Error using setdiff (line 141)
Unknown flag.
Hi Ernie,
You mention in your book that it's hard to find futures that are suitable for pairs trading. Do you think that's true on an intraday time frame too, i.e. with holding period from a few minutes to let's say 3-4 hours?
Thanks
John
Hi Jose,
Make sure you are using the latest version of Matlab.
Ernie
Hi Anon,
It may be easier to find mean-reverting futures pairs intraday.
There may be opportunities during specific hours.
Ernie
Hi Ernie,
I bought your new book. It is a great book, especially the fact that you have introduced so many strategies. Though I have heard of most of them but your wonderful presentation sheds new light on it and clarifies the maths underpinning the models which is very important.
I have heard that Kalman filter can be used for intraday trading. Do you have an example of it where it is used as both momentum and mean-reversion. I am new to Kalman so I am going by what I have read elsewhere.
Thanks
John
Hi Ernie,
I am trying to find files that can be used for loading in stationarytests.m.
inputData_USDCAD is not located at http://epchan.com/book2/
One of your earlier posts said to go to ratedata.gaincapital.com, but I couldn't find out which columns correspond to the 'hhmm' 'tday' and 'cl' variables.
Also in AUDCAD_daily.m, I could not find the AUD_interestRate file in epchan.com/book2.
I did not find any spreadsheets at ratedata.gaincapital.com that corresponding to interest rates.
Thank you!
Matthew
Hi Ernie,
I am trying to find files that can be used for loading in stationarytests.m.
inputData_USDCAD is not located at http://epchan.com/book2/
One of your earlier posts said to go to ratedata.gaincapital.com, but I couldn't find out which columns correspond to the 'hhmm' 'tday' and 'cl' variables.
Also in AUDCAD_daily.m, I could not find the AUD_interestRate file in epchan.com/book2.
I did not find any spreadsheets at ratedata.gaincapital.com that corresponding to interest rates.
Thank you!
Matthew
Hi John,
Certainly, Kalman filter can be used for intraday trading: there is no restriction on the frequency of Kalman updates.
The 2 examples in my book are both mean-reverting. Offhand, I don't know of any papers using it for momentum trading - however, if the state variable represents returns instead of prices, you can conceivably build a momentum assuming that the returns are as predicted by the equations.
Ernie
Hi Matthew,
The AUD_interestRate.mat file is definitely on epchan.com/book2.
For the other files, they are too big to load to that server. Please email me, together with the password you used to access book2, and I will provide you with a link to another server.
Ernie
Thank you Ernie!
What email address can I use to reach you?
Hi Matthew,
You can find my email here:
http://www.epchan.com/e-mail/
Ernie
Hi Ernie¡
I would like to program events news strategies for futures similar as you say in your book for stocks. Do you know a web site where I can download historical world economics news?
Thanks;
Hi Jose,
Try http://biz.yahoo.com/c/e.html for US economic calendar.
Many FX brokers' websites also have global macroeconomic data release calendar.
Ernie
Hi Ernie.
Do you know what happened to Alphacet? Was it bought out or something?
Also, what do you think about this for orderbook simulation with matlab?
http://www.mathworks.com/matlabcentral/fileexchange/23514-order-book-simulation
Hi Anon,
Yes, as far as I know Alphacet is looking for a buyer, or already bought.
Thanks for telling us about the Matlab order book simulator. It looks interesting, though I haven't used it and don't see any other reviews or comments about it.
Ernie
Hi Ernie,
I believe I never or seldom see you mention volume in your blog or books, so how do you think about volume? Base on my own testing so far, I cannot find any way to put volume in my analysis to have good result. You and mostly other professional traders you know mainly just use price High Low Open Close as the key analysis?
-HK
Hi HK,
I did mention how we can incorporate volume in order flow calculations in my book. Except for order flow calculations, I have not found volume to be of great predictive power.
Ernie
Hi Ernie,
I haven't finished reading your new book yet so I guess I haven't read that part yet.
-HK
Hi Ernie,
Firstly thank you for the great book!
Secondly, i think there is a problem with parseEarningsCalendarFromEarningsDotCom.m due to changes in output format in prevEarningsFile and todayEarningsFile.
Would you pls let us know how it's supposed to be corrected?
Thank you in advance.
DKNY
Hi DKNY,
Yes, earnings.com has removed a lot of the free info from its website. I think Yahoo Finance and earningswhispers.com both have earnings calendars, but I have not attempted to parse their websites recently.
Ernie
Thank you Ernie! What is the assumed bid/ask in your backtest for that strategy? Its hft more or less so a lot is going to depend upon those assumptions.
BR,
DKNY
Hi Ernie,
Thanks for a very good book. A couple of questions:
1) Do you prefer OLS or Kalman filter to modelling hedge ratio? Could you maybe elaborate why/when one is preferable to the other approach.
2) For FX mean reversion strategies, do you think it's possible to be profitable trading two pairs against each other? Or do people mainly look at a baket of currencies?
3) I've modified some of your examples to include trading costs (spread, transaction fees, slippage). I am using hourly data. Do you think it make sense to e.g. calculate deviation from mean every hour but let's say only recalculate hedge ratio every day? When updating the hedge ratio evert hour I get a very nice spread zig zag pattern around the mean but I'm struggling to create a profitable strategy. I believe mainly due to the hedge ratio changing to much. Maybe my pair is just not good enough? Any help is much appreciated as I'm a bit stuck getting started with my first strategy.
Many thanks
Bobby
Hi DKNY,
I am assuming you are referring to Example 7.2? That is not a high frequency strategy as it enter at the open and exit at the close, so bid-ask is not as important. In the book, I assume no transaction costs, as explained in the Preface, but I normally assume 5 bps one-way transaction costs for equities strategies.
Ernie
Hi Bobby,
1) I normally just use OLS with a moving window, due to its simplicity and fewer parameters. Kalman filter is more principled and does not have the problem of a sharp cutoff of a moving window, but it does have more parameters to optimize, and is more complex.
2) You can certainly trade 2 FX pairs against each other profitably.
3) I think hedge ratio does not need to be updated very often, certainly not intraday.
Ernie
Thank you Ernie! in that case it's not pactical to trade t. Taking into consideration bid/ask, market impact and broker fees it will definitely have negative Sharpe.
BR,
DKNY
Hi Ernie
5bps tcosts sounds low if one considers mkt impact, slippage etc. Have you checked your actual realized tcosts?
Hi Anon,
tcost depends on the stock universe, the time of execution, and the type of orders.
The strategy in discussion trades only SPX stocks, and execute only with MOO and MOC orders. In my experience, under these conditions 5bps per side is appropriate.
Ernie
Hi Ernie,
I am trying to run your Example 1.1 in Matlab 2013a, but got error
Undefined function 'lag'
May I know which version of Matlab you are using? or is that lag function a self-developed function which is not in Matlab package?
Thank you,
WL
Hi Ernie,
I am trying to run your Example 1.1 in Matlab 2013a but got the error
Undefined function 'lag'
Is 'lag' a function you developed so do not exist in matlab or it is version issue?
Thank you,
WL
Hi WL,
Thanks for letting me know.
I have uploaded a new version that uses backshift function instead of the lag function. (The lag function is available from spatial-econometrics.com).
Ernie
Hi Ernie,
Thank you for the prompt reply. spatial-econometrics is very useful.
By the way, in the % Randomized entry trades hypothesis test, I think the vector 'longs' probably should be 'longs_sim', to match with codes before it.
Thanks,
WL
Hi WL,
Actually, I believe in the randomized entry trades hypothesis test, longs_sim=longs(P) and shorts_sim=shorts(P) are correct, since we simply randomized the actual trades here. We shouldn't be randomizing other simulated trades from before.
Ernie
Hi Ernie
In your book you mentioned intraday momentum strategy which uses DRN etf as an inderlying.
As an entry signal you take a 2% move in DRN. Are you sure there is no typo here?
I thought one would need to use 2% move in the index (6% move on the leveraged etf) as an entry signal. I dont have an access to hft data, so I cant check it myself.
Thank you.
DKNY
Hi DKNY,
I don't believe there is a typo for the DRN strategy. The 2% is the entry return for the levered ETF.
Ernie
Thank you Ernie!
How does the Sharpe looks like if you increase the treshold to 4%, 5% etc?
DKNY
Hi DKNY,
There are far fewer trades if you increase the entry threshold to 4% or above. Therefore, the Sharpe ratios dropped.
Ernie
Hi Dr. Chan,
Thanks for your great effort on this book. I'm trying to reproduce your results in example 2.1. But where I can find your "inputData_USDCAD"?
Thanks,
Andrew
Hi Ernie,
You mentioned you trade intraday fx mean reversion. What time frame is that, i.e. minute, hourly etc? Or do you do high frequency? How many trades does your intraday strategies generate?
Thanks for a great blog!
Hi Ernie,
You mentioned you do intraday fx strategies. What time frame is that, i.e. minute, hourly etc? How many trades per day do you typically generate in intraday strategies.
Thanks for a great blog!
Hi Anon,
My intraday FX trades' average holding period is an hour, and there are on average just 1 round trip per day.
Ernie
Hi Andrew,
Due to the large size of that input file, please contact me privately for download.
Ernie
Hi Ernie,
You mention GLD/GDX pair in your book. Do you believe this pair is cointegrated at the moment? A portfolio of this pair for the last 3 years doesn't look very stationary.
Thanks
Hi Ernie,
You mention GLD/GDX pair in your book. Do you believe this pair is cointegrated at the moment? A portfolio of this pair for the last 3 years doesn't look very stationary.
Thanks
Hi Anon,
Actually, GDX-GLD has stopped cointegrating back in 2008, for reasons discussed in Chapter 4 of my new book.
Ernie
Hi Ernie,
I have just come across this and have purchased the book. I recently finished reading 'Practical Speculation' by Victor Niederhoffer and Laurel Kenner and was wondering have you read it. They had an interesting section on 'Complexity' in relation to trading strategies, specifically the importance of adjusting the strategy for changing circumstances. We summarised the section here if any of your readers are interested in a short read about this: http://www.stocktipsnetwork.com/blog/stock-picks-a-question-on-complexity-you-re-advisor-won-t-want-you-to-ask
I look forward to reading your take on this subject.
Conor.
Hi Conor,
Thanks for mentioning Victor's book. I haven't read it but the idea sounds interesting.
Ernie
Hi Erine,
I have couple strategies where I use MATLAB for analysis and spam CSVs to/from MetaTrading4 FX trading platform. I am sure it is not the best way of doing things.
If you have ever used MATLAB to trade FX, would you shed some lights on which broker you use / relatively more MATLAB friendly?
I think interactive brokers can link to MATLAB but my small capital does not meet their requirement.
Thanks!
IC
Hi IC,
I use Matlab to connect with Interactive Brokers. I do not believe there is any minimum capital requirement for using this method. Check out exchangeapi.com or undocumentedMatlab.com
Ernie
Hi Erine,
Thanks for your last reply, I will look into it. Another one. Do you find long public holidays, say 4 day in a roll thanks giving, influence results/returns of a mean revert strategy that uses cointegration (ie Johansen test)?
Thanks,
IC
Hi IC,
I have not noticed any unusual mean reversion properties due to long holidays.
Ernie
Hi Ernie,
Which data provider do you use for historical intraday ETF data? kibot.com seems to be a popular one. I have not yet used it myself though.
Many thanks
Hi Anon,
Do you need tick level data? If so, Nanex.com and tickdata.com are both good. I heard kibot.com provides that too.
If you need only 1-sec bars, Interactive Brokers will give 1 year of that for free if you use their API.
Ernie
Many thanks. I only need 1-minute bars but ideally 5 years of data. Might go with kibot.com.
Post a Comment