tag:blogger.com,1999:blog-35364652.post2087571512274908786..comments2024-03-15T10:28:18.248-04:00Comments on Quantitative Trading: Beware of Low Frequency DataErnie Chanhttp://www.blogger.com/profile/02747099358519893177noreply@blogger.comBlogger138125tag:blogger.com,1999:blog-35364652.post-42486820213096653222021-07-20T08:12:16.549-04:002021-07-20T08:12:16.549-04:00Hi Mark,
Generally, we like IQFeed / DTN's dat...Hi Mark,<br />Generally, we like IQFeed / DTN's data feed.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-56430087719536607902021-07-20T01:43:06.588-04:002021-07-20T01:43:06.588-04:00Hi Ernie,
For those of us requiring less granular...Hi Ernie,<br /><br />For those of us requiring less granularity in our data (EOD futures data), how do you like Kinetick (DTN IQ)? You list a number of data providers across your 3 books, but this is not one of them.Marknoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-83948539951608191732019-11-01T07:28:01.945-04:002019-11-01T07:28:01.945-04:00Anon,
While most official closing prices in US sto...Anon,<br />While most official closing prices in US stocks will be close to the mid price of the minute bars at the close, they don't match exactly.<br /><br />There are not many hours of overdoing things: either you have the official prices, or you don't. You just have to pay for it (from Bloomberg for e.g.)<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-61060048997803794902019-11-01T07:07:40.604-04:002019-11-01T07:07:40.604-04:00You don't need such data .. you just have to t...You don't need such data .. you just have to think a bit more how to use your 1 min data, a little bit of creativity saves many many hours of over doing things.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-30465874632623651722019-07-24T09:52:54.381-04:002019-07-24T09:52:54.381-04:00Hi James,
Sorry for the late reply ... I just saw ...Hi James,<br />Sorry for the late reply ... I just saw this sitting in my Inbox.<br /><br />5bps is the typical commission + half-spread of a SP500 stock. If you are trading small caps, it could be 4 times larger.<br /><br />As my article suggests, the posted Closing price is often not the same as the true auction price you will get filled MOC. So in effect we still have to allow for half-spread to account for the difference.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-12946387334632319432019-01-22T22:15:28.423-05:002019-01-22T22:15:28.423-05:00Hi Ernie,
Thanks for the great article. I also at...Hi Ernie,<br /><br />Thanks for the great article. I also attended the online meetup where you gave this presentation back in 2015. <br /><br />I have developed and backtested two equity strategies, but do not yet have any trading experience with stocks.<br /><br />You mention above that 5bps is a reasonable cost estimate for S&P 500 stocks, and I have seen this figure in various other places. I am trying to understand where this estimate is coming from?<br /><br />If once uses MOC or MOO orders as in your first two examples, then we "avoid paying the bid-ask spread". I checked the exchange transaction fees at: <br /><br />https://www.nyse.com/publicdocs/nyse/markets/nyse/NYSE_Price_List.pdf<br /><br />and at $0.0004/share for Tier 1 stocks, this will typically be much less than 1bps. The commission fees at IB are also per share, and I believe are also well under 1bps. Is this 5bps cost estimate taking into account market impact? If so, shouldn't this be a function of the amounts being traded?<br /><br />James <br /><br />Anonymoushttps://www.blogger.com/profile/08003874824276906542noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-84051021925179259762018-08-21T10:05:35.101-04:002018-08-21T10:05:35.101-04:00Hi Ben,
Yes, prices on Yahoo and Google *may* diff...Hi Ben,<br />Yes, prices on Yahoo and Google *may* differ from that on Bloomberg IF you select the primary exchange prices on Bloomberg. Not all prices on all stocks differ every day, but they do occasionally for some stocks.<br /><br />This is applicable only to US exchanges. I cannot speak for other exchanges.<br /><br />5 bps is a reasonable cost estimates for S&P 500 stocks. It is difficult to measure market impact on open and close prices. I haven't backtested this strategy recently, so I don't know if it continued to perform well.<br />Ernie<br />Ernie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-71692259874720050952018-08-21T09:14:19.347-04:002018-08-21T09:14:19.347-04:00Hi Ernie,
Are you saying that open/close prices ar...Hi Ernie,<br />Are you saying that open/close prices are different on yahoo or google than they are on Bloomberg HP page? Is this just the case in the US? In your second example is 5bps an accurate assumption of trading costs, did you find there was any market impact, has it continued to perform well 3 years later? <br />Many thanks<br />BenBen Humanhttps://www.blogger.com/profile/10589814609924411774noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-65383565842047866852018-01-19T13:34:54.424-05:002018-01-19T13:34:54.424-05:00Beppe,
Yes, my total returns are an arithmetic sum...Beppe,<br />Yes, my total returns are an arithmetic sum, not a compound return. This is useful for Sharpe ratio calculation, or if the trader wants to bet a fixed amount of capital each day.<br /><br />However, even if you were to compute compound return, it isn't correct to multiply the price with the returns. They are totally different animals. Compound return formula would be just cumret=cumprod(1+retO2C)-1.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-48485781335441800252018-01-19T10:32:59.624-05:002018-01-19T10:32:59.624-05:00Believe it or not, I made the same changes before ...Believe it or not, I made the same changes before your reply!!!<br />Well...to be honest, "almost" the same: I forgot the "minus" sign. Ironically I got a far better APR (more than 20%) while Sharpe almost equal. Unfortunately I still have values too far from yours. It's clear that depends on functions and by some changes I made. No problem, I think for the moment it's more important understand the "meaning" of a strategy. So now I can go on with the book. <br /><br />Now my very last question: <br /> "retO2C=(cl-op)./op;<br /> pnl=sum(positionTable.*(retO2C), 2);<br /> ret=pnl/topN;"<br />You sum returns for every day in order to have a final column vector pnl. It's all ok but...summing percentages from different prices shouldn't require us to multiply these percentages for the opening prices? I mean, that way we have the absolute value we're really gaining/loosing. Or maybe I'm doing confusion because I don't even understand why dividing for 10.<br />Anyway, I abused of your patience so thanks again for your help. Beppenoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-30193391121704540862018-01-19T08:52:49.666-05:002018-01-19T08:52:49.666-05:00Hi Beppe,
I believe you can substitute op(t, :) &l...Hi Beppe,<br />I believe you can substitute op(t, :) < buyPrice(t, :) with retgap < -stdretC2C90d, but note the minus sign. <br /><br />The function lag is in spatial-econometrics.com which you can download for free. But lag(x) can be substituted by backshift(1, x), and backshift is a function from epchan.com/book2.<br /><br />Matlab is mostly backward compatible. If you find a piece of code doesn't work, most likely it is because you need to download some utility functions from my website or from spatial-econometrics.com.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-39918775180665052372018-01-19T08:33:01.184-05:002018-01-19T08:33:01.184-05:00Thanks for your reply!
Yes you're right about ...Thanks for your reply!<br />Yes you're right about the point 2): the moving average is computed over closing prices and not return (sorry yesterday I was clearly done with my mind!!).<br />The point "1" instead...I'm still not sure about it: Shouldn't it be: retgap < stdretC2C90d? Anyway, I'll work on it.<br /><br />The real problem is (for the previous and future strategies I'll read on the book) working with different versions of Matlab (as I can't use any longer function as "lag" etc.) but I thought I managed to change functions in a way to get close to your results. I was clearly wrong (my Sharpe Ratio is only 0,63 compared to your 1,5 and my APR 0,0048 instead of 8,7%!!!). I don't know where, but I've failed into coding. So, is there by any chance, an update version of this book's Matlab code?<br />Thank again for your attention and your patience!<br />BeppeBeppenoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-44702658945473956192018-01-18T21:04:43.244-05:002018-01-18T21:04:43.244-05:00Hi anon,
1) The buyPrice was indeed set to backshi...Hi anon,<br />1) The buyPrice was indeed set to backshift(1, lo).*(1-entryZscore*stdretC2C90d); <br /><br />But note the condition <br /> hasData=find(isfinite(retGap(t, :)) & op(t, :) < buyPrice(t, :) & op(t, :) > ma(t, :));<br /><br />So only those days with op < buyPrice will be used.<br /><br />2) Indeed "open prices to be higher than the 20-day moving average of the closing prices. "<br />and ma is computed as ma=backshift(1, smartMovingAvg(cl, lookback));<br /><br />Note ma is NOT computed as moving average of returns. It is computed as moving average of close prices.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-24456403646567047912018-01-18T11:42:26.270-05:002018-01-18T11:42:26.270-05:00Hi, I'm following your book codes (strategy &q...Hi, I'm following your book codes (strategy "Buy on Gap" pg.93-96) on Matlab but as I'm using the 2016 version I had to rearrange some of your old functions (i.e "lagmatrix" instead of "lag" and "tsmovavg" + "movstd" instead of your personal functions for moving averages and moving standard deviations").<br />Anyway...I honestly find very different results from yours even if using the very same data!<br />Another thing I noticed is it seems you're not following your own conditions: for example:<br />1) You calculate 90 days moving average for daily returns (close to close) and say that we have to consider only (today open/yesterday low) returns < than the 1 standard deviation from 90 days movaverages just computed. But...in your code you make the previous day low - 1 standard deviation. Isn't it wrong??<br /><br />2)in the second condition you say that today low must be < than 20 days mov.average of closing prices. So, why in the code I read ma= moving average of close to close returns????<br /><br />Please, I hope you can help me as I'm just a newbie matlab user and I really love to follow your work. Thanks<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-66967012994137287222016-07-14T08:01:29.295-04:002016-07-14T08:01:29.295-04:00Hi Slav,
Yes, your idea for using a volume bar at ...Hi Slav,<br />Yes, your idea for using a volume bar at the open is certainly a good alternative of using the official open price, which often has limited size. However, even though that auction size is limited, it isn't quite limited to 100 shares. You can typically trade at least $200K to $500K order depending on the market cap of the stock without moving the auction price.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-91183058194846006122016-07-14T00:08:41.144-04:002016-07-14T00:08:41.144-04:00Ernie, I see your point on figuring out the correc...Ernie, I see your point on figuring out the correct open and closing prices in order to conduct a proper backtest. This is where I wanna make a suggestion of forgetting ohlc data, I find liquidity way more important of an issue unless you only trade 100 share or 1 contract positions. Volume based bars or tick bars which are constructed off 1min may be a little better.<br /><br />As an example you would trade 500 share bars, and if you need to execute at open you would take the worst price of the range or median price of 1000share bar range as your price of execution.(latter is better because in reality even if you tried to execute this 500 share trade you wouldn't have been able to within first bars range because that would assume you were only buyer).My point is all these back tests are pointless without tick data - period. <br /><br />slavAnonymoushttps://www.blogger.com/profile/00993561158019019612noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-56249956249680012942016-06-19T19:33:45.987-04:002016-06-19T19:33:45.987-04:00Hi Sqrt Alpha,
I don't exactly know what you m...Hi Sqrt Alpha,<br />I don't exactly know what you meant by order or deal matching. My instinct tells me that my article isn't about what you are thinking.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-63815066851845134432016-06-19T13:49:24.473-04:002016-06-19T13:49:24.473-04:00Hi Ernie,
It seems like a discussion on mechanism ...Hi Ernie,<br />It seems like a discussion on mechanism of order matching. Though low-frequency data like day bar or month bar we backtest, matching a deal through tick data is a necessity. Correct?Sqrt Alphahttps://www.blogger.com/profile/14496960785936577254noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-20318263012880626382016-01-14T16:30:32.105-05:002016-01-14T16:30:32.105-05:00Ernie,
thanks so much for your quick reply! Now t...Ernie, <br />thanks so much for your quick reply! Now this makes sense and the book chapter helped. <br />Nielsnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-9817621216664127742016-01-14T07:37:54.220-05:002016-01-14T07:37:54.220-05:00Hi Niels,
Any "white" data noise always ...Hi Niels,<br />Any "white" data noise always inflates the returns of a mean-reversion strategy. (See explanation on p. 10 of my book Algorithmic Trading.) In fact, white noise is the best kind of noise for mean reversion. Consolidated stock/ETF/CEF data suffers from white noise, while primary exchange data does not.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-19293331222817888472016-01-13T21:25:55.475-05:002016-01-13T21:25:55.475-05:00Ernie,
thanks for the blog and information. About...Ernie, <br />thanks for the blog and information. About the 'CEF Premium Reversion' strategy I wonder if you have an explanation why the different price changes the outcome significantly? I understand that the price is different but why does it result in a disadvantage, in particular when you trade long and short positions? <br />Thanks, NielsNielsnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-9629410108032392522015-06-16T20:17:00.280-04:002015-06-16T20:17:00.280-04:00Hi Karen,
If you have generated a vector of tradin...Hi Karen,<br />If you have generated a vector of trading signals (buys or sells), you should use actual contract prices to determine the returns based on these trading signals.<br /><br />But in generating a trading signal at time t, you can back-adjust futures for all time t' <= t to create a continuous contract. In particular, if you want to run a Johansen test on spreads of different futures, the backadjustment should involve only additions. If your signal is generated based on ratio of prices, the backadjustment should involve only multiplications.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-65716970345046702632015-06-16T16:10:07.192-04:002015-06-16T16:10:07.192-04:00Hello Erine
I'm reading your book, at Page 14...Hello Erine<br /><br />I'm reading your book, at Page 14, you mentioned of continuous future contracts for arb trading.<br /><br />You said : If your strategy generates trading signals based on the<br />price difference between two contracts, then you must choose the price<br />back-adjustment method; otherwise, the price difference may be wrong and<br />generate a wrong trading signal..... However, if your strategy generates trading signals based on the ratio of prices between two contracts, then you must choose the return<br />back-adjustment method"<br /><br />But, after reading what you replied above, saying that professional futures traders stick to actual contract to backtest, and manage the roll over to the next contract by buying/selling on the rollover day.<br /><br />I'm a little bit confused of what should be the correct method to backtest, Let say I have 3 years data of future contract of commodity A and commodity B, and the data shows that Feb and August are the active contracts for both of A and B usually. Therefore I pick 2012 Feb, 2012 Aug, 2013 Feb, 2013 Aug, and 2014 Feb, 2014 Aug for commodity A , same for commodity B for backtesting<br /><br />Based on what you mentioned above, I shouldn't do any back-adjustment of the return series from A and B before applying any ADF, johansen Test to it? Please kindly advise. Thank You<br /><br />Karen<br /><br /><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-73536164244231195502015-06-08T15:38:25.497-04:002015-06-08T15:38:25.497-04:00If you back-adjust futures prices by eliminating t...If you back-adjust futures prices by eliminating the return gap during rollover, you will create a price gap, and vice versa. There is no way to avoid both, and so both cases involve look-ahead bias. Hence professional futures traders stick to actual contract prices.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-10843099494063849412015-06-08T15:37:09.233-04:002015-06-08T15:37:09.233-04:00Hi David,
NN requires a large amount of data to av...Hi David,<br />NN requires a large amount of data to avoid overfitting financial data. Hence it may be useful in high frequency trading.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.com