tag:blogger.com,1999:blog-35364652.post9161010336089772883..comments2024-04-16T09:30:34.503-04:00Comments on Quantitative Trading: The Pseudo-science of Hypothesis TestingErnie Chanhttp://www.blogger.com/profile/02747099358519893177noreply@blogger.comBlogger90125tag:blogger.com,1999:blog-35364652.post-25387952582380428822023-12-27T08:08:16.457-05:002023-12-27T08:08:16.457-05:00Hi Anon,
If the null hypothesis is the mean return...Hi Anon,<br />If the null hypothesis is the mean return of a strategy being zero (or some generalization of that), and you rejected it, that doesn't mean that your mean return is really non-zero. However, if you can't reject it, you can be pretty sure the strategy is very weak or is just random.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-21640868631248918222023-12-26T09:38:07.702-05:002023-12-26T09:38:07.702-05:00Hi Ernie, just curious - if the null hypothesis is...Hi Ernie, just curious - if the null hypothesis is simply the detrended strategy return (as recommended in “Evidence Based Technical Analysis), would this not be sufficient to support a good hypothesis test? The purpose simply being to create a reference sample distribution which can be used to test whether the algo returns are likely significant? This feels like a straightforward and reasonably valuable test to me.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-3942665205642779842013-05-24T08:22:47.562-04:002013-05-24T08:22:47.562-04:00Hi RM,
Thanks for the clarification.
In my opinio...Hi RM,<br />Thanks for the clarification.<br /><br />In my opinion, hypothesis testing can still be useful in backtest as a way to reject weak strategies, though it can't positively affirm a good strategy is not due to luck alone.<br /><br />Also, as my forthcoming book will show, sometimes the failure to reject a null hypothesis lead to interesting new insights about what drives the profits of a strategy!<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-35502231287376379732013-05-24T03:16:30.460-04:002013-05-24T03:16:30.460-04:00Hi Dr. Chan,
I think there is a difference betwee...Hi Dr. Chan,<br /><br />I think there is a difference between the two examples.<br /><br />Example 1:<br />1) If a person is an American then it is highly unlikely she is a member of Congress.<br />2) The person is a member of Congress.<br />3) Therefore it is highly unlikely she is an American.<br /><br /><br />Example 2:<br />1) If a returns distribution is normal, then it is highly unlikely we will have a 6-sigma return.<br />2) Our return is 6-sigma.<br />3) Therefore it is highly unlikely the returns distribution is normal.<br /><br />In the first example, Gill states 2), but when he refers to a member of congress, he is referring to a member of only American congressmen. The congressmen he uses for 2) are a strict subset of Americans, the group in 1), (he does not allow the group "Americans" to vary in any way). The universe for congressmen is too restrictive by being only American.<br /><br />In your example, example 2, a "6-sigma return" in 2) is not a strict subset of the normal distribution, the group in 1). In other words, there are 6-sigma returns for the Cauchy distribution, exponential distribution, uniform, etc.. <br /><br />In my opinion, we should make the analogy:<br /><br />Example 1:<br />1) If a person is an American then it is highly unlikely she is a member of Congress.<br />2) The person is a member of Congress, including Congresses from various countries.<br />3) Therefore it is highly unlikely she is an American.<br /><br />and Example 2:<br />1) If a returns distribution is normal, then it is highly unlikely we will have a 6-sigma return.<br />2) Our return is 6-sigma, including 6-sigma returns from various statistical distributions.<br />3) Therefore it is highly unlikely the returns distribution is normal.<br /><br />I agree with your sentiment that rejection of the null hypothesis is clearly not enough for backtesting strategies (I am planning to comment again tomorrow to ask for your ideas on what else we can do), but have to disagree with Jeff Gill's opinions regarding hypothesis testing. On the other hand, some push-back is definitely needed for the number of researchers who blindly run regressions until their p-value drops below that magical arbitrary threshold of 0.05 so that they can tell a story to fill in the rest. <br />There has also been papers recently which have studied the number academic papers reporting various p-values which shows a very obvious game being played, I'll try to remember the name.Reid Mintohttps://www.blogger.com/profile/07159127479675176880noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-35149153220225858922013-05-23T09:48:33.426-04:002013-05-23T09:48:33.426-04:00Hi RM,
Let me rephrase your H0 about stock returns...Hi RM,<br />Let me rephrase your H0 about stock returns distributions.<br /><br />1) If a returns distribution is normal, then it is highly unlikely we will have a 6-sigma return.<br />2) Our return is 6-sigma.<br />3) Therefore it is highly unlikely the returns distribution is normal.<br /><br />Do you agree this is the logic?<br /><br />If you substitute "returns distribution is normal" with "a person is an American", and "a 6-sigma return" with "is a member of Congress", then we are back to the probabilistic syllogism which you have regarded as absurd.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-55950841347389324362013-05-23T06:11:58.933-04:002013-05-23T06:11:58.933-04:00Hi all,
There seems to be much confusion here reg...Hi all,<br /><br />There seems to be much confusion here regarding what a hypothesis test is and the conclusions we should draw from the test. <br /><br />First, the example<br /><br />"1) If a person is an American then it is highly unlikely she is a member of Congress.<br />2) The person is a member of Congress.<br />3) Therefore it is highly unlikely she is an American."<br /><br />in no way, shape, or form imitates the logic of a hypothesis test. In terms of probabilities, what the example is saying is<br /><br />1) P(C|A) is low<br />2)&3) P(C|A)==> P(A|C) is low, which is absurd<br /><br />It makes little sense to talk about probabilities of the null hypothesis, P(H0). A null hypothesis is usually a distributional statement, not a random variable to which we can assign probabilities. The null hypothesis is either true or it's not (without getting too philosophical). <br /><br />Hypothesis testing uses the statistical analogue of a proof technique in mathematics called proof by contradiction. For our hypothesis test, we first say okay, let's assume that our null hypothesis H0 is true.<br /><br />1) Now we are in a world where H0 is true, or that some distributional statement holds. This is the truth in our world now.<br />1)* stock returns are normally distributed<br /><br />2) If H0 is true, then the probability of this event of happening is extremely low, or P(A|H0) = extremely low<br />2)* If stock returns follow normal dist., then we should rarely see eight sigma events in a 250 trading year, if at all in our lifetime<br /><br />3) Therefore, we have statistically convincing but not definitive evidence that H0 is not true<br />3)* We observe (clueless guess) multiple eight sigma events a year/decade etc.., therefore it's reasonable to think that stock returns are indeed not normally distributed<br /><br />In short, all a hypothesis test is saying is that suppose someone wins the powerball lottery eleven times, wouldn't you question that it is not due to random chance but to cheating, or not as hard to win lottery as you thought, etc. ?<br /><br />But I completely agree researchers abuse hypothesis tests when they have no idea how to properly use them....Reid Mintohttps://www.blogger.com/profile/07159127479675176880noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-40919963414937076412013-03-22T08:11:03.383-04:002013-03-22T08:11:03.383-04:00Hi Winfred,
HTB stock list is easy to get on a dai...Hi Winfred,<br />HTB stock list is easy to get on a daily basis, even from Interactive Brokers' website. But it is hard to find historical records of it. So you have to save them yourself going forward.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-14368703276072622842013-03-21T23:35:51.331-04:002013-03-21T23:35:51.331-04:00Hi Ernie,
I see. To filter out the HTB stocks, wh...Hi Ernie,<br /><br />I see. To filter out the HTB stocks, where could we get the HTB stock list? I think it is easy for hedge fund to get it from stock loan desk of brokage firm. But it may be quite difficult for individual to use that channel? <br /><br />Thanks,<br />WinfredWinfrednoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-334446643041170882013-03-21T10:13:55.220-04:002013-03-21T10:13:55.220-04:00Hi Winfred,
The cost of shorting common stocks dep...Hi Winfred,<br />The cost of shorting common stocks depend on the stock, in particular, it depends on whether the stock is hard-to-borrow.<br /><br />Pair trading can still work if you pick the stocks that are not HTB.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-26058268095225975852013-03-21T06:03:33.613-04:002013-03-21T06:03:33.613-04:00Hi Ernie,
You said it is very expensive to borrow...Hi Ernie,<br /><br />You said it is very expensive to borrow Inverse ETFs. How about short common stock? I think in Asian markets, it is also costly and difficult for individual investor? Then it means those pairs trading strategies, mean reverting strategies (normally requires short and long) would not be an option for an individual? <br /><br />Winfred<br /><br /><br /><br />Winfrednoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-36615753888397224592013-02-05T13:58:48.554-05:002013-02-05T13:58:48.554-05:00Peter H. claims that a job like that should take a...Peter H. claims that a job like that should take around ~$600.<br /><br />http://epchan.blogspot.com/2009/05/matlab-as-automated-execution-system.html<br /><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-38368203642180822022013-02-05T13:49:13.647-05:002013-02-05T13:49:13.647-05:00Morningstar, I know, offers bulk data downloads of...Morningstar, I know, offers bulk data downloads of tick data at the end-of-day via FTP. So you can start using that in matlab or python, right away.<br /><br />With DTN, I believe it is API-access only, so unless you know c or c++(I think DTN also has the api accessible via VB), you're going to have to hire someone to wrap the c/c++ code for you via swig in python or mex-files in matlab.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-6998494631662992692013-02-05T08:00:25.600-05:002013-02-05T08:00:25.600-05:00Anon,
Yes, the 1 minute time stamps of CQF are ann...Anon,<br />Yes, the 1 minute time stamps of CQF are annoying. Thanks for the tip about Morningstar and DTN.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-67869358590927312962013-02-04T22:13:20.231-05:002013-02-04T22:13:20.231-05:00DTN/Nanex nxcore is also said to be another decent...DTN/Nanex nxcore is also said to be another decent historical tick data provider. <br /><br />Both morningstar and DTN/nanex also offer real-time data.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-22232845373046628022013-02-04T22:07:25.274-05:002013-02-04T22:07:25.274-05:00Ernie,
cqgdatafactory
CQG offers historical tick...Ernie,<br /><br />cqgdatafactory<br /><br />CQG offers historical tick data, but only at 60 second time-stamps.<br /><br />Morningstar Quotes (formerly known as Tenfore) seems to offer full historical tick data at even the millisecond and sub-millisecond level.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-39919727285155778972013-02-03T02:06:14.282-05:002013-02-03T02:06:14.282-05:00Hi Ernie,
In your book, you mention that AUD/CAD ...Hi Ernie,<br /><br />In your book, you mention that AUD/CAD is relatively stationary.<br /><br />Do you know any other currency pairs which is relatively stationary, like AUD/CAD?<br /><br />Thanks a lot.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-30392061916192354392013-01-30T12:52:30.301-05:002013-01-30T12:52:30.301-05:00Ferdi,
Interesting book - thanks.
ErnieFerdi,<br />Interesting book - thanks.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-31117697197972912012013-01-30T12:29:36.726-05:002013-01-30T12:29:36.726-05:00You should read The cult of statistical significan...You should read The cult of statistical significance by Ziliak and McCloskeyFerdinoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-35985382656006420582013-01-30T10:21:40.326-05:002013-01-30T10:21:40.326-05:00@anon,
The last 2 papers that you referenced are t...@anon,<br />The last 2 papers that you referenced are the same: is that intentional?<br />Thanks,<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-17957813112504791172013-01-30T07:39:55.556-05:002013-01-30T07:39:55.556-05:00Anon,
Thanks for the references. I will study them...Anon,<br />Thanks for the references. I will study them and perhaps post my opinions in the next blog post.<br /><br />Ken: One of my consulting clients signed up to be the seeder of my first fund. Generally investors approach me out-of-the-blue because they know me through my blog, book, and workshops.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-47143988457724561872013-01-29T18:15:35.979-05:002013-01-29T18:15:35.979-05:00Ernie,
I am trading futures on commodities. I did...Ernie,<br /><br />I am trading futures on commodities. I didnt find any website close to currensee for commodities.<br /><br />Can you tell us how did you manage to seed your own fund ?<br /><br />ThanksKenhttps://www.blogger.com/profile/02203529699193949304noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-31576834159764591132013-01-29T12:40:17.691-05:002013-01-29T12:40:17.691-05:00Hi Dr. Chan,
My apologies about "Dear Chan&q...Hi Dr. Chan,<br /><br />My apologies about "Dear Chan" in my previous message it was meant to say "Dr. Chan". <br /><br />In regards to some of the material that I have found so far pertaining to using options to help one choose an entry point:<br /><br />This is an old article from trading markets but it is a primer on the concept<br /><br />http://www.tradingmarkets.com/.site/stocks/education/strategies/01042000-3274.cfm<br /><br />and these are the other ones<br /><br />http://web.ics.purdue.edu/~zhang654/jfqa_option.pdf<br /><br />This one was my favourite so far<br /><br />http://www.ruf.rice.edu/~yxing/option-skew-FINAL.pdf<br /><br />The majority of the research done so far is applied towards event trading (i.e. earnings releases); however, as a prop trader who trades intraday and needs to earn a return daily and monthly... I was wondering what your opinion might be on some of the implications towards intraday trading. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-13376187828630397132013-01-28T18:48:53.824-05:002013-01-28T18:48:53.824-05:00Ken,
If it is an FX strategy, check out websites s...Ken,<br />If it is an FX strategy, check out websites such as currensee.com. I am sure similar sites for equities strategies exist.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-19601476088766590192013-01-28T18:35:29.979-05:002013-01-28T18:35:29.979-05:00@ Andrew
I have three big kinds of strategies. I ...@ Andrew<br /><br />I have three big kinds of strategies. I feel that I can have between 5 and 100 times more capital depending on the strategies.<br /><br />I plan to begin taking on outside allocation soon but I dont want to rush it because its a big responsability. Even if I say to my seeder that I can loose money I know that they dream of my past result. So I want to have more track record and experience to have the maximum security for them.<br />As I am young and without a long and classic background, its also quite difficult to find seed money !<br /><br />@ Ernie & All<br /><br />I begin to look at seeding solutions but it seems that everyting is done for big players around 50 millions of $.<br />How can you seed smaller amounts ? (outside family and friends) <br /><br /><br />ThanksKenhttps://www.blogger.com/profile/02203529699193949304noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-52579564565886939522013-01-28T09:41:59.498-05:002013-01-28T09:41:59.498-05:00Hi Peter,
The delay should not be more than a few ...Hi Peter,<br />The delay should not be more than a few minutes from the original, otherwise it would be a totally different strategy!<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.com