tag:blogger.com,1999:blog-35364652.post7058330961437969371..comments2024-06-13T11:58:17.293-04:00Comments on Quantitative Trading: Hidden Markov model applied to FX predictionErnie Chanhttp://www.blogger.com/profile/02747099358519893177noreply@blogger.comBlogger36125tag:blogger.com,1999:blog-35364652.post-6612060498713179522021-08-16T16:50:05.459-04:002021-08-16T16:50:05.459-04:00Hi Richard,
I am waiting for the commenter to clar...Hi Richard,<br />I am waiting for the commenter to clarify what exactly s/he meant by PPMC. My guess is that s/he meant PFMC (particle filter monte carlo) which is a well-established technqiue.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-5836336174170144922021-08-11T12:55:25.271-04:002021-08-11T12:55:25.271-04:00Hi Ernie,
Regarding the discussion of the "p...Hi Ernie,<br /><br />Regarding the discussion of the "ppmc" in connection to markov models above, did you ever do any more research into this and/or figure out exactly what ppmc means in the realm of markov models? A few things I've found googling are:<br />posterior predictive model checking,<br />particle photon monte carlo,<br />parallel processing of markov chain,<br />prediction by partial match method-C,<br />pairwise partically markov chains, and finally<br />pearson product moment correlation<br /><br /><br />Thanks,<br /><br />RichardRichard Harrisonhttps://www.blogger.com/profile/09474977811542757789noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-77108109111923490732015-01-09T14:47:55.860-05:002015-01-09T14:47:55.860-05:00Well, once the subject/research directly relates t...Well, once the subject/research directly relates to money making opportunity it is totally pointless to expect any kind of useful feedback/contribution: fools contribute, smarts make money...<br />If someone has a working idea it's a very simple to validate - make money; the alternative would be to contribute and to have a lot of nice talk...Stefanhttps://www.blogger.com/profile/09443967412957527553noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-56164701256035625502014-11-23T07:56:22.637-05:002014-11-23T07:56:22.637-05:00All intraday backtesting should be done with quote...All intraday backtesting should be done with quotes instead of trades.<br /><br />Quotes are always present at 9:30am.<br /><br />Ernie<br />Ernie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-71394616911652110872014-11-23T07:55:08.344-05:002014-11-23T07:55:08.344-05:00Hi Ernie,
You mentioned in your book that You used...Hi Ernie,<br />You mentioned in your book that You used "Buy on gap" strategy in live trading.<br />How do you handle a case where there are no trades/quotes for one or more instruments during pre opening session?<br />Analyzing historical data, this case is sometimes true. An another problem occurs when there are trades/quotes but they are too old, for instance timestamp is equal 08:55 Am.<br /><br /><br />I'll be grateful for the helpAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-16000333661957270542014-11-23T07:54:25.409-05:002014-11-23T07:54:25.409-05:00Hi Ernie,
You mentioned in your book that You used...Hi Ernie,<br />You mentioned in your book that You used "Buy on gap" strategy in live trading.<br />How do you handle a case where there are no trades/quotes for one or more instruments during pre opening session?<br />Analyzing historical data, this case is sometimes true. An another problem occurs when there are trades/quotes but they are too old, for instance timestamp is equal 08:55 Am.<br /><br /><br />I'll be grateful for the helpAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-23385245055757218672014-09-20T08:05:42.753-04:002014-09-20T08:05:42.753-04:00Thanks for your report of success with the HMM mod...Thanks for your report of success with the HMM model!<br />By PPMC, do you mean particle filter Monte Carlo?<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-53608521340374190932014-09-20T05:54:32.096-04:002014-09-20T05:54:32.096-04:00This is interesting. I implemented my version of t...This is interesting. I implemented my version of the markov model and backtests gave me results of an average of 66% win rate on an hourly trading period over a cumulative trading period of 5 years. I then applied a ppmc method to these results and the win rate went up to an average of 83%. In terms of actual trading I've been trading for 7 months now and the average win ratio is 69% using both methods. It gets better with time and similarly adapts to changing market conditions so I'm confident in it. Anyways just saying that it is possible to do this thing.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-68451852451968953572014-02-15T10:57:11.848-05:002014-02-15T10:57:11.848-05:00Hi Laurent,
I have actually read this paper before...Hi Laurent,<br />I have actually read this paper before. In fact, some collaborators and I have tried to replicate and extend the results to more stocks. The effort was a failure, and reinforced my opinion that machine learning techniques that directly learn rules are unsuitable for trading.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-25375430143777096482014-02-15T03:16:05.276-05:002014-02-15T03:16:05.276-05:00Hi Ernie,
I think this paper from Jerry Hong is w...Hi Ernie,<br /><br />I think this paper from Jerry Hong is worth reading for you, very interesting (on HMM and SVM) : http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-63.pdf<br /><br />Laurent Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-57826954436230720172013-05-27T07:39:27.162-04:002013-05-27T07:39:27.162-04:00Anon,
To determine what a state variable should be...Anon,<br />To determine what a state variable should be, often you need some domain knowledge. I.e., you need more than HMM to constrain your model. A good example is given in Chapter 3 of my new book, which illustrates the use of HMM in finding the hedge ratio of a cointegrating pair of ETFs. The state variable chosen in this case is not arbitrary at all. Also, in this case, the objective is not in predicting the next measurement, though you can choose to do so.<br /><br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-34697032418427971142013-05-27T00:41:16.834-04:002013-05-27T00:41:16.834-04:00Thanks Ernie. Those functions are provided by Matl...Thanks Ernie. Those functions are provided by Matlab Statistics toolbox. There are five functions available there.<br /><br /> *<br /><br /> hmmgenerate — Generates a sequence of states and emissions from a Markov model<br /> *<br /><br /> hmmestimate — Calculates maximum likelihood estimates of transition and emission probabilities from a sequence of emissions and a known sequence of states<br /> *<br /><br /> hmmtrain — Calculates maximum likelihood estimates of transition and emission probabilities from a sequence of emissions<br /> *<br /><br /> hmmviterbi — Calculates the most probable state path for a hidden Markov model<br /> *<br /><br /> hmmdecode — Calculates the posterior state probabilities of a sequence of emissions<br /><br /><br />Regarding your comment on Predicting the State Variables, the reality is that we have no idea what are the states and how many of them should that be? so do people just assume some arbitrary states "Sunny, Rainy, Cloudy" or ie (RiskOn, RiskOff, RiskNeutral) type scenario. <br /><br />For me to get the most likely states, I need to use the Viterbi function. <br /><br />likelystates = hmmviterbi(seq, TRANS, EMIS).<br /><br />But I will need to first find out those TRANS, EMIS probability matrix given our own seq. of observations. <br /><br />[TRANS_EST2, EMIS_EST2] = hmmtrain(seq, TRANS_GUESS, EMIS_GUESS)<br /><br />After all, it sounds like there will be quite a bit of estimating guessing work here. You estimate the probability matrix, and use the estimated probability matrix to deduce your states. <br /><br />After all these hardwork, what you can find is a bunch of State numbers which they call it "Most Likely" state given "What had happened"? <br /><br />Question is how do we use it NOW for the future prediction?<br /><br />Am I missing something here? <br /><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-41380458457186379622013-05-24T10:02:57.589-04:002013-05-24T10:02:57.589-04:00Anon,
I am not familiar with the specific Matlab f...Anon,<br />I am not familiar with the specific Matlab function that you use (I use a free package instead), but generally speaking, yes, if you want to predict the next measurement variable, that's what you do. In other applications, traders are more interested in the state variable (e.g. a hedge ratio, which is not directly observable and thus "hidden"), and the state variable prediction would be the focus.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-47073315655582146102013-05-24T08:58:48.375-04:002013-05-24T08:58:48.375-04:00Ernie,
I am trying to use Matlab's HMM functi...Ernie,<br /><br />I am trying to use Matlab's HMM function to do some simple modeling. I am still trying to understand how to use all the functions to make the prediction. Say I have a time series of daily return, I change it to Up, Flat or Down (1, 0, -1) as my observation. Say I have a simple 2 states model. Now I can put the entire observation series along with some initial guess values for the Emission Probability and Transition Probability to estimate the Transition and Emission Probability matrix.<br /><br /> [TRANS_EST2, EMIS_EST2] = hmmtrain(seq, TRANS_GUESS, EMIS_GUESS)<br /><br />Now, with these two matrix, what do you do to create the new prediction? <br /><br />Do you just run [seq,states] = hmmgenerate(1,TRANS,EMIS) to generate 1 number which is your next observation sequence and call it your prediction?<br /><br />ThanksAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-25733398301241514742012-04-11T17:29:53.820-04:002012-04-11T17:29:53.820-04:00Anon,
It is up to the organizer of the workshops, ...Anon,<br />It is up to the organizer of the workshops, Technical Analyst magazine. If you are interested, please request a New York or Chicago workshop at training@technicalanalyst.co.uk<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-91910537403819468492012-04-11T13:38:56.493-04:002012-04-11T13:38:56.493-04:00Ernies,
when do you come to USA to teach Quantitat...Ernies,<br />when do you come to USA to teach Quantitative Trading class?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-35098974384609403922012-03-22T08:06:41.506-04:002012-03-22T08:06:41.506-04:00Hi Ruthstein,
I am glad you found a bug. If the pr...Hi Ruthstein,<br />I am glad you found a bug. If the programming logic are the same in Matlab and MT, then the only reason results can be different is the input data is wrong.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-38683184506752779462012-03-21T22:42:13.065-04:002012-03-21T22:42:13.065-04:00Hi ernie, thank you very much for your comments. s...Hi ernie, thank you very much for your comments. someone help me out with his plug-in for the time part and there was a a very slight error in the time preparation in MATHLAB. Still, the results remain inconsistent. But surprisingly now, the Sharpe Ratio is almost the same value for the top 5 minimal drawdown passes! but not in terms of profits, though. <br /><br />On the bright side, this makes choices way easier than before, since I just decide in terms of safest drawdown, since the sharpe ratio for all them are pretty acceptable. <br /><br />Again, thanks for your kind help and I must say, your book is a good read... I will have no doubt that I buy again your next book!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-38887869485866376432012-03-21T08:44:51.601-04:002012-03-21T08:44:51.601-04:00Hi Ruthstein,
Yes, it is likely that errors in dat...Hi Ruthstein,<br />Yes, it is likely that errors in data preparation is what caused the differences. In Metatrader, data is installed as part of the program. But Matlab is a general computing platform, much like a calculator. You have to be very careful in preparing data for input into Matlab.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-58401957238646536262012-03-21T02:45:09.734-04:002012-03-21T02:45:09.734-04:00Ernie Chan said...
Hi Anon,
When you said the resu...Ernie Chan said...<br />Hi Anon,<br />When you said the results from Matlab differs from Metatrader, can you be more specific? Are you sure that the logic in the 2 programs are identical?<br /><br />-------------<br /><br />Yes, Im very sure they are. <br /><br />Ok, i be more specific. My strategy is extremely simple, but profitable (at least for me) - just 2 lines of logic, 2 integer parameters. I cant see how or why such simple logic differs greatly, between the two. <br /><br />The difference is that in MT4 I get hundreds passes, but in MATHLAB, I only get around 50 passes. In MATHLAB, one of the 1 year test pass return a balance of 200+K from initial capital of 10K, but in MT4, the balances is within range 50K-100K, for all the passes. <br /><br />One more thing, in MT4, time of the bars are considered within the tester. I dont need to re-program anything. But in MATHLAB, I have to separate this data set. Maybe thats why the difference?<br /><br />Thx again for your kind help.<br /><br />Rgds<br />RuthsteinAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-7972888567552569122012-03-20T16:42:34.872-04:002012-03-20T16:42:34.872-04:00I also thought that the Sharpe ratio could still b...I also thought that the Sharpe ratio could still be employed in any program. Is it really just limited to Mathlab?farmland investment in Australiahttp://www.greenworldbvi.com/alternative-investments-options/agricultural-farmland/australian/noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-70142361958396591312012-03-20T08:01:50.106-04:002012-03-20T08:01:50.106-04:00Hi Anon,
When you said the results from Matlab dif...Hi Anon,<br />When you said the results from Matlab differs from Metatrader, can you be more specific? Are you sure that the logic in the 2 programs are identical?<br /><br />You can employ Sharpe ratio in any programs you choose, not necessarily Matlab. It is just mean return divided by standard deviation.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-16480968362980146402012-03-20T03:53:14.944-04:002012-03-20T03:53:14.944-04:00Hi ernie:
I am currently reading your book called ...Hi ernie:<br />I am currently reading your book called "quantitative trading", and already programmed and tried MATHLAB for backtesting. However, the results differs from MetaTrader Strategy tester/Optimization. <br /><br />In MT4, I have hundreds of passes which agree with most of my real trades (thankfully) but the latter is not as positive. I use the same dataset, which I track from 2001-2009.<br /><br />The main reason why MATHLAB is that i wish to employ Sharpe Ratio. Usually, in MT4, choosing my parameters is fairly easy, straightforward. I choose the ones with minimal drawdown + best returns, and then run separate copies of them. <br /><br />After reading your book, I was thinking of choosing parameters with:<br />1) Minimal drawdown<br />2) Best returns<br /><br />and add a third criteria, Sharpe Ratio. This way, I feel I can increase my returns, no? The formula looks complicated but nonetheless, its no harm trying. What do you think? And thanks!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-35364652.post-6082598318116842082012-03-09T08:14:01.133-05:002012-03-09T08:14:01.133-05:00I have no empirical evidence and financial predict...I have no empirical evidence and financial prediction isn't really my area of expertise. It is just that in my few attempts at using machine learning for financial predictions, I learnt that the amount of noise tends to swamp out any trends the market may have. As a result most learners tend to perform really poorly, quite possibly due to over-fitting to the training data. <br /><br />So one of my ideas is to use techniques like Maximum Entropy to reduce the degree of over-fitting. However, I have not actually tried this out.Davehttps://www.blogger.com/profile/15475375181573991056noreply@blogger.comtag:blogger.com,1999:blog-35364652.post-17611073681146609272012-03-09T07:14:25.217-05:002012-03-09T07:14:25.217-05:00Dave,
Why do you think maximum entropy HMM will wo...Dave,<br />Why do you think maximum entropy HMM will work better? It seems to be just another method to estimate the parameters.<br />ErnieErnie Chanhttps://www.blogger.com/profile/02747099358519893177noreply@blogger.com