Friday, September 05, 2014

Moving Average Crossover = Triangle Filter on 1-Period Returns

Many traders who use technical analysis favor the Moving Average Crossover as a momentum indicator. They compute the short-term minus the long-term moving averages of prices, and go long if this indicator just turns positive, or go short if it turns negative. This seems intuitive enough. What isn't obvious, however, is that MA Crossover is nothing more than an estimate of the recent average compound return.

But just when you might be tempted to ditch this indicator in favor of the average compound return, it can be shown that the MA Crossover is also a triangle filter on the 1-period returns. (A triangle filter in signal processing is a set of weights imposed on a time series that increases linearly with time up to some point, and then decreases linearly with time up to the present time. See the diagram at the end of this article.) Why is this interpretation interesting? That's because it leads us to consider other, more sophisticated filters (such as the least square, Kalman, or wavelet filters) as possible momentum indicators. In collaboration with my former workshop participant Alex W. who was inspired by this paper by Bruder et. al., we present the derivations below.

===

First, note that we will compute the moving average of log prices y, not raw prices. There is of course no loss or gain in information going from prices to log prices, but it will make our analysis possible. (The exact time of the crossover, though, will depend on whether we use prices or log prices.) If we write MA(t, n1) to denote the moving average of n1 log prices ending at time t, then the moving average crossover is MA(t, n1)-MA(t, n2), assuming n1< n2.  By definition,

MA(t, n1)=(y(t)+y(t-1)+...+y(t-n1+1))/n1
MA(t, n2)=(y(t)+y(t-1)+...+y(t-n1+1)+y(t-n1)+...+y(t-n2+1)/n2

MA(t, n1)-MA(t, n2)
=[(n2-n1)/(n1*n2)] *[y(t)+y(t-1)+...+y(t-n1+1)] - (1/n2)*[y(t-n1)+...+y(t-n2+1)]    
=[(n2-n1)/n2] *MA(t, n1)-[(n2-n1)/n2]*MA(t-n1, n2-n1)
=[(n2-n1)/n2]*[MA(t, n1)-MA(t-n1, n2-n1)]

If we interpret MA(t, n1) as an approximation of the log price at the midpoint (n1-1)/2 of the time interval [t-n1+1, t], and MA(t-n1, n2-n1) as an approximation of the log price at the midpoint (n2-n1-1)/2 of the time interval [t-n1, t-(n2-n1)], then [MA(t, n1)-MA(t-n1, n2-n1)] is an approximation of the total return over a time period of n2/2. If we write this total return as an average compound growth rate r multiplied by the period n2/2, we get

MA(t, n1)-MA(t, n2)  ≈ [(n2-n1)/n2]*(n2/2)*r

r ≈ [2/(n2-n1)]*[MA(t, n1)-MA(t, n2)]

as shown in Equation 4 of the paper cited above. (Note the roles of n1 and n2 are reversed in that paper.)

===

Next, we will show why the MA crossover is also a triangle filter on 1-period returns. Simplifying notation by fixing t to be 0,

MA(t=0, n1)
=(y(0)+y(-1)+...+y(-n1+1))/n1
=(1/n1)*[(y(0)-y(-1))+2(y(-1)-y(-2))+...+n1*(y(-n1+1)-y(-n1))]+y(-n1)

Writing the returns from t-1 to t as R(t), this becomes

MA(t=0, n1)=(1/n1)*[R(0)+2*R(-1)+...+n1*R(-n1+1)]+y(-n1)

Similarly,

MA(t=0, n2)=(1/n2)*[R(0)+2*R(-1)+...+n2*R(-n2+1)]+y(-n2)

So MA(0, n1)-MA(0, n2)
=(1/n1-1/n2)*[R(0)+2*R(-1)+...+n1*R(-n1+1)]
 -(1/n2)*[(n1+1)*R(-n1)+(n1+2)*R(-n1-1)+...+n2*R(-n2+1)]
+y(-n1)-y(-n2)

Note that the last line above is just the total cumulative return from -n2 to -n1, which can be written as

y(-n1)-y(-n2)=R(-n1)+R(-n1-1)+...+R(-n2+1)

Hence we can absorb that into the expression prior to that

MA(0, n1)-MA(0, n2)
=(1/n1-1/n2)*[R(0)+2*R(-1)+...+n1*R(-n1+1)]
 -(1/n2)*[(n1+1-n2)*R(-n1)+(n1+2-n2)*R(-n1-1)+...+(-1)*R(-n2+2)]
=(1/n1-1/n2)*[R(0)+2*R(-1)+...+n1*R(-n1+1)]
 +(1/n2)*[(n2-n1-1)*R(-n1)+(n2-n1-2)*R(-n1-1)+...+R(-n2+2)]

We can see the coefficients of R's from t=-n2+2 to -n1 form the left side of an triangle with positive slope, and those from  t=-n1+1 to 0 form the right side of the triangle with negative slope. The plot (click to enlarge) below shows the coefficients as a function of time, with n2=10, n1=7, and current time as t=0. The right-most point is the weight for R(0): the return from t=-1 to 0.


Q.E.D. Now I hope you are ready to move on to a wavelet filter!

P.S. It is wonderful to be able to check the correctness of messy algebra like those above with a simple Matlab program!

===
New Service Announcement

Our firm QTS Capital Management has recently launched a FX Managed Accounts program. It uses one of the mean-reverting strategies we have been trading successfully in our fund for the last three years, and is still going strong despite the low volatility in the markets. The benefits of a managed account are that clients retain full ownership and control of their funds at all times, and they can decide what level of leverage they are comfortable with. Unlike certain offshore FX operators, QTS is a CPO/CTA regulated by the National Futures Association and the Commodity Futures Trading Commission.

===
Workshops Update

Readers may be interested in my next workshop series to be held in London, November 3-7. Please follow the link at the bottom of this page for information.

===
Follow me on Twitter: @chanep

145 comments:

  1. Hi Ernie,

    I wonder if moving average crossover strategy can still make profits in short-term time frame. I thought it is
    at least a mid-term or even long-term trend following strategy.

    Thanks.

    ReplyDelete
  2. That may well be the case: you can optimize n1 and n2 and find out what values work presently.

    Ernie

    ReplyDelete
  3. Hi Ernie,

    May I ask where we can get historical fundamental data for companies, such as
    EBITDA, P/E ratio?

    ReplyDelete
  4. Quandl.com has free historical fundamental data. Quality is not assured.

    Compustat sells them for a fee. Quality is industrial strength.

    Ernie

    ReplyDelete
  5. Hi Erine

    I would like to know how much is the fee that you charge on the FX managed Acct, How's the average yearly return of the past few years?
    Investor must open an account with IB?

    Thanks

    ReplyDelete
  6. Incentive fee of 20% subject to high watermark, annual management fee of 1%.

    The annual returns are listed in the YTD column of the Monthly % Returns table on epchan.com/accounts. Please note all returns are calculated net of fees.

    Ernie

    ReplyDelete
  7. IB account is not necessary, but would certainly simplify things.


    Ernie

    ReplyDelete
  8. Hi Ernie,

    What is high watermark?

    ReplyDelete
  9. High watermark means the highest cumulative return since inception up to the previous month's end. No incentive fee will be charged unless the cumulative return up to the current month's end is higher than the high watermark.

    Ernie

    ReplyDelete
  10. Hi Erine

    Which currency pairs that you FX fund is trading for this intraday mean revision? Any hard stoploss?

    What is the minimum funding required for subscription?

    Thx

    ReplyDelete
  11. This strategy trades G7 currencies plus the most liquid EM currency.

    There is a hard stoploss of 15%.

    Please email me to discuss minimum investment, as it depends on various factors.

    Ernie

    ReplyDelete
  12. Hi Ernie,

    To deal with TAQ data, shall we focus on primary exchange or all exchanges?

    What is the difference?

    ReplyDelete
  13. It depends on what order types you are using, which broker you are using, and where you/they are routing your orders. If you are sending MOO, LOO, MOC, or LOC, then you have to use primary exchange data only.

    Ernie

    ReplyDelete
  14. Hi Ernie,

    In IB, the round trip transaction cost per share for US stocks is $0.01.

    For US stock pairs trading in backtestiung, how much do we need to use for slippage besides tc?

    ReplyDelete
  15. Slippage depends on your order size, and the specific stock's bid-ask spread. In general, I assume 5bps all-in one-way tcost for the average SP500 stock.

    Ernie

    ReplyDelete
  16. Hi Ernie,

    My order size is small, less than 200 shares per trade, but I may trade small cap stocks. (300k daily volume)

    I use Bid/Ask bars to compute P&L, and we know commissions in IB per share is $0.005. Therefore, I only need to estimate slippage. How about $0.01 one way for slippage?
    Thanks.

    ReplyDelete
  17. The top of book size is typically just 100 shares. So yes, slippage is at least $0.01 per side, or $0.02 round trip, even if you use quotes for backtesting.

    Ernie

    ReplyDelete
  18. Hi Ernie,

    I've been live paper trading (using IB's simulated trading account) a mean reversion intra day FX strategy for some time now, with promising results. Could you comment on how realistic these results really are? Thanks.

    -T

    ReplyDelete
  19. Hi T,
    I think FX paper trading in IB is pretty realistic. On the other hand, paper trading in many FX ECN's won't be, due to an effect I explained in some details in my MFT course.
    Ernie

    ReplyDelete
  20. Hi Ernie,

    May I ask where we can get historical FX intraday bars?

    ReplyDelete
  21. Historical FX data is best obtained free from your own broker, since every broker has different prices.

    Otherwise ratedata.gaincapital.com has free data.

    Ernie

    ReplyDelete
  22. Hi Erine

    You trade mean revision of the spread of 2 different currency pairs or just trading the mean revision of an outright position of a single currency pair?

    Likes GBPUSD, it has fallen steeply in recent days under the worry of the potential implications of Scottish independence. In your opinion, besides putting a hard stoploss in the trading program, what other algorithms that can help when this kind of unexpected breakdown occurs? Thx

    ReplyDelete
  23. We trade spreads of currency pairs.

    Generally speaking, it is not a good idea to enter into a mean-reverting trade when we anticipate significant news that may affect a country's currency.

    Ernie

    ReplyDelete
  24. Hi Ernie,

    For US stocks, small order size (less than 200 shares), is it realistic to set slippage $0.02 per share per side, or $0.04 round trip?

    Besides slippage, I already use quotes to compute P&L, and $0.01 per share for round trip commissions.
    So, tc = spread + slippage + commissions.

    My execution is fully automated via IB TWS.

    ReplyDelete
  25. For 200 shares, a slippage of $0.01 per trade is sufficient.

    Ernie

    ReplyDelete
  26. Hi Ernie,

    According to your book, when we trade FX, we need to close positions before 17:00 ET to avoid rollover interests?

    Is this lose or gain significant?

    Thanks.

    ReplyDelete
  27. The loss or gain is not significant if the differential interest rate is small.

    Ernie

    ReplyDelete
  28. Hi Ernie,

    What's a typical allocation of equity in a startup hedge fund? Suppose it's a discretionary traded quant driven fund with revenues split threeways between a portfolio manager, a seed capital provider and a quant researcher/trader.

    Does 60/35/5% seem too unfair?

    I appreciate any input I can get!
    /the quant

    ReplyDelete
  29. Hi quant,
    Whether 5% is fair or not depends on how much of the revenue comes from your contribution. For e.g. if they are trading a model that you created, you could argue that you should deserve more share.
    Ernie

    ReplyDelete
  30. Hi Ernie,

    May I ask how to do divergence stock pairs trading ( the opposite of mean reverting) ?

    It seems to be easy to set entry, but how do we exit the position? Do we have to do stoploss and profit taking?

    Thanks

    ReplyDelete
  31. Divergence stock pairs trading is a momentum strategy, often triggered by a fundamental event on one or both stocks. E.g. M&A event. As with most momentum strategies, stoploss is recommended, but profit taking is not. Exit is usually triggered by the completion of the event (e.g. closing of M&A deal), or a fixed holding period.

    Ernie

    ReplyDelete
  32. Hi Ernie,

    I have a question about example3_6.m in your first book.

    The way I read the hedgeRatio is you calculate it based on the entire trainset data. Then, this hedgeRatio value is used to backtest that trainset data. Isn't this a lookahead bias?
    I meant it seems it is using hedgeRatio value that is obtained from the entire data that we are backtesting from older data to newer ones. In real trading we can't determine the hedgeRatio using future data, since it is the future and not yet available.

    Or, am I being confused here ? :)

    ReplyDelete
  33. Hi Sonny,
    You are right. As I wrote in my introduction to my second book, many of my examples has this type of subtle look-ahead bias. The reason is for simplicity, and an implicit exercise for readers to figure out these subtleties, as you just did.

    Ernie

    ReplyDelete
  34. Hi Ernie,

    How do we calculate Sortino ratio?

    How do we set target return for computing downside deviation?

    Assume we use data of 1 hour returns , and daily returns.


    ReplyDelete
  35. You can calculate Sortino ratio the same way as Sharpe ratio, except that you should include only negative deviations from the mean in computing the standard deviation of returns in the denominator.

    Ernie

    ReplyDelete
  36. Thanks Ernie!
    After trying to understand your code for that example 3.6, my understanding is...

    - You are using hedgeRatio calculated from the trainingset and use it for testset. This is why you seperate the data to training and test set, because I thought there are no difference in both sets, but actually the test set is tested using hedgeratio, spreadmean, and spreadStd from trainingset.

    - In real world trading, how do we adjust the hedgeratio, spreadmean and spreadstd? Every day we calculate the new numbers using updated daily data from yesterday,... or ?

    - Here the exit is only determined if the zscore comes back below abs(1). Therefore, the trades are guaranteed winner even if we have to wait for a long time for zscore to drop below abs(1). Is this correct analysis?
    In real trading, do you apply stop loss if zscore climbs above abs(3), for example, or any other methods you use?

    Thanks for all the kind helps. You are very generous for retail people.

    ReplyDelete
  37. Hi Sonny,
    True: whenever I separate training and testing data and used the parameters obtained solely from the training set, there is no look-ahead bias in the test set.

    In live trading, you can certainly using a moving training set every day to retrain your data.

    Also in live trading, we use Bollinger bands to determine exits, hence the average is a moving average, and exits do not guarantee profits. There is an implicit stoploss embedded in Bollinger bands.

    Ernie

    ReplyDelete
  38. Hi Ernie,
    I am reading your second book right now. It is a very interesting and inspiring book.

    I am having some difficulties in reproducing your results in certain examples of the book. Since your book's website only provides data files in Matlab format, which I could not read or use. In your book the data for
    EWC during the periods (04/04/2006-04/09/2012) had some highs very close to 35$, but the data for EWC which I downloaded from Yahoo had only reached $32 or so. I was wondering if there is any possibility that you could provide data EWC-EWA pair in the above periods in ASCI format so that I could use, I would greatly appreciate.

    Thanks

    Pete

    ReplyDelete
  39. Hi Ernie,

    If the order size is not big, what is the slippage to trade G7 and commodities currencies?

    Thanks.

    ReplyDelete
  40. Hi Pete,
    The difference in data is due to the dividend adjustments to the prices after my book was published. You should get very similar results for the backtests even though the prices are different.
    Ernie

    ReplyDelete
  41. The slippage in trading currencies if your order size is small is due solely to the latency in data feed and order submission, and is entirely specific to your network and broker's system. Generally, we can assume the slippage is zero and just take into account bid-ask spread and commissions in such backtests.

    Ernie

    ReplyDelete
  42. Hi Ernie,

    Would you please explain what happen to stocks prices in M&A event?

    ReplyDelete
  43. In an M&A event, it was found that the acquirer stock price will typically go up, while the acquiree's price will go down. (This was research done by Ravenpack, contrary to popular beliefs.)
    This movement will continue until closing of the transaction.

    Ernie

    ReplyDelete
  44. Hi Ernie,

    Could we know when M&A starts and when it ends?
    Do they do disclosure as public information?

    Btw, do you trade parameters with highest Sortino ratio?

    ReplyDelete
  45. The announcement of an M&A is a closely guarded secret, but you can enter a trade after the announcement as a momentum trade. The expected closing of a M&A is announced, but there is no guarantee that it will proceed as announced.

    I typically approximately optimize our parameters based on Sharpe ratio.

    Ernie

    ReplyDelete
  46. Thanks, Ernie.
    The cointegration test results are pretty similar. But I could not reproduce the KF test results as your book showed. I could not understand the jump (~100%) in the cumret plot.

    Thanks again.
    Pete

    ReplyDelete
  47. Hi Pete,
    Please email me if you wish to compare results in more details.
    Ernie

    ReplyDelete
  48. Hi Ernie,

    It seems using Kelly formula is to maximize Sharpe ratio.
    May I ask if you know any similar research or formula which is used to maximize Sortino ratio?

    Thanks.

    ReplyDelete
  49. Kelly formula is not used for maximizing Sharpe ratio. It is used for optimizing the leverage in order to maximize the long term compounded growth. No matter what leverage one uses, the Sharpe ratio of a strategy will be the same. The only way to increase the Sharpe ratio is to change your strategy.

    Ernie

    ReplyDelete
  50. Hi Ernie,

    When we do parameters optimization,
    usually how long is the historical data we use for Daily bars, 1 hour bars and 1 min bars?

    Thanks

    ReplyDelete
  51. The lookback used for parameter optimization is itself a parameter to be optimized!

    To avoid going in circles, you can choose to optimize the lookback in a fixed in-sample period.

    Ernie

    ReplyDelete
  52. Hi Ernie,

    What do you mean, "to avoid going in circles"?

    ReplyDelete
  53. If you consider the lookback for optimizing parameters as a parameter to be optimized, then one has to consider what lookback to optimize this lookback parameter. Thus the circle.

    Ernie

    ReplyDelete
  54. Hi Ernie,

    For Kalman filter, as you said before,
    there is no distinction between in sample and out sample data, so there is no lookback window to choose.

    Therefore, we just feed KF with all data to optimize its parameter(which control speed of beta change)?

    Thanks.

    ReplyDelete
  55. For KF, the effective lookback is always the beginning of the data to the last bar.

    Ernie

    ReplyDelete
  56. Hi Ernie,

    To some extent, I find Sharpe ratio and Sortino ratio are negative correlated.

    High Sharpe ratio makes you trade more (smooth equity curve) while high Sortino ratio makes you trade less (more positive jumps on less smooth equity curve). Calmer ratio is more correlated to Sortino ratio.

    How do you choose parameters between different ratios?

    ReplyDelete
  57. Since I am increasingly cognizant of abnormal risks, I tend to optimize Calmer ratio of late. But there is a problem with Calmer ratio: over what period of max drawdown should we use? Evidently, the longer the period, the bigger the max drawdown. So any period is an arbitrary number. Sharpe ratio has much more sensible mathematical properties that do not depend on lookback.

    Ernie

    ReplyDelete
  58. Hi Ernie,

    I have a quick follow-up question from your reply above
    "Also in live trading, we use Bollinger bands to determine exits, hence the average is a moving average, and exits do not guarantee profits. There is an implicit stoploss embedded in Bollinger bands.
    "

    About the moving average, what is usual length or is it dynamically changed?

    From the backtest code pnl, I assume that even if the pairs converge from std 2 into std 1, that pair still can lose money?

    last question....
    Using cadf conintegration test, is it valididea if I test 1 year of data, 6 months of pair data, and 3 months of pair data and use the 3 months cointegration result as the most weight to trade the pair ?
    any suggestion?

    thanks

    ReplyDelete
  59. Hi Sonny,
    The lookback length is a parameter to be optimized in-sample.

    Mean reversion to a Bollinger band does not guarantee profit.

    We cannot determine cointegration using 3 months. 1 year is minimum, and 3 years is optimal.

    Ernie

    ReplyDelete
  60. Hi Ernie,

    You imply that wavelet filter is a better choice. Could you please suggest several useful mother wavelet functions?

    Thanks,

    ReplyDelete
  61. Hi Chengxi,
    I don't know for a fact that wavelet filter will be better - I merely suggested it is worth trying.

    I did try it many many years ago, but it didn't produce great results. But that maybe because I was naive and foolish back then.

    Ernie

    ReplyDelete
  62. Hi Ernie,

    Usually, how many years of data are used to backtesting?

    ReplyDelete
  63. One should always start from at least 2007 in a backtest in order to see how model performs under stress.

    Ernie

    ReplyDelete
  64. Hi Ernie,

    As you say, from 2007, could we use "walk forward" to backtesting?

    For example, 2 years in sample (optimization parameters), 1 year out sample(trading), then all move forward 1 year, repeat until today.

    Collect all statistics from all 1 year out sample.

    ReplyDelete
  65. You can use walk-forward, but the out-of-sample period must include 2007 and onwards.

    Ernie

    ReplyDelete
  66. Hi Ernie,

    May I ask what are the momentum strategies for US stocks market?

    ReplyDelete
  67. Please read my book Algorithmic Trading, chapter 6 and 7 on momentum strategies, mostly for US markets.

    Ernie

    ReplyDelete
  68. Hi Ernie,

    Is it possible to do intraday trading using market depth?

    Is there any books or papers we could read about it?

    ReplyDelete
  69. Sure, please google Hellstrom and Simonsen, 2006.

    Ernie

    ReplyDelete
  70. Hi Ernie,

    Thank you for the information.

    It seems they report their results at the level of 1 min, 2 mins etc..

    If we buy an extra fast real-time data feed, maybe we can trade market depth on Interactive Brokers? Have you traded market depth on any markets (forex, stocks, futures) on IB? Or with other brokers.

    Many thanks.

    ReplyDelete
  71. You should be able to improve results with a fast data feed. But with IB's order handling latency at >= 10ms, it is not the ideal broker for such HF strategies.

    Ernie

    ReplyDelete
  72. Hi Ernie,

    How do we choose "in sample period" for stock pairs trading? 1 year, 3 years or .... where we can optimize lookback period.

    ReplyDelete
  73. We typically use half of all the data for in-sample training.

    Ernie

    ReplyDelete
  74. Hi Ernie,

    For intraday strategies, I could get higher Sharpe ratio using the last year for in-sample training while I get lower Sharpe ratio using the last 3 years.

    Which parameter set shall I use for real trading?

    Thanks.

    ReplyDelete
  75. I suggest you can use 1 year rolling in-sample data for optimization.

    Ernie

    ReplyDelete
  76. Hi Ernie,

    So it is like walk forward optimization?

    Maybe I could use 1 year rolling in-sample data, and half-year ( or 1 year) out of sample.
    So I need to do re-optimization every half-year (1-year) in real trading.

    Thanks.


    ReplyDelete
  77. I don't know how you can use out-of-sample data for optimization. Out-of-sample is your live trading record.

    Ernie

    ReplyDelete
  78. Hi Ernie,

    It seems that no parameter set could last forever for a stock pairs.

    May I ask when we need to re-optimize to get a new parameter set or even stop trading the stock pairs?

    Thanks.

    ReplyDelete
  79. One should discover a fundamental or economic reason why a stock pair stops working. That way, one may be able to introduce a new variable that will repair the cointegration relationship. Re-optimizing the parameters is not the best solution.
    See p. 91 of my book Algorithmic Trading.

    Ernie

    ReplyDelete
  80. Hi Ernie,

    Thank you for quick response.

    It seems to be easier to analyse fundamental or economic reasons between ETFs as you mention in your book.

    Recently, I focus on intraday stock pairs trading. I just run program to search a whole sector (such as energy sector) to get "good" stock pairs based on Sharpe ratio among thousands of combinations.

    It seems it is intraday trading, and there are hundreds of stocks in a sector, and I am not very good at industry analysis.

    I wonder how we could trade intraday stock pairs properly. Or we need to trade interday stock pairs and ETFs.

    Thanks.

    ReplyDelete
  81. It is for the exact reason you mentioned that I don't trade stock pairs!

    Ernie

    ReplyDelete
  82. Hi Ernie,

    Usually, what is the ratio between in-sample and out of sample data length?

    ReplyDelete
  83. Hi Ernie,

    It seems there are 2 approaches to systems parameters:

    1. Decide on a set of parameters for the strategy and stick to it.

    2. Keep changing the system parameters based on the latest market conditions (chameleon approach).

    Which one do you prefer? And why?

    ReplyDelete
  84. You can indeed continuously adapt your parameters based on a rolling lookback period, but the lookback period should be long (> 1 year), since many parameters do mean-revert. Using such a long lookback means that the parameters typically do not change much. So in practice, I just keep them fixed.

    Ernie

    ReplyDelete
  85. Hi Ernie,

    Have you heard about walk forward test by Robert Pardo?

    What is the pros and cons of this method?

    Thanks.

    ReplyDelete
  86. If I understand correctly, Robert Pardo's walk forward analysis is just backtesting with parameters optimized in a rolling lookback period.

    This is certainly reasonable, but does not eliminate certain data snooping biases that are not due to parameters, but to strategy logic. For example, if you find that your strategy doesn't work well in a walk forward test, won't you change a few rules and run the test again? Aha! There you have introduced data snooping bias.

    Ernie

    ReplyDelete
  87. Hi Ernie,

    Thanks for response.

    Also, the walk forward window is a parameter itself. The rule of thumb is to choose 25% of optimization data set. Ideally, we need to know the "speed" of the market we trade.

    Moreover, the other con is that it is computation intense, and it takes extra effort to build a walk forward trading system.

    ReplyDelete
  88. Hi Ernie,

    Usually, different instruments have different margin requirement, which means we could get different leverage. For example, if we trade forex (40x), futures(30x), and intraday stocks (4x), how could we allocate our buying powers to different types of assets via Kelly formula?

    It seems the buying powers is not fixed.

    Many thanks.



    ReplyDelete
  89. Kelly formula tells you what leverage you should deploy for each instrument. It is independent of that instrument's margin requirement. Of course, your broker will impose different constraints on margin (or equivalently, maximum leverage) on different instruments, so you should scale down the overall leverage of your portfolio to meet those constraints. Please see page 98 of my book Quantitative Trading which discussed this precise issue in some depth.

    Ernie

    ReplyDelete
  90. Hi Ernie,

    Are there slippage for trading US ETFs when we just send market orders?

    It seems ETFs have bigger bid/ask size than US stocks.

    ReplyDelete
  91. No, you won't incur slippage trading ETF with MKT orders, only bid-ask spread, unless your order size is huge.

    Ernie

    ReplyDelete
  92. Hi Ernie,

    In your book, one of the basic strategies you discuss involves adjusting changing the hedge ratios daily. Ignoring transaction costs, should we expect the cash flows from these hedge ratio changes to be significant/material? How do we account for them when looking at the profit/loss from mean reverting strategies?

    Thanks,
    flapjackery

    ReplyDelete
  93. Hi flapjackery,
    There shouldn't be significant cash flow from adjusting the hedge ratios and rebalancing. But in any case, I would have accounted for any such cash flow in the P&L the way I computed it.
    Ernie

    ReplyDelete
  94. Hi Ernie,

    For pairs trading, usually, how often do we need to evaluate pairs traded live and think about replacing bad performers of live portfolio by good performers of paper portfolio?

    Thanks.

    ReplyDelete
  95. It depends on the max drawdown duration in your backtest. Any drawdown longer than that requires a re-examination.

    Another measure is the half-life of mean-reversion. A drawdown that is an order of magnitude larger than halflife should trigger alarm bells.

    Ernie

    ReplyDelete
  96. Hi Ernie,

    Thank you for response.

    "Another measure is the half-life of mean-reversion. A drawdown that is an order of magnitude larger than halflife should trigger alarm bells."

    I do not understand the paragraph above.

    For example, half-life is 5 days.
    What do you mean?

    ReplyDelete
  97. You should be concerned if drawdown is 50 days.
    Ernie

    ReplyDelete
  98. Hi Ernie,

    Could we use stocks "prices" to compute correlations?

    ReplyDelete
  99. No, you have to turn stock prices into returns in order to compute correlations.

    Ernie

    ReplyDelete
  100. Hi
    If anyone knows...
    How can I get primary exchange EOD data (NYSE)?

    ReplyDelete
  101. My strategy uses open and close prices, MOO and MOC orders.
    Should I simulate bid-ask spread and slippage?

    ReplyDelete
  102. MOO and MOC orders do not suffer slippage nor bid-ask spread. They get the auction price.

    Ernie

    ReplyDelete
  103. Primary exchange EOD data can be obtained from Bloomberg. Otherwise you have to buy tick data and find the trade tick with the largest volume at the close.
    Ernie

    ReplyDelete
  104. Hi Ernie,

    Could we send MOO and MOC in Interactive Brokers via API?

    ReplyDelete
  105. Hi Ernie,

    Some people trade cointegrated stocks pairs with high correlations.

    Does that make sense?

    ReplyDelete
  106. Hi Ernie,

    Thank you for response.

    I guess that the advantage is that
    even when cointegration is broken, if the correlation still holds, then the loss is limited.

    ReplyDelete
  107. I have a question regarding IB paper trading. Market On Open orders are not executed some times (I get message: "At the open order could not be executed") and finally orders are canceled. Does anyone know what might be causing this?

    ReplyDelete
  108. It is just a technical issue that
    IB does not support MOO orders in paper trading. There is no deep explanation for it: after all this is just paper trading.
    Ernie

    ReplyDelete
  109. Hi Ernie,

    Any idea why historic closing prices often differ between IB and Google/Yahoo?

    Thanks

    ReplyDelete
  110. What Yahoo, Google or IB report as closing price is just the last price captured on any one of the dozens of exchanges in the US. These prices tend to generate erroneous backtest performance, since we cannot be assured that we can execute at that price historically.

    The only closing price that matters is the "official" closing price from the primary exchange, obtainable if you subscribe to Bloomberg, or if you have access to high frequency (1ms) historical data.

    Ernie

    ReplyDelete
  111. Thanks for your reply Ernie.

    Another thing:
    Do you find the metric Max Drawdown/Volatility useful? Max Drawdown scales with vol (not linearly though) which means one need to standardize drawdowns somehow.

    Thanks

    ReplyDelete
  112. The usefulness of maximum drawdown is to see what extreme volatility can do to your equity curve. So I won't advise dividing it by volatility.

    Ernie

    ReplyDelete
  113. However when comparing different strategies with different volatilities one need to standardize drawdowns somehow.

    ReplyDelete
  114. Hi,

    I just want to share I had a similar idea before on the crossover and I eventually ended up optimizing filter weights of Finite Impulse Filters (with N-weighted sum of N historical prices). Long/short decision is triggered when comparing the current price against the weighted sum. The weights are "calibrated" using raw historical closing prices with the objective function being optimizing sharp ratio. A sharp ratio of 5+ was easily achievable with couple thousand percent annual return even with harsh simulated commissions. Then I extended the idea to a multi-dimension model where I added more market info. Thus even more parameters to be calibrated. It was a 60taps by 8market info by 1000stocks “matrix”. And it resulted even higher sharp ratio, ONLY when looking at calibration results. :P

    The calibrated model performance is same as a coin flipping decision process in backtesting. It was my first time experiencing over fitting first hand out side of class and got fooled by randomness. It was fun tho. I Just want to FYI of what i tried if anyone is thinking about this route and hope this is helpful.

    This post is very refreshing. Will dig up my decade old codes and try the method in this post. Nice post. Thanks.

    Irvin

    ReplyDelete
  115. The standard way to compare max drawdowns for strategies with different volatilities is to use Calmar ratio (google it).

    Ernie

    ReplyDelete
  116. Hi Irvin,
    Thanks for sharing your idea!

    If I understand you correctly, your filter is actually something like a feed forward neural network?

    When you mentioned overfitting, are you suggesting that the live trading results (or live walk-forward testing results) are much worse than the +5 Sharpe ratio that you found in-sample?

    Ernie

    ReplyDelete
  117. Hi Ernie,

    I suppose we can call it a NN, it is more of a single layer one with each "neuron/synapses" a impulse function. It is a feed-forward filter.

    In sample performance was unrealistically good. Out of sample performance did not generate statistically significant return even without considering commission. So I didn't bother to try it in a live account.

    I've also tried to use daily index data as far back as 30yrs to expand the sample size for calibration. However, it also failed to work as soon as it touches the out of sample data.

    Irvin


    ReplyDelete
  118. Why don't people give up on these neural networks. They just don't work out of sample or are highly unstable. Simplicity is a good thing. Occam's razor always applies.

    ReplyDelete
  119. Hi Erine

    What timeframe you prefer to use for calculating the spread for trading Mean Revision of FX usually?

    ReplyDelete
  120. Hi Ernie,

    Been following your blog for quite a while now and it has been very informative. Thanks for sharing all this great info with us.

    I have recently finished writing software for automated intraday trading & analysis in Matlab using IB (with the standard 100 contracts real time data). Now I am trying to hone in on the most effective strategies.

    You discuss the continuing effectiveness of mean reversion, despite what you say is "low volatility in the markets." My question is, if mean-reversion has a higher return with increased volatility, why not use extremely volatile leveraged etfs like junior gold miners or some such thing? Is it because of drawdown or black swan issues?

    On a related note, can you give any generally effective ways of choosing stocks that work for mean reversion, or at least things to avoid?

    For example, can historically high amounts of "reversion to the mean" be an indicator? A high standard dev but a small kurtosis? I see a lot of information about mean reversion as a trading concept, but little in the way of choosing the most effective time scales and contracts for performing it. Thanks for your help!

    Tyler

    ReplyDelete
  121. Hi Tyler,
    Sure, if you find mean reversion in leveraged ETF, you can certainly trade them. (I haven't).

    There is strong cross-sectional mean reversion of stocks intraday. Please see the example strategy in the chapter on stock mean reversion in my book Algorithmic Trading.

    To discover cross-sectional mean reversion, you don't need any indicator. Just backtest a simple long-short strategy will do.
    Ernie

    ReplyDelete
  122. Hi Ernie,

    How Premiums and Discounts of ETF affect strategies? Is Premiums and Discounts arbitrage possible between two ETF of the same asset/index?

    -HK

    ReplyDelete
  123. Hi HK,
    There are unlikely to be much premium/discount in ETFs left for retail investors to arbitrage. Fund managers have Authorized Participants (typically big banks) that specialize in minimizing these premium/discount intraday to by buying and selling basket of stocks.
    Ernie

    ReplyDelete
  124. Hi Ernie,

    Could we assume that there is no individual default risk for intraday stocks trading?

    Thanks.

    ReplyDelete
  125. Even though stocks typically default after market close, there is nothing to prevent bad news from leaking out during market hours and cause a stock to plunge 99%.

    Ernie

    ReplyDelete
  126. Hi Ernie,

    Do you have any comment about post–earnings-announcement drift, or PEAD ?

    Thanks.

    ReplyDelete
  127. PEAD continues to work, provided some bells and whistles are added.

    Ernie

    ReplyDelete
  128. Hi Ernie,

    Thank you for response.

    Shall we avoid stocks pairs trading after earnings announcement?

    ReplyDelete
  129. Yes, avoiding mean reversion trades immediately after any corporate event is a good idea.

    Ernie

    ReplyDelete
  130. Hello Ernie, with regard to currency pairs like EUR/MXN, CHF/NOK, bid/ask + slippage costs are probably more significant than for EUR/USD. My question is, is there an estimate available such as 3 basis points in each direction - I understand this may depend on the time of day, but I was looking for an estimate.

    Thanks.

    ReplyDelete
  131. The bid-ask spread for any FX pair is highly dependent on the broker you use. In fact, some brokers do not charge commission, and earn all their profits from the spreads. So you have to go to your broker and observe the actual spread.

    Ernie

    ReplyDelete
  132. Thanks for the answer. But what about slippage? The broker is IB, but its just that I have limited myself to EUR/USD, EUR/CAD etc. and was concerned about the non-majors - the spread itself seems quite low after adjusting it for the value of the underlying currency.. Also the planned number of round trips can go > 300 in a year, so this can get tricky.. I am going to paper trade with them but that will take some time to tell how strong the strategy will be :-)

    ReplyDelete
  133. By slippage, I think you mean the difference in the price that triggered your order, and the execution price. This slippage of course depends on how long it takes for your order to travel from your computer to IB's server. It could be anywhere from 0ms to 100ms. And thus the slippage could be 0 to 2 pips.

    Ernie

    ReplyDelete
  134. Hi Ernie,

    What happen to ATHL in US stocks market today?

    There are no quotes.

    ReplyDelete
  135. Hi Ernie,

    always good to read your articles and comments about others' strategies.

    But in regards to your FX managed account, dont you think 30+% is quite much?

    ReplyDelete
  136. What is this 30% figure you are referring to?

    Ernie

    ReplyDelete
  137. Hi Ernie and congratulations for this post, it s very interesting. I don't know what is a wavelet filter, there is some paper or book which explain it? thank you

    Ciro

    ReplyDelete
  138. Hi Ciro,
    Thanks for your kind words!

    There is no better intro than Wikipedia:
    https://en.wikipedia.org/wiki/Wavelet

    Matlab also has a wavelet toolbox with good documentation:
    https://www.mathworks.com/help/wavelet/index.html

    Ernie

    ReplyDelete
  139. Thank you Ernie,

    so if i understood well, this filter could be suitable for a more statistical cyclical analysis, could be?

    ReplyDelete
  140. Hi Ciro,
    Yes, wavelets transform is similar to Fourier transform. Both can be used to filter out "noise" of various frequencies.
    Ernie

    ReplyDelete