Everybody know that volatility depends
on the measurement frequency: the standard deviation of 5-minute returns is
different from that of daily returns. To be precise, if z is the log price,
then volatility, sampled at intervals of τ, is

volatility(τ)=√(Var(z(t)-z(t-τ)))

where Var means taking the variance over many sample times. If the prices really follow a geometric random walk, then Var(τ)≡Var((z(t)-z(t-τ)) ∝ τ, and the volatility simply scales with the square root of the sampling interval. This is why if we measure daily returns, we need to multiply the daily volatility by √252 to obtain the annualized volatility.

Traders also know that prices do not really follow a geometric random walk. If prices are mean reverting, we will find that they do not wander away from their initial value as fast as a random walk. If prices are trending, they wander away

*faster*. In general, we can write

Var(τ) ∝ τ^(2H)

where H is called the "Hurst exponent", and it is equal to 0.5 for a true geometric random walk, but will be less than 0.5 for mean reverting prices, and greater than 0.5 for trending prices.

If we annualize the volatility of a mean-reverting price series, it will end up having a lower annualized volatility than that of a geometric random walk, even if both have exactly the same volatility measured at, say, 5-min bars. The opposite is true for a trending price series. For example, if we try this on AUDCAD, an obviously mean-reverting time series, we will get H=0.43.

All of the above are well-known to many traders, and are in fact discussed in my book. But what is more interesting is that the Hurst exponent itself can change at some time scale, and this change sometimes signals a shift from a mean reversion to a momentum regime, or vice versa. To see this, let's plot volatility (or more conveniently, variance) as a function of τ. This is often called the term structure of (realized) volatility.

Start with the familiar SPY. we can compute the intraday returns using midprices from 1 minutes to 2^10 minutes (~17 hrs), and plot the log(Var(τ)) against log(τ). The fit, shown below, is excellent. (Click figure to enlarge). The slope, divided by 2, is the Hurst exponent, which turns out to be 0.494±0.003, which is very slightly mean-reverting.

But if we do the same for daily returns of
SPY, for intervals of 1 day up to 2^8 (=256) days, we find that H is
now 0.469±0.007, which is

*significantly*mean reverting.
Conclusion: mean reversion
strategies on SPY should work better interday than intraday.

We can do the same analysis for USO (the WTI crude oil futures ETF). The intraday H is 0.515±0.001, indicating significant trending behavior. The daily H is 0.56±0.02, even more significantly trending. So momentum strategies should work for crude oil futures at any reasonable time scales.

Let's
turn now to GLD, the gold ETF. Intraday H=0.505±0.002, which is slightly
trending. But daily H=0.469±0.007: significantly mean reverting! Momentum
strategies on gold may work intraday, but mean reversion strategies certainly
work better over multiple days. Where does the transition occur? We can examine
the term structure closely:

We can see that at around 16-32 days, the volatilities depart from
straight line extrapolated from intraday frequencies. That's where we should switch from momentum to mean reversion strategies.

One side note of interest: when we compute
the variance of returns over periods that straddle two trading days and plot them as function of log(τ), should τ
include the hours when the market was closed? It turns out that the answer is
yes, but not completely. In order to produce the chart above where the
daily variances initially fall on the same straight line as the intraday
variances, we have to count 1 trading day as equivalent to 10 trading hours.
Not 6.5 (for the US equities/ETF markets), and not 24. The precise number of equivalent trading hours, of course, varies across different instruments.

===

**Industry Update**

- Nick over at mintegration.eu discusses the new intraday databases at Quandl and Kerf.
- Factorwave.com
(Euan Sinclair's creation) started a new forum: slack.factorwave.com.
It has some very active and in-depth discussions of many trading and
investing topics.
- Prof. Matthew
Lyle at Kellogg School of Management has a new paper out that relates fundamentals to variance risk
premiums: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2696183.

**Upcoming Workshops**

- April 20-21: Mean Reversion Strategies.

There are a lot more to mean reversion strategies than just pairs
trading. Find out how to thrive in the current low volatility environment favorable to
this type of strategies.

## 95 comments:

How did you calculate the 10 hours = 1 day, simply by fitting the 1 day point onto your linear fit, solving for it to put it on the line? Isn't there a risk one will wash out significant overnight trending/mean-reversion that way? And, I suppose more generally, are there any references for how to deal with overnight gaps when trying to do trading on the 3-5 day timeframe using intraday data?

Hi one_fell_swoop,

You are right that 10 hours= 1 day is determined by trying different numbers, and finding the one where intraday and daily variances coincide at the same T where they overlap.

Indeed, this does not capture seasonal mean reversion such as what may occur only overnight. This study is only concerned with "time-invariant" mean reversion, or "homogeneous" mean reversion.

The general way to deal with overnight gap is that you can to treat the overnight bar differently from the intraday bars, by multiplying its variances by some adjustment factor. This factor is determined by a procedure outlined above. (Instead of adjusting the equivalent time, you can adjust the equivalent variance.)

Ernie

Hi Ernie,

how long is the window length over which you calculate the variance?

You only state "variance over many sample times"... Or do you want to keep it a secret? :)

I figured, if the slope is near 1.0, varying the window length may push the slope above or below 1 randomly. How do you handle this? (But SPY seems always to be mean reverting on an interday basis?!)

BR!

Hi BR,

I computed the variances over 20130116-20160115.

Obviously, a different period might give a different answer. But within this period, you can see that I have included the standard errors (SE) in the slope. For many of the results (though not SPY intraday), a slope of 1 is not within 2x SE.

Ernie

Hello Ernie,

Interesting article. Is there a problem with the x-axis of the GLD chart? 16 and 32 days cannot have a base 2 log between 13 and 15, if I get this right of course.

Thanks, Michael.

The units of x-axis (t) is actually in minutes. Apologies for not making that explicit.

So at log2(t)=14, we have t=2*14/10/16 days=27 days. (The factor of 10 due to the equivalence of 10 trading hours and 1 trading day.)

Ernie

Thank, now it makes sense.

Actually, I meant t=2*14/10/60 days=27 days above!

ERnie

Hi Ernie!

Nice artical again, but I have one remark. Hurst exponent is different for brownian motions with different distributions. You can simulate it with Monte Carlo and test out.

Hi Danil,

Are you referring to fractional Brownian motion?

Ernie

I refer to the article of this guy http://mechanicalforex.com/2016/03/the-hurst-exponent-and-forex-trading-instruments.html

Hi there, Dr Chan,

Have you considered using this measurement of "momentum or MR indicator" as possible half life for cointegrated residues?

Or, have you/do you use the Hurst exponent in cointegrated pairs or residues in order to further filter them as "more" mean reversal if they show an H<.5?

Thanks

Hi Eduardo,

No, the transition time scale does not indicate how long it would take for a price series to mean-revert. For that, you have to use Ornstein-Uhlenbeck equation.

As we approach the transition time from the mean reverting region, we should expect the half life to approach infinity.

But yes, the lower the Hurst exponent, the shorter the half life of mean reversion.

Ernie

Hi Ernie,

No matter intraday or swing trades, is that pair or more than pair kind of mean reverse strategies would need much smaller chance to cut lose than momentum? Seem like even the best momentum traders would cut lose 4 out of 10 times which is mentally hard to cut lose that often.

-HK

Hi HK,

I do not recommend stop loss for mean reversion strategy in general, unless it is never expected to be triggered. See last chapter of my second book Algorithmic Trading.

Ernie

Thanks for the answer, Dr.!

I too use the OU equation for HL of Mr pairs trading.

What I tried to imply was that, maybe the transition time scale is an additional filter for the filtering of possible pairs in the half life part. Like, if a pair has a HL (OU Calculated) of 20 days, but the transition time scale shows that in approximately 10 days there is a shift from MR to Momentum, maybe one should not hold the MR trade in the pair/etf/etc for more than the latter, in this case, 10 days.

I may be wrong, just thinking out loud here

Thanks

Hi Eduardo,

That's a good point. Yes, you can use the transition time as an upper limit to your holding period.

Ernie

Hi there, Great blog you have there, really. I learned so much from your posts already so please juse keep up the good work! :)

Thanks, Gustavo!

Ernie

Great article! Very helpful information. I was trying to reproduce the SPY interday plot. I used log daily price of SPY. But seems I get a different result. For example, the first data point, 1 day lag, the log variance I got is -9.61, but your plot shows should be around -8.9. Am I missing anything? Thanks

Hi Ernie,

Have you tried Digital Signal Processing (DSP) trading strategies?

Are they profitable?

Thanks.

Also interested in hearing Ernie's opinion on this..

(the financial hacker blog / Zorro platform lead programmer's blog talks extensively about this subject, very interesting)

Best

Hi Dawei,

Thanks for your kind words.

Did you use consolidated close price for your daily plot? If so, they always exaggerate mean reversion, due to noise (see my article on Beware of Low Frequency Data, April 2015). I use midprice for my intraday plots.

Also, I don't know if you use the same data period as my 20130116-20160115 for SPY?

In any case, one data point isn't that important. What's important is the Hurst exponent, which averages out the noise.

Ernie

Hi Eduardo,

Many techniques in trading can be considered Digital Signal Processing. E.g. Kalman filter, wavelets, etc. Some of them are useful, others are not. So one needs to be specific about the technique.

Ernie

Hi Ernie,

Have you read books (Digital Signal Processing) written by John Ehlers?

http://www.mesasoftware.com/ehlers_books.htm

He mentioned Sinewave indicator in his books.

Thanks.

No, I haven't read John Ehlers.

Do these indicators work for you?

Ernie

Hi Ernie,

His books provide codes for Tradestation and Multicharts.

He also sells his codes on his website.

I just begin to read his books and test his codes.

Basically, he said his indicators can tell Trend mode and Cycle mode.

Sorry, I was referring specifically to high/low bypass filters and the very well documented FFT.

Here's the link from another blog http://www.financial-hacker.com/build-better-strategies-part-2-model-based-systems/

Hi Eduardo,

Thanks for the link.

No, I haven't tried hi/lo pass filters or FFT in trading. I have tried wavelets, but with no major gains.

Ernie

Dr. Chan

Thank you very much for your reply. I was using the same date period SPY data. Since I was trying to do the interday plot, so I used SPY daily price data. As you suggested, I changed to midprice. (I used 0.5*(High+Low)). But the hurst exponent I got is only about 0.387.

Hi Dawei,

Midprice does not mean the mid of high and low. It means the mid of bid and ask at the market close.

Ernie

Hi Ernie,

May I ask what is your intraday trends strategy of E-mini S&P 500 futures?

Thanks.

Please feel free to email me to discuss that.

Ernie

Hi Ernie,

I plotted the autorcorrelation of daily returns of USO and found a statistically significant peak at a lag of 1 day. However the value is negative indicating mean reversion. So I tested two simple strategies to confirm this daily MR behavior and the difference in equity curves confirms this small daily MR behavior. Not sure how to connect autocorrelation of returns with the Hurst parameter and your results.

Thanks,

Steve

Hi Steven,

Your result is indeed contrary to that from Hurst exponent.

However, as an additional test, I would suggest you test using the midprice of the bid-ask at market close, not the consolidated close.

As I wrote before on this blog, consolidated closes have a tendency to reveal false mean reversion that nobody can trade on.

Ernie

Hi Ernie,

Do you trade long/short equity strategies?

Thanks.

Yes, I do.

Ernie

Hi Ernie,

What kinds of factors do you use in Long/Short equity strategies?

Thanks.

Mainly lagged returns.

Ernie

Hi Ernie,

Why don't you use fundamental factors, such as ROE?

For lagged returns, do you use PCA?

Thanks.

We prefer short term trading, hence fundamental factors are not of much help.

We don't use PCA currently, but it is under active research.

Ernie

Hi Ernie,

May I ask how long is "short-term" trading in your definition and strategies?

Thanks.

Short term to us is intraday.

Ernie

Hi Ernie,

Do we usually include intercept when we run linear regression?

Thanks.

Yes, including intercept is usually recommended, unless you have a fundamental reason not to.

Ernie

Hi Ernie,

For stocks pairs trading, do we need to include intercept when we run rolling linear regression?

Thanks.

It depends on the exact pair. But generally I won't.

Ernie

Hi Ernie,

Many thanks for a great blog!

I'm looking for an FX broker to trade a G10 strategy. Do you think IB is the best broker for an account of around $100k? What other alternatives should I look at? Do you have a good set up to recommend for low latency?

Many thanks

Yes, IB is as good as any in terms of FX for accounts <= $100K.

For low latency connection, ask Sam at speedytradingservers.com.

Ernie

Just curious, what options would you look at for accounts >$100K?

If you are an Eligible Contract Participant, as defined by CFTC, then you can open any prime broker account, and access any FX ECN directly such as HotspotFX, LMAX, EBS, etc.

Ernie

Hi Ernie,

What is the reasonable transaction cost assumption (including commissions, spread, and slippage) for S&P 500 stocks?

IB is the broker. Thanks.

We typically assume 5bps one way transaction cost for SP500 stocks.

Ernie

Hi Ernie,

It seems intraday long-short mean reverting strategy does notwork for SP500 stocks recently.

Is that right? Thanks.

I agree it is hard to make long-short stock strategies work this year (see hedge fund reports also on this category).

Ernie

Hi Dr,

Barging in this subject about SP500 and plain vanilla LS..

Have you ever tried to pair/triplet trade them in a very short term? Not HFT sub ms, us, but minutes wise, instead of EOD.

I had some success in doing so in 2014 and 2015, although not in the US Market (didn't try it). Looking for really small distortions from the average mean, using real time bid and ask can get you in and out very fast, may me worthwhile if your costs are not high. I stopped doing so because as a retail trader my discounts were not big enough to make it worthwhile, gross profit was real, nevertheless.

Hi Eduardo,

No, we haven't tried day trading stocks pairs like you suggested. I agree there may still be opportunities there.

Ernie

Hi Ernie,

Do you know why it is hard to make long-short stock strategies work this year?

How do we deal with it? Thanks.

Typically, long-short strategies depend on volatility to earn returns. Volatility in the stock market has been very low in the last few months.

You can always run a short volatility strategy in this market condition.

Ernie

Hi Ernie,

Thank you for quick response!

May I ask what are short volatility strategies you would recommend?

Thanks.

See the VX strategy discussed around Figure 5.12 in my book Algorithmic Trading.

ERnie

Hi Ernie,

I find that intraday long-short mean reverting strategy does not work for SP500 stocks since May 2010.

Thanks.

Thanks for sharing your experience!

Ernie

You have amazing explanations of price actions. anyone who wants to learn forex trading should look at your stuff which has helped me tremendously with my trading.

wow good

Hi Ernie,

What is the reasonable assumption of transaction costs for Russell 2000 stocks?

Thanks.

10 bps one way.

Ernie

Hi Ernie,

Do you trade only intraday strategies?

Are the capacity of intraday strategies limited?

Thanks.

We trade mostly intraday strategies, because of their higher statistical significance and lower risk. Yes, they do have lower capacity, but then we don't have billions to manage at this point. We are, however, working on strategies with longer holding period and higher capacity, and will be able to launch soon.

Ernie

Hi Ernie,

Do you trade Russell 2000?

Thanks.

Not at the moment.

Ernie

Hi Ernie,

Thank you for quick response.

Is it because of ill-liquid for small cap stocks?

Thanks.

You are correct.

Ernie

Great article!

Hi Ernie,

If I understand it correctly from your second book p. 45, the Hurst Exponent can be between +1/-1. When 0.50 is random walk, it strikes be as bold to claim that H = 0.56 is STRONGLY trending as you do in the above article. I would have guessed that strongly trending would be H = 0.8 or thereabouts??

Chis.

Hi Chris,

Actually H is between 0 and 1. It isn't realistic to have negative H, because that would imply prices remain constant over the long term.

Whether a price series is trending or not depends on whether it is statistically significantly greater than 0.5. Some of the price series I noted in the article passed this significance test by a good margin. However, I failed to find the adjective "strongly" mentioned in my article above. Can you please point out the sentence?

Ernie

Sorry, between 0 and 1, my bad.

I was referring to the following you wrote: "We can do the same analysis for USO (the WTI crude oil futures ETF). The intraday H is 0.515±0.001, indicating significant trending behavior. The daily H is 0.56±0.02, even more significantly trending."

Chris

Hi Chris,

Yes, I use the word "significant" in a specific sense. It means that it is more than 2 standard deviations away from the mean.

Ernie

The mean of what exactly?

Chris,

The significance testing in this specific context is to see if the Hurst exponent for random data of the same size will have the same value as what we obtained. The conclusion is that if we assume Gaussian distribution of such values, the chance that this happens is less than 2.5%. Hence with better than 97.5% probability this is a trending price series.

See also p. 16 of my second book, section on Statistical Significance of Backtesting: Hypothesis Testing.

Ernie

Hi Ernie,

I was trying to replicate your results and noticed that because of you are taking logarithm from time based on 2, i.e. log2(Time), you should also take log2 from variance, i.e. log2(Var). Please correct me if I'm wrong. You didn't make this explicit, so I was straggling a bit. Thank you for the great topic.

Pavel

Hi Pavel,

The log2(t) on the x-axis is for display purposes only. In my actual linear regression, I have taken the natural log of both variance and timescale.

Hope this helps.

Ernie

Hi Ernie

I have a question regarding this formula. I am aiming to replicate your study:

volatility(τ)=√(Var(z(t)-z(t-τ)))

z = log price

τ = time interval?

t = what plugs into t ?

if i understand the process is below:

Take variance over many sample times, in this case... mid price of 1 minute to 2^10 (1024 minutes or 17 hours) for intraday

Once we do the above calculation, we need to multiply the daily volatility by √252? to obtain annualized volatility? or is that not needed?

for the plot:

plot the log(Var(τ)) against log(τ)

is this a rolling window across a start date to end date in your example:

2013-01-06 to 2016-01-15

for τ

would i take the width of the window, lets say for intraday.. 1 minute to 2^10. And i slide that width of window up to the end date. So a rolling variance you could say?

Or do i just take to and from and work out plot the log(Var(τ)) against log(τ)?

If that is the case, then how would i plug in the time periods into τ??

It would be log variance of mid price from 1 minute to 2^10 2013-01-06 to 2016-01-15 to enter in log(Var(τ))... then against log(τ).. what would I enter in here?

Let me know if you can perhaps make it in simple terms for someone like me :)

Thanks a lot!

Andrew

Hi Andrew,

No, you do not annualize the volatility in this study. The whole point of the exercise is that we should not assume a Gaussian diffusion process for the log prices. I.e. Hurst exponent is not necessarily 0.5.

Yes, the window for computation is entire data set.

For any given time t taken from the data set, the time bar for the computation of log return is tau. I don't want to call it a "window", since it is just a bar (1 minute? 1 hour?) So it is just log(price(t+tau))-log(price(t)). You will have as many data points as the number of t in your data set. You will compute the variance of these data points (log returns). For different tau, you get different variances. These different variances vs tau form the plot.

Ernie

Ok so for each day I would , lets say mid prices of 1 minutes to 2^10 minutes (1024 minutes)

Lets say start date of: 2013-01-06

1. Compute log returns from 1 minute to 1024 minutes ( so 1 minute bar for each incremental step to 1024 minutes)

2. Compute the variance of those returns

3. Subtract the variance of those returns - the log returns?

How does one form the plot especially over date range: 2013-01-06 to 2016-01-15

We are plotting the variance of log returns against the log returns right?

Just struggling with how to structure it and how 1 minute to 1024 minutes relate over the 'n' range period.

OR

for the first 1 minute bar to the last 1024 1 minute bar...

we do the log returns and the variance of the log returns...

From date range: 2013-01-06 to 2016-01-15

log(Var(1min bar)) - log(1min bar)

We do that for every bar... first 1 minute bar up to the last 1024 minute bar....

and the result of that... is what we plot?

I can get this, just need a lil more 'dumbing down' again!

I do see the value in it, if i could view the markets nature in this way, it means could 'fit' a model suited to exploit the characteristics of the that market. At least that's the initial thinking

Andrew,

Your first scheme is closer to the way.

You don't just compute the returns on one day. You should, for e.g., compute the 1-min log returns on all days, and then compute the variance of them. There is also no need for 3): no need to subtract anything from the variance.

You have now a set of tau (1-min, 1-hour, 1-day, etc.), and a corresponding set of variances. Plot the log of those variances against the log of tau.

Ernie

Hi Ernie,

Great blog - I'm a fan of your books as well. I was wondering for some clarification on this part of your post:

If we annualize the volatility of a mean-reverting price series, it will end up having a lower annualized volatility than that of a geometric random walk, even if both have exactly the same volatility measured at, say, 5-min bars. The opposite is true for a trending price series.

Are you saying that when returns are negatively autocorrelated (mean reverting time series), if you take the standard deviation as a measure of volatility then you will underestimate the "true" standard deviation of the process? And vice versa for positively autocorrelated (trending) returns?

This makes sense intuitively, but I ran some simulations in R and am not finding results that match with my intuition. We can simulate an AR(1) model in r with positive and negative autocorrelation:

> sd(arima.sim(model=list(ar=.5),sd=1,n=1000))

> sd(arima.sim(model=list(ar=-.5),sd=1,n=1000))

and it turns out both of these actually overestimate the true standard deviation (which is 1). Does this match with your intuition? Or am I examining a different phenomenon

Also, if you look at the wikipedia article here

https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation#Effect_of_autocorrelation_.28serial_correlation.29

it seems like from the equation stated at the top - having a negative autocorrelation parameter would lead to sample variance overestimating the true variance and vice versa - which I think is opposite to your statement about mean reverting/trend following variance.

Hi Cherkassky,

Thank you for your kind words.

There is no "true" volatility. Any volatility measurement is a function of the time scale it is measured. You should use the volatility for the time scale of your trading strategy.

In particular, I was not comparing sample (unconditional) variance vs the conditional variance (1 in your example) of an ARIMA process. I was measuring the unconditional variance at various time scales.

Hope this clarifies.

Ernie

Hi Ernie

I feel I was able to create the hurst exponent by your procedure:

https://flare9xblog.wordpress.com/2017/09/24/modelling-the-hurst-exponent/

Hi Andrew,

Thanks for letting us know!

Ernie

Hi Ernie!

The literature says rescaled range (R/S) (and therefore Hurst…) is a measure for “long-term dependence” or long memory. Lo (1991) then modified the test to be robust to short-term dependence.

Do you know if there is an argument against using R/S or Hurst on a short-horizon (i.e. why must Hurst describe long-term and not short-term dependence?)

Thank you!

Felix

Hi Felix,

Such diffusive processes typically exhibit power-low dependence only when diffusion is a bit far away from the original (long-horizon). In the short term, there are other complicated dynamics at work requiring complicated functions to describe the time dependence of their distance from the origin.

Ernie

Post a Comment