Tuesday, January 25, 2011

High frequency trading ideas

I just started reading Larry Harris' book "Trading and Exchanges" (thanks to Max Dama's glowing book review) and already a couple of potential high frequency trading techniques stood out:

"Quote matching" - a technique whereby front-runners place a limit buy order just a cent (for stocks) higher than the best bid price. If the order is filled, they then place a limit sell order just a cent lower than the best ask. Assuming the best bid-ask quotes don't move, the worst they can do is to lose 1 cent by selling the share back to the best bidder, while the most profit they can make is the bid-ask spread plus rebates for providing liquidity minus 2 cents by having the sell long limit order filled. This could work out quite profitably if the bid-ask spread is wide. But of course, the best bid-ask do change constantly, so front-runners would need to cancel and correct their limit orders constantly, and the optimal algorithm for doing this could get quite complicated. Meanwhile, if you are a bona fide liquidity provider, you would have to avoid providing this free option to the front-runners by constantly monitoring who is in front of you. As usual, this chess game can quickly degenerate into an HFT arms race.

"Manipulation of stop orders" - a.k.a. "gunning the market", a technique whereby the market gunners buy aggressively so as to trigger large buy stop orders that they believe are in place at a higher price. When these buy stop orders are filled, the prices are driven higher still, and these manipulators then sell their position profitably.

One of my old momentum strategies was a victim of these market gunners, and after that sad experience I refused to use stop orders any more, at least for stocks. However, here is a question for our knowledgeable readers: can other traders actually see what stop orders there are on an order book (whether for stocks, futures, or Forex markets)? And if so, would a trading robot that simulates stop orders by sending out buy market orders when the stop price is touched work better than manually placing a buy stop order on the order book?

57 comments:

aaron said...

I dont think they can see a stop order but I think they are pretty good at fishing the locations of stops like right above the high or low of the last few minutes or around whole numbers. I'd be willing to bet that there are also some really smart market makers that keep track of market orders that occured recently of 1000 share lots. They know that people who use 1000 share lots are usually traders that can only take so much pain.

phil said...

"This could work out quite profitably if the bid-ask spread is wide."

I think you are missing the key factor. Obviously the wider the spread the better (all else being the same) but more importantly, performance is a factor of the correlation factor between the sign(buy/sell) of sequential orders. A perfect -1 would be, well, perfect.

Ernie Chan said...

Aaron:
Good point!

Phil:
Certainly no profit can be made if my buy or sell limit order is not filled. But if it is not filled, there will not be any sequential order of the same sign executed in my account. So by design, every execution will have perfect -1 serial correlation, for my account.

Ernie

Shaun Overton said...

It's certainly true on the forex side that some groups can see a representative sample of the stop orders. Those privileged groups are the large retail shops and technology providers.

I've seen the whole book before. It makes your jaw drop when you see how densely all the stop orders cluster. No wonder retail loses all the time.

That said, my experience is that the slippage on market orders outweigh the effects of stop hunting. Your stop is still in the same spot whether the broker has it on hand or you're hiding it as a future market order. Removing one order out of 100,000 doesn't exactly affect the statistics.

Ernie Chan said...

Shaun,
Thanks for your insider's view!
Ernie

antoine said...

Here's Oanda's (retail fx broker) enlightening historical open order book plotting limits, stopins and stopouts.

http://fxtrade.oanda.com/analysis/historical-open-orders

Ernie Chan said...

antoine,
The charts are very interesting -- thanks!
Ernie

Marco Aurélio said...

I already heard about the bid-ask method. It was made by a guy trading a med-liq OTM options near of 0.25 cents.
He send orders in both sides, for example b:.24 s:.25, and them stay looking for the book. If one side is filled (ex.: .25), wait for the other get filled, and maybe add new order in the 0.25. If the market start running against him, there are feel .25 orders lefting he close this position just paying comitions.
Sorry if dint make myself clear i not this good explainig this stuffs in english.

A Humble Analyst said...

FX markets are full of gunning it, they will gun their own clients. Bruce Kovner, like many smart traders have a stop buy they dont leave it with any brokers because brokers will gun your stop.

A Humble Analyst said...

Aaron, like Shaun said, the brokers / dealers can see the order book for their clients especially so in FX. in FX many commercial clients (not institutional) have reasonably big size and leave their stops with the dealers. When markets are quiet dealers tend to run these stops.

Anonymous said...

It looks like we started reading Harris' book around the same time! Since I finished reading your book Quantitative Trading, I've started trading at a mid-size equities prop firm and Harris' book has been eye-opening.

I've found the quote matching strategy to be riskier than it appears because often the large sitting orders will vanish and the price subsequently moves rapidly against you. I tend to believe this is another strategy to profit at the expense of quote-matchers. Stop loss orders seem to exarcerbate the loss when they get filled at a very bad price before the price moves back in the direction you had hoped. However, not having a stop loss order in place may be even riskier.

I haven't traded this strategy in the HFT context so I'd be very interested to see more details from that perspective.

Puzzled said...

Hi Ernie,

a somewhat off topic question, regarding picking parameters for pair trading.

Back in October I ran exhaustive searches over the space of possible parameter values for the pairs GLD-GDX and EZA-EWA. I picked these two pairs after filtering many other pairs, as these two seemed more stable that others. The parameters I had were window-size (for averaging), open threshold, close threshold, max number of parallel open trades, and holding period. I considered three year period 2007-2008, 2008-2009, 2009-2010 to check for bias and avoid over-fit. Once I had the parameters, I was running trades in simulation mode since October. The results were horrible, with significant lose.

So I guess I am still as puzzled as ever. How should one pick the parameter values in such a way as to generate revenue and at the same time to avoid over-fit? What are the considerations that one needs to keep in mind? Are there other criteria beside the profit that one should optimize based on?

Thanks for your tips.

Ernie Chan said...

Hi Puzzled,
If you optimal pairs and parameters were profitable in each of the three 1-year periods, and you experience a deep drawdown since October, then there are a couple of possibilities:
1) Maybe the drawdown is expected -- have you checked the maximum drawdown in backtest?
2) Maybe the backtest is flawed -- was there look-ahead bias in the backtest? Was the data adjusted and clean?

Instead of optimizing returns, optimizing Sharpe ratio generally gives better results going forward, and it will also give better compounded return if you follow Kelly formula for leverage.

Ernie

Anonymous said...

Have you made a cointegration test on recent data? Maybe they are just not cointegrated anymore...

Ernie Chan said...

Anon,
Sometimes it is hard to tell whether a pair has lost cointegration until perhaps half a year after the fact!

If the pair has diverged for several times longer than its halflife for mean-reversion, then perhaps it has lost cointegration.
Ernie

Puzzled said...

Hi Ernie,

I double checked the maximum draw-down in the back-test, and it seems that the current lose is small than that. It still feels fishy, however, that first time one trades with these parameters and one ends up with such a lose. Is there a statistical method to measure based on the back-test what is the likelihood of such a lose?

Regarding picking the parameters, I re-run the back-test, and I noticed that there is a lot of influence between the values of the different measurement one gets from the back-test.

For example, I tried to pick parameter values that reduce the maximum draw-down. Such parameters resulted in small number of trades per year (less than 40 trades / year). Also, the maximum number of parallel trades is small (which I guess is good), and so is the maximum holding period. But, with such few trades, one ends up having a small sharpe-ratio.

If on the other hand, one picks parameter values that maximizes the sharpe-ratio, then one ends up with much more trades per year (~140/year), much bigger maximum parallel trades (which I guess is not good), much bigger maximum draw-down, and longer maximum holding period.

I guess other ways of exploring the space of parameter values would result in different properties. What is to be done? Are there any combined measures that somehow take into account, say, the values of the sharpe, the maximum draw-down, the maximum parallel trade count, the trade count per year, and the yearly return? Or at least, how does one compare a high sharpe-ratio resulting from a lot of trades per year with a small sharpe-ratio resulting from a few trades per year?

Thanks.

Ernie Chan said...

Hi Puzzled,
The best performance criterion to pick depends on your specific situation. If you are trading for yourself, there should be less worry about drawdown, so maximizing Sharpe ratio will maximize your long term growth. If you are running a fund, then maybe you have to pay more attention to drawdown, in which case Constant Proportion Portfolio Insurance can be considered (I talked about that in a previous blog post) together with maximizing Sharpe ratio.
Ernie

Milk Trader said...

Traders will put stops in obvious places such as $18.97 instead of $19.00, because even numbers are obvious and they don't want to be obvious. But generally, the place where stops congregate are places where the trader is 'wrong' in his analysis, such as when a trendline breaks or a pivot point doesn't hold. The best way to locate these stops is to think like that trader and ask yourself where you would be 'wrong'.

Anonymous said...

Hi Ernest:

I have just read your book and I think it is one of the best books I have read about the markets in quite a long time * tips hat *

This is way off-topic, but I thought this would be the best place to ask.

I am an engineer that is relatively new to the markets, I havent invested a dime yet because I know this world has many ins and outs that I am still learning. The good thing is that on the way I am learning about many math tools that are useful for my profession as well, so this , for me is a quite interesting hobby at the moment.

I have also been reading some of your posts + comments and I can see that the experience and knowledge you have in this field is enormous, so I figured out that I would ask you some general questions to know your opinion, since these are questions that appear in my mind and I feel am not qualified enough to answer properly.

Here I go:

1) Is there a real significant edge with the use of complex math tools ( complex for the vast majority of ppl, like kalman filter to name one ), compared to, for example, people that use the classical technical indicators like psar, macd, moving averages, rsi, etc and back test/ forward test their systems properly? Or ppl that use trading bots after optimizing its parameters etc? Is it even possible to make money out of these tools? I know many are just pure marketing... If you have any ideas or experiences about this it would be greatly appreciated.

2)If the algo trading, as it seems, is taking more and more weight over time in the markets, would there be any consequences on the real economy? I think mainly about the currencies exchange and how they affect international commerce for example. Before algo trading, I believe the speculation was mostly done by fundamental analysis, based on observations of the real economy, however algo trading here strikes me as something that could lead these markets way off their "real" balance, causing overpriced/underpriced currencies to diverge a lot from what it "should" be ( Balances, exports, imports etc )

3)Markets change a lot from time to time, I understand how quants can adapt to these changes rapidly, because basically they all operate in the short term, as far as i know ( meaning they hold positions intraday or intramonth as much, in general ) How is it then, explainable the consistent success of ppl like Buffet, Soros, Kostolany etc based mainly ( or that is what they say ) on "fundamental analysis" , " value investing" and so on?, I think their success is quite large to call it luck...

Please be as insightful as you can, reading your comments/posts is a mind opener for me :)

Thanks for your time and excuse any possible mistakes in the text, my mother tongue is not english.

Have a nice day!

Milk Trader said...

@anon, I'll take a stab at #2. Whether it's the Chinese farmer hording cotton in hopes the prices continue to skyrocket so he can sell at even higher prices, or the European hedge fund buying euro in the face of sovereign defaults across the continent because the spread between euro and 2-year swaps is "rich", markets often trade anywhere but where the fundamentals suggest. They have a nasty habit of misbehaving, if I were to borrow a phrase from Mandelbrot.

Ernie Chan said...

Hi Anon,
Thank you for your kind comments on my book.

I agree with Milk Trader's answer on #2. Algorithmic traders are typically short-term traders, and as such, they provide liquidity to other investors or speculators who hold longer term. When there are too many algo traders, the price of liquidity become very cheap, and most algo traders will be driven out of the market, restoring equilibrium to the ecosystem.

On #1, so far I have not seen any evidence that sophisticated machine learning tools are able to outperform technical indicators. See my many other blog posts on AI and similar topics.

On #3, there is no contradiction in both techniques earning good returns. Fundamental investors earn returns on their economic insights, while short-term traders earn returns on providing liquidity to these fundamental investors. It is, however, not a zero sum game if you believe that ultimately fundamental investors help the economy grow.

Ernie

Anonymous said...

Hi Ernest, I was googling about automated trading and found your blog, looks quite nice.

Could you tell me what are the expected returns of an algo trading strategy? I read many places where they claim to sell soft that hits 10% a week or more, however I think that is quite difficult. ( everyone would be rich :) )

As an expert in this field, could you give me a real "range" of expected payoffs? ( using proper money management ) and how long should one expect a single strategy to be profitable?

Or at least a number from which you can imply that the software is a scam with reasonable certainty?

Thanks in advance!

John Marino said...

Hello all:

I a noob trying to learn enough about this topic to get started.

Can a book that was written nearly 10 years ago on as fast-moving subject as algo trading be useful today? I would think that any strategy that works would eventually stop working due to too many participants chasing the effect. That said, if the benefit is structural (eg: short-term traders get paid to source liquidity to the long-term guys), I can see it persisting longer, but eventually there can bee too much supply of liquidity providers, driving down the price of their service.

John Marino said...

Hmmm. "Heather Johnson" posted on iStockAnalyst an exact copy of Ernie's post with no attribution.
http://www.istockanalyst.com/article/viewarticle/articleid/4834578

Ernie Chan said...

Hi Anon,
From my own experience in the last year, an average equities strategy can expect about 10% a year unlevered, and FX strategies can expect about 20% unlevered.
Ernie

Ernie Chan said...

Hi John,
Yes, some books on algo trading can remain useful even if written more than 10 years ago. An example would be the Market Models book on my Recommended List on the right sidebar of my blog.
Ernie

P.S. Thanks for the alert on Heather Johnson's copycat blog. It is actually quite disgraceful since she was a guest blogger of mine some time ago.

Bernd said...

I love comments here, learning a lot from all of you guys!!

Ernie , one off-topic question, regarding FX, have you managed to trade the London Breakout? It has a lot of relevance in most fx forums as far as I have seen.

From a math point of view, it would be nice to try to "model" the impulse, but am guessing things are much easier with other techniques.

If you have any exp with this, could you give me some hint? just risk management? technical indicators? price action / chartism?

btw, what is your opinion about price action / chartism? does it work for you? ( head & shoulders, tweezer tops and so on...), given the fact that so many ppl know it, i guess it shouldnt work, however am just a newbie in this field, so your insights are very appreciated!

Regards and thanks for your blog!

Ernie Chan said...

Hi Bernd,
Thanks for your tip about the London Breakout. I haven't heard of this strategy before, and not being a chartist, I am not even sure that I can interpret the terminology correctly.

However, I think I can recast the strategy in a quant's language (anyone more familiar with chartist terminology please correct me here!). So here goes:

Take the previous daily high and the 5 highs of the 1-hour bars from 0:00 ET to 4:00 ET and use linear regression to fit a straight line through them. This is the "downtrend line". If the slope is positive, then do not trade this morning. Similarly, construct the "uptrend line" using the previous daily low and the 5 recent 1-hour lows.

At any time going forward, enter buy market order if the last price is higher than the uptrend line, and vice versa for downtrend line.

For the simple version of the strategy, use a fixed profit target of, say, 20 pips for EURUSD. Close out any existing position 3 hours after entry.

Am I paraphrasing the strategy correctly?

Ernie

Bernd said...

Hi Ernie:

Basically that is correct, some people use Fibo levels as well, some go for 10 pips instead of 20,some use other pairs, etc, but basically what you expose is correct. ( there is also a version for the beginning of the american session )

As far as i know, the logic under this strategy lies in the fact that when the London market or Frankfurt market open (1 hour away one from the other ), there is a "rush" of liquidity in the market, so all the technical figures + indicators are more efficient due to the fact that they are just a reflection of "mass psychology". That is what i have read, dont know for sure, but seems at least logical.

I have read a ton about this, they recommend NOT to play it on Fridays, X-mas , Summer holidays and so on due to the lack of liquidity, ( all the big boys are in Hawaii enjoying their bonuses! ), being careful when relevant news like NFP are about to be broadcast, fiscal year closings etc.

There is a guy who made a complete "forex training" page about this, it was funny to later find out that from all the places in the world, he lives around 15 min-bus ride from me :D. He uses technical figures like double bottoms / double tops / 1-2-3 patterns + MACD normally.

http://blogs.forex.es/vidaoperando/

Site is in spanish, but pics are pretty much self explanatory, if you need help with translation just let me know and I will gladly help.

If you are interested in more forex info, I have some more that i find pretty interesting, but havent put to a test yet, let me know about it and i will post it here as well or mail to you, as you prefer.

Thanks for your interest and hope you find this info useful ;)

PS:
If you find a tweak to make it super profitable dont forget to mail me a couple of millions! :D

Take care.

Bernd said...

Hi Ernie, my previous post vanished :(, blogger error i guess.

Yes, you are right, there are many tweaks to it, but the basic strategy is supposed to work like that. The opening of the London/Frankfurt markets create a rush of liquidity that makes technical figures more reliable since they are based on mass psychology ( that is the main idea behind the strategy as far as i know ), so trading it on holidays is not recommended.

Some ppl use fibo levels, 1-2-3 patterns, double tops,bottoms, etc to trade it. I havent really put a lot of effort into it, but am planning to, however, backtesting strategies like that is quite painful since it is not an automated approach.

Any ideas to light this problem with tools you may know?

Bernd.

Paul said...

Regarding that copycat post, I don't think it was Johnson that posted it. That site is a content farm and it mistakenly took Johnson's name as the author. If you look, Johnson's name links back to here.

There are plenty of sites like that. They all have an obscure link back to the original post to avoid being charged as blatant plagerism.

Just pointing this out so that the innocent (Johnson) doesn't get blamed.

Ernie Chan said...

Bernd,
Actually, the way that I described the London Breakout is quite suitable for automated backtesting, as long as your have access to linear regression function (e.g. via Matlab).

I will give it a try when I have some time and post the results here.

Ernie

Ernie Chan said...

Thanks, Paul, for clearing things up!
Ernie

Ernie Chan said...

Bernd,
I could not make the London Breakout strategy work using the trend lines as I described.

However, I found that it does work well using generic volatility breakout methods for momentum trading, entering at around 9am GMT as in the original description, and holding a maximum of 3 hours. So I consider this strategy verified!

Ernie

Bernd said...

Hi Ernie:

"generic volatility breakout methods for momentum trading"

when you say this are you talking about any matlab tool to help you do this? or is it the basic approach of buying @ highest high , waiting to a channel breakout, etc.

I remember i tried the later approach (with paper money) buying/selling with technical indicators/chart patterns...0 sum game for me. Many people claim they are able to do this on many forums, but i find their approach quite subjective, sometimes they use fibo channels, sometimes trendline ruptures, sometimes MACD... so they do not have a system that is easily automated.

If you could elaborate a little bit more it would be nice, i dont really care about the money, but the problem solving itself.

Do you need to get that "intuition" to operate manually? my logic says no, but maybe i just need more training...

thanks for the light!!

Bernd said...

Hi Ernie!

thanks for the feedback!

when you say:

"generic volatility breakout methods for momentum trading"

I dont really know what you mean, could you put an example or point me to some place where i can learn that? or elaborate the concept a little bit?

thanks in advance!

Ernie Chan said...

Hi Bernd,
A generic breakout method can be, for e.g., whenever the price breaks above the upper Bollinger band, we buy. The Bollinger bands can be computed based on previous days' prices at some specific time.

As with most momentum strategies, I also find that both profit caps and stoplosses are needed.
Ernie

Bernd said...

Thanks for the tip Ernie!

Another thing I would like to ask you..why MatLab and not mql?

Isnt the use of matlab "tying" you to specific brokers who offer the API to interact with their platform?

MQL would tie you to brokers using metatrader, as far as i know, but you would still earn some power in case things go wrong with your specific broker.

Matlab obviously has much more math power inside, but for short term strategies (intraday), it is mostly a matter of finding the right strategy using standard indicators,right? mql would be really fast for this i think.

For intraday trading do you also use matlab? or would you recommend any other tool?

PS: I think that if you posted about " developing an intraday strategy 101" with hands-on examples, it would raise a lot of interest for a lot of ppl.

thanks ;)

Ernie Chan said...

Bernd,
Since I don't know anything about mql, I can't comment on whether it is better or worse than Matlab. But one thing I know: I can implement any strategy with Matlab, whether intraday or not, and for any market. Also, many equities strategies (as opposed to futures/FX strategies) cannot be reduced to technical indicators, and so they cannot be handled by many standard trading software. Using Matlab does not tie to you a brokerage, as long as you can write a Java-to-Matlab link.

But if trading FX is the only concern, and if mql is easier to learn than Matlab, then I am all for it!

Ernie

Dr. Troy Lau said...

The question is... what percentage of the time will your stop loss order be executed before the sell at the ask? Even if you have a potential 5* profit with the spread, if the stop loss gets executed 95%t of the time it's a losing strategy.

Intra-day price data coupled with intra-day level II data, back tested will answer this question.

Read my quantitative investing blog at

http://thequantinvestor.blogspot.com/

Anonymous said...

Hi Ernie

This is a somewhat off-topic post, mainly because I didn't know where else to post it.

Have you read "Ben Graham Was A Quant" by Steven Greiner? I was looking forward to your review on the book.

Thanks

http://www.amazon.com/Ben-Graham-Was-Quant-Intelligent/dp/0470642076/ref=sr_1_1?ie=UTF8&qid=1299656959&sr=8-1

Rahul said...

Hi

I required
Perry Kaufman New trading system and methods

Please someone give me download links for that

If u have this book or any link please send me on my email-id

its rahul.dhawan09@gmail.com

thanks in Advance

Ernie Chan said...

Hi Anon,
Thanks for telling us about "Ben Graham was a Quant". From what I see, the book isn't published yet. But I will keep an eye out for it.
Ernie

fdax said...

and what about about cash vs futures or cash vs etfs ???

i guess this is the perfect topic for a frequency trading idea ...

fdax

Ernie Chan said...

fdax,
Yes, theoretically index arb seems an attractive HFT idea. In practice, it is not easy to make it work, especially for independent traders, due to brokerage latency and other problems.
Ernie

fdax said...

sure,
i'm just trying to bring another idea of HFT, i've heard index arb is the main activity in the HFT field,
do you have any experience with that type of trading ?

fdax

Ernie Chan said...

fdax,
I have written about an index arb strategy in the Premium Content area of my website:

http://www.epchan.com/subscription/IndexTracking.pdf

Subscription info is here: http://www.epchan.com/subscriptions.html

Ernie

Anonymous said...

Hi Ernie

The book "Ben Graham was a Quant" is available as a Kindle ebook in certain regions.

Thanks.

Anonymous said...

The strategy of gunning stop orders, strictly speaking illegal market manipulation. SEC recently fined high-frequency shop for this type of thing.

spritrig said...

The advantaged of a robot simulating a stop and entering a market order is not giving your adversary prior information about your next move.

Stop orders could be used as bait. Close your position and cancel your stop before the price gets there.

Muahahhahaa!

spritrig said...
This comment has been removed by the author.
Anonymous said...

Hi Bernd,
A generic breakout method can be, for e.g., whenever the price breaks above the upper Bollinger band, we buy. The Bollinger bands can be computed based on previous days' prices at some specific time.

Hi Ernie,

Could you give an example of how you compute Boll bands as mentioned above? Do you take one price (say at 9:00 GMT) per day, over say a month and then figure out the st dev and moving averages using those 20 closing prices?

thx

Ernie Chan said...

Hi Anon,
Yes, your example of a Bollinger band computation is similar to what I have in mind. In both of my books, I have given numerous examples of Bollinger band strategies.
Ernie

Anonymous said...

Hi Ernie,
I have an enquiry about mean reversion strategy. I have developed a program to detect Mean Reversion. But I do not know where to start applying this strategy to HFT. thanks for your help.
Amin

Ernie Chan said...

Hi Amin,
Have you backtested your mean reversion strategy on high frequency data?
Or have you paper traded this strategy using live data?
Ernie

Unknown said...

Hi Ernie,
Is it possible now to do quote matching without ultra hft (about half a second delay) in slow illiquid stocks with wide spreads?

Ernie Chan said...

Hi Danil,
I haven't tried, so I don't know. But it does seem plausible.
Ernie