Thursday, April 04, 2013

An Integrated Development Environment for High Frequency Strategies

I have come across many software platforms that allow traders to first specify and backtest a strategy and then, with the push of a button, turn the backtest strategy into a live trading program that can automatically submit orders to their favorite broker. (See all my articles on this topic here.)  I called these platforms "Integrated Development Environment" (IDE) in my new book, and they range from the familiar and retail-oriented (e.g. MetaTrader, NinjaTrader, TradeStation), to the professional but skills-demanding (e.g. ActiveQuant, Marketcetera, TradeLink),  and finally to the comprehensive and industrial-strength (e.g. Deltix, Progress Apama, QuantHouse, RTD Tango). Some of these require no programming skills at all, allowing you to construct strategies by dragging-and-dropping, others use some simple scripting languages like Python, and yet others demand full-blown programming abilities in Java, C#, or C++. But which of these allow us to backtest and execute high frequency strategies?

To state the obvious: backtesting HFstrategies is quite hard. The volume of data is one issue. But in addition, the execution details are very important to such strategies: details such as the exact exchange/venue to which we are routing our orders, the precise state of the order book that triggers our orders, the order types we are using, and finally the probability of getting filled if we use non-marketable orders. Messing up one of these details and the backtest will be far from realistic. I often tell people that it is easier to paper trade a HF strategy than to backtest one. While many of the platforms I reported above do allow backtesting using tick data, I don't know that they enable backtesting using the full order book and choice of execution venue. With this background, I am happy to report I have recently come across just such a platform called Lime Strategy Studio.

First, the bad news. LimeTrader is useful only to traders who trade with Lime Brokerage, as it is configured to send live orders to Lime only. [UPDATE: I have since learned that there are adapters available for 3rd party brokers.] However, if you are going to trade HF stocks and futures strategies, why not go with Lime, since they provide you with a comprehensive API, direct ultra-low latency feeds from the exchanges, and allow (nay, insist on) colocation either at the exchanges or at their data center at a reasonable fee? (Full Disclosure: I have no current business relationship with Lime, though I was a customer.) Another piece of bad news: the specification of the strategy must be in C++.

But once you get over these two hurdles, the benefits are manifold. Every detail that you can specify for a live trading strategy can be specified for the backtest and paper trading. As I said, these details may include order type, trading venue, state of order book, and even statistics of the order book, not to mention fundamental data such as earnings, corporate actions, and other user-provided data such as news. A fill simulator is included for your non-marketable orders. As with other IDEs, once you backtested a strategy in its every detail and are satisfied with the performance metrics, you can go live (either for paper or production trading) with the push of a button.

If any reader know of other IDEs that have similar features and useful for backtesting HF strategies, please let us know!

===

Speaking of HF strategies, traders often lament the ultra-high secrecy around them and the difficulty of gathering knowledge in this field. A friend (hat tip: Dave) referred me to this paper by Prof. Dragos Bozdog et. al. that gives a flavor of what sort of modeling may be involved. I find it very readable and thought-provoking.

===

There are still 2 slots available in my online Mean Reversion Strategies workshop scheduled for May.



69 comments:

Anonymous said...

Could you share your opinion on PRIMUS Trade? Maybe recommend a similar but better (or more cost efficient platform)? It is similar to the platforms listed above and the company I work for currently uses it. Link to the platform: http://www.primustrade.com/

(it was designed for equity trading)

experquisite said...

Heya Ernie, thanks again for an informative post. Is there middle ground for developing a HF trading system from a standing start, other than opening an account with Lime? ie -- is there a (recommended) source for tick data that can be downloaded and backtested against offline, and how does bootstrap an analytics platform/IDE with it?

Ernie Chan said...

Anon,
I took a quick look at PRIMUS, but am not sure that it is suitable for HFT. I do not see any mention of access to historical order book info, nor the ability to direct an order to specific venue only.
Ernie

Ernie Chan said...

experquisite,
If you are patient enough, it is possible to backtest HF strategies using Matlab/Python, but you still won't have a good fill simulator. Of course, this won't matter if you are sending MKT orders.

If your strategy involves the order book, this would become even more difficult.
Ernie

experquisite said...

Understood, but where is the best place to get the raw data if you don't want to choose your broker yet? I suppose the most effective thing would be to get the tick data archives into an amazon instance, but then you have to choose the data format, the provider, all these details -- is there any primer on this sort of thing? Should one be targeting some kind of KDB, or HDF5, or something else?

For prototyping without orderbook (assuming fill simulation is not the most important aspect right now), can one still use python pandas or does it hit a wall? What about strategies and transforms to operate on live 'stateless' data, aka using exponential moving averages instead of simple, except for other common stats operations?

Background info/references on that sort of stuff would be awesome!

Ernie Chan said...

experquisite,
For raw tick data, it is available from tickdata.com, cqgdatafactory.com, and nanex.net.
They have vastly different pricing and data formats - be careful with your choice!

Python is possible for strategies that do not involve order book.
Ernie

Jane Fox said...

Ernie,

This is a tad off topic because it doesn't involve HF strategies.

I noticed you mentioned some trading/development platforms but did not mention Amibroker. Amibroker does not require C++ code but does require code in its own AFL language.

Have you heard of Amibroker (of course you have)? I would like to know yours thoughts and if you think its as good as the ones you mentioned.

Jane

Ernie Chan said...

Jane,
Yes, indeed I have heard of Amibroker. But not having used it myself, I can't say much about it. Perhaps other readers can comment?
Ernie

Anonymous said...

Hi Ernie

Have you ever heard of this website

http://orderbookfx.com/

Any Comments??

Thx

Ernie Chan said...

Hi anon,
After a quick browse, I cannot figure out which exact ECNs and FX platforms orderbookfx.com actually aggregate. Can you?

A comparable product is Streambase, which no doubt is many times more expensive. The difference, though, is that we can actually send orders to hit those quotes on Streambase.
Ernie

Anonymous said...

manifold -> manyfold

Ernie Chan said...

Thanks for spotting the typo, Anon!
Ernie

Anonymous said...

Will your new book be available on Kindle?

Ernie Chan said...

Hi Anon,
Yes, my book should be available on Kindle and other electronic readers.
Ernie

Jane Fox said...

Ernie,

How do you deal with "holes" in your data? Holes can occur when a ticker isn't trading or when the data feed is interrupted. They can last for any length of time and can occur in any timeframe.

In Amibroker, you can "Pad" the holes with data from previous bars.

Jane

Ernie Chan said...

Jane,
As I use Matlab for backtesting, it is very easy to fill forward data to fill any holes.

In fact, any reader of my book has access to the function fillMissingData downloadable from epchan.com/book.

Ernie

Minsheng Wu said...

Hi Ernie,
Will you have courses in Singapore? I saw that you used to conduct courses in Sinapore. I have read from your old book and find it very useful. - Minsheng

Ernie Chan said...

Hi Minsheng,
Yes, I will teaching algo trading at NTU July 25-28. For more info and registration, please email nscfe@ntu.edu.sg
Hope to see you there!
Ernie

Anonymous said...

Hi Ernie, I am from Hong Kong and nice to meet you:) Why do you focus on MATLAB but not building a total new system with C++/C#/Java? Isn't that MATLAB focuses on complicated math calculation for research kind instead of real time trading?

Ernie Chan said...

Hi Anon,
Indeed it is much more efficient to do research and backtesting in MATLAB than in C++. For low frequency executions MATLAB also works well. We only need C++ if we want to build a HF executions systems.
Ernie

HK said...

Hi Ernie,

Thanks for your quick reply!

then what do you use for your live system? I am planning to use the opensource Tradelink to backtest and build an automated system. It will also improve my C# skill:)

Thanks!
-HK

Victor said...

Hi Ernie, one IDE worth taking a look at is Multicharts (http://multicharts.com). They recently released a .NET version where you can design indicators and signals in C#. I just downloaded a free starter version (just for 2 symbols) and I'm trying it right now with Interactive Brokers. So far so good.

Ernie Chan said...

Hi HK,
I use Matlab for my live low frequency systems, and C# and C++ for higher frequency ones.

I heard TradeLink is a good API too.
Ernie

Ernie Chan said...

Hi Victor,
Does multicharts allow you to backtest portfolio strategies, where the trading signals of one stock may depend on the price history of possibly all other stocks in the universe?
Ernie

Anonymous said...

Ernie,

Multicharts is basically a Tradestation clone that isn't tied to a broker.

Ernie Chan said...

Anon,
It is cool that multicharts is broker-neutral, but can it backtest portfolio strategies?
Ernie

Anonymous said...

It can backtest, but there are serious limitations.

http://www.youtube.com/watch?v=PMxVmdtzXxA

http://www.youtube.com/watch?v=ZdJ-Orj5R-U

As a discretionary retail trading platform, it is not horrible. For algorithmic trading, you really don't need it if you are already using Matlab or Python.

Something to keep in mind about these retail platforms is that they are not optimized for performance, they are just optimized to work with high latency broker apis like IB's. Plus, neither multicharts nor tradelink have fix connectors. So, you can't plugin your broker via fix unlike with matlab or python. Still, I would say that metatrader is at least, semi-decent, compared to the rest.

If you really need faster execution, outside of paying for some all-in-one suite from quantfactory, tbricks or deltix, it's best to just go the EMS route. From what I heard about all-in-ones, quantfactory and tbricks tend to be cheaper than Deltix (you can look into that). So, you can still model strategies in matlab, but for execution, you can use something like xtrader, cqg, bloomberg emsx, etc.

Also, here is something else that you might want to look into. It is a demo of Simulink being used for financial modeling:

http://www.4shared.com/video/eJEOyBbJ/file.html

http://www.mathworks.com/matlabcentral/fileexchange/24320-algorithmic-trading-with-matlab-2009-update

In theory, you should be able to simulate an orderbook and do fill simulation with simulink and stateflow (part of matlab).

Victor said...

I read in the forum that Multicharts will add portfolio trading in version 9.0, that is expected in the first semester of 2013.

HK said...

Hi Ernie,

How do you have a high frequency system at home? I think:
1. A normal home internet speed may not have HFT kind of speed
2. Home computer may not be fast enough to do that kind of real time analysis/offline backtest.
3. High frequency system would take $0.001 profit in each trade, while retail brokerage would charge much higher than that for each trade.

Welcome to share how can you build that, I thought it is only possible with big financial institution.

-HK

Ernie Chan said...

Hi Anon,
Thank for your detailed comments on various IDE.
Regarding Tradelink, I think they advertised that it is FIX-enabled. Do you find that not true?
Ernie

Ernie Chan said...

Hi HK,
You can't of course literally run HFT strategies "at home". But you can build such strategies and run them at a collocated server.

It is also true that most retail brokerages would not be suitable due to high commissions or high latency. For stocks, you need a larger account so that you can trade at Lime or some other institutional brokerage. For FX and futures, there are more retail brokerages that allow you to collocate and have reasonably small commissions.

When I said HFT, I am referring to trades that take place between 1-100 ms. For sub-ms trades, I don't think retail traders should attempt them.

Ernie

Anonymous said...

@Ernie

Lightspeed is a retail broker, it also offers HFT capabilities and co-location.

The Tradelink project overadvertises its capabilities. It doesn't have a working fix connector. What they have is a 'blank template', basically you need to make a fix connector. Very deceptive. Plus, it is a very unstable program.

Have you considered trying out Simulink/Stateflow for order book and fill simulation?

Ernie Chan said...

Anon,
Good to know that Lightspeed offer colocation for stock traders.

It is perhaps possible to build a realistic fill simulator using Mathworks Stateflow, but I would rather leave that to 3rd party firm or a broker to do. Especially for Forex, each broker/ECN will need to be tuned independently for this simulator to work.

Ernie

HK said...

Hi Ernie,

Thanks for your answer. I am reading your book and you mentioned a good profitable trading strategy can be/should be "Make everything as simple as possible, but not simpler". Is that because a very complicate strategy would be curve fitting?

Why do banks/ibanks hire proprietary traders to manually trade, especially we have much higher technology for trading in these decades? I saw a main casino company in Macau hires proprietary trader trainee to trade stocks/derivative in HK market. The post mentioned they taught systemic method, unfortunately I couldn't get an interview for this job.

Here is a little history of me:
I had interviews for trader assistant of a worldwide bank and trading system developer of a main global trading system company, both interviews were many rounds and hours but finally I couldn't get it. My IT career is alright and I keep moving on but never had a chance to be in a serious trader related position. I was in an OTC derivative broker trainee position but it was all about trading speed and relationship, that's not what I am good at and want to be so I got another job.

-HK

Ernie Chan said...

Hi HK,
Yes, simplicity is a way to avoid overfitting.

A discretionary trader may beat an algo system if 1) the trader possess fundamental market knowledge that the algo system doesn't, or 2) the market characteristics change rapidly so that the algo system cannot catch up but the infinitely adaptable human can.

But a lot of times, the "human skills" are illusory and may simply be good luck. The realization of this leads to the proliferation of algo trading across the globe.

Ernie

HK said...

Hi Ernie,

I see, so only very few human in the world can be true master of manual discretionary trading under high pressure with super rational thinking and fundamental knowledge, and there is almost no way to tell if it is luck or real skill. Thanks and I am reading your book again:).

-HK

vincegata said...

Dr. Chan,

What is the better way to analyze quotes, in the raw form or consolidated into, say, 1 sec OHLC? If I do not consolidate then the number of quotes per given time interval will be different e.g. EUR/USD 2000 quotes 9 am-10 am but 1200 quotes 3 pm - 4 pm. Also, if I do not consolidate then the number of quotes will be different for different symbols, so how can I calculate their correlation?

Thank you.

Ernie Chan said...

Vincegata,
To calculate correlations, you can indeed consolidate quotes into fixed time intervals, whether they are 1 sec or 1 min, depending on your trading frequency.
Ernie

HK said...

Hi Ernie,

Other than Monte Carlo method, would you suggest some other famous algorithms or theories that we can study? Thanks.

-HK

Ernie Chan said...

Hi HK,
Actually, I am not sure what you are going to use Monte Carlo methods for?
Generally, I don't recommend using any fancy methods for quant trading. The simpler the method, the likelier it will actually make money.
Ernie

HK said...

Hi Ernie,

I see. I will back-test and see if methods like Monte carlo doesn't help much:)

Ed Seykota doesn't believe day-trade is a good way to have profit:
http://www.seykota.com/tribe/faq/2007_Nov/01/index.htm

Do you think day-trade is not effective?

For subjective ways like Elliott Wave and W.D. Gann, do institutional traders use these methods?

How about Thomas DeMark?

-HK

Ernie Chan said...

HK,
Sorry, I don't agree with most of what Seykota, Elliott, Gann, and DeMark said. My trading principles are mostly based on academic research coming out of the finance departments of various universities and the research desks of the US Federal Reserve Banks.
Ernie

HK said...

Hi Ernie,

May I know how to get the information of the research desks of the US Federal Reserve Banks?

If there is a surprising big accident happens, would you choose to not let your computer trades because the accident effect would abnormally affect the market?

-HK

Ernie Chan said...

Hi HK,
Google, for e.g. "Federal Reserve Bank, St. Louis, research".

In theory, we might say that the markets are abnormal in the aftermath of a surprise, and we shouldn't trade our models. In practice, it is hard to determine what exactly constitutes a surprise, and I have seldom found any benefits in overriding one's models.
Ernie

HK said...

Hi Ernie,

So basically we don't need to care financial news at all? Would you focus on corporates' P/E, dividends, and government release data? For me, I think both corporates and government can release not real data or hide whatever they want, especially government has the authority to make stuff up to stimulate the market or control crowd expectation. It would be hard to fake dividends in long term because corporates really need to pay the money.

-HK

Ernie Chan said...

Hi HK,
Your question was about surprises, not about financial news.
In the aftermath of true surprises, there is not much we can do. But for expected news releases, we can refrain from trading ahead of time, and this can be part of the trading strategy that can be backtested.

Ernie

HK said...

Hi Ernie,

I get your meaning. So the effect of expected news like FOMC meetings can be back-tested and be part of a strategy. It doesn't matter if gov releases real data reflects the true economics or not, the effect can be back-tested.

Comparing to stock market, would FX future market be more predictable and has trend characteristic?

Stocks in Hong Kong are very tricky and not fair at all. The stock exchange over here is very loose. I trade ETF and futures but don't like to trade stocks in Hong Kong, because a stock of a single company is so risky even not small stock. Stocks that constructs Hang Seng index can be very risky too, like 330 ESPRIT or 267 CITIC PACIFIC.

-HK

Ernie Chan said...

Hi HK,
Yes, futures market has more trending behavior.

Like you, I haven't trade individual stocks for some time, even in the US market.
Ernie

HK said...

Hi Ernie,

If a strategy with no more than 5 variables is the best, then why financial institutions keep spending so much money to hire phd quants, programmers, traders for high/low frequency trading? Those financial giants should have knew what could be the best.

-HK

Ernie Chan said...

Hi HK,
Most of the quants in investment banks are hired to model derivatives and fixed income models, which are highly nonlinear, and much more complex than cash equities or spot FX models. For the latter 2, linear models are preferred.
Ernie

HK said...

Hi Ernie,

I will definitely read your coming new book. Any other reading suggestion?

-HK

HK said...

Hi Ernie,

It is so hard to even find a job post for getting into proprietary trader or related programmer industry. almost all are always hiring very experienced people.

Look like banking are hiring less people for years after financial crisis. It has been many years but look like it is still so bad. I am not sure why especially I am in Asia and economics over here doesn't look like that bad.

-HK

Ernie Chan said...

Hi HK,
You can find other suggested books on the Recommended Books list on the right sidebar of my blog.
Ernie

HK said...

Hi Ernie,

Is that very common for a professional including trader in proprietary trading industry becomes an independent trader?

-HK

Ernie Chan said...

Hi HK,
I wouldn't say it is very common. Most traders working for big banks enjoy their big salary and bonus there.
Ernie

Anonymous said...

RTS' RTD Tango has full-depth backtesting, including FIFO and pro rata matching algorithms.

Anonymous said...

Just read the news from Dow Jones, EBS is going to deploy a new system to rein in high-frequency traders...see below
--------------------------------
Foreign-exchange platform EBS is considering strict new handicaps on high-frequency trading.
ICAP PLC (IAP.LN, IAPLY)-owned EBS is planning changes to how it processes orders to buy and sell currencies that it says would eliminate certain advantages for the fastest traders.
The platform would do away with the "first in, first out" system of order processing, which gives the trader who places an order first access to the most favorable exchange rate. EBS says this system has been exploited by increasingly nimble programs that can buy and sell positions in microseconds, before other market players can react. Instead, EBS would bunch client orders together as they come in and execute them randomly, a spokesman said.
EBS is considering launching the new system by summer, the platform confirmed.
High-frequency trading has become an increasingly significant part of the foreign-exchange market in recent years, and it now makes up about 40% of all trading, nearly double the share in 2007, according to consultancy Aite Group. These firms have been able to pump out orders and snap up prices at such speed that they have nibbled away at banks' large orders and made it difficult for them to complete large trades at the price they want.
EBS has taken steps to rein in high-frequency traders before, as its core clients, including large banks, complained that the platform wasn't doing enough to protect their ability to trade. Last September, EBS said it would expel users who launch a swarm of rapid-fire trade requests they have no intention of completing. The tactic is a common one for high-frequency algorithms, which use it to gauge the immediate direction of a market, while frustrating lower-speed traders, as it pre-empts their efforts to execute sizable orders. The exchange also dropped the fifth decimal place in its currency quotes, removing most of the finer price points that work best for high-speed computer-based traders.
Spot foreign-exchange volume on EBS was down 2% in March from the previous year and 23% from 2011. Meanwhile, currencies-trading volumes on systems owned by EBS's top competitor Thomson Reuters Corp. (TRI, TRI.T) have grown since Thomson Reuters bought FXall, another dealing system.
On April 18, Tradition, a brokerage, launched a rival currency-trading system called ParFX. Tradition's platform is backed by some of the biggest dealers in the $4 trillion-a-day foreign-exchange market, including Deutsche Bank AG (DB, DBK.XE), Barclays PLC (BCS, BARC.LN) and UBS AG (UBS, UBSN.VX).



(END) Dow Jones Newswires
April 29, 2013 13:24 ET (17:24 GMT)

Anonymous said...

Hi Ernie,

Would you know the main reason(s) that the futures market trends more?

IC

Ernie Chan said...

Hi IC,
That is due to the roll returns of futures, which typically persist with the same sign for long periods.
Ernie

Anonymous said...

Thank Erine,

Look forward in your new book.

IC

Jared said...

I'd recommend QuantConnect to your backtesting IDE list; we're the only one with 100% free tick data for 16,000 equities and FX. Just C# now but we're adding multi-languages soon. Backtesting tick simulations in minutes vs hours on your home desktop PC.- Jared (Founder @ QC).

Ernie Chan said...

Hi Jared,
Thanks for the heads-up!
Ernie

danice said...

Hi,
I think that discussion should begin by explaining what HFT is. Even if you're using tick data, it still may not have to much with HF. There is a lot of misunderstandings, but of HF strategies typically require management of queue in order book.

R, Matlab and python are commonly used in HFT. It is important to be able to mix script code with C/C++. To test you need exchange simulator (Tradestation, Amibroker etc. completely unsuitable for this).

Daniel

Anonymous said...

Hey Ernie,

Do you know if MetaTrader can be used for arbitrage? What do you think are its major limitations? Can it be connected to R?

Regards.

Ernie Chan said...

Hi Anon,
I am not familiar with Metatrader, but with sufficient tinkering, I believe most platforms allow you to trade some sort of arbitrage strategies.

I will leave the R-Metatrader question to other readers who may be familiar with these platforms.

Ernie

Bjorn said...

Hi Ernest,



I’ve been following your blog for a while with much interest.

I am in to FX Spot algo trading since 5 years and I am heading towards HFT and MFT more and more.



I don’t know if you have time to answer this, my problem is that I don’t know of wich platform to work with.



Problem is that I am working with mql4 algos and I have a bunch of HFT-MFT that theoreticly and in backtests work very well.

Research gives that mt4 brokers, no matter if they claim to be ecn. Stp, mtf, exchange and/or if they are connected to currenex, still seems to filter tick data live feed.



Comparing tick data from mt4 ecn brokers including LMAX wich claim to be MTF and the first exchange broker on spot fx retail, I still see that tickdata differs a lot when compared to LMAX API .net plattform. Analyse shows that there are 10 times more ticks/sec on LMAX api .net solution



I need to have a backtesting ability, a paper trade ability and also ability to optimize the algo. And it has to be a platform that supports Spot FX.



Do you know wich plattform that supports all this?

And also: I have noticed that you have a course in MFT trading in October, would that one be suitable for me?

I am very interested to participate, but I assume it’s a question of platform and language to work with.



Best regards

Bjorn

Duke said...

IF source code size is very big,CPU needs time to decode it and Quote chart will become very slow, How to slove this issue? ( I have not supercomputer)

"On his last day of work at New York-based Goldman Sachs in June 2009, Aleynikov uploaded hundreds of thousands of lines of source code from the firm’s high-frequency trading system"
http://www.bloomberg.com/news/2012-02-17/ex-goldman-programmer-s-conviction-overturned-on-appeal.html

Ernie Chan said...

Duke,
If high performance is required, do not use charting while performing live trading. Let your algorithm makes the trading decision: it will on average be better than discretionary ones.
Ernie

Anonymous said...

Ernie,

I'm glad you liked Lime Strategy Studio. I wrote the product a while back.

I came across your blog randomly while searching for something else related to finance. Quite the surprise to see the thing you built reviewed by someone else!

Nathan