Friday, November 11, 2011

Trading platform and EC2 revisited

Recently I opened a discussion on the various software platforms which allow the programmers among us to build trading strategies easily. Here is one other addition: Quantopian. It is only in alpha stage, but I did get a preview of its features:

1) You can code in Python, which is an easier language to learn than Java, but no less powerful. In fact, I know of a superb programmer who uses Python to backtest HF strategies.

2) It is web-based, which means you can take advantage of collocation on a server much more stable than your own desktops. (For those who worry about the confidentiality of your strategies, the founder indicated to me that they can run an image of the software on an Amazon EC2 account that you owned so they won't have access to your codes. As for the confidentiality of codes residing on EC2 itself, please see below*.)

3) It is event-driven (or for those who like the latest jargon: CEP-enabled), like all the Java API's that I discussed in the previous article.

4) They have 1-min US equities data for backtesting. Tick-level data will be available soon.

5) Toolboxes for common technical indicators, mathematical algorithms, etc. will be available soon.

6) They will run a competition for trading models which makes it easier for independent traders to become trading advisers to others, or to raise money for their own funds.

Unfortunately, live walk-forward testing is not yet available.

* Some readers have wondered whether it is safe to run their trading models on Amazon's EC2. Won't Amazon's employees have access to their wildly profitable strategies? The answer is no: Amazon's security policy:


Guest Operating System: Virtual instances are completely controlled by the customer. Customers have full root access 
or administrative control over accounts, services, and applications. AWS does not have any access rights to customer 
instances and cannot log into the guest OS....

Thanks to a reader OL from France who provided me with this info. He also told me that: 

"So, I finally deployed my momentum strategy on a Linux instance of EC2 (which is free btw).
I wrote it based on the java demo application provided by Interactive Brokers and some parts of Algoquant.
So far, I use a European instance of EC2 which alas doesn't have the best latency to IB US servers (90 ms) but still better than my bedroom connection. 
A test ping from a US instance to IB US servers results in only 15 ms ..."

So there you go: Java+Algoquant+IB+EC2=profit.

64 comments:

olafhel said...

Quantopian seems to be available on invitation for the moment.
Will take a look at it later,
thanks Ernie !!!

Soom alias OL ;o)

Douglas Paul said...

Hi, I would like access to Quantopian to check it out. my email is info @ agarwoodnews .com

Ernie Chan said...

Douglas,
I think you should email Quantopian directly for your request.
Ernie

fawce said...

Ernie, thanks for the post. We are still invitation only, but if your readers email me (fawce) I can send them an invite.

Anonymous said...

Hi fawce,
Could you tell me your email address. Thanks

Ernie Chan said...

The email is fawce @ quantopian.com

Anonymous said...

Thanks Ernie for the email address. Also I have enjoyed your postings. Thanks

Dave newbie said...

I have been thinking about security of strategies on hosted systems for several years. My day job is information security. Virtuallization systems like EC2 are easy for any staff with access to review any data that is on the disk of an EC2 image. It is not likely but it is possible. Plus with the network monitoring equipment it is easy to see which EC2 client images are actively "talking" to IB. Those images would be of most interest to any irresponsible employ. My favorite strategy is to create an encrypted volume on the EC2 server and mount the encrypted volume at boot time. If someone hacks the system they can review the data but if they are browsing data on the disk it will be encyrpted. The encryption key can be setup to be automaticallly applied to the volume mounting process at boot time, but from another system.

Anonymous said...

Dave,
I happend to agree with you. Of course unless you don't care about losing your things.

Anonymous said...

Virtualization cannot be safe. EC2 relies on XEN. You are what is called a dom-U user, and the real hardware is dom-0. Whoever has access to the dom-0 (and certain Amazon employees have) has full access to your memory pages from there. Sorry guys.

Anonymous said...

Please be advised that I emailed fawce@quantopian.com, and now this email account of mine, which I used, has been compromised.

arienai said...

My personal favorite for security is to use
http://www.boxcryptor.com/ as a folder on dropbox.

With dropbox installed on the EC2 machine you can place files via box cryptor to encrypt on a per file basis.

This gives you an encrypted auto-sync to all your EBS volumes but you don't need to encrypt each and every file (i.e. the less important ones) or mount disks

Ernie Chan said...

arienai,
That's a great piece of infomation.
Thanks!
Ernie

Anonymous said...

Not related to this, it's about pairs trading. In your book you look at pairs trading and the timescale can vary.

My question is, given that going short when using a retail broker essentially involves stock lending, would the broker impose a limit on the days the short leg can be guaranteed (I'd guess 3 days).

If so, how hard have you found it in practice to keep the short leg in pairs trades where the convergence can be e.g. 1 month? From your experience, is pairs trading best suited for intraday or 1-2 day strategies?

Ernie Chan said...

Anon,
I often have difficulty locating the short side of a ETF pair on Interactive Brokers. But once located, I never had it called back.

In general, I prefer pairs that revert in under a month. But if you can find intraday reverting pairs, more power to you!

Ernie

fawce said...

Hi everyone,
Just a quick update - thank you to everyone who emailed me, I'm thrilled by the response (thanks again Ernie). I will be replying to each and every request directly, but I wanted to cross-post here. I have a new release nearly ready and I will be sending invites once that release is launched on the site, probably next week. Thank you for your interest and patience!

thanks,
fawce

Nedzad said...

Ernie,

I love your site, but I am not sure why you are having a link to Larry Connor's book called High Probability ETF Trading. His strategies have a lot of practical issues i.e. they are impossible to implement. For example, the author advocates buying an ETF on the market close, but his decision making depends on the daily high, low, and the close price which are not available by the time you are required to submit a MOC order.

For $49.95, these authors are selling scam book!

Ernie Chan said...

Nedzad,

I must disagree with your assessment of Larry Connor's book.

As I emphasized in my own book, most published strategies cannot be traded "as-is". They should be viewed as a inspirational, as prototypes for traders to refine and improve. I personally have found a couple of useful ideas from Connor's book that I traded previously.

Concerning your specific criticism of orders at the close using OHLC as signals, there is no reason you have to implement them as MOC orders. You can easily send market or limit orders a few seconds before the close. The slippage involved is small at least for liquid stocks.

Ernie

Nedzad said...

Point and lesson well taken Ernie.

Ken said...

Hi,


You are talking on Amazon spying other users. It seems more likely and easier for IB to spy what you are doing because the can retrieve your sharpe ratio and trade easily.

Do you know any stories on broker reusing their customer strategies (particularly IB) ?

Is there any safer broker ?

Thanks in advance for your (always) good answers,

YopYop

Ernie Chan said...

Yopyop,
Your point is well-taken.

I have not heard of anyone complaining about IB spying on their customers, but one of my ex-partners is quite wary of a certain brokerage nicknamed the Giant Vampire Squid spying on their customers, as this firm has an enormous prop trading desk.

I encourage any readers who have heard of any rumours on any firms to post their comments here.

Ernie

Ken said...

Ernst,

I d like to add that most of the big trading houses (Vampire Squid, international commodities trader) have their own brokerage.
All the big quant funds have multiple brokers (to split their trades between multiple brokers ?).

YopYop

maninmoon said...

Hi Ernie, I am interested in testing a HF strategy on FX. The data I have comes in every X millisecs. If I assume a latency of Y ms and Y < X. I get 100% fill rate. That can't be right! But my X is large to my known Y. How can I make my backtest more realistic?

Ernie Chan said...

Yopyop,
Yes, splitting trades between brokers is certainly one good way to conceal your true P&L.

Ernie

Ernie Chan said...

Unknown,
When you said your "data", do you have every bid/ask quote as well? Just having transactions data won't guarantee you will be filled.
Ernie

fawce said...

Hi Everyone, www.quantopian.com now accepts requests for alpha access, so you no longer need to email me.

thanks,
fawce

maninmoon said...

Ernie,

I have "snapshot" data - i.e. best bid and offer in x ms timeslice.

Anonymous said...

Hello Earnie,

thanks for your reply on the short leg. For how many days do brokerages (e.g. IB) guarantee to keep a short position once established.

Also regarding the rate they charge for lending the stock, is it the same for all stocks traded in the same exchange or the rate varies per stock.

Finally, is the lending rate guaranteed to stay as it was on the day the trade was booked? I'd imagine e.g. in a severe crash people loosing money from lending and wanting to charge more for it, don't know if they can do it though.

Many thanks

Anonymous said...

in my previous post I meant *Ernie*, apologies for the typo on your name, quite an annoying mistake to make.

Ken said...

Hi everyone,

Because IB lacks a lof of futures/markets I am currently looking for a broker with a strong API and access to all or most of agricultural commodities market.
Any idea ?

Thanks,

Ernie Chan said...

Unknown,
If your strategy submits buy limit orders at price higher than the current ask quote, then assuming that latency has not changed the real ask quote, your order should be filled. There is nothing unrealistic about a 100% fill rate in this situation. If you are concerned that the ask quote actually has changed before the next bar, you can double-check the ask quote for the next bar, and reject the fill if that is now higher than your buy limit price.
Ernie

Ernie Chan said...

Hi Anon,
Brokerages do not guarantee the number of days you can hold a short position before being called. If all their customers who long the stock want to sell it, your short position will be forcibly closed.

The lending rate differs for each stock depending on whether they are hard to locate. The lending rate may also change from day to day, but you should be notified if it changes.

Ernie

Anonymous said...

Hi Ernie,

Your ex-partner has good reasons to worry about Giant Vampire Squid. It's one of our brokers. To attract more commission, it often tips off its best-paying customers about other lowly, dumb money's large order flow. Apparently, it considers letting customer A front run customer B as legitimate business practice. Whether it tips off its prop traders is anyone's guess. People been to the trading floor knows the "Chinese Wall" exists only if you are deaf and blind.

trade futures said...

Sounds like a pretty good robust platform. Hopefully, it would be more popular soon. Is there access to historical futures data?

Zoomly said...

Still can't get an access to Quantopian. Write few letters to them, but there's still silence(

Anonymous said...

Is there a cost breakdown for this platform once it goes live? will users be able to download data locally on top of any "cloud" offerings?

Also Ernie, being the stat arb expert, would you have any papers to recommend on the performance of stat arb with regard to market factors (e.g. market volatility etc).

fawce said...

@zoomly - I was sure I replied to everyone's requests, I am very sorry that I missed yours. If you haven't already, could you please submit your request via http://www.quantopian.com? I'll be sure to respond quickly.

@anonymous - we'll update everyone on the business model and costs when the site goes public. Please stay tuned!

Ernie Chan said...

Anon,
I generally don't find factor model to be profitable. But if you really want to read up on it, try the book "Active Portfolio Management" by Grinold and Kahn.
Ernie

Anonymous said...

How can any email account be compromised by sending email to fawce@quantopian.com?

Anonymous said...

Hello fellow trader. Please visit my blog at http://openbooktrader.wordpress.com. Thanks!

Billy Tsoi said...

Hi Ernie

I have brought your book too. It really opens up my mind given i am really boring to work as IT guy in i.bank.

Anyway, here is my question, if you use matlab to build your ATS, is there any Complex Event Processing feature on Matlab? Is it possible for Matlab to handle event driven strategy ?

Ernie Chan said...

Billy,
You can program some simple events-driven strategy in Matlab, but no, it is not good at efficient processing of complex events. However, google "Matlab CEP" and you may find some useful examples out there.
Ernie

Anonymous said...

Ernie,

Does Matlab have the ability to backtest multiple strategies (intraday momentem, long term trend following, etc) on a portfolio (40 or so) of futures contracts simultaneoulsy and return one set of performance data (sharpe, max drawdown, etc) for all the strategies combined as per Mechanica or Trading Blox? Also, is it difficult to set up futures contract data (contract size, currency conversions, etc) in Matlab?

thx

Ernie Chan said...

Anon,
Matlab is a general-purpose computational platform as opposed to a special-purpose trading software: you can program it to be as complicated or as simple as you want. I personally have used it for everything from backtesting to live execution for many different instruments/markets and brokerages.
Ernie

Will Nelson said...

Ernie,
Is there a place where people have published specific strategies that they used in the past, saying how profitable they were and how long they remained valid, in a way that could be backtested and checked. I guess you could say I'm experiencing a lack of trust regarding the literature I have read, because it's starting to sound a bit like parapsychology ("it worked...I saw it work...but nobody else saw it and in fact it doesn't work if anyone else is watching because the subject gets nervous, or some other reason, etc etc") I too am a former physicist, by the way.

Ernie Chan said...

Will,
I know of a few sites which track people's real investment track records (e.g. Currensee.com, myfxbook.com, strategyxchange.com), but none that require traders to disclose their strategies.

But obviously, if someone has disclosed their strategies to you, you won't need an independent website to verify its performance: you can just repeat the backtest yourself!

The only caveat may be that you think perhaps a recently disclosed strategy is full of "data snooping bias" (i.e. overfitted to past data, or "datamined"). To avoid this, you can look up strategies that have been disclosed publicly a few years ago, and then backtest its performance in the years since publication.

Ernie

Andrew said...

hi ernie,

been a long time fan and have avidly reading your book. I have been trying to move my trading into the quantitative direction. At the moment I am trying to decide b/w Python and matlab for implementing my trades with IB. However, it seems that the free matlab to IB api's that were once available have all been taken down; i was hoping if you could point me in the direction of any that you can think of? I would strongly prefer to stay with matlab as my previous programming has been in this area. thx!

Ernie Chan said...

Hi Andrew,

You can try matlab-trading.blogspot.com or leptokurtosis.com for free Matlab ActiveX API. Also, search for "Interactive Brokers API" on Matlab File Exchange will get you to one too.

Ernie

Anonymous said...

What other options are there besides EC2 is you want faster execution besides colocation.

In terms of colacation are there any services like ec2 specifically for trading?

What are the best colocation options are out there. Last I looked it was around 10k a month but it seemed hard to find information. Also concerned about security. If this was closer to 1-2k I'd be in. I mainly trade futures so looking in chicago area.

Thanks all your responses have been very helpful as people typical ask other questions I have. Great blog!

Ernie Chan said...

Anon,
The best colocation option is one that is offered by your own broker, since your market data and orders all travel within the same local network. Lime brokerage offers colocation, so does FXCM for institutional clients. I have heard rumours that even IB is going to offer this service.

If your broker does not offer this, then Equinix or Telx also offers data centers close to various exchanges and ECN's. They have lots of clients in trading. Monthly rate starts at about $1K.

Other readers have previously suggested slicehost.com and gogrid.com as alternatives to EC2 as well.

Ernie

Andrew said...

hey ernie,

thx for your reply. i checked out matlab-trading.blogspot.com i'm not sure if there's matlab code still available for linking with the IB api? also would you have experience getting activeX working on a linux sys? i'm trying to port this over to a linux os as letting it run on win os hasn't been that stable long term wise. thx!

Ernie Chan said...

Andrew,
Have you tried downloading the file http://quantum.meplaza.nl/ib_matlab_tutorial.zip?

I don't think ActiveX can work in Linux, but maybe other readers who know more about this can comment here?

Ernie

Clark said...

Can anyone point me to publications or share their knowledge on modeling slippage? I am working on equities statarb system (with a few twists) that is trading a small account (less than $1MM). Backtesting performance is good, live performance is good, but I need to get a better grasp on slippage to know if it can scale. It trades daily (not intra-day). I have run some live tests on Interactive Brokers proprietary order execution algos and tested Interactive Broker's OPG orders (Open Price Guarantee) execution, but my test results are inconclusive. My tests would indicate that slippage is 0 to slightly positive, but that can't be right even at the current small size of trades. I would greatly appreciate any advice, and thanks in advance for any pointers.

Regards,
Clark

Ernie Chan said...

Clark,
My question to you is: slippage relative to what?

If you are sending OPG orders, your execution price will be exactly equal to the official exchange open price.

So do you mean slippage relative to the mid point just before/after the open?

Ernie

Clark said...

Ernie,

"Relative to what" is exactly what I am struggling with. In my test trades, about 75% of my trades receive the "official open" the other 25% are higher and lower in approximately equal proportions. I guess OPG is not really OPG.

If my trade volumes increase, it would impact what the "official open" is, so I am trying to estimate this. It is likely that I need to go read some more research papers that describe modeling slippage, so if you can recommend those I would greatly appreciate it.

Additionally, if you have any ideas regarding specific Interactive Brokers order algos, I am all ears.

Regards,
Clark

Ernie Chan said...

Clark,
If some of your OPG orders did not receive the official open prices, this is a major bug/issue that you need to work with IB to resolve.

On the other hand, if your order size is a substantial portion of the opening print, it will definitely affect the open price itself (independently of the bug above). It should not really be called "slippage" in this case. It should be called "market impact". I would recommend you set your order size to be no more than 5% of the opening print. The size of the opening trade can be found either on Bloomberg, or by looking at the volume immediately after the open.

Ernie

Clark said...

Thanks for the feedback. I have an open call in with IB about the OPG issue. No answer yet...

I will research your open print volume input/idea further.

Thanks again.

Clark

reptile said...

Sooo.. what is the best platform for algo ? :)

Ernie Chan said...

reptile,
There is no "best" platform, since it all depends on your own skills. For me, Matlab is the language/platform of choice. But once I master Python, I may try one of those open source IDE such as Marketcetera or Tradelink because of their FIX connectivity.
Ernie

Anonymous said...

I'm running an automated trading strategy currently from home. My setup uses my broker's Windows-based trading platform and I've written the automation using my brokers API. I want to reduce some of the technological risks such as internet connection or power going down. I am US based.

One option is colocation at my brokers datacenter but that is more expensive than I'd like at this stage. Another is a dedicated server at a generic datacenter but that is not much cheaper and doesn't offer the latency advantages.

Do you think using Amazon EC2 is a reasonable replacement for a home desktop computer-based infrastructure?

What issues are there with EC2? I'm not familiar with cloud services but do you have to worry about the system slowing down for a few seconds because someone else is doing heavy CPU/IO/networking? The system pausing for a few seconds could be bad.

Do you find the connection from EC2 to broker typically better than connection from home to broker?

Do you have to worry about the instance being shutdown while trading?

Overall am I stepping up my infrastructure by moving from home computer to EC2? Or am I just swapping one set of issues for another?

What is a ballpark monthly cost for an EC2 instance that could replace a home trading system?

Ernie Chan said...

Yes, EC2 is a reasonable choice, and is a step-up from home computer. It doesn't do much to reduce your latency to your broker, but of course it is much more stable than home computer.

I haven't experienced any issues with cloud services in general, such as the ones your mentioned. In particular, I never had an instance got shut down.

EC2 is not the only cloud service. If you google VPS, there are many providers which will provide you with a permanently running service (no starting or killing instances).

The typical monthly cost is $100-$400 depending on if you are using CPU at all times.

Ernie

Anonymous said...

Thanks for your reply, why did you choose Amazon EC2 over traditional VPS?

Also what type of instance do you run http://aws.amazon.com/ec2/instance-types/ ?

Ernie Chan said...

Anon,
I used to use Amazon EC2 because I only run strategies during the market hours and I can save on some CPU time at night. But since I now run strategies round the clock, I switch to a VPS, which is more convenient.

I used a M1 large Windows instance when I was with EC2.
Ernie

Anonymous said...

Which VPS hosting company do you use?