As an algorithmic trader, I am constantly in search of a better physical infrastructure where I can connect via the internet to my execution broker at the highest speed and with the least possibility of outage, and at a reasonable cost.
To that end, I would like to mention Fios, a fiber-optics service from Verizon with download speed of 50 Mpbs, upload speed is 20 Mbps, both faster than your typical T-1 line (1.5 Mbps). Furthermore, it costs only $45/month. Hey, even Paul Krugman has installed it at his home!
(I haven't tried it myself, and would like to hear from those of you who have and see if it is time to say goodbye to T-1.)
And as I have reported earlier, I am also constantly looking for a good cloud computing platform so that I can run more strategies without cluttering my office with computers. Finding one will obviate the need for any big investment in internet connectivity at the office.
To that end, I have been trying out Amazon's EC2 for several months. I use it to run one of our strateiges, and I have to report that my experience is mixed.
Firstly, if you are not an IT person, it does take a lot of time (8 person-hours?) to get set up and running, especially with their securities precautions. The learning curve is steep.
Secondly, and more annoyingly, the instances sometimes fail to start properly, or fail to bundle properly. (Bundling means saving the software configuration for future use.) I am using Windows instances. Maybe those who use Linux instances have better experiences?
Thirdly, and most annoyingly, when a new instance is started, Windows often cannot automatically synchronize its clock with time.windows.com or any other internet clock. As a result, the time is often wrong. Now, this may not be a big deal for usual office work. But when your automated trading strategy depends crucially on the time of the day, it can be quite fatal to your profit. If anyone has experienced a similar problem with Window's clock and know a fix, please let me know!
Despite all these hassles, I am still running strategies on EC2, hoping that once EC2 get past the beta release, things will be better.
I would love to try quantitative trading one day
ReplyDeleteThe linux ec2 rocks. Never tried the windows version but maybe that's what's causing difficulty...
ReplyDeleteFIOS is excellent. I have been using it for a couple years. Never down and always at top speed.
ReplyDeleteEC2 is mainly designed for distributed computing. What you need is a simpler virtual private server (VPS). For example http://www.slicehost.com/. Depending on your profit margins it you might even want your own colocated server. Our broker allows us to place our servers in their internal network with a direct fiber connection to the exchange. We even get to connect to the exchange directly and only clear the trades with our broker. In US there is Lime Brokerage offering DMA with colocation at Wall Street for about $200/month and about $200 additionaly per month for direct market feed for most securities.
ReplyDeleteThanks for all the feedback!
ReplyDeleteerkki: I did check out slicehost.com and Lime Brokerage: both look very good. We are going to try them out at some point.
Hello Ernie,
ReplyDeleteSometime when a Windows server is not a member of Active Directory domain, time synchronization can be a bit tricky. I would recommend Microsoft's article http://support.microsoft.com/kb/307897.
Btw I loved your book; it is excellent.
Regards
i share your concerns on AWS; awesome idea, but implemention needs a bit of work.
ReplyDeletefor time synchronization, you shouldnt have any problems finding freeware apps that synchronize time at preset intervals, i used one in my nt4 days.
Hi Anonymous,
ReplyDeleteYour advice about time synchronization is very helpful -- thanks! Will try it out when I next have that problem.
Ernie
Hi Ernie,
ReplyDeleteI also use EC2, but only for back-testing... I'm "stuck" on a back-testing platform that doesn't scale well, so the solution is to run multiple boxes.
For my actual automated trading, I also use a remote hosted service. I'm a little paranoid about using VPS... I like the idea of having my own physical hardware. This also makes it unnecessary to spend money on building a big network pipe going to my house... a big network pipe that can still go away, if my neighbor has bad aim with his weed-whacker!
Hi cctang,
ReplyDeleteI have fixed most of the problems running automated trading programs on EC2 (e.g. using http://www.sharewareconnection.com/small-sntp-agent.htm to automatically sync the clock on startup.) So far, running on EC2 beats running on my own desktop because of the stability and reliable connection, and also I can run as many strategies as I like without fuss.
Ernie
I use the following little program for clock synchronization and it seems to work really well:
ReplyDeletehttp://www.worldtimeserver.com/atomic-clock/
Ernie - I found an app called Paglo (http://www.paglo.com/opensource/paglocrawler) that uses Amazon's Cloud Watch to monitor EC2 servers and displays the data in an easy to use format. I am going to be using this to make sure everything is working as I expect it to with my machine & make sure the machine is not overloaded.
ReplyDeleteI've been pleased with the setup process for EC2 so far.
Eric
Eric,
ReplyDeleteThanks for the post. Sounds like a great resource!
Ernie
If you're looking for a different cloud that has a stronger 100% Uptime, 10,000% Guaranteed SLA, check out gogrid.com. I first heard about them at the google IO conference a few months ago, supposedly they are very good.
ReplyDeleteJason,
ReplyDeletegogrid.com looks like a good cloud platform. Thanks for the suggestion.
Ernie
Yes i agree Verizon FiOS is really fast. I signed up for FiOS Internet and part of FiOS deals got $150 back
ReplyDeleteHi Ernie. I wondered if you could give us an update on how things are going with Amazon's Cloud. I have been considering migrating my trading to the cloud. I trade a portfolio of pairs automated using MS Excel and VBA. I also run the Redi platform.
ReplyDeleteBtw, my Xmas present to myself in '08 was your book, and I thoroughly enjoyed it! Your blog is also my favorite trading blog.
Steve
Hi Steve,
ReplyDeleteI continue to use EC2 for some of my trading strategies, and have no problems so far! I do recommend it.
Thanks for your kind words on my book and blog.
Ernie
perhaps we can help each other out. I have developed a data warehouse solution that lives in the Amazon EC2 cloud. I think it would be perfect for quants. I can take care of all the data connections, instancing, security, provisioning, etc. You can do all the black box stuff you want. I just want to own the process of creating the warehouse.
ReplyDeleteJust started experimenting using an EC2 Linux instance two days ago. EC2 will be a critical for me, as I am moving to a location with horrible internet providers.
ReplyDeleteI am able to configure an instance manually in less than ten minutes, thanks to the wonderful Linux 'yum' command and an initial learning curve. I'm sure I'll be able to automate it. I haven't yet learned how to use the S3 non-volatile storage yet, nor to bundle an AMI yet.
I think EC2 is great. I still have more to learn.
These are the two web pages that got things going for me after having an unused EC2 account for over a year.
http://www.robertsosinski.com/2008/01/26/starting-amazon-ec2-with-mac-os-x/
http://blogs.deepal.org/2009/03/amazon-ec2-building-apache-web-server.html
Hi Ernie was thinking of setting up servers on aws.amazon following udemy course Forex Trading Secrets of the Pros With Amazon's AWS, but have read some mixed feedback of aws ec2. I wonder how you are doing with it now, and if you have any other suggestions? Regards Catherine
ReplyDeleteHi Catherine,
ReplyDeleteYou should use EC2 if you can tolerate about 20 ms latency in connection to your broker. Otherwise, I suggest using a VPS located near your broker's server.
Ernie