Saturday, May 25, 2013

My new book on Algorithmic Trading is out

A reader (Hat tip: Ken) told me that my new book Algorithmic Trading: Winning Strategies and Their Rationale is now available for purchase at Amazon.com. The difference with my previous book? A lot more sample strategies with an emphasis on their "rationale", and more advanced techniques. It covers stocks, futures, and FX. A big thank-you to my editors, reviewers, and you, the reader, for your on-going support.

And when you are done with it, please post a review on Amazon whether you like it or hate it!

Also, I am now offering a live online course on Backtesting in June. It covers in excruciating details the various nuances of conducting a correct backtest and the numerous pitfalls one can encounter when backtesting different types of strategies and asset classes. For syllabus and registration details, please visit my website.



260 comments:

«Oldest   ‹Older   201 – 260 of 260
Anonymous said...

I thought I'd share my latest findings. iQFeed seems to be a good alternative to e.g. kibot.com. Not very expensive and they also have support for Matlab. http://www.iqfeed.net/matlab/index.cfm

Anonymous said...

Hi Ernie,

Thanks for writing this book, I think it is a very nice exposition of practical applications. Very few books in this area have decent examples, so thank you for that. I was reading through your Kalman filter example on page 80, and I wondered if you could answer a quick question on the entry/exit signals:

I understand the idea is to buy the spread (long EWC, short EWA) when:
longsEntry=e < -sqrt(Q). This is intuitive. However, the exit is written as:
longsExit=e > -sqrt(Q).

Why wouldn't you prefer to wait until the error bounced much further in the opposite direction, so that it is instead greater than the positive variance, i.e.:
longsExit=e > sqrt(Q) ?

The impressive plot of cumulative returns tells me that there is no mistake, but I was hoping you could please explain this part in a bit more detail?

Thanks again for writing a great book.

anonymous_TX

Ernie Chan said...

Hi anon_TX,
You are right. We can set longsExit=e >=0 or >= sqrt(Q), and vice versa for shortsExit.

I also changed that in my recent talk on this topic.

Thanks for the heads-up.
Ernie

Anonymous said...

Hi Ernie,

Thanks for a very good book.

What's your experience with short selling ETFs with IB? For example the ones you mention in the book like EWC, IGE and EWA. What's the cost of holding a short position for a few days? Have you always been able to short these?

Many thanks

Ernie Chan said...

Hi Anon,
IB does not generally charge borrow fee for most ETFs. However, they may not be available for borrow at all times. I had no problem shorting EWC and EWA, but have not tried IGE recently.

All stock loan availability and costs are visible here: https://www.interactivebrokers.com/en/index.php?f=shortableStocks&p=overview

Ernie

Anonymous said...

Hi Ernie,

In calculating the returns from a short sale, I see lots of people multiplying the daily returns by -1, but I've come to think that this isn't the right thing to do. I was wondering if you had any advice on the proper way to do this in a backtest (short of simulating actual transactions).

Thanks,
Frank

Ernie Chan said...

Hi Frank,
Whether a position (measured in $) is long or short, the P&L of that position is position*(cl-prevCl)/prevCl, where cl=close price of bar, prevCl=close price of previous bar, and the return is just this P&L divided by abs(position).

Ernie

Anonymous said...

Thanks for the quick reply. I see where I tripped myself up -- multiplying arithmetic returns by -1 does work, but multiplying log differences (which I use mostly interchangeably) by -1 does not.

Thanks again

Ernie Chan said...

DKNY wrote:
"Hi Ernie

Would you be able by any chance to update the data on the web to see how the strategy for the last year? I guess it would be beneficial for many readers. For example I dont have an access to futures market data and I cant update the table myself.
Thank you.
DKNY "

DKNY: Do you mean the data I uploaded to epchan.com/book2?
I am afraid that I cannot update those data due to licensing issues. A free source of futures data is http://www.tradingblox.com/tblox/?page_id=218

Ernie

Joey Zhao said...

Hi Ernest,

I bought your book and it is quite helpful with my research program. But I have a question about replicating your example to get post-earnings announcement logic matrix because I found that earnings.com is redirected to Reuters and web has been changed. Do you have any update about getting these data? I would greatly appreciate your help if you can provide some solutions for this.

Ernie Chan said...

Hi Joey,
You can get earnings data from
http://biz.yahoo.com/research/earncal/today.html

Ernie

Anonymous said...

Hi Ernie,

I am reading your new book recently.

I read Montana paper you mention in the book.

It seems in Montana paper, they use "log-return" instead of "log-price" in their S&P 500 Futures Index example.


What do you think about this?

Usually log-return itself is already stationary.

Thank you.


Anonymous said...

Hi Ernie,

In Montana paper you mention,
it seems they use "log-return" instead of "log-price" in their S&P 500 Futures Index example.

Is that ok to do long/short trading this way?

Thank you.

Ernie Chan said...

Hi Anon,
What Montana et al meant by "log-return" is the difference of log prices, hence it is the same as what we all understand as ordinary returns.
Ernie

Anonymous said...

Hi Ernie,

What I mean is that we usually use price (log-price) to test cointegration, that is,

logF = a + logS + e

However, in Montana, at pages 2826-2827, they use return (log-return) to create spread, that is

b(t) = logh(t) - logh(t-1)
r(t) = logP(t) -logP(t-1)

and

b(t) = c + r(t) + w

which is weird because usually return series is already stationary itself.

Ernie Chan said...

Hi Anon,
This issue was addressed in Chapter 3 of my book "Algorithmic Trading". If one uses log price to form a spread, that means one needs to constantly rebalance the market values of both legs to maintain a constant hedge ratio. In any case, I hope to publish a blog post on this topic with a newer way to look at this soon.

Ernie

Anonymous said...

Hi Ernie,

In your new book,

positions=repmat(numUnits, [1 size(y2, 2)]).*[-beta(1, :)' ones(size(beta(1, :)'))].*y2;

According to positions calculation above, it implies that we adjust our positions every time when hedge ratios change, right?

Also, it seems if half-life is short (like 10 days), Kalman filter generates long/short signals almost every day?

Thanks

Unknown said...

Hi Ernie,
I run Johansen test for a pair of stock in my country(Thailand). The result shows that I can reject the null hypothesis that r<=0 at over 99% level but can't reject r<=1. What is this mean? Can I conclude that there are two cointegrating relationships between this pair? I also test this portfolio with linear strategy and get APR = 26.12% with sharpe ratio 2.31.
Thank you very much

Anonymous said...

Hi Ernie,

In pairs trading, after sending limit orders for two legs, if only one leg is fulfilled (usually, how long do you wait before taking action?), what do we do?

It seems we have 3 choices:

1. leave it alone, so it is not dollar neutral.

2. unwind the fulfilled leg, and may lose money.

3. send another market order for the unfulfilled leg, and may lose some money too.

Usually, what do you do?

Ernie Chan said...

Hi Anon,
Yes, we have to adjust the portfolio every time the hedge ratio changes.

Certainly if the halflife is short, there will be more frequent trading.

Ernie

Ernie Chan said...

Hi piranut,
Rejecting null hypothesis for r<=0 but not r<=1 means that you have one and only 1 coint relationship.

Congrats for finding a great stock pair!
Ernie

Ernie Chan said...

Hi Anon,
If you send limit orders for both legs and only 1 is filled, you should send MKT order for the unfilled sides immediately.
Ernie

Anonymous said...

Hi Ernie,

Have you tried to use Kalman filter and Bollinger Bands together?

Ernie Chan said...

Hi Anon,
We use Kalman filter to find the hedge ratio, and the entry and exit thresholds based on the expected mean and standard deviation of the spread because we may believe those values are more accurate than just using moving averages and standard deviations. So there is no reason to revert to these "Bollinger band" values when using Kalman filter.
Ernie

Anonymous said...

Hi Ernie,

When we want to execute a stock pair, we send a limit order for one leg first, and then send a market order for the other leg.

I wonder when is the best timing to do that. One hour before the market close (4 pm), or 10 minutes before 4 pm, or half hour, or other choice.

It seems that we may need some time for the limit order to be fulfilled.

Also, when we want to unwind a pair, what do we do? Just send 2 market orders or 2 limit orders?

Ernie Chan said...

Hi Anon,
An entry into a pair should be decided by the spread. If you wait half an hour for the price of one leg to come to you, the spread value may no longer be there.

The limit price should be aggressive, i.e. equal to the ask price if you are selling, to ensure execution within a minute or so.

The same goes for the exit trade.

Ernie

JackD said...

Hi Ernie, I am experimenting with some of the strategies in your new (and BTW excllent) book and have questions regarding the matlab code implementation dealing with cross sectional intraday Long-Short model- andrew_lo.m. The dataset you are loading is InputDataOHLCDaily_20120424.mat.

In your matlab code you adjust for transaction costs @ .0005 one-way as follows:

"dailyretMinuscost = dailyret - smartsum(abs(weights./cl-backshift(1,weights)./backshift(1,cl)).* backshift(1,cl),2).*onewaycost./smartsum(abs(weights),2); %transaction costs are only incurred when the weights change."

First, would I be correct if I assume that the last term i.e. smartsum(abs(weights),2) simply equals 1.0 since the weights were normalized previously when you divided weights by expression smartsum(abs(weights),2) earlier in your code.

Second, you seem to be differencing the weights from one day to the next, but I don't see where the onewaycost factor is applied twice when using the cl and op version of the intraday model. In that model you take a position near the open and close the position at near the close.

Third, I think it would be very instructive for your readers, if you could provide more algebraic formulas that you are implementing in your matlab code to help in understanding it. Currently, the reader has to decipher the matlab code to get the formulas.

Forth and and most important, Thanks and I look forward to your next book.

Jack D

JackD said...

Hi Ernie, I am experimenting with some of the strategies in your new (and BTW excllent) book and have questions regarding the matlab code implementation dealing with cross sectional intraday Long-Short model- andrew_lo.m. The dataset you are loading is InputDataOHLCDaily_20120424.mat.

In your matlab code you adjust for transaction costs @ .0005 one-way as follows:

"dailyretMinuscost = dailyret - smartsum(abs(weights./cl-backshift(1,weights)./backshift(1,cl)).* backshift(1,cl),2).*onewaycost./smartsum(abs(weights),2); %transaction costs are only incurred when the weights change."

First, would I be correct if I assume that the last term i.e. smartsum(abs(weights),2) simply equals 1.0 since the weights were normalized previously when you divided weights by expression smartsum(abs(weights),2) earlier in your code.

Second, you seem to be differencing the weights from one day to the next, but I don't see where the onewaycost factor is applied twice when using the cl and op version of the intraday model. In that model you take a position near the open and close the position at near the close.

Third, I think it would be very instructive for your readers, if you could provide more algebraic formulas that you are implementing in your matlab code to help in understanding it. Currently, the reader has to decipher the matlab code to get the formulas.

Forth and and most important, Thanks and I look forward to your next book.

Jack D

Ernie Chan said...

Hi JackD,
Thanks for your compliments!

1) Yes, the sum of absolute value of weights is 1 if we have normalized them first.

2) I believe you are confusing 2 models that I described. The linear long-short model described in Example 4.3 and contained in andrewlo_2007_2012.m is not an intraday model. It holds positions overnight, and thus we do have to difference the weights between today and yesterday. We do not trade at the open in this model. Only the buy-on-gap model described in Example 4.1 enters at the open and exits at the close. If you want to change Example 4.3 to an intraday model, then indeed the transaction cost will be always 10 bps each day, and there would be no need for diffencing.

3) The only important formula for this strategy is equation 4.1.

Ernie

Unknown said...

Hi Ernie,

I am a huge fan of your work, got hooked by your first book and currently reading your second.
I have a question in regards to the code of the short-on-gap model.
I do not obtain the same equity curve than figure 4.2 when changing bog.m to a short version.
Please find below bog.m adapted to a short version.
The 5 lines where changes have been input are market with a "%".
Am I missing out on something ? Thank you very much.
Sebastian

clear;
topN=10;
entryZscore=1;
lookback=20;
load('inputDataOHLCDaily_stocks_20120424', 'stocks', 'tday', 'op', 'hi', 'lo', 'cl');
stdretC2C90d=backshift(1, smartMovingStd(calculateReturns(cl, 1), 90));

sellPrice=backshift(1, hi).*(1+entryZscore*stdretC2C90d); %
retGap=(op-backshift(1, hi))./backshift(1, hi); %

pnl=zeros(length(tday), 1);
positionTable=zeros(size(cl));
ma=backshift(1, smartMovingAvg(cl, lookback));
for t=2:size(cl, 1)

hasData=find(isfinite(retGap(t, :)) & op(t, :) > sellPrice(t, :) & op(t, :) < ma(t, :)); %
[foo idxSort]=sort(retGap(t, hasData), 'descend'); %

positionTable(t, hasData(idxSort(1:min(topN, length(idxSort)))))=1;
end

retO2C=(op-cl)./op; %

pnl=smartsum(positionTable.*(retO2C), 2);
ret=pnl/topN;
ret(isnan(ret))=0;
fprintf(1, '%i - %i\n', tday(1), tday(end));
fprintf(1, 'APR=%10.4f\n', prod(1+ret).^(252/length(ret))-1);
fprintf(1, 'Sharpe=%4.2f\n', mean(ret)*sqrt(252)/std(ret));
cumret=cumprod(1+ret)-1;
plot(cumret);

Ernie Chan said...

Hi Sebastian,
Your codes look right, but I would make it clearer by setting positionTable to -1, and leaving retO2C to (cl-op)./op.
Ernie

Frank said...

Hi Ernie,

Thanks for your book. It is a really great writing.

I am working through some of you examples and I have a few questions concerning Example 5.4 mean revision trading of calendar spreads.

1) On p123 you say

"We run the ADF test for 12-month log calendar spread of CL, and discovered that it is indeed stationary with 99 percent probability, and a half-life of 36 days."

but then in the code you run the adf test on the roll return, gamma, rather then the calendar spread itself (something that you do later for VX). When I do that, that is, run the Dickey-Fuller test on gamma, I get 91% probability to reject the null hypothesis that the gamma time series is not stationary. Which is still kind of ok, but definitely not 99%. Am I doing something wrong?

2) I am re-writing your code in R since I don't have access to Matlab right now, thought, I used to use it a lot earlier. R is cheaper, but it takes thinking to translate your code, which may be a good thing. It seems like you use backshift(.,1), lag(.) and lag(.,1), which, as far as I understand, the same, at least implying from what you mean to do in the code.

Anyway, I am able to reproduce your graph of gamma, lookback=36 days, but I can't reproduce the cumulative returns graph. For me the code does not quite match the 3-point criteria. 1) Holding period is 61 days, while holddays=3*61 in the code. And in the code the holding period is not either of these but from 10 days before expiry of future #1 to 10 days before expiry of the appropriate next future (2 or 3 months later expiry future) in the for loop setting up the positions matrix: "myEndIdx-myStartIdx >= holddays". So you are holding a future longer than 3*21 days and it can be much longer. 2) Then you introduce the mean-reverting strategy: "positions(zScore > 0, :)=-positions(zScore > 0, :);", so you are not actually holding the future for 63 days or so but take long or short position of the spread every day. I am confused here.

3) On a less technical note, I don't completely get it why the strategy is based on gamma's mean reversion. If the futures is in backwardation, the long calendar spread has positive roll return, while this strategy shorts it when the deviation from gamma is positive, that is, when it is in an even stronger backwardation. Same for contango, and negative deviation from gamma force long position. So I am confused by this paradox.

Thanks for your clarifications in advance. I am sort of new to algo trading.

Frank

Ernie Chan said...

Frank,
1) You are correct to run the adf test on the calendar spread itself. I was just taking a shortcut.

2) Where I wrote "The holding period for a pair of contracts is 3 months (61 trading days)." I should have wrote "The maximum holding period..." Indeed we can hold as long as the rolling date for the front contract, but that won't be very much longer than 3 months.

And yes, we don't actually hold it that long: it is just a programming trick. We will exit when the Bollinger band signal tells us to exit.

3) For a calendar spread, the first order effect of roll returns is cancelled out because of the long-short positions. We need to look at the second order effect of the change in roll returns in order to generate total returns.

Ernie

Anonymous said...

Hi Ernie,

where is your data file "inputData_USDCAD" ... definitely not on your site.

Having bought the book I would expect ALL the data to be available

Thx

Ernie Chan said...

Hi Anon,
As I wrote in these comments previously, those files are too large to be uploaded to my website. Interested readers are welcome to email me privately for a separate download link.
Ernie

Unknown said...

Hi Ernie,

i recently read your books

in chapter 3, when calculating the price spread, you recalculate the hedge ratio every day using a short look-back period.

so, should i rebalancing the portfolio everyday with changin beta?

Ernie Chan said...

Hi Ilseo,
Yes, you should rebalance portfolio everyday.
Ernie

cheerful said...

Dr Ernie,

Would give a short data on USDCAD? I tired to use ratedat@GainCapital but run into simulation error at load(xxx) and y=cl(hhmm=1659). Variable tday not found, undefined function or variable 'hhmm'

Ernie Chan said...

Hi cheerful,
Please email me privately if you would like a link to the USDCAD data.
Ernie

Unknown said...

Hi Ernie,

Nice book and great contents. However, I cant figure out the password for your site from example 1. May I know the username/password of your site pls?

Thanks,
KL

Ernie Chan said...

Hi Kinson,
Thank you for your kind words.
The userid/password are on page. 199. If you can't find it, please email me.
Ernie

Unknown said...

Hi Ernie,
The book is great. And I have questions about mean reversion.
In the Calender spread mean reverting chapter, you used roll return for signals for trading. Isn't that the strategy a momentum one? I just don't understand why in a contango situaton shorting futures is momentum. Since the Futures price is higher than the spot. Isn't that a mean reversion in the price? I understand the strategy and I agree with the strategy. However this conceptual problem has been with me for a long time.

Unknown said...

Hi Ernie,
while in a contango situation, why is shorting a future a momentum strategy? Since the price is higher than the spot. isn't it a mean reverting strategy? And in the Calender spread mean reverting strategy you use z>0 for signal. Isn't this a momentum one? Thank You
And it is a really great book.

Unknown said...

Hi Ernie,
I just do not understand why shorting a future in a contango situation is a momentum strategy since the price is higher than the spot and we are waiting for it to revert to the spot. And in the Mean Reverting calender spread chapter. It seems to me that using gamma as a signal is a momentum strategy.
And this is a great book. Thank You.

Ernie Chan said...

Hi Zhou,
If the roll return is negative (i.e. contango), we short the future to profit. Hence we are shorting in the same direction as the existing return. How is that not momentum?

Also, I now disagree with my own writing on calendar spreads. The only way we can profit from trading it is if the forward curve has non-zero curvature. Just having non-zero roll returns will not be enough: the roll returns must differ at different maturities.

Ernie

Sqrt Alpha said...

Hi Ernie,

I found some guys discussed on window length when computing hurst exponent. Some say the value makes sense strongly. Does it mean the max value of $\tau$ chose when calculating variance?

In plus, is just hurst exponent enough to judge MR or Momentum? I found that you mentioned statistical significance of the exponent. It hints that we should do variance ratio test, not just look at the exponent value, correct?

Sqrt Alpha said...

Hi Ernie,

I found some guys discussed on window length when computing hurst exponent. Some say the value makes sense strongly. Does it mean the max value of $\tau$ chose when calculating variance?

In plus, is just hurst exponent enough to judge MR or Momentum? I found that you mentioned statistical significance of the exponent. It hints that we should do variance ratio test, not just look at the exponent value, correct?

Ernie Chan said...

Hi Sqrt Alpha,
Yes, variance ratio test is a good significance test for the Hurst exponent. We should compute that in addition.
Ernie

Sqrt Alpha said...

Thanks for your friendliness, Ernie.

What do you recommend about scaling window?, please kindly refer to the link
https://www.researchgate.net/post/Can_anybody_help_with_Hurst-Exponent_algorithms_and_scaling_window

Sqrt Alpha said...

Thanks for your friendliness, Ernie.

What do you recommend about scaling window?, please kindly refer to the link
https://www.researchgate.net/post/Can_anybody_help_with_Hurst-Exponent_algorithms_and_scaling_window

Ernie Chan said...

Hi Sqrt Alpha,
If by "scaling window" you meant the sampling frequency one should use for computing Hurst exponent, please see my recent article http://epchan.blogspot.com/2016/04/mean-reversion-momentum-and-volatility.html.

Ernie

Unknown said...

Ernie!!

Not sure if this comment will be seen! But I've got a question for you about a set of examples in your book. In "Algorithmic Trading" example 2.8, the cointegration strategy between EWA, EWC, and IGE, you compute the sharpe ratio of the strategy = 1.4. Is this the annualized sharpe? Or is it the sharpe for the entire time period of the strategy? Because I get 1.4 when I compute it for the entire strategy period, but I get 0.55 for the annual period.

Similarly, in example 3.1 with the spread strategy, you compute the sharpe as 0.59, which seems to be the annualized sharpe. Because when I do the backtest, I get 1.3 for annual sharpe, and 0.55 for the entire period.

So the main question is: are you using annual sharpe to evaluate these strategies? Or are you using a different time period. Or is there something wrong in my calculations?

Thanks a lot! Super glad I found your books.
Regards,

--Brandon

Ernie Chan said...

Hi Brandon,
Thanks for your kind words.

All my Sharpe ratio calculations are annualized, based on the entire period. I don't quite understand what you meant by getting 0.55 for an annual period. Which annual period? And why only that annual period?

Ernie

Unknown said...

Hi Ernie,

I come back to you to seek help for a technical problem. Many times in your Matlab codes and on your website, you use a dataset called USD_CAD. However, this dataset can't be found on your website. Could you please upload it or provide a way to find it?

Thanks in advance,

Best regards,

Alberto

Ernie Chan said...

Hi Alberto,
Please email me for that data.
Ernie

Sulfred said...

Hi Ernie,

I find the information about the website from the book on page 199 but I found that only page 1 can be viewed and downloaded. Is there a way to get the contents of page 2 and 3.

Thank you very much.

Best regards,
Sulfred

Ernie Chan said...

Hi Sulfred,
Please press Next (at top right corner) if you want to go to the next page on epchan.com/book2.
Ernie

Sulfred said...

Hi Ernie,

Thank you so much for your reply. I am sorry about that I didn't make myself clear.

I have pressed Next at the top right corner but it doesn't work. Seems the link is missing.

Thank you very much.

Best regards,
Sulfred

Ernie Chan said...

Hi Sulfred,
I tried pressing Next on two different servers, and it advances to next page without problem. Are you using Chrome?
Ernie

Sulfred said...

Hi Ernie,

Yes you are right. I am using Chrome. I download all the files through IE successfully.
Thank you very much.

Sulfred

«Oldest ‹Older   201 – 260 of 260   Newer› Newest»