The story is now familiar: Zillow Group built a home price prediction system based on AI in order to become a market-maker in the housing industry. As a market maker, the goal is simply to buy low and sell high, quickly, and with minimal transaction cost. Backtests showed that its AI model's predictive accuracy was over 96% (Hat tip: Peter U., for that article). In reality, though, it lost half a billion dollars.
This is a cautionary tale for anyone using AI to predict prices or returns, including those of us in more liquid markets than housing. Despite Zillow’s failure, the root cause of this discrepancy between backtest and live market-making is well-known, and it has nothing to do with machine learning or AI. Their failure was due to adverse selection, which can happen to any market maker, whether human or machine. In this context, "market maker" is used in a broad sense - a market maker provides liquidity to the market using limit orders. For instance, any mean-reversion trader is a market maker. As long as the market maker is trading against a counterparty who has more information (a.k.a. the "informed trader"), adverse selection will take money away from the market maker and give it to the informed trader. This is because as market makers, the only model is to buy when prices are cheap, no matter why they are cheap. In contrast, the informed traders may know why the asset is cheap and if it will get cheaper, so they are happy to sell to a market maker. In the opposite situation, if the informed traders believe that the current prices are cheap, but will get higher, they will refrain from selling. In this case, the limit order will not get executed, and market makers suffer from "opportunity cost". In Zillow’s case, the informed traders are the homeowners who have a better understanding of the value of their own home due to qualitative factors (e.g. views, interior design, neighborhood safety, etc.) outside of Zillow’s model.
In my book Machine Trading, I wrote, "Adverse selection happens when prices on average go down after we buy something, and go up when we sell something". Therefore, adverse selection can be measured quite easily by computing the difference between the (paper) P&L of unfilled orders and the P&L of filled orders over a short time frame. In order to determine whether your AI predictive model will work in reality, it is ideal to deploy it live in a small capacity, and measure the differences over time. If there is significant adverse selection, the trader can always choose not to participate in the market. For example, it is legendary that high frequency traders stopped providing liquidity to the market during extreme events such as flash crashes. Traders don't want to be the suckers at the game. Unfortunately for Zillow, they weren’t aware of the well-practiced art of market making.
Another common way to reduce adverse selection is to keep a close tab on your inventory. If, in a short period of time, inventory suddenly changes significantly compared to average trends, it may indicate that there is new information arriving on the market that you are not aware of (e.g. mortgage rate going up by 1%). In this situation, it would be wise to cancel your limit orders until the coast clears. For a mathematical interpretation of this concept, view the formulation by Avellaneda and Sasha. Inventory management was a key technique that Zillow did not adopt, which could have minimized their adverse selection risk.
AI has been a major asset in numerous business processes, including market making, but it is just one part of complex production machinery. As we can see from Zillow’s use case, predictions, even accurate ones, are not enough to generate profits. As I explained in my previous blog post, we at Predictnow.ai don't think that AI is the be-all and end-all of decision making. Instead, we believe the value of AI lies in its ability to correct human-made decisions. But, an even larger lesson here is that experts in one industry (e.g. housing) can benefit from the knowledge of experts in another industry (e.g. quantitative finance). This transdisciplinary knowledge is exactly what Predictnow.ai offers enterprises to improve and enhance their processes.
2 comments:
How would you calculate the "P&L of unfilled orders"? So what would be the time stamp of opening and closing the position?
I was thinking, one possible way to simulate paper P&L corresponding to an unfilled limit order would be, to open a synthetic position at a time when the limit order was sent (opening time stamp) and close the position when the limit order was cancelled (closing time stamp). A drawback though is that this depends on your logic to cancel limit orders. Any suggestions?
P&L of an unfilled order is to assume that it were filled (at the limit price) and compute the unrealized P&L over some period based on your exit logic, or based on various time horizons.
Post a Comment