An interesting feature emerged from this extended analysis. CL and XLE are still found to be cointegrated over this long period, albeit with a slightly lower probability (90%). However, we can see something of a regime shift around mid-2002, when CL went from generally under-valued to over-valued relative to XLE. (Even after including this regime with lower relative crude oil prices in my calculations, I still find the current spread to be undervalued by about $10,521 as of the close of Nov 17, which is near its 6-year low.)
What was the reason for this apparent shift in mid-2002? And are we in the middle of a similar regime shift in the opposite direction? Maybe our readers who have a better grasp of the economic fundamentals of the energy markets can shed light on this.