[MUSIC] And I can get rid the fixed size here because I'm just going to use the same plot. So by the way, when I put two of these things in the same plot, in the same cell, and I do that, Matplotlib is smart enough to print them in the same plot, okay? So you can see what's going on, this is essentially a moving average. So all I did here, this is a fancy way of plotting the index and a moving average at a trailing 36-month moving average of the index, right, mean over 36. And that's why if you look in here at the very beginning, at the very start here, there's nothing for the first 36 months. And that's because it takes 36 months to get the first data point, right? The first data point requires 36 months, so you lose that 36 months worth of data. And that's why the moving average that we generated out of that rolling window lacks data for the first 36 months. Okay, so now that we're done with that, we can actually look at the returns. The returns of that index, not the index itself, right? So let's let's do that, so let's look at total market index trailing 36-month returns, shall we call it that? Okay, so the trailing 36-month returns at any point in time, right? I'll just call it 36 returns, right? Well, how do we do that? Well, that is nothing more, we already have the total market index returns. So I'm going to look at the actual return, and we need a rolling window, right, we know that. The window size is going to be 36 months, we know that. But what do we want to do? We don't want to just compute the mean, we want to compound those returns over time, right? Now mean is built-in, so you could just say mean. But obviously the sort of compounding isn't built-in, but what is built-in is our good old friend aggregate, right? So we can aggregate each of those 36-month windows, right? So this is applied on each window, right? So what do we want to do for each window? How do we want to aggregate them? We want to aggregate them by just compounding them, right? And we already wrote code to do that, if you remember, that is nothing more than erk dot, what was it, annualize? Yeah annualize returns, right? So we're just going to call that function. So what's going to happen is, it's going to call erk.annualize_rets. That's what it's going to use to aggregate each 36-month window. Remember that we have to give it the periods per year, that was that was the additional periods_per_year=12, okay? So that'll give us the trailing 36 month returns, make sense? All right, so that we should be able to plot right away, right? So let's just plot that. So tmi_tr36rets.plot, let's just look at that, right? And let me make my figsize again (12,6), and what else? Let's give it, well, let's give it a label, because I'm going to add a second plot to it right now. And so I will say something like Tr36 mo Ret. Okay, good? So let's just look at this and see what we get. All right, so that looks somewhat reasonable, that's compounded returns over three years, it starts in 1929, looks good, right? But I actually want to see something else at the same time. I want to look at the actual returns at the same time. So why don't I do, why don't I plot the total market return as well? And I'm going to plot that as well. Let's give it a different label. Just the standard one, and that should be a lot more busy, that's exactly what I was hoping to see. Can't see which one is which, so let's say legend=True, and, Legend=True for that, okay? All right, good. That is exactly what I was hoping to see, right? You see no data for the first three years, and then it pops in. This is the actual returns, and this is the compounded returns, right? So good, so now we have looked, we've got a nice chart, right? The blue line, at least, so let me get rid of this. Let me get rid of that. So that blue line gives me a nice chart of the trailing 36 months. Now, all I need to do is, you can see that there's times when it's crashing, the market is crashing. So the trailing 36 months have been pretty bad. What I want to see is, is there some relationship between this and correlation? Because if correlations are going to be able to bail us out, we want to make sure that there's still correlation potential in these down markets, right? That's really what want to to check. So to be able to do that, we need some measure of are stocks correlating? Or are they decorrelating during these up markets and down markets? Or is there just no relationship, right? So what we want to do now, the last step in this, is to measure the average correlation across these industries. I'm just going to use industries here. And let's see what happens if there's some sort of relationship between the average correlation that was observed over time and the returns that were observed over time, okay? That is our final order of business. But to be able to do that, I need to introduce one sort of complicated concept. And I'm going to use this as an excuse to introduce the concept of a multi-index, so let's do that next. So let me do this, let me say, just so we know what we're doing here. We're talking about rolling correlations, that's what we want to do. Just like we've done rolling returns over there, we want to do rolling correlations. And we're going to be doing, along with, what are we going to call this? We're going to say MultiIndex. And .groupby as well, we're going to need that when we have a multi-index. So this is another cool thing. All right, that's what we're going to be doing next. Okay, so let's now look at correlations, right? So what do we want? We want to generate some sort of a time series of correlations, right? Let's call it ts_corr, right? So let's see if the obvious thing works, what is the obvious thing? Well, we've got the returns, so let's just do industry return, right? Let's try the same trick, right? 36, sorry, windows=36, right? And then remember we did mean, so now we should be able to just do corr, right? And let's look at what that looks like. So let's do ts_corr, and let's look at the very end of that, right? All right, so what it's going to do is run this correlation over time. And what we're seeing is a little disconcerting, because it's doing exactly what we want. It's producing a time, see when we did mean, it was computing the mean of the entire, It was computing the mean of that entire set of returns, right? Here what we're doing is we're computing the correlation. So the correlation is itself, A matrix, right, it's a square matrix. So what we're seeing here is it's a time series of matrices. It's not a time series of numbers, right? That's the problem, and the way that NumPy, I'm sorry, pandas is returning that is you see here, there are actually two indices here? The first index is a date, and then the second index is the industry, and the columns are the industries. So what that's saying is, for this date, wholesale and food had a correlation of 0.47. At this date, retail and beer had a correlation of 0.4. At this date, so on and so forth, finance and healthcare had a correlation of 0.65, okay? So that's what is a multi-index. Whenever you see two of these, you know that's a multi-index. Let's compare that, for example, with just the industry return itself, right? And let's just look at the head of that, right? And you'll see you don't have multi indices, there's just a single row index. Here, it's as if you had 2018 Wholesale, 2018 12 Wholesale, 2018 12 Retail, 2018 12 Meals, 2018 12 Fin, you get that? That's what a multi-index is, okay? Now our problem is, what do we want? We want to look at the average correlation, the average of that matrix, that whole big fat matrix of all of these. We want to look at the average correlation in 2018 12, right? So what we have to do is, we've got to group. So one of the things that we haven't really looked at so far is there's a really nice feature in pandas where you can just say group by. And group by is very much like rolling. In fact, rolling, you can think of it as a group by, where rolling is basically saying group by all the rows that fall into this window, right? Here you can say group by, and you give it what you want to group by. And since this is a multi-level index, you'll have to say which level you want to group by. And so let's do exactly that, you'll see it what I mean when I'm actually typing this in. Okay, so well, let's do one more thing, just to make sure that we are understanding this thing. So if I do ts_index, sorry, ts_corr I think is what I, yeah, ts_corr. And I look at the index, right, and you look at the names of that index, right? So here this is this index here, but ts_corr has two indices, right? And you can see that there's no names here, right? And that's why it didn't give you a name here, right? So let's give it a name, that'll make it a little easier to work with. So I'm going to say ts_corr.index.names, and I'm just going to assign it to, what was it? Well, the first index is the date, and the second index is the industry, right, something like that? Right, you understand what I'm doing here? This index, I'm giving it a name, and this index, I'm giving it an industry. So let's just do that, and let's look at the same thing that we did over there, right? Let's just look at ts_corr.tail again, right? Right, and now you see it's got a nice little label there, yeah? Maybe it's a little bit more clear when I do that. Okay, so now that we have that, we can actually do our trailing 36 correlation, right? So let's try that, so what does that look like, what do we really want? We want the average correlation over trailing 36 months, but we want the correlations this time, right? So we want the extended average. We've got this ts_corr stuff, but that's a multi-index. So the first thing we want to do is want to to group by date, right? So we want all of the things for a particular date. So you say groupby, right? And the level is a multi-level index. So it's called a level, right, is date, okay? So this will give you now a data frame. So whatever I'm going to call now here is going to be given a data frame that has all of the rows for that particular date, for each date, okay? Now, what do we want to do with that? Well, we want to apply some sort of function on that, right? And we've talked about these lambda functions before, this is a perfect example of where we can use a lambda function, okay? So what do we want? It's actually very simple, we want, so we're going to be given this kind of a covariance matrix as the input. So let's apply this lambda function. And what is this lambda function going to be called with? It's going to be called with all the rows for that date and all the columns for that date. Well, that's exactly the covariance matrix, right? So let's say it's given some covariance matrix, right? And what do we want to do, right? Actually, it's not a covariance matrix, sorry, it's a correlation matrix, right? This is this is not a covariance matrix, this is a correlation matrix. The entries here are correlations. Okay, so what do we want to do with the correlation matrix? Well, i'ts super simple, we just want the mean correlation, right? So coirmat dot, let's just say values.mean. Right, so far so good? Okay, so now I've got my industry. Let's just look at it before we do anything with it. So I'm going to say ind_tr36corr, and let me just plot it, just to just to make sure that I've got the right thing. And there you go, okay, so that's good. So now let's let superimpose these plots, let's make it look a little nicer. So first what I'm going to do is I'm going to do my total market index returns. But we want the 36-month returns, right? And I'm going to plot, right? Well, okay, this is going to be tricky, because we've got two different units. So well, let's just plot it and then we'll fix it, okay? So let's do the same thing we had before, label is Tr36 Mo Rets, right, that's the label. And let's do my figsize=(12,6). Okay, great, now let's plot on that same thing, Not the returns, but we want to do, what, this thing that we just generated, which ind_tr36corr, right? So we want to plot that, so this is the trailing 36-month correlations, say Corr, right? And yeah, let's say legend=True here, and let's do that as well, there, okay? And let's plot it, and you'll see that it's not really great. But let's do it and then see what happens. All right, so all right, this is okay, it's interesting. But you see what's happening is that this is in units of return, right? So this is, sorry, the blue line is in units of return, so it's going up and down, centered around maybe a little above zero. And on the other hand, our other thing is in units of correlation. So it's not really obvious what's going on. I want them to sort of superimpose, and I want something on the x-axis and I want the other one on the y-axis. Well luckily that is super easy to do. So all I have to do is, in this first one, I just have to say secondary_y axis? This is the danger of just doing it without looking, but let's try that. And if I get an error, I will go look up the manual. Beautiful, fantastic. All right, so you see what happened here, I gave it a secondary y-axis, that's all that happened. I just said there's a secondary y-axis here, and it now uses the y-axis for returns. The right axis for returns and the left axis for. We should probably do the other one, we should switch that around. Let's say returns on the left, just to be nice. Okay, there you go, right? So these are my returns on the left, right? And these are my correlations on the right, right? And my God, this is about as clear picture as were ever going to see, isn't it, right? What are we seeing? We're seeing that when the market is tanking, correlations are rising. When the market is sort of rising, then correlations are going sort of back to normal, right? And this is, remember trailing 36-month average correlations. And I think this is about as vivid a picture as you will see of why diversification fails you just when you need it most. When the market is falling apart, that huge plummet, right? That's when you really want correlations to bail you out, right? And look at what's happening to correlations. Now it's not always the case, I'm not saying this is exactly what happens, right? But you can see, certainly, look at that plunge from here to here, right? In 2008, what happened, right? So in fact, let's just plot it from 2008 onwards. So what are we doing here? So let's do let's do just 2007 onwards. Right, so I'm just going to plot that part of it. I'm going to zoom in, so to speak, right? Okay, and you can see that as the market is plummeting in 2008-2009, right, correlations are spiking up fairly dramatically, right? And they just keep going, right? They go as high as, how high? 0.75 is at its peak, correlations were 0.75, that's crazy. So obviously diversification is not going to help you. So let's try and put a number to that. A simple way for me to do that would be, let's take the correlation between these two series. So I've got my total market index, what is it, 36rets? Yeah, that's exactly right. And I compute the correlation of that with my industry 36 correlation. So I'm computing the correlation between the trailing returns and the trailing correlations, right? So it's a correlation with the correlation with returns, and you can see that it's actually negative, right? And that is really the problem with diversification. Diversification does not help you when markets are crashing, all right? So just be very clear about what this is. This is the correlation between the returns and the average correlation. So this is not the correlation of industry, this is not the average correlation across the industries. This is the correlation between the average correlation of industries and returns, okay? That, go back here, let me take that out, right? That orange line is the average correlation across industries. Good, I'm going to stop here. Hopefully I've made the point, and I've also use this as an excuse to introduce rolling windows and multi-level indices. So we did a lot of pandas stuff here. Play around with rolling indices, do, say, moving averages on your own data. And it will be really helpful to play around with this. I will see you next time, and see you then. [MUSIC]
