All right, well, this is going to be, again, a little bit of a long session, because the goal here is to really look at what happens when markets fall, in terms of the correlations across the stocks in the market. Now, the way I'm going to do this, I'm going to use this as an excuse to sort of, actually build a cap-weighted market index, and I'll give you a sense of how that's done. And we're going to do a fair amount of Pandas stuff that's interesting and new as well. So I think I should jump right in. And this stuff is standard, I think we've seen this now enough times that I don't need to talk about it. What I'm going to do is, here, let me copy that. Because that is the part I'm going to start modifying. What I want to load here is, in addition to the industry returns, I want to introduce two new files that I've made available. The first is, so this is the file that we've been working with so far, that's the returns of each industry in any month, of the portfolio that consists the stocks in that industry for that month. Now, the way they computed this return is, they constructed a portfolio that was consisting of all the stocks in the food industry, and then they weighed those stocks by the market cap of each of those stocks. So the market cap of the stock is simply the number of shares outstanding multiplied by the price. So, one way for you to think about the market cap of the stock is, it's the total value of the company. In other words, if you bought every single share of that stock that's out there, you own 100% of that company, that's what it would cost you. So that's the market capitalization, and in a sense, it is really the market price of the company at that moment, all right? The number of shares outstanding multiplied by the price. Okay, so that is the market cap, and so if you, in a sense, want to look at the total value of the market, all you do is, you add up all the market caps of all the companies, and that is the total market size. If you're Building a portfolio where all the components of that portfolio, all the individual stocks in that portfolio, are proportionate to how big they are, how much value is embedded in that stock price. Then you would weigh it by the market caps, and that's how the S&P 500, or the FTSE 100, or the MSCI indices, that's exactly how they're done, with a minor correction for liquidity, which is what they call the float adjustment. We're not going to do a float adjustment, but short of that, we're going to do everything else. We're going to construct a market index very similar to a cap-weighted index, and then we're going to use that to understand what happens to correlations when that stock market index is rising, and when that stock market index is falling, so you can see we've got a lot of work to do, okay? So, to be able to construct that market index, what do we need? We've got to have two things to figure out what weight should I put in this food industry portfolio. Well, you need to know what is the total market capitalization of all of the stocks in that food industry, so what do you need? So the first thing you need is the number of firms that made up that, so let's look at that file that's contained in this nfirms file, and the nfirms file tells us that at that moment in time, that is in July of 1926, we had 43 stocks in that food portfolio. We had three beer companies, and we had 16 tobacco companies, and so on and so forth. Now, what is the size of those companies? Well, that's what the size file tells you, and it says that, on average the size was 7.12 for those three companies, the average size was 7.12. Don't worry about what the units of that are, it doesn't matter because we're going to be looking at the market cap weight, so we can ignore the units. So, what I'm going to do now is exactly that, so the first order of business is, I need to read these three files in. And so for now, let me do something super simple, because these files all have the same size. So I'm just going to go in here. This is that thing that we have, this is the one that we just created. I'm just going to make, this is a terrible way of doing it, but why don't we just do it this way? So I'm going to do size. I'm going to add a new function that returns the size. And so, instead of the valuated reads, I'm going to read the size file. For the size file, we don't really need to divide by 100, so I'll take that out, but everything else, it looks exactly the same. And then let's also do, let me get rid of this doc string, because this is definitely wrong at this point. And like I said, I don't really like copy-pasting like that, but I just want you to see that really it's the exact same thing, so that's why I don't want to refactor this code. And I'll do and I'll put that in the material, but for now just just let's keep it simple. So this is nfirms, so we did the size up there, and again, we don't want to divide by 100 there. Right, and there you go, okay? Let's fix this, that's nfirms. Save it, all right. Now let's go in here, and just as we've got this, let's also pull in the additional two files that we've got, nfirms, and what did we call the other one? Size, right? And so, we want nfirms to be nfirms, and size here, okay, good. Let's see what happens when you do that. So, what we should do is, we should get three new variables. So let's just look at the industry return shape, and we should make sure that these have the same shape, let me get rid of the space here. Okay, so, we should have the same shape for the nfirms file. Yep, that's good. And, I'll remember to do copy-paste this time. And this one should be size, there should be no surprises at, okay, good. So that means we've been able to load these files in properly. So now let's just look at them, just to make sure that we also got, so I'll say, ind_nfirms, let's look at the head, and yeah, that's exactly what we were expecting, so, good. So now we have the source data that we need, the raw data, so what do we need next? So, the first thing we need to do to construct our sort of market cap index is, we need to find out the market capitalization of each one of these, well, that's super easy. So let's compute the market capitalization, so market cap of each one of these industry portfolios over time, that's what we want, and that's what, that's the number of firms times the average size. So that's that, and let's just look at the shape of that, it should be, just to make sure. Yep, that's good, that's exactly what we wanted. And now what do we want? So if you want to find out the fraction of the total market capitalization that is in each industry, what do we need? We need the total market capitalization. So let's compute the total market cap, total market cap, and what is the total market cap? It's nothing more than each Industriy's market cap, so what do we call that industry market cap, right? And you want to sum it, but you don't want to sum it over the industry, you don't want the sum of the columns, you want to sum across columns. So what you do is, you say, axis, the axis with which across which you want to sum is the columns. And if this worked well, well, let's just plot it and see. And yes, as you should expect, that the market capitalization has of course dramatically gone up over time. Let's make this a little bigger. Figsize is, let's call it (12,6), yeah, so there you go, looks nice now. And you can see that when these crashes happen, which is what we're going to be talking about, you lose a lot of market capitalization. Okay, good. So, now what is the next order of business? We've been able to compute the market capitalization, but what we really want is the weight of each one of these industries over time, so we want the capitalization weight. So what do we want? We've got the total and we've got each individual one, so all we have to do is divide each industry's market capitalization by the total market capitalization at that point in time. So let's compute the cap weights, so ind_capweight, is what? It is the industry market cap. And now we want to divide that by the total, we want every single one of those columns to be divided by the total market cap column. So, the way you do that is by calling the divide function, or the divide method, and you give it what you want to divide by, so it's the total market cap. But again, remember that the axis we want to use is what? This time we want to do it over rows, because what we've got here, the total market cap is one big column, so you want to do this. The access that you want to do this is over the rows, so you want to do this for every row, in other words, makes sense? Okay, so let's do that, and what do we get? So let's just make sure that this capweight.head. So what does that mean? Good, this is looking exactly the way I wanted, what this is saying, the way to interpret this is, in July of 1926, food was 0.05, that means 5.8% of the total market cap. Let's say, chemicals were 3%, so let's just try something just to make sure that, let's look at this industry capweight, and let's just look at it in 1926. And let's just sum, I'm just going to sum across all of these, and make sure that I get to 100%. So I want to sum, and the axis that I want to use is columns here, and I get 100%. That means, what that's saying is, if you add up all of the capweight of food plus the capweight of beer plus the capweight of smoke plus the capweight of games at that point in time, you add them all up, you get 100%, which is exactly what I want to see. Good, that tells us that we're in good shape. Let's just look at something just for fun. So for example, let's look at the capweight of, let's look at finance, and let's compare it to something like, I don't know, steel or coal, or something like that, and let's just plot that over time. Yeah, and what that's telling you here, figsize=(12,6). So what that's telling you is, back in 1929 and 26, steel was a huge part of the market cap. So steel companies were a big part of the stock market, almost 10%, little less than 10%, today they are much less. On the other hand, finance companies, financial companies were just a teeny tiny bit of the total market, and of course, they're a much bigger part of the market today. Okay, so that's that's how you interpret that. So, where the finances now is something between 15 and 20% of the total market. The total market capitalization of finance companies is about 20% of 15% of the total market capitalization market. Okay, so we got that, so now we've got all the capweights for the individual components, so we can construct a cap-weighted market index, just like the S&P 500, what does the S&P 500 do? It takes 500 stocks, the top 500 stocks, and then it computes a portfolio that consists of the prices of all those companies, or the values of all of those companies, proportional to their market capitalization. We're going to do exactly the same thing. So we're going to compute, let's call it a total_market index. So, how do we compute the returns of that index? Well, let's first compute the return. What's the total Market return that we got? The total Market return is nothing more than the weighted average of each one of these industries. So let's let's just do a weighted average. So how do you do a weighted average? Well, it is, you take the capweight, and you multiply by the return of that industry. So you take the capweight of the industry multiplied by the return industry, and then you just add them up. And again, (axis="columns"). So, over time all we're doing is, you're just computing the weighted average return of each industry. So the total weighted average return across all Industries, and you want to look at that over time, so that's your total market return. So, let's see if that looks like a return, so let's say, total_market_return, and let me plot that, just to see if, oops, sorry. Right, okay, so, yeah, that looks kind of busy and looks like a return, so far so good, well, what do we want to do? We want to take these returns and convert it into a total market index, so we could just compound the returns, but I'm just going to remind you that we actually wrote the code to do that some time ago, so let's just use that. So the total_market_index is nothing more than the wealth index derived from those returns. So we already wrote that code at some point, when we were doing drawdowns, if you remember, if I give it the total market returns now. And if you remember, one of the things we computed the drawdowns, but we also computed the wealth, the wealth Index. Well, that's exactly what we're looking for, so, let's just do that, let's let's take that total_market_index and plot it. Okay, and I'll make my figsize=(10,5) and let's maybe give it a title, just to look nice, title=Total Market Capweighted Index. And that's from 1926-2018. Okay, let's see what that looks like, and there you go. Good. That is our cap-weighted index, very much like, think of it as a broader market, like the Wilshire index or something, it's all stocks, because the industry portfolio is constructed from all stocks. Okay, so now what do we want to do next? What we want to do is, we want to look at the returns of the market over a three year period. So what is the endgame here? What I want to do really is, I want to look at how returns is related to market correlations. We'll get to correlations in a second, but let's start by looking at the sort of trailing 36 month return, okay? And the way I would do this, so let's start with something simple. Let me start by saying, let me start over a period of time, so I can say, I've got my total market index, total_market_index. I've got my index. Let's just look at it from, I don't know, 1980 onwards, just to look at it over a period of time. So from 1980 onwards, what do I have? I have, well, let's just plot that in. Let me, again, put a larger figsize so that you can see it, the same (12,6), okay? Now, on that same plot, on that exact same plot I'm going to plot the exact same thing, except I'm going to use this thing called rolling. So what does rolling do? What rolling does is, it's a rolling window, and it starts with a time series, this is something that works if your index is a time series, and that's why we went through the pain of making sure that the index for this variable, for this data frame was a time series. So, the row index. So the row index for this is a time series, we know that, that's why we get nice plots and things like that, but we also get the advantage of being able to do this rolling business. So what I'm going to do is, I'm going to give it a window size of 36, this is monthly returns if you remember, right? So 36 months is 3 years. So what it's going to do is, it's going to produce another time series, and that time series, each element of that time series is computed out of a window of 36 months. And so, that gives us the 36 month window, and what do we want in that 36 month window? Well, we can do various things, but the simplest thing, I'm just going to compute the mean return, okay? So, what does this look like? [MUSIC]