Hi. Welcome back. In this set of lectures we're talking about Markov models. And in the previous lecture I introduced what they were. How they are these finite set of states and there's transition probabilities between those states. What I want to do in this lecture, is just show you how a simple Markov process works. So we're gonna take a very, very simple Markov process and work through it. See exactly how dynamics unfold. And in doing so we're going to learn how to use matrices. How to actually, specifically how to multiply matrices. Okay, so let's take our simple example, and let's do the case of the alert and bored students. So I'm teaching a class, like many people do. This is an online class. There's some percentage of people that are alert, and there's some percentage of people that are bored. And what's going to happen, at any given moment in time, someone whose alert could switch and become bored, and someone who's bored could switch and become alert. And what we want to do is we want to understand the dynamics of that process. We want a model that helps us understand these dynamic processes where there's these states, alert and bored, and people move between them. So we got to make some assumptions. We've got the two states, alert and bored. Now we need to understand the transition probabilities. We need to assume something about transition probabilities. So let's assume the following is true: that 20 percent of the alert students become bored, but the 25 percent of the bored students suddenly say, "hey, that sounds interesting, these Markov processes sound really cool", and they become alert. So, that's what we wanna model. So we can think of that as, we've got this set of alert students. 20 percent of them are gonna become bored. And of the bored students, 25 percent are gonna become alert. So we can draw this picture, but this doesn't help us much. We wanna sort of figure out what's gonna happen. And this is where the matrices will be useful. Now, before we do the matrices, let's just try to do it by hand. So, let's start with this scenario. You've got 100 alert students. So, if I have 100 alert. Let's call this A, is 100. And bored is zero. And I know that 20 percent of the alert become bored and the rest are going to stay alert. So that means what's going to happen is that I have 80 alert and 20 bored. We're now going to think, okay, what happens next? What happens next, I know that of the 80 alert, I know that 20%, which is 16, are going to become bored. And the rest, 64, will be alert. Now the 20 that are bored, I know that 25%, 25% of that is 5, so I'm going to put 5 of those, become alert and I'm gonna know that 75 percent of it, which is 15, stay bored. So what I'm gonna get is 69 alert and 31 bored. And I can say, okay now I've got alert, 69, and bored, I've got 31. And now I've gotta do this again. I've gotta think, okay, well, 20% of these, which is gonna be 13.8, become bored, and so on. And this, you think, okay, this gets really complicated. And I've got numbers all over the place. Maybe there's a better way. Maybe there's, instead of writing all these numbers with these zeroes, there must be a simpler way to keep track of all this. Well, there is, and the idea is something called a Markov Transition Matrix. And here's the idea. We basically write down the probabilities of moving from state to state. So, these columns tell us what's true at time t. And the rows tells us what's true at time t+1. So if you're alert at time t, there's an 80% chance you stay alert and a 20% chance you become bored. If you're bored at time t, there's a 25% chance you become alert and a 75% chance that you stay bored. So this gives us the matrix representations, simple representation of all these transition probabilities. Now the reason this is useful is, then we can just multiply these matrices to see how the transitions unfold. So, here's an example. Suppose I start with 100%, or one. This is just a probability, so probability one summons alert. And I want to figure out how many alert people are next time. This number, this point eight, tells me the percentage of alert people that stay alert. So I'm going to get, so I take point eight and I multiply it by the one. That gives me point eight. And I want to ask, how many bored people become alert? Well, 25 percent do. And how many bored people were there? Zero. So I'm going to get point 25. Times zero, so I end up with point eight. So the way I multiply matrices is I basically take this row here, and I multiply it by this column. Now, let's make this more formal. So, what I do to multiply these things out is, I've got here's where people go at times t+1. This is how many are gonna be alert at time t+1. 80 percent of the alert people, and 25 percent of the bored people. Here is the percentage of alert people, and the percentage of the bored people. So I want to know how many are going to be alert next time. 80 percent of the one, and 25 percent of the zero, and that's going to mean point eight. Now I want to ask how many are going to be bored. Well, 20 percent of the alert people, and 75 percent of the bored people. So I take this row, and also multiply by the composite. Take point two times one, and point 75 times zero, and I get point two. So what you get is, when I multiply this matrix, the transition matrix here by this initial vector one, zero. I get 80, 20 just like I did before. Remember the way I did that is I take this row, multiplied by the column and then I take this row, multiplied by the column. Watch now, I can do it again. Now I'm at 80, 20. And I wanna ask how many am I gonna get next period. Well I take this row 80, 25 and multiply it and see 80 percent of the people are alert, 80 percent of them stay alert. So that's right there, 20 percent are bored, 25 percent become alert. So that's right there. And when I add those up, I'm going to get 69. And here I'll get 31. We'll see what happens in the next period. I again just take this row times this column. So of the 69 percent that are alert, 80 percent of them stay alert. And of the 31 percent that are bored, 25 percent of them become alert. And I get that 63 percent are alert. And then I can do it again. Take point eight times the 63, point 25 times the point 37, then I get the new percentage that's alert. Which is going to be 60%. And I can keep going, and going, and going, and if I do it one more time, I end up with 58%. So what does that tell us? That tells us that if we started with all alert students, after six periods, I'm gonna end up with 58 percent of students being alert. Now we wanna know, where does this process stop? Is it going to go end up with nobody being alert? Well, let's think that through. Let's suppose that we started out with nobody being alert. And we can ask what happened. So I started with all bored students. What's gonna happen? Well now all I do is put a zero for the alert and a one for the bored and ask what happens next. Well, 80 percent of these alert students will stay alert, but that's zero. So it's .8 times zero. And 25 percent of the bored students will become alert. That means I'm gonna get .25. And that means, since this sums to one, I'm gonna get .75 over here. Next period I've got: 25% of the students are alert, and 75 percent are bored. Well, now I can just put that here as my population at time 2, and I can think, okay, of these 25% that are bored, or alert, I'm sorry, 80 percent will stay alert. Of the 75 percent that are bored, 25 percent become alert. And if I multiply all that out, I get that 45 percent are alert, 55 percent are bored. If I do it again, I'll end up with an even number of alert and bored. And if I do it one more time, I'll get that 53 percent are alert, and 47 percent are bored. So it sorta looks like this is going to an equilibrium. When I started off with everybody alert, I got down to 58 percent alert. And when I started out with everybody being bored, I ended up with 52 and a half percent being alert. So it looks like it's converging somewhere between 53 and 58%. Well, how do I find what that equilibrium is? This is where the matrices become really powerful. So, let's think of it this way. There's some percentage of people that are alert. That's p. There's some percentage of people that are bored, that's 1-p. What I'd like is, after this process takes place, for the same percentage to be alert. So how many people are going to be alert? Well, that's going to be .8 p plus .25 times one minus p. So what would it mean for there to be an equilibrium? The equilibrium would mean that after I multiply this out, I have the same percentage of people being alert. So the equilibrium, I've put a little star here, is gonna be the p-star, actually .8 p-star plus .25 times one minus p-star. Just gives me p-star back. That I end up with the same percentage of people alert that I started. This just becomes algebra. I can now write this out. I've got an equation where this is my Markov transition matrix. And I want some probabilities p of people being alert. So I should after the transition, I've got p back. Right? So that's just gonna be .8 p plus .25 times one minus p, should equal p. I wanna find the p that solves this. Well, let's multiply through by 20, just to make this simpler. So I'm gonna get 16 p plus five times one minus p equals 20 p. So that's going to give me 16 p plus five minus 5 p equals 20 p. And so we bring all the "p"s over to the one side, and we get five equals 9 p, so p equals 5/9. So what that says is that if I take with 5/9 of the people being alert, I'm going to end up with 5/9 of the people being alert. So let's think about how that works precisely. So 5/9 of the people alert. What do we know? We know that 20% of them are gonna become bored. So that means 20% means that each period, one-ninth of the population will become, move from alert to bored. I also know that 25% of the bored people will become alert. What's 25% of 4/9? That's also 1/9. All right? So what I, is that, so each, each period, 1/9 of the people are moving from alert to bored, and 1/9 of the people are moving from bored to alert, which means that exactly 5/9 stay alert and exactly 4/9 stay bored. Now notice what this equilibrium is like. It's a statistical equilibrium. So if we can think of an equilibrium point where nothing changes, here the thing that's not changing is the probability. So the population is still churning. People are moving from alert to bored. But if I think in terms of probabilities, that probability is staying fixed. That probability is staying fixed at 5/9. People are moving around, but the probability's staying fixed. That's why this is sometimes called a statistical equilibrium, 'cause the statistic p, the probability of someone being alert, is the thing that doesn't change. Okay, pretty involved, right? What we did is, we wrote down the Markov transition matrix. And we showed how using that matrix, we could solve for an equilibrium. And we saw, at least in the simple example of alert and bored students, that the process went to an equilibrium, and it was fairly straightforward to solve for. What we want to do next is we want to do [a] slightly more sophisticated model that involves multiple states instead of just two, involves three states, and we'll see how that process also converges to an equilibrium. Thanks.