Hi. Welcome back. In this set of lectures we're talking about Markov models. And in

the previous lecture I introduced what they were. How they are these finite set

of states and there's transition probabilities between those states. What I

want to do in this lecture, is just show you how a simple Markov process works. So

we're gonna take a very, very simple Markov process and work through it. See

exactly how dynamics unfold. And in doing so we're going to learn how to use

matrices. How to actually, specifically how to multiply matrices. Okay, so let's

take our simple example, and let's do the case of the alert and bored students. So

I'm teaching a class, like many people do. This is an online class. There's some

percentage of people that are alert, and there's some percentage of people that are

bored. And what's going to happen, at any given moment in time, someone whose alert

could switch and become bored, and someone who's bored could switch and become alert.

And what we want to do is we want to understand the dynamics of that process. We want a

model that helps us understand these dynamic processes where there's these

states, alert and bored, and people move between them. So we got to make some

assumptions. We've got the two states, alert and bored. Now we need

to understand the transition probabilities. We need to assume something

about transition probabilities. So let's assume the following is true: that 20

percent of the alert students become bored, but the 25 percent of the bored

students suddenly say, "hey, that sounds interesting, these Markov processes sound

really cool", and they become alert. So, that's what we wanna model. So we can

think of that as, we've got this set of alert students. 20 percent of them are

gonna become bored. And of the bored students, 25 percent are gonna become

alert. So we can draw this picture, but this doesn't help us much. We wanna sort of

figure out what's gonna happen. And this is where the matrices will be useful. Now,

before we do the matrices, let's just try to do it by hand. So, let's start with

this scenario. You've got 100 alert students. So, if I have 100 alert. Let's

call this A, is 100. And bored is zero. And I know that 20 percent of

the alert become bored and the rest are going to stay alert. So that means what's

going to happen is that I have 80 alert and 20 bored. We're now going to

think, okay, what happens next? What happens next, I know that of the 80 alert,

I know that 20%, which is 16, are going to become bored. And the rest, 64,

will be alert. Now the 20 that are bored, I know that 25%, 25% of that

is 5, so I'm going to put 5 of those, become alert and I'm gonna know

that 75 percent of it, which is 15, stay bored. So what I'm gonna get

is 69 alert and 31 bored. And I can say, okay now I've got alert, 69, and bored,

I've got 31. And now I've gotta do this again. I've gotta think, okay, well,

20% of these, which is gonna be 13.8, become bored, and so on. And this,

you think, okay, this gets really complicated. And I've got numbers all over

the place. Maybe there's a better way. Maybe there's, instead of writing all

these numbers with these zeroes, there must be a simpler way to keep track of all

this. Well, there is, and the idea is something called a Markov Transition

Matrix. And here's the idea. We basically write down the probabilities of moving

from state to state. So, these columns tell us what's true at time t. And the

rows tells us what's true at time t+1. So if you're alert at time t, there's an 80%

chance you stay alert and a 20% chance you become bored. If you're

bored at time t, there's a 25% chance you become alert and a 75%

chance that you stay bored. So this gives us the matrix representations, simple

representation of all these transition probabilities. Now the reason this is

useful is, then we can just multiply these matrices to see how the transitions

unfold. So, here's an example. Suppose I start with 100%, or one. This is just a

probability, so probability one summons alert. And I want to figure out how many

alert people are next time. This number, this point eight, tells me the percentage

of alert people that stay alert. So I'm going to get, so I take point eight and I

multiply it by the one. That gives me point eight. And I want to ask, how many

bored people become alert? Well, 25 percent do. And how many bored people were

there? Zero. So I'm going to get point 25. Times zero, so I end up with point eight.

So the way I multiply matrices is I basically take this row here, and I

multiply it by this column. Now, let's make this more formal. So, what I do to

multiply these things out is, I've got here's where people go at times t+1.

This is how many are gonna be alert at time t+1. 80 percent of the

alert people, and 25 percent of the bored people. Here is the percentage of alert

people, and the percentage of the bored people. So I want to know how many are

going to be alert next time. 80 percent of the one, and 25 percent of the zero, and

that's going to mean point eight. Now I want to ask how many are going to be

bored. Well, 20 percent of the alert people, and 75 percent of the bored

people. So I take this row, and also multiply by the composite. Take point two

times one, and point 75 times zero, and I get point two. So what you get is, when I

multiply this matrix, the transition matrix here by this initial

vector one, zero. I get 80, 20 just like I did before. Remember the way I did

that is I take this row, multiplied by the column and then I take this row,

multiplied by the column. Watch now, I can do it again. Now I'm at 80, 20. And I

wanna ask how many am I gonna get next period. Well I take this row 80, 25 and

multiply it and see 80 percent of the people are alert, 80 percent of them stay

alert. So that's right there, 20 percent are bored, 25 percent

become alert. So that's right there. And when I add those up, I'm going to get 69.

And here I'll get 31. We'll see what happens in the next period. I again just

take this row times this column. So of the 69 percent that are alert, 80 percent of

them stay alert. And of the 31 percent that are bored, 25 percent of them become

alert. And I get that 63 percent are alert. And then I can do it again. Take

point eight times the 63, point 25 times the point 37, then I get the new

percentage that's alert. Which is going to be 60%. And I can keep going, and going,

and going, and if I do it one more time, I end up with 58%. So what does that tell

us? That tells us that if we started with all alert students, after six periods, I'm

gonna end up with 58 percent of students being alert. Now we wanna know, where does

this process stop? Is it going to go end up with nobody being alert? Well, let's

think that through. Let's suppose that we started out with nobody being alert. And

we can ask what happened. So I started with all bored students. What's gonna

happen? Well now all I do is put a zero for the alert and a one for the bored and

ask what happens next. Well, 80 percent of these alert students will stay alert, but

that's zero. So it's .8 times zero. And 25 percent of the bored students will become

alert. That means I'm gonna get .25. And that means, since this sums to one, I'm

gonna get .75 over here. Next period I've got: 25% of the students are

alert, and 75 percent are bored. Well, now I can just put that here as my population

at time 2, and I can think, okay, of these 25% that are bored, or alert,

I'm sorry, 80 percent will stay alert. Of the 75 percent that are bored, 25 percent

become alert. And if I multiply all that out, I get that 45 percent are alert, 55

percent are bored. If I do it again, I'll end up with an even number of alert and

bored. And if I do it one more time, I'll get that 53 percent are alert, and 47

percent are bored. So it sorta looks like this is going to an equilibrium. When I

started off with everybody alert, I got down to 58 percent alert. And when I

started out with everybody being bored, I ended up with 52 and a half percent being

alert. So it looks like it's converging somewhere between 53 and 58%. Well, how do

I find what that equilibrium is? This is where the matrices become really powerful.

So, let's think of it this way. There's some percentage of people that are alert.

That's p. There's some percentage of people that are bored, that's 1-p. What

I'd like is, after this process takes place, for the same percentage to be

alert. So how many people are going to be alert? Well, that's going to be .8 p plus

.25 times one minus p. So what would it mean for there to be an equilibrium? The

equilibrium would mean that after I multiply this out, I have the same

percentage of people being alert. So the equilibrium, I've put a little star here,

is gonna be the p-star, actually .8 p-star plus .25 times one minus p-star. Just

gives me p-star back. That I end up with the same percentage of people alert that I

started. This just becomes algebra. I can now write this out. I've got an equation

where this is my Markov transition matrix. And I want some probabilities p of people

being alert. So I should after the transition, I've got p back. Right? So

that's just gonna be .8 p plus .25 times one minus p, should

equal p. I wanna find the p that solves this. Well, let's multiply through by

20, just to make this simpler. So I'm gonna get 16 p plus five times one

minus p equals 20 p. So that's going to give me 16 p plus five minus 5 p equals 20 p.

And so we bring all the "p"s over to the one side, and we get five equals 9 p, so p

equals 5/9. So what that says is that if I take with 5/9 of the people being

alert, I'm going to end up with 5/9 of the people being alert. So let's think

about how that works precisely. So 5/9 of the people alert. What do we know? We

know that 20% of them are gonna become bored. So that means 20%

means that each period, one-ninth of the population will become,

move from alert to bored. I also know that 25% of the bored people will

become alert. What's 25% of 4/9? That's also 1/9. All

right? So what I, is that, so each, each period, 1/9 of the people are moving

from alert to bored, and 1/9 of the people are moving from bored to alert,

which means that exactly 5/9 stay alert and exactly 4/9 stay bored. Now notice

what this equilibrium is like. It's a statistical equilibrium. So if we can

think of an equilibrium point where nothing changes, here the thing that's not

changing is the probability. So the population is still churning. People are

moving from alert to bored. But if I think in terms of probabilities, that

probability is staying fixed. That probability is staying fixed

at 5/9. People are moving around, but the probability's staying fixed.

That's why this is sometimes called a statistical equilibrium, 'cause the

statistic p, the probability of someone being alert, is the thing that doesn't

change. Okay, pretty involved, right? What we did is, we wrote down the Markov

transition matrix. And we showed how using that matrix, we could solve for

an equilibrium. And we saw, at least in the simple example of alert and bored

students, that the process went to an equilibrium, and it was fairly

straightforward to solve for. What we want to do next is we want to do [a] slightly more

sophisticated model that involves multiple states instead of just two, involves three

states, and we'll see how that process also converges to an equilibrium. Thanks.