案例学习：预测房价

Loading...

來自 University of Washington 的課程

机器学习：回归

3700 個評分

案例学习：预测房价

從本節課中

Assessing Performance

Having learned about linear regression models and algorithms for estimating the parameters of such models, you are now ready to assess how well your considered method should perform in predicting new data. You are also ready to select amongst possible models to choose the best performing. <p> This module is all about these important topics of model selection and assessment. You will examine both theoretical and practical aspects of such analyses. You will first explore the concept of measuring the "loss" of your predictions, and use this to define training, test, and generalization error. For these measures of error, you will analyze how they vary with model complexity and how they might be utilized to form a valid assessment of predictive performance. This leads directly to an important conversation about the bias-variance tradeoff, which is fundamental to machine learning. Finally, you will devise a method to first select amongst models and then assess the performance of the selected model. <p>The concepts described in this module are key to all machine learning problems, well-beyond the regression setting addressed in this course.

- Emily FoxAmazon Professor of Machine Learning

Statistics - Carlos GuestrinAmazon Professor of Machine Learning

Computer Science and Engineering

[MUSIC]

Okay, so another thing we can look at now that we

understand this notion of bias are measures of error as a function of

the number of data points in our training data set.

Okay, so let's start with looking at our true error or generalization error.

But first, I want to make sure its clear that we are looking

at these errors for a fixed model complexity.

So previously we were looking at errors versus model complexity, but

now we are fixing the model complexity.

Looking as a function of the number of data points.

Fixed model complexity.

And let's look at our true error.

And our true error starts somewhere.

And it's somewhere high, because when we have very few data points, our

fitted function is a pretty poor estimate of the true relationship between x and y.

So our true error's gonna be pretty high,

so let's say that w hat is not approximated well from few points.

But as we get more and more data, we get a better and

better approximation of our model and our true error decreases.

But it decreases to some limit.

And what is that limit?

So this is true error, just to be clear.

Well that limit is the bias plus the noise inherent in the data.

Because as we get tons and tons of observations, well,

we're taking our model and fitting it as well as we could ever hope to fit it,

because we have every observation out there in the world.

But the model might just not be flexible enough to capture the true relationship

between x and y, and that is our notion of bias.

Plus, of course,

there's the error just from the noise in observations that other contribution.

Okay, so this difference here Is the bias

of the model and noise of the data.

Okay so that might make sense as a plot of true error versus number of data points,

but now let's look at training error.

And this might look a little more surprising.

So let's say our training error starts somewhere.

But what ends up happening is training error goes up as you get more and

more data points.

Cuz remember, we're keeping the model complexity fixed.

So when we have few data points,

so with few data points,

a fixed complexity model can fit them,

I guess, reasonably well, whatever it is.

Fit these points reasonably well, where reasonably of

course depends on what the complexity of the model is.

But as I get more and more and more data points, that same

complexity of the model can't hope to fit all these points perfectly well.

And what is the limit?

What is the limit of training error?

Let me just annotate this as training error.

Well that limit is exactly the same as the limit of our true error.

And why is that?

Well, I have tons and tons of points there.

That's all points that there could ever be possibly in the world, and

I fit my model to it.

And if I measure training error,

I'm running it to all the possible points there are out there in the world.

And that's exactly what our definition of true error is.

So they converge to exactly the same point in the limit.

Where that difference again, is the bias inherent from the lack of

flexibility of the model, plus the noise inherent in the data.

Okay, so just to write this down in the limit, I'm getting lots and

lots of data points, this curve is gonna flatten out.

To how well model

can fit true

relationship f

sub true.

Okay, so I feel like I should annotate here also,

saying in the limit our true

error equals our training error.

Okay so what we've seen so

far in this module are three different measures of error.

Our training, our true generalization error

as well as our test error approximation of generalization error.

And we've seen three different contributions to our errors.

Thinking about that inherent noise in the data and

then thinking about this notion of bias in variance.

And we finally concluded with this discussion on the tradeoff between

bias in variance and how bias appears no matter how much data we have.

We can't escape the bias from having a specified model of a given complexity.

Okay, and in the subsequent few videos,

we 're gonna look at these notions with formalism.

[MUSIC]