One thing I could do is look at how well my fifth order

polynomial hypothesis had done on my test set.

But the problem is this will not be a fair estimate of how well my

hypothesis generalizes.

And the reason is what we've done is we've fit this extra parameter d,

that is this degree of polynomial.

And what fits that parameter d, using the test set, namely,

we chose the value of d that gave us the best possible performance on the test set.

And so, the performance of my parameter vector theta5, on the test set,

that's likely to be an overly optimistic estimate of generalization error.

Right, so, that because I had fit this parameter d to my test set is no longer

fair to evaluate my hypothesis on this test set, because I fit my parameters

to this test set, I've chose the degree d of polynomial using the test set.

And so my hypothesis is likely to do better on

this test set than it would on new examples that it hasn't seen before, and

that's which is, which is what I really care about.

So just to reiterate, on the previous slide, we saw that if we fit some set of

parameters, you know, say theta0, theta1, and so on, to some training set,

then the performance of the fitted model on the training set is not predictive of

how well the hypothesis will generalize to new examples.

Is because these parameters were fit to the training set,

so they're likely to do well on the training set,

even if the parameters don't do well on other examples.

And, in the procedure I just described on this line, we just did the same thing.

And specifically, what we did was, we fit this parameter d to the test set.

And by having fit the parameter to the test set, this means that

the performance of the hypothesis on that test set may not be a fair estimate of how

well the hypothesis is, is likely to do on examples we haven't seen before.

To address this problem, in a model selection setting,

if we want to evaluate a hypothesis, this is what we usually do instead.

Given the data set, instead of just splitting into a training test set,

what we're going to do is then split it into three pieces.

And the first piece is going to be called the training set as usual.