Explore
For Enterprise
Join for Free
Log In

艺术与人文
商务
计算机科学
数据科学
信息技术
健康
数学和逻辑
个人发展
物理科学与工程
社会科学
语言学习
學位
證書

瀏覽 Coursera 的全部課程

瀏覽
搜索
企業版
登錄
免費加入

Prediction study design

Loading...

实用机器学习

约翰霍普金斯大学

4.5（2,578 個評分） | 99K 名學生已註冊

課程 8（共 10 門，数据科学專項課程）

One of the most common tasks performed by data scientists and data analysts are prediction and machine learning. This course will cover the basic components of building and applying prediction functions with an emphasis on practical applications. The course will provide basic grounding in concepts such as training and tests sets, overfitting, and error rates. The course will also introduce a range of model based and algorithmic machine learning methods including regression, classification trees, Naive Bayes, and random forests. The course will cover the complete process of building prediction functions including data collection, feature creation, algorithms, and evaluation.

查看授課大綱

您將學習的技能

Random Forest, Machine Learning (ML) Algorithms, Machine Learning, R Programming

審閱

4.5（2,578 個評分）

5 stars
1,685 ratings
4 stars
608 ratings
3 stars
189 ratings
2 stars
58 ratings
1 star
38 ratings

DH

Jun 18, 2018

Excellent introduction to basic ML techniques. A lot of material covered in a short period of time! I will definitely seek more advanced training out of the inspiration provided by this class.

HP

Jan 16, 2017

It was like opening up a door to a whole new world. I have discovered new tools that I will thoroughly enjoy to use for the exploration of data and for predictions. Thanks Team Coursera !

從本節課中

Week 1: Prediction, Errors, and Cross Validation

This week will cover prediction, relative importance of steps, errors, and cross validation.

Prediction motivation8:26

What is prediction?8:39

Relative importance of steps9:45

In and out of sample errors6:57

Prediction study design9:05

Types of errors10:35

Receiver Operating Characteristic5:03

Cross validation8:20

What data should you use?6:00

教學方

Jeff Leek, PhD
Associate Professor, Biostatistics
Roger D. Peng, PhD
Associate Professor, Biostatistics
Brian Caffo, PhD
Professor, Biostatistics

腳本

In the last lecture we learned about in sample

and out of sample errors as well as over fitting.

In this lecture we'll talk about prediction study design or how to minimize

the problems that can be caused by in sample verses out of sample errors.

So in prediction study design the first thing that

we need to do is to find our error rate.

So in this lecture we're just going to use a generic error rate, but in the

next lecture we'll talk about what are

the different possible error rates you can choose.

Then you need to split your data, the data that you're going to be using

to build and validate your model into three

components, well, two components and one optional component.

So there's a training set that must be created

in order to build your model, a testing set to

evaluate your model, and optionally a validation set as well,

which is also going to be used to validate your model.

So what you do is on the training

set, you pick features using cross-validation, for example.

We'll talk about what cross-validation is later in

the course, but the idea is basically to

use that training set to pick which of

the features that are most important in your model.

Then use that same technique to actually pick the prediction function

and estimate all the parameters that you might be interested in.

We build a model using the training set.

If there's no validation set, then we apply the best

model that we have to our test set exactly one time.

And so the, why do we only apply it one time?

If we applied multiple models to our testing set, then, and pick the best

one, then we're using the test set, in some sense, to train the model.

In other words, we're still getting an optimistic view of

what the data error would be on a completely new dataset.

So you should apply the prediction model to the test set exactly one time.

If there's a validation set and a test set then you might apply your

best prediction models all to your test set and refine them a little bit.

So what you might find is that some features don't work so well when you're

doing out of sample prediction and you might

refine and adjust your model a little bit.

But now, again, like I said, your test set error is going to be

a little bit optimistic error for what your actual out of sample error will be.

And so what we do is we again, apply our model to exactly

one time to the validation set, only the best one to get our prediction.

So in other words, the idea is that there is one dataset that's

held out from the very start, that you only apply exactly one model

to, and you never do any training or tuning or testing to, and

that will give you a good estimate of your out of sample error rates.

So an important point to keep in mind when

you doing this is to know what the benchmarks are.

So this is an example of a leaderboard from a [UNKNOWN] competition.

And so they often give you benchmarks, marking grey here on this plot

that shows what happens if you say make all the values equal to 0.

And it's a good idea to know

what the prediction benchmark are because sometimes if

you prediction algorithm is performing way better than it should be or way worse than

it should be, the benchmarks will give you some idea of what you should be

doing wrong, and they'll tell you if

your prediction algorithm is kind of going astray.

So this is the studied design that was used in the Netflix prize, so they had

a 100 million ident-user-ident pairs, in other words,

movies that people had given their preference on.

They split that into a training data set, which they were going

to give to, users that we're going to build models for prediction.

Then, they held out, a bunch of ratings that we're not going

to be shown at all to the people that are building the models.

And so, what they provided for people who were building motels,

was a training set, and what they called a probe data set.

So, you would train your model on the training data

set and then you would apply it to the probe

data set to get some idea of what the out

of sample error would be, before submitting it to Netflix.

Then what Netflix would do is they would take your

predictions and they would apply it only to a quiz set.

So this quiz set you didn't get at get to actually see, and you couldn't build your

model on, but if we give you some better

idea of how well your model would perform at

a sample, but in general people could submit multiple

submissions to this quiz set and so they might

actually tune their models a little bit and get

a little bit better performance on the quiz set.

And so what they did ultimately was for the

final evaluation of all the different teams, they applied the

model just one time to this test set that

was held out to the very end of the competition.

And so at the very end of the competition

they got an unbiased estimate of how well this model

would work on a completely new set of data that

the participants in the competition never had a look at.

So this idea was actually very important in their study design and it

actually turned out that some teams that did better on the quiz set, actually

didn't do quite as well on the test set and that was because they

were tuning in their models or over fitting their models to the quiz set.

So this is an important take home message for

you, in the sense that if you're building prediction

models, you always have to hold one data set,

and leave it completely aside while you're building your models.

This is now used by group professionals so

Kaggle is a company that actually runs lots

of co, competitions for prediction competitions for a

whole bunch of different data sets provided by companies.

And they always do a similar model in the sense that they always have a leader board

that consists of the predictions that people have

submitted for a validation data set that they have.

But it's not necessarily the data set that they'll

use to validate their methods at the very end.

That data set is held out until the very end, and each

person gets to apply their algorithm to that data set only one time.

Something to keep in mind is that when you're splitting your

data sets up into training, testing and validation sets, they can

get a little bit small, but you need to avoid small

sample sizes, particularly if you're dealing with the test set size.

And the reason why is, suppose you were predicting a binary outcome, so in my

case, a very common thing to try to do is to predict diseased versus healthy.

And in general, it might be something like, whether people will

click on an ad, or whether they won't click on an ad.

Then one classifier is just flipping a coin.

You could always just flip a coin, and say, they'll be diseased if

the coin is heads, and not diseased if the coin comes out tails.

And so the probability of a perfect classification, using this

really silly algorithm is one half raised to the sample size.

In other words, half the time you'll be

right, just by chance by flipping the coin.

And each time, supposing each prediction is independent,

then each time that you flip a coin, then

you'll get one half times that, a number will

be the decrease in accuracy that you would get.

So if you were pr-, test set has only one sample in

it, then you have about a 50/50 chance of getting that sample right.

So, even if you got prediction accuracy of 100% on the test set,

you would have a 50% chance of that, even with a coin flip.

With n equals 2, you only have a, you

still only have a 25% chance of 100% accuracy.

And with n equals 10 in your test set, now

you have, only about a .1% chance of getting 100% accuracy.

So if you see that 100% accuracy you'll feel a little bit

more confident that it's actually true and it's not something that's just random.

So this suggests that we should make sure

that especially our test sizes are of relatively

large size so we can be sure that

we're not just getting good prediction accuracy by chance.

So some rules of thumb, these are by

no means set in stone, but they are reasonable

rules of thumb that I've used and I think a lot of people have used similar ones.

So you set, when you get a new data set,

if it's large enough, you'll set 60% of your data set

to be training, 20% of your data set to be

test, and 20% of your valid, data set to be validation.

This is again assuming that your test and validation

data sets won't be too small if you do that.

If you have a medium sample size what you might do is you might take 60%

of your data set to be training and 40% of your data set to be testing.

This means you don't get to refine your models in

a test set and then apply them to a validation set.

But it might insure that your testing site is of sufficient size.

Finally, if you have a very small sample size.

First of all, you might reconsider whether you have enough samples

to be able to build a prediction algorithm in the first place.

But suppose your dead set on building a prediction or machine learning

algorithm, then the idea might be to do cross validation and report the

caveat of the small sample size and the fact that you never got

to predict this in an out of sample or a testing data set.

So

some principles to remember are the test set or the validation set

should be set aside and never looked at when building your model.

In other words, you would need to have one

data set which you apply only one model to, only

one time, and that data set should be completely

independent of anything you use to build the prediction model.

In general you want to randomly sample the training and test set and

random might depend on the type of sampling that you want to do.

So for example, if you have, time fit time force

data, in other words you have, data collected over time, you

might want to build your, training set in chunks of time, but

again, random chunks of time and build them on random predictions.

Your data set much reflect the structure of the problem.

In other words, if you want to sample any data set that might have sources

of dependence over time or across space, you need to sample your data in chunks.

This is called backtesting in finance.

And it's basically the idea that you want to be able

to use chunks of data that consist of observations over time.

All subsets should re, reflect as much diversity as possible.

If you do random assignment, it does this.

You might also try balancing by features.

This can be a little bit tricky, but it often is a useful idea.

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

Coursera 致力於普及全世界最好的教育，它與全球一流大學和機構合作提供在線課程。

© 2019 Coursera Inc. 保留所有權利。

通過 App Store 下載

通過 Google Play 獲取

Coursera
關於
管理團隊
工作機會
目錄
證書
MasterTrack™ 證書
學位
企業版
政府版

社區
Learners
Partners
Developers
Beta Testers
Translators

連接
博客
Facebook
領英
Twitter
Instagram
YouTube
技術博客

更多
條款
隱私
幫助
內容訪問
媒體
聯繫我們
目錄
附屬公司