Hi, in the previous module we started talking about the basic steps that you need to do in order to perform an MVPA analysis. In this module we'll continue. So recall that the process of performing an MVPA analysis follows a series of steps. The first is defining features and classes. The second is feature selection. The third is choosing a classifier. The fourth is training and testing the classifier. And the final step is examining the results. So we talked about the first three steps in the last module and this time we'll talk about the final two steps. So to accurately asses the performance of a classifier when applied to a new data set, it's critical to use separate data to train and test the classifier. Ideally, we'd like to train a classifier using as much of the available data as possible. However, this is going to leave very little to test with. One approach to balance this problem is to use cross validation. So k-fold cross-validation is performed in the following manner. We first divide the data set into K different parts, or folds. We then leave one fold out, and then train on the remaining K-1 folds. We predict the observation on the omitted fold, and we repeat for each fold in turn. So using cross-validation approaches, each part of the data is is is at some time in the test data set. And multiple times in the training data set. And then finally, we compute accuracy of all the predictions made. If unbiased accuracy assessment is important, then we tend to want to use more folds. because there's less error using large training data. If on the other hand, parameter optimization is important, then we tend to use fewer folds. We get a more stable estimate of the area using large test data. In practice, 5 or 10 folds is often used as a compromise. When performing cross-validation, it's important that each fold contains observations from each class. For example, if we're doing a classification of faces versus houses, you want each fold to contain examples of both faces and houses. The classes should be roughly balanced in the cross-validation procedure. And stratification can be used to guarantee that each class is adequately represented. I want you however to be careful to include correlated observations in the same fold because otherwise you may bias your test data set. Cross-validation provides a method for choosing between different types of classifiers and determining certain parameter estimates. Final classifier weights can be obtained in a number of ways, such as averaging weights across folds. We often want to determine the accuracy of a classifier to determine whether it works better than chance. A simple approach is just to use simple binomial test with p(success)=0.05 per trial. In the case that we have two different observations. So, if our classifier doesn't do better than a coin flip in separating houses versus faces, then it's really of no use to us. A more accurate way to quantify performance is to use resampling methods. It's important to examine the accuracy for all classes of observations, rather than computing the overall accuracy across observations. If almost all of the observations fall in the same class, then assigning all new observations to that class will tend to give a high overall accuracy. For example, if 90% of all observations fall in class A then a classifier that always assigns new observations to this class will have a 90% overall accuracy. It will also have a 100% accuracy of correctly classifying class A observations. However, it will have 0% accuracy of correctly identifying class B observations because the classifier always tells you be in class A. So this is not so good. An important question is determining which voxels drive the classification. And so the classifier weights can be mapped back onto the brain to provide information about each voxels contribution to the classifier performance. So, different classifiers may provide different maps as they're sensitive to different features in the data. So, in this particular example, we might have the w here, which are the classifier weights, which again, let's just say for sake of argument that it's a vector of length v, where v corresponds to all the voxels of the brain. We can then map this back on to the brain to see the relative contribution of each voxel. And we can use this for interpretation purposes. In fMRI it is important to make analysis choices that balance interpretability with predictive power. Certain methods may give good predictions but the resulting voxel weights maybe difficult to interpret and may not generalize to new subjects. So we always want to balance those things to have a classifier that has a good accuracy with one that tells us something about the underlying neural science. Okay, so that's the end of this module and this was the second module on how to perform an MBPA analysis. Okay, thank you, bye.
