We asked if we have more than one sample, the test we've seen so far have just been for one sample of preferences in this case, but what if we have more than one? And also within those different samples, might we not have more than two response categories possible? So for these, we're going to need two sample tests of proportions. Or in general an end sample test to proportion. And we can go through that here when we revisit our data file that had preferences for website A or B. But now, we also add to that the sex of the respondent, whether they were male or female. So let's go ahead and load prefs AB sex which will give us that information and we'll see if there're differences in preferences by male or female. We can view that file as is good practice here and we can see again, it's nice to have our subject column on the left where we can make sure that each subject is responding once. We have the preference and that's the same as before. And now we have their sex as well, male or female. So we have two preference categories and of course two sex categories there. As is good practice we'll recode subject as a factor rather than a numeric value since it's just a number or thinks it's just a numeric value. Incidentally if you want to avoid that you can give a letter in your subject identifiers. So you might want to do s1 through s60, rather than just 1 through 60 and then you don't need to worry about that. It's not an issue for these analyses, we're just doing it here for good practice. But in future analyses we will use subject directly in our formulas. Let's go ahead and get a little summary as we've done. So we see there's again 14 preferences for A. 46 preferences for website design B. Just like we had before, but now we know there are 31 females and 29 males in the response. So lets go ahead and take a look at the graphs that show those responses by sex. So the way we do that is we ask for a plot, and we ask for it over this data. And we reach inside the table with these brackets and we say we want the rows where the sex column, the row value equals in the first case M for male. And then we ask for the preferences of those rows, so that's what the notation's doing there. Remember the dollar sign kind of reaches in to a field inside. And so, you'll notice also there's a comma and nothing after it here. It's a row and then a comma and then a column. We want all rows, but if you needed to specify a specific cell in a table, that's how you would do that. You do need that comma, or it won't work. So let's go ahead and look at the preference for males, and you can see the graph there. Males seem to prefer website B more. Of course, we don't know if that's statistically significant. Let's check out for females. And we can see, there was a pretty dramatic drop in their preference for website A there, and that would be certainly very interesting to find. Maybe there's something about it that doesn't make females happy about the design or it's less usable for some reason. Of course, that would be pretty interesting to find. So we can do again our Pierson chi Square test, now its a two sample test to proportions. For that we're going to use the x tabs function again to create a cross tabs or cross tabulation table. And they're more interesting now because there are two samples at work rather than just one. So let's go ahead and view that in the Viewer. And we can see that, for preferences for A and B, here on the left, and for each sex we can see the four cells. And usually these tables are drawn out as a kind of two by two grid. You can do it that way as well. And so we can see here's the preference that females had for website A is only a count of two. So, obviously very small, so that's the kind of interesting change here. So, we can run our chi squared test as we've done before, and I'll take this opportunity to point out that some of you may be used to, in programming languages, the period, or the dot, being a scoping operator. kind of like the dollar sign is here, where we can reach inside that data table and specify say a column or a row. The dot in R is just a character. It's like an underscore if you're used to C programming and its derivations. So if I double click, for example, on this term, it highlights the whole thing. It doesn't scope to just one of the words or the other because the period is just, in fact, another character. That takes some people some getting used to if you're used to other programming languages. Okay, so we'll go ahead and run the chi square test. And here we see a result that is statistically significant. And what that tells us again is just that are some differences here. And we would need to follow up with a further analysis if we wanted to see exactly where those differences lay. But we can that in fact there is a difference in the preferences. And from the graphs we can pretty easily tell that difference is strong in large part because of the female's lack of preference for website A. Now another test that's catching on and becoming more popular is the G-test, which is like the chi squared test. It is an asymptotic test as well. But it's meant to be more accurate, and it's more of a kind of newer version of a similar thing. We can access that test in this library, rvaidememoire. A little bit hard to say there, but we can load that in. And because we've done the work to assemble this PRFS variable, we can run a G test directly. And we can see that it produces a similar result to what we've seen before with a two sample chi square test. Now there's also an exact test, just like we used the binomial test in the past and the multinomial test, but those only work for one sample. We can use Fishers' exact test. Originally developed for two by two tests like we have here, but actually can be generalized to kind of r by c, that is row by column, of any number becomes somewhat computationally intensive as those grow, but for most of the kinds of tests you would be looking at, if you have a small number of samples, then this will work well. And the R version of this test is already generalized to be on just two by two tables. So we can run Fisher's exact test as well and get some output there. Showing an exact P value here of 0.0018. Clearly below 0.075, confidence interval, and some other output there. All of those are alternatives to looking at ways of analyzing a two sample test. Now what happens again if we have three response categories? Well, here again we can go back to our data with three response categories, press a, b, c but encode the sex column as well, so we have the sex of the respondent. And so, we'll load, press ABC sex, and we'll continue with our usual process of viewing that. And we can see here that we have the preferences as before, but the sex column as well. Always a good idea to scroll to the bottom to make sure you have the number of rows that you expect, and we do with 60. Go ahead and recode our subject factor as we've done. And look at a brief summary here. We can see have the 8 21 and 31 accounts for preferences of website A B and C. And the sex of 29 females 31 males. Okay, so let's go ahead and do a Pierson chi squared test just like we've done before. So now this should be familiar to you. We'll create the cross tabulation. We'll take a quick view of that. And you can see it's a little bigger because we have three categories. So this would be a three by two contingency table or cross tabulation. And that's how it usually would be drawn out is as a three by two grid in this case. And then we can run the chi square test and see that in fact there is, we get a little warning here about the approximation and that's what I mentioned. It's not an exact test. But we can see at least with this result we do have a P value of less than 0.05. So we can also run our G test. We can see it's a 0.02. And, then the exact test will give is the exact P value for Fisher's exact test, 0.03. So, when all three tests are similar, it does give us some confidence, about what we're seeing here. We can also do these post hock tests because recall that a test over these values would only tell us that there's some difference in the table but if we want to look further we can see where those differences lie. So we can run this post hock binomial tests to see how much do the responses differ by chance? We can see for this that within we can test within males and within females, and we can see the preferences within males for A, B and C website, and within females for A, B and C website against all of their rows. So, we're going to go ahead and do those. And then, as before, we'll do a post hock adjustment of each of those p-values by the home sequential Bonferroni procedure. An adjustment for multiple comparisons. So if we take a look there and we get this output, this bottom row here tells us the three different P values after adjustment, not so males who preferred A, were not statistically significant from chance, neither for B, but for website C they were. And we can go back and look at the counts to see that they preferred C highly. We can do the same for females and see that they were statistically significantly different from chance in their lack of interest in website A, but not for website B, and not for website C. That's where the significant differences lie. Let's go ahead and go back now to our table of analyses. And see that throughout this process of analyzing user preferences, counting samples of their preferences and their sex. We've actually now done our third row here. We had more than one sample. We had in some cases more than two response categories. We've looked at an N-sample chi squared test. In this case a two sample. The G-test and Fisher's exact test. And that covers our test of proportions. Next we'll be looking at analysis of variance, which will be more common when we run experiments where we have subjects give us more than their preference. They actually perform, do tasks, we measure those and then we go about analyzing the results. We'll come to that next.