In the previous lecture, I referred to John Loannidis' suggestion that most scientific results are false. Now I want to explore the factors that have given rise to this pessimistic view. So let us look at one of the main problems. Nowadays, many experimental tests depends on Statistical Evidence. When we pose a question to nature, nature hardly ever says yes or no or even probably yes or probably no. But rather, something like, if the effect that we are looking for does not really exist, the probability of getting these results or even stronger ones is pretty small. Now this probability of a so-called false positive of getting results, suggesting an effect that isn't really there, is known as the p-value. In many fields, if the p-value is equal to or smaller than 5%, the result is considered significant. In other words, we conclude that the effect is real, but such a conclusion may be highly premature. For instance, you would expect that, in general, only 5% of all published results is false. But this is not the case, as a simple example can show. Suppose that of all the hypotheses, we would like to test 1 in 10 is true. Now imagine tests on 1,000 hypotheses, 100 of which are true, and therefore 900 are false. With a false positive rate of 5%, we may expect 5% of 900, or 45 false positives. Suppose, furthermore, that the tests have a power of 80%. That is, they confirm 80%, in this case, 80 of the 2 hypotheses, thereby producing 20 false negatives. Not knowing what is false and what is true, the researchers will conclude that 125 hypotheses are true, 45 of which are not. As negative results are rarely published, the proportion of the published results, which is false, will be much higher than 5%. In this case, even more than 30%. Now, in reality, the situation is probably even worse. As researchers are only human, they tend to be liable to what is known as confirmation bias. In other words, they tend to find what they expect or hope to find. Scientists have to make constant judgments about reliability of their data during the experiments. Results often evoke more trust if they fit theoretical predictions or expectations. Whereas in case of unexpected results, we are more liable to re-run the experiment or to find reasons why we should actually discard the unwelcome results. And this, of course, may lead to selective use of data. Confirmation bias may affect the judgement of the most scrupulous scientists. Finally, not all scientists are that scrupulous. Again, being only human, they may succumb to what is known as Publication Pressure. As we have seen, they need lots of publications in order to further their career. Now in general, scientists need to produce positive results in order to be able to publish their results. It is, therefore, tempting to slightly improve your results, so as to make them publishable. And we have seen a clear example of such behavior in the film where Ponter instructs Rebecca to polish her results. Everybody does it. Now there are many ways to improve your results, even without committing fraud. That is, without fabricating or falsifying data. Some ways of polishing your data are known as p-hacking. There are many different varieties. They usually involve selective use of data. For instance, we can test a huge number of hypotheses about a single data set by exhaustively searching for combinations of variables that show a meaningful correlation. This is also known as data dredging. We can also try different tests on the same hypotheses, but only report the most significant test. Or we can only analyze an interesting subset of the data in which a pattern can be found. And finally, we can analyze a data set and decide on whether to collect more data or not depending on the data collected so far. Now such practices are far from rare, as recent surveys have shown. They are known as questionable research practices or 'sloppy science'. This is the gray area between fraud and responsible research conduct. So maybe the suggestion that most published results are false is not that far off the mark. Indeed, recent attempts to reproduce generally accepted results in social sciences and biomedical sciences failed in more than half of the cases. Still, as fallible as science may be, it is still the best way we have to produce knowledge. But there's always room for improvement. In fact, many steps are taken to improve the situation. One way to do so is to make it easier to publish negative results. Another one is to put less emphasis on the number of publications as a quality measure. Above all, as scientists, we need to be honest to others and to ourselves.
