Saturday, March 14, 2015

What is Evidence?

In God we trust, all others bring data (Edward Deming).

I was discussing testing hypotheses with Damoon several days ago and I thought it is not clear how one can judge a theory using random observations. I came up with a simple example and thought better to share it with you.

It is counter-intuitive how we can extract evidence from data, if somebody brings it. There are many paradigms, I do not want to go into the details to make long and boring statements. I suggest the classic example: tossing a coin.
Lets ask a simple question: what sort of data is an evidence against fairness (or unfairness) of a coin? From fair I mean 1/2 chance of getting a Head, and 1/2 chance of getting a Tail.

Suppose somebody tosses a coin 4 times and gets:  Tail Tail Tail Tail
Such a coin looks suspicious, right? It seems we tend to believe the coin produces more Tails than Heads.

It is widely accepted  we vote against a theory (a theory sometimes is called assumption, sometimes called hypothesis) that produces suspicious results; from the result I mean data.

To have a better understanding of a suspicious result, lets compute what is the probability that a fair coin gives 4 Tails in 4 trials. 
(1/2)^4 ~ 0.06

Usually the threshold between being suspicious and being evidence is 0.05 (sometimes this value is 0.01 if a scientist is conservative). This quantity is related to p-value and testing statistical hypothesis. Interpretation of p-value is difficult, if you are interested, see this paper

Therefore, a scientist lives with the fair coin hypothesis if tosses a coin 4 times and gets 4 tails.

Now suppose we toss the coin five times and get Tail Tail Tail Tail Tail. Then what would be the decision of a scientist? Such data can be produced under the fair coin hypothesis with probability
(1/2)^5 ~ 0.03<0.05. So a scientist will believe that the coin is unfair!

I suggest you to toss a coin 5 times, I bet you get at least one Head or Tail, try it if you do not believe me.


  1. I made a simple C++ program that generate 1 billion 0 or 1 value with a 0.5 bernoulli distribution. Then a counted the amount of time the same value is repeate in a row.
    Here's the result. It turns out that 5 times in a row happens pretty often with a large sample set.
    1 : 250002548
    2 : 124999371
    3 : 62502606
    4 : 31250409
    5 : 15623294
    6 : 7812596
    7 : 3906084
    8 : 1952366
    9 : 976342
    10 : 488475
    11 : 244310
    12 : 122270
    13 : 60995
    14 : 30522
    15 : 15302
    16 : 7628
    17 : 3882
    18 : 1873
    19 : 945
    20 : 491
    21 : 216
    22 : 126
    23 : 59
    24 : 21
    25 : 16
    26 : 9
    27 : 3
    28 : 1

    The question becomes how much are you ready to bet that i get at least one Head or Tail if i throw it 5 times.

    1. Hey Alex, thanks for the comment: the probability of having the same value must be 2*(1/2)^n, where n is the number of times you run the experiment.