## Saturday, March 14, 2015

### What is Evidence?

In God we trust, all others bring data (Edward Deming).

I was discussing testing hypotheses with Damoon several days ago and I thought it is not clear how one can judge a theory using random observations. I came up with a simple example and thought better to share it with you.

It is counter-intuitive how we can extract evidence from data, if somebody brings it. There are many paradigms, I do not want to go into the details to make long and boring statements. I suggest the classic example: tossing a coin.
Lets ask a simple question: what sort of data is an evidence against fairness (or unfairness) of a coin? From fair I mean 1/2 chance of getting a Head, and 1/2 chance of getting a Tail.

Suppose somebody tosses a coin 4 times and gets:  Tail Tail Tail Tail
Such a coin looks suspicious, right? It seems we tend to believe the coin produces more Tails than Heads.

It is widely accepted  we vote against a theory (a theory sometimes is called assumption, sometimes called hypothesis) that produces suspicious results; from the result I mean data.

To have a better understanding of a suspicious result, lets compute what is the probability that a fair coin gives 4 Tails in 4 trials.
(1/2)^4 ~ 0.06

Usually the threshold between being suspicious and being evidence is 0.05 (sometimes this value is 0.01 if a scientist is conservative). This quantity is related to p-value and testing statistical hypothesis. Interpretation of p-value is difficult, if you are interested, see this paper

Therefore, a scientist lives with the fair coin hypothesis if tosses a coin 4 times and gets 4 tails.

Now suppose we toss the coin five times and get Tail Tail Tail Tail Tail. Then what would be the decision of a scientist? Such data can be produced under the fair coin hypothesis with probability
(1/2)^5 ~ 0.03<0.05. So a scientist will believe that the coin is unfair!

I suggest you to toss a coin 5 times, I bet you get at least one Head or Tail, try it if you do not believe me.

1. I made a simple C++ program that generate 1 billion 0 or 1 value with a 0.5 bernoulli distribution. Then a counted the amount of time the same value is repeate in a row.
Here's the result. It turns out that 5 times in a row happens pretty often with a large sample set.
1 : 250002548
2 : 124999371
3 : 62502606
4 : 31250409
5 : 15623294
6 : 7812596
7 : 3906084
8 : 1952366
9 : 976342
10 : 488475
11 : 244310
12 : 122270
13 : 60995
14 : 30522
15 : 15302
16 : 7628
17 : 3882
18 : 1873
19 : 945
20 : 491
21 : 216
22 : 126
23 : 59
24 : 21
25 : 16
26 : 9
27 : 3
28 : 1

The question becomes how much are you ready to bet that i get at least one Head or Tail if i throw it 5 times.

1. Hey Alex, thanks for the comment: the probability of having the same value must be 2*(1/2)^n, where n is the number of times you run the experiment.