Permutation Test for Matched Pairs Data

Permutation tests can also be used on matched pairs data. Instead of randomly assigning all of the results to the two groups, you need to keep the pairs together. So we randomly assign one outcome from each pair to the two groups. This is the same as randomly choosing the sign (positive or negative) for each individual difference.

Example: Helium Filled Footballs

Some people believe that footballs filled with helium will travel farther than regular footballs when kicked. The Columbus Dispatch did an experiment to find out. Two identical footballs were tested, one filled with helium, the other with ordinary air. A novice kicker punted the two balls several times without knowing which ball was which. In total, each ball was kicked 39 times and the distance travelled was measured in yards. The results are in the data frame football.

football = read.csv("http://people.hsc.edu/faculty-staff/blins/StatsExamples/football.txt")
head(football)

##   Trial Air Helium difference
## 1     1  25     25          0
## 2     2  23     16         -7
## 3     3  18     25          7
## 4     4  16     14         -2
## 5     5  35     23        -12
## 6     6  15     29         14

hist(football$difference,col='gray')

qqnorm(football$difference)
qqline(football$difference)

With this data, we could probably trust a two-sample t-test. But a permutation test is another option.

Step 1

The mean difference in our sample was 0.4615385 yards.

Step 2

Now we will simulate many different trials where the differences are the same, but the which of the two balls goes farther (helium or regular) is randomized.

results = c()
for (i in 1:5000) {
  permutedData = sample(c(1,-1),39,replace=T)*football$difference
  results = c(mean(permutedData),results)
}
hist(results,col='gray',main="Permutation Distribution",xlab="Simulated difference")

Step 3

The p-value is:

mean(results >= 0.4615385)

## [1] 0.3346

That’s not even close to being statistically significant. Using the t-test, we get almost the same p-value.

t.test(football$difference,alternative='greater')

## 
##  One Sample t-test
## 
## data:  football$difference
## t = 0.41976, df = 38, p-value = 0.3385
## alternative hypothesis: true mean is greater than 0
## 95 percent confidence interval:
##  -1.392221       Inf
## sample estimates:
## mean of x 
## 0.4615385

Think of the permutation test as a way to double check the t-test.