Permutation Tests

The permutation test is an alternative to classical hypothesis testing methods. Permutation tests use computer simulations to find a p-value instead of relying on the theory of probability distributions like the normal distribution. You can use permutation tests in the following situations:

To compare two samples
To compare matched pairs data
To test for association between two variables

Here we’ll focus on the first case: comparing two different samples.

Steps

Here are the steps for a permutation test.

Calculate the statistic of interest.
Take all of the results in both groups and simulate randomly reassigning them to the two groups. Do this many times and keep a record of the outcomes for the simulated statistic of interest.
The p-value is the percent of simulations with a statistic of interest as extreme or more than the actual value calculated in step 1.

Advantages

Permutation tests don’t make any assumptions about the distribution of the data in the population.

Disadvantages

The null hypothesis in a permutation test is more specific. You have to assume that both samples have the exact same distribution (same shape, center, and spread) not just the same value for one parameter. A permutation test assumes that the only difference between the two groups is the random assignment. This is often appropriate in experiments.
Permutation tests also require a computer.

Main Application

Permutation tests are often used to double check other more traditional hypothesis test methods. If both tests are significant, then you can be pretty confident about your results.

Example: Dolphin Therapy

A 2005 study looked at whether swimming with Dolphins for two weeks in Honduras was more beneficial for patients with clinical depression than swimming a similar amount in Honduras, but without dolphins. A group of 30 volunteers with clinical depression were randomly assigned to either swim with dolphins, or to swim without. After two weeks, they were tested to see if their symptoms of depression had improved. Here are the results from the study.

dolphinData = matrix(c(10,3,5,12),ncol=2,byrow=T)
colnames(dolphinData)=c('Dolphin Therapy','Control Group')
rownames(dolphinData)=c('Depression Improved','Did not improve')
dolphinData

##                     Dolphin Therapy Control Group
## Depression Improved              10             3
## Did not improve                   5            12

barplot(dolphinData,legend=T)

Permutation Test

Step 1

We need to find the statistic of interest. Since we have a categorical variable (depression status) and two groups, a natural choice would be the difference in proportions between the two groups: \(\hat{p}_\text{dolphin} - \hat{p}_\text{control}\). Calculate this!

Step 2

Imagine that 13 of our subjects were going to improve, no matter what. Dolphin therapy had no effect. This basically describes our null hypothesis: dolphin therapy is no different than the control for improving depression symptoms. So if we have these 13 people who will improve, we can simulate how likely it is for 10 of them to end up in the dolphin group. You can randomly permute the order of data in a vector by using the sample command as follows:

# The experiment had 13 success and 17 failures:
outcomes = c(rep(1,13),rep(0,17)) 

# A sample of the same length as the vector randomly permutes the entries:
simulatedOutcomes = sample(outcomes,length(outcomes)) 
simulatedOutcomes

##  [1] 0 0 0 0 1 1 1 1 0 0 1 1 0 0 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1

Write a for-loop to simulate 5000 different permutations of the results. Divide the results into two groups of 15 and record the simulated difference in proportions. Hint: to divide the entries of a vector with 30 entries into two groups of 15, you can use something like:

group1 = simulatedOutcomes[1:15]
group2 = simulatedOutcomes[16:30]

Step 3

What percent of the 5000 the simulated statistics were as extreme or more than the actual results of the experiment? That’s our p-value!

Example 2: Cloud Seeding

Permutation tests also work with quantitative data. Take a look at this example: Cloud seeding.