Family Wise Error Rate controlled by method of maximum statistics

user_1722950 0
Viewed 189

I am flipping each coin 100 times in a bag of 50 coins and then I want to use the Method of Maximum statistics in order to determine the Family Wise Error Rate. However, I keep getting an FWER of 1 which feels wrong.

  coins <- rbinom(50, 100, 0.5)

So I start by defining a new function where we input how many times we do randomizations, the coins themselves, and how many times we flip them.

simulate_max <- function(n_number_of_randomizations, input_coins, N_number_of_tosses, alpha = 0.05) {

  maxList <- NULL

Then we do a for loop for every time we have specified.

  for (iteration in 1:n_number_of_randomizations){

Now we shuffle the list of coins

  CoinIteration <- sample(input_coins)

Now we apply the binary test to every coin in the bag

testresults <- map_df(CoinIteration, function(x) tidy(binom.test(x,N_number_of_tosses,p=alpha)) )

Now we want to add the maximum result from every test to the max list.

  thisRandMax <- max(testresults$statistic)
  maxList <- c(maxList, thisRandMax)

}

Finally, we iterate through every member of the maximum list to subtract the expected value of heads (ie 50 for 50% chance * 100 tosses.

  for (iterator2 in 1:length(maxList)){

maxList[iterator2]<-maxList[iterator2]-(0.5*N_number_of_tosses)  
  
  }

Return the output from the function

return(data.frame(maxList))
}

Now we apply this simulation for each of the requested iterations.

repsmax = map_df(1:Nreps, ~simulate_max(Nrandomizations,coins,Ntosses))

Now we calculate the fwer by dividing the increased amount by the total number of cells.

fwer = sum(repsmax>0) / (Nreps*Nrandomizations)
statistics r
1 Answers

There are some issues that I think would be good to clarify.

A FWER of ~1 seems about right to me given the parameters of your experiment. FWER relates to Type I error, and for a single normally distributed test at alpha = 0.05, FWER = 1 - P(Type I error = 0); FWER = 1 - 0.95 = 0.05. For two tests at alpha = 0.05, FWER = 1 - P(Type I error = 0); FWER = 1 - 0.95^2 = 0.0975. You have 50 coins (50 tests), so your FWER at alpha = 0.05 is 1 - 0.95^50 = 0.923. If your code treats the 100 coins as 100 tests, your FWER will be = 0.996 (~1).

You can control for Type I error (account for multiple testing) by using e.g. the Bonferroni correction (alpha / n). If you change your alpha to "0.05 / 50" = 0.001, you will control your FWER (reduce it) to 0.05 (1 - 0.999^50 = ~0.049). I suspect this is the answer you are looking for: if alpha = 0.001 then FWER = 0.05 and you have an acceptable chance of incorrectly rejecting the null hypothesis.

I don't know what the "maximum estimate of the effect size" is, or how to calculate it, but given that the two distributions are approximately identical, the effect size will be ~ 0. It then makes sense that controlling FWER to 0.05 (by adjusting alpha to 0.001) is the 'answer' to the question and if you can get your code to reflect that logic, I think you'll have your solution.

user_129573400
Related