Multiple comparisons Theory

I had a disagreement with someone about how much to correct for some
results and I wanted to see if I could get some insight before
discussing again. Here was my thinking:

I have a very large analysis which makes a number of independent
claims based on what was significantly different (p<0.05). Assume all
of my variables are independent so I use Bonferroni. The question was
whether or not I correct over sections of my results that have nothing
to do with each other. For instance, let's say I compare 3 tree types
to each other, Tree A vs B, A vs C, and B vs C. In another section, I
also check if 3 bush types are bigger than each other (D,E,F). Do I
correct all comparisons w/ a factor of 3 or 6?

My thinking was, if I am making 6 claims, then I'd expect 5% of this
total to false positives. Therefore, I need to correct with a factor
of 6 for my example. The issue is I actually have 307 claims (i.e. 307
times that I check for a significant difference) in my project. If I
use p<0.05, then I'd expect to get about 0.05(307)= ~15 false
positives in my entire analysis. So, I'd want to correct using a
factor of 307 (To be clear, I made sure to only test things where
actually finding a significant difference would be worth reporting, so
I already filtered out things that I wouldn't end up "claiming").

I thought for awhile before doing using 307 as my factor, and it just
seemed to me that how related the conclusions are doesn't matter. If I
put out 10 papers on completely different subjects and didn't correct
for multiple comparisons, I'd expect the number of false positives
within those 10 papers to add up based on their entire sum. So why
should I isolate individual results to choose my correction factor?
This of course leads to absurdity since then all scientific claims
ever made would need to be corrected together.

Now I know I'm thinking wrong here, and it has to do with the
perspective I am taking. For instance, it doesn't make much sense to
correct all 10 paper's claims unless I'm making some sort of meta-
claim about all of them together. But I can't seem to follow my own
thoughts through.

Can someone help me correct my thinking?

Thanks,
kb

 

Look up "false discovery rate"

 

I've used FDR many times particularly where the variables used in the
comparisons have some dependencies. If I can safely assume
independence among my variables (e.g. centipedes and bears), does FDR
in any way provide an explanation to my question about what should be
contained in the set of comparisons I correct for? If so, then it'd be
nice to know how since I will certainly end up using FDR in some cases
(I used Bonferroni in my post to keep things simple)?

Thanks,
kb

 

[snip, much]
kb >

Do take note that there are several varieties of FDR available for
use, and they are *not* equivalent in assumptions or stringency.

The original, by Benjamini et al, is fairly adaptable for clinical
research. The others that I have seen available for use on-line,
or downloadable, need to be assessed for their own merits.

A few years ago, I was asked to make use of one cited version,
so I did. That one seemed to be useful for something like
Radio Signal Analysis of corrupted signals, but *not* desirable
for evaluation of scientific hypotheses. The evidence -- when I
supplied a set of test results where (as I recall) about 50% of
the tests rejected nominally at the 5% level, the FDR procedure
gave me "adjusted p-values" that were under 5% for almost
everything else, including a test where the nominal p-value
was 0.70. That's "0.70", not "0.07" -- the raw correlation
to be assumed as significant, "consistent with the rest", had
the wrong sign.

The PI agreed with me, that our audience would never consider
this as appropriate.

 

Is there a correction method that simply multiplies the Bonferroni
correction factor by the correlation between variables in order to
account for dependence? For instance, if one test was A1 vs B1 and
another was B1 vs B2, then I could find the correlation b/t A1 and A2
and the correlation b/t B1 and B2, take the average and multiply 1-
Pearson's R with the correction factor (2 in this case) to get the
dependence-adjusted correction factor.

kb

 

addendum to last post: ...assuming the sample size is large enough for
the correlation value to be informative. In fact, it may make more
sense (if it makes sense at all) to only use significant R values, and
treat the non signficant R values as 0. And using whatever correl
method is appropriate (e.g. Pearson, Spearman, Kendall)

kb

kb

 

That sounds over-simplified, but I'm no expert on all the correction
methods. And I don't know whether you are referring to FDR, which
preserves "false conclusions", or to the Bonferroni intent of
preserving the alpha level ("false positives"). FDR methods
certainly make use of a correction factor based on correlation --
usually, the average correlation. (And, how consistent is that
average?)

There are people who make EEG maps which highlight "significant"
regions, using proprietary algorithms on their name-brand machines.
I read something non-formal on that in CHANCE, which suggested
that Bonferroni (or similar) was used on Principal Components, and
some other correction was used (based on average correlation)
within each component. That's what I recall, from what I read
quite a while ago.

That sentence has at least one typo and one think-o, so I
can't have any confidence in how I read it. But I think the
answer for any two-test case is probably No, you can't use
any correlations to reduce the correction. - If the variables
are really highly correlated, I would guess that you probably
should be testing an overall hypothesis based on composite
scores or a composite test.

 

You may find a similar discussion in another group interesting:
https://groups.google.com/group/medstats/browse_frm/thread/ee604b6699d461f1?hl=en#