Multiple comparisons Theory

Discussion in 'Probability' started by Kerrio Brown, Oct 4, 2011.

  1. Kerrio Brown

    Kerrio Brown Guest

    I had a disagreement with someone about how much to correct for some
    results and I wanted to see if I could get some insight before
    discussing again. Here was my thinking:

    I have a very large analysis which makes a number of independent
    claims based on what was significantly different (p<0.05). Assume all
    of my variables are independent so I use Bonferroni. The question was
    whether or not I correct over sections of my results that have nothing
    to do with each other. For instance, let's say I compare 3 tree types
    to each other, Tree A vs B, A vs C, and B vs C. In another section, I
    also check if 3 bush types are bigger than each other (D,E,F). Do I
    correct all comparisons w/ a factor of 3 or 6?

    My thinking was, if I am making 6 claims, then I'd expect 5% of this
    total to false positives. Therefore, I need to correct with a factor
    of 6 for my example. The issue is I actually have 307 claims (i.e. 307
    times that I check for a significant difference) in my project. If I
    use p<0.05, then I'd expect to get about 0.05(307)= ~15 false
    positives in my entire analysis. So, I'd want to correct using a
    factor of 307 (To be clear, I made sure to only test things where
    actually finding a significant difference would be worth reporting, so
    I already filtered out things that I wouldn't end up "claiming").

    I thought for awhile before doing using 307 as my factor, and it just
    seemed to me that how related the conclusions are doesn't matter. If I
    put out 10 papers on completely different subjects and didn't correct
    for multiple comparisons, I'd expect the number of false positives
    within those 10 papers to add up based on their entire sum. So why
    should I isolate individual results to choose my correction factor?
    This of course leads to absurdity since then all scientific claims
    ever made would need to be corrected together.

    Now I know I'm thinking wrong here, and it has to do with the
    perspective I am taking. For instance, it doesn't make much sense to
    correct all 10 paper's claims unless I'm making some sort of meta-
    claim about all of them together. But I can't seem to follow my own
    thoughts through.

    Can someone help me correct my thinking?

    Thanks,
    kb
     
    Kerrio Brown, Oct 4, 2011
    #1
    1. Advertisements

  2. Kerrio Brown

    Ray Koopman Guest

    Look up "false discovery rate"
     
    Ray Koopman, Oct 6, 2011
    #2
    1. Advertisements

  3. Kerrio Brown

    Kerry Guest

    I've used FDR many times particularly where the variables used in the
    comparisons have some dependencies. If I can safely assume
    independence among my variables (e.g. centipedes and bears), does FDR
    in any way provide an explanation to my question about what should be
    contained in the set of comparisons I correct for? If so, then it'd be
    nice to know how since I will certainly end up using FDR in some cases
    (I used Bonferroni in my post to keep things simple)?

    Thanks,
    kb
     
    Kerry, Oct 14, 2011
    #3
  4. Kerrio Brown

    Rich Ulrich Guest

    [snip, much]
    kb >
    Do take note that there are several varieties of FDR available for
    use, and they are *not* equivalent in assumptions or stringency.

    The original, by Benjamini et al, is fairly adaptable for clinical
    research. The others that I have seen available for use on-line,
    or downloadable, need to be assessed for their own merits.

    A few years ago, I was asked to make use of one cited version,
    so I did. That one seemed to be useful for something like
    Radio Signal Analysis of corrupted signals, but *not* desirable
    for evaluation of scientific hypotheses. The evidence -- when I
    supplied a set of test results where (as I recall) about 50% of
    the tests rejected nominally at the 5% level, the FDR procedure
    gave me "adjusted p-values" that were under 5% for almost
    everything else, including a test where the nominal p-value
    was 0.70. That's "0.70", not "0.07" -- the raw correlation
    to be assumed as significant, "consistent with the rest", had
    the wrong sign.

    The PI agreed with me, that our audience would never consider
    this as appropriate.
     
    Rich Ulrich, Oct 15, 2011
    #4
  5. Kerrio Brown

    Kerry Guest

    Is there a correction method that simply multiplies the Bonferroni
    correction factor by the correlation between variables in order to
    account for dependence? For instance, if one test was A1 vs B1 and
    another was B1 vs B2, then I could find the correlation b/t A1 and A2
    and the correlation b/t B1 and B2, take the average and multiply 1-
    Pearson's R with the correction factor (2 in this case) to get the
    dependence-adjusted correction factor.

    kb
     
    Kerry, Oct 15, 2011
    #5
  6. Kerrio Brown

    Kerry Guest

    addendum to last post: ...assuming the sample size is large enough for
    the correlation value to be informative. In fact, it may make more
    sense (if it makes sense at all) to only use significant R values, and
    treat the non signficant R values as 0. And using whatever correl
    method is appropriate (e.g. Pearson, Spearman, Kendall)

    kb

    kb
     
    Kerry, Oct 15, 2011
    #6
  7. Kerrio Brown

    Rich Ulrich Guest

    That sounds over-simplified, but I'm no expert on all the correction
    methods. And I don't know whether you are referring to FDR, which
    preserves "false conclusions", or to the Bonferroni intent of
    preserving the alpha level ("false positives"). FDR methods
    certainly make use of a correction factor based on correlation --
    usually, the average correlation. (And, how consistent is that
    average?)

    There are people who make EEG maps which highlight "significant"
    regions, using proprietary algorithms on their name-brand machines.
    I read something non-formal on that in CHANCE, which suggested
    that Bonferroni (or similar) was used on Principal Components, and
    some other correction was used (based on average correlation)
    within each component. That's what I recall, from what I read
    quite a while ago.
    That sentence has at least one typo and one think-o, so I
    can't have any confidence in how I read it. But I think the
    answer for any two-test case is probably No, you can't use
    any correlations to reduce the correction. - If the variables
    are really highly correlated, I would guess that you probably
    should be testing an overall hypothesis based on composite
    scores or a composite test.
     
    Rich Ulrich, Oct 15, 2011
    #7
  8. Kerrio Brown

    Ray Koopman Guest

    You may find a similar discussion in another group interesting:
    https://groups.google.com/group/medstats/browse_frm/thread/ee604b6699d461f1?hl=en#
     
    Ray Koopman, Oct 16, 2011
    #8
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.