What is this series

Discussion in 'Scientific Statistics Math' started by David W. Cantrell, Feb 16, 2006.

  1. David Winsemius, Mar 6, 2006
    #41
    1. Advertisements

  2. David W. Cantrell

    Reef Fish Guest

    That was more or less what DZ did, except generating the sequence
    himself.
    Yours seems like a hindsight solution. If I give you a different
    sequence of
    4 digits do you think your suggested method would work?

    Try this: 81 68 81

    What are the next two sets of two digits in the sequence? ::)

    -- Bob.
     
    Reef Fish, Mar 6, 2006
    #42
    1. Advertisements

  3. David W. Cantrell

    Reef Fish Guest

    on that ONE single trial, when all possible 366! permutations are
    which is EQUALLY LIKELY ?

    The question of what to do with randomization that turned out
    to "look odd" drew only one participant Bob Dole. So, that'll
    have to remain (probably forever) an unresolved question about
    Fisher's idea of "randomization" in the design of experiments.

    We had already given up on the discussion of "randomization" within
    Fisher's prerequisite notion of a Designed Experiment.

    In our previous discussions of various aspects of Multiple Regression
    Analysis, I deliberately chose to restrict the discussion to the
    seeking
    of models for the purpose of PREDICTION only, without opening the
    can of worms on how to use a designed experiment to "explain"
    any phenomenon, of to ascertain "cause".


    Savage said,

    I know Jerry Dallal (and probably others) are patiently waiting
    to discuss the "design" issue, especially relative to the analysis
    done in conjunction with regression data.

    So, I'll just leave the above cans of worms, with can-openers, for
    anyone to open their favorite can and expound on, their worms.

    -- Bob.
     
    Reef Fish, Mar 6, 2006
    #43
  4. David W. Cantrell

    Reef Fish Guest

    Sorry, that was only 3 numbers.
    I think either the database method failed or everyone gave up.
    This will be your last chance:

    81 68 81 27

    I'll even give you the ANSWER! How do you get the next two
    numbers in the sequence to be 46 and 21 ? :)

    Hint: This is both very advanced mathematics and very silly
    mathematics!

    -- Bob.
     
    Reef Fish, Mar 7, 2006
    #44
  5. To call it a "solution" praises it too highly. It happened to be the only
    offered hit from AT&T's Encyclopedia of Integer Sequences when queried with
    your sequence.
     
    David Winsemius, Mar 7, 2006
    #45
  6. David W. Cantrell

    Reef Fish Guest

    Thank you for this additional explanation. I wasn't paying much
    attention
    to your original suggestion. Now that you had my interest piqued to
    take
    a second look at this AT&T Encyclopedia of integerSequences search,
    I found first of all, that it failed in the example I gave,

    81, 68, 81, 27, 46, 21 ,,, no matter how many terms I gave or where I
    start.

    Then I put in simply the first pairs of digits of pi: 31,41,59

    and search found many unexpected results, such as:

    Primes with prime subscripts: 3, 5, 11, 17, 31, 41, 59, 67, 83, ...

    Primes p such that Fibonacci(p)-1 is divisible by p.
    2, 11, 19, 29, 31, 41, 59, 61, 71, ...

    before it found the digital expansion of pi.
    31, 41, 59, 26, 53, 58, 97, 93, 23, 84, 62, 64, 33, 83, ...

    The late Paul Erdos used ot offer $$ awards for his problems, ranging
    from a few dollars to a few hundred dollars, depending on how difficult
    he judged the problem to be.

    I hereby offer $20 to the first person who can even identify where
    this unsequenced number comes from:

    13816881274621 ...

    since the sequence search failed in 81, 68, 81, 27, 46, 21 ,,,
    no matter how many terms I gave or where I started

    If the sequence 18, 28, 18, 28, ... did not show the silliness of IQ
    type
    of questions for "fill in the next term", you can keep the present
    problem
    for a "bar bet" for $10 -- if someone identified it, then you get my
    $20,
    and still get to keep $10 after paying your losing bet. :)

    -- Bob.
     
    Reef Fish, Mar 7, 2006
    #46
  7. ++++++++++++++++++++++++++++
    Several points.

    1. I don't think there is any validity in arguing your misuses from a
    general philosophy viewpoint. The only time it has any meaning is in a very
    specific situation, a specific case.

    2. From your summary of Savage, I would say he was to harsh on Fisher. He
    takes a contemporary statistics viewpoint, when Fisher was working and
    devising methods well before contemporary statistics. Given Fisher's time
    and the dominance of Pearson's ideas and writings, Fisher did very well.

    3. Fisher's view of randomness was basic. It had to do with making
    assumptions about the behavior (and distribution) of the variations in a set
    of measurements. It was the classical view of probability in Gaussian terms
    and of observations on games. (chances of a certain card being drawn, e,g.
    deck shuffled as a meaning of randomness). Fisher was both fascinated by
    Bayes ideas and repelled by them.

    4. Fisher's pre-occupation with computing (or as a programmer) is no
    different from our contemporary fascination with computers and iPods and
    Blackberry's. The reduction to numbers is anyways the analyst's goal, and is
    the "thing" that is presented to your boss.

    5. The Neyman-Pearson and Fisher issues re hypothesis testing was actually a
    situation running on many levels. Just to view it in literal terms overlooks
    all the historic and side-line issues, including a strong difference in
    basic philosophies. These philosophical issues remain with us today, and
    have not been resolved. We just continue to argue violently (we just shout
    at each other in papers) about hypothesis testing. NHST is the basic glue
    that supports social, psychological and medical research. It's a love-hate
    relationship that will always be with us.

    6. The issues about the randomness of 18121812. If I drew these cards in
    sequence from the top of a well shuffled card deck, would you say the
    shuffling was valid? The point is as I said above, it depends on just
    exactly all the circumstances were, and what these numbers are to be used
    for. In many cases, it is not as simple as Bob Dole says. Experimental
    design is one approach to getting around some of these dependencies on
    randomness.

    David Heiser
     
    David A. Heiser, Mar 8, 2006
    #47
  8. David W. Cantrell

    Reef Fish Guest

    But you very statement is too GENERAL. :) Lacking specifics!

    It's not clear what you are referring to here, in terms of "your
    misuses".
    Are you talking about the discarding of random outcome that "looked
    nonrandom"?

    If so, if a random permutation is used soldiers to war in Iraq,
    Everyone agrees, before hand, that a well-known and well-tested
    pseudorandom generator will be used to generate the drafting sequence:

    and 1, 2, ...,366 turned up. Specific enough? What do you do now?

    Harsh, perhaps. But Fisher was criticizing other mathematicians and
    critisizing the Neyman-Pearson approach, when according to Savage,
    Fisher was WRONG, in the mathematics Savage knew well, and in
    the Neyman-Pearson approach we know well.

    In those respect, Savage's criticism of Fisher's ignorance was
    warranted.
    Fisher should NOT have criticized others on matter in which he was
    ignorant. Had he NOT mis-criticized others, then what you said
    about "Fisher did very well" would be a fitting comment.

    But how does Fisher address the ODD looking random subsets?
    Would he have argued that the 1, 2, ..,366 permutation drawn on a
    single trial, be used?
    I don't think Savage was criticizing Fisher's desk calculator computing
    per
    se, as much as Fisher's failure to keep pace with newer, modern ideas.

    Chester I. Bliss, a Lecturer at Yale (terminated 1967) was teaching a
    graduate course from his book. Students in his course had to spend
    hours a week doing problems on the desk calculators, as required by
    Bliss, when the Computer Center at Yale, with its up-to-date support
    and statistical software were not tapped. I viewed the weekly lab
    sessions as a complete waste of time in the first place (because I
    had never punched another key on those calculators since the
    time I touched one at that lab :), and worse if a similar problem
    had to be done.

    In one semester, I programmed (together with examples in Bliss's
    book tested), rather than wasting my time at the desk calculator
    lab, with Savage overlooking what I had done. Bliss was not only
    not interested, he downed graded me even though I scored the
    highest on the only exam (Final exam) in the course. Bliss never
    taught another course at Yale thereafter. I suspect he was fired.
    THAT's the kind of criticism Savage had on Fisher, I am quite
    sure -- using old obsolete desk calculator methods as well as not
    keeping in time with the modern theory and methods, such as
    the examination of residuals.
    All of what you said in the preceding paragraph is very true.

    That was the example used to criticize the "what's the next number"
    type
    of questions in pattern recognition and other kinds of tests. Not
    about
    randomization.

    1. This is a "statistical" question about the randomness of the
    "generator".

    2. Randomization is the generation of a "subset" by a proven-valid
    "random generation process". (Such as the mechanical card-
    shuffling machines at Blackjack table in Las Vegas that
    continuously shuffle 8 decks)

    The answer to (2) is to keep whatever comes out as the random outcome.
    The answer to (1) is there MAY be cause to question the validity of
    the generator.

    That's too easy an excuse for NOT facing the issue.

    To talk seriously about randomization, you MUST assume that there is
    a random generation process beyond suspicion of being RANDOM.
    Then ask yourself what you would do if you see some "odd pattern"
    from that process.

    There are now "pseudorandom" generators that had been thoroughly
    tested to be trustworthy for serious applications.

    Suppose you ask one of those generators to produce a random
    sequence of 10 two digit numbers, and you get:

    11, 12, 13, 14, 15, 16, 17, 18, 19, 20.

    What justification can you give for NOT accepting it as a random
    sequence?

    I think you're thinking about a different issue -- that of using
    "stratification" as a method to ensure oddities NOT to occur,
    but WITHIN each strata, you still have to rely on "simple random"
    selections -- so you are back to the same unresolved question
    of do you keep or do you throw away?

    -- Bob.
     
    Reef Fish, Mar 8, 2006
    #48
  9. David W. Cantrell

    Herman Rubin Guest

    I believe that things may have been tested 100 trillion times
    (American) which is 10^11; I doubt if the European version,
    which is 10^20, has been done. But 366! ~ 10^781.

    The only reasonable treatment of that is Bayesian.


    What is surprising about that?


    One thing about Fisher which not too many know was that he
    was essentially one to castigate anything which was not of
    his making.

    Fisher insisted on randomized designs, because he was not
    familiar with the general ideas of probability. He did
    consider that some designs are better than others, but
    only if they fell into the class of the ones he liked.
    Huh? Not that Pearson and his group were the greatest, but
    they did not have Fisher's arrogance, nor the disdain for
    higher mathematics.
    Gaussian? I know what Gauss did, and characterizations of
    the normal distribution are not that important. The
    Gauss-Markov Theorem in fact goes in the other direction;
    it is unclear as to how much effect it had, as the
    contribution of Gauss was pointed out after Markov's had
    been found.

    Laplace had difficulty in understanding Bayes, but at
    least was willing to admit it, and he observed that it
    contradicted classical testing, which was already
    present in his time.
    I disagree.

    However, Fisher defined sufficiency in terms of the
    factorization in computing the maximum likelihood estimate;
    he had the average value of X_i as sufficient for the mean,
    even if the variance was unknown.
    Unfortunately. Fisher properly criticized Neyman-Pearson
    for not considering all other alternatives, but could not
    see past his classical approach; this cannot really be done
    below the prior Bayes level, which I advocate. This states
    that one should use a linear combination of the error
    probabilities in the various states, and is far more robust
    than appears at first glance. But the latent idea that the
    P-value was in some sense a statement about the probability
    of the null dominated. One can get reasonable procedures
    only by looking at the errors of both kinds simultaneously,
    and anything like a fixed significance level will use a
    very unreasonable loss-prior combination, with a heavy
    emphasis on alternatives which are extremely close to the
    point null.
     
    Herman Rubin, Mar 9, 2006
    #49
  10. David W. Cantrell

    Reef Fish Guest

    The 10^781 is actually quite irrelevant. There are at least a dozen
    tests for randomness, and I am not even sure if there are pseudo-
    random number generators with cycles that long. The point was
    simply some adequate number of tests that would satisfy every
    practical simulation expert except perhaps the extreme nitpicker.

    That of course is the Bayesian point of view, that randomization is
    neither necessary nor sufficient for a well-designed experiment.
    A bit of an overkill, given the other substantive points which had
    already
    more or less stripped Fisher naked. :)
    That came out rather loudly and clearly from Savage's "Re-reading
    R.A. Fisher" paper. But then I am sure each of us know some much
    lesser men of Statistical stature who do the same.
    I am not sufficiently familiar with this part of Fisher's idea about
    randomized designs, but I don't seem to recall any resolution
    about the Knut Vik square Savage kept harping Fisher about.

    Just to make sure that reader knows:

    Herman is now commenting on David Heiser's comments about my
    comments on Savage's comments on Fisher. :)
    I rather agree, with Herman and Savage here. I don't have anything
    to add to the rest of the discourse between Herman and David Heiser.

    -- Bob.
     
    Reef Fish, Mar 9, 2006
    #50
  11. David W. Cantrell

    Herman Rubin Guest

     
    Herman Rubin, Mar 10, 2006
    #51
  12. David W. Cantrell

    Reef Fish Guest

     
    Reef Fish, Mar 10, 2006
    #52
  13. David W. Cantrell

    Bob Dole Guest

    No, it seldom is. ;)

    On my wall is a framed cartoon given to me by co-workers showing two
    scientist-types admiring a bunch of incomprehensible cartoon math on a
    board that does, I admit, bear a superficial resemblance to mine. One
    says to the other: "Ah, if only it were so simple."

    Bob Dole
     
    Bob Dole, Mar 11, 2006
    #53
  14. David W. Cantrell

    Reef Fish Guest

    Excellent, Bob. I always appreciate and admire those who have a
    good of humor to admit to, and accompany, their discussions that
    were just sensible "discussions".

    I am still looking forward to David Heiser's follow-up to HIS
    discussion, which drew some detailed comments from me and
    from Herman Rubin, on his many tenuous positions that need
    justifification on his part.

    -- Bob.
     
    Reef Fish, Mar 11, 2006
    #54
  15. David W. Cantrell

    Abe Kohen Guest

    Reefer,

    Stop wasting everyone's time with your silly excerpts from google groups.

    If the thread is over (per your pronouncement, King Fish) then why did you
    continue your silly postings to this thread?

    Kerplunk, little fish.
     
    Abe Kohen, Mar 12, 2006
    #55
  16. David W. Cantrell

    Reef Fish Guest


    As you requested, this is Google's report of your profile: a TOTAL of

    2 messages in sci.stat. in 2006 -- and BOTH of them appeared today.

    -----------------------------------------------------------
    Recent Posts:
    Posts in All 3 Groups -- 380 messages

    soc.culture.jewish.moderated -- 378 messages
    sci.stat.math -- 1 message
    sci.stat.edu -- 1 message


    Anti-religious soc.culture.jewish.moderated moments ago
    Anti-religious soc.culture.jewish.moderated moments ago
    Purim is Coming! soc.culture.jewish.moderated moments ago
    SHOAH soc.culture.jewish.moderated moments ago
    SHOAH soc.culture.jewish.moderated moments ago
    Does anyone have any problem with Bush pushing the UAE deal?
    soc.culture.jewish.moderated 2 days ago
    Anti-religious soc.culture.jewish.moderated 2 days ago
    Islam's War Against the Jews soc.culture.jewish.moderated 2 days
    ago
    Anti-religious soc.culture.jewish.moderated 2 days ago
    SHOAH soc.culture.jewish.moderated 2 days ago
    Show all »
    Post Activity
    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
    2005 1 83
    2006 129 115 52

    ---------------------------------------------------------------------------­--


    Was jewish culture getting too boring for you?

    Try PORTUGUESE culture. You'll probably meet lots of mad men
    like you and Luis A. Afonso there. LOL!

    -- Bob.
     
    Reef Fish, Mar 12, 2006
    #56
  17. David W. Cantrell

    Herman Rubin Guest

    This is unfortunate. Any pseudo-random numbers will fail
    at least one test; that they were generated that way.

    This may or may not be important. But if one has a somewhat
    complicated problem, there is good reason to be skeptical.
    There are NO adequate tests. That is why I suggest that
    one never use pseudo-random numbers alone for production
    work. What I advocate one use is an exclusive or, or
    similar procedure, with pseudo-random and physical random
    numbers. There are problems with physical random numbers
    alone; however, they are not the ones with pseudo-random
    numbers; good long-period pseudo-random numbers will get
    the needed uniformity, and the physical random numbers
    will break up the dependency.

    If one has a simple problem, I suggest that one consider
    using quasi-random numbers; they are extremely uniform,
    but not at all random. They have their problems, and in
    particular, acceptance-rejection is a major problem with
    them. As most non-uniform distributions are efficiently
    obtained by acceptance-rejection, know what you are doing.
    Read about the properties of quasi-random numbers before
    you decide your problem is "simple" enough.
     
    Herman Rubin, Mar 12, 2006
    #57
  18. David W. Cantrell

    Reef Fish Guest

    Depends on how many tests you run and what you set your
    standards for "failing". That's Monte Carlo Methods 101, Herman!
    So, what's your POINT about inadequate testing for pseudorandom
    number generators for a generation of a pseudorandom permutation
    of 366 integers?


    Herman, I think you are busy ARGUING with YOURSELF!

    How do you adequately test for a "physical random" generation
    process? That was the process that was actually used for
    the lottery and was challenged and found reasons to suspect
    (but not prove) non-random behavior.

    The point of the "randomization" discussion is to ASSUME that
    we have a process of generation that is agreed upon before
    hand. For that discussion, we could use Herman Rubin's
    favorite, if it meets no serious objection from others.

    What would YOU do if you used YOUR "trusted" generator
    and obtained (1, 2, 3, ..., 366) ?

    What you have brought into this discussion was your irrelevant
    red-herring, which is not only not the issue with randomization,
    but a non-issue with those in the serious business of using
    Monte Carlo methods.
    Whether the problem is simple, or the generator is physical,
    pseudo, or quasi, just START with anyone that suited YOU,
    and then ask the question

    What would YOU do if you used YOUR "trusted" generator
    and obtained (1, 2, 3, ..., 366) ?

    THAT is the issue being discussed here, not your pseudo-
    quasi knowledge about random number generators.

    -- Bob.
     
    Reef Fish, Mar 12, 2006
    #58
  19. David W. Cantrell

    Herman Rubin Guest

     
    Herman Rubin, Mar 13, 2006
    #59
  20. David W. Cantrell

    Reef Fish Guest

    But as I had already explained, this point is moot -- in that we
    were NOT discussing the best way to produce a random or
    pseudorandom number generator -- the point of the discussion
    is WHAT WOULD YOU DO, once you decided a ganerator is
    acceptable (to YOU), and you generated a perfectly
    non-random looking sequence of (1, 2, ..., 366).

    I think I made THAT point rather clear, for you to come back to
    the same red-herring again this time, don't you think?

    You snipped the last paragraphs of mine in the preceding post,
    which should have terminated your rehash of the irrelevant point
    this time:

    ==== BEFIN excerpt of what's in my LAST post, snipped by Herman

    What you have brought into this discussion was your irrelevant
    red-herring, which is not only not the issue with randomization,
    but a non-issue with those in the serious business of using
    Monte Carlo methods.

    Whether the problem is simple, or the generator is physical,
    pseudo, or quasi, just START with anyone that suited YOU,
    and then ask the question

    What would YOU do if you used YOUR "trusted" generator
    and obtained (1, 2, 3, ..., 366) ?

    THAT is the issue being discussed here, not your pseudo-
    quasi knowledge about random number generators.

    ==== END excerpt of what's in my LAST post, snipped by Herman

    You did retain the part below. So,please re-read it carefully:
    In other words, you would "get another sample", just as Bob Dole
    said he would, and I suspect almost everyone who sings the
    tune of "randomization" a la Fisher would, except MYSELF.

    You either believe in randomization or you don't. If you do, you
    are duty bound (not to mention theory bound) to accept the
    randomized outcome you got -- having no reason to doubt the
    generating process being faulty.

    The inevitable question for me to you (and Bob Dole) and anyone
    else who would "get another sample" if confronted with the
    (1, 2, 3, ..., 366) sample:

    How many OTHER patterns would you discard ALSO? E.g.,
    (366,365, ..., 4,3,2,1)
    (1.3.5.7....,361,363,365, 2, 4,6,...360, 362,364) or its reverse

    or TRILLIONS and TRILLIONS of other easily recognizable to
    be patterned ones that looked "too non-random". I think you
    get the idea.

    What fraction of all the possible 366! permutations would have
    been EXCLUDED (a priori) from the full candidate set for reasons
    of "it looked non-random" -- this latter part is definitely NOT a
    condition for exclusion in Fisher's or anyone else's definition
    of "randomization" in a designed experiment.

    THAT is the issue of what I called the "unresolved paradox" in
    "randomization", because what "looks odd" to one may not
    "look odd' or "sufficiently odd" to another ... thus throwing the
    whole idea of "randomization" to an ad hoc, ill-defined, and
    undefined process.

    -- Bob.
     
    Reef Fish, Mar 13, 2006
    #60
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.