Claimed approximation for Kullback-Leibler distance D(p||q) when pand q are close, apparently based

Discussion in 'Math Research' started by jdm, Jan 1, 2012.

  1. jdm

    jdm Guest

    The following claim featured in a research paper I've been studying -
    however, no proof accompanied it beyond a statement that the
    approximation could be obtained using Taylor series at order 2 - and
    it wasn't clear what the variable was supposed to be or around which
    point.

    Let p and q be discrete probability distributions of random variables
    taking on values from a set with M+1 elements;

    p=(p_0, ..., p_M)

    (p_i = P(random variable with distribution p is equal to i))

    Likewise, q=(q_0, ..., q_M)

    Where p and q are close - defined as |p_{i} - q_{i}| << q_{i} \forall
    i - let e_i denote the value (p_i - q_i).

    Then, according to the paper, D(p||q) \approx D(q||p) \approx sum_{i=0}
    ^{M}(e_{i}^{2}/q_{i})/2.

    I haven't been able to verify this approximation for myself, as I
    stated, and if anyone reading this can help (even by arguing that the
    approximation isn't in fact valid) it would be much appreciated!

    Many thanks,

    James McLaughlin.
     
    jdm, Jan 1, 2012
    #1
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.