Claimed approximation for Kullback-Leibler distance D(p||q) when pand q are close, apparently based

Discussion in 'Math Research' started by jdm, Jan 1, 2012.

  1. jdm

    jdm Guest

    The following claim featured in a research paper I've been studying -
    however, no proof accompanied it beyond a statement that the
    approximation could be obtained using Taylor series at order 2 - and
    it wasn't clear what the variable was supposed to be or around which

    Let p and q be discrete probability distributions of random variables
    taking on values from a set with M+1 elements;

    p=(p_0, ..., p_M)

    (p_i = P(random variable with distribution p is equal to i))

    Likewise, q=(q_0, ..., q_M)

    Where p and q are close - defined as |p_{i} - q_{i}| << q_{i} \forall
    i - let e_i denote the value (p_i - q_i).

    Then, according to the paper, D(p||q) \approx D(q||p) \approx sum_{i=0}

    I haven't been able to verify this approximation for myself, as I
    stated, and if anyone reading this can help (even by arguing that the
    approximation isn't in fact valid) it would be much appreciated!

    Many thanks,

    James McLaughlin.
    jdm, Jan 1, 2012
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.