For random variable \(X\) and \(Y\), the mutual information between them is defined as:

\[ \begin{aligned} I(X;Y) = \sum_{x,y}p(x,y)\,\log{p(x,y) \over p(x)\,p(y)} = E \log{p(X,Y) \over p(X)\,p(Y)} \end{aligned} \]

which is symmetrical. Alternatively,

\[ \begin{aligned} I(X;Y) = \sum_{x,y}p(x,y)\,\log{p(x\,|\,y) \over p(x)} = E \log{p(X|Y) \over p(X)} \end{aligned} \]

Interestingly, the mutual information of a random variable with itself, called the self-information, is the entropy of the variable, ie \(I(X;X) = H(X)\), for

\[ \begin{aligned} I(X;X) &= E \log{p(X,X) \over p(X) p(X)} = E \log{p(X) \over p(X) p(X)} \\ &= - E \log{p(X)} \\ &= H(X) \\ \end{aligned} \]

Proposition 2.19, provided the entropies and conditional entropies are finite:

\[ \begin{aligned} I(X;Y) &= H(X) - H(X|\,Y) \\ \end{aligned} \]

and

\[ \begin{aligned} I(X;Y) &= H(X) + H(Y) - H(X,Y) \\ \end{aligned} \]

Both equations can be easily verified. For example,

\[ \begin{aligned} H(X) - H(X|\,Y) &= -E\log p(X) + E \log{p(X|\,Y)} \\ &= -E\log p(X) + E \log{p(X,Y) \over p(Y)} \\ &= E \log{p(X,Y) \over p(X) p(Y)} \\ &= I(X;Y) \\ \end{aligned} \]

Furthermore, for random variable \(X, Y\) and \(Z\), the mutual information between \(X\) and \(Y\) conditioning on \(Z\) is defined as:

\[ \begin{aligned} I(X;Y\,|\,Z) = \sum_{x,y,z}p(x,y,z)\,\log{p(x,y\,|\,z) \over p(x\,|\,z)\,p(y\,|\,z)} = E \log{p(X,Y\,|\,Z) \over p(X\,|\,Z)\,p(Y\,|\,Z)} \end{aligned} \]

Source: Information Theory.

# posted by rot13(Unafba Pune) @ 5:55 PM