For random variable $X$ and $Y$, the mutual information between them is defined as:

$$\begin{aligned} I(X;Y) = \sum_{x,y}p(x,y)\,\log{p(x,y) \over p(x)\,p(y)} = E \log{p(X,Y) \over p(X)\,p(Y)} \end{aligned}$$

which is symmetrical. Alternatively,

$$\begin{aligned} I(X;Y) = \sum_{x,y}p(x,y)\,\log{p(x\,|\,y) \over p(x)} = E \log{p(X|Y) \over p(X)} \end{aligned}$$

Interestingly, the mutual information of a random variable with itself, called the self-information, is the entropy of the variable, ie $I(X;X) = H(X)$, for

$$\begin{aligned} I(X;X) &= E \log{p(X,X) \over p(X) p(X)} = E \log{p(X) \over p(X) p(X)} \\ &= - E \log{p(X)} \\ &= H(X) \\ \end{aligned}$$

Proposition 2.19, provided the entropies and conditional entropies are finite:

$$\begin{aligned} I(X;Y) &= H(X) - H(X|\,Y) \\ \end{aligned}$$

and

$$\begin{aligned} I(X;Y) &= H(X) + H(Y) - H(X,Y) \\ \end{aligned}$$

Both equations can be easily verified. For example,

$$\begin{aligned} H(X) - H(X|\,Y) &= -E\log p(X) + E \log{p(X|\,Y)} \\ &= -E\log p(X) + E \log{p(X,Y) \over p(Y)} \\ &= E \log{p(X,Y) \over p(X) p(Y)} \\ &= I(X;Y) \\ \end{aligned}$$

Furthermore, for random variable $X, Y$ and $Z$, the mutual information between $X$ and $Y$ conditioning on $Z$ is defined as:

$$\begin{aligned} I(X;Y\,|\,Z) = \sum_{x,y,z}p(x,y,z)\,\log{p(x,y\,|\,z) \over p(x\,|\,z)\,p(y\,|\,z)} = E \log{p(X,Y\,|\,Z) \over p(X\,|\,Z)\,p(Y\,|\,Z)} \end{aligned}$$

Source: Information Theory.

# posted by rot13(Unafba Pune) @ 5:55 PM