Some Interaction Information Examples

Interaction information between three random variables has a much less immediate interpretation compared to other information quantities. This makes it more tricky to work with. An i.i. term \(I(X;Y;Z)\) between \(X,\ Y\) and \(Z\), could be either negative:

If we are trying to specify \(X\) with \(Y\) and \(Z\), we expect \(Z\) to mostly be redundant to part of \(Y\)

or positive:

If we are trying to specify \(X\) with \(Y\) and \(Z\), we expect \(Z\) to mostly add new information to that from \(Y\).

(Interaction information is commutative in its arguments, so the variable's can be switched around in these interpretations.)

Knowing how the joint distribution of \((X,Y,Z)\) factors can tell you whether an interaction information is positive or negative. Here are two simple and useful cases.

Negative: \(X\rightarrow Y \rightarrow Z\) is a Markov chain

In this situation, \(P_{(X,Y,Z)}=P_X\cdot P_{Y|X}\cdot P_{Z|Y}.\) Then \(I(X;Y)\geq I(X;Y|Z)\) so \(I(X;Y;Z)\leq 0.\)

An easy example of such variables is \(X=(A_1,A_2,A_3), Y=(A_1,A_2), Z=A_1\) for any random (A_i).

Positive: \(X\perp Y\) and \(Z\) is derived from \(X\) and \(Y\)

In this situation, \(P_{(X,Y,Z)}=P_X\cdot P_Y\cdot P_{Z|(X,Y)}.\) Notice that \(H(X)=H(X|Y),\) and that \(H(X|Z)\geq H(X|Y,Z).\) Then \(I(X;Z) \leq I(X;Z|Y)\) so \(I(X;Y;Z)\geq 0.\)

A trivial example is any \(X\perp Y\) and \(Z=(X,Y).\) A more interesting one is \(N_1,N_2\) iid \(\sim\mathcal{N}(0,1)\) with \(X=N_1+N_2,\ Y=N_1-N_2\) and \(Z=N_1.\)