Extra Problems for Chapter 9

T.M. Cover and J.A. Thomas

1. Variational inequality: Verify, for positive random variables X, that

   

   where and , and the supremum is over all , . It is enough to extremize .

2. Estimation.
   Here is the estimation counterpart to Fano's inequality. Let X be a random variable with differential entropy h(X). Let be an estimate of X, and let be the expected prediction error.
   1. Show

      

   2. Given side information Y and estimator , show

      

      
      

3. Differential entropy bound on discrete entropy:   Let X be a discrete random variable on the set with . Show that

   

   Moreover, for every permutation ,

   

   Hint: Construct a random variable X' such that . Let U be an uniform(0,1] random variable and let Y = X' + U, where X' and U are independent. Use the maximum entropy bound on Y to obtain the bounds in the problem.

4. Channel with uniformly distributed noise: Consider a additive channel whose input alphabet , and whose output Y = X + Z, where Z is uniformly distributed over the interval [-1,1]. Thus the input of the channel is a discrete random variable, while the output is continuous. Calculate the capacity of this channel.

Latex file

Postscript file