Extra Problems for Chapter 11

T.M. Cover and J.A. Thomas

1. Mean squared error.
   Let satisfy . Consider linear predictors for , i.e.

   

   Assume n>p. Find

   

   where the minimum is over all linear predictors b and the maximum is over all densities f satisfying .

2. Maximum entropy characteristic functions.
   We ask for the maximum entropy density satisfying a constraint on the characteristic function . The answers need be given only in parametric form.
   1. Find the maximum entropy f satisfying , at a specified point .
   2. Find the maximum entropy f satisfying .
   3. Find the maximum entropy density having a given value of the characteristic function at a specified point .
   4. What problem is encountered if ?
3. Maximum entropy processes.
   1. Find the maximum entropy rate binary stochastic process , satisfying for all i.
   2. What is the resulting entropy rate?
4. Maximum entropy of sums Let Find the maximum entropy density for Y under the constraint , ,
   1. if and are independent.
   2. if and are allowed to be dependent.
   3. Prove part (a).
5. Maximum entropy Markov chain.

   Let be a stationary Markov chain with . Let for all n.

   1. What is the maximum entropy rate process satisfying this constraint?
   2. What if , for all n for some given value of , ?
6. An entropy bound on prediction error. Let be an arbitrary real valued stochastic process. Let . Thus the conditional mean is a random variable depending on the n-past . Here is the minimum mean squared error prediction of given the past.
   1. Find a lower bound on the conditional variance in terms of the conditional differential entropy .
   2. Is equality achieved when is a Gaussian stochastic process?
7. Maximum entropy rate. What is the maximum entropy rate stochastic process over the symbol set for which the probability that 00 occurs in a sequence is zero?
8. Maximum entropy.
   1. What is the parametric form maximum entropy density f(x) satisfying the two conditions

      

   2. What is the maximum entropy density satisfying the condition

      

   3. Which entropy is higher?
9. Maximum entropy. Find the parametric form of the maximum entropy density f satisfying the Laplace transform condition

   

   and give the constraints on the parameter.

Latex file

Postscript file