Information Complexity and Estimation

Abstract

We consider an input $x$ generated by an unknown stationary ergodic source $X$ that enters a signal processing system $J$, resulting in $w=J(x)$. We observe $w$ through a noisy channel, $y=z(w)$; our goal is to estimate x from $y$, $J$, and knowledge of $f_{Y|W}$. This is universal estimation, because $f_X$ is unknown. We provide a formulation that describes a trade-off between information complexity and noise. Initial theoretical, algorithmic, and experimental evidence is presented in support of our approach.

Appears at WITMSE 2011, The Fourth Workshop on Information Theoretic Methods in Science and Engineering, 7-10 August 2011, Helsinki, Finland. Note that the WITMSE version is 4 pages, owing to different formatting