A fundamental theorem in information theory – the data processing inequality – states that deterministic processing cannot increase the amount of information contained in a random variable or a stochastic process. The task of signal processing is to operate on the physical representation of information such that the intended user can access this information with little effort. In the light of the data processing inequality, this can be viewed as the task of removing irrelevant information, while preserving as much relevant information as possible.

This thesis defines information loss for memoryless systems processing random variables or stochastic processes, both with and without a notion of relevance. These definitions are the basis of an information-theoretic systems theory, which complements the currently prevailing energy-centered approaches. The results thus developed are used to analyze various systems in the signal processor’s toolbox: polynomials, quantizers, rectifiers, linear filters with and without quantization effects, principal components analysis, multirate systems, etc. The analysis not only focuses on the information processing capabilities of these systems: It also highlights differences and similarities between design principles based on information-theoretic quantities and those based on energetic measures, such as the mean-squared error. It is shown that, at least in some cases, simple energetic design can be justified information-theoretically.

As a side result, this thesis presents two approaches to model complexity reduction for time-homogeneous, finite Markov chains. While one approach preserves full model information with the cost of losing the (first-order) Markov property, the other approach yields a Markov chain on a smaller state space with reduced model information. Finally, this thesis presents an information-theoretic characterization of strong lumpability, the case where the function of a Markov chain is Markov (of some order).

The full text of the thesis can be downloaded here.