\ A fundamental theorem in information theory \– the data processing inequality \– states that deterministic processing cannot increase the amount of information contained in a random variable\ or a stochastic process. The task of signal processing is to operate on the physical representation\ of information such that the intended user can access this information with little effort. In the\ light of the data processing inequality, this can be viewed as the task of removing irrelevant\ information, while preserving as much relevant information as possible.

This thesis defines information loss for memoryless systems processing random variables or\ stochastic processes, both with and without a notion of relevance. These definitions are the\ basis of an information-theoretic systems theory, which complements the currently prevailing\ energy-centered approaches. The results thus developed are used to analyze various systems in\ the signal processor\’s toolbox: polynomials, quantizers, rectifiers, linear filters with and without\ quantization effects, principal components analysis, multirate systems, etc. The analysis not only\ focuses on the information processing capabilities of these systems: It also highlights differences\ and similarities between design principles based on information-theoretic quantities and those

based on energetic measures, such as the mean-squared error. It is shown that, at least in some\ cases, simple energetic design can be justified information-theoretically.

\

As a side result, this thesis presents two approaches to model complexity reduction for time-homogeneous, finite Markov chains. While one approach preserves full model information with\ the cost of losing the (first-order) Markov property, the other approach yields a Markov chain on a\ smaller state space with reduced model information. Finally, this thesis presents an information-theoretic characterization of strong lumpability, the case where the function of a Markov chain\ is Markov (of some order).

},
attachments = {https://www.spsc.tugraz.at/sites/default/files/Geiger_Thesis.pdf},
author = {Bernhard Geiger}
}