Information-Theoretic System Analysis and Design

The seminar touches on the following topics:

• Information measures including Information loss, information dimension, information rate, differential entropy
• Behavior of information measures when subjected to nonlinear transforms
• Information loss in simple linear and non-linear transforms : full-wave and half-wave rectifier, quantizer, linear filter
• The problem of relevance: Information Bottleneck Method and signal enhancement
• Principal components analysis and anti-aliasing low-pass filters: optimality regarding mean squared-error, criteria for information-theoretic optimality

The seminar allows us to have an informal syllabus, adapting to prior knowledge and interest of prospective students. The teaching style will be a mix between an ordinary lecture, students solving small examples on the blackboard, and discussions. Grades are obtained by submitting two homework examples (on which you can work in pairs). The grade can be improved by volunteering to solve examples on the blackboard.

Prerequisites

You are expected to know the fundamentals of probability theory (random variable, expectation, stochastic process) and of signal processing (PCA, linear filters, quantization, static non-linear systems). Some knowledge in information theory (entropy, mutual information) is beneficial, but not necessary.

Course Notes/Slides

The course notes can be downloaded from TUGOnline (you need to log in). They are an excerpt of this book.

Homework