Selected Topics Signal, Biosignal and Speech Processing: Information-Theoretic System Analysis and Design

Instructor: 

Information-Theoretic System Analysis and Design

The seminar touches on the following topics:

  • Information measures including Information loss, information dimension, information rate, differential entropy
  • Behavior of information measures when subjected to nonlinear transforms
  • Information loss in simple linear and non-linear transforms : full-wave and half-wave rectifier, quantizer, linear filter
  • The problem of relevance: Information Bottleneck Method and signal enhancement
  • Principal components analysis and anti-aliasing low-pass filters: optimality regarding mean squared-error, criteria for information-theoretic optimality

The seminar allows us to have an informal syllabus, adapting to prior knowledge and interest of prospective students. The teaching style will be a mix between an ordinary lecture, students solving small examples on the blackboard, and discussions. Grades are obtained by submitting two homework examples (on which you can work in pairs). The grade can be improved by volunteering to solve examples on the blackboard.

Prerequisites

You are expected to know the fundamentals of probability theory (random variable, expectation, stochastic process) and of signal processing (PCA, linear filters, quantization, static non-linear systems). Some knowledge in information theory (entropy, mutual information) is beneficial, but not necessary.

Course Notes/Slides

The course notes can be downloaded from TUGOnline (you need to log in). They are an excerpt of this book.

Chapter Slides

1) Motivation and Introduction

PDF

2) Piecewise Bijective Functions

PDF

3) General Input Distributions

Supplement, PDF

4) Information Dimension

PDF

5) Relative Information Loss

PDF

6) Relevant Information Loss

PDF

7) PCA: MSRE and Relative Information Loss

PDF

8) PCA: Relevant Information Loss

PDF

Homework

  Assignment Deadline

Homework

PDF, Files  15.02.2018

 

Term: 
Winter
Education Level: 
Master Level
PhD Level