Neural networks are central to modern AI, from language modelling to speech recognition and computer vision, yet why they work so well is only partly understood. Their elementary operations are mathematically simple, but large-scale models behave in complex ways during and after training, which limits their transparency and trustworthiness. At the same time, their growing size comes with substantial energy consumption and a rising environmental footprint.

This thesis approaches these issues from an information-theoretic perspective, with a focus on redundancy: the same task-relevant information being encoded multiple times across neurons, channels, layers or feature groups. Modern networks are heavily overparameterized, but it is unclear whether such redundancy is a byproduct of training, a source of robustness or merely unused capacity. Partial information decomposition (PID) makes this question precise by splitting the information that several sources carry about a target into unique, redundant and synergistic parts. Estimating these quantities reliably in high-dimensional, deterministic networks is itself an open problem.

The thesis first develops robust methods to estimate redundancy in learned representations. It then studies where and how redundancy arises, how it evolves during training, how it depends on architecture and training, and whether it can be influenced through suitable training strategies. Related work on information planes and the information bottleneck, which asks whether networks benefit from discarding irrelevant information, provides further context, as do information-theoretic generalization bounds, which relate generalization to the dependence between training data and learned parameters. At a later stage, the project will assess whether these insights can support more sustainable deep learning, for example by identifying and removing redundant elements or by encouraging more efficient representations during training.

The thesis is part of the FWF project Information Planes and Decompositions, and includes a research stay at Chalmers University of Technology (Gothenburg) in 2027 within the ELLIS PhD programme.