Published

Tue, Sep 01, 2026

Tags

rotm

Contact

• Bernhard Geiger

There has been a long-standing discussion whether information-theoretic compression in latent representations leads to improved generalization performance. While intuitive—compressed representations do not contain irrelevant input information on which the remaining part of a neural network can overfit—there is now substantial evidence that information-theoretic compression is not necessary for good generalization. Indeed, since many information estimators inherently capture geometric properties of the latent representations, geometric compression (i.e., clustering) has recently been proposed as an explanation of generalization.

In this paper, we investigate the interplay between information-theoretic compression, clustering, and generalization performance in stochastic neural networks. Considering both adaptive additive and fixed multiplicative noise at each neuron output, we show that the connection between these three concepts is more nuanced than initially believed.

The table shows rank correlations between mutual information (MI, low=compressed), a measure for neural collapse (NC, low=clustered), accuracy, and the generalization gap, for conditional entropy bottleneck (CEB) and Gaussian dropout networks. It can be seen that information-theoretic compression is not a reliable indicator for accuracy, with signs of the rank correlations switching between model classes. In contrast, geometric compression appears to be connected to good generalization performance. Most surprisingly, information-theoretic and geometric compression are negatively correlated, suggesting that clustered representations consume large values of mutual information. This result is in contrast to the prevalent belief that information-theoretic and geometric compression are monotonically linked.

The paper (with links to code and data) is available on arXiv and was recently presented at the ECML-PKDD.

Browse the Results of the Month archive.