A Theoretical Analysis of Pooling Operation Using Information Theory

C. Nalmpantis, A. Lentzas and D. Vrakas, "A Theoretical Analysis of Pooling Operation Using Information Theory," 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), Portland, OR, USA, 2019, pp. 1729-1733

Feature Pooling, Deep Learning, Representation Learning, Information Theory, Entropy, Channel Capacity

Several modern deep learning architectures incorporate the operation of pooling in order to achieve sufficient, minimal and invariant representations. Nevertheless, its importance has only been verified empirically, without solid theoretical evidence. This paper presents a comprehensive theoretical analysis, investigates the mechanism of pooling from the information theory point of view and proposes a novel pooling operation based on entropy. In comparison with other versions of pooling, it automatically adapts to the features and creates more compact representations. The theoretical outcomes are validated utilizing both shallow and deep architectures.