C. Nalmpantis, A. Lentzas and D. Vrakas, "A Theoretical Analysis of Pooling Operation Using Information Theory," 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), Portland, OR, USA, 2019, pp. 1729-1733
Author(s): C. Nalmpantis, A. Lentzas and D. Vrakas
Keywords: Feature Pooling, Deep Learning, Representation Learning, Information Theory, Entropy, Channel Capacity
Tags:
Abstract: Several modern deep learning architectures incorporate the operation of pooling in order to achieve sufficient, minimal and invariant representations. Nevertheless, its importance has only been verified empirically, without solid theoretical evidence. This paper presents a comprehensive theoretical analysis, investigates the mechanism of pooling from the information theory point of view and proposes a novel pooling operation based on entropy. In comparison with other versions of pooling, it automatically adapts to the features and creates more compact representations. The theoretical outcomes are validated utilizing both shallow and deep architectures.