Katsaki, S., Aivazidis, C., Mylonas, N., Mollas, I., Tsoumakas, G. (2023) On the Adaptability of Attention-Based Interpretability in Different Transformer Architectures for Multi-Class Classification Tasks, Proceedings of the AIMLAI Workshop of ECML PKDD 2023.

You are here: Home » Publications » Katsaki, S., Aivazidis, C., Mylonas, N., Mollas, I., Tsoumakas, G. (2023) On the Adaptability of Attention-Based Interpretability in Different Transformer Architectures for Multi-Class Classification Tasks, Proceedings of the AIMLAI Workshop of ECML PKDD 2023.

Author(s): Sofia Katsaki, Christos Aivazidis, Nikolaos Mylonas, Ioannis Mollas, and Grigorios Tsoumakas

Keywords: Interpretable Machine Learning, Local Interpretability, Transformers, Attention, Text Classification

Tags:

2023, aivazidis, katsaki, Mollas, Mylonas, Tsoumakas

Abstract: Transformers are widely recognized as leading models for NLP tasks due to their attention-based architecture. However, their complexity and numerous parameters hinder the understanding of their decision-making processes, restricting their use in high-risk domains where accurate explanations are crucial. To overcome this challenge, a technique named Optimus was introduced recently. Optimus provides an adap- tive selection of head, layer, and matrix operations, to provide feature importance based interpretations for transformers. This work extends Optimus, adapting to two new transformer models, as well as the new task of multi-class classification, while also optimizing the time response of the technique. Experiments showed that the performance of Optimus remains consistent through different encoder-based transformer models and classification tasks