DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
arXiv:1907.11065
The paper introduces DropAttention, a dropout-based regularization technique applied to the attention weights of fully‑connected self‑attention layers in Transformers, showing improved performance and reduced overfitting across multiple tasks.
Abstract
Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.