#knowledge distillation
6 resultsToken-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
Hao Sun, Xu Tan, Jun-Wei Gan +4
The paper introduces a token‑level ensemble distillation method using Transformers to improve grapheme‑to‑phoneme conversion accuracy and reduce model size, leveraging unlabeled da…
A Comprehensive Overhaul of Feature Distillation
Byeongho Heo, Jeesoo Kim, Sangdoo Yun +3
The paper proposes a new feature distillation loss that combines a margin ReLU transform, a novel feature position, and a partial L2 distance to improve network compression, achiev…
Self-Knowledge Distillation in Natural Language Processing
Sangchul Hahn, Heeyoul Choi
The paper introduces a self‑knowledge distillation technique that uses a model’s own soft target probabilities from the embedding space to improve training, and demonstrates its ef…
Learning Lightweight Lane Detection CNNs by Self Attention Distillation
Yuenan Hou, Zheng Ma, Chunxiao Liu +1
The paper introduces Self Attention Distillation (SAD), a knowledge‑distillation technique that uses a model’s own attention maps as free supervision to improve lightweight lane‑de…
Similarity-Preserving Knowledge Distillation
Frederick Tung, Greg Mori
The paper introduces a knowledge distillation method that trains a student network to preserve the pairwise similarity relationships of activations observed in a teacher network.
Distill-to-Label: Weakly Supervised Instance Labeling Using Knowledge Distillation
Jayaraman J. Thiagarajan, Satyananda Kashyap, Alexandros Karagyris
The paper proposes a weakly supervised method that uses multiple instance learning and knowledge distillation to generate instance-level labels from only image-level annotations, d…