复旦大学-浙江师范大学“机器学习与视觉前沿论坛”系列报告六

报告题目：SOFT: Softmax-free Transformer with Linear Complexity

报告专家：张力（复旦大学）

报告时间：11月29日16:00-16:45

报告地点：腾讯会议号766-826-483（复旦浙师大MLV前沿论坛）

报告摘要：ViTs have pushed the state-of-the-art for various visual recognition tasks. However, the employment of self-attention modules results in a quadratic complexity. An in-depth analysis in this work shows that they are either theoretically flawed or empirically ineffective for visual recognition. We further identify that their limitations are rooted in keeping the softmax self-attention during approximations. Keeping this softmax operation challenges any subsequent linearization efforts. Based on this insight, a softmax-free transformer or SOFT is proposed. To remove softmax in self-attention, Gaussian kernel function is used to replace the dot-product similarity without further normalization. This enables a full self-attention to be approximated via a low-rank matrix decomposition. The robustness of the approximation is achieved by calculating its Moore-Penrose inverse using a Newton-Raphson method. Extensive experiments on ImageNet show that SOFT significantly improves the computational efficiency of existing ViT variants, resulting in superior trade-off between accuracy and complexity.

邀请人:郑忠龙