Papers Published

1. Si, S; Wang, R; Wosik, J; Zhang, H; Dov, D; Wang, G; Henao, R; Carin, L, Students Need More Attention: BERT-based Attention Model for Small Data with Application to Automatic Patient Message Triage, Proceedings of Machine Learning Research, vol. 126 (January, 2020), pp. 436-456 .
   (last updated on 2024/12/31)

   Abstract:
   Small and imbalanced datasets commonly seen in healthcare represent a challenge when training classifiers based on deep learning models. So motivated, we propose a novel framework based on BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical TextMining). Specifically, (i) we introduce Label Embeddings for Self-Attention in each layer of BERT, which we call LESA-BERT, and (ii) by distilling LESA-BERT to smaller variants, we aim to reduce overfitting and model size when working on small datasets. As an application, our framework is utilized to build a model for patient portal message triage that classifies the urgency of a message into three categories: non-urgent, medium and urgent. Experiments demonstrate that our approach can outperform several strong baseline classifiers by a significant margin of 4.3% in terms of macro F1 score.