# Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record

**Authors:** Christopher McMaster, David FL Liew, Douglas EV Pires  
**arXiv:** [2303.14920](https://arxiv.org/abs/2303.14920)  
**Date:** 2023-03-27  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2303.14920v1/

## Key points

- The study proposes an approach for adapting pretrained language models to solve tabular prediction problems in the electronic health record (EHR) domain
- DeBERTa model is adapted using domain adaptation techniques to predict emergency department outcomes using the MIMIC-IV-ED dataset
- Pretraining a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts
- Proposed approach outperforms other models on two of three benchmark tasks for emergency department outcomes (p<0.001) and matches performance on the third task
- Inclusion of free text data and descriptive columns version of the dataset resulted in better model performance, highlighting the importance of data processing and mixing free text and tabular data in EHR datasets
- Small DeBERTa models can achieve competitive performance when fine-tuned on EHR datasets, which is important for compute-constrained settings such as hospitals
- Attribution scores were used to determine the importance of different input features in machine learning models, providing insights into factors influencing patient outcomes and pinpointing potentially modifiable risk factors
- This study presents a promising approach for adapting pretrained language models to solve tabular prediction problems in EHR domains with potential applications in improving decision-making and patient outcomes in healthcare settings while further evaluation is needed on a wider range of tasks and direct comparison with larger models

## Summary

This study proposes an approach for adapting pretrained language models to solve tabular prediction problems in the electronic health record (EHR) domain. Specifically, the DeBERTa model is adapted using domain adaptation techniques to predict emergency department outcomes using the MIMIC-IV-ED dataset. The proposed approach involves pretraining a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts. The performance of this model is compared with a DeBERTa model pre-trained on clinical texts from the institutional EHR (MeDeBERTa) and an XGBoost model. The results show that the proposed approach outperforms other models on two of three benchmark tasks for emergency department outcomes (p<0.001) and matches performance on the third task. The inclusion of free text data and descriptive columns version of the dataset resulted in better model performance, highlighting the importance of data processing and mixing free text and tabular data in EHR datasets. The study also demonstrates that small DeBERTa models can achieve competitive performance when fine-tuned on EHR datasets, which is important for compute-constrained settings such as hospitals. Attribution scores were used to determine the importance of different input features in machine learning models, providing insights into factors influencing patient outcomes and pinpointing potentially modifiable risk factors. Overall, this study presents a promising approach for adapting pretrained language models to solve tabular prediction problems in EHR domains. This approach has potential applications in improving decision-making and patient outcomes in healthcare settings while further evaluation is needed on a wider range of tasks and direct comparison with larger models.