Advancing Confidence Calibration and Quantification in Medication Recommendation

Medication recommendation (MR) has undergone rapid advancement in recent years, driven by its significant practical implications in healthcare. However, such high-risk scenarios still experience two critical yet overlooked challenges: the prevalent overconfidence in raw confidence for individual medications and the lack of a robust solution for confidence quantification in medication combinations. This paper represents the first in-depth study addressing this gap. We introduce two innovative methodologies tailored to the unique challenges of MR scenarios: 1) A discernible binning-based calibration method with theoretical guarantees for the confidence of individual medication. It guarantees distinct accuracy levels between adjacent bins and maintains consistent statistical reliability across calibration and test data, enabling calibrated confidence to reflect the correctness of medication recommendations distinctively. 2) A sample-based quantification method for the set confidence of medication combination, which is applicable for various existing performance metrics in MR. Utilizing representative deep MR models as backbones and conducting extensive experiments on the widely recognized MIMIC datasets, we empirically prove the effectiveness and robustness of our proposed methods. Our approaches not only improve the reliability of MR but also pave the way for more informed decision-making in clinical settings.