RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports

Jiawei Du; Jia Guo; Weihang Zhang; Shengzhu Yang; Hanruo Liu; Huiqi Li; Ningli Wang

doi:10.1007/978-3-031-72390-2_66

RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports

Jiawei Du
, Jia Guo
, Weihang Zhang
, Shengzhu Yang
, Hanruo Liu
, Huiqi Li^*
, Ningli Wang

^*Corresponding author for this work

Beijing Institute of Technology
Capital Medical University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

22 Citations (Scopus)

Abstract

The Vision-Language Foundation model is increasingly investigated in the fields of computer vision and natural language processing, yet its exploration in ophthalmology and broader medical applications remains limited. The challenge is the lack of labeled data for the training of foundation model. To handle this issue, a CLIP-style retinal image foundation model is developed in this paper. Our foundation model, RET-CLIP, is specifically trained on a dataset of 193,865 patients to extract general features of color fundus photographs (CFPs), employing a tripartite optimization strategy to focus on left eye, right eye, and patient level to reflect real-world clinical scenarios. Extensive experiments demonstrate that RET-CLIP outperforms existing benchmarks across eight diverse datasets spanning four critical diagnostic categories: diabetic retinopathy, glaucoma, multiple disease diagnosis, and multi-label classification of multiple diseases, which demonstrate the performance and generality of our foundation model. The sourse code and pre-trained model are available at https://github.com/sStonemason/RET-CLIP.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings
Editors	Marius George Linguraru, Qi Dou, Aasa Feragen, Stamatia Giannarou, Ben Glocker, Karim Lekadir, Julia A. Schnabel
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	709-719
Number of pages	11
ISBN (Print)	9783031723896
DOIs	https://doi.org/10.1007/978-3-031-72390-2_66
Publication status	Published - 2024
Event	27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024 - Marrakesh, Morocco Duration: 6 Oct 2024 → 10 Oct 2024

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15012 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024
Country/Territory	Morocco
City	Marrakesh
Period	6/10/24 → 10/10/24

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Keywords

Foundation Model
Retinal Fundus Image
Vision-Language Pre-training

Access to Document

10.1007/978-3-031-72390-2_66

Cite this

Du, J., Guo, J., Zhang, W., Yang, S., Liu, H., Li, H., & Wang, N. (2024). RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports. In M. G. Linguraru, Q. Dou, A. Feragen, S. Giannarou, B. Glocker, K. Lekadir, & J. A. Schnabel (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings (pp. 709-719). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15012 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-72390-2_66

Du, Jiawei ; Guo, Jia ; Zhang, Weihang et al. / RET-CLIP : A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings. editor / Marius George Linguraru ; Qi Dou ; Aasa Feragen ; Stamatia Giannarou ; Ben Glocker ; Karim Lekadir ; Julia A. Schnabel. Springer Science and Business Media Deutschland GmbH, 2024. pp. 709-719 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c3e0b48fb2f0466aa5283e5359b987cd,

title = "RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports",

abstract = "The Vision-Language Foundation model is increasingly investigated in the fields of computer vision and natural language processing, yet its exploration in ophthalmology and broader medical applications remains limited. The challenge is the lack of labeled data for the training of foundation model. To handle this issue, a CLIP-style retinal image foundation model is developed in this paper. Our foundation model, RET-CLIP, is specifically trained on a dataset of 193,865 patients to extract general features of color fundus photographs (CFPs), employing a tripartite optimization strategy to focus on left eye, right eye, and patient level to reflect real-world clinical scenarios. Extensive experiments demonstrate that RET-CLIP outperforms existing benchmarks across eight diverse datasets spanning four critical diagnostic categories: diabetic retinopathy, glaucoma, multiple disease diagnosis, and multi-label classification of multiple diseases, which demonstrate the performance and generality of our foundation model. The sourse code and pre-trained model are available at https://github.com/sStonemason/RET-CLIP.",

keywords = "Foundation Model, Retinal Fundus Image, Vision-Language Pre-training",

author = "Jiawei Du and Jia Guo and Weihang Zhang and Shengzhu Yang and Hanruo Liu and Huiqi Li and Ningli Wang",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024 ; Conference date: 06-10-2024 Through 10-10-2024",

year = "2024",

doi = "10.1007/978-3-031-72390-2\_66",

language = "English",

isbn = "9783031723896",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "709--719",

editor = "Linguraru, \{Marius George\} and Qi Dou and Aasa Feragen and Stamatia Giannarou and Ben Glocker and Karim Lekadir and Schnabel, \{Julia A.\}",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings",

address = "Germany",

}

Du, J, Guo, J, Zhang, W, Yang, S, Liu, H, Li, H & Wang, N 2024, RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports. in MG Linguraru, Q Dou, A Feragen, S Giannarou, B Glocker, K Lekadir & JA Schnabel (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15012 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 709-719, 27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024, Marrakesh, Morocco, 6/10/24. https://doi.org/10.1007/978-3-031-72390-2_66

RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports. / Du, Jiawei; Guo, Jia; Zhang, Weihang et al.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings. ed. / Marius George Linguraru; Qi Dou; Aasa Feragen; Stamatia Giannarou; Ben Glocker; Karim Lekadir; Julia A. Schnabel. Springer Science and Business Media Deutschland GmbH, 2024. p. 709-719 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15012 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - RET-CLIP

T2 - 27th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2024

AU - Du, Jiawei

AU - Guo, Jia

AU - Zhang, Weihang

AU - Yang, Shengzhu

AU - Liu, Hanruo

AU - Li, Huiqi

AU - Wang, Ningli

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024

Y1 - 2024

N2 - The Vision-Language Foundation model is increasingly investigated in the fields of computer vision and natural language processing, yet its exploration in ophthalmology and broader medical applications remains limited. The challenge is the lack of labeled data for the training of foundation model. To handle this issue, a CLIP-style retinal image foundation model is developed in this paper. Our foundation model, RET-CLIP, is specifically trained on a dataset of 193,865 patients to extract general features of color fundus photographs (CFPs), employing a tripartite optimization strategy to focus on left eye, right eye, and patient level to reflect real-world clinical scenarios. Extensive experiments demonstrate that RET-CLIP outperforms existing benchmarks across eight diverse datasets spanning four critical diagnostic categories: diabetic retinopathy, glaucoma, multiple disease diagnosis, and multi-label classification of multiple diseases, which demonstrate the performance and generality of our foundation model. The sourse code and pre-trained model are available at https://github.com/sStonemason/RET-CLIP.

AB - The Vision-Language Foundation model is increasingly investigated in the fields of computer vision and natural language processing, yet its exploration in ophthalmology and broader medical applications remains limited. The challenge is the lack of labeled data for the training of foundation model. To handle this issue, a CLIP-style retinal image foundation model is developed in this paper. Our foundation model, RET-CLIP, is specifically trained on a dataset of 193,865 patients to extract general features of color fundus photographs (CFPs), employing a tripartite optimization strategy to focus on left eye, right eye, and patient level to reflect real-world clinical scenarios. Extensive experiments demonstrate that RET-CLIP outperforms existing benchmarks across eight diverse datasets spanning four critical diagnostic categories: diabetic retinopathy, glaucoma, multiple disease diagnosis, and multi-label classification of multiple diseases, which demonstrate the performance and generality of our foundation model. The sourse code and pre-trained model are available at https://github.com/sStonemason/RET-CLIP.

KW - Foundation Model

KW - Retinal Fundus Image

KW - Vision-Language Pre-training

UR - https://www.scopus.com/pages/publications/85208174466

U2 - 10.1007/978-3-031-72390-2_66

DO - 10.1007/978-3-031-72390-2_66

M3 - Conference contribution

AN - SCOPUS:85208174466

SN - 9783031723896

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 709

EP - 719

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings

A2 - Linguraru, Marius George

A2 - Dou, Qi

A2 - Feragen, Aasa

A2 - Giannarou, Stamatia

A2 - Glocker, Ben

A2 - Lekadir, Karim

A2 - Schnabel, Julia A.

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 6 October 2024 through 10 October 2024

ER -

Du J, Guo J, Zhang W, Yang S, Liu H, Li H et al. RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports. In Linguraru MG, Dou Q, Feragen A, Giannarou S, Glocker B, Lekadir K, Schnabel JA, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 - 27th International Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 709-719. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-72390-2_66