TY - JOUR
T1 - Integrating clinical knowledge and imaging for medical report generation
AU - Zhao, Meng
AU - Liu, Juncai
AU - Shen, Hongyu
AU - Yan, Bin
AU - Pei, Mingtao
AU - Wang, Yi
N1 - Publisher Copyright:
© 2025 Elsevier B.V.
PY - 2025/9
Y1 - 2025/9
N2 - Medical report generation is an important cross-modal task in the field of medicine, aiming to automatically generate professional and accurate reports for given medical images. Integrating clinical knowledge into the task of medical report generation can enhance the semantic accuracy of medical image feature descriptions and improves the interpretability and robustness of the model. In this paper, we propose to integrate clinical knowledge and image content to generate medical report. The clinical knowledge is presented by a clinical knowledge graph, where anatomical structures and observations are represented as nodes, and their relationships are represented as edges. We design a graph generation module to dynamically generate relevant knowledge graphs specific to each image, which can diversify the knowledge graph structures and expand the coverage of clinical knowledge to provide a more extensive range of clinical knowledge during report generation. Furthermore, we design a graph attention module to facilitate optimized feature representation within the clinical knowledge graph by incorporating message passing between nodes and edges. This fosters a more comprehensive understanding of the relationships and significance of clinical information. Experimental evaluations conducted on the IU X-Ray and MIMIC CXR datasets demonstrate the superiority of the proposed method in generating medical reports. The results highlight the potential of leveraging clinical knowledge for enhancing the precision and clinical relevance of the generated reports.
AB - Medical report generation is an important cross-modal task in the field of medicine, aiming to automatically generate professional and accurate reports for given medical images. Integrating clinical knowledge into the task of medical report generation can enhance the semantic accuracy of medical image feature descriptions and improves the interpretability and robustness of the model. In this paper, we propose to integrate clinical knowledge and image content to generate medical report. The clinical knowledge is presented by a clinical knowledge graph, where anatomical structures and observations are represented as nodes, and their relationships are represented as edges. We design a graph generation module to dynamically generate relevant knowledge graphs specific to each image, which can diversify the knowledge graph structures and expand the coverage of clinical knowledge to provide a more extensive range of clinical knowledge during report generation. Furthermore, we design a graph attention module to facilitate optimized feature representation within the clinical knowledge graph by incorporating message passing between nodes and edges. This fosters a more comprehensive understanding of the relationships and significance of clinical information. Experimental evaluations conducted on the IU X-Ray and MIMIC CXR datasets demonstrate the superiority of the proposed method in generating medical reports. The results highlight the potential of leveraging clinical knowledge for enhancing the precision and clinical relevance of the generated reports.
KW - Clinical knowledge graph
KW - Graph attention
KW - Radiology reports generation
UR - http://www.scopus.com/inward/record.url?scp=105005520521&partnerID=8YFLogxK
U2 - 10.1016/j.patrec.2025.04.036
DO - 10.1016/j.patrec.2025.04.036
M3 - Article
AN - SCOPUS:105005520521
SN - 0167-8655
VL - 195
SP - 59
EP - 65
JO - Pattern Recognition Letters
JF - Pattern Recognition Letters
ER -