Cross-modal Network of Mining Text-knowledge for Radiology Report Generation

To reduce the burden on radiologists and improve the accuracy of disease diagnosis, generating radiology reports automatically aims to automate the generation of accurate and seamless diagnostic reports from radiology images. However, the field still faces several challenges. Firstly, medical images are very similar, and fine-grained visual differences and data bias in the dataset can result in disease details being neglected. In addition, medical reports require a detailed and fluent representation of long paragraphs rather than a single short-sentence description. To address these limitations, this paper proposes a cross-modal network based on text-knowledge mining for radiology report generation. The model uses a Cross-modal Memory Network to facilitate image-text interaction. Then, we cluster ground truth reports and use the clustering results as a second label to learn fine-grained visual details related to the text. In addition, medical guiding vocabulary is introduced to improve the images' encoding capabilities to mitigate data bias. Our proposed method performs well on the benchmark dataset IU X-Ray, outperforming many state-of-the-art models. Furthermore, we provide ablation experiments to demonstrate the effectiveness of the proposed components.