An Inclusive Task-Aware Framework for Radiology Report Generation

To avoid the tedious and laborious radiology report writing, the automatic generation of radiology reports has drawn great attention recently. Previous studies attempted to directly transfer the image captioning method to radiology report generation given the apparent similarity between these two tasks. Although these methods can generate fluent descriptions, their accuracy for abnormal structure identification is limited due to the neglecting of the highly structured property and extreme data imbalance of the radiology report generation task. Therefore, we propose a novel task-aware framework to address the above two issues, composed of a task distillation module turning the image-level report to structure-level description, a task-aware report generation module for the generation of structure-specific descriptions, along with a classification token to identify and emphasize the abnormality of each structure, and an auto-balance mask loss to alleviate the serious data imbalance between normal/abnormal descriptions as well as the imbalance among different structures. Comprehensive experiments conducted on two public datasets demonstrate that the proposed method outperforms the state-of-the-art methods by a large margin (3.5% BLEU-1 improvement on MIMIC-CXR dataset) and can effectively improve the accuracy regarding the abnormal structures. The code is available at https://github.com/Reremee/ITA.