Focusing on Abnormal: Visual Contrastive Classification and Semantic Enhancement for Medical Report Generation

Automating medical report generation from radiology images is vital for accurate, standardized diagnoses. Abnormal regions in medical images, often small and rare, are challenging to detect, leading models to overlook critical disease features and generate repetitive healthy content. We propose V-C2SE, a novel model that enhances abnormality detection through two key strategies within the visual modality: 1) Visual Contrastive Classification (VC2), which aligns disease-specific features across random samples using contrastive learning, improving the model’s focus on abnormal semantics during encoding; 2) Visual Semantic Enhancement (VSE), which constructs healthy templates to amplify abnormal features in a feature-space augmentation paradigm, ensuring precise report generation. By leveraging contrastive learning and healthy templates, V-C2SE detects subtle abnormalities with high precision and generates clinically relevant reports. Evaluated on IU X-Ray and MIMIC-CXR datasets, V-C2SE achieves competitive results with state-of-the-art methods across natural language generation (NLG) and clinical efficacy (CE) metrics, producing high-quality, semantically accurate reports. Our approach addresses the critical challenge of focusing on rare abnormalities and enhancing diagnostic efficiency.