Impact of annotation quality on model performance of welding defect detection using deep learning

The use of X-ray-based non-destructive testing (NDT) methods is widespread in the task of welding defect detection. Many scholars have turned to deep-learning computer vision models for defect detection in weld radiographic images in recent years. Before model training, annotating the collected image data is often necessary. We need to use annotation information to guide the model for effective learning. However, many researchers have been focused on developing better models or refining training strategies, often overlooking the quality of data annotation. This paper delved into the impact of eight types of low-quality annotations on the accuracy of object detection models. In comparison to accurate annotations, inaccuracies in the annotated locations significantly impact model performance, while errors in category annotations have a minor effect on model performance. Incorrect location affects both the recall and precision of the model, while incorrect categorization only impacts the precision of the model. Additionally, we observed that the extent of the impact of location errors is related to the detection accuracy of individual classes, with classes having higher original detection AP experiencing more substantial decreases in AP under location errors. Finally, we analyzed the influence of annotator habits on model performance. The study examines the effects of various types of low-quality annotations on model training and their impact on individual detection categories. Annotator habits lead to the left boundary of annotated boxes being less accurate than the right boundary, resulting in a greater impact of annotations biased to the left than those biased to the right. Based on experiments and analysis, we proposed annotation guidelines for weld defect detection tasks: prioritize the quality of location annotations over category accuracy and strive to include all objects, including those with ambiguous boundaries.