Metal surface defects can easily lead to structural failure in aircraft and spacecraft manufacturing, which brings great accident risk. Aiming at the poor performance of traditional YOLOv5s algorithm in detecting metal surface punching, filamentous spots and crescent notch, an improved YOLOv5s algorithm is proposed. This algorithm adds Ghost network to Backbone. The MHSA attention mechanism was added in the Neck network. Replace the original loss function with DIOU. The experimental results show that the improved network, compared with the original network, improves the detection accuracy of the model by 3.2%, and has low false detection and missed detection rates, which confirms the effectiveness of the proposed method in metal surface defect detection.