On the basis of semantic segmentation models, a method is proposed to transfer and improve defect recognition knowledge from complex networks to simple networks through channel wise distillation. This method is based on the teacher-student network, and during the training process, the student network model imitates the defect recognition method of the teacher network model. The KL divergence is used to measure differences, and the jointly optimized student network model has a lightweight model structure and high defect recognition ability. The experimental verification on the dataset of steel surface defects shows that after distillation optimization, the average similarity coefficient mDice of the ResUnet+ConvNeXt-T semantic segmentation benchmark network for defect segmentation is increased from 0.7503 to 0.7711. The inference speed of the student network model is improved by about 77% compared to the teacher network model, effectively improving the recognition ability of lightweight student network models for steel surface defects.