Metal Defect Image Recognition Method Based on Shallow CNN Fusion Transformer

doi:10.3969/j.issn.1004-132X.2022.19.003

Abstract

Abstract: Aiming at the problems of large amount of parameters and calculation in the field of metal defect recognition， a metal defect recognition method was proposed based on shallow CNN and Transformer model. The shallow CNN was used to learn the local information and position information of the images， and the Transformer was used to learn the global information of the images. At the same time， the channel attention module SE was introduced to pay attention to the important feature channels to realize the defect image recognition. The effectiveness of this method was verified by introducing the open defect data set， and the universality of this method was verified by using the self-built defect ultrasonic data set. The experimental results show that the proposed method has strong universality and may effectively identify metal defect images on small and medium-sized data sets.

Key words: metal defect identification, deep learning, convolutional neural network（CNN）, Transformer model, multi-head attention

摘要： 针对金属缺陷识别领域中传统深度学习方法存在参数量多、计算量大的问题，提出了一种浅层卷积神经网络融合Transformer模型的金属缺陷识别方法。利用浅层卷积神经网络学习图像局部信息与位置信息，通过Transformer学习图像全局信息，同时引入通道注意力模块SE关注重要特征通道，实现缺陷图像识别。通过引入公开缺陷数据集验证该方法的有效性，同时利用自建缺陷超声数据集验证所提方法的通用性。实验结果表明，在中小规模数据集上，该方法通用性较强，能够对金属缺陷图像进行有效识别。

关键词: 金属缺陷识别, 深度学习, 卷积神经网络, Transformer模型, 多头注意力

CLC Number:

TP391

TANG Donglin, YANG Zhou, CHENG Heng, LIU Mingxuan, ZHOU Li, DING Chao. Metal Defect Image Recognition Method Based on Shallow CNN Fusion Transformer[J]. China Mechanical Engineering, 2022, 33(19): 2298-2305，2316.

唐东林, 杨洲, 程衡, 刘铭璇, 周立, 丁超. 浅层卷积神经网络融合Transformer的金属缺陷图像识别方法[J]. 中国机械工程, 2022, 33(19): 2298-2305，2316.

References

［1］LYU Xiaoming， DUAN Fajie， JIANG Jiajia， et al. Deep Metallic Surface Defect Detection：the New Benchmark and Detection Network［J］. Sensors， 2020， 20（6）：1562.
［2］LIN H I， WIBOWO F S. Image Data Assessment Approach for Deep Learning-based Metal Surface Defect-detection Systems［J］. IEEE Access， 2021， 9：47621-47638.
［3］唐东林，周立，吴续龙，等. CNN融合PCA-DT模型的金属缺陷识别研究［J/OL］. 机械科学与技术：1-8［2021-11-12］. https：∥doi. org/10. 13433/j. cnki. 1003-8728. 20200478.
TANG Donglin， ZHOU Li， WU Xulong， et al. Research on Metal Defect Recognition of CNN Fusion PCA-DT Model［J/OL］. Mechanical Science and Technology for Aerospace Engineering：1-8［2021-11-12］. https：∥doi. org/10. 13433/j. cnki. 1003-8728. 20200478.
［4］GRZEGORZ P. Multi-Sensor Data Integration Using Deep Learning for Characterization of Defects in Steel Elements［J］. Sensors， 2018， 18（2）：292.
［5］WANG J， FU P， GAO R X. Machine Vision Intelligence for Product Defect Inspection Based on Deep Learning and Hough Transform［J］. Journal of Manufacturing Systems， 2019， 51：52-60.
［6］WEI X， YANG Z， LIU Y， et al. Railway Track Fastener Defect Detection Based on Image Processing and Deep Learning Techniques：a comparative study［J］. Engineering Applications of Artificial Intelligence， 2019， 80：66-81.
［7］GU J， WANG Z， KUEN J， et al. Recent Advances in Convolutional Neural Networks［J］. Pattern Recognition：the Journal of the Pattern Recognition Society， 2018， 77：354-377.
［8］TAO X， ZHANG D， MA W， et al. Automatic Metallic Surface Defect Detection and Recognition with Convolutional Neural Networks［J］. Applied Sciences， 2018， 8（9）：1575.
［9］GUANS， LEI M， LU H. A Steel Surface Defect Recognition Algorithm Based on Improved Deep Learning Network Model Using Feature Visualization and Quality Evaluation［J］. IEEE Access， 2020， 8：49585-49895.
［10］WANG S， XIA X， YE L， et al. Automatic Detection and Classification of Steel Surface Defect Using Deep Convolutional Neural Networks［J］. Metals -open Access Metallurgy Journal， 2021， 11（3）：388.
［11］姜洪权，贺帅，高建民，等. 一种改进卷积神经网络模型的焊缝缺陷识别方法［J］. 机械工程学报， 2020， 56（8）：235-242.
JIANG Hongquan， HE Shuai， GAO Jianmin， et al. An Improved Convolutional Neural Network for Weld Defect Recognition［J］. Journal of Mechanical Engineering， 2020， 56（8）：235-242.
［12］HE D， XU K， WANG D. Design of Multi-scale Receptive Field Convolutional Neural Network for Surface Inspection of Hot Rolled Steels［J］. Imageand Vision Computing， 2019， 89：12-20.
［13］刘涵，郭润元. 基于X射线图像和卷积神经网络的石油钢管焊缝缺陷检测与识别［J］. 仪器仪表学报， 2018， 39（4）：247-256.
LIU Han， GUO Runyuan. Detection and Identification of SAWH Pipe Weld Defects Based on X-ray Image and CNN［J］. Chinese Journal of Scientific Instrument， 2018， 39（4）：247-256.
［14］樊丁，胡桉得，黄健康，等. 基于改进卷积神经网络的管焊缝X射线图像缺陷识别方法［J］. 焊接学报， 2020， 41（1）：7-11，97.
FAN Ding， HU Ande， HUANG Jiankang， et al. X-ray Image Defect Recognition Method for Pipe Weld Based on Improved Convolutional Neural Network［J］. Transactions of the China Welding Institution， 2020， 41（1）：7-11，97.
［15］YUN J P， SHIN W C， KOO G， et al. Automated Defect Inspection System for Metal Surfaces Based on Deep Learning and Data Augmentation［J］. Journal of Manufacturing Systems， 2020， 55：317-324.
［16］VASWANI A， SHAZEER N， PARMAR N， et al. Attention is All You Need［C］∥Advances in Neural Information Processing Systems. Long Beach， CA， 2017：6000-6010.
［17］DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An Image is Worth 16×16 Words：Transformers for Image Recognition at Scale［J］. arXiv Preprint， arXiv：2010. 11929， 2020.
［18］TOUVRON H, CORD M, DOUZE M, et al. Training Data-efficient Image Transformers &Distillation through Attention［C］∥Proceedings of the International Conference on Machine Learning (ICML). ［Online Confenerce］, 2021, 139:10347-10357.
［19］ZHOU D， KANG B， JIN X， et al. DeepViT：towards Deeper Vision Transformer［J］. arXiv Preprint， arXiv：2013. 11886， 2021.
［20］TOUVRON H, CORDM, SABLAYROLLES A, et al. Going Deeper with Image Transformers［C］∥Proceedings of the 18th IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada, 2021:32-42.
［21］HU J， SHEN L， SUN G. Squeeze-and-excitation Networks［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Salt Lake City， UT, 2018：7132-7141.
［22］HE Y， SONG K， MENG Q， et al. An End-to-end Steel Surface Defect Detection Approach via Fusing Multiple Hierarchical Features［J］. IEEE Transactions on Instrumentation and Measurement， 2019， 69（4）：1493-1504.
［23］SELVARAJU R R， COGSWELL M， DAS A， et al. Grad-cam：Visual Explanations from Deep Networks via Gradient-based Localization［C］∥Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy, 2017：618-626.
［24］SIMONYAN K， ZISSERMAN A. Very Deep Convolutional Networks for Large-scale Image Recognition［J］. arXiv Preprint, arXiv:1409. 1556, 2014.
［25］SZEGEDY C, LIU W, JIA Y Q, et al. Going Deeper with Convolutions［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Boston, MA, 2015:1-9.
［26］HE K， ZHANG X， REN S， et al. Deep Residual Learning for Image Recognition［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Seattle, WA, 2016：770-778.
［27］SANDLER M， HOWARD A， ZHU M， et al. MobileNetv2：Inverted Residuals and Linear Bottlenecks［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, 2018：4510-4520.
［28］MA N， ZHANG X， ZHENG H T， et al. ShuffleNet v2：Practical Guidelines for Efficient CNN Architecture Design［C］∥Proceedings of the European Conference on Computer Vision（ECCV）. Munich, Germany, 2018：116-131.

[1]	WANG Hao, LUO Haodong, SHI Yazhong, WANG Liwen, ZHANG Wei, WANG Zhong. A Topology Optimization Method Based on End-to-end Deep Learning Framework TOPO-U-Net [J]. China Mechanical Engineering, 2026, 37(1): 174-183.
[2]	XIAO Yufeng, ZHANG Chaoyong, Saixiyalatu , MENG Yifan, ZHU Chuanjun. Multi-step Ahead Real-time Prediction of Tool Wear Based on YOLOv11-Seg and Transformer Model [J]. China Mechanical Engineering, 2025, 36(12): 2944-2951.
[3]	Zulong YAN, Qilong PANG, Jianlong XIONG. Prediction of Three-dimensional Machined Surface Topography of KDP Crystals Based on Deep Learning [J]. China Mechanical Engineering, 2025, 36(10): 2329-2334.
[4]	ZHAO Yunjie, HE Yansong, ZHANG Zhifei, XU Zhongming. 3D Beamforming Map Compression Method Based on Generative Model [J]. China Mechanical Engineering, 2025, 36(07): 1520-1529.
[5]	YANG Heran1, 2, ZHANG Peijie1, 2, SUN Xingwei1, 2, PAN Fei1, 2, LIU Yin1, 2. Surface Roughness Prediction for Screw Belt Grinding Based on Improved CNN [J]. China Mechanical Engineering, 2025, 36(02): 325-332.
[6]	ZENG Hao, CAO Huajun, DONG Jianxiong. Tool Wear Prediction Method Based on ISABO-IBiLSTM Model [J]. China Mechanical Engineering, 2024, 35(11): 1995-2006.
[7]	LI Yue1, 2, XIE Heng1, ZHOU Gongbo1, 2, ZHOU Ping1, 2, LI Menggang1, 2. Soft Sensor Modeling and Uncertainty Analysis Approach of Tool Wear Based on Semi-supervised Bayesian Transformer [J]. China Mechanical Engineering, 2024, 35(11): 2015-2025.
[8]	NIE Xin, TAN Tian, SHEN Danfeng. Research on Stamping Springback of Automobile Beam Parts Based on Deep Learning [J]. China Mechanical Engineering, 2023, 34(07): 838-846.
[9]	LIU Huiyong, ZHANG Song, LI Jianfeng, LUAN Xiaona, . Tool Wear Detection Based on Improved CNN-BiLSTM Model [J]. China Mechanical Engineering, 2022, 33(16): 1940-1947，1956.
[10]	YU Hao, HUANG Huagui, ZHENG Jiali, ZHAO Tielin, ZHOU Xinliang. Non-contact On-line Inspection Method for Surface Defects of Cross-rolling Piercing Plugs for Seamless Steel Tubes [J]. China Mechanical Engineering, 2022, 33(14): 1717-1724.
[11]	YANG Guangyou, LIU Lang, XI Chenbo. Bearing Fault Diagnosis Based on SA-ACGAN Data Generation Model [J]. China Mechanical Engineering, 2022, 33(13): 1613-1621.
[12]	AI Qingbo, ZHANG Jie, CHENG Hui, LYU Youlong, ZUO Liling, HU Lan. Analysis for Roll-bending Forming Quality of Spaceflight Thin-walled Cylindrical Workpieces Based on PointCPP-LSF Method#br# [J]. China Mechanical Engineering, 2022, 33(08): 977-985.
[13]	HE Yan, XIAO Zhen, LI Yufeng, WU Pengcheng, LIU Degao, DU Jiang. An Assembly Quality Prediction Method for Automotive Instrument Clusters Using CNN-SVR [J]. China Mechanical Engineering, 2022, 33(07): 825-833.
[14]	ZHANG Shengwen, ZHOU Xi, LI Bincheng, CHENG Dejun, CHEN Wendi. Information Extraction Method of Part Machining Features Based on Image Deep Learning [J]. China Mechanical Engineering, 2022, 33(03): 348-355.
[15]	LI Congbo, LONG Yun, CUI Jiabin, ZHAO Xikun, ZHAO De. Surface Roughness Prediction Method of CNC Milling Based on Multi-source Heterogeneous Data [J]. China Mechanical Engineering, 2022, 33(03): 318-328.