Weld seam profile extraction using top-down visual attention and fault detection and diagnosis via EWMA for the stable robotic welding process

  • Yinshui He
  • Zhuohua Yu
  • Jian Li
  • Guohong MaEmail author


Laser vision-sensing technologies are the most widely used to detect weld seam profiles during the intelligentized robotic welding process (IRWP) with thick steel plates, in which the weld seam profile extraction technology plays a crucial role for guiding the welding torch in real time. This paper presents an effective method to extract the weld seam profile from the intense arc background. To emphasize the weld seam profile in images and produce saliency maps at the initial stage, a top-down visual attention model is proposed using the target-driven characteristics of the weld seam profile and splashes. Due to the interference data surviving in the saliency map, a visual attention–based strategy is suggested to gradually discern the larger segments of the weld seam profile through local competition of dynamic saliency based on clustering results. For ineffective weld seam profile extraction resulting from empirical parameters used in the weld seam profile extraction process, the exponentially weighted moving average (EWMA) control chart is employed to implement fault detection and diagnosis (FDD) by monitoring irregular changes of slopes of the extracted weld seam profile. In the final stage, a novel step is arranged to retrieve the possible loss of the weld seam profile. Using the proposed method, validations are carried out using the welding experiments with T-joints and butt joints. Experimental results show that the ratio of successful extraction is over 97% and more stable welding processes with better welds are obtained. This method lays a good foundation for the general weld seam profile extraction process and shows a potential industrial application to the IRWP.


Weld seam profile extraction Fault detection and diagnosis Top-down visual attention Exponentially weighted moving average control chart Robotic welding 


Funding information

This work is financially supported by the National Natural Science Foundation of China under the grant no. 51575349, 51665037, and 51575348, and is partly supported by the State Key Laboratory of Smart Manufacturing for Special Vehicles and Transmission System (GZ2016KF002).


  1. 1.
    Chen SB, Lv N (2014) Research evolution on intelligentized technologies for arc welding process. J Manuf Process 16(1):109–122CrossRefGoogle Scholar
  2. 2.
    Teimouri R, Baseri H (2015) Forward and backward predictions of the friction stir welding parameters using fuzzy-artificial bee colony-imperialist competitive algorithm systems. J Intell Manuf 26(2):307–319CrossRefGoogle Scholar
  3. 3.
    Pashazadeh H, Gheisari Y, Hamedi M (2016) Statistical modeling and optimization of resistance spot welding process parameters using neural networks and multi-objective genetic algorithm. J Intell Manuf 27(3):549–559CrossRefGoogle Scholar
  4. 4.
    Xu Y, Fang G, Chen S, Zou JJ, Ye Z (2014) Real-time image processing for vision-based weld seam tracking in robotic GMAW. Int J Adv Manuf Technol 73(9–12):1413–1425CrossRefGoogle Scholar
  5. 5.
    Muhammad J, Altun H, Abo-Serie E (2017) Welding seam profiling techniques based on active vision sensing for intelligent robotic welding. Int J Adv Manuf Technol 88(1–4):127–145CrossRefGoogle Scholar
  6. 6.
    He L, Wu S, Wu C (2017) Robust laser stripe extraction for three-dimensional reconstruction based on a cross-structured light sensor. Appl Opt 56(4):823–832CrossRefGoogle Scholar
  7. 7.
    Zeng J, Chang B, Du D, Wang L, Chang S, Peng G, Wang W (2018) A weld position recognition method based on directional and structured light information fusion in multi-layer/multi-pass welding. Sensors 18(1):129CrossRefGoogle Scholar
  8. 8.
    Kiddee P, Fang Z, Tan M (2016) An automated weld seam tracking system for thick plate using cross mark structured light. Int J Adv Manuf Technol 87(9–12):3589–3603CrossRefGoogle Scholar
  9. 9.
    Yamazaki K, Suzuki R, Shimizu H, Koshiishi F (2012) Spatter and fume reduction in Co2 gas- shielded arc welding by regulated globular transfer. Weld World 56:12–19CrossRefGoogle Scholar
  10. 10.
    Pritschow G, Mueller S, Horber H (2002) Fast and robust image processing for laser stripe-sensors in arc welding automation in industrial electronics, 2002. ISIE 2002. Proceedings of the 2002 IEEE International Symposium on 2Google Scholar
  11. 11.
    Moon HS, Kim YB, Beattie RJ (2006) Multi sensor data fusion for improving performance and reliability of fully automatic welding system. Int J Adv Manuf Technol 28(3–4):286–293CrossRefGoogle Scholar
  12. 12.
    Gu WP, Xiong ZY, Wan W (2013) Autonomous seam acquisition and tracking system for multi-pass welding based on vision sensor. Int J Adv Manuf Technol 69(1–4):451–460CrossRefGoogle Scholar
  13. 13.
    Nguyen HC, Lee BR (2014) Laser-vision-based quality inspection system for small-bead laser welding. Int J Precis Eng Manuf 15(3):415–423CrossRefGoogle Scholar
  14. 14.
    Chen H, Liu W, Huang L, Xing G, Wang M, Sun H (2015) The decoupling visual feature extraction of dynamic three-dimensional V-type seam for gantry welding robot. Int J Adv Manuf Technol 80(9–12):1741–1749CrossRefGoogle Scholar
  15. 15.
    Yin XQ, Tao W, Feng YY, Gao Q, He QZ, Zhao H (2017) Laser stripe extraction method in industrial environments utilizing self-adaptive convolution technique. Appl Opt 56(10):2653–2660CrossRefGoogle Scholar
  16. 16.
    Ye Z, Fang G, Chen S, Zou JJ (2013) Passive vision based seam tracking system for pulse-MAG welding. Int J Adv Manuf Technol 67(9–12):1987–1996CrossRefGoogle Scholar
  17. 17.
    Zhou PY, Li J, Shen NM, Li F (2014) An improved weld seam extraction method using saliency detection for pipe-line welding based on GMAW and passive light. Appl Mech Mater 598:160–163CrossRefGoogle Scholar
  18. 18.
    Gharsallah MB, Braiek EB (2015) Weld inspection based on radiography image segmentation with level set active contour guided off-center saliency map. Adv Mater Sci Eng 2015:1–10CrossRefGoogle Scholar
  19. 19.
    Li N, Wang Z, Xu H, Sun L, Chen G (2016) Weld seam detection based on visual saliency for autonomous welding robots. In: 2016 IEEE workshop on advanced robotics and its social impacts (ARSO). IEEEGoogle Scholar
  20. 20.
    He YS, Chen YX, Wu D, Huang YM, Chen SB, Han Y (2015) A detection framework for weld seam profiles based on visual saliency. In: Tarn TJ, Chen SB, Chen XQ (eds) Robotic welding, intelligence and automation. RWIA 2014. Advances in intelligent systems and computing, vol 363. Springer, ChamGoogle Scholar
  21. 21.
    He Y, Chen H, Huang Y, Wu D, Chen S (2016) Parameter self-optimizing clustering for autonomous extraction of the weld seam based on orientation saliency in robotic MAG welding. J Intell Robot Syst 83(2):219–237CrossRefGoogle Scholar
  22. 22.
    He Y, Chen Y, Xu Y, Huang Y, Chen S (2016) Autonomous detection of weld seam profiles via a model of saliency-based visual attention for robotic arc welding. J Intell Robot Syst 81(3–4):395–406CrossRefGoogle Scholar
  23. 23.
    He Y, Xu Y, Chen Y, Chen H, Chen S (2016) Weld seam profile detection and feature point extraction for multi-pass route planning based on visual attention model. Robot Comput Integr Manuf 37:251–261CrossRefGoogle Scholar
  24. 24.
    Yan Z, Na L, Huang Y, Chen S (2014) Feature characters extraction with visual attention method based on three-light-path weld pool images. Trans CHN Weld Inst 35(8):53–56Google Scholar
  25. 25.
    Gong Y, Dai X, Li X (2010) Structured-light based joint recognition using bottom-up and top-down combined visual processing in image analysis and signal processing (IASP) in 2010 International Conference on Image Analysis & Signal Processing, IEEEGoogle Scholar
  26. 26.
    Danelljan M, Robinson A, Khan FS, Felsberg M (2016) Beyond correlation filters: learning continuous convolution operators for visual tracking in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)Google Scholar
  27. 27.
    Zou Y, Chen T (2018) Laser vision seam tracking system based on image processing and continuous convolution operator tracker. Opt Lasers Eng 105:141–149CrossRefGoogle Scholar
  28. 28.
    Li X, Li X, Ge S, Khyam MO, Luo C (2017) Automatic welding seam tracking and identification. IEEE Trans Ind Electron 64:7261–7271CrossRefGoogle Scholar
  29. 29.
    Ramirezmoreno DF, Schwartz O, Ramirezvillegas JF (2013) A saliency-based bottom-up visual attention model for dynamic scenes analysis. Biol Cybern 107(2):141–160MathSciNetCrossRefGoogle Scholar
  30. 30.
    Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal 20:1254–1259CrossRefGoogle Scholar
  31. 31.
    Itti L (2005) Models of bottom-up attention and saliency. Neurobiol Attention:576–582Google Scholar
  32. 32.
    Bernhard S, John P, Thomas H (2007) Graph-based visual saliency. In advances in neural information processing systems 19:Proceedings of the 2006 ConferenceMIT Press 545-552Google Scholar
  33. 33.
    Roberts SW (1959) Control chart tests based on exponentially weighted moving average. Technometrics 1:239–250CrossRefGoogle Scholar
  34. 34.
    Thomson M, Twigg PM, Majeed BA, Ruck N (2000) Statistical process control based fault detection of CHP units. Control Eng Pract 8:13–20CrossRefGoogle Scholar
  35. 35.
    Zhao Y, Wang S, Xiao F (2013) A statistical fault detection and diagnosis method for centrifugal chillers based on exponentially-weighted moving average control charts and support vector regression. Appl Therm Eng 51:560–572CrossRefGoogle Scholar
  36. 36.
    He Y, Yu Z, Li J, Ma G, Xu Y (2019) Fault correction of algorithm implementation for intelligentized robotic multipass welding process based on finite state machines. Robot Comput Integr Manuf 59:28–35CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Environment and Chemical EngineeringNanchang UniversityNanchangChina
  2. 2.School of Mechanical EngineeringNanchang University, Key Laboratory of Lightweight and High Strength Structural Materials of Jiangxi ProvinceNanchangChina
  3. 3.Institute of TechnologyEast China Jiao Tong UniversityNanchangChina

Personalised recommendations