Advertisement

Signal, Image and Video Processing

, Volume 13, Issue 3, pp 423–430 | Cite as

Motion-resistant heart rate measurement from face videos using patch-based fusion

  • Zhao Yang
  • Xuezhi YangEmail author
  • Jing Jin
  • Xiu Wu
Original Paper
  • 191 Downloads

Abstract

The ability to measure heart rate (HR) from face videos is useful in applications such as neonatal monitoring, telemedicine and affective computing. In the realistic environments, subjects often have spontaneous head movements and facial expressions which severely degrade the performances of the current methods. We propose a novel patch-based fusion framework for estimating accurate HR from face videos in the presence of subjects’ motions. The wavelet time–frequency analysis is applied on the raw blood volume pulse (BVP) signals for selecting less contaminated patches. Furthermore, a weighted fusion formula is constructed to obtain the final precise BVP signal, which is based on frequency and gradient information. Our method is validated on both our self-collected dataset and public dataset MAHNOB-HCI. Compared with the state of the art, experimental results show that the proposed method has an obvious superiority in the accuracy and robustness.

Keywords

Photoplethysmography Imaging Wavelet time–frequency analysis Fusion Heart rate estimation 

Notes

Acknowledgements

We acknowledge funding support from: Training Programme Foundation for Application of Scientific and Technological Achievements of Hefei University of Technology (JZ2018YYPY0289) and the Specialized Research Fund for the Doctoral Program of Higher Education of China (JZ2018HGBZ0186).

References

  1. 1.
    Kannel, W.B., Kannel, C., Paffenbarger, R.S., Cupples, L.A.: Heart rate and cardiovascular mortality: the Framingham study. Am. Heart J. 113(6), 1489–1494 (1953).  https://doi.org/10.1016/0002-8703(87)90666-1 CrossRefGoogle Scholar
  2. 2.
    Temko, A.: Accurate heart rate monitoring during physical exercises using PPG. IEEE Trans. Biomed. Eng. 64(9), 2016–2024 (2017).  https://doi.org/10.1109/TBME.2017.2676243 CrossRefGoogle Scholar
  3. 3.
    Guven, G., Gurkan, H., Guz, U.: Biometric identification using fingertip electrocardiogram signals. Signal Image Video Process. 12, 1–8 (2018).  https://doi.org/10.1007/s11760-018-1238-4 CrossRefGoogle Scholar
  4. 4.
    Allen, J.: Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 28(3), R1-39 (2007).  https://doi.org/10.1088/0967-3334/28/3/R01 CrossRefGoogle Scholar
  5. 5.
    Wu, T., Blazek, V., Schmitt, H.: Photoplethysmography imaging: a new noninvasive and non-contact method for mapping of the dermal perfusion changes. Proc. SPIE 4163, 62–70 (2000).  https://doi.org/10.1117/12.407646 CrossRefGoogle Scholar
  6. 6.
    Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008).  https://doi.org/10.1364/OE.16.021434 CrossRefGoogle Scholar
  7. 7.
    Poh, M.Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010).  https://doi.org/10.1364/OE.18.010762 CrossRefGoogle Scholar
  8. 8.
    Kwon, S., Kim, H., Park K.S.: Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone. In: IEEE Engineering in Medicine and Biology Society (EMBC). San Diego, CA, USA, pp. 2174–2177 (2012). https://doi.org/10.1109/EMBC.2012.6346392
  9. 9.
    Poh, M.Z., McDuff, D.J., Picard, R.W.: Advancements in non-contact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011).  https://doi.org/10.1109/TBME.2010.2086456 CrossRefGoogle Scholar
  10. 10.
    Li, X.B., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Columbus, OH, USA, pp. 4264–4271 (2014). https://doi.org/10.1109/CVPR.2014.543
  11. 11.
    De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013).  https://doi.org/10.1109/TBME.2013.2266196 CrossRefGoogle Scholar
  12. 12.
    Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Algorithmic principles of remote-PPG. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017).  https://doi.org/10.1109/TBME.2016.2609282 CrossRefGoogle Scholar
  13. 13.
    Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Amplitude-selective filtering for remote-PPG. Biomed. Opt. Express 8(3), 1965–1980 (2017).  https://doi.org/10.1364/BOE.8.001965 CrossRefGoogle Scholar
  14. 14.
    Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: IEEE International Conference on Computer Vision (ICCV). Washington, DC, USA, pp. 3640–3648 (2015). https://doi.org/10.1109/ICCV.2015.415
  15. 15.
    Kumar, M., Veeraraghavan, A., Sabharwal, A.: DistancePPG: robust non-contact vital signs monitoring using a camera. Biomed. Opt. Express 6(5), 1565–1588 (2015).  https://doi.org/10.1364/BOE.6.001565 CrossRefGoogle Scholar
  16. 16.
    Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, pp. 2396–2404 (2016). https://doi.org/10.1109/CVPR.2016.263
  17. 17.
    Ha, R.Y., Nojima, K., Adams, W.J., Brown, S.A.: Analysis of facial skin thickness: defining the relative thickness index. Plast. Reconstr. Surg. 115(6), 1769–1773 (2005).  https://doi.org/10.1097/01.PRS.0000161682.63535.9B CrossRefGoogle Scholar
  18. 18.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). Kauai, HI, USA, pp. 511–518 (2001). https://doi.org/10.1109/CVPR.2001.990517
  19. 19.
    Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Portland, OR, USA, pp. 3444–3451 (2013). https://doi.org/10.1109/CVPR.2013.442
  20. 20.
    Tomasi, C., Kanade, T.: Detection and tracking of point features. Technical Report CMU-CS-91-132, Carnegie Mellon University (1991)Google Scholar
  21. 21.
    Daubechies, I., Heil, C.: Ten Lectures on Wavelets. Capital City Press, Vermont (1992)CrossRefzbMATHGoogle Scholar
  22. 22.
    Tarvainen, M.P., Ranta-aho, P.O., Karjalainen, P.A.: An advanced detrending method with application to HRV analysis. IEEE Trans. Biomed. Eng. 49(2), 172–175 (2002).  https://doi.org/10.1109/10.979357 CrossRefGoogle Scholar
  23. 23.
    Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012).  https://doi.org/10.1109/T-AFFC.2011.25 CrossRefGoogle Scholar
  24. 24.
    Huang, S.J., Hsieh, C.T., Huang, C.L.: Application of Morlet wavelets to supervise power system disturbances. IEEE Trans. Power Deliv. 14(1), 235–243 (1999).  https://doi.org/10.1109/61.736728 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Computer and InformationHefei University of TechnologyHefeiChina
  2. 2.Anhui Province Key Laboratory of Industry Safety and Emergency TechnologyHefeiChina

Personalised recommendations