Continuous respiratory rate monitoring during an acute hypoxic challenge using a depth sensing camera
- 74 Downloads
Respiratory rate is a well-known to be a clinically important parameter with numerous clinical uses including the assessment of disease state and the prediction of deterioration. It is frequently monitored using simple spot checks where reporting is intermittent and often prone to error. We report here on an algorithm to determine respiratory rate continuously and robustly using a non-contact method based on depth sensing camera technology. The respiratory rate of 14 healthy volunteers was studied during an acute hypoxic challenge where blood oxygen saturation was reduced in steps to a target 70% oxygen saturation and which elicited a wide range of respiratory rates. Depth sensing data streams were acquired and processed to generate a respiratory rate (RRdepth). This was compared to a reference respiratory rate determined from a capnograph (RRcap). The bias and root mean squared difference (RMSD) accuracy between RRdepth and the reference RRcap was found to be 0.04 bpm and 0.66 bpm respectively. The least squares fit regression equation was determined to be: RRdepth = 0.99 × RRcap + 0.13 and the resulting Pearson correlation coefficient, R, was 0.99 (p < 0.001). These results were achieved with a 100% reporting uptime. In conclusion, excellent agreement was found between RRdepth and RRcap. Further work should include a larger cohort combined with a protocol to further test algorithmic performance in the face of motion and interference typical of that experienced in the clinical setting.
KeywordsNon-contact monitoring Depth sensing Respiratory rate Hypoxic challenge
The clinical importance of respiratory rate (RR) is well known as it provides important information regarding many aspects of a patient’s respiratory status. Changes in RR are often one of the earliest and more important indicators that precedes major clinical manifestations of serious complications such as respiratory tract infections, respiratory depression associated with opioid consumption, anaesthesia and/or sedation, as well as respiratory failure [1, 2, 3]. A wide range of methods have been proposed for the determination of respiratory rate using non-contact means including RGB video camera systems [4, 5], infrared camera systems , laser vibrometry , piezoelectric bed sensors , doppler radar , thermal imaging  and acoustic sensors . The determination of respiratory information from depth data has received relatively less attention than RGB video methods, although such systems are well suited to this task.
The inflation and deflation of the lung during a respiratory cycle is a phenomenon measurable with a depth camera. Many studies have compared tidal volume, as measured by a reference system (e.g. a spirometer), with the tidal volume extracted by a depth camera system based on morphological changes in the chest wall. One of the earliest tidal volume measurements extracted in this way was carried out by Yu et al. . They assessed a Kinect V1 based system against a spirometer and achieved a correlation coefficient, R = 0.97 (p < 0.001), based on 12 healthy subjects undertaking a range or respiratory activities including shallow, middle and deep breathing as well as isovolume maneuvers. Aoki et al.  performed a similar study using a Kinect V1 system comparing the tidal volumes to that measured using a gas analyser. Four healthy subjects were monitored over a 180 s acquisition period and they obtained an R = 0.99 correlation coefficient. A more recent study by Seppänen et al.  evaluated depth camera extracted tidal volume over a range of respiratory rates. Part of the data was used to train a bank of FIR filters from which the depth data volume was extracted. Various filter configurations were tested. Their best model produced R2 = 0.932 (R = 0.965) and a tidal volume accuracy error of 9.4%. In another study, Soleimani et al.  obtained volume and flow signals by processing Kinect V2 depth data to produce parameters typically associated with a pulmonary function testing. They evaluated their system on 40 COPD patients. Using a spirometer as a reference, they demonstrated that the forced vital capacity and slow vital capacity test both produced correlation coefficients of 0.999, (although they included a rescaling based upon the spirometry measurements to achieve this). Harte et al.  used four Kinect V1 cameras arranged in a cross configuration pointing towards the subject located in the centre. Their study included thirteen healthy subjects and nine cystic fibrosis patients. Based upon a one-way ANOVA test (F[1,51] = 7.5783; p = 0.0082), they concluded there was a significant difference in the sample means of the two groups. Transue et al.  evaluated a Kinect V2 based system against a spirometer for tidal volume accuracy. Their system performed a general volume estimation of the chest and then required per-subject training against a spirometer as reference to offer more accurate individual results. The dataset consisted of multiple trials of 20 s duration with participants in a standing position. They obtained accuracies ranging from 92.2 to 94.2% (corresponding to absolute errors in tidal volume from 0.055 to 0.079 l) when assessing four healthy subjects against a spirometer.
A number of research groups have focused specifically on determining a respiratory rate using depth sensing camera equipment. Bernacchia et al.  assessed ten healthy young adult subjects and found good agreement between the breath periods derived from a Kinect depth sensing system and a spirometer reference. They achieved a 9.7% RMSD for the breath periods between the two devices. During the tests, which lasted only 40 s per acquisition, the subjects were asked to maintain ‘regular respiratory activity’. In a study of young children (between 1 and 5 years) Al-Naji et al.  found excellent agreement between depth-sensing RR and a piezo-belt reference with correlation coefficients ranging from 0.97 to 0.99 depending on whether bed sheets were used and the background lighting levels. The study was, however, limited to five healthy volunteers in relatively benign conditions. Rezaei et al.  studied the respiratory rate of restrained rodents when subjected to fear-inducing predatory odours. They found they could measure respiratory rate with an accuracy of 94.8% using a reference RR from visual observation. Martinez and Stiefelhagen  utilised a depth camera to extract respiratory rate data from 94 sleep analysis sessions from 67 patients in a sleep clinic. They found their system to be 85.9% accurate when compared to a reference thermistor placed at the nose. However, their depth system achieved similar results to that of a contact chest band sensor. Monitoring the respiratory rates of three preterm infants was the focus of a study by Cenci et al.  where each infant was assessed in five 30 s intervals. They found an overall correlation coefficient of R = 0.95 between their system output and the respiratory rate derived from ECG impedance pneumography.
Much of the early work involving depth sensing cameras focused on the determination of tidal volume. More recent studies which have considered respiratory rate are, in general, limited to relatively benign conditions, short periods of time, limited numbers of subjects/patients and/or poor reference measures. The work reported here extends current research in this area by studying a cohort of healthy volunteers exhibiting a wide range of respiratory rates resulting from being subjected to a rigorous, protocolized hypoxic challenge.
2.1 Clinical study
The data was collected opportunistically during a non-related hypoxia (‘breathe-down’) study to evaluate a pulse-oximeter sensor. This parallel study protocol includes a desaturation event comprising a series of step-changes in oxygen saturation. Approval was given for the use of depth camera data acquisition and no other alteration to the existing protocol was made.
Participant demographic information
2.2 Data acquisition and processing
The depth data was captured using a Kinect V2 camera (Microsoft Corporation, Redmond, WA, USA) connected to a laptop and at a frame rate of 30 fps. The camera was mounted on a tripod and placed in front of each subject. The data was collected over several days and the distance between the camera and subjects varied between 1.2 and 2.0 m over the collection period and positioned vertically at approximately chest height. The subjects were seated in a slightly reclined position. The room was illuminated with standard ceiling mounted fluorescent lights. Other than starting and stopping the recording process no other intervention or calibration was required over the study period.
The next steps in the algorithm (outlined in Fig. 2) extract a robust value of respiratory rate (RR) from the volume signal. The respiratory volume signal is first filtered by a low pass filter (Butterworth, 5th order, cut-off 0.67 Hz). The peaks of this signal are then identified and the respiratory periods (RPs) calculated as the time difference between successive peaks to produce a “per breath” RP signal. This RP signal is low-pass filtered (Butterworth, 5th order, cut-off 0.67 Hz) to smooth the periods. The RR signal is then calculated by multiplying the reciprocal of the RP signal by 60. The final step removes the effect of outliers in the RR signal by averaging over a 60 s sliding window only those points that are within the 25th and 75th percentiles of the values. (We have found that these outliers may arise if non-prominent peaks are not successfully eliminated during the initial stages of the algorithm and this outlier removal step successfully deals with these.) This processing produces the output RRdepth signal, an example of which is shown in Fig. 3c.
The capnograph provides a reference respiratory rate on a per-second basis. This output reporting time step duration is relatively typical for medical monitoring devices for screen updating. We therefore resampled the output of the depth sensing RR to match this. The two respiratory rate signals, RRdepth and RRcap, required synchronization prior to statistical analysis as the depth camera and capnograph signals were collected independently on separate acquisition systems. This was carried out using cross-correlation of the two signals.
2.3 Data analysis
The latter expression is a root mean square deviation (RMSD) and represents a combination of the systematic and random components of the differences between the corresponding readings from the two devices.
Least-squares linear regression was performed to obtain the line of best fit between the video and reference parameters from which the gradient, intercept, Pearson correlation coefficient, R, and associated p values were computed. In this work p < 0.05 was considered statistically significant. A Bland–Altman analysis of the data was also performed using the method of Bland and Altman  which compensates for within-subject longitudinal correlation in the data. SD of the bias and corresponding limits of agreement were calculated using this methodology.
Matlab (R2018b) was used to process the data and perform the statistical analysis. An in-house developed C++ application was used to capture the depth data.
Uptime was calculated to be 100% for every case.
A relatively small data set (N = 14) was used in developing the algorithm with no data withheld for separate blind testing. Over-training is a therefore possibility, and the results must be viewed in this context. However, the respiratory signal was manifestly obvious across all patients and a relatively simple algorithm was sufficient to identify individual breaths and produce a respiratory rate. In addition, off-the-shelf depth sensing technology was employed and the equipment is simple to set up with minimal instruction provided to the clinical research staff for its use. (They were asked to set up the camera at chest height within 1 to 2 m of the subject, ensure that the subject was in the frame and turn the acquisition on; with no other intervention or calibration required.)
Motion was relatively restricted as the subjects were seated, attached to a face mask and also had pulse oximeter probes attached to each hand. The participants therefore remained relatively immobile during the acquisition. In clinical practice, however, an algorithm would have to cope with more significant patient motion, including change of posture or position in bed, hand and limb movements, (including hand movements within the line of sight) and interactions with clinical staff. Recently, our group proposed a novel motion protocol for RGB video monitoring including yaw, pitch and roll maneuvers of the head . Although this is at a very early stage and specific to idealized maneuvers for the subject’s head it could perhaps be extended to whole body maneuvers for testing depth-based respiratory technology. However, it is in general very difficult to synthesize the wide variety of complex activities observed in clinical practice and exposing the algorithm to large amounts of patient data acquired from across the spectrum of patient care is ultimately the best way to develop a robust technology.
It is an open question whether depth sensing monitoring could replace other modalities for monitoring respiratory rate in the clinical environment. The technology may be adopted more rapidly in its current state for specific use cases including post-anesthesia respiratory depression indications, neonatal monitoring (where there is a need to avoid excessive contact with the neonatal skin), sleep monitoring and home monitoring of, for example, elderly, post-surgical and/or respiratory patients. The technology also has the advantage of having additional physiological and patient contextual information available from the same modality. These include tidal volume trending, apnea detection, patient activity monitoring, fall detection and bed posture monitoring. A final positive attribute of the technology which should be commented on is its ease of use. Ultimately, the technology should require nothing more than to be aimed at the subject and turned on and it works through patient clothing and bed sheets and with the lights turned off (both of which RGB methods cannot do).
The results strongly indicate the potential for a robust respiratory rate monitoring technology based on depth sensing camera equipment. Future work should attempt to fully test this technology in a more rigorous fashion through a range of confounders typically exhibited in the clinical setting including motion and interference.
Compliance with ethical standards
Conflict of interest
All authors are employees of Medtronic which sponsored the research.
- 3.Michard F, Gan TJ, Bellomo R. Protecting ward patients—the case for continuous monitoring. ICU—Manage Pract. 2019;9(1):10–4.Google Scholar
- 4.McDuff DJ, Estepp JR, Piasecki AM, Blackford EB. A survey of remote optical photoplethysmographic imaging methods. In: 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC); 2015. pp. 6398–404.Google Scholar
- 6.Li MH, Azadeh Y, Babak T. A non-contact vision-based system for respiratory rate estimation. In: 36th annual international conference of the IEEE engineering in medicine and biology society; 2014. pp. 2119–22.Google Scholar
- 7.Scalise L, Ercoli I, Marchionni P, Tomasini EP. Measurement of respiration rate in preterm infants by laser Doppler vibrometry. In: International symposium on medical measurements and applications; 2011. pp. 657–61.Google Scholar
- 8.Bu N, Ueno N, Fukuda O. Monitoring of respiration and heartbeat during sleep using a flexible piezoelectric film sensor and empirical mode decomposition. In: 29th annual international conference of the IEEE engineering in medicine and biology society; 2017. pp. 1362–66.Google Scholar
- 9.Droitcour AD, Seto TB, Park BK, Yamada S, Vergara A, El Hourani C, et al. Non-contact respiratory rate measurement validation for hospitalized patients. In: Annual international conference of the IEEE engineering in medicine and biology society; 2009. pp. 4812–15.Google Scholar
- 12.Yu MC, Liou JL, Kuo SW, Lee MS, Hung YP. Noncontact respiratory measurement of volume change using depth camera. In: Annual international conference of the IEEE engineering in medicine and biology society; 2012. pp. 2371–74.Google Scholar
- 13.Aoki H, Miyazaki M, Nakamura H, Furukawa R, Sagawa R, Kawasaki H. Non-contact respiration measurement using structured light 3-d sensor. In: 2012 Proceedings of SICE annual conference (SICE); 2012. pp. 614–8.Google Scholar
- 14.Seppänen TM, Kananen J, Noponen K, Alho OP, Seppänen T. Accurate measurement of respiratory airflow waveforms using depth data. In: 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC); 2015. pp. 7857–60.Google Scholar
- 15.Soleimani V, Mirmehdi M, Damen D, Hannuna S, Camplani M, Viner J, Dodd J. Remote pulmonary function testing using a depth sensor. In: IEEE biomedical circuits and systems conference (BioCAS); 2015. pp. 1–4.Google Scholar
- 17.Transue S, Nguyen P, Vu T, Choi MH. Real-time tidal volume estimation using iso-surface reconstruction. In; IEEE first international conference on connected health: applications, systems and engineering technologies (CHASE); 2016. pp. 209–18.Google Scholar
- 18.Bernacchia N, Scalise L, Casacanditella L, Ercoli I, Marchionni P, Tomasini EP. Non contact measurement of heart and respiration rates based on Kinect™. In: IEEE international symposium on medical measurements and applications (MeMeA); 2014. pp. 1–5.Google Scholar
- 20.Rezaei B, Lowe J, Yee JR, Porges S, Ostadabbas S. Non-contact automatic respiration monitoring in restrained rodents. In: 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC); 2016. pp. 4946–50.Google Scholar
- 21.Martinez M, Stiefelhagen R. Breathing rate monitoring during sleep from a depth camera under real-life conditions. In: IEEE winter conference on applications of computer vision (WACV); 2017. pp. 1168–76.Google Scholar
- 22.Cenci A, Liciotti D, Frontoni E, Mancini A, Zingaretti P. Non-contact monitoring of preterm infants using rgb-d camera. In: ASME international design engineering technical conferences and computers and information in engineering conference. American Society of Mechanical Engineers; 2015; pp. V009T07A003-V009T07A003.Google Scholar
- 23.Bland J, Altman D. Agreement between methods of measurement with multiple observations per individual. J Pharm Stat. 2007;17:571–82.Google Scholar
- 24.Addison PS, Foo DMH, Jacquel D. Running wavelet archetype aids the determination of heart rate from the video photoplethysmogram during motion. In: 39th annual international conference of the IEEE engineering in medicine and biology society; 2017. pp. 734–7.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.