Human segmentation by geometrically fusing visible-light and thermal imageries

Abstract

From depth sensors to thermal cameras, the increased availability of camera sensors beyond the visible spectrum has created many exciting applications. Most of these applications require combining information from these hyperspectral cameras with a regular RGB camera. Information fusion from multiple heterogeneous cameras can be a very complex problem. They can be fused at different levels from pixel to voxel or even semantic objects, with large variations in accuracy, communication, and computation costs. In this paper, we propose a system for robust segmentation of human figures in video sequences by fusing visible-light and thermal imageries. Our system focuses on the geometric transformation between visual blobs corresponding to human figures observed at both cameras. This approach provides the most reliable fusion at the expense of high computation and communication costs. To reduce the computational complexity of the geometric fusion, an efficient calibration procedure is first applied to rectify the two camera views without the complex procedure of estimating the intrinsic parameters of the cameras. To geometrically register different blobs at the pixel level, a blob-to-blob homography in the rectified domain is then computed in real-time by estimating the disparity for each blob-pair. Precise segmentation is finally achieved using a two-tier tracking algorithm and a unified background model. Our experimental results show that our proposed system provides significant improvements over existing schemes under various conditions.

Appendix

1.1 A Proof of Theorem 31

Since the homography matrix H′ is up to scale, we can assume it is in the form of

$$H'=\left[ \begin{array}{ccc} a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & 1 \end{array} \right] $$

According to the definition of image rectification, epipoles of the two image is at infinity and in the form of [1 0 0]^T and [a 0 0]^T also subject to the homography. Plug them in (3) we have

$$\begin{array}{lll} a_{11}& =& a \\ a_{21}& = &0 \\ a_{31}& =& 0 \end{array}$$

since

$$ y_2'=\frac{a_{22}y_1'+a_{23}}{a_{32}y_1'+1}=y_1' $$

the following equation will always hold,

$$ a_{32}y_1'^2-(a_{22}-1)y_1'-a_{23}=0 $$

Therefore, all the coefficients for different order have to be zero. We have a ₃₂ = 0, a ₂₂ = 1, a ₂₃ = 0.