Single-View 3D Reconstruction by Learning 3D Game Scenes

Abstract

We report a method for reconstructing a three-dimensional (3D) depth map using a single two-dimensional (2D) image. Our method is designed to reconstruct manmade objects, such as buildings. We first estimate the normal map, and then integrate it to obtain the depth map. To estimate the normal map, we analyze the co-occurrence relation between the normal vectors and image features in a training dataset. We consider the corners and lines to be the image features. The training dataset is formed of 3D game scenes. In the offline learning phase, we detect corners and lines in the normal map of each game scene using a detection algorithm, and observe the normal vectors around them. Then we construct a database of the co-occurrence relations, i.e., how frequently each corner or line appears with each normal vector. In the online reconstruction phase, given an input image, we detect the corners and lines using the same detection algorithm, and estimate the normal vectors around them based on the learned co-occurrence relation. We formulate this estimation using a Markov random field. Finally, the estimated normal map is integrated by solving Poisson’s equation, and we obtain a depth map.