Integrated Frameworks for Three-Dimensional Scene Reconstruction

Abstract

It has been shown in the previous chapters that the problem of three-dimensional scene reconstruction can be addressed with a variety of approaches. Triangulation-based approaches rely on correspondences of points or higher-order features between several images of a scene acquired either with a moving camera or with several cameras from different viewpoints. These methods are accurate and do not require a-priori knowledge about the scene or the cameras used. On the contrary, as long as the scene points are suitably distributed they do not only yield the scene structure but also the intrinsic and extrinsic camera parameters, i.e. they perform a camera calibration simultaneously with the scene reconstruction. Triangulation-based approaches, however, are restricted to parts of the scene with a sufficient amount of texture to decide which part of a certain image belongs to which part of another image. Occlusions may occur, such that corresponding points or features are hidden in some images, the appearance of the objects may change from image to image due to perspective distortions, and in the presence of objects with non-Lambertian surface properties the observed pixel grey values may vary strongly from image to image, such that establishing correspondences between images becomes inaccurate or impossible at all. Intensity-based approaches to three-dimensional scene reconstruction exploit the observed reflectance by determining the surface normal for each image pixel. They are especially suited for textureless parts of the scene, but if several images of the scene are available, it is also possible to separate texture from shading effects. Drawbacks are that the reflectance properties of the regarded surfaces commonly need to be known, the reconstructed scene structure may be ambiguous especially with respect to its large-scale properties, and small systematic errors of the estimated surface gradients may cumulate into large depth errors on large scales. PSF-based approaches directly estimate the depth of scene points based on several images acquired at different focus settings. Depth from defocus can be easily applied and no a-priori knowledge about the scene needs to be available, but a sufficient amount of surface texture is required. Due to the fact that estimation of the PSF is sensitive with respect to pixel noise, the resulting depth values tend to be rather inaccurate. Depth from focus is very accurate but also time-consuming due to the large number of images required. These considerations illustrate that each of the described approaches has its specific advantages and drawbacks. Some of the techniques are complementary; as an example, triangulation-based methods yield three-dimensional point clouds describing textured parts of the scene while intensity-based methods may be able to reconstruct textureless regions between the points. Hence, just like the human visual system achieves a dense three-dimensional scene reconstruction based on combinations of different cues, it appears to be favourable for computer vision systems to integrate different three-dimensional scene reconstruction methods into a unifying framework. This chapter describes several approaches of this kind and discusses their specific preconditions, advantages, limitations, and preferential application domains.