Evaluation of Using Mobile Devices for 3D Reconstruction of Cultural Heritage Artifacts
This work aims to examine the reliability of smartphones that incorporate 3D depth sensors for 3D reconstruction of cultural heritage objects. The main focus is to compare the models generated with two image-based methods: photogrammetry and Tango Constructor application. The result are promising, showings that Tango-based method is an efficient way for 3D reconstruction of historical artifacts and is able to provide morphometric data comparable with photogrammetry-based data. The method can provide restorers a quick way to record vast amount of data, combined with sufficient accuracy and ease-of-use and this make it a potential alternative to conventional methods, like photogrammetry or laser scanning.
Keywords3D reconstruction Mobile devices Cultural heritage
In the Cultural Heritage (CH) 3D reconstructed models provide an important support for procedures like historical reconstruction, analysis of artifacts and architecture, documentation of archaeological sites, preservation, marketing of museum etc. [1, 10]. The 3D models of cultural heritage objects are generated by using dedicated software graphical applications, image based methods of specialized sensors .
Many 3D digital models of cultural heritage objects like buildings, statues and historical places are developed using image based methods, active sensors like laser scanners, total stations or combination of them [2, 11, 13]. Although laser scanners are more accurate than image-based systems, the latter are preferred on the strength of their cost, ease-of-use and time saving. As mentioned in , some of the disadvantage of laser scanning are that many blind spots are generated and a great deal of time is required to obtain the data.
The possibility to reconstruct reliable 3D models by simply using consumer low cost devices is a great opportunity. Smartphones are widely used and offer the possibility to access a variety of applications for accomplishing a large number of tasks. Due to the development of camera features and the usage of multiple cameras, smartphones became a common device used for image acquiring.
In order to perform 3D reconstruction, the user can use a smartphone equipped with a depth sensor  or attach a depth sensor to the smartphone or simply make photos of the target and upload the pictures on a cloud server and then run a web-based application to create the 3D reconstruction.
To obtain a high accuracy for the 3D reconstructed object, a set of the most representative selection frames is uploaded to the cloud where they are processed and the 3D model is obtain . For volumetric 3D models of realistic scenes, like whole buildings, many agents using augmented reality smartphones collaborate through an online pose optimization . In  multiple users with smartphones collaborate to provide the best frames for a 3D reconstruction pipeline that are uploaded on a cloud-based server and processed through a Structure form Motion (SfM) and Dense Image Matching (DIM) procedures.
Another method of reconstructing outdoor scenes of large proportions is by computing depth maps at interactive frame rates on the GPU of a Google Project Tango Tablet 4 through motion stereo . In  is presented a comparison between the usage of Tango tablet and ZED camera in 3D reconstruction. The Tango tablet offers better accuracy of generated point clouds for outdoor environments and a lower value of the average error. The ZED camera per-forms better indoors because it has to be connected to a computer while scanning.
Because the lack of appropriate light source can be a disadvantage for 3D reconstruction using smartphones, in  was presented 3D structured light scanning that represents the combination between 3D reconstruction and the registration on Lenovo Yoga Tab 3 Pro to achieve a 3D point cloud model.
Due to the unfavorable environment conditions or object materials (e.g. poor lighting, shiny surface, homogenous textures), some authors proposed improved approaches for point cloud generation , but it is still room for new improvements of algorithms and used technology to overcome some limitations, like, for instance, transparent surfaces, that cannot be detected by the devices.
It should be noted that the point cloud’s resolution depends on the user needs and the application in CH: the reconstructed 3D models can be used for documentation and analysis, for creating digital archives, for providing digital replicas for exhibitions and so on. Nevertheless, the quality of the resulted model should be as good as possible in all cases.
The aim of this research study is to investigate the accuracy of smartphones that incorporate 3D depth sensors for 3D reconstruction of cultural heritage artifacts compared with photogrammetry based approach. Using these techniques, two 3D measurements have been carried out and the goal was to analyze whether the first method, based on Google’s Tango technology, is suitable to reproduce geometrical volumes of objects in CH preservation sector, using photogrammetry data as reference.
2 Android Applications (Apps) for 3D Reconstruction
Following is a brief overview of the 3D reconstruction Android apps available on the Google Play store with examples of their output. The most significant applications are presented, but it is not an exhaustive overview, there are many other applications that can be used for 3D scanning of objects.
2.1 Tango® Constructor
2.2 Open Constructor for Tango
2.4 Matterport Scenes
2.5 Scandy Pro
3 Materials and Data Collection
With the advent of the 3D data capture image-based methodologies, new scenarios are opening up for the construction of true 3D models of artifacts, statues, buildings or other object belonging to cultural heritage. The input images are acquired in different experimental configuration, with different cameras and the proposed procedure has the purpose to compare 3D models starting from the two input data (Tango and Photogrammetry).
For depth perception, Tango devices use three technologies: structured light, time-of-flight and stereo. The first two require the use of an infrared sensor to estimate the distance to the surrounding objects from the time difference between the emission of the infrared wave and its return to the sensor as a result of the reflection. Stereo technique uses two cameras for taking pictures and calculates the distance between them, like to the human eyes do. Project Tango’s developers provides APIs that return point clouds from the depth sensor in the form of xyz coordinates.
For photogrammetry method, the images were collected using a Samsung Galaxy S7 smartphone, that employs a ISOCELL S5K2L1 sensor (1.4 µm pixel size), with a maximum resolution of 12 effective Megapixels (4032 × 3024). In order to reduce the errors of 3D model, the smartphone camera was placed at close distance from the object (about 1.5 m). The images were captured at different angles with respect to the statue surface. The smartphone camera was rotated horizontally and vertically with 10 degrees, maintaining a 70–80% overlap between consecutive photos. Having this overlapping percentage, the acquisition of 173 images was enough for obtaining a good 3D model of the statue.
The Tango model (Fig. 8b) was obtained using Lenovo Phab2, a large Android smartphone with numerous cameras and infrared sensors that could make use of the Google’s Tango software. It has a 16 MP rear camera, depth sensor, motion tracking camera and a 6.4” QHD display. The phablet not only tracks motion, but also has depth perception thanks to an IR emitter and other sensors, remembering the space around it. Another features are accelerometer, gyroscope, ambient light sensor, compass, a powerful Qualcomm Snapdragon, 652 processor optimized for Tango, and an 8 MP front camera. With this device, you are able even to measure distances in the environment.
The image sets were acquired in different sessions, keeping the same procedural steps in order to obtain comparable results for the two methods. The models are imported first in MeshLab , an advanced 3D mesh processing software system, free and open source, to improve the quality of the meshes.
The 3D point clouds generated from the two different recording techniques were analyzed for variations in their quality, to determine which method is most appropriate for generating 3D models in an easy and fast way.
In a first stage, the .obj files obtained from Tango Constructor and from PhotoScan were imported in CloudCompare and they are converted in .pts files. When comparing two models obtained with different methods, the first problem that arises is the difference of resolution of the two corresponding point clouds. The RGB model acquired with the Samsung camera has about 400 times more points in the dense cloud than the one obtained with Tango camera, 5530620 and 13855 points, respectively. Before doing the comparison, the points on the mesh were sampled at 3000000 points.
After registering the two 3D clouds, a quantitatively and qualitatively evaluation of their difference was performed by calculating the Hausdorff distance and the Cloud to Cloud (C2C) distance between the two models. C2C distance computation calculates the nearest neighbor distance, i.e. for each point of the compared cloud, the software looks for the nearest point in the reference cloud and then computes their distance.
The models are very similar, showing the larger deviations on the edges of the model where no perfect match was achieved. The model produced by photogrammetry is of higher quality than the Tango-based one and this can be easily seen even when looking at them (e.g. visual comparison of the two models in terms of geometrical details and lack of noise). But this thing is due to the surface properties of the statue: there are no big changes in terms of color – the surface is almost white, there are reflections occurring in some areas or shaded surfaces. Moreover, as there are many powerful features, the accuracy would be better.
The one-sided Hausdorff distance was also computed in order to analyze the differences and similarities between the reconstructed 3D models . The geometric difference between the 3D models was achieved by calculating the distance between each point of the models and then measuring the actual dimension in millimeters. The models surfaces at a point are similar if the distance between two points is smaller. To compute the Hausdorff distance was used the feature integrated into MeshLab in the filter Sampling -> Hausdorff Distance.
In order to better visualize the error, the results of the calculated Hausdorff distance were displayed by using red-green-blue colormap. For this representation, the red color is maximum error and the blue color is minimum error. Results of the geometric differences applying Hausdorff distance are presented in Fig. 11c. The maximum error between the 3D models is 0.96% and the average error is 0.16%. Thus, the models generated using Tango technology and photogrammetry are geometrically approximately similar.
However, a critical part of Tango scanning is the influence of lighting conditions and object material. We have chosen an object whose surface is very little reflective and the captures where made in proper conditions, with enough light and with clouds. As is shown in [16, 33], there is an obvious difference in the performance of a Tango device in low light and high light conditions. When there are insufficient lighting conditions, the obtained models have certainly a reduced geometric quality. Moreover, Tango-based device is not able to detect surfaces that are directly illuminated by sunlight or transparent surfaces. Even with these limitations, using this method the restorers have a quick way to record vast amount of data, combined with sufficient accuracy and ease-of-use.
The result shows that it is possible to obtain 3D models that are morphometrically comparable using both image-based techniques. The photogrammetry technique has created clearer details in relief of the artifact and their depth seems much higher than the model captured with depth sensor. However, the conducted analysis shows that the differences between the two models are very small and can be considered as acceptable.
As mentioned above, the transparent, shiny, reflective, but also dark surfaces are problematic for image-based 3D reconstruction. They influence the quality of the resulting model, but there are expectations both on the part of the hardware and the improved algorithms that next devices will be able to overcome the different unfavorable conditions and can provide solutions for the realization of 3D models of artifacts in CH field. In this way, the scientists or the conservators can easily compare different information given by different techniques.
Moreover, even if the native models produce by Tango are not rich in information enough, there are various approaches in the literature that can enrich them with semantic and topological information . The method is very suitable when the time is a critical aspect.
The 3D reconstruction is very important to analyze, document and to preserve Cultural Heritage. The aim of this paper was to investigate the application of an image-based reconstruction methodology based on Google Tango for 3D reconstruction of historical artifacts. We have attempted to evaluate the quality of the data generated by Google’s Tango Constructor Android application compared to photogrammetry-based 3D reconstruction using commercial software for a relatively large artifact with low features of the surface.
The 3D data evaluation comparison indicates that Tango-based method is an efficient way for 3D reconstruction of historical artifacts and is able to provide morphometric data comparable with photogrammetry-based data. The analysis was performed by comparing the point clouds obtained using the two mentioned methods and the variation of them in terms of distances between point is relatively small and thus quite accurate.
The performance of Tango Constructor free standalone application was evaluated for an outdoor artifact that was considered suitable for 3D reconstruction due to the bright color and low reflectivity of the surface. Tango Constructor allows users to capture and view 3D models of objects or of their surroundings, and the models can be shared as mesh files. This is a quick way of creating 3D models, which can be used in other applications or software environments.
The presented method has a series of advantages, like saving time required for data collection, fast generation of the 3D model, cost saving, less maintenance, and ease of use. However, there are certain limitations of this method: the accuracy of the 3D reconstructed model is influenced by object materials and lighting conditions. Another limitation is related to the size of the object. If the object to be scanned is small or very small, it is reconstructed with very few details, distorted or even it cannot be identifiable in the scanned scene.
It is certain fact that mobile devices equipped with sensors and appropriate software for depth sensing can find their application in CH field. Since laser scanners have already proven their usefulness and potential on acquiring high quality 3D data, and photogrammetry or image-based techniques have their advantages, but a combination of them lead to better results .
Concluding, due to the versatility in terms of cost, time and ease of use, Google Tango represent a viable solution for conservators and restorers, enlarging their choice in the range of 3D reconstruction instruments. However, as a preliminary test, this study need to be extended in order to asses if Tango devices are adequate for specific applications and to determine whether this low cost solution has the potential to make the difference in 3D reconstruction.
As future work, we plan to continue the investigation by using different scenarios (e.g. indoor, outdoor), more objects with different shapes and different lightning conditions. We also intend to conduct a study to evaluate the performance of various reconstruction applications that use Google Tango technology and which were presented (some of them) in Sect. 2 of this article.
This paper is supported by European Union’s Horizon 2020 research and innovation programme under grant agreement No. 692103, project eHERITAGE (Expanding the Research and Innovation Capacity in Cultural Heritage Virtual Reality Applications).
- 4.Chung, M., Callin, J.: Point Cloud Framework for Rendering 3D Models Using Google Tango. Thesis, Santa Clara University (2017)Google Scholar
- 5.Diakite, A., Zlatanova, S.: First experiments with the Tango tanglet for indoor mapping. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 3(4), 67–72 (2016)Google Scholar
- 6.Donlic, M., Petkovic, T., Pribanic, T.: On tablet 3D structured light reconstruction and registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462–2471 (2017)Google Scholar
- 7.Erica, N., et al.: 3D reconstruction with a collaborative approach based on smartphones and a cloud-based server. In: 5th International Workshop LowCost 3D-Sensors, Algorithms, Applications, vol. 42, pp. 187–194 (2017)Google Scholar
- 8.Froehlich, M., Azhar, S., Vanture, M.: An investigation of Google Tango® tablet for low cost 3D scanning. In: Proceedings of the International Symposium on Automation and Robotics in Construction, ISARC, vol. 34. Vilnius Gediminas Technical University, Department of Construction Economics & Property (2017)Google Scholar
- 16.Senthilvel, M., Soman, R.K., Varghese, K.: Comparison of handheld devices for 3D reconstruction in construction. In: Proceedings of the International Symposium on Automation and Robotics in Construction, ISARC, vol. 34. Vilnius Gediminas Technical University, Department of Construction Economics & Property (2017)Google Scholar
- 17.RTB-Map. https://play.google.com/store/apps/details?id=com.introlab.rtabmap&hl=en. Accessed 21 Nov 2017
- 18.RTB-Map 3D model example. https://sketchfab.com/models/b782694a0e93428ebbe8df688634f7a3?ref=related. Accessed 21 Nov 2017
- 19.Tango Constructor App. https://play.google.com/store/apps/details?id=com.projecttango.constructor&hl=en. Accessed 21 Nov 2017
- 20.Tango Constructor. https://sketchfab.com/models/cd5fe7388c16419caf4945837a40cbec. Accessed 21 Nov 2017
- 21.Open-Constructor. https://github.com/lvonasek/tango/wiki/Open-Constructor. Accessed 21 Nov 2017
- 22.Open-Constructor App. https://play.google.com/store/apps/details?id=com.lvonasek.openconstructor&hl=en. Accessed 21 Nov 2017
- 23.Open-Constructor example. https://sketchfab.com/models/4e9804f548454c30b06ad31f37a8638c?ref=related. Accessed 21 Nov 2017
- 24.Matterport Scenes. https://play.google.com/store/apps/details?id=com.matterport.capture&hl=en. Accessed 24 Nov 2017
- 25.Matterport Scenes example. https://sketchfab.com/models/a5e3dd2e31194a1e982e75cacb747fb8?ref=related. Accessed 24 Nov 2017
- 26.Scandy Pro. https://play.google.com/store/apps/details?id=co.scandy.scandypro.production&hl=en. Accessed 16 Jan 2018
- 27.Scandy Pro. https://www.scandy.co/blog/four-reasons-for-scandy-pro/. Accessed 16 Jan 2018
- 28.Scandy Pro example. https://sketchfab.com/models/bfd1841b23174ab09e82086bccecdd36. Accessed 16 Jan 2018
- 29.Dryanovski, I.: 3D reconstruction with tango. In: 2016 IEEE Hot Chips 28 Symposium (HCS) (2016)Google Scholar
- 30.Agisoft PhotoScan. http://www.agisoft.com/. Accessed 21 Apr 2018
- 31.MeshLab. http://www.meshlab.net/. Accessed 21 Apr 2018
- 32.CloudCompare. http://www.danielgm.net/cc/. Accessed 8 May 2018
- 33.Gülch, E.: Investigations on Google Tango development kit for personal indoor mapping. In: Proceedings of the 19th AGILE International Conference on Geographic Information Science, Helsinki, 14–17 June 2016Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.