Realistic Benchmarks for Point Cloud Data Management Systems

Abstract

Lidar, photogrammetry, and various other survey technologies enable the collection of massive point clouds. Faced with hundreds of billions or trillions of points the traditional solutions for handling point clouds usually under-perform even for classical loading and retrieving operations. To obtain insight in the features affecting performance the authors carried out single-user tests with different storage models on various systems, including Oracle Spatial and Graph, PostgreSQL-PostGIS, MonetDB and LAStools (during the second half of 2014). In the summer of 2015, the tests are further extended with the latest developments of the systems, including the new version of Point Data Abstraction Library (PDAL) with efficient compression. Web services based on point cloud data are becoming popular and they have requirements that most of the available point cloud data management systems can not fulfil. This means that specific custom-made solutions are constructed. We identify the requirements of these web services and propose a realistic benchmark extension, including multi-user and level-of-detail queries. This helps in defining the future lines of work for more generic point cloud data management systems, supporting such increasingly demanded web services.

Appendix

1.1 Appendix A: Executable Benchmark Data Sets and Queries

Table 6 Data sets name, benchmarks in which they are used, number of files and disk size

Table 7 Data sets area and description

Fig. 7

Approximated projection of the extents of the used datasets in Google Maps: Purple area is for 20M dataset, cyan area is for 210M dataset, green area is for 2201M dataset and red area is for 23090M dataset

Fig. 8

Queries used in the medium benchmark (up to 210M extent)

Fig. 9

Queries used in the medium benchmark

1.1.1 Data sets

Tables 6 and 7 contains information on the used data sets in the executable benchmark and their usage in the different stages. Figure 7 shows the extent of the used data sets.

Note that for the full AHN2 we include two versions of the data set. The first one, 639478M was used in our previous full-benchmark execution while the second one, 638860Mc is the one used for in the new execution and does not contain erroneous and duplicate points that are found in the first version. Also note the difference in the data sets sizes. This is due to the fact that for the cleaning process that was required to generate the second (cleaned) version of the full AHN2 data set the points in the files needed to be resorted and that affected dramatically the compression performance of LAZ. In the first version the files were separated according to their nature, in object and terrain files, and that improved the compressor performance. However, as part of the cleaning processes, these files were joined and the compression ratio was affected. The compression ratio improves when the data is resorted by LAStools as part of the benchmark execution (see Table 1) but even in that case it is not optimal due to the mixing of point cloud from different nature (total size 1,66 Tb).

1.1.2 Queries

Figures 8 and 9 show the first 20 query geometries that were used in the several benchmark stages. Table 8 describes all of them, their ID, the number of points in the boundary of the query geometry (Pnts) and the test data set name in which the query geometry is located.

1.2 Appendix B: Executable Benchmark Loading Results

Table 9 contains the loading details of the medium-benchmark execution for various PCDMS’s and data sets. The results of LAStools are when using LAS (instead of LAZ). The PCDMS using the blocks model were using the compression available at that time (second half 2014) and with optimal block sizes previously computed. Note that all the Oracle Exadata approaches (oe* on the table) run in a different hardware than the other approaches.

Table 8 Description of the different queries

Table 9 Times and sizes of the data loading procedure for the different PCDMSs and datasets. The names of approaches encode the PCDMS name (o for Oracle, p for PostgreSQL, etc.), flat or blocked model (f and b, respectively), and the dataset name. For example ob2201M stands for the dataset 2201M loaded in the Oracle blocks PCDMS

Table 10 contains the loading details of the full-benchmark execution that was done with LAStools and Oracle Exadata PCDMS’s. Note that for this execution the 6394784M data set was used, i.e. the AHN2 version with duplicate and erroneous points. For an in-deep analysis of these results we refer the reader to our previous work (van Oosterom et al. 2015).

1.3 Appendix C: Executable Benchmark Querying Results

Table 11 contains the number of returned points and the response times of the first seven queries for the different PCDMS’s and data sets. Note that each query was executed twice, the numbers in the table are from the second execution, usually called hot query because of the fact that the PCDMS may be able to reuse cached data either by the PCDMS itself or the file system or the operative system (OS).

Table 12 contains the number of returned points and the response times of the execution of the 30 full-benchmark queries for the LAStools and Oracle Exadata PCDMS. Note that for LAStools two columns are given. The first one is when using a DBMS in a pre-filtering step for the queries and the other is without it. For an in-deep analysis of these results we refer the reader to our previous work (van Oosterom et al. 2015).

Table 10 Full-benchmark loading results for the LAStools and Oracle Exadata PCDMSs

Table 11 Comparison of number of points returned and response times by the hot queries 1 to 7 in the different approaches

Table 12 Full benchmark query results of LAStools and Oracle Exadata. Notes (a.) Nearest neighbours queries (#18, #19 and #20) were not executed as functionality was not implemented, and (b.) Oracle Exadata query #25 was also re-run using an MBR instead of a geometry close to an MBR with and improved the time to 353.93 s with 3.6546E+10 selected points