Keywords

1 Introduction

There are many surgical navigation systems [1, 2]. In almost all cases, we used 3D mechanical or 2D non-mechanical probes with ultrasonic sensors. Unfortunately, since the image resolution of an ultrasonic sensor is not accurate, we cannot detect the position, orientation, and shape of a real liver precisely for manipulating it in surgical navigation systems. In addition, the calculation of the position, orientation, and shape of the liver is immensely time consuming by many approaches based on combination calculation between enormous cloud points [3, 4].

To address this, we manufactured a liver surgery navigator having the goal of decreasing surgical procedural risk for the liver. We then evaluated its precision using a mechanical system that moves the liver replica translationally and rotationally with a high degree of precision. We previously demonstrated that depth image-matching is quantitatively more useful than matching based on comparing point groups (i.e., the point cloud library) for enabling a virtual liver (i.e., liver polyhedron standard triangulated language (STL) form) to follow an actual liver (i.e., liver replica) at high speed [5]. We also investigated two statements about this virtual world: that depth image-matching is useful [6]; and that sufficient following precision can be obtained by evaluating a liver-following algorithm based on the method of steepest descent under natural light or fluorescent lamps in a laboratory [7]. Next, we compared following algorithms based on the method of steepest descent [8] and simulated annealing [9] under shadow-less lamps in an actual operating room, evaluating the latter to be more precise. This revealed that even better following precision is obtained when shadow-less lamps are covered with shade filters [10].

For this study, we manufacture a mechanical system that moves the replica of an actual liver translationally and rotationally with a high degree of precision. We run a simulated annealing algorithm that uses, as a search evaluation indicator, the difference between an actual depth image obtained from the Kinect depth sensor and a virtual depth image based on the z-buffer obtained from OpenGL. When an actual liver is accurately moved translationally and rotationally using a mechanical system that move the replica likewise, we evaluate the precision using the actual amount of translational and rotational movement, rather than previous concordance rates.

Section 2 of this paper describes experimental elements, such as the virtual liver, the actual liver, and the shade filter. Section 3 describes the liver surgery navigator. Then, Sect. 4 describes the results of experiments evaluating the liver navigator. We conclude with a summary of our study in Sect. 5.

2 Experimental Elements

This chapter describes the experimental elements used in this study (i.e., the virtual liver, the actual liver, and the shade filter).

2.1 Liver Polyhedron STL Form (i.e., Virtual Liver)

STL is a system that manifests three-dimensional (3D) shapes, including models from small triangular aggregates, having become the standard format in the rapid prototyping industry. In this study, STL created from a hospital patient’s magnetic resonance imagery tomography images are used as a liver polyhedron STL form. Figure 1(a) shows the STL human liver model.

Fig. 1.
figure 1

(a) Liver polyhedron STL Form, (b) Skin gel liver model.

Full size image

2.2 Skin Gel Liver Replica (i.e., Actual Liver)

The STL skin liver model shown in Fig. 1(b) was created using a 3D printer, and skin gel was poured into the concave form of the generated model to produce the skin-gel liver model. The liver model produced using skin gel is comparable to an actual liver during surgery and is used as the liver replica.

2.3 Shade Filter SL999

SL999 is a heat shielding and insulating film sold by Nextfil. This film can shield and insulate infrared light, UV light, and heat, without blocking visible light rays. By affixing it to the target object, we can block 99.9% of infrared light over 1,000 nm and 98% of UV light. Table 1 shows the performance of SL999, whereas Fig. 2(a) shows a graph of its spectrum transmittance.

Table 1. Performance chart of the shade filter SL999.
Full size table
Fig. 2.
figure 2

(a) Spectrum transmittance of SL999, (b) Spectrum transmittance of TS6080.

Full size image

2.4 Shade Filter TS6080

TS6080 is a heat shielding and insulating film sold by Cyber Reps. This film can shield and insulate infrared light, UV light, and heat without blocking visible light rays. By affixing it to the target object, we can block 98% of UV light. Table 2 shows the performance of TS6080, whereas Fig. 2(b) shows a graph of its spectrum transmittance.

Table 2. Performance chart of the shade filter TS6080.
Full size table

3 Liver Surgery Navigator

This chapter first describes each type of graphical window of the liver surgery navigator. Then, it explains how to find the concordance rate among them.

3.1 Windows of the Liver Surgery Navigator

This section describes our liver navigator, which semi-automatically performs initial alignment of the liver replica and the liver polyhedron STL form. While looking at an image of the livers, the operator first superimposes the liver replica on the liver polyhedron STL form over as wide an area as possible. Next, the liver navigator automatically moves the liver polyhedron STL form and minimizes the difference between it and the depth image. Finally, when the concordance rate is large enough (i.e., 90% or more), the liver replica is moved and interlocking precision evaluation begins.

Figure 3 illustrates the various windows that control our liver navigator. The top-right window displays the liver replica and a depth image of the liver polyhedron STL form. It also shows the concordance rate of the liver replica and the liver polyhedron STL form. In the window located to the right-of-center, on the bottom, concordance rate evaluation and the range of initial alignment can be set. The colored image window on the bottom-right is a real-time image taken by Kinect v2; it allows researchers to monitor the movements of the liver replica. Finally in the window located to the left-of-bottom, we can see positions and angles of 5 degrees-of-freedom concerning to translational and rotational movements.

Fig. 3.
figure 3

Windows of the liver surgery navigator that uses simulated annealing.

Full size image

3.2 Precision Evaluation in Past Following Algorithms (i.e., Concordance Rate)

This study uses the difference in movement distance between the liver replica and the liver polyhedron STL form to evaluate liver navigator precision. Evaluation is conducted by measuring the degree of distance from the axis position of the liver polyhedron STL form in the liver navigator and the liver replica in the translational/rotational movement generator. We then calculate the difference between the movement distance estimated from the initial position of the liver replica and the movement distance of the liver polyhedron STL form. The initial position of the concordance rate is set to 90% or greater. The concordance rate is a value derived from the difference between the depth value and the z-buffer. Figure 4 shows an example of how to find it. First, the depth value obtained using the Kinect to photograph the liver replica and the z-buffer value of the virtual liver STL polyhedron are compared pixel-by-pixel. When a1 = 80 mm, a2 = 92 mm, a3 = 85 mm, a4 = 79 mm, a5 = 75 mm, b1 = 73, b2 = 93, b3 = 82, b4 = 68, and b5 = 62, the difference between configuration a and configuration b is 7, 1, 3, 11, and 13. If the difference derived is less than 10 mm of the threshold set in the liver navigator, it is concordant and is expressed as a percentage. Here, the concordance rate is 60% (i.e., 3/5).

Fig. 4.
figure 4

Concordance rate of actual liver and virtual liver

Full size image

4 Experiments to Evaluate the Liver Surgery Navigator

In this study, experiments are performed in a laboratory and an operating room. In the laboratory, experiments are conducted using an operating bed and shadow-less lamps. To move the liver translationally and rotationally, a translational/rotational movement generator is set atop the operating bed. A 25-cm square acrylic plate 2 cm thick is placed on top; the liver replica is placed on top of that. A robotic arm capable of up and down movement is mounted on the Kinect and set so that the distance between the Kinect and the liver replica can be changed as needed. The Kinect v2 is set up horizontally, in relation to the operating bed, at a position 90 cm above the liver replica atop the translational/rotational movement generator. The Kinect v2 is fixed to the operating bed by placing a metal rod in a clamp installed on the operating bed and fixing it to the rod.

Our experimental methodology follows. First, the axes of the translational/rotational movement-measuring device are adjusted to the initial movement position, the acrylic plate is set up, and the liver replica is placed atop the plate. Next, the initial positions of the liver replica and the liver polyhedron STL form are aligned with the liver navigator that uses simulated annealing, and the concordance rate is set to 90% or greater. Then, measurement of the various axes (i.e., X, Y, yaw) begins. As for movement, both X and Y axes are moved 1 cm at-a-time until they reach 7 cm. Although 360° of movement is possible for the yaw axis, owing to the range of movement of an actual liver, we rotate the yaw axis 5° at a time until it reaches 45° (Fig. 5). Figure 6(a) shows what the experiment looks like in the laboratory, whereas Fig. 6(b) shows what the experiment looks like in the operating room.

Fig. 5.
figure 5

(a) Translational experiments, (b) Rotational axis of the liver model.

Full size image
Fig. 6.
figure 6

(a) Experiment in the laboratory, (b) Experiment in the operating room.

Full size image

4.1 Experiment with Exposure to Two Shadow-Less Lamps

This experiment was conducted in a space lit with two shadow-less lamps and operation room lighting. Kinect v2 took images, starting from the initial alignment of the liver replica, using it and the translational/rotational movement generator, until the conclusion of translational and rotational movement. The experiment was conducted using this imaging data. Figure 7(a) shows the lighting from the two shadow-less lamps.

Fig. 7.
figure 7

(a) Lighting from two shadow-less lamps, (b) Shadow-less lamp covered with TS6080.

Full size image

4.2 Experiment with Two Shadow-Less Lamps Covered with TS6080 and SL999

This experiment was conducted in a space lit by two shadow-less lamps and operation room lighting using TS6080 and SL999. As with the experiment sans-lighting from shadow-less lamps, to compare the movement distance of the liver replica to the movement distance of the liver polyhedron STL form, images of the experiment were taken with the Kinect v2 and the captured data was used for evaluation. We attached the TS6080 and SL999 with tape so that they covered the shadow-less lamps; the separator side faced the shadow-less lamps. The TS6080 was affixed to both shadow-less lamps. Figure 7(b) shows what a shadow-less lamp looks like when covered with TS6080.

4.3 Translational Movement Experiment on the X-Axis

Here, we evaluated how accurately the liver polyhedron STL form followed the liver replica when it was moved translationally along the X axis in several environments: in a laboratory under natural light or fluorescent lamps, in an operating room under two shadow-less lamps, and in an operating room under two shadow-less lamps covered with two types of shade filters.

Experimental Results in the Laboratory

We moved the liver replica translationally and confirmed that the liver polyhedron STL form was linked to that movement. As shown in Fig. 8(a), the concordance rate during movement consistently exceeded 90%. As for changes in the amount of movement from the liver polyhedron STL form, the error value increased as movement distance increased, as shown in Fig. 8(b). The largest error observed, in comparison to the amount of liver replica movement, was 2.9 mm. This result is slightly better than the experimental results from the operating room, as described in the next section.

Fig. 8.
figure 8

We describe translational experimental results along the x-axis in the laboratory. (a) Changes in the concordance rate manifested by the liver polyhedron STL when the liver replica is moved (Vertical axis: concordance rate (%); horizontal axis: movement distance (mm)), (b) Movement distance errors (mm) and error rate (mm).

Full size image

Experimental Results Using SL999

We moved the liver replica translationally and confirmed that the liver polyhedron STL form was linked to that movement. The concordance rate during movement was about the same as the rate without shadow-less lamps and, as shown in Fig. 9(a), it consistently exceeded 90%. As seen in Fig. 9(b), results regarding changes in the amount of movement from the liver polyhedron STL form differed from the results for the TS6080 shade filter. Thus, with the SL999 shade filter, there was less fluctuation in errors for each movement distance, and there were fewer errors overall. The largest error, compared to the amount of liver replica movement, was −3.6 mm, similar to those obtained in the TS6080 shade filter experiment. Additionally, the concordance rate was also slightly lower, meaning that it also functioned as an evaluation indicator of movement error.

Fig. 9.
figure 9

We represent translational experimental results along the x-axis in a surgical room with two lamps enclosed of SL999. (a) Changes in the concordance rate manifested by the liver polyhedron STL when the liver replica is moved (Vertical axis: concordance rate (%); horizontal axis: movement distance (mm)), (b) Movement distance errors (mm) and error rate (mm).

Full size image

4.4 Translational Movement Experiment on the Y-Axis

Here, we evaluated how accurately the liver polyhedron STL form followed the liver replica when it was moved translationally along the Y axis in several environments: in a laboratory under natural light or fluorescent lamps, in an operating room under two shadow-less lamps, and in an operating room under two shadow-less lamps covered with two types of shade filters.

Experimental Results in the Laboratory

We moved the liver replica translationally and confirmed that the liver polyhedron STL form was linked to that movement. As shown in Fig. 10(a), the concordance rate during movement consistently exceeded 90%, as measured on the X axis. As can be seen in Fig. 10(b), overall errors were observed in results regarding changes in the amount of movement from the liver polyhedron STL form. Furthermore, large errors were observed from 4 cm-on, with a maximum error of −10.3 mm.

Fig. 10.
figure 10

We describe translational experimental results along the y-axis in the laboratory. (a) Changes in the concordance rate manifested by the liver polyhedron STL when the liver replica is moved (Vertical axis: concordance rate (%); horizontal axis: movement distance (mm)), (b) Movement distance errors (mm) and error rate (mm).

Full size image

Experimental Results Using SL999

We moved the liver replica translationally and confirmed that the liver polyhedron STL form was linked to that movement. As shown in Fig. 11(a), the concordance rate during movement consistently exceeded 90%, just like the rate without shadow-less lamps. When compared to the TS6080 shade filter, the SL999 results were superior, with errors sustained below half, as shown in Fig. 11(b). There were almost no large errors. The largest error, compared to the amount of liver replica movement, was only −5.3 mm. This was better than any other experimental result to date. However, the concordance rate was not good, demonstrating that it is not necessarily appropriate as an error evaluation indicator for translational movement.

Fig. 11.
figure 11

We represent translational experimental results along the y-axis in a surgical room with two lamps enclosed of SL999. (a) Changes in the concordance rate manifested by the liver polyhedron STL when the liver replica is moved (Vertical axis: concordance rate (%); horizontal axis: movement distance (mm)), (b) Movement distance errors (mm) and error rate (mm).

Full size image

4.5 Experiments on Rotational Movement Around the Yaw Axis

Here, we evaluated how accurately the liver polyhedron STL form followed the liver replica when the liver replica was moved rotationally around the yaw axis in several environments: a laboratory under natural light or fluorescent lamps, an operating room under two shadow-less lamps, in an operating room under two shadow-less lamps are covered with two types of shade filters.

Experimental Results in the Laboratory

We moved the liver replica rotationally and confirmed that the liver polyhedron STL form was linked to that movement. As shown in Fig. 12(a), the concordance rate during movement consistently exceeded 90%, just as measured on the X axis. As can be seen in Fig. 12(b), changes in the amount of rotational movement from the liver polyhedron STL form generally followed predictions calculated from initial position. There were few errors that excluded initial movement, and we confirmed that it was more consistent that translational movement. The largest error, compared to the amount of liver replica movement, was 2.3°.

Fig. 12.
figure 12

We describe rotational experimental results along the yaw-axis in the laboratory. (a) Changes in the concordance rate manifested by the liver polyhedron STL when the liver replica is moved (Vertical axis: concordance rate (%), horizontal axis: (°)), (b) Movement distance errors (°) and error rate (°).

Full size image

Experimental Results Using Two Shadow-Less Lamps

We moved the liver replica rotationally and confirmed that the liver polyhedron STL form was linked to that movement. As demonstrated by Fig. 13(a), the concordance rate during movement was better than in previous results and nearly always exceeded 96%.

Fig. 13.
figure 13

We represent rotational experimental results along the yaw-axis in a surgical room with two lamps. (a) Changes in the concordance rate manifested by the liver polyhedron STL when the liver replica is moved (Vertical axis: concordance rate (%), horizontal axis: (°)), (b) Movement distance errors (°) and error rate (°).

Full size image

Meanwhile, as shown in Fig. 13(b), errors were gradually added to the initial value as the amount of rotational movement from the liver polyhedron STL form changed. Although there were few errors for each movement, errors accumulated as the angle of movement increased, and large errors ultimately remained. No noticeable differences were observed when using just one shadow-less lamp. The largest error, compared to the amount of liver replica movement, was −6.7°.

To summarize these results, whereas the concordance rate was better than in experimental results from the laboratory and there were no large errors for rotational movement, the errors grew larger. Therefore, we learned that concordance rate is not an indicator for error evaluation in rotational movement.

5 Conclusion

Using a mechanical system that precisely generates translational/rotational movement, this study evaluated the precision of a liver-following algorithm based on simulated annealing. The results empirically demonstrated the following three things.

  1. (1)

    The experimental results in the operating room under shadow-less lamps were often superior to the experimental results in the laboratory under natural light. Additionally, using filters to block the infrared spectrum from the shadow-less lamps produced better results. We are currently selecting a shade filter that can reliably remove, from the shadow-less lamps, spectra interfering with the infrared pattern projected by the Kinect v2.

  2. (2)

    Concordance rate, which has been used to evaluate liver following algorithms in the past, often could not be used (i.e. was not correlated) for evaluating error in translational movement or rotational movement, globally (i.e., overall comparison of each experiment) or locally (i.e., comparison of multiple times in each experiment).

  3. (3)

    When the algorithm was evaluated in the past using concordance rate, rotational movement was evaluated as more precise than translational movement in nearly all cases. However, the results of this study showed the opposite. The movement precision of rotation, especially the degree of roll and pitch, was poor. This revealed that, when depth images are compared, small angle differences between the liver replica and liver polyhedron STL form are not being successfully evaluated.

Moving forward, we will first investigate shade filters that are better fitted to cameras that measure depth images. Next, unlike the lever for translational movement, the lever for rotational movement is in an extremely small area. This causes the operator to move the liver replica by accident. Additionally, because the graduation of rotational movement is done in far smaller increments than that of translational movement, the proper amount of rotational movement is not obtained, particularly for the angles of roll and pitch. This urgently needs improvement.