Keywords

1 Introduction

The liver has several blood vessels such as arteries, veins, portals, etc. and these are not externally visible. By using X-ray imaging, computed tomography (CT) or magnetic resonance imaging (MRI) etc., the position of these vessels in the liver is checked preoperatively. During surgery, surgeons touch the incision part of the liver using their fingers, feel for a pulse-beat, and perceive the proximity of the vessels. However, this method is dangerous and can injury to the blood vessels and result in bleeding.

Surgical support systems for orthopedic, dental, or brain surgery are developed [1,2,3]. These systems determine the position of the surgical tools, synchronize with the tomographic images and navigate the tool to the target position. The target body parts of these support systems are rigid and exhibit negligible deformation during surgery.

However, the liver is soft and changes it shape during operation and the existing support systems cannot be applied to the liver surgery. Research targeted on the surgical support systems for soft organs has been limited, and no effective system have been developed yet. We have been developing a liver surgery support system.

In this paper, the detail of our liver surgery support system and newly developed wireless knife attachment are described. It can alert the proximity of the target parts by lighting LEDs.

2 Our Liver Surgical Support System

The detail of our liver surgical support system was described in [4]. It uses two depth cameras with different characteristics. The first camera is lower precision, but a wide measurement range used for determining the shape of the liver during surgery. Kinect for Windows v2 developed by Microsoft is currently used. Three-dimensional (3D) polyhedron model is generated from tomographic images preoperatively. The 3D model contains the shape of the liver, inner blood vessels, and tumors. By matching the depth images from the depth camera and Z-buffer of the 3D polyhedron model during surgery, the positions of the liver is determined using simulated annealing algorithm and the location of the inner blood vessels and the tumors are estimated [5,6,7,8]. The second camera is higher precision and performs single point measurements using single markers [9] to determine the tip position of the surgical knife. MicronTraker3 (type H3-60) by ClaroNav is currently used. By merging all information, the distance between the knife tip and the vessels or tumors is calculated [10] and the proximity of the knife to the target parts is determined. To indicate the proximity, the LEDs on our surgical knife attachment are gradually illuminated (Fig. 1).

Fig. 1.
figure 1

Overview of our liver surgical support system

Full size image

3 Conventional Surgical Knife Attachment

The conventional surgical knife attachment was presented in [4]. The size and weight of the attachment are approximately 70 × 70 × 70 mm and 261 g respectively. The size of marker for MicronTracker3 is 30 × 40 mm. An LED bar module (OSX10201-GYR1) [11] and a piezoelectric speaker are mounted in the attachment and a microcomputer (Arduino Nano) controls them. The LED module has 10 LEDs (five greens, three yellows and two reds). The attachment has no internal power source. The electric power and control commands are externally supplied from the main PC through USB cable (Fig. 2). Specification details are listed in Table 1.

Fig. 2.
figure 2

Conventional surgical knife attachment

Full size image
Table 1. Specification of conventional surgical knife attachment
Full size table

4 Wireless Surgical Knife Attachment with Proximity Indicators Using ArUco Marker

In the new attachment, the case was made smaller and lighter, and a microcomputer with a Wi-Fi module and a rechargeable battery were built in the case and controlled wirelessly (Fig. 3). We confirmed that this battery drive for more than 6 h. For the knife position measurement system, ArUco [12] was used. ArUco is a marker recognition library for AR which can estimate postures of cameras and markers. ArUco is built in the OpenCV library, it can be easily used with an inexpensive USB camera. ArUco uses black and white square markers (ArUco marker, Fig. 4). Specification details are listed in Table 2. The circuit diagram of the attachment is illustrated in Fig. 5.

Fig. 3.
figure 3

Tool for navigation experiment

Full size image
Fig. 4.
figure 4

Sample of ArUco marker (DICT_4X4, ID0)

Full size image
Table 2. Specification of new surgical knife attachment
Full size table
Fig. 5.
figure 5

Circuit diagram

Full size image

5 Knife Tip Position Calibration

To measure the relative vector from the marker on the attachment to the knife tip, the calibration procedure described in [4] was basically used. Camera and knife coordinate systems are defined as \( \sum_{c} \) and \( \sum_{k} \) respectively. The knife attached marker and the table fixed marker are represented as \( {\text{M}}_{knife} \) and \( {\text{M}}_{table} \) respectively.

To acquire a relative vector from \( {\text{M}}_{knife} \) to the knife tip, the knife tip is placed at the origin point \( p_{table}^{c} \) of \( {\text{M}}_{table} \), and the position and orientation of each marker are measured in \( \sum_{c} \) coordinates (Fig. 6). The position and orientation of \( {\text{M}}_{knife} \) measured in \( \sum_{c} \) coordinates are defined \( P_{knife}^{c} \) and \( R_{knife}^{c} \) respectively. The relative vector \( P_{rel}^{c} \) is calculated by the following equation. \( {\text{P}}_{\text{rel}}^{\text{k}} \)

Fig. 6.
figure 6

Knife tip position calibration

Full size image
$$ P_{rel}^{c} = P_{table}^{c} - P_{knife}^{c} $$

To convert \( P_{rel}^{c} \) in \( \sum_{k} \) coordinates, the following is used.

$$ P_{rel}^{k} = \left( {R_{knife}^{c} } \right)^{ - 1} \cdot P_{rel}^{c} $$

Finally, the knife tip position \( P_{tip}^{c} \) in \( \sum_{c} \) coordinates is estimated as follows.

$$ P_{tip}^{c} = R_{knife}^{c} \cdot P_{rel}^{k} + P_{knife}^{c} $$

Note that the OpenCV returns the orientation of the ArUco marker as 3 × 1 rotation vector with reference to the marker coordinate system. In this rotation vector, the direction of the vector shows the rotation axis and the norm of the vector shows the rotation angle. To convert the rotation vector into a rotation matrix in the camera coordinate system, the Rodrigues’ rotation formula was used.

6 Preliminary Experiments

6.1 Positional Precision in Distance Between Camera and Marker

We verify the difference in position measurement of the ArUco marker depending on the distance between the ArUco marker and the camera. Since it is quite difficult to verify the absolute accuracy of the distance, we will examine the repetitive accuracy.

The experimental procedure is as follows. 30 × 30 mm sized ArUco marker (ID 0 of DICT_4 × 4) was set on the horizontal table. A USB camera (Logicool C615, resolution 1600 × 896 pixels) was attached to a tripod and placed above the marker. By capturing ArUco marker from the camera, the position of the marker was measured. The distance between the marker and the camera was set to about 500, 600, 700 mm (Fig. 7), and the position of the marker was measured 100 times respectively.

Fig. 7.
figure 7

Captured camera images of ArUco marker set on the flat table

Full size image

The measurement result of the marker position is shown in Fig. 8. The positive direction of the x, y and z coordinate axis is the right, the downward and the backward direction with the camera position as the origin respectively. You can see that the measurement distance is almost correct from the graph. The standard deviation of the measurement distance of each axis is shown in Fig. 9. The standard deviation in the z direction is larger than in the x and y directions in all measurement distances. However, all standard deviations are less than 1 mm. From this result, it was found that the ArUco has a high repetitive accuracy of distance and the accuracy is not much different with the distance of the camera under 700 mm.

Fig. 8.
figure 8

Measurement result of ArUco marker position for 100 times in the distance between the marker and camera of 500, 600 and 700 mm approx.

Full size image
Fig. 9.
figure 9

Standard deviation of marker position in x, y and z axis

Full size image

6.2 Positional Precision in Changing Angle of Marker

We verify changes in marker positional precision due to camera and marker angle. As the method in the previous section, the position of the ArUco marker was measured by the camera above the ArUco marker. The marker was tilted to 0, 15, 30, 45, 60 and 75º around x axis (Fig. 10), and 100 times of the marker position were measured for each.

Fig. 10.
figure 10

Captured camera images of the angled ArUco marker

Full size image

The measurement result of the marker position is shown in Fig. 11. You can see that the measurement distance is almost correct from the graph. The standard deviation of the measurement distance of each axis is shown in Fig. 12. As the marker angle increased, the standard deviation of the Z axis tends to decrease. On the other hand, the marker detection rate tends to decrease as the marker angle increased (Fig. 13). Summarize these results, it is better to use ArUco markers at around 60º.

Fig. 11.
figure 11

Captured ArUco maker images

Full size image
Fig. 12.
figure 12

Standard deviation of angled ArUco marker position in x, y and z axis

Full size image
Fig. 13.
figure 13

Marker detection ratio

Full size image

7 Navigation Experiment

We conducted experiments to validate operator navigation using the new attachment. The task is to trace an invisible target circle with the knife tip based on the LED information. The target circle is set on a flat horizontal table in front of the subject. The diameter of the circle is 80 mm. The attachment cube was fixed to a hard steel rod with 130 mm in length, 6 mm in diameter.

With \( \left( {x_{\text{tip}} , y_{\text{tip}} } \right) \) as the knife tip position, \( \left( {x_{\text{c}} , y_{\text{c}} } \right) \) as the center of the circle, and \( r \) as the radius of the circle, a distance \( {\text{L}} \) between the knife tip and the circle is calculated from the following equation.

$$ {\text{L}} = \left| {r - \sqrt {(x_{c} - x_{tip} )^{2} + \left( {y_{c} - y_{tip} } \right)^{2} } } \right| $$

The LED array contains 10 LEDs. However, the microcomputer of new attachment has only 9 digital I/O. For this reason, the LEDs are gradually illuminated between \( 1 \le {\text{L}} \le 9 \) mm and can navigate over a range in \( \pm 9 \) mm of the circle.

The five subjects (A to E) are undergraduate students from the Osaka Electro-Communication University, not surgeons. The experimental results of the trajectory of the tip position are shown in Fig. 14. The green circle is the target circle and the purple dots show the trajectory. The LEDs turn on within the range between the blue and yellow circles. The maximum and averaged navigation errors for each subject are shown in Fig. 15. The results show that it was possible to navigate with the maximum error of 9.3 mm only with the output of the LED array. This is nearly within the navigable range by the LED array, and the usefulness of the new device was shown.

Fig. 14.
figure 14

Experimental results of trajectory of each subject (Color figure online)

Full size image
Fig. 15.
figure 15

Navigation error of each subject

Full size image

8 Conclusion

We developed a compact, lightweight, inexpensive wireless surgical knife attachment which can display proximity by gradually illuminating LEDs using ArUco markers and a microcomputer with Wi-Fi module. To investigate the characteristics of the positioning of the ArUco marker, we measured the repetitive accuracy of the position of the marker by changing statically in the position and posture of the marker, and clarified the marker has a suitable angle for stable and robust use. We also conducted a navigation experiment to five subjects. The subject held the knife with the attachment in hand and traced the knife tip to the invisible circle by watching the illuminated LEDs on the attachment. The LEDs are gradually illuminated according to the distance between the knife tip and the circle. As a result, it was shown that navigation is possible within the presentable range and the new attachment has comparable navigation accuracy to the previous one.

In the future, we will consider the use of high resolution USB camera, examination of marker size, use of multiple cameras for more precision, robustness and stabilization.