Keywords

1 Motivation

With the advent of medical device interoperability and of online workflow recognition, methods for configurable remote control [1, 2] and context-awareness [3, 4] have been discussed, and ways to automate supportive tasks and human-machine interaction [5] have been demonstrated recently. Operating rooms have begun to emerge from a set of monolithic medical devices, through an ensemble of interoperable connected devices, to a distributed system with consistent behavior. The research efforts aim to assist the surgical personnel and to keep the ever increasing complexity manageable. Instead of technology, the patient should be the focus. However, automation usually bears severe risks that limit the scope. Hence, we propose a method for the context-aware assignment of medical device functions to input devices in integrated operating rooms. By means of that, surgeon’s direct control can be increased with a limited interaction complexity.

2 Materials and Methods

The designed approach relies on medical device interoperability and the availability of intraoperative technical context-awareness. Thus, the basic principles of both prerequisites and their existing implementations are briefly summarized.

2.1 Medical Device Interoperability and Context-Awareness

An openly integrated technical environment is an essential prerequisite for a comprehensive workflow recognition based on data from various sources as well as for the implementation of remote control. The emerging IEEE 11073 SDC standards family for medical device interoperability introduces the service-oriented architecture paradigm to operating rooms [6] and will frame the access to data and control across vendors. In this context, input devices, such as buttons and pedals [7], act as service providers analogously to medical devices offering control functionalities, such as parameter settings and operations. Following the concept discussed in [1], we developed an orchestration component that enables a dynamic configuration of remote control.

The context-aware online assignment of controls requires the intraoperative provision of information on the operational context of the medical devices. The implementation is based on an existing context-awareness pipeline that gathers data for recognition, performs a mapping to low-level tasks, processes a network of process models and additional components to provide a comprehensive situation description (Surgical Process Context), and shares these data via SDC network [5]. The contextual information also include predictions of upcoming work steps based on the method described in [8]. The proposed dynamic assignment is based on the situation description, especially the ongoing low-level work step and the potential upcoming tasks.

2.2 Modeling of User Interaction Needs

We propose a novel modeling approach for the realization of dynamic functionality assignment. The human-machine interaction is described as a set of interaction use cases, for instance setting the shaver’s parameters and using it to remove soft tissue. The use case can be further decomposed into a set of atomic interactions, such as the step-by-step decrease or increase of the revolution limit and the continuous motor control. The interactions are described by a basic type, a target, such as the device setting, and a categorical input type to distinguish between triggers and continuous control.

The interactions are associated to the surgical workflow by mapping them to low-level tasks. To that end, the modeling approach includes pre-configuration and initiating interactions (pre-step interactions) as well as interactions during an ongoing work step (intra-step interactions). For example, the revolution limit of the shaver might be configured prior the actual work step of tissue removal. Hence, the decrease and increase interactions are required for pre-configuration. The motor control via foot pedal initiates the tissue removal work step and is thus also required prior to the work step (pre-step need). In this example, all three interactions are as well required during the actual removal work step (intra-step need).

In the formal modeling, the probability for each pre-step interaction as well as the probability of each intra-step interaction are provided. In addition, the probabilities are modeled user-dependent to represent each member of the surgical team individually. If recordings of surgeries are available, the probabilities may be determined empirically. The aspects of human-machine interaction that are essential for a dynamic assignment of controls during surgery may then be described in terms of user’s interactions associated to low-level tasks. The resulting interaction use cases are user-specific and depend on the type of surgery.

2.3 Modeling of Interaction Profiles

Besides the users’ needs modeled so far, a representation of available input devices, such as switches at handles or foot pedals, is necessary for a dynamic assignment. The required risk management for remote control poses considerable challenges for the modeling. For example, an input device needs to respect the risk class of the controlled medical device. Otherwise, an assignment shall be prohibited. Furthermore, the assignments should be consistent in each situation to achieve a sufficient user acceptance. For example, decreasing and increasing a device parameter should be assigned to a pair of co-located buttons.

To ensure consistent and safe configurations, a pre-definition of interaction profiles is proposed. When designing such a profile, functions are assigned to input devices allowing for a comprehensive risk management and the preservation of consistency. Whilst for each input device, the modeling complexity is reduced to the interaction assignment and an access probability for each user. By means of an access probabilities, input devices may be mainly or exclusively associated to certain users, for instance the buttons on the endoscope camera head will be used by the surgeon exclusively. The interaction profiles add another layer of specificity to the interaction use cases, as they depend on the available input devices and their accessibility by the team members.

2.4 Intraoperative Profile Selection

The dynamic assignment of functions to input devices is realized by a context-aware automated selection of the most appropriate interaction profile. The OR system performs the selection at the beginning of each low-level task. The set of interactions that the optimal profile would have to cover consists of the intra-step interactions of the actual work step and the pre-step interactions of the upcoming next task. Thus, the suitability of an interaction profile depends on the actual work step as well as on the upcoming task, which is yet not known. An online profile selection method must rely on predictions, especially a probability distribution over potentially forthcoming tasks.

We designed a score for an online assessment of the suitability of each interaction profile. The scoring uses the ongoing workflow, the prediction of forthcoming tasks, the modeled user interaction needs, and the interaction assignments of the profiles. The recently begun work step and the prediction are considered to be given as they are part of the Surgical Process Context [5]. For a given work step \(s_t\) and the potential upcoming tasks \(s_{t+1}\), a score \(\omega _u \) can be calculated for a profile \(\pi \) with pre-step interactions \(\bar{i}\) and interaction during the work step i as follows for each user.

$$\begin{aligned} \omega _u ( \pi ) = \frac{ \sum _{i \in \pi }{p(i|s_t)} }{ \sum _{i}{p(i|s_t)} } \end{aligned}$$
(1)

The required conditional probabilities \(p(i|s_t)\) are given by the modeling of the user needs. Equation 1 represents the coverage of the user’s current needs. By design, there should be at least one interaction profile with full coverage.

$$\begin{aligned} \bar{\omega }_u ( \pi ) = \sum _{s_{t+1}}{ \left( p(s_{t+1}|s_t) \frac{ \sum _{\bar{i} \in \pi }{ \left( p_d (\bar{i}) \cdot p(\bar{i}|s_{t+1}) \right) } }{ \sum _{i}{p(i|s_{t+1})} } \right) } \end{aligned}$$
(2)

Equation 2 represents the coverage of needs for potential upcoming tasks, which are weight by the corresponding transition probability and the accessibility of the input device \(p_d(\bar{i})\) that the interaction is assigned to. The score of a profile \(\omega ( \pi )\) may then be calculated as a weighted sum over all user-dependent scores \(\omega _u\) and \(\bar{\omega }_u\) respectively. Both, \(\omega _u\) and \(\bar{\omega }_u\) are defined to be zero if no interaction is required for the given task. The user’s priority is expressed as \( \beta _u \). The parameter \(\alpha \in [0;1]\) allows to balance between the interactions needed in the ongoing work step and those required to initiate potential upcoming tasks.

$$\begin{aligned} \omega ( \pi ) = \sum _{u}{ \beta _u \cdot ( \alpha \cdot \omega _u + (1 - \alpha ) \bar{\omega }_u )} \end{aligned}$$
(3)

Finally, the interaction profile with the highest score is selected. To resolve inconclusive cases, an appropriate linear ordering is defined and the candidate profile with the smallest distance to the previously active profile is selected.

2.5 Technical Integration and Risk Management

The proposed scoring method needs to be integrated into the service-oriented operating room infrastructure. To that end, an orchestration component acts as a service consumer to the input devices, the medical devices, and the context provider. Based on the current interaction profile, the component must react to communicated user interactions by remote controlling the corresponding medical device. Therefore, the orchestration component becomes part of the command processing chain and must respect the risk class of the controlled device, just as the input device has to. Whenever a new low-level work step has been recognized, the most suitable interaction profile is automatically selected. However, such a dynamic changing of control assignments bears severe risks. Essentially, the user needs to be informed about the current assignment of the available input devices. This may be realized with a continuous display, as for instance proposed in [7]. To avoid an additional display, we propose a two-level risk mitigation based on a preliminary risk analysis. Whenever the interaction profile is changed, a temporary overlay is shown in the field of view of the surgeon, which displays the input device assignments. And in case of uncertainties, a static interaction can be used to show the overlay on demand. Figure 1 depicts to examples of temporary overlays provided by the orchestration component to communicate the recently changed assignments to the user.

Fig. 1.
figure 1

Overlays of a new interaction assignment of the spin-click wheel on the Visus JiveX viewer during preparation (left) and of a new assignment of the buttons at the endoscope camera head during sinus surgery (right).

Full size image

Furthermore, the orchestration component needs to mitigate the risk of unintended actions due to profile changes simultaneously to user interactions. To that end, the profile is not changed while an interaction is ongoing, for instance a foot pedal is pressed down. Additionally, an input device is blocked for several seconds after its assignment has been changed automatically, and a visual feedback is given in the video overlay. The feasibility of the approach may depend on the risk management for a concrete assignment in a concrete clinical use case. For assorted functions, for instance coagulation, the blocking time might not be feasible and functions like these must be excluded from the dynamic assignment. However, we expect that a broad set of functions will be manageable.

3 Experiments

We have conducted initial experiments for the Functional Endoscopic Sinus Surgery (FESS) - a common surgery in ENT - in a demonstration setup. The ventilation and the drainage of the paranasal sinuses are restored under endoscopic vision with powered shavers, suction, and other instruments. Optionally, a surgical navigation system is used. The demonstration setup included all necessary devices and a viewer for medical records. For the initial technical validation presented here, recordings of twenty-four previously simulated workflows were used [5]. The workflows covered the essential surgical activities with respect to the technical limitations. The procedures consisted of thirty-five distinguishable activities including preparation tasks, tissue removal, cavity traversal, occasional endoscope cleanings, medical record access, and navigation usage.

In the present experiments, we focused on the surgeon’s interactions. The technical setup included three relevant input devices: a spin-click wheel with three interactions, a foot switch with three pedals, and two buttons at the endoscope camera head. However, the spin-click wheel is not accessible to the surgeon while using the endoscope and the endoscope camera buttons are not accessible otherwise. Effectively, six input devices are available during the initial patient preparation and the final after care, and five input device are available during the endoscopic phase. The modeling of the human-machine interaction included twenty atomic interactions, among them shaver settings and motor control, displaying the navigation or the medical record viewer on the secondary display, or scrolling through documents and pages of the medical record. The useful combinations of these interactions were represented in fourteen interaction profiles. Especially the interactions for forthcoming tasks in various combinations increased the amount of required interaction profiles.

A leave-one-out cross scenario was used to evaluate the performance of the proposed scoring approach for the online selection of an appropriate interaction profile. Although the simulated workflows were based on former recordings of real interventions, the probabilities for interactions could not be determined empirically. Hence, for the technical validation we assume every interaction defined in the interaction use cases is always required, especially \(p(i|s_t) \in \lbrace 0,1 \rbrace \). Furthermore, we weighted the coverage of the current needs and the forthcoming interactions equally (\(\alpha = 0.5\), see Eq. 3). We analyzed the rate of availability of the required pre-step and intra-step interactions.

4 Results

In the cross validation, 1245 work steps were analyzed, of which 1162 can include human-machine interaction, such as capture an endoscopic image for documentation purposes or configure the parameters of a medical device. Table 1 lists the availability of the interactions in the experiments with the proposed scoring approach. As already discussed, the interaction profiles are designed so that at least one profile covers all required intra-step interactions for each interaction use case. The results for intra-step interactions show that the method always selects a profile with full coverage of the intra-step needs.

Table 1. The availability of interactions to pre-configure and initiate (pre-step) and of interactions required during the work steps (intra-step).
Full size table

Due to the limited number of available input devices (five during endoscopy, six otherwise), the method has to rely on the predictions of forthcoming steps. None of the interaction profiles is capable of providing all potentially required pre-step interactions. In 57 work steps, at least one of the required interactions was not directly available for the surgeon. Overall, 2655 of the 2724 required interactions were available (97.5%). Most of the misses occurred for the increase of the shaver revolution limit during suction of nasal cavities and for the switching of the secondary screen. In the suction-related profiles, there is no input device left to also provide the revolution limit increase, which is the least critical pre-step interaction. The switching of the secondary screen is assigned to a single button in most profiles; hence, the system needs to predict whether the navigation or the medical record will be needed next, which tends to be challenging resulting in 30 misses. However, predictions significantly contribute to the availability rates by enforcing profiles with probable pre-step interactions.

5 Conclusion

The presented method for online selection of interaction profiles aims to simplify human-machine interaction in increasingly complex surgical working environments. The results of the preliminary technical validation indicate that the proposed approach has the potential to increase the surgeon’s direct control while preserving operability with a reasonable amount of input devices. However, a careful design of the interaction profiles with respect to the clinical use case and the technical setting is still essential. The effectiveness of empirically determined probabilities in the model and multi-user scenarios need to be evaluated, both technically and pre-clinically on phantoms.

The integration of the method into a interoperable medical device ensemble realizes an additional aspect of context-aware assistance in the operating room. The context-aware assignment of functions to input devices will ease the complexity for the staff in cases where potential risks make automation impossible.