Keywords

1 Introduction

Teleoperation is always a challenging task due to the lack of sense of immediacy and insufficient information for operation. Robotic arm operation in space used for transporting astronauts during the extra-vehicle activities or docking spaceships is one of such tasks. Safe and efficient control of the robotic arm is heavily dependent on the spatial skills of the operator [1], so it is important to make it clear the key cognitive factors in the space teleopertation task, especially for improving the astronauts’ training efficiency.

According to NASA’s related report [2], during the robotic arm operation task in space, astronauts usually have to make decisions only relying on the visual feedback from the cameras mounted on the robotic arm and at various locations on the space station exterior to learn about the spatial relationship of the arm with the surrounding structure. However, there are usually only three camera viewpoints available at any moment [3], and the cameras are not always placed at the optimal locations for astronauts to observe the clearance from the structure. Addition to these, astronauts also have to memorize the location of these cameras and switch between them, which undoubtedly increases the operators’ mental workload during the operation. To avoid the danger of colliding with structure or singularities during the operation, the operators need to confirm they handle the hand controllers correctly before they give that command, and the procedures may also require a second operator to provide additional monitoring of the scene. At the same time, the movements of the robotic arm are made very slow and after the operators have established situation awareness of the spatial relationship [4] to guarantee safety.

Under this circumstance, the preflight robotic arm training on the ground becomes very important for the operation success in the orbit. Astronauts must experience sufficient training first before they execute a real task so that they can be skilled enough to operate successfully. The robotic arm training for the astronauts in NASA began with a generic robot arm simulation, which having a 6 degrees-of-freedom robot arm and different camera views available as well as two hand controllers [5]. Trainees needed to learn how to visualize the clearance and choose the ideal cameras during the operation process. They also needed to learn how to make the right control commands via the hand controllers to avoid collisions or singularities. During the training process, the operator’s performance was usually evaluated by a robotics instructor and an instructor astronaut according to standard criteria covering all aspects of operations [6]. The criteria could include spatial/visual perception, situational awareness and appropriate input of the controllers. However, the performance scores given by instructors could be subjective sometimes, so we also need some objective criteria, especially relating to the operation process and operation efficiency.

Usually, the training process took much time. In order to improve the training efficiency and make the training course more customized, the key cognitive factors are needed to be identified first. Previous studies showed that individual spatial ability might be related with the operation performance [7]. In this study, we mainly focused on analyzing the key cognitive factors influencing the space teleoperation task for training, which was important to cultivate and maintain the operation skills of operator. To achieve this, we needed to observe the operation training procedure. But on the ground it was not easy to reproduce the working environment physically, so we built up a simulated space robotic arm operation platform by means of computer simulation, which could provide various operating conditions similar to a real space robotic arm aboard the space station and the operator could use it via the hand controllers. With the operating experience in this simulated environment, we constructed a hierarchical model that described the space robotic arm operation task. And then the key cognitive factors that might affect the task training process were proposed according to the hierarchical model. Lastly an experiment in this simulated operation platform was conducted to testify the effect of these cognitive factors.

2 The Hierarchical Model for the Robotic Arm Operation Task

In order to analyse the key cognitive factors influencing the space teleoperation task training, we should firstly learn about the procedure in the robotic arm operation. From the review of the in-orbit robotic arm operation activities, the kinds of operation tasks could be various [8, 9]. For example, the robotic arm could be used for satellite deployment and retrieval, docking with the space station and transporting astronauts during extra-vehicle-activities and so on. Although the contents of the operation might be varied in different tasks, they also had some potential characteristic in common. During each operation task, the operator needed to observe the status (e.g. position and attitude) of the robotic arm firstly, so that the operator could be aware of the spatial relationship between the arm and the exterior of the space station. This was the fundamental step of the whole subsequent operation steps, and we named this step of the operation as “Evaluate the present situation”; and then, immediately after this step the operator would be able to establish the awareness of the present situation, which made him/her possible to foresee the integrated route for the movement of the robotic arm and plan for the operation commands for next few steps, so we classified this step of the operation as “Plan for the following operation commands”; and finally, the operator needed to transform the operation plan into real control commands via hand controllers, and confirmed whether the outcome of the control command met his/her expectations, and we defined this final step as the “Execute the commands and confirm the realtime outcome”. The three steps described above could be regarded as a small operation unit of the whole robotic arm operation. During the whole operation procedure, this operation unit would be repeated periodically until the operation ended in success (Fig. 1).

Fig. 1.
figure 1

The illustration of the hierarchical model that described the process of the space teleoperation task investigated in this study.

Full size image

3 The Cognitive Factor Analysis in Space Teleoperation Task Training

From the hierarchical model described above, we could extract the key factors that influenced the operation process.

In the first step, namely the step “Evaluate the present situation”, the operator would observe the status of the robotic arm and its spatial relationship with the exterior of the surrounding structure via different cameras mounted at different locations. These cameras could be located at the exterior of the space station as well as at the end of the robotic arm, and the operator could view the images from several different cameras at the same time. Due to the number of the operation monitors in the robotic arm workstation, usually there were only three or four images available at any moment, so this would require a camera selection process when evaluating the present situation [10]. The operator must select the most suitable three or four camera images from the whole cameras for observing the position and attitude of the robotic arm and determining the clearance distances.

The cameras could be divided into two categories. The first kind were the cameras located at the exterior of the space station or the base of the robotic arm. The images from this kind of the camera were presented from a fixed view and would not change as the robotic arm moved. These cameras could provide stable view images for the operators, which could give operators an exocentric view frame of the robotic arm movement [11]. The second kind of the cameras were the ones located at the end of the robotic arm. The images from this kind of the camera could provide a view attached to the end of the arm, from which the operators would obtain a sense of egocentric frame understanding of the environment. These two different categories of the cameras could result in two different perception mode. The first perception mode was related to the first kind of cameras. In this mode, the operator would perceive the status of the robotic arm in an exocentric way, which might help the operator form an overview of the situation. The second perception mode was related to the second kind of cameras, and in this mode, the operator perceived the situation of the robotic arm in an egocentric way, which could provide the operator a more specific view about the situation of the surroundings around the arm. As these two different perception modes might influence the way operators observe the robotic arm status, so the perception mode could be one of the factors influencing the teleoperation task.

In the second step, the operator needed to plan for the following operation commands. During this stage, the operator should form a plan for how to give the appropriate following operation commands based on the evaluation results from the first step. For example, if the robotic arm was evaluated to be pitched up too much in the previous step, then the operator needed to know how to adjust its attitude via proper operation commands in this step. In order to achieve this, the operator needed to take advantage of the mental imagery to make a rehearsal of the robotic arm’s trajectory as it was hard to predict the movement of the arm accurately and in time through other methods. As a result, the operator should be able to image what the position or attitude of the robotic arm would be after certain commands as well as the perspective of the arm from other views that was not presented in the current monitors. This mental imagery transformation process was similar with the two common-used individual spatial ability factors, namely spatial visualization and perspective-taking abilities [12]. These two kinds of spatial abilities might be the two related cognitive factors. Spatial visualization ability was also usually presented by mental rotation ability. It was referred to the ability to mentally manipulate an array of objects. The manipulation was in a fixed egocentric reference frame. Perspective-taking ability described the ability to imagine how an object or scene looked from perspectives different to the observer’s current view. It demanded a transformation process in the egocentric reference frame while the world coordinate frame was fixed. These two abilities could be regarded as logically equivalent, the only critical difference was in the coordinate frame which was manipulated to obtain the final view. Previous studies showed that although the performance was also highly correlated, a measurable distinction between spatial visualization and perspective-taking ability was found [13]. So we needed to consider these two cognitive factors in separated ways.

In the third step, the operator’s task was to execute the commands and confirm the realtime outcome. When executing the commands during this procedure, there existed two different possible types of control modes. The main difference between these two modes was the coordinate system used. This could effect the methods of adjusting the position and attitude of the arm. In the first mode, the origin of the control coordinate system was at the end of the robotic arm and the norm’s direction changed accordingly while the attitude of the arm varied, we named this mode as the “end-control-mode”. Under this circumstance, the control direction of the position and attitude was coupled. For example, in this control mode, the upward direction would changed for 90 degrees after pitching the same extent. So it would require the operator to transform the mental representation of the control direction constantly with the movement of the arm, which might increase the mental workload, but on the other side, could give the direct commands relating to the arm’s current status, and it might be helpful for improving the situation awareness of the operator. In the other control mode, the origin of the control coordinate system was located at some point at the exterior of the space station, e.g. it could be at the base of the robotic arm and so it would be stable during the whole process, and we named this mode as the “global-control-mode”. And also the norm of the coordinate system would keep stable. This could provide a constant reference frame for the robotic arm’s operating and none of the control directions was changeable no matter how the movement rotated. In this situation, the control direction of the position and attitude was decoupled. This could be helpful when the operator received the images from the exocentric cameras, but might also confuse the operator when the images was presented in an egocentric camera. Although in the second step the strategy used for operating was formed, the control mode was also needed to be taken into consideration to obtain the final commands to be executed. So we considered that the control mode could influence the space teleoperation task operation.

From the analysis above, we concluded that there might exist three different kinds of factors that influenced the space teleoperation task, namely perception mode, individual spatial ability and control mode. These three steps composed an operation unit during a complete teleoperation task and this unit was repeated periodically in the task. The task training process was also consisted of large amount of these repeated unit, so we could regard it that these factors drawn from the hierarchical model would likewise influence the space teleoperation task training process.

It was important and useful to identify these factors in theory as this was the first step for the continued study. And to go further, we conducted an experiment to testify the influence of the individual spatial ability factor. In this experiment, we built up a simulated space teleoperation platform that could be controlled via two hand controllers.

4 The Experiment Validation

4.1 Materials and Methods

Participants.

Twenty-four adults (mean age = 26.3, SD = 2.75, ranging from 23 to 31) with college-level education participated the experiment. None of these participants had conducted the space teleoperation task in simulated or physics environment before the experiment. The study was approved by the IRB and all participants signed the informed consent prior to the experiment.

Measurement of Spatial Ability.

Because two factors of the individual spatial ability, mental rotation and perspective-taking, were concerned by us mostly in this experiment, we measured both in their 2D and 3D versions. The 3D Mental Rotation Ability (MRA) was measured by Cube Comparison Test (CCT) on computer, and the 2D MRA was also measured by Card Rotation Test (CRT) through computer. The 3D Perspective-taking Ability (PTA) was measured by the paradigm developed by Guay [14] on computer, and the 2D PTA was also measured by the paradigm developed by Kozhevnikov and Hegarty [15] using specially developed software. The specific parameters used in the spatial ability tests were shown in Table 1.

Table 1. Parameters setting in the four spatial ability tests
Full size table

Simulated Space Teleoperation Task Platform.

In this study, we developed a space teleoperation task platform via computer simulation. The participants could use the hand controllers to manipulate the simulated robotic arm in the in-orbit background. The difficulty of the tasks could be set differently with various parameter settings through the platform. These main parameters included the position of the target, the initial status of the robotic arm and the camera selection at the beginning and the two control modes as described above and so on. The user interface of the simulated platform was shown in Fig. 2. To evaluate the performance of the participants, the time consumed during the whole operation task and whether each task was finished successfully was recorded.

Fig. 2.
figure 2

Theillustration of the overview [16] (the upper left) and the user interface of the simulated robotic arm platform. This illustration showed the camera views that available by the participant during one trial. This was the final stage of one trial and the red cross/star marked the target point that needed to approach to. (Color figure online)

Full size image

Experiment Procedure.

The whole experiment was conducted in two periods. In the first period, we measured the spatial ability of all the participants and calculated the participants’ scores in the spatial ability tests; and then in the second period, after identifying all the participants’ spatial ability scores were valid, the participants were arranged to take the simulated robotic arm operation experiment.

To be specific, during the first period, participants’ spatial ability in 2D & 3D MRA and PTA were tested. Their scores in each test were obtained using the performance indicators as shown in Table 1. Then in the second period, there were twelve formal trials and all the participants needed to operate the simulated robotic arm twelve times after six practices. During the practice stage, the participant could ask the experimenter questions about how to use the hand controllers, but no help about the manipulation strategy were provided.

As in this experiment we mainly focused on the influence of individual spatial abilities on the task, we provided the same parameters about the initial status of the robotic arm as well as the camera selection at the beginning and the control mode for all the participants. To be specific, in all the twelve tasks, we provided the participants four camera views, two of which provided the exocentric perception mode and the rest two provided the egocentric perception mode. And the control mode was set to be the “global-control-mode”. The task in each trial was to transform the simulated astronaut on the end of the effector to the target point. The main differences among the twelve formal tasks were the locations of the targets. The participants needed to transform the simulated robotic arm from the beginning position towards the target in each task. And in order to have enough complexity for each task, the locations were set to be accessible only after a combination of operations including pitch, yaw as well as roll. In order to avoid the learning effect, the sequence of the tasks were randomized for each participants.

4.2 Results

Statistical analysis was conducted using SPSS. We first observed the participants’ performance in spatial ability tests. As both the latency and percent correct indicator could reflect the participants’ performance, we use the ratio of latency and percent correct as the performance indicator to take both indicators into consideration at the same time, and marked as spatial ability synthetical indicator. The higher value of this indicator meant the better performance of the participant. Then we investigated the participants’ performance in simulated robotic arm operation. As we provided enough practices before formal trial and did not set the time limitation to finish each trial, almost all the trials of every participant’s was ended in success (only 2 of the all 288 trials were failed and were due to the inattentive judgement during the final stage). So we took the average time consumed in the operation task as the indicator that reflected the participants’ performance in the task.

After the indicators was built up, the correlation between participants’ spatial ability and simulated space teleoperation performance was analyzed. The correlation coefficients were shown in Table 2. From these results we could see that, except the result of the 2D mental rotation test, the other three of all the four spatial ability tests were correlated significantly.

Table 2. Pearson correlation coefficients for spatial ability test scores and simulated robotic arm operation task performance
Full size table

4.3 Discussion

From the result described above, we could conclude that the mental rotation and perspective-taking ability of the individual spatial abilities were correlated with the performance in the simulated robotic arm operation. This result was consistent with the analysis we conducted above using the hierarchical model. Although the result of the 2D mental rotation was not correlated with the performance in robotic arm operation significantly, we thought this might be due to the robotic arm operation was mainly in the three-dimensional environment, and especially the transformation of the arm movement’s mental representation seldom happened just in a two-dimensional world. However, the perspective-taking process in 2D and 3D were essentially related with each other closely, and the result also showed that both these two components were correlated with the operation performance.

5 Conclusions

Space teleoperation task is challenging for astronauts. It is important to identify the factors that may influence the operators’ performance. In this study, we proposed a different way to explore the potential cognitive factors that might effect the robotic arm operation. Firstly, a hierarchical model was proposed and the operation process was divided and described by this model. Then according to the analysis of this model, we proposed some factors that might influence the teleoperation task training process. And finally we conducted an experiment via the computer-simulated method to identify one of the factors we proposed in theory, and the result was consistent with our previous analysis generally and the detailed discussion was given.

Except for the individual spatial ability factor, the remaining two other factors were still needed to be tested and verified in future studies. And in this study, we just focused on the space operation without time delay or the object on the end effector of the space robotic arm was cooperative, so the hierarchical model could was simplified at some extent. The method to establish a complete model with the time delay and cooperative object is also needed to be studied in the future.