Augmented Reality for Human-Robot Interaction in Industry
Augmented reality for human-robot interaction in industry is the usage of augmented reality technologies for displaying computer-generated digital contents, correctly aligned to real objects, in order to enhance and enrich the communication interface of users, which operates robots in an industrial environment.
The industry domain has taken advantage of augmented reality (AR) since its origin (Sutherland 1968): technicians are often involved in complex assembly, repair, or maintenance procedures, and they need to refer to instruction manuals to complete their tasks (Caudell and Mizell 1992). However, these activities often require a high cognitive load due to a continuous switch of the attention between the physical device and the paper manual (Henderson and Feiner 2007). AR can efficiently overcome this issue providing the same content in a digital form and properly displayed on the physical object involved in the task.
In this entry, AR technologies adopted to visualize digital contents in industry, the different kinds of robots involved in, and the most common tasks which benefit of AR contents are discussed.
Among human sensory inputs, sight, hearing, and touch are currently the senses enhanced by an AR system through digital contents. However, since industrial environments may provide different kinds of limitations depending on the task, such as anti-noise headphones or gloves, the AR content is usually provided visually. Different devices can be adopted to display the AR content depending on the task and the environment. Overall, an AR system is characterized by three blocks: a tracking system, a content generator, and a combiner.
The tracking system has the task to establish an absolute reference system, based on some features related to the environment: this reference system is fundamental to properly display the augmented reality content with respect to the user’s view and the real-world objects. The tracking system is called object dependent if the reference system depends on the position of a real object: this kind of systems adopts computer graphics algorithms to identify the object from frames of the environment provided by a digital camera. Other, less common solutions consist of inertial, mechanical, and magnetic tracking systems or involve the usage of absolute references such as GPS coordinates (Foxlin 2002).
The content generator has the task to compute the graphical content to be displayed depending on the coordinates provided by the tracking system and the frames provided by the digital camera.
The combiner has the task to overlap the assets to the user view. It acts in different ways according to the used AR paradigm.
AR technologies are usually applied to human-robot interaction (HRI) with two categories of industrial robots: robotic arm manipulators and automated guided vehicles (AGVs).
A robotic arm manipulator is defined as an n-degree-of-freedom (nDoF) arm robot. It is composed of links connected by joints that are controlled by using either DC or brushless electric motors. Their positions are sampled by means of encoders, and the joints’ velocities are measured with tachometers. Joints are divided into two different categories: revolute and prismatic. Revolute joints allow rotations along one local axis (usually the z-axis), whereas prismatic joints allow translations along one local axis (usually the z-axis). Robotic arms are also equipped by different custom tools depending on the task to accomplish: these tools are positioned at the end of the kinematic chain and are called end-effectors.
AGVs are vehicles that can move along predefined path and predetermined directions automatically and autonomously, without human interference. They are usually equipped by sensors that allow them to identify and eventually avoid obstacles along their path. Their primary task is to transport equipment around a manufacturing facility. AGVs are equipped with automatic guiding, either electromagnetic or optical, they can follow a predefined path through visual analysis of a familiar environment, or they can use vision system to understand their location in an unknown environment.
Depending on the type of robot considered and its task, various types of AR systems are adopted, and different AR contents are provided to the user. When used with robotic arm manipulators, technicians can benefit from AR systems to visualize one or more of the following features: end-effector path, end-effector direction, object(s) manipulated or involved in the task, workspace involved in the robot task, forces applied by the end-effector, and faults that occur on the industrial manipulator. When used with AGVs, the features visualized through AR usually consist of the path and workspace of the robot. In the following, each feature will be introduced and explained.
Depending on the types of paths and trajectories, different AR systems can be adopted. Trajectories can be divided in 2D paths and 3D paths. The first ones are usually visualized on 2D areas using projectors mounted directly on the industrial manipulator or on appropriate supports placed nearby the robotic arm. They normally consist of one or more connected lines of the same color. The 3D paths are visualized in the real environment by means of wearable or handheld devices. It is possible to interact with both types of path using some specific tracked devices to modify them: this allows users to change the trajectories of the industrial manipulator through the interaction with the augmented reality paths.
Objects that are going to be manipulated by the robotic arms can be highlighted using both 2D and 3D features. The 2D features are commonly represented by means of icons or geometric planar shapes projected directly on the object (Fig. 5). The projectors are mounted directly on the industrial manipulator or on appropriate supports placed near the robotic arm. The 3D features are commonly represented by 3D virtual replicas of the real objects that are superimposed on the object manipulated by the robotic arm (Akan and Çürüklü 2010).
It is possible to identify two different workspaces: the first one is the workspace of a robot arm manipulator, defined as the set that comprehend all the positions it can reach; the second one is the “collaborative workspace,” defined as the working area in which both the human operator and the industrial manipulator work together.
The robot workspace is commonly visualized using handheld or wearable devices. The operating area of the industrial manipulator can be represented using a 3D sphere, centered in the base of the industrial manipulator. The diameter of the sphere can also vary, depending on the movements of the end-effector.
Forces applied by the end-effector can be monitored and visualized using wearable or handheld devices. Forces can be represented using 3D virtual vectors, applied on the tool center (Fig. 4). The force components in X, Y, and Z are displayed as well as the resulting vector. Furthermore, the components are colored with different colors, depending on the intensity of the force (Mateo et al. 2014).
When a fault occurs on an industrial manipulator that is working side by side with a human operator, stress and anxiety may increase in the worker because he is not able to realize in real time which is the cause of the error. Moreover, since it is not possible to realize the cause of the fault immediately, the time and resources required to solve it strongly increase.
There are at least four different categories of faults that can be visualized using AR technologies: faults on the velocity sensor, faults on the actuation system, faults due to overloading problems, and faults caused by collision (De Pace et al. 2018). These typologies of errors can be visualized using both hand-handled and wearable devices. Each fault is represented using a specific 3D asset, superimposed on the error’s location. Faults on the velocity sensors are represented using 3D circular arrows placed near the robotic arm’s joints. These arrows rotate as long as the velocity sensor is acquiring correct data from the motor encoder (Fig. 6, green arrow). When a fault occurs on the velocity sensor, the arrows stop moving, and they change their color to red to highlight the problem (Fig. 6, red arrow). Faults on the actuation system can affect the joint’s motor. If an error occurs on it, a 3D model representing the motor is superimposed on the real motor. Moreover, it starts blinking to emphasize the location of the fault. Faults due to collision can be represented using a 3D sphere center at the base of the industrial manipulator. A collaborative manipulator is able to foresee collisions, and when it detects an unexpected object, it stops its movements to avoid the collision. Human operators may not understand why the manipulator has stopped its movements, misunderstanding its actions. To avoid these misjudgments, when the manipulator foresees the collision, the 3D sphere starts blinking to highlight the intentions of the industrial robotic arm. Finally, errors due to overloading problems suddenly stop the movements of the manipulator. When this type of fault occurs, a 3D anvil, along with a warning signal, is superimposed on the payload.
- Akan, B., Çürüklü, B.: Augmented reality meets industry: interactive robot programming. In: Proceedings of SIGRAD, vol. 52, pp. 55–58. Link¨oping University Electronic Press, Vsters (2010)Google Scholar
- Caudell, T.P., Mizell, D.W.: Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of HICSS, vol. 2, pp. 659–669. IEEE, Kauai (1992)Google Scholar
- Chadalavada, R.T., Andreasson, H., Krug, R., Lilienthal, A.J.: That’s on my mind! robot to human intention communication through onboard projection on shared floor space. In: Proceedings of ECMR, pp. 1–6. IEEE, Lincolnm (2015)Google Scholar
- De Pace, F., Manuri, F., Sanna, A., Zappia, D.: An augmented interface to display industrial robot faults. In: International Conference on Augmented Reality, Virtual Reality and Computer Graphics, vol. 2, pp. 403–421. Springer (2018)Google Scholar
- Foxlin, E.: Motion Tracking Requirements and Technologies. Handbook of Virtual Environment Technology, vol. 8, pp. 163–210. Mahwah, N.J.; London: Lawrence Erlbaum Associates (2002)Google Scholar
- Henderson, S.J., Feiner, S.K.: Augmented reality for maintenance and repair (armar). Technical report, DTIC document (2007)Google Scholar
- Humanrobo.: TOSY Industrial Robot: Arm Robot by Humanrobo. https://commons.wikimedia.org/wiki/File:TI_A620-30.JPG CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0/deed.en (2009)
- Mateo, C., Brunete, A., Gambao, E., Hernando, M.: Hammer: an android based application for end-user industrial robot programming. In: Proceedings of MESA, pp. 1–6. IEEE, Senigallia (2014).Google Scholar
- Matsumaru, T.: Mobile robot with preliminary-announcement and display function of forthcoming motion using projection equipment. In: Proceedings of ROMAN, pp. 443–450. IEEE, Hatfield (2006)Google Scholar
- RoboCup.: BvOF RoboCup2013 – Junior Rescue by RoboCup2013, https://www.flickr.com/photos/robocup2013/9154255582/in/photostream/ CC BY 2.0 https://creativecommons.org/licenses/by/2.0/ (2013)
- Sutherland, I.E.: A head-mounted three dimensional display. In: Proceedings of AFIPS, pp. 757–764. ACM, San Francisco (1968)Google Scholar
- Vogel, C., Poggendorf, M., Walter, C., Elkmann, N.: Towards safe physical human-robot collaboration: a projection-based safety system. In: Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on, pp. 3355–3360. IEEE (2011)Google Scholar