Keywords

1 Introduction

The augmented reality (AR) is a technology to display the overlaying of virtual content on the real environment. The AR applications have been utilized for the designing or assembling the real environment, reducing designing time, enhancing the manufacturing processes, and improving the quality. The user can interact with the augmented reality directly and intuitively. Several research works presented the applications on display devices, such as head-mounted displays (HMDs) [1,2,3], handheld devices [4, 5], and projectors [6]. In general, AR systems required the head-mounted displays (HMDs) that provided three main characteristics such as the visualization of combining the real and virtual object into the real environment, the interaction in real time, and the registration of virtual object in 3D space [7]. The optical see-through HMDs provide the AR overlay through a transparent screen without the delay of video image. In the past, the HMDs had limitations of field of view, low frame rate, size, weight, and low resolution. Especially, the low frame rate of rendering holographic objects causes eye fatigue. The Microsoft HoloLens HMD as self-contained holographic computer can render holographic objects at 60 fps that enables the 3D design in a highly realistic environment [8].

In general, 3D models or virtual objects are typically made by complicated 3D modeling software, e.g., SolidWork, AutoCAD, Blender, Maya, and Tinkercad, which provides a simple graphical user interfaces (GUI) and precision of 3D model. Usually, mouse and keyboard are used to interaction with those software but this kind of interaction has a lack of immersion and intuition during modeling. The interaction in AR applications helps the novice to understand the designing and creating virtual object. The hand gesture recognition is one of the interactions which the users can interact with the virtual object in mid-air [9,10,11,12]. The Leap Motion is an input device used to track hand gestures and describe the interaction in millimeter scale [13]. The Leap Motion device has been applied in the virtual reality application for designing of 3D model and manipulating the game objects.

The studies of 3D modeling can be divided into two main approaches. First, sketch-based modeling is the creation of 3D models from 2D sketches. This modeling applies the freeform strokes in mid-air and operations such as extrusion, bending, and cutting [14, 15]. Secondly, geometric modeling or clay sculpting is modification or manipulation of the initial 3D shapes [12, 16]. This approach manipulates 3D shapes by moving, increasing, and removing the components of 3D models such as vertex, line, and surface. In addition, the gesture interaction technique was applied in the sketch-based modeling [14, 15] and geometric modeling [12, 16,17,18]. This proposed research work applied the geometric modeling approach for modifying any 3D model shapes with gesture interaction technique. In addition, several 3D prototypes can be made by those 3D models.

Recently, the 3D printing technology has been utilized to produce 3D prototypes easily. The user required the software tool for designing and customizing variety of the 3D prototype models. The immersive modeling systems using virtual reality techniques have been developed for rapid prototyping of 3D objects. The systems were able to manipulate object’s position, orientation, and size. Furthermore, augmented reality can show the 3D model on a mobile device using AR-marker [19]. However, the previous work could not modify the 3D model intuitively before printing.

Hence, this paper presented the design of 3D model for making a 3D prototype using the augmented reality. The Microsoft HoloLens was used to execute the application, receive the data from the sensors and connect to the Leap Motion and 3D printer. The hand gesture interaction was used to manipulate the 3D model including the translation, rotation, scaling, and modifying of the holographic object. This paper explained the system setup and the interaction method for augmented object and physical object in the actual environment. The results showed the system performance and usability of the proposed system related to the designing of holographic object and its 3D printed prototype.

2 System Overview

In general, the designing of 3D prototype using 3D printer has complicated processes such as, learning how to utilize user interface in CAD software, exporting STL file format from CAD software to Slicer software, setting properties of 3D printer in Slicer software, exporting G-code file format from Slicer software to 3D printer controller, and calibration of 3D printer. The beginner, especially non-technical user, spent more time on understanding those processes. The proposed system simplifies those processes by synchronizing between the applications of 3D model designing and 3D printer. In the proposed system, the hologram was used to display the targeted 3D model for being merged with the real environment while the hand gesture recognition was used for user’s interaction with the 3D model. The Microsoft HoloLens, an optical see-through HMD, was used to display the hologram of 3D model. There are two core components of gestures such as air tap and bloom were used to provide natural interaction [20]. The air tap gesture is tapping index finger down similar to a mouse click. The bloom gesture is holding out the hand with palm up and moving fingertips together. In addition, the combination of gaze and air tap gesture was used as the gaze-and-commit function. The combination of air tap and holding gesture allows the function of click and drag. However, the hand gestures were unable to describe the location in space precisely. The Leap Motion was then chosen to operate the interaction with hologram in millimeter unit and increase more various hand gesture interaction.

The proposed system overview in Fig. 1 shows that the user can design any 3D printed prototypes naturally by visualizing the hologram of 3D model and operating with hand gesture recognition. The system devices consist of the Leap Motion, the Microsoft HoloLens, the actual 3D printed cube, a 3D printer, and a computer. The Leap Motion was attached to the Microsoft HoloLens in order to transfer the gesture commands via web socket. The Leap Motion was connected to the computer with 2.8 GHz Intel Core i7 CPU, 8 GB RAM, and NVIDIA Ge-Force GTX 1050 Ti graphics card via micro USB 3.0. It was mounted on the top of Microsoft HoloLens about 0.05 m with declining angle about 25°. The user wore the head-mounted display for visualization and interaction between the virtual hands and holographic objects. The designed holographic objects can be selected and then were printed through web server automatically. The 3D printer was connected to the computer which had a web server running backend application online. The backend application called the data and file from the web server and then opened the slicer program for printing 3D prototype automatically.

Fig. 1.
figure 1

System overview

Full size image

The system consists of three main software parts which are hand gesture manager, graphics manager, and 3D printing manager. The hand gesture manager was used to manage the connection between Leap Motion and the HoloLens application. Leap Motion API was used in hand gesture manager to manipulate 3D holographic object intuitively instead of Gaze-and-Commit interaction from Microsoft HoloLens API. In the preprocessing phase of Leap Motion, it provided the positions and orientations of finger and hand movements. Then, the hand gesture manager sent the virtual hands’ positions and orientations to the graphics manager which took care of the image registration of virtual hand. Vuforia [21, 22] is software for creating the augmented environment by tracking the AR marker in the physical environment for rendering any holographic objects to Microsoft HoloLens. The graphics manager was used to display or visualize the 3D model such as holographic object, virtual hand, and user interface on the Microsoft HoloLens and build a structure of 3D model. Then, the graphics manager sent the structure of 3D model to the 3D printing manager providing the CAD file generation in the form of ASCII STL format by applying the slicing model algorithm [23,24,25,26]. The system converted the structure of 3D model to ASCII Stereo Lithography (STL) standard. Next, the user saved and uploaded 3D printed prototype file from the 3D printing manager to the web server with the timestamp order. The backend application was an application for calling Cura slicer application and uploaded the latest STL file from the database into the Cura slicer application. The Cura slicer converted that file to the G-code and ordered to print the 3D prototype on the 3D printer as shown in Fig. 2.

Fig. 2.
figure 2

System data flow

Full size image

3 Implementations

3.1 Hand Gesture Manager

The hand gesture manager consists of preprocessing data through Leap Motion API in the Orion version. The hand tracking applied the skeleton-based algorithm that provided the positions and orientations of hands and fingers. The Leap Motion device was mounted on the Microsoft HoloLens. The Leap Motion controller implemented the right-handed coordinate system shown in Fig. 3. The coordinate frame was set at the center of Leap Motion. In this proposed system, Unity3D used the left-handed system so its z-axis was the opposite to the Leap Motion controller’s. The Leap motion API helped to solve this problem. The hand gesture manager provided an ability of manipulating with 3D model in 3D space by six features as shown in Table 1.

Fig. 3.
figure 3

Leap Motion controller with right-handed coordinate system

Full size image
Table 1. Gesture commands
Full size table

3.2 Graphics Manager

The graphics manager was used to display or visualize the 3D models such as holographic object, virtual hands, and user interface on the Microsoft HoloLens. Unity3D game engine and Microsoft Visual Studio have supported the development of HoloLens application. Unity3D game engine allows to build any three-dimensional worlds but it is not a CAD software. In addition, the graphics manager was used to create and deform the structure of 3D model sent to the 3D printing manager.

Setting the Holographic Environment.

The graphic manager took care of displaying the holographic object in three dimensions. The visualization of holographic object in this system required the scene configurations of HoloLens camera, AR camera, and Leap Motion camera. The scene properties of HoloLens camera, the main camera, was configured based on the HoloLens guideline. The coordinate system of HoloLens applied a meter as the unit. The AR camera was the scene camera for Vuforia. The Vuforia library supported the registration of holographic object in real world. The main camera was combined with the AR camera by transformation of the target poses into the Microsoft HoloLens coordinate system. The properties of AR camera, such as app license key, datasets, and rendering behavior of the scene, was configured for providing the position of holographic object related with the AR-maker. Next, Leap Motion camera was added into the scene for displaying the virtual hands. Leap Motion camera was translated in Y axis of Microsoft HoloLens about 0.05 unit and rotated with declining angle about 25°. The pose of Microsoft HoloLens had to be determined relatively with the Leap Motion’s pose via Perspective-n-Point (PnP). In the holographic application, it displayed the virtual hand as shown in Fig. 4.

Fig. 4.
figure 4

Virtual hand

Full size image

Mesh Deformation.

The proposed system implemented the vertices mesh deformation based on mathematical operations, combinatorics, and logical connections for mesh manipulation. This mesh modeling provided the cube shape and allowed the gesture commands to manipulate each vertex (Vx, Vy, Vz) of cube in 3D space. The modified cube shape can be translated, rotated, and scaled. The original and modified cubes are shown in Fig. 5.

Fig. 5.
figure 5

Mesh deformation

Full size image

3.3 3D Printing Manager

The 3D printing manager was used to generate a STL file for 3D Prototype. After the holographic cube modification, the user needed to save 3D model file. In general, Unity3D does not provide the feature for saving 3D prototypes file format. Most of 3D modeling methods for designing 3D prototypes apply the Stereo Lithography (STL) standard. The proposed system created the STL format file from a list of the triangular surfaces which describe a computer generated solid model. Usually, this kind of format has two types which are an ASCII (printable character) format and a binary format. In this research, ASCII STL format was used and its syntax structure can be shown in Fig. 6.

Fig. 6.
figure 6

Syntax of mesh modeling in ASCII

Full size image

Figure 7 illustrates the process of 3D prototype making. The process starts from creating of the virtual object in CAD software and exporting it into the STL file format. Next, the user opens the slicer software, configurates the work space of 3D printer, opens STL file, slices the 3D model, and saves G-code for printing. In this proposed system, the 3D printing manager received the structure of 3D model from the graphics manager and converted that structure file into a STL file. Next, the STL file was uploaded to the web server as shown in Fig. 8. The 3D printer was connected to the computer via wireless network. That computer also run the backend application for retrieving the STL file from the database of web server. When the backend application found the new list as shown in Fig. 9, it uploaded the file to Cura slicer for obtaining the G-code and printing the 3D prototype on the 3D printer.

Fig. 7.
figure 7

The process of 3D prototyping

Full size image
Fig. 8.
figure 8

Sending STL file to web server

Full size image
Fig. 9.
figure 9

List of STL files

Full size image

4 Experimental Setup and Evaluation

This section presents the effectiveness of the proposed holographic application in real environment. In experiment, the actual 3D printed cube was put on the table as an initial shape for the evaluation of system. The user wore the proposed device, Microsoft HoloLens attached with the Leap Motion, in Fig. 1. The initial user’s position was far from the table by 2 m when the holographic application began. Afterwards, the user could move to any positions. The optimal zone of operating the proposed system was in the range of 0.85 to 2 m from the center of the actual 3D printed cube to the user. The evaluations investigated the system performance and usability.

4.1 System Performances

The system performance was tested including frame rate, the interaction tasks, and visualization of hologram. The frame rate is the most important for visualization of hologram. It has the effects on a hologram display such as shaking, real time operating, and causing of eye fatigue. The Microsoft HoloLens guide recommended the frame rate should be at least 60 frames per second (fps).

In the proposed holographic application, the normal frame rate was equal to 65 frames per second (fps). The frame rate was 60 to 62 fps when the virtual hands and hologram were displayed. Further, it was 55 to 60 fps when the virtual hand interacted with the holographic object or virtual object. The results of averaged frame rate are shown in Fig. 10. These averaged frame rates were still acceptable for rendering the holographic object during various activities.

Fig. 10.
figure 10

The averaged frame rate of holographic application

Full size image

The interaction tasks were evaluated the performance of the system’s gesture recognition. The user did actual gesture 30 times/command and the detected gestures were counted. The gesture commands consist of six commands such as selecting, translating, rotation, scaling, operating for deformation, and displaying the hand panel. The results showed that the accuracy of gesture command detection in the proposed system was good as shown in Table 2.

Table 2. Gesture command detection in holographic application
Full size table

4.2 Usability

The usability was evaluated using the proposed holographic application for creating some 3D prototypes. In this experiment, a physical object, an actual 3D printed cube, was put on the table as an initial shape and list of three targeted shapes was given to the participants for designing of 3D prototypes. The participants were eight people participated in the usability test. They were all volunteers aged between 19 and 28 at the Institute of Field Robotics. The participants were entry level or non-experience CAD designer. Two participants have used the Microsoft HoloLens while six persons never used it.

First, the participants designed holographic cube to fit with the actual 3D printed cube via HoloStudio application and proposed holographic application. The HoloStudio application provided hand gesture interaction with core hand gestures of HoloLens. The hologram was manipulated by gaze and commit interaction of HoloStudio application. The proposed holographic application provided the hand gesture interaction using both hands via the Leap Motion API. Its hologram was manipulated by gesture commands in Table 1. Designing times were limited to 15 min. The holographic object was saved in STL file format from both applications and printed as 3D prototype. The sizes of holographic objects were compared with the size of actual 3D printed cube for measuring of accuracy along with averaged time of designing as shown in Table 3. The result showed that some users spent times of designing less than the limited time. The shorter time implies that one application would be easier for designing of 3D model than another one.

Table 3. Comparison of holographic object’s size
Full size table

Secondly, the participants were asked to design and deform the original holographic cubes into three targeted shapes in Table 4. Next, those holographic objects were saved in STL file format from the proposed applications and printed as 3D prototypes. It was found that the HoloStudio application could not perform this task due to lack of deformation capability. The outputs from this experiment were the real 3D printed prototypes obtained from the designed holographic object. In addition, the 3D prototype was assembled with the physical object as shown in Fig. 11.

Table 4. Averaged time of designing
Full size table
Fig. 11.
figure 11

3D printed prototype assembled with physical object

Full size image

Afterwards, the participants were asked to fill the questionnaire for evaluating the level of user’s satisfaction. The questionnaire rating started from 1 (strongly disagree) to 7 (strongly agree). The results from the questionnaires are shown in Table 5.

Table 5. Results from questionnaires
Full size table

5 Conclusions

This research proposed the design of 3D model and build of 3D prototypes by merging technologies such as augmented reality (AR), hand gesture recognition, and 3D printing. The proposed system consisted of three main software parts which were hand gesture manager, graphics manager, and 3D printing manager involving with the Leap Motion, Microsoft HoloLens, and 3D Printer, respectively. Some experiments were conducted to evaluate the proposed system performance and usability of designing of 3D prototypes using augmented reality. The experimental results showed that the normal frame rate was equal to 65 fps which was more than Microsoft HoloLens guide recommended. Although the frame rate during interaction between virtual hands and holographic object was dropped to 55 fps, it was still acceptable to provide a good experience to the user. Moreover, after comparison of the uses of HoloStudio and the proposed application for designing of 3D model, it was found that the proposed system was better than the HoloStudio due to the shorter times of designing and more available features of hand gesture interaction. In addition, the size of 3D prototype built by the proposed system was nearly with the size of targeted object. The result from the questionnaire showed that the proposed system was able to reduce designing time, enhance user’s experience, and improve the usability.

In the future, several initial 3D shapes should be available in this proposed system so that the user can design and build different complex 3D prototypes. Furthermore, the system devices could be reconfigured to be a wearable system.