Keywords

1 Introduction

LEGO bricks are very popular among children, and for those with autism spectrum disorder (ASD), the bricks could be a good auxiliary toy to improve their social communication skills [1,2,3]. In this paper, we proposed a system to assist children with ASD playing with LEGO bricks, in which it could recognize a 2D structure (n × m bricks) and generate the whole layout through a camera automatically. More specifically, the system allows teachers to create a task (structure), tracks the bricks played by children in real time and give children feedback according to their built result. The proposed system can also reduce the teacher’s workload and allow them to monitor more children simultaneously.

At the current stage, we have implemented a prototype for detecting 2D structures containing up to eleven bricks in three different colors and three different dimensions. We also made a model for the educators to design their unique brick structures for the children. The system can load a pre-defined (target) layout and then generate instructions for the users in both text and speech. In the future, we will expand the system to detect 3D structures with more brick colors and shapes. More interactive methods will also be introduced to facilitate the learning process.

2 Related Work

2.1 LEGO Therapy for Children with ASD

Many studies have shown that early intervention could be beneficial for children with autism as reported in [4], which could be performed for children as young as 18 months age [5]. LEGO bricks, which are very popular for toddlers, have been used as a therapeutic medium to enhanced children’s social communication skills [1,2,3]. In the playing process, children are assigned with various roles and expected to collaborate to build a structure following a set of clear instructions provided by their teacher. Given a clear and predictable task in the LEGO play, children with ASD would feel more comfortable compared to in a free-play one; hence, would be more likely to initiate communication with their peers.

Recently, technology-based intervention using LEGO play, including LEGO robots, have also been introduced to children with ASD [6, 7]. However, using LEGO in an intelligent tutoring system for improving their social communication skills is rare. In this project, we attempt to fill this void.

2.2 The Adoption of Learning Technology

The increasing number of children diagnosed with ASD has become an issue around the world, especially for the educators [8, 9]. For instance, in the U.S., the prevalence of children aged 8 years with autism was 6.7 per 1,000 in 2000, and increased to 14.7 per 1,000 in 2010 [10]. Since the increase would bring more pressure to the special-education teachers, many attempts have been proposed to reduce their workload through the introduction of various assistive learning technology, including self-monitoring systems [11], electronic AAC [12, 13], collaborative-learning applications [14, 15], and intelligent tutoring systems, as in our case here. In fact, some applications are proven effective in improving children’s social communication skills. In addition, some researches have also demonstrated a better acceptance of electronic technologies to the youth with autism [16, 17], which also motivates us to implement the system.

3 Motivation and Design Rationale

Our proposed system is intended for a small classroom of 8 to 10 students (preschool children age 3 to 5) and a supervising teacher. During the activity, students will be divided into 4 to 5 pairs, and each pair will be assigned to collaboratively build a LEGO structure on a special desk equipped with the system. Ideally, all five systems could simultaneously notify the teacher any important progress made by their students, such as the completion time, frequency and type of mistakes, etc. The teacher, according to their students’ learning progress, may (re-)design appropriate learning scheme for them. Each learning scheme may consist of a set of LEGO structures to be built within a certain duration (e.g. 30 min). The system, on the other side, may automatically tag each learning scheme with its average completion time, the frequency of mistakes and/or some common mistake(s) in building them, so that it can be evaluated/redesigned in the future.

Since many preschool children with ASD exhibit delay or deficit in gross and fine motor skill, and many comorbid with ADHD and/or dyspraxia, the instructions used in our system must be comprehensible for them, for instance, using simple words delivered in low-volume voice, avoiding sensitive colors (e.g. yellow), providing repetitive instructions, etc. Finally, the material must be affordable for a special-education (public) school; for example, the current prototype consists of a decent laptop with an ordinary webcam, and some LEGO Duplo bricks.

4 System Implementation

The test set of bricks is shown in Fig. 1. Eleven basic LEGO Duplo bricks with three different shapes (short, long, curved) and colors (red, green, blue) are used in the testing. Figure 2 shows a USB camera, connected to a computer, is put in front of the bricks to track the changes. In terms of the software, there are two separate parts for both teachers and children. Teachers can use an interactive program to create a unique structure of bricks (designer mode). The module designed for children can then load the structure and generate the guidance automatically (player mode).

Fig. 1.
figure 1

The blocks used in the test. (Color figure online)

Full size image
Fig. 2.
figure 2

An example of LEGO structures being tested.

Full size image

4.1 Brick Recognition Module

The proposed system uses several functions, including some from the OpenCV library, to process frames from the camera and generate contours. Figure 3 shows the flowchart of the image processing. First, the pixels are filtered based on their RGB values. Since only red, blue, and green colors appear in the test set, pixels containing other color will be replaced with a white one. Then, the derivatives on x- and y-axis are calculated using an extended Sobel operator. The two derivatives will be subtracted to sharpen the contours of bricks. Threshold and blur functions are also used to remove the noise in the frame. The resulted grayscale image will then be passed to OpenCV’s findContours function. For each contour, a minimum rotated rectangle contains a closed contour is computed, and the program will try to figure out whether the contour belongs to a LEGO brick or not based on properties such as the rectangle’s area, slope, and length-width ratio. The program also samples points inside a rectangle to determine the brick’s color. Figure 4 gives the program’s final label on one frame (the labels are not displayed in a released version). Some critical properties to recognize the bricks are introduced in Table 1.

Fig. 3.
figure 3

The pre-processing of an input frame. (Color figure online)

Full size image
Fig. 4.
figure 4

The result of block recognition.

Full size image
Table 1. Some properties used to recognize the bricks.
Full size table

4.2 Layout Recognition Module

To determine whether a brick is at a correct position or not, we introduce a coordinate system to represent its position relative to the base. Since all bricks have the same height, we use it as the measurement unit of their length as shown in Fig. 4. In this example, the size of the short bricks is rounded to 1 × 2, hence, the long one is 1 × 4. Specifically, the coordinate (x, y) of a brick contains its row number and the left corner’s projection on the x-axis. In this way, a specific composition of bricks on one side can be represented in numbers and easily compared with target layouts.

4.3 Interaction Module

The user interface is generated using cvuiFootnote 1, a UI library built on OpenCV. Before children can use the system, their teacher must create a target layout (learning scheme) using the designer module (will be explained in the next section) and store it into the system. For the sake of maintainability, the target layout will be encoded in the format [size] [color] [x-value] to describe every block and saved in an unencrypted text file. For instance, a layout in Fig. 5 will be written as:

Fig. 5.
figure 5

The coordinate system used in our prototype to represent a brick layout. (Color figure online)

Full size image

  • long red 0

  • long blue 0

  • short green 0 short red 2

Using a plain text file, the system could easily generate textual (or speech) instructions for each building step, for instance, we could read the first line above, interpret and insert it into a textual template to generate a statement “put a [long red] brick []”. The second line can be interpreted and used to generate statement “put a [long blue] brick [on top of the [long red] one]”; followed by “put a [short green] brick [on top of the [long blue] one]”, and “put a [short red] brick [beside the [short green] one]”, and so on. The system will allow teachers to change the textual templates to fit their students’ language abilities.

When a wrong brick is placed, the system can also provide feedback and wait for the user to correct it. The checking process is done by comparing the output of layout recognition module with the target layout in real time.

To make the instructions more attractive and comprehensible, our system uses various text-to-speech modules, for example, a sentient child tone or male/female adult voice with pace appropriate for the target children. At the current stage, three types of voice feedback are provided in our system, some examples are shown in Table 2. The voice is generated using text to speech service from BaiduTM.

Table 2. The voice feedbacks in the system.
Full size table

5 System Prototype

5.1 Designer Mode

Figure 6 shows the user interface of the basic designer module intended for creating learning schemes. Teachers can select a brick with specific shapes and colors on the right pane and then put it on the left pane (a 10 × 10 grid), layer by layer up to eleven layers. After they finish designing, they can click ‘Output Scheme’ to save the target layout in the filesystem which can be retrieved in the future.

Fig. 6.
figure 6

The designer module.

Full size image

5.2 Player Mode

Figure 7 demonstrates the user interface of the system in action. The main part on the interface is a camera view on the left pane, which will be used to track the building steps and provide visual notifications, if any.

Fig. 7.
figure 7

The user interface of the player module.

Full size image

Indeed, the camera must be adjusted manually to cover the working area or the system may not work properly. At this stage, no calibration is required; hence, a light color background (e.g. white) with bright ambient light is suggested for an optimal recognition. After adjusting the camera, the user may load a target layout, which be automatically shown in the right pane (see Fig. 8). In this example, the layout consists of six bricks, including a curved green brick on the top.

Fig. 8.
figure 8

The system is notifying a mistake. (Color figure online)

Full size image

Instructions on what brick should be picked will be given in both textual (e.g. “Get one Blue long block” in Fig. 8) and auditory form (via computer speaker), because most young children may not read but their supervising teacher may read the message later (recall that a teacher may supervise five groups of students, hence, s/he may not remember the latest instruction of all five). During the building process, the system will monitor the whole structure and provide feedback accordingly, for example, in the right pane of Fig. 8 the correct red brick is ticked and the current target blue brick is marked with a yellow star. If a brick is incorrectly placed, the system will highlight the wrong brick in the camera view (left pane) and provide an instruction to check and replace it. In Fig. 8, the red long brick in the camera view is in a correct position, but the green short brick is incorrectly placed - it should be a blue long brick, instead. In this example, the mistake is marked clearly in the camera view along with a message ‘check again!’.

When the target structure has been correctly built as shown in Fig. 9, all bricks in the right pane are ticked and the notification ‘you have completed all of the tasks’ is given to the user, also in both textual and auditory forms. In this case, the system may load a new layout, or the teacher may assign a new scheme for the children or exit the system.

Fig. 9.
figure 9

The notification when all steps are completed.

Full size image

6 Concluding Remarks

In this paper we have demonstrated a prototype that can assist children to build a trivial LEGO structure. The system enables teachers to design some target structures, assign one of them, and guide children to build it step by step. Currently, the system can recognize the brick structure from one side based on frames recorded by a webcam. Using the system, it may reduce teachers’ workload, yet capable of tracking more children simultaneously. In addition, the dialog module for enhancing children’s interaction with the system is under construction.

In the future, some improvements will be made in order to generate 3D structures to support a richer combination of LEGO bricks. Also, more instructions in speech will be added so that the children may focus more on their work instead of the screen.