Abstract
Speech processing techniques help improving real-time communication between human to human, human and machine, machine to machine. If these techniques are used integrated with robots, then their physical flexibility and wider reach can enable a wide range of real-time applications. In this paper, we propose an ‘Intelligent Cry Detection Robotic System’ (ICDRS) for real-monitoring of child-beating in classrooms, in order to facilitate prevention of child-abuse prevalent in this form. The proposed system has two major modules: the ‘Cry-Detection System’ (CDS) and a ‘Smart Robotic System’ (SRS) equipped with audio-visual sensors. The CDS unit present in the classroom, consist of three parts. First, the cry-recording unit (CRU) records the audio signals and sends to ‘Signal Processing Unit’ (SPU). Then SPU applies signal processing techniques, and intelligently detects the cry events using the features extracted from the acoustic signal. If the system detects a cry, then it further sends the control commands to the ‘Signal Transmission Unit’ (STU), which sends an automatic SMS to the Vice-Principal or Supervisor-Teacher, i.e., the person-in-charge, thereby alerting him/her about the child cry in a particular classroom. The controlling person can give control-commands to the SRS from a web-application and can get the live-stream of the video from the classroom. A Wi-Fi module acts to facilitate the communication between this controller and the Robot (SRS). The initial performance evaluation results are very much encouraging. The proposed system can have potential applications in the schools, hospitals and child care centers etc. Hopefully, this prototype can be a useful step towards preventing child-abuse, prevalent in different forms in our society.
1 Introduction
Generally robots are highly sophisticated and powerful machines which are designed to perform tedious work In addition, they assist the human beings in many aspects of the life. Initial robots were developed in 1940s. They were mainly used for handling the radioactive materials and then in late 1960s these were also used for picking and placing of the objects in different places. Connectivity between the controller and the robot can be through some wired connection or through wireless communication [1]. With advancement of technology connectivity has shifted to wireless communication because of very less range of control operation of wired networks. Through wireless Communication range of control can be increased. Nevertheless, it is also limited. Wireless Communication can take place using different technologies like Bluetooth, WiFi, RF, IR etc.
On the other hand, speech is one of the easiest ways of communication between people for expressing views, ideas and thoughts. Speech Signals can be classified into voiced [2] and unvoiced sounds. Voiced speech signals are because of the vibration of the vocal folds, for example these would be vowels. Unvoiced speech signals don’t involve the vocal folds vibrations, for example these would be fricative sounds [3]. In the same way Cry, Shout, Laughter also falls under the category of paralinguistic speech signals, where these kind of signals consists of some voicing activity in them.
Now a days child abuse and beating of children is getting more predominant in every part of the world especially in schools. Sometimes, higher authorities may not know about the issue and may not be able control situation, which can lead to any dangerous situation. In order to handle such issues, most of the high end schools have installed surveillant cameras in the classrooms for monitoring the students, However deploying these cameras in the every classroom is costly and it also requires an human to watch the surveillance footage every moment in order to monitor the students in the classrooms which is difficult task. This paper proposes an Intelligent Cry Detection Robotic System (ICDRS) in order to address the above mentioned problem, This automated system will monitor the classroom, However the Cry Detection System deployed in the classroom will detect the cry and notify the controller with an alert message.
In this paper, we developed an ICDRS for classroom monitoring. This integrates speech signal processing techniques and wifi controlled robot for this application. It processes cry signal (paralinguistic speech signal) [4] and extracts the energy feature in the CDS. Signal energy is the combined feature of source and system [5]. The developed prototype validates the cry using this combined feature i.e., Energy of the signal for detecting and analyzing the cry.
ICDRS is divided into two parts, one is the Robotic system and another is the Cry detecting system. The Robotic system can be remotely operated by the controller using WiFi communication medium. Robotic system is controlled by the controller and it travels to the classrooms according to the control commands it receives. This will live stream the video of the classroom with the camera mounted on it. Before the robotic system starts its operation, the Cry Detection system will be recording the sound signals and analyzing them on the microcontroller, if the cry detection system finds any cry in the classroom it will notify the controller with an SMS saying that child is crying in the classroom with the help of the GSM module integrated in the system.
This paper is organized as follows. In Sect. 2, it gives an overview of the design aspects of the Intelligent Cry Detection Robotic System (ICDRS) in both hardware and the software aspects for developing the Cry detection System and the Robotic platform. In Sect. 3, Intelligent Detection and Control operation, Cry detection, web based control and also the video/image streaming are discussed. Section 4, gives the performance evaluation which includes the various results and experiments carried out in the ICDRS. Explanation about the advantages of the prototype are given in the Sect. 5. Possible applications are discussed in detail in the Sect. 6 and followed by summary and future scope in the Sect. 7.
2 Design Details of Intelligent Cry Detection Robotic System (ICDRS)
This section explains about the hardware and software architectures followed in building the ICDRS. Figure 1, explains about the functional block diagram of the ICDRS. Figure 2, gives the overall working architectural design of the ICDRS prototype which includes all the key components being used. The robotic system block explains how the robot is controlled using WiFi connecting medium and also the cry recording, processing [6] and generation of SMS is explained in the cry detection system block.
2.1 Design Details of the Smartbot Prototype
The key components used in building this part are Arduino an 8-bit microcontroller with a clock speed of 16 MHz, ESP8266 is an low cost WiFi module for the connecting medium, which as a specialty of acting as both client and server. L293D, A dual H- bridge driver integrated circuit is used to amplify the low current control signal from the microcontroller and produce a high current control signal for the DC motors movement. DC motors are fixed to the robotic chases [7] and pins of the motors are connected to the motor driver integrated chip for receiving the control commands from the microcontroller [8]. Table 2, gives all the necessary AT commands for configuration of the WiFi module to network [9].
2.2 Design Details of the Cry Detection System (CDS)
The key components used in building the CDS are Raspberry pi 3, Model B, which is the heart of the entire system. Matlab Simulink models are deployed into the microcontroller and processed in it [10]. Zebronica microphone is used to record the sound. Another end of the microphone is connected to the 7 channel USB soundcard and further connected to the microcontroller. After processing an control signal is sent to the arduino which is connected to the raspberry pi for the further process. Arduino will communicate with the GSM module and send an SMS to the controller. GSM module used in the system is sim900a, its on the quad band technology which includes bands of 850/900/1800/1900 MHz. AT commands are used for the interaction of the GSM with arduino. Sending an SMS to the controller based on the child cry to the pre-configured number is done using the AT commands. Table 1, consists of all required AT commands for configuration and sending SMS for the GSM module [11].
Initially it record the signal and processed in the matlab Simulink model. MATLAB Simulink [12] is a simulation tool, it is a design environment for embedded systems which are integrated with the MATLAB. This enables to compute MATLAB algorithms. Further this also encourages to support system level design, simulation, code generation and testing of the embedded systems.
For detecting the Cry signal of the child, Combined feature of source and system i.e., Energy of the signal is used in the ICDRS. Energy of the signal is the summation of energy across all the frequency components of signal spectral energy density. For ICDRS, energy is calculated using the auto correlation [13]. Highest peak after applying the auto correlation is the energy of the signal, when the time-lag(m) is zero.
3 Intelligent Detection and Control Operation
3.1 Cry Detection
Cry detection system makes ICDRS as a real time system by processing real time information. Mic present in the system continuously records the audio signals and sends them to the processing unit in buffers. Processing unit analyses this data by extracting features from the data. These features are extracted by applying signal processing techniques on them. This work is done by the SIMULINK model dumped in the processing unit. Later, the values obtained are validated against the data present in the unit, which helps the system to intelligently identify the cry signal. As soon as the unit identifies the cry signal it sends control command to the microcontroller intimating it about the baby cry. This, in turn sends message to the controller, alerting him/her about the child cry in the classroom (Fig. 3).
3.2 Web Based Control
WiFi module i.e., ESP8266 doesnt require any serial communication using wires, it gives a flexibility of wireless [14]. The control commands are sent from the web page with required instructions, those instructions are generated in the form of a HTTP request and it is received by the WiFi module which is mounted on the Smartbot System. Those signals are decoded and processed for the smartbot movement. For movements, forward, backward, left, right directions are written on the web page. Depending on the button operated by the controller on the web page the robots moves in that direction [15]. In the same way the video streaming [16] is also done by clicking of the button the page [17].
3.3 Video Streaming/Images of the Classroom
In order to conform that child is crying in the classroom, the smartbot is sent to the classroom to capture the image or video. This also confirms that if there is any conflict going on in the class for the child cry. In this paper video streaming [18] done using the Internet Protocol(IP) Camera which is compatible with the WiFi [19]. This camera is mounted on the smartbot based on the control command [20] the camera module turns on capture the real time information of the classroom [21]. The captured images or videos can be seen on the smart phone or in the web browser [22]. Figure 2, shows the screen shot of the SMS received by the controller from the ICDRS.
4 Performance Evaluation
Various delays encountered during the operation of the system are tabulated in Table 3. This table consists of an average values of delay time, when the experimentation is carried out for 50 times. It is observed that if network is not good near the ICDRs, then the SMS will not be sent by the system to the controller/parent, because GPRS used in the system requires good network connectivity for the SIM present in the module to communicate to the GPRS module. Time taken for the SMS to transmit to the controller depends on the connectivity of the network.
In addition time taken for the control commands to execute after selecting a button on the web page is about 1.01 s. This is the time taken by WiFi module to send the information to the micro-controller present on the robotic platform [23]. Time taken for the processing of signals is about 0.2 s, which is less compared to all other operations. This may be differed when, size of the data analyzed changes, probably it increases for large data and decreases for small data size. Delay time encountered for live video streaming is on average 2 s, which is more compared to all other operations.
5 Advantages of ICDRS
5.1 Intelligent Smart System
Cry Detection system present in the ICDRS analyzes audio signal, extracts features from it and validates it with the features of the cry signal and decides whether the signal is a cry signal or not.
5.2 Real Time Monitoring
Mic present in the cry Detection system records the audio signals and processes them in real time. In addition, it sends the message to the controller regarding the child cry and alerts him about the situation. Controller gives the control commands to the robot through Wi-Fi [24] and can see the live stream of the video of the classroom [25]. All these operations take place in real time, so the proposed system is a real time monitoring system.
5.3 Future Extension Work
This project is further extended to analyze the infant babies for real time monitoring. This would be extended to monitor the environment and climatic changes around the baby cradle. As a future progress reason for the cry would be analyzed. As previous work [11] helped to classify the baby cry into various categories like pain, discomfort, hungry etc.
6 Possible Applications
The developed Intelligent Cry Detection Robotic System has wide area of applications in day to day life, which is an real-time application of the speech signal processing in the world where this could be built with an low budget. Few important key applications are discussed in this section.
6.1 Prevention of Child Abuse
Physical maltreatment of children is present in all corners of the world, it disturbs the children mentally and physically. In some nations it is even considered as a crime. In most of the scenarios children may not be able to convey their pain to their parents, may be because of fear or any other reason. This case is more in children below age of 5. The proposed ICDRS can be modified to solve these kind of issues, where the system detects the child cry and tries to find the reason behind the cry, identify the maltreatment like beating scenario and notify the parents or to the respective people to aware them about the situation for further reaction.
6.2 Baby Care Centers
Most of the parents are forced to leave their children in baby care centers or with a nanny or with a relative in their absence, because of their tight scheduled work. But, they are panic and are curious about their child, they don’t have any idea whether the person with whom they left their baby are able to take care of their child. In addition they always worry whether their child is crying or is he/she fine. The proposed system can to help parents to get free from their worries. The system can be modeled to monitor babies and detect the child cry and alert the parent [26]. It can even analyze the cry of the baby and find the reason behind their cry, which helps the parents and the care takers to take care of the baby accordingly. System can detect the environmental changes around the baby and notify the parent about the situation, it can also try to calm down the baby by playing light music or blinking different lights [27].
6.3 Applications to Study of Other Paralinguistic Sounds
Cry signal is one of the paralinguistic sound, laughter, cough and shout also falls under this category. The proposed system can be modified and used for applications of other paralinguistic sounds. The work on analysis of cough [4], laughter [28], shout [29] has been done earlier where various features like Energy and Formant frequencies are studied, the analysis on these paralinguistic features can be embedded with the hardware and systems for various applications could be developed.
6.4 Infant Cry Analysis
Infant cry as it is also an audio signal carries some meaningful information, which can be analyzed by using signal processing techniques [30]. The information obtained from the features extracted can be used to identify the reason for baby cry [31], that is analyzing the cause of cry like pain [32], discomfort, anxiety, hunger etc., [33] Analysis can be helpful to identify the baby who is crying from among many babies in a child care center or a play school [34]. In addition we can identify the severity of baby cry and caution the parent or the respective person [35].
7 Summary and Conclusion
An Intelligent Cry Detection Robotic System (ICDRS) for classroom monitoring and real time information is developed in this paper. This system is classified into two parts smart robotic monitoring and Cry detection system. This robotic system can be controlled remotely using the WiFi connectivity using the ESP8266 (WiFi module) mounted on the robot which gives a flexibility of wireless. This paper also explains how the signal processing is embedded into the microcontroller for the cry detection. The developed system will record the signals and process them in the microcontroller. If the system encounters the cry then it would trigger the arduino and generate an SMS to pre-defined number using the GSM module to the concerned people.
The biggest drawback faced is to configuring the WiFi module before processing. For configuring it requires another router. Another limitation in the system is GSM module can send the SMS only when proper network is present near the system. The paper also provide comprehensive view of the embedding the MATLAB Simulink into the microcontroller and the WiFi technologies for the connectivity between the robot and the controller. This research could help the upcoming researchers for conducting tests and experiments.
References
Asthana, S., Varma, N., Mittal, V.K.: An investigation into classification of infant cries using modified signal processing methods. In: 2nd International Conference on Signal Processing and Integrated Networks (SPIN), February 2015, pp. 679-684 (2015)
Rabiner, L.: On the use of auto-correlation analysis for pitch detection. IEEE Trans. Acoust. Speech Sig. Process. 25, 24–33 (1977)
Oppenheim, A.V., Schafer, R.W., Buck, J.R.: Discrete Time Signal Processing. Prentice-Hall, Englewood Cliffs (1989)
Mittal, V.K., Yegnanarayana, B.: Study of characteristics of aperiodicity in Noh voices. J. Acoust. Soc. Am. (JASA) 137(6), 3411–3421 (2012)
Titze, I.R., Story, B.H.: Acoustic interactions of the voice source with the lower vocal tract. J. Acoust. Soc. Am. (JASA) 101, 2234–2243 (1997)
Asthana, S., Varma, N., Mittal, V.K.: Preliminary analysis of causes of infant cry. In: IEEE International Symposium on Signal Processing and Information Technology, ISSPIT, 15–17 December 2014, pp. 468–473 (2014)
Kumar, K., Nandan, S., Mishra, A., Kumar, K., Mittal, V.K.: Voice-controlled object tracking smart robot. In: Proceedings of IEEE 3rd International Conference on Signal Processing, Computing and Control 2015 (ISPCC 2015), JUIT, Waknaghat, India (2015)
Sharma, S., Asthana, S., Mittal, V.K.: A database of infant cry sounds to study the likely cause of cry. In: 12th International Conference on Natural Language Processing (ICON 2015), IIITM-K, Trivandrum, India (2015)
Ryu, D., Kang, S., Kim, M., Song, J.-B.: Multi-modal user interface for teleoperation of ROBHAZ-DT2 field robot system. In: 2004 Proceedings of IEEEIRSJ International Conference on Intelligent Robots and Systems (IROS 2004), vol. 1, pp. 168–173. IEEE (2004)
Raj, P., Rajasree, N., Jayasri, T., Mittal, Y., Mittal, V.K.: A Web Based Intelligent Spybot. In: Proceedings of IEEE 3rd International Mining Intelligence and Knowledge Exploration (MIKE 2015), vol. 9648, pp. 472–481. IIIT Hyderabad, December 2015
Mittal, V.K.: Discriminating the infant cry sounds due to pain vs. discomfort towards assisted clinical diagnosis. In: 7th Workshop on Speech and Language Processing for Assistive Technologies, SLPAT 2016, San Francisco, USA, 13 September 2016, pp. 37–42 (2016)
MATLAB Simulink Projects. https://in.mathworks.com/discovery/simulink-projects.html. Last Viewed 1 June 2017
Mittal, V.K.: Discriminating features of infant cry acoustic signal towards automated diagnosis of cause of crying. In: 10th International Symposium on Chinese Spoken Language Processing (ISCSLP 2016), Tianjin, 17–20 October 2016
Morshed, N.M., Muid-Ur-Rahman, G.M., Karim, M.R., Zaman, H.U.: Microcontroller based home automation system using Bluetooth, GSM, Wi-Fi and DTMF. In: International Conference on Advances in Electrical Engineering (ICAEE) (2009)
Narvaez, L., Llanes, E., Hernandez, C., Poot, R., Chi, V.: Design and implementation of a system for wireless control of a robot. Int. J. Comput. Sci. Issues 7, 191–197 (2010)
Dayma, D., Chavan, B., Kale, S., Tarle, B.S.: Smart Spy Robot. Int. J. Sci. Technol. Manage. 04(02), 93–95 (2015)
Narvaez, L., Lanes, E., Hermandez, C., Poot, R., Chi, V.: Design and implementation of a system for wireless control of a robot. Int. J. Comput. Sci. Issues 7, 191–197 (2010)
Makula, P., Mishra, A., Kumar, A., Karan, K., Mittal, V.K.: Voice-controlled object tracking smart robot. In: Proceedings of IEEE 3rd International Conference on Electronics, Computing and Communication Technologies (CONNECT 2015). IIIT Bangalore, India, 10–11 July 2015
Patoliya, J., Mehta, H., Patel, H.: Arduino controlled war feild spy robot using night vision wireless camera and android application. In: 5th Nirma University International Conference on Engineering (NUiCONE), pp. 1–5
Vaishnav, P., Tiwari, S.: Accelerometer based hand gesture controlled robot. Int. J. Sci. Res. (IJSR) 4(3), 223–226 (2015)
Balakrishnan, M.: A smart Spy robot charged and controlled by wireless systems. In: IEEE Sponsored 2nd International Conference on Innovations in Information Embedded and Communication Systems, ICIIECS 2015, Coimbatore, 19 March 2015
Maheshwari, T., Kumar, U., Nagpal, C., Ojha, C., Mittal, V.K.: Capturing the spied image-video data using a FlexiControlled Spy-robot. In: 2015 Third International Conference on Image Information Processing, pp. 330–335, December 2015
Benefits of Robots - RoboWorx .https://www.robots.com/articles/viewing/benefits-of-robots. Last Viewed 15 May 2017
Mittal, Y., Toshniwal, P., Sharma, S., Singhal, D., Gupta, R., Mittal, V.K.: A voice-controlled multi-functional smart home automation system. In: Proceedings of IEEE 12th International INDICON Conference, Jamia Millia Islamia, New Delhi, India (2015)
Jain, P., Firke, P.N., Patil, T.S., Rode, S.S., Kapadnis, K.N., Kapadnis, K.N., et al.: RF based Spy robot. Int. J. Eng. Res. Appl. 4(4), 06–09 (2014). (Version 2)
Lavner, Y., Cohen, R., Ruinskiy, D., Ijzerman, H.: Baby cry detection in domestic environment using deep learning. In: 2016 IEEE International Conference on the Science of Electrical Engineering (ICSEE), pp. 1–5, November 2016
Skogsdal, Y., Eriksson, M., Schollin, J.: Analgesia in newborns given oral glucose. Acta Paediatrica 86(2), 217–220 (1997). http://dx.doi.org/10.1111/j.1651-2227.1997.tb08872.x
Mittal, V.K., Yegnanarayana, B.: Analysis of production characteristics of laughter. Comput. Speech Lang. 30(1), 99–115 (2015)
Mittal, V.K., Yegnanarayana, B.: Effect of glottal dynamics in the production of shouted speech. J. Acoust. Soc. Am. 133(5), 3050–3061 (2013)
Petroni, M., Malowany, M., Johnston, C., Stevens, B.: A new, robust vocal fundamental frequency (F0) determination method for the analysis of infant cries. In: 1994 Proceedings of IEEE Seventh Symposium on Computer-Based Medical Systems, pp. 223–228 (1994)
Chandralingam, S., Anjaneyulu, T., Satyanarayana, K.: Estimation of fundamental and formant frequencies of infants cries; a study of infants with congenital heart disorder. Indian J. Comput. Sci. Eng. 3(4), 574–582 (2012)
Daga, R.P., Panditrao, A.M.: Acoustical analysis of pain cries in neonates: fundamental frequency. IJCA Spec. Issue Electron., Inf. Commun. Eng. ICEICE 3, 18–21 (2011)
Mima, Y., Arakawa, K.: Cause estimation of younger babies’ cries from the frequency analyses of the voice - classification of hunger, sleepiness, and discomfort. In: 2006 International Symposium on Intelligent Signal Processing and Communications, ISPACS 2006, pp. 29–32, December 2006
Neustein, A. (ed.): Advances in Speech Recognition: Mobile Environments, Call Centers and Clinics. Springer, New York (2010)
Cohen, R., Lavner, Y.: Infant cry analysis and detection. In: IEEE 27th Convention of Electrical & Electronics Engineers in Israel (IEEEI 2012), pp. 1–5. IEEE (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Myakala, P.R., Nalumachu, R., Mittal, V.K. (2018). A Low Cost Intelligent Smart System for Real Time Child Monitoring in the School. In: Thampi, S., Krishnan, S., Corchado Rodriguez, J., Das, S., Wozniak, M., Al-Jumeily, D. (eds) Advances in Signal Processing and Intelligent Recognition Systems. SIRS 2017. Advances in Intelligent Systems and Computing, vol 678. Springer, Cham. https://doi.org/10.1007/978-3-319-67934-1_39
Download citation
DOI: https://doi.org/10.1007/978-3-319-67934-1_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-67933-4
Online ISBN: 978-3-319-67934-1
eBook Packages: EngineeringEngineering (R0)