Integrating the Latest Artificial Intelligence Algorithms into the RoboCup Rescue Simulation Framework

Abstract

The challenge of the Rescue Simulation League is for a team of robots or agents to learn an optimal response to mitigate the effects of natural disasters. To operate optimally, several problems have to be jointly solved like task allocation, path planning, and coalition formation. Solve these difficult problems can be quite overwhelming for newcomer teams. We created a tutorial that demonstrates how these problems can be tackled using artificial intelligence and machine learning algorithms available in the and the . Here we show (1) how to analyze and model disaster scenario data for developing rescue decision-making algorithms, and (2) how to incorporate state-of-the-art machine learning algorithms into Rescue Agent Simulation competition code using the Engine API for Java.

1 Introduction

Urban Search and Rescue (USAR) scenarios offer a great potential to inspire and drive research in multi-agent and multi-robot systems. Since the circumstances during real USAR missions are extraordinarily challenging [8], benchmarks based on them, such as the RoboCup Rescue competitions, are ideal for assessing the capabilities of these systems. Thus, one goal of the RoboCup Rescue competitions is to compare the performance of algorithms that coordinate and control teams of either robots or agents performing disaster mitigation tasks.

In particular, the Rescue Agent Simulation competition aims to simulate large scale natural disasters, such as earthquakes, enabling the exploration of new forms of autonomous coordination of heterogeneous rescue teams under adverse conditions. This competition was first demonstrated in the RoboCup 2000 [12] and officially launched in the RoboCup 2001. Participating teams have their background mainly from artificial intelligence and robotics.

Fig. 1.

View of disaster scenario in the Kobe map after an earthquake.

The competition is based on a simulation platform and a set of complex scenarios representing the conditions of cities after an earthquake (see Fig. 1). In each scenario, fire brigade, police force and ambulance team agents extinguish fires, unblock roads, and rescue civilians trapped inside collapsed buildings, respectively. The final score in the scenario is calculated based on the number of rescued civilians and the number of remaining buildings taking into account the damage caused by the fire. Scenarios typically contain up to 5000 buildings and up to 1000 civilians, as well as agent teams of fire brigades, police forces and ambulance teams composed of up to 50 agents each.

The complexity of these scenarios imposes several challenges to the development of different aspects of multi-agent systems like task allocation with uncertainty, coalition formation, cooperation, distributed control, and communication [1]. Artificial intelligence (AI), in particular machine learning (ML) algorithms are very well suited to cope with some of these challenges. For instance, fire brigades and ambulance teams can optimize their task allocation decisions by estimating, respectively, the danger of fire ignition in different buildings (discrete state—classification) and the chance of rescuing trapped civilians alive (continues state—regression).

The implementation of state-of-the-art AI and ML algorithms, their training, and their integration into the Rescue Agent Simulation competition code can be quite overwhelming for newcomer teams. Hence we propose that competition teams take advantage of existing and well-established AI and ML tools to develop their competition code. Here, we demonstrate¹

• 1. how to use and add-on packages, such as the , to analyze and model disaster scenario data using both interactive design tools (GUIs) and programming code. The analysis and modeling provide support to the development of more elaborate data-driven rescue decision-making algorithms (see Sect. 2).

• 2. how state-of-the-art ML algorithms can be directly incorporated into the Agent Development Framework (ADF) [13] using the Engine API for Java (see Sect. 3).

2 Interactive Approach

and the can be used in an interactive mode to analyze disaster scenario data and create models that agents can use to base their decisions during the unfolding of these disaster scenarios.

2.1 Unsupervised Methods

Unsupervised machine learning methods can be used to analyze and model disaster scenario data. In the Rescue Agent Simulation competition, clustering algorithms are interesting for agents to partition maps into sectors and evenly distribute the search and rescue workload among them [9, 10]. implements several clustering algorithms, such as k-means [6], k-medoids [4], hierarchical clustering [5], Gaussian mixture models [7], and hidden Markov models [2].

Fig. 2.

Partitioning of the buildings in the Paris map using k-means clustering algorithm. (Color figure online)

We can, for instance, use the interactive mode to assess which of these clustering algorithms provides a more evenly distributed number of buildings per sector for a specific city map. This assessment first requires that all (x, y) coordinates of buildings in the city map to be exported into a text file, which can be accomplished including some Java code into precompute phase of the agents code (see AbstractSimpleAgent.java lines 79–106). Next, these coordinates are imported into a matrix in using the textscan command and subsequently partitioned using one of the clustering algorithms available.

Figure 2 shows the buildings from the Paris map partitioned using the k-means clustering algorithm. The different shapes and colors represent the association of each building to a specific sector. This partitioning were created with the [indices,centroids] = kmeans([x,y], 5) command, and the plot generated with the gscatter(x,y,indices) command (see importBuildingsData.m and building_cluster.m scripts).

2.2 Supervised Methods

Supervised machine learning methods can be used to learn associations between variables (part of a causal model of the world). Estimates of variables’ value of a world model can be done with discrete states (classification) or continuous states (regression). In the Rescue Agent Simulation, competition teams can use these methods to assess their strategy and the most relevant predictors, for instance, for ambulance teams to estimate the chance that trapped civilians have to survive to a rescue operation by predicting their remaining health points (HP) at the end of a scenario simulation.

To demonstrate the use of to evaluate the strategy of a simple agent team, we collected several metrics from multiple runs in multiple scenarios of this agent team and assessed them using available classification and regression supervised learning algorithms in . The metrics colleted were (see matlab.generator.simple.agent.ambulance.SimpleAmbulanceTeam.java lines 389–420): start and end time of the rescue operation (sTime and eTime), initial and final Euclidean distance to the nearest refuge (sDist and eDist), initial and final HP (sHP and eHP), initial and final damage level (sDamage and eDamage), and initial buriedness (sBuriedness).

In , data can be preprocessed with dimensionality reduction methods like principal component analysis and singular value decomposition followed by linear or non-linear regression methods. The results can be visualized with ensembles like random forests, boosted and bagged regression trees. To learn those ensembles several optimization algorithms like AdaBoost and TotalBoost are available. We evaluated the accuracy of different combinations of predictor metrics and concluded that only the metrics with values of the beginning of the rescue operation were relevant to the prediction accuracy (i.e., sTime, sDist, sHP, and sDamage).

To use classification, we discretized the eHP according to the ranges: 0 Dead, 1–3000 Critical, 3001–7000 Injured, and 7001–10000 Stable. Then we trained different classification algorithms in using this data and the most accurate classification was obtained using the Weighted K-nearest neighbors (KNN).

Fig. 3.

Classification Learner app showing predictions of the injury class of the civilians at the end of the scenario using Weighted K-Nearest Neighbors. Left panel shows different assessed classifying algorithms and their respective accuracy. Middle panel shows a scatter plot showing the relationship between the initial distance to refuge (sDist) and the time the rescue initiated (sTime) with the model predictions and their correctness. Right panel shows the number of predicted versus true (or correct) classification of rescue civilians. The diagonal (green) shows the number of correct classifications, while all other cells represent the number of misclassifications, how they were classified versus the correct classification. (Color figure online)

Fig. 4.

Regression Learner app showing predictions of the chance to survive (remaining HP) of trapped civilians. Left panel shows different assessed regression algorithms and their respective root mean square error. Right panel shows the prediction versus true HP value of the trapped civilians at the end of a simulation.

Figure 3 shows the Weighted KNN classification used to predict if a civilian would be dead, in a critical state, injured or in a stable state at the end of a scenario simulation. This classification predicts correctly 78.9% of the civilians’ state. Notice, however, that most of the wrong detections (i.e., sum of the numbers in the red cells) are above the diagonal green cells in the right panel of Fig. 3 meaning that this trained classifier predicts a civilian in a less severe state than the civilian really will be. For instance, there are 9 cases in which the civilian will die and the classifier predicted it as injured.

We applied regression methods to the same data without discretazing the eHP and trained different regression algorithms in . Figure 4 shows an ensemble fit into a bagged tree model with the estimate of the remaining health points (HP) at the end of the simulation scenario with a root mean square error (RMSE) between the predicted and true HP values equals 1167.5 (and normalized RMSE equals 0.1228).

2.3 Path Planning

If an agent wants to move to a specific location to perform a task, a path plan to that location has to be defined. Two possible approaches to tackle this problem are (1) to use path planning algorithms from the graph and network algorithms² or (2) to use graph-routines from Peter Corke’s Robotics Toolbox [3].

Fig. 5.

The small Test map of the RoboCup Rescue Agent Simulation competition in as topological (left) and metrical graph (right).

First, however, all the roads of a city map need to be converted to a graph in format. The nodes of the graph are identified by the roads ID and they also store the actual (x, y) location of the road to facilitate the visualization of the results (see Fig. 5). The Java code to generate such a graph is called during the precompute phase of the Rescue Agent Simulation simulation, and its pseudo-code is:

Once created, the graph in can be queried, for instance to get the shortest path between two nodes. This can be done by calling a script which contains the function short_path = getPath(from,targets), that loads the graph G, calls the method [TR,D]=shortestpathtree and sorts the resulting paths TR based on the distance D. It is possible to specify in the algorithm to use (Breadth-first or Dijkstra). It is also possible to use A*, which is available in Peter Corke’s robotics toolbox [3]. The code of this algorithm is open source and well documented making it possible to modify the A* algorithm to Dijkstra’s algorithm (by removing the heuristics) or breadth-first (by not sorting the frontier on distance so far). The only thing needed is a script to translate from native graph-format to Peter Corke’s Pgraph-format. For smaller competition maps like Kobe this can be done in 13 s (measurement with a computer with a Intel Core i7-8550U processor), for larger maps like Paris 22 s are needed for this conversion (see Fig. 6). Both are fast enough for the precompute phase of the competition.

Fig. 6.

Maps of Kobe (left) and Paris (right) in Peter Corke’s Pgraph-format.

An advantage of this approach compared to the path-planning methods typically applied by the Rescue Agent Simulation competition teams is that each agent can load this a priori map and modify the edges based on the blockades observed and/or communicated. This information can even be updated when police force agents clear part of the road.

2.4 ROS Interface

A challenge in the Virtual Robot competition is that whenever an agent reaches a building, it has to enter that building [11]. The Robotics System Toolbox allows to directly control robots and realistic simulation via the Robotics Operating System (ROS) interface, as demonstrated in the Future of RoboCup Rescue workshop [14] and the RoboCup@Home Education workshop³.

Another challenge in this competition is the detection of buried victims from camera images. In the same workshops, victims detection has been demonstrated using the deep learning capabilities, a combination of the Neural Network Toolbox, Parallel Computing Toolbox, GPU Coder, and Computer Vision System Toolbox. Notice that these toolboxes run models deployed to GPU faster than TensorFlow or Caffe, which is a highly desirable for robotic applications⁴.

3 ADF Integration

In addition to using the models and algorithms in interactive mode, they can also be integrated into the Agent Development Framework (ADF) [13] to run during the simulation execution. ADF is the mandatory agent architecture for all competition teams participating in the Rescue Agent Simulation competition. This agent architecture is composed of several, highly specialized modules responsible for different data processing and decision-making tasks, such as clustering, path planning and task allocation.

The integration of models into the ADF is based on the Engine API for Java^®5, which enables Java programs via MatlabEngine class to interact with synchronously (startMatlab method) or asynchronously (startMatlabAsync method). In addition to start , there is also a possibility to connect synchronously (connectMatlab method) or asynchronously (connectMatlabAsync method) to an existing shared instance. To share a instance, enter the command matlab.engine.shareEngine in the command window. Once connected, it is possible to evaluate a function with arguments (feval and fevalAsync functions) or evaluate a expression as a string (eval and evalAsync functions). Additionally, it is possible to interact with the workspace by getting (getVariable and getVariableAsync functions) or setting (setVariable and setVariableAsync functions) variables. Once finished the interaction, disconnect from the current session using disconnect, quit, or close functions.

In Sect. 3.1 we show how to integrate the k-means clustering into rescue agents, and in Sect. 3.2 how ambulance team agents can use a trained classifier to decide which trapped civilian has a better chance of surviving a rescue operation.

3.1 Clustering Integration

Currently, competition teams need to implement their own version of standard artificial intelligence algorithms from scratch to solve common tasks, such as k-means clustering. However, provides more diverse and robust implementations of these standard algorithms that teams may benefit of to prioritize the development of high-level strategies.

Although diverse and robust, the time constraint imposed on rescue agents demands a more elaborate assessment of the efficiency of the algorithms integrated to the ADF. Here, we have assessed the performance of the k-means clustering algorithm implemented in the Sample ADF using pure Java and in measured in a computer with Intel Core i7 6700HQ 2.6 GHz (8 cores) processor and 16 GB RAM using Arch Linux, Oracle Java JDK 8 and R2017b.

Figure 7 shows the result of this assessment in which the k-means clustering algorithm executes in less time than the Sample ADF Java implementation for all 83 agents. There was significant difference on the execution average time for the ($6,095.87 \pm 1,178.51$ ms) and the Java ($8,783.36 \pm 1,188.22$ ms) implementations; t(164) = 14.63, p < 0.05. Hence, we can conclude that using k-means clustering reduces the effort and maintenance, and increases the performance of the agent teams.

Fig. 7.

Performance of the k-means implementation in Java and in performed in sequence by 83 agents during the initialization stage of the execution phase of the scenario simulation for the Kobe map.

The k-means clustering can be integrated into the ADF and executed in the precompute phase or execution phase of the scenario simulation. Here, we show how to integrate the k-means clustering algorithm in the agents’ initialization stage of the execution phase. The Java code for such integration is

This code connects the agents to and prepares entities data for clustering (i.e., the x and y entity location). In the initialization, agents are executed in sequence avoiding concurrent connections. Next, the function kmeans is evaluated using the feval method with several parameters: the dimension of the k-means output (set to 2), the number of clusters (set to 10), the distance metric (set to cityblock), and the maximum number of interaction (set to 100). Once executed, the feval method returns an object array with the indices in the position 0 and the centers in the position 1 that are cast to their respective data types. Finally, the engine is closed, and the indices and the centers can be used to assign agents to specific partitions of the map.

The integration of the clustering algorithm requires only a single call per agent to the engine as its results are stored and used in the remainder of the simulation run. Path planning algorithms, for example, are affected by changes in the environment, and would require reprocessing to account for changes. Because of this characteristic, the path planning algorithm would need to be called every time the agent has to calculate a path in the execution phase of the simulation run, even though the first execution can be performed during the precompute phase. Please see Sect. 2.3 to further details about the path planning.

3.2 Classifier Integration

Ambulance teams can also benefit of to optimize their rescue operations by predicting more accurately the chance of trapped civilians to survive a rescue operation. First, however, it is necessary to train a classifier with data collected from earlier runs. This training is performed using the Classification Learner app as described in Sect. 2.2.

We have trained a classifier using the data from rescued civilians collected from several simulation executions of the Paris map using a simple rescue team. The data collected was the time (sTime), the distance to the nearest refuge (sDist), the civilian HP (sHP) and the damage (sDamage) at the start of the rescue operation and the HP of the civilian (eHP) at the end of the rescue operation. We discretized the final HP (eHP) according to the ranges: 0 Dead, 1–3000 Critical, 3001–7000 Injured, and 7001–10000 Stable using the code

The trained classifier model (targetSelectorModel) is exported using the Export Model - Export Compact Model feature and validated against a validation dataset with the code

The training and validation steps comprise an iterative process whose cycle should be repeated until the validation accuracy is satisfactory. Then, the exported model can be saved as a file (targetSelectorModel.mat) and invoked in the function

The predictions function can then be called inside the calc method of the HumanDetector class for the ambulance team agents in the ADF using the code

This code executes the function selectTargets using data about a specific victim and returns a prediction about the state of the victim at the end of the rescue operation coded as 0 Dead, 1 Critical, 2 Injured, and 3 Stable. The ambulance team can then combine this prediction with several other information about other victims to determine which victim is worth rescuing first. Possible strategies to use this classification includes (1) classify all known victims, (2) discard the predicted dead, and (3)

• a. select one randomly among them

• b. select the closest one

• c. select the closest one that is predicted Critical

Notice that we use MatlabEngine.findMatlab() and connectMatlab() methods instead of MatlabEngine.startMatlab(). This requires that a session is running and shared to the code to work. To share a session, open , enter the command matlab.engine.shareEngine in its command window, and leave it open during the execution of the simulation.

4 Conclusion

This paper describes the possible uses of existing artificial intelligence (AI) and machine learning (ML) tools to analyze and model disaster scenario data as well as the integration of these tools to the competition code. The examples provided tackle common challenges of the Rescue Agent Simulation competition in which AI and ML tools suit. The approach, however, is extensible to any other algorithm available in or any other tool that provides an interface in Java. For instance, this approach can be extended to integrate deep learning, state machines, and graph node refining algorithms, which may increase the scientific outcomes of the Rescue Simulation League as

• (1) teams may focus on high-level strategies to solve rescue challenges and

• (2) will provide a performance benchmark against which teams can show their improvements.