1 Introduction

The RoboCup@Home competition [1] aims at bringing robotic platforms to use in realistic domestic environments. In contrast to other leagues like soccer – which predefine and standardize the playing field – here robots need to deal with different apartment layouts, changing decorations, unknown sites, unstructured public spaces, and interfering or cooperating humans who are only very briefly – or not at all – instructed how to interact with the robot. The set of benchmarking tasks is adapted or newly defined each year. These typically require multiple capabilities, like navigation and mapping, person recognition and tracking, speech understanding and simple dialogues, object recognition and manipulation. The integration of these capabilities towards a coherent system behavior that deals with any kind of exception or interference during a task is one of the major challenges of RoboCup@Home. The competition is organized into different stages. Within the first stage, tests focus on a small set of capabilities (e.g. person following and guiding or object recognition and manipulation) scoring the best two tries out of three. The stage is finalized by an integration challenge (GPSR – General Purpose Service Robot) where robots have no predefined task, but need to autonomously sequence a task given by speech. The best 50% of the teams proceed to the second stage. Here, robots are tested in an enhanced and longer version of GPSR (EE-GPSR), in a real restaurant as a waiter, and in an individual open performance (Open Challenge). The final is an extended open challenge that is judged by an internal and external jury.

The Team of Bielefeld (ToBI) was founded in 2009 and successfully participated in the RoboCup German Open as well as the RoboCup World Cup from 2009 to 2015. In 2016, the team finally won the global competition and ended first in several of the individual tests (Navigation, Person Recognition, GPSR, EE-GPSR, Restaurant). There are multiple reasons for the performance gain in 2016. Some of these will be discussed in the following sections. Bielefeld University is involved in research on human-robot interaction since more than 20 years especially gaining experience in experimental studies with integrated robotic systems [2, 3]. An important lesson learned is that the reproducibility of robotic systems is critical to show the incremental progress – but that this is rarely achieved [4]. This applies to experimentation in robotics as well as to RoboCup. A Technical Description Paper (e.g. [5]) – as typically submitted to RoboCup competitions – is by far not sufficient to describe or even reproduce a robotic system with all its artifacts. The introduction of a systematic approach towards reproducible robotic experiments [6] has been turned out as a key factor to maximally stabilize basic capabilities like, e.g., navigation or person following. Together with appropriate simulation engines [7] it paves the way to an automated testing of complete RoboCup@Home tasks.

In the Open Challenge and the Final, we introduced a multi-robot collaboration scenario that combines small mobile sensor devices with human-sized service robots showing the scalability of the communication [8] and behavior [9] framework. This is an essential contribution to the development of the @Home league because our future homes will be equipped with numerous intelligent devices. A service robot will act in a more human-aware manner if it already knows where people are instead of searching for them. We show how this can be realized in a very flexible manner without completely relying on fragile communication channels.

2 Robot Platforms

In 2016, ToBIparticipated in RoboCup@Home with the two service robots Biron and Floka. Those were assisted by multiple instances of the smaller AMiRo as an extended mobile sensor platform. Figure 1 gives on overview of the three mentioned platforms. While we focus on navigation and HRI with Biron, the body structure of Floka allows advanced bi-manual manipulation.

Fig. 1.
figure 1

Robotic platforms of ToBI. The overall height of Biron is \({\approx }140\, \text {cm}\). The Floka platform has an adjustable height between \({\approx }160\,\text {cm}\) and \({\approx }200\,\text {cm}\). The AMiRo has a diameter of 10 cm.

Full size image

The robot platform Biron (cf. Fig. 1(a)) is based on the research platform GuiaBot by adept/mobilerobots customized and equipped with sensors that allow analysis of the current situation. The Biron platform has been continuously developed since 2001 and has been used in RoboCup@Home since 2009.

The robot base is a PatrolBot which is maneuverable with \(\approx \)1.7 \(\text {ms}^{-1}\) maximum linear velocity and \({>}5\,\text {rad s}^{-1}\) rotational velocity. The drive is a two-wheel differential drive with two passive rear casters for balance. Inside the base there are two laser range finders with a coverage of 6.28 rad around the robot with a scanning height of \(\approx \) \(30 \text {cm}\) above the floor (SICK LMS in the front + Hokuyo UBG-04LX in the back). The upper part of the robot also houses a touch screen (\(24 \text {cm} \times 18 \text {cm}\)) as well as the system speaker. In contrast to most other PatrolBots, Biron does not accommodate an internal computer. Instead, two laptops are attached to platform. These are equipped with Intel Core i7-4810MQ processors and 16 GB main memory are running a standard Ubuntu Linux. For person detection/recognition we use a full HD webcam of the type Logitech HD Pro Webcam C920. For object recognition we use a 24 Mpx DSLM camera (Sony Alpha \(\alpha \)6000). Two Asus Xtion PRO LIVE Sensors on top of the base provide RGBD data for the robot. As microphones two Sennheiser MKE 400 are installed front- and back-facing, supported by two AKG C 400 BL on the sides. Additionally, the robot is equipped with the Neuronics Katana 450 arm.

Our robot Floka (cf. Fig. 1(b)) is based on the Meka M1 Mobile Manipulator robotic-platform. It has been continuously enhanced within the ‘Cognitive Service Robotics Apartment as Ambient Host’ project to explore research questions related to human-robot interaction in smart-home environments [10, 11].

An omni-directional base with Holomni’s caster-wheels and a lift-controlled torso enable navigating in complex environments. In total, the robot has 37 DoF, which break down to joints. It has 7 per arm, 5 per hand, 2 for the head, 2 in the torso, and 9 joints actuate the base including the z-lift. The motors in the arms, torso and hands are Series Elastic Actuators (SEAs), which enable force sensing. The four-fingered underactuated anthropomorphic hands are attached to an ATI Industrial Automation Mini40 force/torque sensor. The sensor-head contains a Primesense Carmine 1.09 short-range RGBD camera and a Ximea MQ042CG-CM 4.2 Mpx Color CMOS Camera. Two AKG C 400 BL are attached to the shoulders. The base is equipped with a Hokuyo UTM-30LX scanning range finder on the front top and a SICK TiM571 integrated in its back. For processing it’s back houses a custom processing PC with Intel Core i7-6700 and 16 GB of RAM and Zotac ZBOX SN970 for NVIDIA CUDA. A third PC for real-time control of the actuators with Intel Core i5-3470S and 8 GB of RAM is integrated in the base. All the components controller boards are interfaced over Ethercat with the Meka M3 control framework. This framework is building up on RTAI for Linux to enable a 1 kHz control-loop.

The AMiRo as used in RoboCup@Home is a two wheeled robot with a physical cylindrical shape [12]. It extends and enhances the capabilities of Biron and Floka in many tasks within the Open Challenge and Final. Commonly, multiple AMiRos are applied in conjunction to build a multi-robotic setup which is interconnected via Wi-Fi. AMiRo (cf. Fig. 1(c)) was developed at Bielefeld University with the main objective of research and education. It consists of a set of stackable electronic modules for sensor processing, actuator control, and behavior coordination that fully utilize currently available electronics technology for the construction of mini-robots which are able to show rich autonomous behaviors.

Additionally, practically all common USB or serial device can be attached to extend its sensor and actor capabilities. To name some applied extensions for the RoboCup, the SICK TiM571 is used to perform online SLAM and the captured video is offered via a WebSocket based webserver. An ORBBEC Astra S RGBD camera is used for high precision table top detection and interaction.

3 System Architecture

Our service robots employ distributed systems with multiple clients sharing information over network. On these clients there are numerous software components written in different programming languages. Such heterogeneous systems require abstraction on several levels.

Fig. 2.
figure 2

System architecture of ToBI’s service robots (Color figure online)

Full size image

Figure 2 gives an overview of the multiple layers of abstraction in the cooperating robot systems. Each column represents one type of robot. The behavior level (blue) represents the highest level of abstraction for all robots. This can be skills or complex behaviors. The robot specific software (green) and hardware component interfaces (red) are unified with the BonSAI Sensor Actuator Abstraction Layer (yellow). Even skills from the small AMiRo can be seamlessly integrated into the behavior of the service robots.

Thus, software components can be easily exchanged without changing any behaviors. The system architecture also abstracts from the middleware which is handled on the component layer. The software dependency tree of the system is completely modeled in the description of a system distribution which consists of a collection of so called recipes [6]. In order to foster reproducibility/traceability and potential software (component) re-use of the ToBI system, we provide a full specification in our online catalog platformFootnote 1.

The catalog provides detailed information about the soft- and hardware system including all utilized software components, as well as the facility to execute live system tests and experiments remotely. In order to gain access to our remote experiment execution infrastructure please contact the authors.

3.1 Communication Architecture

The communication between software components is mainly based on the Robotic Service Bus (RSB) [8] providing an API which abstracts from different transports (e.g. in-process, socket, Spread) and integrates well with ROS-based nodes. Figure 3 shows the current middleware-oriented communication infrastructure. Numerous essential applications for Floka and Biron have been realized using RSB and ROS. Thus, we decided to run these two systems in conjunction locally on every robot. The message exchange between ROS nodes and RSB participants is handled by ROS4RSB applications, being message bridges living in both habitats. The only exception is the AMiRo due to its hardware constraints in processing power. Therefore, RSB has been chosen as the sole communication bus locally because of its more resource efficient handling of messages [8].

Facing diverse multi-robot setups, ToBIrelies on RSB over Wi-Fi for robot-robot communication. The decision has been made due to the Spread transport mechanism for group communication [13] which is one of the exchangeable transport mechanisms of RSB. Spread supports a rich fault model that includes process crashes and recoveries, and network partitions and merges. Thus, RSB outperforms ROS’s multi-master [14] approach in our use case, since ROS still uses a centralized approach for syncing robots which join the network as well as peer-to-peer socket connection between remote nodes. Therefore, the Wi-Fi communication channel is handled by RSB exclusively. Since the small channel bandwidth between the robots does not allow massive data exchange, only selected messages are forwarded via an RSB Bridge from the highly loaded internal communications to the external common bus.

Fig. 3.
figure 3

Intra- and inter-communication architecture of ToBI

Full size image

3.2 Reusable Behavior Modeling

For modeling the robot behavior in a flexible manner ToBI uses the BonSAI framework. It is a domain-specific library that builds up on the concept of sensors and actuators that allows the linking of perception to action [15]. These are organized into robot skills that represent the basic unit of the robot’s actions. These basic units are combined into behaviors with certain strategies for an informed decision making.

To support the easy construction of more complex robot behavior we have improved the control level abstraction of the framework. BonSAI supports modeling of the control-flow using State Chart XML which can be combined, hierarchically. The coordination engine serves as a sequencer for the overall system by executing BonSAI skills to construct the desired robot behavior. Skills can be triggered asynchronously and communicate via events with the coordination engine. This also allows a flexible delegation to other robotic platforms. The BonSAI framework has been released under an open source license.

4 Selected Robot Skills

The performance in the Finals targeted a futuristic apartment scenario where a heterogeneous set of robots is available for diverse tasks (cf. Fig. 4(a)). All robots shared their position and state via a common RSB connection using Wi-Fi so that an additional PC was able to visualize the current common inner world model representation of all robots (cf. Fig. 4(b)).

Biron acted as a common domestic worker performing GPSR, Floka as a novel robot with sophisticated grasping skills which were going to be presented, and AMiRos equipped with a LIDAR being the apartment’s watchdogs. One human host resided in the apartment’s living room with Floka, Biron, and one AMiRo. Additionally, one visiting guest entered the apartment through a long hall way (\({\approx }8 \text {m}\)) where another two AMiRos were located.

Fig. 4.
figure 4

Final task scenario showing Biron, Floka, and AMiRo

Full size image

4.1 Multi-robot Interaction

Multi-robot interaction comes into play if parallel but coordinated tasks need to be performed or an event requires an instant reaction at different places. This is shown in the first part of the Final. When a guest appears at the door, the AMiRos and Biron robots collectively solve the task of accompanying the guest on the way to their owner. AMiRos waited in the hall way, acting as a mobile tag while attaching to traversing legs. The first AMiRo followed the guest for a certain distance until it handed over the guest seamlessly to the next one. Positioning information were transferred to Biron, so that it was finally able to take over the guest and introduce him to the owner.

The crucial part of applying multi-robotics is the provision of a fail-safe communication channel and the task allocation. Intercommunication between the robots via Wi-Fi is inherently fragile due to instability in the RoboCup@Home arena caused by noise or coverage, joining and detaching robots, or hardware failures. These link layer interferences cannot be completely eliminated by tweaking the network drivers, e.g. maximizing channel utilization, minimizing connection abortion, or reducing reconnection delays. Thus, we solved the remaining connection abortions on the application layer by our proposed communication (cf. Fig. 3) and behavior architecture (cf. Fig. 2) to keep the multi-robot setup running. Facing the task allocation, we decided to design every robot fully autonomous with state machines loosely linked to each other. Thereby, information exchange is reduced to percepts or symbols representing the current state or requested task allocations.

On every robot runs a stand-alone behavior based on a state-machine representing the robot’s capabilities relevant for the task. On the basis of societal agents [16], BonSAI realizes the connection and abstraction layer to allow transparent access to the sensorimotor features, skills, or behaviors of its own or every other remotely accessible feature. This approach makes no difference between the methods used to be triggered by intra- or inter-robot behaviors. If communication breaks, the behaviors define fall-back strategies similar to those which may deal with blocked navigation goals or other unexpected events (Fig. 4).

4.2 Error Handling and Reporting in GPSR

The GPSR task is a challenge to foster the ability of robots to interpret a natural language command and solve the task by combining capabilities. We propose an approach that not only extracts the task from a given sentence but even allows the robot to handle unexpected situations and report by verbal feedback to the operator. Hierarchical grammars are used to cluster the task into eight main-categories based on the sentence’s predicate. For each category a set of verbs is defined directly in the grammar. Subcategories of actions are defined by type (person, object or location) and number of the objects in the sentence. Pronouns are replaced by object designators. All gathered information about the task is then stored in a memory as a sequence of action subcategories. Based on this subcategory, BonSAI behaviors are executed which directly trigger skills or re-usable behaviors of Stage 1 (Sect. 3.2). These also store information to memory about their success or failure. Thus, the behavior framework allows to trace back a task performance and verbally report about each subtask. ToBI was one of the first teams realizing such a generic approach within the EE-GPSR test.

Fig. 5.
figure 5

The grasping pipeline. On the left output of the CLAFU component is displayed. The RVIZ environment including Floka, the detected planar surface as well as fitted superquadrics are visible in green. (Color figure online)

Full size image

4.3 Flexible Grasping Considering Object Shapes

Our previous grasping pipeline involved a basic grasp generator producing many potential grasp poses (up to thousands) around a given center pose of the target object. The number of generated grasp poses was implicitly determined by an equidistant sampling around the view-aligned bounding box of the object’s point cloud, i.e. neither shape nor orientation of the object were considered. All potential grasps were fed into the MoveIt! [17] planning pipeline to check for feasibility. The first adequate grasp pose was finally selected for grasping. In order to improve the pipeline in speed as well as precision, a new grasp generator was adapted from work by Haschke [18]. The grasp generator is composed of two nodes, an object fitting node and a grasp generation node.

A preprocessed point cloud alongside additional information regarding the recognized type of object(s) is generated by the Classification Fusion (CLAFU) [19] component. The point cloud of objects newly encountered are further fitted by superquadrics. Considered shapes are boxes, spheres and cylinders. The grasp generator analytically generates a reduced set of grasp poses exploiting the shape information and object dimensions. For a box, grasps will be aligned to opposing faces; for cylinders and spheres, the arm will approach from the most comfortable direction and thus resolve the redundancy introduced by the shape’s symmetry. The grasps will be filtered for feasibility and collision with the table exploiting the object’s and hand’s bounding boxes. All remaining grasps are ranked according to several criteria, including preference for side vs. top grasp, comfort, and motion distance of the hand. Thus, only a reduced set of 1 to 10 grasps per object is fed into the MoveIt! [17] planning pipeline for a final feasibility check, determining a collision-free grasp trajectory.

Fig. 6.
figure 6

Accumulated scores from the pre-defined tests in RoboCup@Home. The numbers on the x-axis refer to tests, the \(X^*\)-tests have been won by the ToBI-Team.

Full size image

5 Analysis and Lessons Learned

In Fig. 6 the results of the 5 best teams in the predefined tests of RoboCup@Home 2016 are shown. ToBIachieved best performances in the Person Recognition, Navigation, GPSR, Restaurant, and EE-GPSR tests. For most of the required capabilities, available standard libraries were used. Thus, it were not the algorithmic parts in the components that made any difference. The reasons why teams fail in some of the challenges are multifaceted. In many cases, bugs in the code are a major cause. In other cases, unexpected interferences, external noise, and untested environmental conditions lead to a break-down. At the RoboCup site in Leipzig, our team experienced several of these but learned to deal with it. For example, the floor of the hall was very slippery for the wheels of the Floka robot; thus, we needed to change the drive strategy and odometry model of its omnidirectional base; the multi-robot coordination relied on a very fragile Wi-Fi on-site, so that the middleware was required to deal with partial communication break-downs (cf. Sect. 3.1). This year, we changed to a new people detection and tracking framework [20]. Based on this, a re-usable following behavior was implemented that also included strategies for re-initialization, if a person was lost. This BonSAI behavior is re-used in the Navigation, Following and Guiding, GPSR, and Restaurant tasks. Introducing new basic skills typically requires a high need for bug-fixing until all possible side effects have been explored. This emphasizes the importance of the development framework used within the competition. We exploited a dedicated toolchain for reproducible experimentation in robotics [21]. In the Cognitive Interaction Toolkit (CITK), there is a versioned description of any – incrementally developed – system distribution including all software and data dependencies which is automatically built on a continuous integration (CI) server. This allows to track down any system change that might have caused an error or repaired it. The CITK also provides an automated testing environment [7]. For RoboCup@Home, we implemented a complete person-following test utilizing a simulation environment based on Morse which is included in the CITK catalog entry referenced in Sect. 3. This test was automatically executed by the CI server as a replicated experiment which helped to stabilize the person detection and tracking skills as well as the following behavior.

Another important aspect is to design robot behaviors for the complete competition and not only for a single test. For example, furniture or persons sometimes blocked navigation goals in the apartment; thus, the robot needed to deal with these situations all the time – not only when mentioned in the rulebook. A general behavior for memorizing and reporting about these events even achieved additional points for the team in the EE-GPSR test. The reasoning about task performances will be a critical capability for future RoboCup@Home developments. This includes dealing with situations that require the delegation subtasks because a robot is stuck, needs help, or is too far away. Here concepts from multi-robotics come into play as already argued before.

6 Conclusion

In this paper, we have described the main features of the ToBI system which won the RoboCup@Home competition in 2016. This result has been achieved with a team mostly consisting of bachelor and master students that were completely new to RoboCup, with a newly introduced Floka robot, and with a Final performance that heavily relied on a fragile WiFi communication between multiple robots.

Nevertheless, the ToBI robots gave the most stable performance throughout the competition. New elements have been introduced like reporting on success and failure of tasks assigned to the robot in GSPR, re-entering into the arena through a closed door in the Navigation task, the flexible grasping considering object shapes in the Final together with a multi-robot cooperation where a person is handed over from one robot device to the next one. There are a couple of essential elements that made up the success of the team. First of all, we improved the robustness of the ToBI system on all levels, starting with a fail-prove communication framework, an automated testing procedure for new component skills, a re-usable behavior definition that is used and tested across multiple tests, and a development approach that aims at reproducible robotic experiments. Finally, there was an inherent team spirit which highly motivated each individual team member. All these aspects contributed to ToBI’s overall success.