1 Introduction

Intelligent devices continuously find their way into our homes. As such, we envision also personal service robots to solve mobile manipulation tasks within our future smart homes. In this paper, we consider the problem of localizing and retrieving items with a personal robot. Instead of tedious visual search, we propose to attach low-cost Bluetooth Low Energy tags to objects. The robot operates in an environment that is instrumented with static wireless devices to estimate the position of the tags. The coarse position estimate hints to typical placement locations in the environment, from which the robot can retrieve the objects.

Imagine a scenario in which you may have displaced your wallet. You command your personal service robot to find and fetch it. A Bluetooth tag included in the wallet acts as a beacon which is detected by static Bluetooth receiving devices in the surrounding. The receivers measure the signal strength of the Bluetooth tag. Using knowledge about the position of the receivers in the environment, the position of the tag can be estimated through lateration. The robot assesses the signal strengths and estimates the coarse position of the tag. It searches for the wallet close to the detected position, grasps it, and happily brings it to you.

Technically, our low-cost object localization system consists of small, cheap Bluetooth Low Energy (BLE) tags and at least four receivers. The receivers are built from Raspberry Pi computers. We propose to estimate the position of the tags through lateration of received signal strengths. We assess the localization performance of our system in terms of accuracy and timing. We also report on the successful demonstration of our approach at the RoboCup German Open 2014 competition in Magdeburg.

2 Related Work

Since GPS is not available indoors, other approaches need to be taken for localizing objects in indoor environments. In recent years, some approaches have been developed that find wireless devices.

A simple approach is to use proximity detection [1], e.g. to detect RFID tags close to one of several RFID readers that have been placed in an environment. The location of the object is then simply associated with the position of the reader that best detected the tag. Approaches that utilize less instrumentation of the environment are methods based on propagation principles [2–4] and fingerprints [5].

Propagation-based approaches use distance and angle information estimated from the electrodynamical properties of the wireless signals. Most common is the use of received signal strength (RSS, e.g., [2]) which is related to the distance between the signal emitter and the receiver. Other approaches are received signal phase methods (RSP, e.g., [3]), which measure the phase-shift of signals, or time of arrival methods (TOA, e.g. [4]) that directly measure the propagation time of the signal. The common principle behind these methods is that they physically explain measured properties of the signal, and determine a continuous estimate of the position of the wireless emitter. All propagation-based methods make model assumptions that need to be fit to actual data acquired for the specific wireless setup and environment.

Fingerprinting methods (e.g., [5]), on the other hand, do not make such model assumptions, but directly represent a map of typical signal measurements, when the emitter is at specific locations. To find the emitter for new measurements, the location with the best matching example measurements is retrieved. The advantage of fingerprinting methods is that they do not make strong model assumptions and can work even in difficult environments with e.g. shadowing or multi-path effects. However, they are tedious to train—signal strengths need to be measured from the emitter placed at a dense grid of locations in the environment—and require adaptation to every change of wireless setup and environment.

Using Bluetooth to localize objects [5, 6] has a number of advantages over using other wireless modalities. Bluetooth is a low-cost, low-energy technology. Many mobile devices and computers are nowadays equipped with Bluetooth, which makes Bluetooth solutions widely applicable without a special augmentation of the environment with e.g. RFID tags [7]. In our work, we deploy Bluetooth devices within a smart environment. For instance, small active Bluetooth tags can be attached to objects such as wallets, keys, or other items that users frequently displace. A cognitive service robot can then localize objects of interest, and retrieve them. While a robot could search visually for the object [8], using wireless sensing simplifies search significantly, due to the large field-of-view and the unique identification of Bluetooth devices.

3 System Description

3.1 Cognitive Service Robot Cosero

Fig. 1.
figure 1

The cognitive service robot Cosero. Left: Cosero opens a bottle during the final of the RoboCup@Home German Open competition 2014 in Magdeburg. Right: Cosero pushes a chair to its location during the Demo Challenge of RoboCup 2013, Eindhoven.

Full size image

We designed our cognitive service robot Cosero, shown in Fig. 1, to cover a wide range of tasks in domestic indoor environments [9, 10]. It has been equipped with a flexible torso and two anthropomorphic arms that provide human-like reach. A linear actuator moves the whole upper body up and down, allowing the robot to grasp objects from a wide range of heights—even from the floor. Cosero’s anthropomorphic upper body is mounted on a mobile base with narrow footprint and omnidirectional driving capabilities. By this, the robot can maneuver through narrow passages that are typically found in indoor environments, and it is not limited in its mobile manipulation capabilities by holonomic constraints. The human-like appearance of our robot also supports intuitive interaction with human users.

For perceiving its environment, we equipped the robot with diverse sensors. Multiple laser scanners on the ground, on top of the mobile base, and in the torso measure objects, persons, or obstacles for navigation purposes. We use a Microsoft Kinect RGB-D camera in the head to perceive tabletop objects and persons.

Cosero navigates in indoor environments on horizontal surfaces. Hence, we use the 2D laser scanner on the mobile base as the main sensor for mapping and localization. It navigates to goal poses by planning obstacle-free paths in the environment map, extracting waypoints, and following them. Obstacle-free local driving commands are derived from paths that are planned towards the next waypoint in a local collision map that includes 3D measurements from the lasers in the robot.

After approaching the estimated coarse location of the object, the robot detects objects in its vicinity either by segmenting 2D scans of its torso laser or the depth images of its head-mounted RGB-D camera. It selects a grasp on the detected object and executes a parametrized motion primitive to grasp it [11].

3.2 Bluetooth Localization

Bluetooth Low Energy (BLE) was chosen as the technology for wireless communication because it offers several advantages. Firstly, its low transmit power allows for very small tags with long lifetimes, which is crucial in our application. If the tag batteries would have to be replaced every week, this would be an additional burden on the user, which we want to avoid.

Secondly, BLE allows direct signal strength measurement in a standardized way. Bluetooth adapters report an RSSI (Received Signal Strength Indication) for each scan answer they receive from a Bluetooth tag. This RSSI value is directly related to the signal strength and, hence, to the distance of the tag from the receiver.

Thirdly, Bluetooth components are available off-the-shelf at a very low price and in small form factors, e.g. in coin size (see Fig. 2). Finally, the system can also be used to track other Bluetooth devices, such as most smart phones, tablets and other mobile equipment.

Our system is set up as follows: We employ a number of receivers, which are mounted at fixed locations in the apartment. The locations are chosen such that the distance between receivers is large. The receivers consist of a small single board computer (Raspberry Pi) and a Bluetooth USB receiver. The Raspberry Pi already has an Ethernet connector for relaying the data. If wireless communication is desired, a USB WiFi adapter can be added. All in all, we can buy a receiver set for under 50 US$.

We simplify the problem with the assumption that receivers and tag lie in a horizontal plane. Otherwise, we would have to consider the optimization problem in 3D. Because our achieved accuracy lies in the range of 1 m, this simplification does not impede successful application of the method in reasonable room sizes.

The receivers send the detected signals and their strength to a central location estimation unit (LEU). This LEU could be, for instance, the robot, or a designated receiver. The LEU then estimates tag positions using the lateration approach described below and forwards the result to the robot or other data consumers.

Fig. 2.
figure 2

Used Bluetooth tags (left: Estimote Inc., right: StickNFind) with coin for size comparison.

Full size image

3.3 Handheld Teleoperation Interface

To extend the possibilities of human-robot interaction, we also developed a handheld teleoperation interface on a tablet computer, which can be used to remotely interact with the robot. The general system, which is targeted at immobile users, is described in [12].

The interface offers three levels of teleoperation: On the body level, the user can directly operate parts of the robot such as the omnidirectional drive, gaze direction and the end-effectors. The skill level can be used to parametrize and trigger robot skills, e.g. by setting navigation goals or commanding objects to be grasped. Finally, the operator can configure autonomous high-level behaviors that sequence skills on the task level.

Situational awareness is gained through several visualizations of the robot perceptions. We also integrated the localization system into the user interface. The tag locations are always displayed to the user within a map of the environment (see Fig. 3). Appropriate actions (e.g. combining a navigation and grasping command to retrieve the object) can then be triggered using the already present user interface components for command input.

Fig. 3.
figure 3

Tablet user interface showing the tag locations on a map of the RoboCup@Home arena at German Open 2014 in Magdeburg.

Full size image

4 From Signal Strength to Relative Distance

In the LEU, the RSSI readings of all BLE receivers are processed. In a first step, we filter the measured signal strength by finding the highest three signal strengths observed in a time window of 10 s. We average those three highest measurements to obtain a robust signal strength estimate. The rationale behind this approach is that while signal strength is easily underestimated (e.g., because of temporal occlusions or multi-path echoes), it is difficult to overestimate it.

In our observations, the measured signal strength does not directly correlate with distance. There are other factors, such as occlusions of the line-of-sight between tag and receiver, receiver angle dependencies, and so forth. Because of this, we estimate a relative distance from the signal strength in a first step. This relative distance is assumed to be proportional to the real distance, with the same proportionality factor over all receivers. We make the assumption here that a loss of signal strength is always isotropic, i.e. experienced by all receivers at the same degree. This assumption is correct in many cases, such as objects inside a container.

For an isotropic sender of electromagnetic waves, the output power \(P\) perceived at a distance \(r\) follows the proportionality:

$$\begin{aligned} P \sim \frac{1}{r^2} \Leftrightarrow r \sim \frac{1}{\sqrt{P}}, \end{aligned}$$
(1)

which we can turn into an equality for the estimated relative distance \(D\):

$$\begin{aligned} D := r \cdot k = \frac{1}{\sqrt{P}}. \end{aligned}$$
(2)

The linear relationship between distance \(r\) and estimated relative distance \(D\) was experimentally verified in a lab experiment (see Fig. 4). A Bluetooth tag was placed sequentially on a grid with 1 m resolution in our lab room (7 \(\times \) 5 m). Four receivers mounted in the corners of the room were used to obtain signal strength data for each tag location. Furthermore, ground truth distance of the tag to the receivers was measured.

Fig. 4.
figure 4

Relationship of estimated relative distance D to real distance. The tag was placed at varying positions and ground truth distance to the receivers was measured. The red line shows the linear trend observed (Color figure online).

Full size image

5 Position Estimation

Fig. 5.
figure 5

Solutions for the lateration problem found by the optimization method. Green rectangles are the Bluetooth receivers, the cyan rectangle denotes the estimated tag position. The black circles show the relative distance \(D\) scaled by the found \(k\) parameter. (a) solution for an ideal problem (single intersection point); (b) solution for real-world data (Color figure online).

Full size image
Table 1. Deviation from ground truth and standard deviation of measurements in m from Fig. 6
Full size table
Table 2. Deviation from ground truth and standard deviation of windowed means in m from Fig. 7
Full size table
Fig. 6.
figure 6

Evaluation of position estimation (raw optimization results). Items with same color belong to one tag position (Color figure online).

Full size image
Fig. 7.
figure 7

Evaluation of position estimation (windowed means). Items with same color belong to one tag position (Color figure online).

Full size image

For position estimation, the over-constrained lateration problem needs to be solved. The receivers report relative distances \(D_i\) to the tag. With perfect data, a single coefficient \(k\) would exist, which would make the circles around each receiver with radius \(\frac{D_i}{k}\) intersect in a single point (see Fig. 5(a)). Because the data is not perfect, we treat the problem as an optimization problem. We simultaneously optimize for the tag position \(x\) and coefficient \(k\).

The minimized cost function is specified as:

$$\begin{aligned} f(x, k) := \sqrt{ \sum _{i=1}^n \left( \frac{|| x - p_i ||_2 - k \cdot D_i}{\ln {(1 + D_i)}} \right) ^2 }, \end{aligned}$$
(3)

where \(D_i\) denotes the relative distance estimated by receiver \(i\), and \(p_i\) is the position of receiver \(i\). We divide by \(\ln (1+{D_i})\) to give more weight to short-range measurements, as they are more precise. The numerical error present in the measurements scales logarithmically with estimated distance, since the measurement is retrieved as an integer decibel number from the Bluetooth receivers.

For minimization, we use the implementation of the Simplex algorithm of Nelder and Mead [13] available in the GNU scientific library. We initialize the simplex to the center of the environment, in order to find a reasonable minimum of the cost function. The resulting position estimate is still noisy, so we smooth it with a windowed mean over 30 s.

6 Results

6.1 Evaluation

We evaluated the accuracy and timing of our localization method in a lab experiment. The tag was sequentially placed on six positions spaced evenly in a 5 \(\times \) 7 m room. The raw position estimates obtained without temporal smoothing are shown in Fig. 6 and their accuracy can be seen in Table 1. The estimates with 30 s windowed mean smoothing can be seen in Fig. 7 and Table 2. In almost all cases, the deviation between the ground truth tag position and the estimated position is below 1 m. This is more than sufficient for our application, since our robot can locate objects in close vicinity using its other sensors.

The StickNFind tags send out measurable packets every 1000 ms, which allows even for tracking of slowly moving objects.

6.2 Public Demonstration

Fig. 8.
figure 8

Cosero localizes and retrieves medicine using our approach during the Demo Challenge of the RoboCup@Home competition at German Open 2014 in Magdeburg.

Full size image

We demonstrated our object localization approach publicly during the Demo Challenge of the RoboCup@Home league at the 2014 RoboCup German Open competition in Magdeburg (see Fig. 8). A user asked Cosero to retrieve his medicine that he could not find. The medicine had been placed at one of two locations which was chosen by a jury member. The jury consisted of members of the executive and technical committees of the league. A Bluetooth tag had been attached to the medicine, which a LEU localized using four RaspberryPi receivers that were mounted in the corners of the room. The LEU coarsely localized the object at the right spot, and Cosero drove to the table close to its localization. The robot searched on the table, grasped the medicine, and brought it to the user. In a second run, the robot localized and retrieved the object from the other location. In the meantime, the jury members could watch the estimated position of the tag and the progress of Cosero’s task execution on a mobile tablet interface. The demonstration was well received by the jury who awarded the highest score in this test to our team.

7 Conclusions

We proposed a simple and affordable system for object localization and retrieval with a personal service robot using Bluetooth Low Energy devices. Our method places static receiver devices in a smart home environment to measure the signal strength of active Bluetooth tags that can be attached to objects. The signal strength is interpreted as relative distance, which we use to laterate the position of the tag. In addition to the location of the tag, we also estimate the proportionality factor between the signal strength and the distance to the tag. In experiments, we evaluate accuracy and timing of our localization approach. We also report on the successful demonstration of our integrated object localization and retrieval approach at RoboCup German Open 2014.

Ideas for future work include the investigation of static reference tags for system calibration. It might also be possible to combine lateration with fingerprints or local models in order to improve localization accuracy. Finally, it could be worth to equip the robot itself with Bluetooth receivers, which would make signal strength-based taxis towards tagged objects in its vicinity possible.