Keywords

1 Introduction

In this paper, we report on a first exploratory study using IoT devices and related long-range (LoRa) wireless communication devices used in a marine setting, in collaboration with marine biologists. To the best of our knowledge, this is the first attempt to develop interactive systems that operate in the previously described setting, i.e., using low-cost IoT and LoRa technologies while providing marine biologists with a unique interface to explore coastal areas using remotely operated unmanned vehicles. A second contribution is the design of a modular platform (named, Seamote), which serves to allow remote real-time control and environmental telemetry of other IoT devices to be used in marine environments. Seamote acts as a data mule, capable of communicating with a wide range of control devices (e.g. mobile applications, remote controllers, artificial intelligence automated inputs, etc.), converting its’ payloads to control the end-devices used in marine setting (e.g. IoT devices for environmental telemetry, biota active telemetry and tracking, USVs - Unmanned Surface Vehicles, etc). In this study, Seamote is designed to be used by marine biologists and other stakeholders (i.e. tourists and citizen scientists), with the ultimate goal to obtain higher spatial-temporal resolution data (e.g. marine mammal footage, acoustic recordings and environmental data). An overarching contribution of this paper is to raise awareness in the design and HCI communities for the opportunities which lie outside of urban environments. We achieve this through a systematic review of the literature and challenges for marine science and oceanography. Then we relate to these challenges and technological opportunities for the design and HCI communities. Seamote depicts three major key research questions:

  • [RQ1]. How to design a low-cost LoRa-based Internet of Water Things (IoWT)?

  • [RQ2]. Which interaction challenges emerge from remotely operated devices?

  • [RQ3]. How to leverage environmental telemetry using LoRa?

2 Related Work

2.1 State of the Art in Aquatic Setting

Current research in oceanography greatly depends on satellite imagery, sensors and deployment platforms (e.g. Acoustic Doppler Current Profilers, Conductivity, Temperature and Depth profilers, Autonomous Underwater Vehicles, oceanographic buoys) to collect crucial information, such as Sea Surface Temperature [16, 32], variations in salinity and temperature over depth [15, 17, 29] and currents direction and intensity [4, 6], for their studies. Similarly, scientists focusing in the biology and ecology of marine organisms and habitats, also rely on multiple tools, instruments and sensors to collect data and better understand the links between abiotic and biotic factors and variables. Technological advances over the last decades have allowed deep-sea researchers to survey habitats, conduct experiments and collect samples remotely (e.g. with Remote Operated Vehicles - ROVs, drop-cameras, multi-beam sonar) and/or from the safety of custom designed submarines with comprehensive payloads designed for scientific purposes [12, 26, 30, 31, 33].

2.2 IoT in Aquatic Setting

Several studies have been reported to use UAV’s for environmental telemetry [4, 24, 28]. Other recent works are also reported to use IoT and provide low-cost solutions for water monitoring [1, 9, 13, 19] as well as tackling the problem of marine litter [8]. ROVs and USVs can be used for surveillance applications in the ocean, not only on the surface, but also underwater. These types of vehicles can be operated remotely be wired or wireless communication systems. Mahfuzh and colleges developed an ROV [10, 21] to make maneuvers underwater and on the surface. Their vehicle has 6 motor actuators on two motor controllers, commanded by a microcontroller. It is remotely controlled by a joystick and the radio commands are sent via a 2.4 GHz wireless communication, which has a very limited range [25]. Various aquatic technologies have been used for oceanographic data collection, offshore exploration, surveillance, surface water quality and navigation [2, 3, 14, 20]. While IoT provides an enormous potential allowing remote technical diagnostics and improved safety, including management of the energy distribution, monitoring equipment, improving passenger experience, enhancing navigation and tracking cargo [11, 18], IoT remains scarce in aquatic applications.

2.3 HCI in Aquatic Setting

Instead, when focusing on HCI applications, several studies have been reported to already use ROV’s. For instance, a work explored the usage of GUI for the control of UAV’s [7] providing the usability studies of persons in charge. Similarly, another work explored the similar approach used for ground control station staff [34]. Also, HCI applications have been reported to use gestural inputs to control remote operated vehicles [5, 23]. However, these all reported studies either rely on desktop applications, providing complex GUI applications and or these studies are not seen in aquatic settings. When dealing with marine environment, a recent study provided the interface and system for gathering and classifying cetacean acoustics using Wi-Fi. This study used whale watching tourists as citizen scientists to obtain these data [27]. In our study, we designed a system that goes one step further, allowing citizen scientists and marine biologists to collect larger corpora of cetaceans. Although interactive applications in HCI communities have been also reported to be tested within the water, such as the multimedia sensory table based on total internal reflection [22]. Conversely, no previous studies explored the potential of combining the HCI with LoRa, IoT, and embedding it into aquatic setting. In our study, we design the remote operating system and modular platform for allowing such interactions.

3 Methodology

This section describes the system apparatus, architecture and an overview of the designed Internet of Water Things (IoWT) system. The location chosen for the pilot tests was the local marina with an entrance to the sea. Subjects of the study were 3 participants (with age range between 24–34) who were approached on the spot. They were 2 males, including a sailor and computer engineer, and 1 female with linguistic background. All participants reported to have previous experience with the usage of smart phone applications. For the pilot test, all participants were using Think-aloud protocol and were asked to express their opinion about the effectiveness, efficiency and satisfaction when using the Seamote apparatus. They were given the Seamote App coupled with Seamote Bridge to perform initial usability tests. The given task was to operate the USV to a specific boat, explore the nearby waters, and to return, avoiding the obstacles found on the sea surface. Before each test run, to each participant a demo run was provided, depicting the maneuverability of the Seamote. Seamote apparatus is designed from three main components: (i) Seamote App, a mobile application for remote control of the device; (ii) Seamote Bridge, a designed case with microcontroller serving to receive signals from the Seamote App and communicate with the device using LoRa; and (iii) Semote USV, an IoWT remotely operated device for collecting the data.

Fig. 1.
figure 1

Seamote Bridge. An evolution of special design case with the embedded microcontroller and LoRa antenna, capable to be mounted to the smartphone using suction cups.

Full size image

Seamote Bridge. Being a custom designed case, it serves to contain basic microcontroller hardware. It is comprised of a single LoPy4 with an antenna and battery, shown in Fig. 1. Acting as a network integrator, Seamote Bridge runs a HTTP server, which receives commands from the control devices, in the form of POST requests, and forwards them in further via LoRa payloads to the USV end devices. Algorithm is designed to stop sending motor commands with included lag (between 1–2 s) should the participant stop using the Seamote App. In this setup, LoRa is not being used with the LoRaWAN network, and is used instead as a point-to-point communication, thus not requiring a gateway with an internet connection. The access to the HTTP server is done via an Access Point (AP) running on the Seamote Bridge, which also features a captive portal, popping-up on a Seamote App as a dialog upon connecting to the AP. Also, we designed an encasing of the Seamote Bridge which resembles a marine creature, in this case a tiger shark. It serves to contain the electronics inside, being coupled with the smartphone via suction cups, and being supported by the tail and fins, as depicted in Fig. 1.

Fig. 2.
figure 2

Seamote USV - deployed in-situ, used for the preliminary tests with participants. Top: Seamote App, used from the marina standpoint. Middle: Seamote USV buoyancy and stability tests. Bottom: Underwater imagery collected by the participants and Seamote USV against other marine vessels found in local marina.

Full size image

Seamote USV. In this study, device is a surface vehicle with enabled LoRa communication capable of reaching the remote land locations. This device is based on an existing Nikko Sub-168, a narrow range ROV toy, with dimensions of \(17\times 7\times 11\) cm. Rationale for using this device was the low-cost, out-of-the-box solution which contains two propellers of two blades each. Existing radio antenna and batteries have been replaced with new hardware based on a Pycom LoPy4 microcontroller. The purpose of using this microcontroller was due to it being a low-cost LoRa enabled chip which can support the testing controls of motors using LoRa. It is in further equipped with a PySense board, which includes an accelerometer, temperature, light, humidity and battery voltage sensors. Aim of the collected environmental telemetry is to be in future compared with external data, verifying to which extent does the casing and microcontroller biases the PySense sensors.

Fig. 3.
figure 3

Seamote App - an evolution of a mobile application GUI. From left to right: using a joystick mounted with a tangible analog stick using silicon suction cups. Image to the right: a map depicting the latitude and longitude of the Seamote USV.

Full size image

Seamote App. This study proposed the usage of simplistic smartphone application to be connected to the Seamote Bridge, allowing the Seamote USV to be fully operative from single screen. Mobile application and GUI are designed to encompass 4 core functionalities: (i) a joystick, used for controlling the end device, in this case the Seamote USV; (ii) ongoing battery level indicator, allowing the user to understand the battery level for both logic and motor batteries; (iii) a map, pointing out the past trajectories and current GPS coordinates obtained from the Seamote USV; and (iv) environmental sensor data, being displayed on the header of the application. The rationale of using the digital joystick was to simplify the usual four stock buttons, allowing the user to more easier maneuver the Seamote USV with a single hand while having the location indicator on map. Seamote App is in further connected via Wi-Fi to the access point located on our Seamote Bridge (Fig. 3).

4 Results

4.1 IoWT LoRa Deployment [RQ1]

Design of the Seamote system provided to be robust enough to withstand the Beaufort scale 1, as well as to successfully pass the buoyancy and water-tightness tests. In Fig. 2, it is possible to observe the Seamote system deployed in-situ. Understanding the collected images, Seamote system successfully managed to sustain the deployment, collection of data, environmental telemetry and retrieval to the surface, and all by the remote control from the remote marina location using LoRa. SD cards were also used, allowing the long-term collection of parameters capable for later retrieval, such as the environmental sensor data, application commands and board meta-data (battery levels and LoRa parameters). In Fig. 4, we depict the paths taken by the three participants performed from the origin to the goal points denoted in colors. The points shown in the figure were captured by the GPS (with its associated error) aboard the Seamote USV and recorded in the Seamote Bridge, while also being forwarded to the Seamote App. The distance from the start to each goal was on average 19.8 meters and the tests took approximately 5 min each, where participants were exploring additional circling and maneuverability of the Seamote USV.

4.2 User Interaction Observations [RQ2]

Using Think-aloud protocol, all three participants reported the Seamote apparatus to be effective (having the data in real-time). Regarding the efficiency, they stated that more instantaneous responsiveness should be added to the Seamote USV. Moreover, they expressed high satisfaction (enjoying in overall the remote control). In further, subjects suggested the option for autopilot mode with predefined routes in the Seamote App, pointing that this way Seamote USV could perform a more efficient long-range survey. Interestingly, all three participants expressed the need to have a button to stop the Seamote USV, even if our system already stops the engines if not being used for 2 s. Also, there were additional concerns with the suction cups used as a tangible joystick was not calibrated to run the Seamote USV with full throttle, where 2 participants removed the suction cup and used the joystick on the GUI.

4.3 Environmental Telemetry [RQ3]

In Fig. 5 we portray the temperature and humidity, obtained from the Seamote USV during the tests. Data shows temperature changes with peaks when taken out of the water between tests, as well as the influence of heat by the motors. Also, we observe nuances in humidity readings indicating the time spent in water. In Fig. 6, we plot the RSSI against the SNR. Although the short distance did not allow for the RSSI to be weak, this shows that the noise was kept consistent in the aquatic environment. The SNR showed values ranging from 5 to 7, indicating a good ratio that allows the radio to capture the signal against the environment noise - RSSI (M = −44.8, SD = 7.9) and SNR (M = 6.3, SD = 0.5) with n = 265. Figure 7 depicts the most used commands issued from the application, having forwards at full speed more used. Figure 8 depicts the pitch and roll recorded during the tests, showing a clear tendency of the Seamote USV to lean backwards. This is consistent with control commands from Seamote App, lifting the nose and dipping the back of the USV. Roll parameter proves to be stable in aquatic settings, as observed in the tests, only being tilted by abrupt corners, and always remaining in the correct position, due to the underwater camera stabilizing the center of mass.

Fig. 4.
figure 4

Seamote In-situ test - 3 GPS trips denoted in colors from one point of the harbour to the other and back.

Full size image
Fig. 5.
figure 5

Environmental telemetry sensor data obtained from the Seamote USV.

Full size image
Fig. 6.
figure 6

Signal to Noise Ratio (SNR) consistent across different Received Signal Strength Indicator (RSSI) in water environments.

Full size image
Fig. 7.
figure 7

X and Y map of the commands sent from the mobile application. Color and size denote the usage frequency of those positions (green indicating more). (Color figure online)

Full size image
Fig. 8.
figure 8

USV pitch and roll indicating a tendency to lean backwards, as the primary motion in the tests was a forwards motion.

Full size image

5 Discussion

In this study we present Seamote, an integrative LoRa-based IoWT apparatus for aquatic deployment, consisted from 3 designed modules: a mobile application (Seamote App), casing for the phone (Seamote Bridge), and an Internet of Water Things (IoWT) low-cost device, used for aquatic surveys (Seamote USV). While observing the results of our research questions, we find that it is possible to design and deploy IoWT applications [RQ1]. Moreover, LoRa proved to be adequate for the remote control of the USV, as reported by the participants [RQ2]. Conversely, obtained environmental telemetry shows the feasibility of collecting such data [RQ3]. More studies, and usability tests need to be performed to provide a more clear insight to the best tangible user interface and feedback. Currently, there are several limitations of Seamote apparatus such as the USV size and speed, as the design of circuits has been reappropriated to match into the existing toy casing. This hinders the Seamote to travel the larger distances which is allowed by LoRa. Also, such device can withstand solely Beaufort scale 1, avoiding the risk of permanent device dislocation. Future controller should support stronger casings while Seamote App needs to be tailored to allow multiple devices. Additional optimization should be performed to allow constant exchange of payloads among the devices. Future work will focus on integration of active tracking of acoustic tags, providing a flexible network of receivers capable of transmitting data in real-time. Seamote apparatus will be scaled up to support sun/wind power generation for long-term deployment and remote monitoring of environmental conditions. With Seamote apparatus, it is possible to sense and react to captured data, used in survey missions in marine biology. Also, Seamote can control multiple USVs and UAVs. Drones can be used to capture aerial images of the ocean, using real-time image vision algorithms to detect marine litter. Other versatile sensory input can also be mounted to the current microcontroller, e.g. dissolved oxygen, hydrophone, sonar, salinity, turbidity, plankton collectors, etc. Finally, Seamote apparatus provided in this pilot study has a threefold impact on aforementioned related work: (i) it provides a low-cost solution for the state of the art sensors found on market, as sensing the oceans remains still expensive; (ii) it challenges the IoT applications to be design for aquatic environment, allowing the new research direction in Internet of Water Things (IoWT); and finally (iii) it opens the door for the new interaction interfaces and novel HCI applications which are to be applied in challenging oceanic environments.