1 Introduction

For decades, monitoring movement parameters has been of interest to athletes, coaches and researchers alike. Athletes are interested in both the information layer (distance, speed, pace, etc.) and the social aspect (this particularly concerns recreational/amateur athletes) [1, 2]. Coaches are interested in numbers (data) because without knowing “how it is”, it is difficult to properly apply changes in an athlete’s training and monitor them so that progress is made [3]. In turn, the scientific field often covers both numbers and devices, assessing their usefulness in professional training, including the reliability and failure-free performance of particular models.

Providing civilian users with the first global navigation satellite system (GNSS), namely GPS NAVSTAR, in the mid-1990s, particularly after the elimination of the so-called selective availability (a deliberate reduction in accuracy; May 2000), resulted in an increase in their popularity. What was of great importance to the development of this technology were the advances manifested in the miniaturisation of devices and increasing their computing capabilities and the achievement of full operational capability by three successive systems (Russian GLONASS in 2011, European Galileo 2016, and Chinese BeiDou 2020). As well as the fields of professional land surveying and navigation, they have also found applications in sports and recreation.

Popular receivers, commonly used for recording and monitoring movement parameters, are code devices in which the determination of position coordinates is based on a measurement of the pseudorange and is computed as the product of the duration of signal transmission on the satellite-receiver path. A greater number of satellites received (‘seen’) by a GNSS receiver allows observations to be carried out continuously and more reliably, which is particularly important in areas with limited horizon visibility (e.g. mountains, urbanised areas), has an effect on the equipment initialization duration, and contributes to reducing the duration of the measurement. Manufacturers of the leading sports and recreational receivers usually rely on two systems (most frequently, GPS+Glonass or GPS+Galileo), although triple- and quadruple-system devices are available. In some cases, human movement information extracted from receivers leads to advanced analyses [4, 5]. Also, an increasing number of receivers use additional sensors, such as accelerometers or gyroscopes [6, 7].

GNSS receivers, in which the coordinate determination is based on a variable number of observed satellites and, thus, changing pseudoranges, are specific measuring instruments because they do not yield a repeatable measured value. It is also known that in view of geometrical determinants, the vertical accuracy is inferior to the horizontal accuracy [8, e.g. tables 3.8-3, p. 58]. It is, therefore, necessary to test receivers, preferably using procedures developed to this end (and described, e.g. in [9], as static and dynamic testing). The accuracy of position coordinate determination by receivers is most fully defined by the sample distribution, and its measures include, e.g. Root Mean Square (RMS), Circular Error Probable (CEP), Spherical Error Probable (SEP), Distance Root Mean Square (DRMS), etc., described, for example, in reference [10]. In many operating manuals for datalogger-type receivers, popular in the 2010s, and for certain sports receivers, these values were published, yet this practice has stopped over time.

There are several methods for assessing the usefulness of GNSS receivers in monitoring sports and recreation [11]. The easiest approach taken by researchers is to test a particular receiver in a set of movement activities specific to a particular discipline/event, and then to relate the values obtained to the desired values (e.g. total distance; [12, 13]). This method, however, has a serious drawback, as it is easy to imagine a situation in which a receiver, when determining the total distance based on erroneous coordinates, is close to the reference distance (which provides no information on the receiver quality). In the most advanced receiver testing attitude for each point of the coordinates determined by the tested receiver, a reference point is established at the same moment of time (and, obviously, in the same coordinate system). A task of this kind requires both the use of precise GNSS receivers operating in the Differential GPS/Real Time Kinematic mode and the performance of complex computations [14,15,16,17,18].

Many sports disciplines in which the athletes’ locomotion is monitored are carried out on a two-dimensional, horizontal plane. This is the case with football, marathon running, open-water swimming, sailing races, etc. Another group of disciplines are those in which the elevation component occurs (the route being covered exists in a three-dimensional space) but has no significant effect on the outcomes achieved or is insignificant in terms of the knowledge being gained (cross-country skiing, certain car and cycling races). The third group is sport disciplines/events for which the elevation component is of utmost importance, with examples including alpine skiing, paragliding, downhill mountain biking and mountain running. The latter, in particular, has been characterised by growing popularity in recent years, as evidenced, for example, by its introduction into the World Athletics’ competition cycle [19], with the competitions of this kind being divided into two categories, namely ‘uphill’ and ‘up&down’, and defined appropriately (Rule 251 in [20]). For certain popular organisations (e.g. International Skyrunning Federation, ISF), many more mountain running events are distinguished, with their framework strictly defined (e.g. for the Vertical category: ‘uphill only races with 1,000 m vertical climb’; [21], par. 2.3.4)

When describing the locomotion changes in climbing by an athlete, different measures and indices are applied. The simplest one is to determine the absolute height (above sea level, ASL) or the relative height as the difference between heights (e.g. between the end point and start point) or the so-called elevation gain (EG) or total elevation gain (TEG) being the sum of all climbs/uphill runs. Progress in reaching elevations is also sometimes described using the VAM index (originally from the Italian language, ‘velocità ascensionale media’ (Vertical Ascent in Meters, unit—vertical metres per hour), commonly applied in road bicycle racing. It is also used in mountain climbing under the name of ‘vertical climbing speed’ (the same unit, [22]).

The reading (estimation) of a single height by a sports and recreational GNSS receiver will not be identical to the height known from topographic maps. In common terms, the ‘elevation’ is the difference between the Earth’s surface (adopted as the mean sea level, MSL) and a point above or below it. A typical sports GNSS receiver determines, by performing computations, the longitude and latitude as well as the so-called ellipsoidal height—the distance between the receiver and the surface of the rotational ellipsoid, a virtual surface described by the WGS-84 reference system (a standard used in geodesy, cartography, and satellite navigation). Since the Earth’s surface, however, is highly variable, an additional theoretical concept of a geoid is being introduced into the field of Earth sciences, defined as a surface to which the force of gravity is everywhere perpendicular. A model of such a geoid, with high resolution and accuracy, is then used in land surveying, geological, geophysical and oceanographical works. The distance between the receiver and the geoid surface is referred to as the orthometric height. These quantities are described by the formula [23, p. 53]:

$$H\approx h-N$$
(1)

where H is the orthometric height to the geoid; h is the ellipsoidal height, and N is the undulation value (the difference between geoid and ellipsoid).

If a GNSS receiver has (or is capable of) downloading an accurate geoid model for a predetermined longitude and latitude, it is also capable of providing the user with height in relation to MSL. For many popular sports and recreational GNSS receivers, the solution to improving the elevation determination accuracy is the use of a barometric sensor which either operates automatically (when starting the activity) or is initiated by the user. Certain models also have the ability to manually calibrate the starting point against the DEM (Digital Elevation Model of the Earth) in place, but the type and accuracy of such terrain models, covering large areas of the world, are not provided by manufacturers.

Regardless of how an athlete’s progress in reaching an elevation is assessed, what is crucial in this analysis is the correct determination of this elevation by devices used by the athlete, as the distribution of effort on individual sections (e.g. of a mountain run) is an important aspect of achieving the optimal final result. Bearing this in mind, this article has two aims: (1) to propose a methodology for assessing sports and recreational GNSS receivers in the context of their determination of the elevation component, with attention paid to the appropriate measures for assessing these receivers, and guidelines on reference systems. This methodology is based on the idea, which has been under development for almost two decades, of creating high-accuracy digital terrain models for individual countries, which, by supplying geographic information systems (GIS), allows complex spatial analyses to be carried out; (2) to apply the aforementioned methodology for assessing wrist-worn GNSS receivers used by the participants of a selected mountain run.

2 Methods

2.1 Reference surface and test procedure

The study was based on the Digital Terrain Model (DTM) of Poland, which is a discrete (point-based) representation of the topographic elevation of the land surface developed on behalf of state institutions and made freely available by the Head Office of Geodesy and Cartography (pol. GUGiK, gugik.gov.pl). A digital model in the ARC/INFO ASCII GRID format was used, in which text files contain point elevation values in a regular grid with a 1-m mesh. The points are generated based on aerial laser scanning (LIDAR) in the reference system PL-EVRF2007-NH. The mean elevation error for the aforementioned model falls within a range of up to 0.2 m [24], which meets the recommendation described in reference [9], according to which the reference system should be 10 times better than the accuracy of the device being tested. Data packages for the Tatras area were downloaded in April 2022 (valid for the year 2021).

The following test procedure was adopted: for each point of the geographic coordinates determined by the test receiver, a height resulting from the DTM used was automatically determined using the GlobalMapper program (version 22). Both elevation data sets were then compared in a spreadsheet by computing the following measures: root mean square (RMS, associated with a probability level of 68%), mean elevation error, and total elevation gain error. The latter measure is presented in the following form:

$$\mathrm{TEG }(\mathrm{\%})=\left(\frac{{\sum }_{i=1}^{n}EGrec \times 100}{{\sum }_{i=1}^{n}EGpseudoref}\right)-100$$
(2)

where EGrec is the single elevation gain error specified by the tested receiver; EGpseudoref is the single elevation gain error based on DEM for coordinates specified by the tested receiver and n is the number of observations.

The procedure based on the DTM reference surface was checked during preliminary tests with the use of three wrist-worn, multi-system receivers (Garmin Forerunner 920xt, 935, and 945).

2.2 Mountain uphill run

The running event held in Poland’s highest Tatra mountains (Tatra Uphill Run - Poland’s Skyrunning Vertical Championship, www.biegnakasprowy.pl, 25 September 2021) was selected due to being a competition that reflects well the participants’ demand for the elevation parameter. This is the longest uphill run in Poland and, according to the organiser, over a distance of about 8.5 km, runners reach the estimated elevation of about 1072 m. Figure 1 shows the route of the run (dark blue line), its key points (start—Zakopane city, checkpoint—Myślenickie Crags, finish—Kasprowy Peak) and elevation profile. The terrain visualisation was generated from the aforementioned DTM using GlobalMapper.

Fig. 1
figure 1

Key locations of the run against the terrain generated by the GlobalMapper from a DTM point cloud (below) and the elevation profile (above)

Full size image

The athletes who recorded their track using wrist-held GNSS receivers were contacted after the run and asked to transfer the track in the original version (recorded using software dedicated for a particular device), indicating the manufacturer as well as the receiver model and version and providing information as to whether the receivers were set to default mode, or whether other settings were used. A total of 20 tracks from among 295 race participants were collected, including tracks from seven athletes from the top 20 of the list. The devices included various models manufactured by Garmin, Suunto, and Polar, which were capable of receiving signals from one (2 devices), two (7 devices), or four global satellite navigation systems (5 devices). As the settings of the athletes’ receivers were not discussed with them before the event, the settings preferred by individual participants were taken into account. It appeared that, in the context of GNSS selection, nineteen people (95%) used default settings (which in 14 cases—70%—was the GPS system, while in five—25%—GPS+Glonass) as well as an automatic elevation determination support mode (barometric altimeter, available in most receivers). One of the users changed the reception of two systems into the reception of one (GPS). Other settings were not noted, despite potential opportunities. The recording frequency varied for different receivers while amounting to 1/s for most of them, which translated into a mean number of about 2.2 thousand coordinate determinations per receiver (a large number of data points is important, as any individual/single application of DEM can be affected by errors [25]). Each track was trimmed, with only the section corresponding to the duration of the run being left. The computations were corrected for an offset of 1.0 m (the average difference between the receiver location and the ground). The local Ethics Committee made no objections to the research.

3 Results

Table 1 presents the results of sports receivers, computed based on the tracks recorded during the mountain run, in relation to the reference surface. The results were sorted in descending order according to the RMS measure.

Table 1 The key settings and results obtained by sport receivers during uphill mountain run
Full size table

As can be seen, in terms of elevation determination during the run, the most accurate ones were three S9B (same model) receivers (#278, #20 and #4) in which the RMS result obtained amounted to 3.6, 4.0, and 4.1 m, respectively. The highest RMS values were obtained by devices that did not use a barometric altimeter (#5, #35, #196—22.1, 19.7, and 14.0 m), although the individual examples of #45 and #98 (6.4 and 7.6 m) show that acceptable results can also be achieved without using this sensor. Higher values of the mean elevation error were a characteristic of the second part of the run (checkpoint to finish), which is interesting because this section of the route is covered over the mountain ridge (Fig. 1, on the right), and the availability of GNSS satellites should be (and is) better than that in the first part (where the route is running through a wide, extensive valley). Based on the computations presented in the last column, it can be noted that eighteen receivers (90%) underestimated the TEG. Neither the reception of two GNSS systems nor the use of a barometric altimeter appears to affect this value.

4 Discussion

The issue of accuracy of widely available, code-based satellite navigation receivers has been known for a long time and concerns both horizontal and vertical accuracy. As successive GNSSs have reached their full operational capability, and as the systems and electronic devices used for their reception have become increasingly advanced, modern receivers perform better at determining the position coordinates and elevation, particularly under difficult conditions (forests, mountain valleys; [26,27,28]). This does not mean, however, that they have become tools which require no attention in terms of handling and settings.

The methodology for assessing the determination of the elevation component by sports GNSS receivers, proposed in this article, is relatively simple to implement. As for Poland, the DTM is freely available, and individual data packages are precisely characterised and processable in the aforementioned program or its freeware equivalent (e.g. QGIS). Importantly, there is also a specified mean elevation error of this model, falling within a range of up to 0.2 m, which successfully meets the requirement stating that the reference system must offer accuracy ten times greater than that of the tested receiver. These assumptions enabled a comparison of the receivers used by selected participants of the ‘Tatra Uphill Run’, which showed clear differences in the accuracy characteristics of the devices (Table 1). For example, the adopted RMS measure allowed better results obtained by the models with an integrated barometric sensor to be noted, while no advantage was noted for dual-system GNSS receivers as compared to the single-system ones, even though the GPS-only receivers with no barometric altimeter were mainly found at the bottom of the list. As regards the TEG measure, the general conclusion was that the receivers commonly underestimated this value by several percent. In a similar study, in which inference was based on elevation corrections with DTM ([29], unfortunately, neither the terrain model on which the study was based nor its accuracy was provided), TEG in cycling was compared. It was noted that the results were relatively consistent among a group of devices of the same manufacturer, with differences only occurring between individual manufacturers. This study did not note this relationship, but it should be stressed that the receivers under assessment were different devices (manufacturer, model, model version) with different settings (recording frequency, software version, etc.), and the analysis of the tracks was conducted with no previous arrangements with the participants.

Interestingly, however, when the size and characteristics of receiver errors over the entire distance of the ‘Tatra Uphill Run’ were analysed, it was found that receivers of particular manufacturers had a characteristic repetitive profile, which is shown in Fig. 2. As regards the Garmin devices, it was the underestimation of the elevation, which increased with the distance starting from the halfway of the run, while for the Suunto devices the situation was similar, but the error value in plus/in minus was not that significant. In contrast, the Polar receivers showed variable error characteristics which were unrelated, it appears, to the course of the mountain run route. It is most probable that the observed profiles are related to the manner in which the receivers of individual manufacturers ‘choose’ satellites for the determination of their geographical position, based on both hardware and firmware [30, 31]. Thus, this would partially confirm the observation of reference [29] on certain receiver characteristics that are consistent within a group of producers.

Fig. 2
figure 2

The size and characteristics of errors of selected Suunto (a), Garmin (b), and Polar (c) receivers over the entire distance of the mountain run route (vertical axis: a uniform scale from −20 to +20 m)

Full size image

Despite the range of available settings of the receivers used to record their run, most users used default settings, e.g. for the number of GNSS used. It is possible that this is the result of the nomenclature being commonly (and erroneously) used in society, according to which the term ‘GPS’ has become a synonym of a GNSSs in general, so that the typical user is unaware that there are other, equivalent GNSSs, whose reception can improve the capabilities of the device ([17], table 9). Thus, the devices may not have been set up optimally for this activity, which could affect the achieved results. Meanwhile, the proper selection of movement-monitoring devices and their settings are important because they can further provide the basis for more complex analyses of the loads or biomechanics of running athletes (e.g. [32, 33]), and have an influence on safety-related decisions being taken, particularly under difficult, e.g. mountainous, conditions.

5 Conclusion

The use of DEM with good accuracy (mean error 0.2 m) enabled a comparison of sports and recreational GNSS receivers during a mountain run. The best results in terms of determining the elevation component were obtained by receivers supported by a barometric altimeter (RMS value of 3.6–4.1 m, same model), while the worst values were achieved by receivers without this sensor (14–22.1 m). It was also observed that the elevation error characteristics were common within the group of receivers of a particular manufacturer. While objective measure values were obtained, and the methodology allowed the receivers to be distinguished, the study itself is subject to certain limitations and comments. Firstly, the receivers under test should be prepared (navigation settings, up-to-date firmware, charge level) and supervised (until the data are exported) by the researcher, not the user. Secondly, it would be worthwhile, in future research with the use of DTM, to pay attention to acquiring an accurate (horizontally) route of the run. Thirdly, it should be investigated whether the (vertical) fluctuations of the wrist-worn receiver affect the altitude determination. The final comment concerns the receivers themselves: if the study aimed to determine the accuracy of a particular model, more than one unit would have to be gathered.