Keywords

1 Introduction

Road surface conditions have long been a concern in society because they have a significant impact on transport safety and driving comfort, especially in snowfall areas. It can be seen that about 50% of accidents have occurred on frozen road surfaces. Therefore, it is important to detect frozen road surfaces effectively.

In areas of snowfall, the road surface can have many different states, which will change with weather and volume of traffic. The changes are mainly influenced by two factors: (1) The substance that covers the road surface, such as asphalt, water, snow, and ice, which is called the road surface type (RST); and (2) The shape of the road surface, such as its roughness or frequency of potholes, which is called the road surface shape (RSS).

Automatically estimating road surface conditions is a critical activity in transport infrastructure management, and many approaches have been proposed. Most make use of expensive sensors to detect road anomalies, or evaluate the road roughness index, however a common problem of these approaches is the high cost of setup and execution. Modern smartphones contain various sensor types such as an accelerometer, gyroscope, and Global Positioning System (GPS), which allow the smartphone to track its position and motion states with a high degree of precision. Because the penetration rate of smartphones is increasing, crowdsourced mobile sensing, used for collecting low-cost smartphone sensor data, has become possible; this allows for the use of an in-vehicle smartphone to monitor and estimate road surface conditions. Estimating road conditions using such sensors, which are usually loosely placed in the car, nonetheless poses a significant challenge.

In this paper, we propose a smart city application “around-the-corner.” which enables citizens to share road conditions including traffic and surface conditions. The keys to establish the application are collecting data on road and estimating road surface conditions, by using a motion sensor embedded in a smartphone. To solve the former issue, we proposed a methodology of crowdsourced mobile sensing framework [1], which can collect sensor data reflecting microscopic roadside phenomena using crowdsourcing. In our study, a published smartphone application called ‘Drive around-the-corner.’ is used. This provides an online driving recorder service to collect both sensor data and videos, recorded from the view of the driver; by using this application, users benefit from a free record of their driving, and we obtain large amounts of low-cost sensor data. Then, we estimate road surface conditions which contain both the RSS and RST factors by analysing such collected sensor data. We have been developing a web application “around-the-corner.” to show detected conditions and road events, such as heavy traffic and road constructions.

2 Related Work

2.1 Estimation of Road Surface Conditions Using Acceleration Sensors

For paved roads, most existing work relates to the RSS; for example, the road roughness, or road anomalies such as potholes. The International Roughness Index (IRI) [14] is a standard global index of road roughness, and study [4] shows that the IRI and the Root Mean Square (RMS) of the vertical component of acceleration values, have a high correlation. Using this relationship, it is possible to calculate approximate values of the IRI, however a limitation of the study [4] is that the parameter must be manually adjusted for different vehicles. Another study [15] provides a spring and damper model, which can automatically estimate vehicle parameters including a damping ratio and resonant frequency, and can then use these parameters to calculate approximate values of the IRI. A further study [11] also evaluates the roughness index, but focuses mainly on detecting the changing road conditions. Other research has studied the detection of road surface anomalies, such as potholes; one study [2] provided an improved Gaussian Mixture Model (GMM) for detection of road potholes.

2.2 Estimation of Road Surface Condition Using Cameras

In contrast to normal paved roads, most work on snow covered roads concerns the RST, and uses image processing techniques. Studies [8, 12, 13] use standard in-vehicle camera devices, such as a driving recorder or smartphone. Among these studies, [8, 12] can determine wet or snowy conditions with a high degree of accuracy; however, they cannot detect frozen roads that are the most dangerous in snowy areas. Study [13] can detect frozen roads, but with an accuracy of less than 60%. Study [16] also uses an in-vehicle camera to estimate the road surface condition with a high level of accuracy, but it is necessary to attach two polarizing films to the lens. Finally, study [7] used a fixed camera placed at representative points on major roads, and to improve accuracy the study also used weather data; the geographical area covered however, was limited.

2.3 Summary

Table 1 shows a summary of the related work discussed in this section. A common problem is that no single study supports both RSS and RST. Additionally, approaches using motion sensors such as an accelerometer are more robust than those using cameras. For these reasons, in our study we have estimated new road surface conditions using both the RSS and the RST. Furthermore, we have provided a method to estimate newly defined road surface conditions using motion sensors only.

Table 1. Summary of related works
Full size table

3 Social Drive Recording Service: “Drive around-the-corner.”

In February 2015, we developed a drive recorder application called “Drive around-the-corner. (Drive ATC)” This application was made available to the public in February 2016Footnote 1. Drive ATC collects driving behavior logs, records events, and delivers information regarding the vehicle’s current position (Fig. 1).

Fig. 1.
figure 1

Smartphone mounting positions.

Full size image
Table 2. Data collected by Drive around-the-corner.
Full size table

The service can be accessed via the iOS application. Before commencing a journey, users mount their smartphone in a holder and connect a power-supply cable if necessary, and then open the application. The application records driving behavior logs and videos, and uploads them to the service platform.

3.1 Sensing Functions

Onboard Location and Motion Sensors. The Drive ATC application obtains location and motion data from onboard sensors. While users are driving and using the application, behavior logs and movies are recorded. The data that are collected are pooled in the local data store and then transmitted to the service platform. The data that are collected are shown in Table 2.

Movies. The Drive ATC application records two types of movies, one to be uploaded and the other to be saved locally. To reduce traffic to the service platform, uploaded movies are transferred intermittently, the frame rate being adjusted in accordance with the speed of the vehicle.

Because these movies are uploaded via a mobile network such as 3G or LTE, they should be compressed. The movies that are saved locally are of higher quality and can be used as evidence in the event of an accident.

3.2 User Functions

Map with Event Information. When the Drive ATC application is opened, it shows a map of the current location (Fig. 2a). Roadside events are retrieved from the service platform and shown on the map. For example, the yellow icon located at the center of Fig. 2a denotes road construction. This information has previously been posted by other Drive ATC users.

Fig. 2.
figure 2

The “Drive around-the-corner.” application. Traffic information, user-posted events, events extracted from sensor data, and footprints, are shown on the main screen map. (Color figure online)

Full size image

Posting Event. To enable users to report a roadside event to others while they are stationary, the application provides them with the ability to post event information. After tapping the footprint marker in the top right corner, users are requested to select an event that they recognize (Fig. 2b). There are eight possible events grouped into three categories: heavy traffic, road condition, and roadblock. The selected event is posted to the service platform with details of the current time and location.

3.3 An Example of Collected Data

Figure 3 shows an example of acceleration data collected from a vehicle in Sapporo. The green line represents the z-axis (vertical) offset acceleration value for gravity. The blue and red lines represent the x-axis (the axle direction of the car) and y-axis (the heading direction), respectively. Figure 4 illustrates scenes from the journey shown in Fig. 3.

Fig. 3.
figure 3

An example of recorded acceleration data (Color figure online)

Full size image
Fig. 4.
figure 4

Example scenes (Color figure online)

Full size image

At first, the car travels on a wide road covered with snow (Fig. 4a). The surface of the road is frozen, but even. The acceleration, represented by the green line in Fig. 3a of this segment oscillates below \(0.3\,\mathrm {m/s^2}\). Next, the car stops for a red light. The car’s engine is automatically shut down by the start–stop system, and the oscillation falls to the minimum level. Then, the car moves to the next intersection and turns left onto a wet road. The increase in transverse acceleration represented by the red line in Fig. 3a indicates the left-hand turn. The acceleration increases significantly because this wet road is more stable than the previous frozen road (Fig. 4b).

The car then turns left onto a narrower road. The road is frozen, but is uneven because the ice is thawing. The car travels very slowly and pitches wildly (Fig. 4c). Its acceleration reaches more than \(0.5\,\mathrm {m/s^2}\), and even rolling and yawing are recognized because the car slips and drifts on the road (Fig. 3b).

4 Detecting Road Surface Conditions

In previous studies, many road monitoring systems have used image processing techniques [3, 16] whereby the video cameras are usually placed at representative points on the main roads or mounted on the dashboards of vehicles. However, the effectiveness of the cameras may be impacted by factors such as low light levels at night time or snowfall. Further, this method cannot identify the surface conditions when a frozen road surface has been covered by a layer of snow.

Fig. 5.
figure 5

Three kinds of road surfaces

Full size image

To enable the detection of road surface conditions from the collected data, we studied a methodology for estimating the condition of snow-covered roads using the collected motion values. This method extracts features from both the time domain and the spectral domain of data collected from accelerometers and gyroscopes.

4.1 Road Surface Condition Definition

We define a set of both road surface types (RST) and road surface shapes (RSS). The set of RST includes four elements: paved, sherbet, compacted snow, and frozen. A sherbet road surface means that the road surface, which is covered by snow, contains water or ice. The set of RSS also includes four elements: Smooth, bumpy, potholes, and potholes/bumpy. A smooth surface shape means that the road is flat, regardless of whether snow has fallen. Tables 3 and 4 show details of these; the new road surface conditions are defined by the cartesian product of these two sets.

Figure 5 shows some examples of the defined road surface conditions. Figure 5b, c, and d show frozen road surfaces with different RSS. In snowy areas, these are the most dangerous road types; even cars fitted with winter tires may slip. Figure 5e shows a sherbet road surface with some potholes; although this kind of road is less hazardous than frozen roads, it will affect vehicle speed. Figure 5f shows a road covered with compacted snow; although the road has a covering of snow, driving on this kind of road is usually normal.

Table 3. Road surface type definitions
Full size table
Table 4. Road surface shape definitions
Full size table

4.2 Estimating Road Surface Conditions

As mentioned in Sect. 2, most related work has focused only on vertical direction acceleration when estimating road surface conditions, when in fact road surface conditions generate other vehicle in-motion values. We, however, realized in our preliminary experiment that not only vertical but also horizontal acceleration values of frozen and rough road are greater than of compacted and smooth road. The reason for the horizontal motion may be due to the car slipping in a horizontal direction despite the driver preference to keep the driving direction forwards only. We, therefore, can assume from this example that the motion values may reflect both the type and shape of the road surface. If this assumption is true, we can use motion values to classify only road surface conditions that include both type and shape.

4.3 Feature Extraction

In the field of human activity recognition, many effective features have been published using motion sensor data; in our study, these were used as an initial feature set. In our method, signals from each axis of the accelerometer and gyroscope are segmented into windows of 2 s, with a 50% overlap between two consecutive windows. We use \(x_a\), \(y_a\), and \(z_a\) to denote the three axes of the accelerometer, and use \(x_g\), \(y_g\), and \(z_g\) to denote the three axes of the gyroscope. For both the accelerometer and gyroscope, the x-axis is the direction of the car axle, the y-axis is the direction in which the car is heading, and the z-axis is the vertical direction. Table 5 shows the details of the initial feature set. The extracted features have 67 dimensions in total.

Table 5. Initial set of features for estimating road surface condition
Full size table

4.4 Features Selection

A good feature set helps to improve the efficiency of the classification algorithms and enables accurate classification. Numerous feature selection algorithms have been published. Among them, the PCA, Relief-F, and SFFS are three popular algorithms for feature selection. To select the best features from the initial feature set, we compared these three algorithms to choose the best one. In this study, we determined the criteria of choice as the number of features that provide the best accuracy with the random forest classifiers. For evaluating the accuracy, we used the ten-fold cross-validation approach.

PCA is a mathematical algorithm that reduces the dimensionality of the data while retaining most of the variation in the data set [6]. It accomplishes this reduction by identifying directions, called principal components, along which the variation in the data is maximal. Using this algorithm to select features, we gradually increased the number of PCA components beginning with two PCA components and calculated the accuracy each time for all the 67 PCA components. Finally, the least number of features that provided maximum accuracy were selected.

Relief-F is a filter-based feature selection method used for the weight estimation of a feature [9, 10]. The weight of a feature of a measurement vector is defined in terms of the feature relevance. The features were sorted according to their relevance in decreasing order. The most relevant feature was first added and the accuracy of the given dataset was found using random forest classifiers. Subsequently, the successive relevant features were added sequentially, and the accuracy was calculated each time until all the 67 features were added. Finally, the least number of features that provided maximum accuracy was selected.

SFFS is a wrapper-based feature selection method [5]. It uses a classification scheme as a wrapper around which the whole feature selection is carried out. It starts with an empty set for the desired selected features “X”. The features are to be selected from a larger set of features “S”. Let’s be the most significant feature in S with respect to X, which provides the least accuracy when included in X. At each iteration, the most significant feature in S is included in X if its inclusion does not increase the accuracy. Similarly, the least significant feature in X is found and removed if its exclusion helps improve the accuracy.

4.5 Classification

In this study, we use the random forest classifiers to classify the road surface conditions based on the folders for cross validation. A random forest classifier is an ensemble learning method that constructs a multitude of decision trees at training time. It is one of the most successful ensemble learning techniques that has proved to be very popular and powerful in pattern recognition and machine learning for high-dimensional classification and imbalanced problems.

According to the study [2], different speeds of a vehicle will affect the values of the frequency and the amplitude of the motion, even for a vehicle driving on the same road under the same conditions. In this study, we divided the velocity domain into intervals of \(10\,\mathrm {km/h}\). The road-surface condition was classified across different speed ranges using the random forest classifier.

4.6 Experimental Results and Discussion

Datasets. The road condition labels were manually generated by three people living in an area of snowfall. The actual ground conditions were determined by voting results from these three people. In addition to the road surface conditions, the acceleration and gyro are also affected by the car, driver, smartphone, and the mount. To reduce these factors, we used the same car and driver, with the same smartphone and mount; differences in motions should thus be affected only by road conditions and driving behavior. We used the dataset from one user, and Fig. 6 shows the numbers of each road condition at different speed ranges. Because the drive recorder application was used during daily driving, imbalances inevitably occurred in the collected data.

Fig. 6.
figure 6

Numbers of each road condition at different speed ranges

Full size image

Feature Selection and Classification Accuracy. To select the best features from the initial feature set, we compared three kinds of feature selection algorithm: The PCA, the Relief-F, and the SFFS. Each algorithm is evaluated against the accuracy of the random forest, with ten folders for cross validation. Figure 7 shows the selected features from three feature selection algorithms, and Fig. 8 shows the performance index comparisons for random forest classifiers, combined with the feature selected algorithms; the SFFS algorithm is most effective, with fewer features and higher accuracy in each speed range.

Fig. 7.
figure 7

Selected features from three feature selection algorithms

Full size image
Fig. 8.
figure 8

Performance index comparisons for random forest classifiers, combined with the feature selected algorithms

Full size image
Fig. 9.
figure 9

Results of the SFFS + Random forest classification, across different speed ranges

Full size image
Fig. 10.
figure 10

Details of the selected features by using the SFFS in different speed ranges

Full size image

Based on the results above, we decided to use SFFS as the feature selection algorithm. In this study, the classification was evaluated against the recall of the random forest, with ten folders for cross validation. The recall is defined as

$$\begin{aligned} Recall= \frac{TP}{TP+FN} \end{aligned}$$
(1)

where TP is the number of true positives, and FN the number of false negatives. Figure 9 shows the results of the SFFS + Random forest classification across different speed ranges; the results show that the best average of recall, of about 91%, was obtained in the \(50\,\mathrm {km/h}\) – \(80\,\mathrm {km/h}\) range. We, therefore, could confirm that the road surface conditions can be effectively classified by using only the motion values when the speed is greater than \(50\,\mathrm {km/h}\). In particular, we could confirm that the most dangerous frozen road can be classified by using only the motion values.

Figure 10 shows the details of the major features selected by using the SFFS. The most common features in different speed ranges are the correlation between the directions of the vertical and the car axle, the standard deviation of the pitching of the vehicle, and the high frequency energy.

Discussion. From the experimental results, we can see that the more complex RSS such as bumpy and potholes are distributed in the low-speed range, and accuracy of the classification between them is very low. One of the reasons is that we have not quantitatively defined the RSS.

Fig. 11.
figure 11

“around-the-corner.” website (Color figure online)

Full size image

In this paper, we have described an estimation method for the road-surface condition by using only one vehicle. However, in the future, we need to estimate the road surface conditions by using multiple vehicles. Many factors influence vertical motion values. One of the factors is the type of vehicle. Length of the wheelbase and the strength of the suspension differ depending on the type of vehicle, and we believe these aspects have an influence on the motion values. The other factor is the individual’s driving style. We infer that the motion values change differently, depending on the individual’s driving technique and experience. We need to verify the relationship between these factors and the motion values in the future.

5 Web Application: “around-the-corner.”

Users can access the service website to check on the current situation and review their journey and driving performanceFootnote 2. Figure 11 illustrates the website, which mainly consists of a map and a list of detected events. All visitors to the website can view event information in either map or list form (Fig. 11a). On the right of the main page, up-to-date events are listed. When they select one event on the list, the icon of the selected event will be placed on the map. Each event is represented on the map by a corresponding icon. Under the map, shown events on the current map are listed. Displayed events can be filtered by users with the controller in the page. Users can set the target time period, time range, days of the week, month, and types of events.

Events include both detected based on collected and anonymized data and posted by users. Figure 11b illustrates some detected events, such as hard braking and lateral vibration, on map.

Registered users can also log in and access their own driving records. Figure 11c illustrates an example of driving records. The drive route is denoted by a red line. And also, they can view a timeline of the drive under the map including uploaded images. They can also play back uploaded movies.

6 Conclusion

This paper introduces a smart city application to share road conditions. The application is based on a mobile sensing framework to collect sensor data reflecting personal-scale, or microscopic, roadside phenomena using crowdsourcing. To motivate users to get involved in the service, it must deliver the useful information or service to them. It, therefore, is quite important to extract useful information from collected data.

In the experiment of detecting road surface conditions, we could obtain the result of about 91% average recall in the \(50\,\mathrm {km/h}\) – \(80\,\mathrm {km/h}\) range. We suppose that the result supports to build a web service that provides detected road conditions from up-to-date crowdsourced mobile sensing application.

We have been improving a method to estimate road conditions. We need to gather more data (and users) to evaluate our methodology and approach.