Automotive Innovation

, Volume 2, Issue 2, pp 121–126 | Cite as

Scale Consistency Quantification for Subjective Evaluation of Vehicle Dynamics

  • Wan’an YangEmail author


The subjective evaluation of vehicle dynamics performance is widely applied in different stages of vehicle development. However, the rating result has frequently been challenged because it is easily affected by various subjective and objective factors. Currently, there is no suitable index for determining evaluator’s consistency when performing a subjective evaluation of vehicle dynamics. This evaluation is quite unique, with limited samples, multiple indices, and poor repeatability, in addition to being stratified and two dimensional. The cross-grouped factor analysis (CFA) method is proposed to identify the scale for a subjective evaluation and to quantify its consistency. An application case study revealed that the proposed method is effective.


Subjective evaluation Consistency Quantification Cross-grouped factor analysis 

1 Introduction

The subjective evaluation of vehicle dynamics performance is important in vehicle development and is performed from the program initial benchmark ride to the final post-SOP (start of product) ride. This evaluation is conducted by tuning engineers or customers to understand the performance of competitive vehicles and the actual physical vehicles at different development stages. The evaluation results inform the next development direction through product target setting and design status review.

Nowadays, digitalization, accompanying intelligent driving, internet connectivity, and new energy vehicle, is becoming a very hot topic in vehicle development. Because it is an established field, many processes in the design of vehicle dynamics have already been digitalized. With regard to the modeling of vehicle parts (suspension, tire, steering, etc.), driver, road surface, and load cases (kinematic and compliance test, ride and handling test, etc.), digitalization has reached a somewhat high precision level and can provide simulation results with high confidence.

Although evaluation is a key issue with regard to vehicle development, it still relies on subjective processes, which are very laborious for professional development engineers. Moreover, only a few indexes can be simulated under certain conditions. Owing to the complexity and coupling of the vehicle dynamics performance items, it is very difficult to form an effective index by conducting objective tests. Additionally, studies have shown that none of current indexes can cover all performance features.

Most subjective evaluation studies have focused on several areas including the evaluation consistency among the group members. Xu and Liu [1] have explained the difference between the interrater agreement and interrater reliability, which may lead to a misuse when aggregating the subjective evaluation into higher level units. By using factor analysis (FA) on the survey results obtained from raters who scored the writing test in the national examination, Li [2] evaluated their scoring process and found that the raters’ own characteristics were affected by six factors. Many studies have been conducted on the correlation between a subjective evaluation and an objective test. Liu [3] built an optimal correlation model to determine the significant orders of several handling indexes (snake test for example) affecting the evaluation results. Yang et al. [4] used neural networks to obtain the correlation directly between the subjective evaluation and the test data obtained when a vehicle passed a step bump. Various studies have focused on weight estimation for multi-objective decisions. Chen et al. [5] proposed a threshold aggregation method for the rank correlation coefficient in the case wherein the subjective evaluation is inconsistent with the test result. However, they did not investigate the consistency of the subjective evaluators themselves. Guo et al. [6] improved the weight determination method by efficiently combining the entropy weight method and the analytic hierarchy process for multi-decisions, in an attempt to reduce the potential unbalanced results. Xu and Zhao [7] developed an evaluation system using the factor analysis method and the analytic hierarchy process to analyze the factors affecting the city home endowment availability.

The abovementioned studies try to achieve a better correlation between the subjective evaluation and the test result, but few of them investigated the personal evaluation scale and its quantitative consistency. The subjective evaluation of vehicles dynamics is relatively unique, stratified, limited samples, and multiple indexes. Thus, it is hard to maintain ideal repeatability and high accuracy, owing to the personal scale stability problem. Thus, the evaluation results have always been challenged during the engineering development. It is a common sense that we must accept that the evaluation results are only comparable within one activity. Hence, to avoid potential conflicts owing to personal scale inconsistencies, a comparison with previous evaluation results is not recommended.

A breakthrough for vehicle dynamics performance design will be achieved if the digitalization of subjective evaluation is realized. In this way, the vehicle performance could be outlined in earlier development phase, which is very helpful in high-precision target setting. This imaginative objective can be achieved by following the key path shown in Fig. 1. Notably, the stability of the evaluation scale is the most fundamental condition.
Fig. 1

Key path for digital tuning of vehicle dynamics

The subjective evaluation results must definitely be unique when attempting to determine the correlation through an objective test. The most typical approach toward this end is to ask several professional engineers to rate a series of performance items for a group of vehicles and summarize the ratings into a common conclusion according to the group average. Moreover, all evaluation members must have a relatively similar taste with regard to the vehicle dynamics performance, after having worked together for a long period. Furthermore, each of them should ensure that their evaluation scale, either short term or long term, is not excessively affected by the vehicle, date, or evaluation site.

However, research results suggest that it is difficult to keep a person’s subjective evaluation scale stable when it is disturbed by a large amount of influences. Apart from the evaluation method, process, environment, performance, etc., it can be affected by a person’s responsibility, mood, self-confidence, experience, persuasion, and on-site control even in subjective terms. These factors tend to produce various negative effects on the stability of the evaluation scale.

In fact, a suitable method of checking the consistency of the personal evaluation scale does not exist. This is an important issue, which has prevented the achievement of a better correlation. This paper introduces a novel method of determining an index capable of characterizing personal features when conducting vehicle dynamics performance evaluation. Additionally, the scale consistency is measured.

2 Cross-grouped Factor Analysis

2.1 Factor Analysis

From experience, the vehicle ride and handling performance are known to be interrelated. Thus, it is difficult to quantitatively identify the correlation on an item-by-item basis, even for a skilled person. By using factor analysis (FA), which is an advance statistical method for data mining, the factors determining the rating score for each vehicle can be easily revealed.

In advance statistics, FA is one of several tools for data reduction and can effectively reduce the dimension of messy and complex data to determine the correlation and factors behind the parameters. The key concept is to interpret the raw data as a linear combination of various common and unique factors, and to obtain the most effective dimension reduction at the balance of minimizing the information loss and reducing more dimensions, that is, the cumulative percentage of the extraction sums of the squared loading in the correlation matrix.

The raw data are set as matrix X. (The parameters X1–X7 described below are set in the columns, and the cases are set in the rows.) It is assumed that these data are the combination of common factors and unique error factors; that is, X = AF + ε, where F represents the common factors, ε denotes the unique error factors to be abnegated, and A denotes the component load matrix (factors in columns, component loads in rows). Data reduction is possible by decoupling the covariance matrix as the eigenvalue λ, and the corresponding eigenvector β as the factors.
$$X^{\text{T}} X = \left[ {\sqrt \lambda_{1} \beta_{1} , \sqrt \lambda_{2} \beta_{2} , \ldots ,\sqrt \lambda_{n} \beta_{n} } \right]\left[ {\begin{array}{*{20}c} {\begin{array}{*{20}c} {\sqrt \lambda_{1} \beta_{1} } \\ {\sqrt \lambda_{2} \beta_{2} } \\ \ldots \\ \end{array} } \\ {\sqrt \lambda_{n} \beta_{n} } \\ \end{array} } \right]$$

There are two common data reduction approaches, namely balancing the reduction efficiency and information loss. It is suggested to keep eigenvectors that have either an eigenvalue greater than 1 or a variance explanation accumulation greater than 85%. This depends on the purpose of applying FA. The eigenvectors with an eigenvalue of less than 1 can still be kept if we accept a less data reduction. The same situation occurs when keeping a higher percentage of variance explained. In this case, all eigenvectors β with an eigenvalue λ greater than 1 are kept as common factors, while the remaining eigenvectors are ignored.

The effectiveness of FA is based on the inner structure of the raw data and is applicable to data with a relatively stronger potential relationship among the parameters. The raw data used in this study had a very ideal distribution and provided good results when subjected to a Bartlett test of sphericity. Hence, FA was applied for further analysis.

The items considered in the vehicle dynamics subjective evaluation are as follows: X1 (overall ride, weighted from its three following sub-items) with X2 (body motion, vertical motion of the vehicle body, and motion smoothness), X3 (isolation includes rolling isolation on different roads and impact hardness when passing a bump) and X4 (shake includes shake from powertrain, unsprung mass, structure, and component, when driving on rough roads) as its sub-items, and X5 (Over Handling, weighted from its two following sub-items) with X6 (steering includes response, torque, friction, and parking effort) and X7 (cornering includes steering response linearity and vehicle roll behavior) as its sub-items. Each person evaluated several selected vehicles. Table 1 presents the raw data during a program benchmark ride on proving ground. Each of the six evaluators (in this case, four professional evaluators vs. two normal engineers) drove seven vehicles one by one, at the same speed, on the ride and handling track, which hosts different rough road events. The subjective feeling was scored item by item from 1 (worst) to 10 (best), immediately after each vehicle was driven.
Table 1

Example of raw evaluation data summary

FA was applied to the raw data of each evaluator. The factors were extracted using the principal components method on its covariance matrix.

Table 2 presents an example of the analysis result for one evaluator. As can be seen, the first two factors (i.e., Factor 1 and Factor 2) with an eigenvalue greater than 1 could explain more than 94% of the variance for all data.
Table 2

Total variance explained of one evaluator’s data

Component (factor)

Initial eigenvalues


% of variance

Cumulative  %





























Similar results were obtained when the FA was applied to the remaining evaluator data. This means that the vehicle dynamics performance parameters X1–X7 were decided only by two factors, which agrees with our experience. In the development of vehicle dynamics, all level 1–3 performance indexes were classified into two major categories (ride or handling) and were mutually influenced owing to the suspension architecture, part performance, and component structure, among other elements. However, they could generally be considered as two independent factors.

Figure 2 shows the factor component load map obtained by considering the first two factors as the X and Y axes, respectively, for the data of one evaluator. This suggests that all loads (overall load or sub-item loads) were distributed in a narrow area close to the circular arc with a radius of 1, which means a high statistics interpretation on data information. The component load angle α between overall ride and overall handling represents the perception relationship with the ride and handling performance and was defined as the key characteristic parameter.
Fig. 2

Factor component load map and key characteristic parameter α

2.2 Cross-grouped FA

To determine the personal scale consistency for a subjective evaluation, the cross-grouped FA was introduced for further investigation.

First, the subgroup was built by extracting one vehicle’s scores from the total data, as shown in Fig. 3. For example, the first subgroup consisted of raw data for vehicles No. 1, 2, 3, and 4, but not for vehicle No. 5.
Fig. 3

Cross-grouping in raw data

Secondly, the same approach was used to build the remaining subgroups; therefore, the total number of subgroups was the same as the number of vehicles.

Finally, to obtain the corresponding key characteristic parameter α for each subgroup of a single evaluator, FA was applied to these cross-grouped data comprised by several vehicle scores awarded by one person.

3 Application

The FA was applied to the subgrouped data to obtain the key characteristic parameter α. Changes were not observed even when Varimax orthogonal rotation was selected. Figure 4 shows the analysis results obtained when FA was applied to the subgrouped data of one evaluator and includes the factor component load and the corresponding characteristic parameters.
Fig. 4

Characteristic parameters and mean and deviation of one evaluator

The cross-grouped FA was applied on the data of all six evaluators. The result reveals that the deviation of the characteristic parameter α of the professional evaluators was obviously smaller than that of normal engineers and that their average values were irregular. There is no direct physical meaning for the factors or for the angle between overall ride and overall handling. However, angle α is strongly correlated with the scale of the personal subjective evaluation. The angle change implies the effect of the evaluator’s mental disturbance on the evaluation scale. By observing this angle change, the scale consistency could be quantized. Therefore, in the case wherein either different vehicles are evaluated at each time or the same vehicle is evaluated at different times, the deviation of the characteristic parameter α is considered as a personal scale consistency index for the subjective evaluation of vehicle dynamics (Fig. 5).
Fig. 5

Comparison of descriptives between profession and normal evaluators

4 Conclusion

This paper introduced a novel method of scale consistency quantification for the subjective evaluation of vehicle dynamics. Using an advanced statistical analysis method, namely factor analysis, the key characteristics parameter was defined to represent the evaluation scale, and its deviation from the cross-grouped FA analysis could be quantitatively determined as a scale consistency index.


Compliance with Ethical Standards

Conflict of interest

On behalf of all authors, the corresponding authors state that there is no conflict of interest.


  1. 1.
    Xu, X.F., Liu, Y.: The research and application of interrater agreement. Psychol. Sci. 30(5), 1175–1178 (2007)Google Scholar
  2. 2.
    Li, B.: The analysis of raters’ own characteristics influencing subjective scoring agreement. J Shanxi Univ (Philos Soc Sci) 33(3), 75–79 (2010)Google Scholar
  3. 3.
    Liu, D., Wang, S.C., Chen, J.J., et al.: Correlation between subjective and objective evaluation data of vehicle handling stability based on snake test. Energy Conserv Environ Prot Transp 13(4), 32–35 (2017)Google Scholar
  4. 4.
    Yang, W.A., Wang, Q., Wang, F.: Correlation of subjective evaluation and objective tests for impact hardness/memory shake when vehicles passing speed bumpers. Noise Vib Control 33(6), 82–86 (2013)Google Scholar
  5. 5.
    Chen, M., Gu, H.J., Zhao, W.: Subjective and objective evaluation method of vehicle based on inconsistency. J Chongqing Univ Technol (Nat Sci) 30(4), 16–20 (2016)Google Scholar
  6. 6.
    Guo, J.W., Pu, X.Q., Gao, X., et al.: Improved method on weights determination of indexes in multi-objective decision. J Xidian Univ 41(6), 118–125 (2014)Google Scholar
  7. 7.
    Xu, L., Zhao, M.X.: City home endowment service availability evaluation system: based on factor analysis and analytic hierarchy process. J Northwest Univ (Philos Soc Sci Ed) 47(6), 63–71 (2017)Google Scholar

Copyright information

© China Society of Automotive Engineers (China SAE) 2019

Authors and Affiliations

  1. 1.Pan Asia Automotive Technical Center Co. Ltd.ShanghaiChina

Personalised recommendations