Data analysis on video streaming QoE over mobile networks
Abstract
One of recent proposals on standardizing quality of user experience (QoE) of video streaming over mobile network is video Mean Opinion Score (vMOS), which can model QoE of video streaming in 5 discrete grades. However, there are few studies on quantifying vMOS and investigating the relationship between vMOS and other quality of service (QoS) parameters. In this paper, we address this concern by proposing a novel data analytical framework based on video streaming QoE data. In particular, our analytical model consists of Kmeans clustering and logistic regression. This model integrates the benefits of both these two models. Moreover, we conduct extensive experiments on realistic dataset and verify the accuracy of our proposed model. The results show that our proposed framework outperforms other existing methods in terms of prediction accuracy. Moreover, our results also show that vMOS is essentially affected by many QoS parameters such as initial buffering latency, stalling ratio, and stalling times. Our results offer a number of insights in improving QoE of video streaming over mobile networks.
Keywords
Quality of service (QoS) Quality of experience (QoE) Kmeans Logistics regression Video streaming Mobile networksAbbreviations
 DL
Download
 E2E
Endtoend
 LR
Logistic regression
 QoS
Quality of service
 QoE
Quality of experience
 Sig
Significance
 SVGOP
SpeedVideo global operating platform
 SVM
Support vector machine
 vMOS
video mean opinion score
 Wal
Wald test value
1 Introduction
Video streaming is becoming one of the most popular services over mobile networks. It is predicted in [1] that the traffic caused by video streaming will occupy more than 77% of all consumer Internet traffic by 2021, among which mobile video traffic will be more than 55% of all video traffic. The growing demands on video streaming over mobile networks inevitably lead to the challenges in optimizing network resource in order to improve the user perceptual experience. Many previous studies mainly focus on improving quality of service (QoS) of video streaming over mobile networks. Typical QoS measures include throughput, bandwidth, outage, jitter, and delay [2]. However, most of these QoS metrics fail to characterize user perceptual experience, which is also called quality of experience (QoE). It is more crucial to conduct video quality assessment from QoE than that from QoS [3, 4] because (i) enhancing QoS does not directly improve QoE [5] and (ii) only improving QoS sometimes significantly increases operating expenditure, consequently decreasing the profit of service providers [6].
Therefore, QoE improvement of video streaming over mobile networks has received extensive attention recently. In particular, the work of [7] investigates QoEdriven crosslayer optimization for video transmission in wireless networks. Ramamurthi et al. [8] propose a resource management scheme at network core in wireless networks to improve video QoE. The work of [9] presents a largescale measurementbased study on the effects of Internet path selection in video QoE and offers several QoE enhancement schemes.
However, the prerequisite of QoE improvement of video streaming is to quantify QoE appropriately. Video QoE assessment schemes can be generally categorized into subjective tests, objective assessments, and datadriven analysis [3]. Compared with subjective tests and objective assessments, datadriven analysis is more promising due to the availability of massive datasets and the accuracy of characterizing user perception while overcoming the drawbacks of subjective tests and objective assessments (such as high cost and insufficient human visual system knowledge). In particular, the work of [6] proposes a datadriven model to quantify the metrics affecting video QoE. Jiang et al. [10] improve video QoE by exploiting datadriven QoE prediction. The work of [11] improves the video bitrate adaptation based on datadriven QoE prediction. Huang et al. [12] propose a dynamic adaptive streaming via HTTP to optimize user QoE. Ref. [13] presents a MultiConstraint QualityofExperience (QoE) centric Routing (MCQR) scheme to improve user video streaming QoE over mobile networks. The work of [14] presents a queuebased model to analyze the video buffer using discretetime analysis. Marai et al. [15] propose a clientserver cooperationbased approach to achieve efficiency, fairness, and stability of video adaptive streaming.
Mean Opinion Score (MOS) evaluation scale
Score  Quality  Distortion  Class 

1  Bad  Very annoying but objectionable  0 
2  Poor  Annoying, but not objectionable  
3  Fair  Perceptible and slightly annoying  1 
4  Good  Just perceptible, but not annoying  
5  Excellent  Imperceptible 
Therefore, this paper aims to conduct datadriven QoE analysis on vMOS. In particular, we obtain a realistic dataset on video QoE based on SpeedVideo Global Operating Platform (SVGOP) established by Huawei. This dataset has the following unique characteristics: (1) heterogeneous data types, (2) positive/negative correlations, and (3) dependence of features; these characteristics result in the difficulties in analyzing video QoE data.
To address the above concerns, we propose a datadriven analysis framework to analyze the relationship between vMOS and other QoS parameters. Although our previous work [20] presented preliminary results on quantifying vMOS and other QoS parameters, this study is significantly different from our previous work in the following aspects: (1) we conduct a data preliminary analysis on vMOS data; this analysis has been ignored in our previous work; (2) we propose a novel analytical framework in this paper, which is significantly different from the previous work; and (3) experiment results have shown that our proposed model can improve the predication accuracy than our previous work.
In addition, this paper has the following research contributions in contrast to other existing studies: (i) our model consists of Kmeans clustering approach and logistic regression; the combination of these two approaches can greatly improve the predication accuracy; (ii) we have conducted extensive experiments on three training datasets and one testing dataset and the experiment results show that our proposed model outperforms other existing methods in terms of predication accuracy; and (iii) moreover, our results also imply that a small set of QoS parameters play an important role in determining vMOS.
The remainder of this paper is organized as follows. Section 2 describes the data used in this paper and identifies the challenges. We then present the overview of our method in Section 3. Section 4 presents the experimental results. Finally, we conclude this paper in Section 6.
2 Data description
Description of dataset
Types  Features 

QoS parameters  Average rate of playing phase (kbps) 
Video total download (DL) rate (kbps)  
Video bitrate (kbps)  
Initial max DL rate (kbps)  
EndtoEnd (E2E) roundtrip time (RTT) (ms)  
Initial buffering latency (ms)  
Video Initial buffer downloaded (byte)  
Playing time (ms)  
Playing total duration  
Stalling times  
Stalling ratio  
Scoring factors  vMOS 

Heterogeneous data types. The preliminary results show that QoS parameters are in different types and different ranges. For example, the initial buffering latency is ranging from 500 ms (millisecond) to 30,000 ms while the average rate of play phase is ranging from 300 kbps (kilobit per second) to 16,000 kbps.

Positive/negative correlations of QoS parameters. As shown in the statistics results, we observe that there are positive or negative correlations between vMOS and other QoS parameters. For example, there is a positive correlation between vMOS and video total DL rate while there is a negative correlation between vMOS and initial buffering latency.

Dependence on QoS parameters. The preliminary statistics results also show that vMOS is essentially affected by multiple factors, such as average rate of playing phase, video total DL rate, endtoend (E2E) roundtrip time, and initial buffering latency.
The above characteristics result in the difficulties in analyzing video QoE data. To address the above challenges, we propose a novel datadriven QoE analysis framework (which will be described in details in Section 3).
3 QoE analysis framework
3.1 Data preprocessing
 Positive values:$$ u_{ij}=\frac{x_{ij}\min\left(x_{ij}\right)}{\max\left(x_{ij}\right)\min\left(x_{ij}\right)}, $$(1)
 Negative values:$$ u_{ij}=\frac{\max\left(x_{ij}\right)x_{ij}}{\max\left(x_{ij}\right)\min\left(x_{ij}\right)}, $$(2)
where x_{ij} represents the original value, u_{ij} represents the value after normalization, min(·) is the minimum value, and max(·) is the maximum value.
3.2 Kmeans + LR method
In this paper, we use logistic regression (LR) mainly to predict whether a user’s QoE of video streaming is “good” or “bad” since it is hard to determine it based on a continuous vMOS value within [ 1,5]. However, logistic regression requires that the dependent variable is dichotomou (binary). Therefore, we first exploit Kmeans to categorize the sample datasets into two groups according to “good” or “bad”. We then use logistic regression to predict the QoE of video streaming.
3.2.1 Kmeans clustering
The main idea of Kmeans algorithm [21] is to find a partition such that squared error between the empirical mean of a cluster and the points in the cluster is minimized. In particular, given a dataset X={x_{i}}, i=1,2,...,m, we partition them into K disjoint clusters so that the sum of the intracluster variances is minimized. We denote the K disjoint clusters by C=C_{1},C_{2},...,C_{k}.
where μ_{k} is the mean of cluster C_{i}.
This clustering process can be completed by alternating between assigning instances to their closest centers and recomputing the centers until a local minimum reaches.
3.2.2 Logistic regression
In this paper, we concern with a binary classification problem of categorizing QoE of a video streaming into two cases of “bad” and “good”. To solve this problem, we exploit logistic regression (LR). In particular, we denote the QoE by a binarydependent variable. This variable only takes two values, either “0” or “1”. Specifically, we use “1” to represent “good” and “0” to represent “bad”. Without loss of generality, we classify a data sample as “bad” when its vMOS score is within [ 1,2] and “good” when its vMOS score is within [ 3,5], as shown in Table 1.
where β_{0} is the offset and β_{i} (i=1,...,n) is the corresponding regression coefficient for each QoS parameter.
where p∈ [ 0,1]. Since the logistic regression model is nonlinear, the maximum likelihood estimation method can be used to estimate the regression coefficient β_{i} (i=0,...,n).
4 Empirical study
As we summarize in Section 2, the dataset has the characteristics such as different types, positive and negative correlations, and dependence. Therefore, we need to use the proposed method to address these concerns. In particular, we describe the experiment settings in Section 4.1. We then show the intermediate results of Kmeans clustering and logistic regression in Section 4.2. We next compare our proposed method with other existing methods in terms of predication accuracy in Section 4.3. Finally, we conduct performance analysis of our proposed method in Section 4.4.
4.1 Experiment settings
Meta data information
Description  No. of samples  No. of features 

Dataset 1  30,000  11 
Dataset 2  30,000  11 
Dataset 3  26,984  11 
Summary of features
Types  Features  Variables 

QoS parameters  Average rate of playing phase (kbps)  x _{1} 
Video total download (DL) rate (kbps)  x _{2}  
Video bitrate (kbps)  x _{3}  
Initial max DL rate (kbps)  x _{4}  
EndtoEnd (E2E) roundtrip time (RTT) (ms)  x _{5}  
Initial buffering latency (ms)  x _{6}  
Video Initial buffer downloaded (byte)  x _{7}  
Playing time(ms)  x _{8}  
Playing total duration  x _{9}  
Stalling times  x _{10}  
Stalling ratio  x _{11} 
4.2 Kmeans LR method
4.2.1 Kmeans Analysis
Kmeans clustering analysis
Dataset  Classification standard  QoE  No. of samples 

Dataset 1  3.90  Bad  10,306 
Good  19,694  
Dataset 2  3.90  Bad  19,833 
Good  10,167  
Dataset 3  3.93  Bad  9,199 
Good  17,785 
4.2.2 Logistics regression analysis
where x_{i} (i=1 to 11) corresponds to each of 11 features as given in Table 4 and β_{i} is the regression coefficient.
Regression coefficients
Coefficient  Dataset 1  Dataset 2  Dataset 3  

Value  Wal  Sig  Value  Wal  Sig  Value  Wal  Sig  
β _{1}  34.43  0.005  0.943  − 15.51  0.272  0.602  − 11.094  0.214  0.643 
β _{2}  − 12.65  0.001  0.98  9.88  0.103  0.748  12.21  0.245  0.621 
β _{3}  105.18  7.374  0.007  1.721  38.976  0  2.263  75.108  0 
β _{4}  − 18.403  0.27  0.599  − 0.632  0.129  0.72  − 0.736  0.152  0.697 
β _{5}  15.89  0.41  0.522  0.313  0.074  0.785  0.943  0.585  0.444 
β _{6}  112529  7.47  0.006  2156.26  404.857  0  2598.501  408.67  0 
β _{7}  − 14.49  0.013  0.909  − 4.778  0.312  0.576  4.1  0.454  0.501 
β _{8}  211762  2.37  0.123  3612.752  6.299  0.012  4204.75  5.961  0.015 
β _{9}  − 3252.45  2.359  0.125  − 56.298  6.409  0.011  − 64.055  5.813  0.016 
β _{10}  − 1911.44  0.265  0.607  19.609  0.062  0.803  − 45.572  4.02  0.045 
β _{11}  − 207352  2.369  0.124  − 3553.314  6.368  0.012  − 4158.129  4.132  0.042 
It is shown in Table 6 that coefficient β_{6} (corresponding to initial buffering latency) has the most significant impact on vMOS. Similarly, we can find that coefficient β_{3} (corresponding to video bitrate) also has a strong influence on vMOS because of high Wal value (i.e., 7.374).
4.3 Performance comparison
We next present the experimental results over the given datasets with performance comparison with conventional traditional methods including multivariate linear regression, logistic regression, support vector machines, Knearestneighbor and Naive Bayes [24].
Performance comparison
Method  Precision  

Dataset 1  Dataset 2  Dataset 3  
Multivariate linear regression  72.55%  72.51%  71.51% 
Logistics regression  74.93%  77.55%  83.55% 
SVM  82.56%  87.17%  85.42% 
Knearestneighbor  52.48%  61.34%  56.77% 
Naive Bayes  88.12%  83.47%  91.09% 
Kmeans + LR (proposed)  96.94%  97.13%  97.54% 
4.4 Performance analysis of our method
4.4.1 Residual plots
4.4.2 Accuracy analysis
where TP is true positive, FP is false positive, FN is false negative, and TN is true negative.
D value and ranking
Features  Dataset 1  Dataset 2  Dataset 3  

D value  Ranking  D value  Ranking  D value  Ranking  
Average rate of playing phase  0.0605  7  0.0662  7  0.0667  6 
Video total DL rate  0.0596  8  0.0651  8  0.0653  8 
Video bitrate  0.1848  3  0.1620  3  0.2160  3 
Initial max DL rate  0.3008  2  0.2997  2  0.3155  2 
E2E RTT  0.5409  1  0.5186  1  0.5602  1 
Initial buffering latency  0.0222  11  0.0222  11  0.0199  11 
Video initial buffer downloaded  0.1020  4  0.1025  4  0.1033  4 
Playing time  − 0.0668  6  − 0.0667  6  − 0.0657  7 
Playing total duration  − 0.0814  5  − 0.0741  5  − 0.0695  5 
Stalling times  − 0.0231  10  − 0.0229  10  − 0.0279  9 
Stalling ratio  − 0.0232  9  − 0.0231  9  − 0.0224  10 
In summary, we can see from Table 8 that initial buffering latency, stalling times, and stalling ratio can significantly affect vMOS. In particular, initial buffering latency, stalling times, and stalling ratio are negatively correlated with vMOS. We have the following major findings: (1) vMOS is affected by multiple QoS parameters together. Essentially, vMOS is affected by 11 QoS parameters as given in Table 4. (2) Small set of QoS parameters dominates the performance of vMOS. Interestingly, we observe that a small set of QoS parameters has the stronger influence on vMOS than other QoS parameters.
5 Discussion and future work

Concentrating on optimizing several dominating QoS parameters in vMOS. For example, we may focus on optimizing the network resource to reduce initial buffering latency, stalling times, and stalling ratio so that we can significantly improve the video QoE while maintaining relatively low operating expenditure. However, it is not an easy task to achieve this goal because the enhancement of these QoS parameters is also involved with many other technologies, such as crosslayer optimization and distributed resource allocation [10, 25, 26].

Identifying QoS bottlenecks. Determining QoS bottlenecks can help to enhance system performance and consequently improving vMOS. However, it is also difficult to identify the QoS bottlenecks since they are often affected by many factors. For example, video stalling is essentially caused by many factors, such as network congestion, network failure, device mobility, and radio spectrum scarcity. There is a challenge in identifying the causality of stalling. In the future, we may apply datadriven approach to identify the reason behind video stalling according to different scenarios.

Distributing the videos appropriately to improve vMOS. For example, we can distribute the most popular videos at the servers close to users so that we can significantly reduce the initial buffering latency. However, to determine the popularity of video streaming is challenging since it requires the extensive efforts in analyzing the massive video data [27].
6 Conclusions
In this paper, we propose a novel data analysis model to analyze video Mean Opinion Score (vMOS), which is an important measure of user quality of experience of video streaming. In particular, our proposed model is a combination of Kmean clustering method and logistic regression method, which can essentially improve the prediction accuracy than other existing methods. We conduct experiments over several realistic datasets. Extensive experiment results show that our proposed method outperforms other existing methods in terms of prediction accuracy. For example, our proposed method has the precision of 96.94, 97.13, and 97.54% on dataset 1, dataset 2, and dataset 3. Our results also show that a small set of QoS parameters play an important role in determining vMOS; this implies that we can concentrate on enhancing these key QoS parameters. It can be achieved by integrating crosslayer optimization and distributed resource allocation schemes together and mitigating QoS bottlenecks.
Our model has a broad range of applications. For example, it can be used to enhance the QoE of video service providers (such as Netflix and YouTube), videocentric mobile applications (including Facebook LIVE, Instagram LIVE, Snapchat, etc.), video game live streaming services (such as Twitch, Hitbox and NetEase Game Lives). Moreover, it can be used to improve the usability of video surveillance systems. For example, the quality of video streaming of video surveillance systems can be helpful in detecting dangers in advance.
Footnotes
Notes
Acknowledgements
The authors would like to express their appreciation for Gordon K.T. Hon for his thoughtful discussions.
Funding
The work described in this paper was partially supported by Macao Science and Technology Development Fund under Grant No. 0026/2018/A1, the National Key R&D Program of China under Grant No. 2016YFB0201900, the National Natural Science Foundation of China (NSFC) under Grant No. 61572538 and Grant No. 61672170, the Fundamental Research Funds for the Central Universities under Grant No. 17LGJC23, the NSFCGuangdong Joint Fund under Grant No. U1401251 and Guangdong Science and Technology Plan with Grant No. 2015B090923004. All the funding bodies are involved with the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.
Authors’ contributions
QW proposed the idea, derived the results, and wrote the paper. HND supervised the work and revised versions. DW gave valuable suggestions on the motivation of conducting data analysis on video streaming over mobile networks. HX contributed to motivating, revising and proofreading of the article. All authors read and approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
 1.Cisco, Visual Network Index. “forecast and methodology, 20152020”. White Paper, 1–41 (2016).Google Scholar
 2.M Venkataraman, M Chatterjee, Inferring video QoE in real time. IEEE Netw. 25(1), 4–13 (2011).CrossRefGoogle Scholar
 3.Y Chen, K Wu, Q Zhang, From QoS to QoE: A tutorial on video quality assessment. IEEE Commun. Surv. Tutorials. 17(2), 1126–1165 (2015).CrossRefGoogle Scholar
 4.H Nam, KH Kim, H Schulzrinne, in IEEE INFOCOM. QoE matters more than QoS: Why people stop watching cat videos (IEEENew Jersey, 2016), pp. 1–9.Google Scholar
 5.V Joseph, G de Veciana, in IEEE INFOCOM. NOVA: QoEdriven optimization of DASHbased video delivery in networks (IEEENew Jersey, 2014), pp. 82–90.Google Scholar
 6.A Balachandran, V Sekar, A Akella, S Seshan, I Stoica, H Zhang, in Proceedings of the ACM SIGCOMM Conference. Developing a Predictive Model of Quality of Experience for Internet Video (ACMNew York, 2013).Google Scholar
 7.S Thakolsri, W Kellerer, E Steinbach, in Proceedings of IEEE International Conference on Communications (ICC). QoEBased CrossLayer Optimization of Wireless Video with Unperceivable Temporal Video Quality Fluctuation (IEEENew Jersey, 2011).Google Scholar
 8.V Ramamurthi, O Oyman, J Foerster, in IEEE Global Communications Conference (GLOBECOM). VideoQoE aware resource management at network core (IEEENew Jersey, 2014), pp. 1418–1423.Google Scholar
 9.M Venkataraman, M Chatterjee, Effects of internet path selection on videoQoE: analysis and improvements. IEEE/ACM Trans. Networking (TON). 22(3), 689–702 (2014).CrossRefGoogle Scholar
 10.J Jiang, V Sekar, H Milner, D Shepherd, I Stoica, H Zhang, in 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI). CFA: A Practical Prediction System for Video QoE Optimization (Berkeley, 2016), pp. 137–150.Google Scholar
 11.Y Sun, X Yin, J Jiang, V Sekar, F Lin, N Wang, et al, in Proceedings of ACM SIGCOMM. CS2P: Improving video bitrate selection and adaptation with datadriven throughput prediction (ACMNew York, 2016), pp. 272–285.Google Scholar
 12.W Huang, Y Zhou, Xie X, D Wu, M Chen, E Ngai, Buffer State is Enough: Simplifying the Design of QoEAware HTTP Adaptive Video Streaming. IEEE Trans. Broadcast. 64(2), 1–12 (2018).CrossRefGoogle Scholar
 13.C Lal, V Laxmi, MS Gaur, M Conti, Enhancing QoE for video streaming in MANETs via multiconstraint routing. Wirel. Netw. 24(1), 235–256 (2018).CrossRefGoogle Scholar
 14.V Burger, T Zinner, L DinhXuan, F Wamser, P TranGia, A Generic Approach to Video Buffer Modeling Using DiscreteTime Analysis. ACM Trans Multimedia Comput Commun Appl. 14(2s), 33:1–33:23 (2018).CrossRefGoogle Scholar
 15.OE Marai, T Taleb, M Menacer, M Koudil, On Improving Video Streaming Efficiency, Fairness, Stability, and Convergence Time Through ClientServer Cooperation. IEEE Trans. Broadcast. 64(1), 11–25 (2018).CrossRefGoogle Scholar
 16.Huawei, “Requirements of Mobile Bearer Network for Mobile Video Service”. White Paper, 1–8 (2016).Google Scholar
 17.International Telecommunication Union (ITU), “Subjective video quality assessment methods for multimedia applications”. ITU Recommendation:1–37.Google Scholar
 18.W Pan, G Cheng, H Wu, Y Tang, in IEEE/ACM 24th International Symposium on Quality of Service (IWQoS). Towards QoE assessment of encrypted YouTube adaptive video streaming in mobile networks (IEEENew Jersey, 2016).Google Scholar
 19.CM Lentisco, L Bellido, QJCCE Pastor, JLA H, QoEBased Analysis of DASH Streaming Parameters Over Mobile Broadcast Networks. IEEE Access. 5:, 20684–20694 (2017).CrossRefGoogle Scholar
 20.Q Wang, HN Dai, H Wang, D Wu, in The 15th IEEE International Symposium on Parallel and Distributed Processing with Applications. DataDriven QoE Analysis on Video Streaming in Mobile Networks (IEEENew Jersey, 2017).Google Scholar
 21.X Wu, V Kumar, JR Quinlan, J Ghosh, Q Yang, H Motoda, et al, Top 10 algorithms in data mining. Knowl. Inf. Syst. 14(1), 1–37 (2008).CrossRefGoogle Scholar
 22.J Friedman, T Hastie, R Tibshirani, The elements of statistical learning. vol. 1. Springer series in statistics New York (2001).Google Scholar
 23.RR Wilcox, Introduction to robust estimation and hypothesis testing, Academic press (Elsevier, MA, USA, 2011).Google Scholar
 24.E Alpaydin, Introduction to machine learning (MIT press, MA, USA, 2014).zbMATHGoogle Scholar
 25.Y Wang, X Lin, Userprovided networking for QoE provisioning in mobile networks. IEEE Wirel. Commun. 22(4), 26–33 (2015).CrossRefGoogle Scholar
 26.Z Su, Q Xu, Q Qi, Big data in mobile social networks: a QoEoriented framework. IEEE Netw. 30(1), 52–57 (2016).CrossRefGoogle Scholar
 27.Z Su, Q Xu, F Hou, Q Yang, Q Qi, Edge Caching for Layered Video Contents in Mobile Social Networks. IEEE Trans. Multimed. 19(10), 2210–2221 (2017).CrossRefGoogle Scholar
Copyright information
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.