Automatic Classification of Bing Answers User Verbatim Feedback

Abstract

Microsoft Windows products receive huge amount of feedback across different channels. The amount of feedback received monthly around 11 K, which is humanly inefficient to analyze. When user issues a technical query (e.g., connect projector in win 10) on Bing search engine, it shows an answer (e.g., steps to connect to projector in Windows 10) to user query. These answers are created by content author’s team. The triggering team shows the relevant answer for a user query. When users are not satisfied with the answer, they might provide feedback. It is very crucial to understand user feedback to improve user experience. The existing approach to analyze user feedback is to go through each piece of feedback and assign it to right team for resolution. This approach is laborious, expensive and does not scale well. We proposed an approach, which understands user query, answer, and feedback and automatically categorize the verbatim feedback into one of the following three categories: authors, triggering, irrelevant (Junk). The classified feedback is routed to the respective team. We trained a supervised machine learning classifier to perform feedback classification. We have extracted different features from query, answer, and user feedback. Our features composed of bag-of-n-grams extracted from verbatim feedback and deep semantic structured model (DSSM) score between query and answer title. We have achieved 82% classification accuracy using support vector machine (SVM) algorithm. This classifier has been improved over the time. Our approach reduced huge amount of manual work. The proposed solution also helped in reduction of dissatisfaction ration (internal success measure) by 2%, which indicates the enhancement in overall user experience with tech answers.