Budget Cost Reduction for Label Collection with Confusability Based Exploration

Abstract

Subjective judgment is an important manner of collecting labels which are used for training and evaluating models. Researchers always use a large enough and fixed number of workers to label an instance and then aggregate the labels of an instance from multiple workers into a single label. However, some easy instances only need a small number of workers and some difficult instances need more workers to reach stable aggregated labels. Using a fixed number of workers cannot efficiently use the limited budget. We thus propose an approach for reducing the cost of label collection by assigning a dynamic number of workers to each instance. We propose an Exploration-Focused Upper-Confidence Bound (EFUCB) approach which tends to explore the stable aggregated labels for all instances in the entire dataset. It iteratively selects an instance to ask the workers for one more label. To select the instances for labeling, in contrast to using the collected labels as the reward directly, it utilizes the disagreement of workers to measure the reward of collecting labels for an instance. The experiments based on real datasets verify that our approach can reduce the same budget cost with fewer influences on the aggregated labels so that the utility of the collected labels is preserved.