U-Measure

Definition

U-measure is a versatile evaluation measure. For a given textual information access task, U-measure first builds a trailtext, which is a concatenation of all the texts that the user has read (which may be constructed under some user behaviour model or based on some real evidence such as clicks and eyetracking or mousetracking data). Then, U-measure computes a score using a decay function based on the positions of relevant items within the trailtext. Formally, a trailtext tt is a concatenation of n strings: tt = s ₁ s ₂ …s _n . The strings may be documents, parts of documents, snippets, sentences, or any other fragments of text that have been read. The offset position of s _k (1 ≤ k ≤ n) is defined as pos(s _k ) = ∑ _{j = 1} ^k | s _j |, where the length of each string is measured in terms of the number of characters. Furthermore, the position-based gain is defined as g(pos(s _k )) = 0 if s _k is considered nonrelevant and g(pos(s _k )) = gain _l if its relevance level is l. For...