Why did you pick that? Visualising relevance criteria in exploratory search

  • Ulises Cerviño BeresiEmail author
  • Yunhyong Kim
  • Dawei Song
  • Ian Ruthven


In this article, we present a set of approaches in analysing data gathered during experimentation with exploratory search systems and users’ acts of judging the relevance of the information retrieved by the system. We present three tools for quantitatively analysing encoded qualitative data: relevance-criteria profile, relevance-judgement complexity and session visualisation. Relevance-criteria profiles capture the prominence of each criterion usage with respect to the search sessions of individuals or selected user groups (e.g. groups defined by the users affiliations and/or level of research experience). Relevance-judgement complexity, on the other hand, reflects the number of criteria involved in a single judgment process. Finally, session visualisation brings these results together in a sequential representation of criteria usage and relevance judgements throughout a session, potentially allowing the researcher to quickly detect emerging patterns with respect to interactions, relevance criteria usage and complexity. The use of these tools is demonstrated using results from a pilot-user study that was conducted at the Robert Gordon University in 2008. We conclude by highlighting how these tools might be used to support the improvement of end-user services in digital libraries.


Relevance criteria Exploratory search Information retrieval Literature-based discovery User study Document valuation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barry C.L.: User-defined relevance criteria: an exploratory study. J. Am. Soc. Inform. Sci. 45(3), 149–159 (1994)CrossRefGoogle Scholar
  2. 2.
    Barry C.L., Schamber L.: Users’criteria for relevance evaluation: a cross-situational comparison. Inform. Process. Manage. 34(2–3), 219–236 (1998)CrossRefGoogle Scholar
  3. 3.
    Borlund P.: The concept of relevance in IR. J. Am. Soc. Inform. Sci. Technol. 54(10), 913–925 (2003)CrossRefGoogle Scholar
  4. 4.
    Borlund P.: The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Inform. Res. 8(3), (2003)Google Scholar
  5. 5.
    Cervino Beresi, U.: Related scientific information: a study on user-defined relevance. Ph.D. thesis (2010)Google Scholar
  6. 6.
    Cleverdon, C.W., Mills, J., Keen, E.M.: Factors determining the performance of indexing systems, vol. 1: design, vol. 2: test results. In: Aslib Cranfield Research Project, Cranfield (1966)Google Scholar
  7. 7.
    Ericsson K.A., Simon H.A.: Protocol analysis: verbal reports as data. MIT Press, Cambridge, MA (1993)Google Scholar
  8. 8.
    Mayer, R., Rauber, A.: Establishing context of digital objects’ creation, content and usage. In: Proceedings of the First International Workshop on Innovation in Digital Preservation, Austin, TX (2009)Google Scholar
  9. 9.
    Hurvich L.M., Jameson D.: An opponent-process theory of color vision. Psychol. Rev. 64, 384–404 (1957)CrossRefGoogle Scholar
  10. 10.
    Kelly D.: Measuring online information seeking context, part 2: findings and discussion. J. Am. Soc. Inform. Sci. Technol. 57(14), 1862–1874 (2006)CrossRefGoogle Scholar
  11. 11.
    Kullback S., Leibler R.A.: On information and sufficiency. Ann. Math. Stat. 22, 79–86 (1951)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Lin J.: Divergence measures based on the shannon entropy. IEEE Trans. Inform. Theory 37, 145–151 (1991)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Savolainen R.: The sense-making theory: reviewing the interests of a user-centered approach to information seeking and use. Inform. Process. Manage. 29, 13–28 (1993)CrossRefGoogle Scholar
  14. 14.
    Schamber, L.: Users’criteria for evaluation in a multimedia environment. In: Proceedings of the 54 Annual Meeting of the American Society for Information Science, vol. 28, pp. 126–133 (1991)Google Scholar
  15. 15.
    Wang P., White M. D.: A cognitive model of document use during a research project. Study II. Decisions at the reading and citing stages. J. Am. Soc. Inform. Sci. 50(2), 98–114 (1999)CrossRefGoogle Scholar
  16. 16.
    Ware C.: Color sequences for univariate maps: theory, experiments and principles. IEEE Comput. Graph. Appl. 8(5), 41–49 (1988)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Ulises Cerviño Beresi
    • 1
    Email author
  • Yunhyong Kim
    • 1
  • Dawei Song
    • 1
  • Ian Ruthven
    • 2
  1. 1.The Robert Gordon University, School of ComputingAberdeenUK
  2. 2.Department of Computer and Information SciencesThe Strathclyde UniversityGlasgowUK

Personalised recommendations