Habitability In Question-Answering Systems

  • William Ogden
  • James Mcdonald
  • Philip Bernick
  • Roger Chadwick
Part of the Text, Speech and Language Technology book series (TLTB, volume 32)

This chapter discusses our work in evaluating question-answering systems. We begin by reviewing evaluation concepts and methods that have been used in the past. We then consider more recent efforts to evaluate search engines that have contributed to the techniques we currently use. Finally we discuss our techniques and discuss results from user studies we have conducted. Implications are discussed for the future design of habitable question-answering systems.


Natural Language Search Engine Information Retrieval System Question Answering Start System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

6. References

  1. Bates, Marcia J. “An Exploratory Paradigm for Online Information Retrieval.” Intelligent Information Systems for the Information Society. Ed. B.C. Brookes. Amsterdam: North-Holland, 1986. 91-99.Google Scholar
  2. Bernick, Philip. “Habitability in Search Engine Interfaces: Characteristics Identified Through Formative Evaluation Techniques.” Doctoral dissertation, New Mexico State University, Las Cruces, NM, 2003.Google Scholar
  3. Catledge, Lara D., and James E. Pitkow. “Characterizing Browsing Strategies in the World-Wide Web.” Computer Networks and ISDN Systems. 27.6 (1995): 1065-73.CrossRefGoogle Scholar
  4. Choo, C.W., B. Detlor, and D. Turnbull. “Information Seeking on the Web-An integrated model of browsing and searching.” Proceedings of the Annual Meeting of the American Society for Information Science (ASIS), 23 June 2003 <>
  5. Chu, H., and Rosenthal, M. 1996. Search Engines for the World Wide Web: A Comparative Study and Evaluation Methodology. In ASIS1996 Annual Conference Proceedings,23 June2003 ElectronicProceedings/chu.html
  6. Cockburn, A., & Jones, S. (1996). Which Way Now? Analysing and Easing Inadequacies in WWW Navigation. International Journal of Human-Computer Studies, 45(1), pp. 105-129.CrossRefGoogle Scholar
  7. Hölscher, Christoph and Gerhard Strube. “Web Search Behavior of Internet Experts and Newbies.” Proceedings of the9th International World Wide Web Conference(WWW9). 23 June 2003
  8. Jansen, B.J., A. Spink, J. Bateman, and T. Saracevic. “Real Life Information Retrieval: A Study of User Queries on the Web.” SIGIR Forum, 32.1 (1998): 5-17.CrossRefGoogle Scholar
  9. Jansen, B.J., A. Spink, and T. Saracevic. “Failure Analysis in Query Constructions: Data and Analysis from a Large Sample of Web Queries.” Digital Libraries 98. New York: ACM, 1998. 289-90.Google Scholar
  10. Katz, Boris, Sue Felshin, Deniz Yuret, Ali Ibrahim, Jimmy Lin, Gregory Marton, Alton Jerome McFarland and Baris Temelkuran. “Omnibase: Uniform Access to Heterogeneous Data for Question Answering.” Proceedings of the 7th International Workshop on Applications of Natural Language to Information Systems (NLDB 2002), June, 2002.Google Scholar
  11. Lewis, Clayton and John Rieman. “Task-Centered User Interface Design: A Practical Introduction.” available via anonymous ftp at: in:/pub/cs/distribs/clewis/HCI-Design-Book. 1994.Google Scholar
  12. Marchionini, G., S. Dwiggins, A. Katz, and X. Lin. “Information Seeking in Full-Text End-User-Oriented Search Systems: The Roles of Domain and Search Expertise.” LISR, 15 (1993): 35-69.Google Scholar
  13. Moldovan, Dan, Marius Pasca, Sanda Harabagiu, and Mihai Surdeanu. Performance Issues and Error Analysis in an Open-Domain Question Answering System. 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), July 2002, Philadelphia, Pennsylvania.Google Scholar
  14. Navarro-Prieto, R., M. Scaife, and Y. Rogers. “Cognitive Strategies in Web Searching.” Proceedings of the 5th Conference on Human Factors and the Web, 23 June 2003 proceedings/navarro-prieto/index.htmlGoogle Scholar
  15. O’day, Vicki, and Robin Jeffries. “Orienteering in an Information Landscape: How Information Seekers Get From Here to There.” HPL-92-127 Hewlett-Packard:n.p, 1992.Google Scholar
  16. Ogden, William C., and Philp Bernick. “Oleada: User-Centered Tipster Technology for Language Instruction.” Proceedings of the Tipster Phase II 24 Month Workshop. Tysons Corner, VA, 1996.Google Scholar
  17. Ogden, William C., and Philp Bernick. “Tabula Rasa Meta-Tool: Text Extraction Toolbuilder Toolkit.” Technical Report MCCS-97-305, Computing Research Laboratory, New Mexico State University, Las Cruces, NM, 1997.Google Scholar
  18. Ogden, William. C., and Philip Bernick. “Using Natural Language Interfaces.” Handbook of Human Computer Interaction. Eds. Helender, M., T. Landauer, and P. Prabhu, 2nd Edition, Amsterdam: North Holland, 1997, 137-61.Google Scholar
  19. Spink, A., D. Wolfram, B.J Jansen, and T. Saracevic. “Searching the Web: The Public and Their Queries.” Journal of the American Society for Information Science and Technology, 52 (2001): 226-34.CrossRefGoogle Scholar
  20. Tauscher, Linda and Saul Greenberg. “How people revisit Web pages: Empirical findings and implications for the design of history systems.” International Journal of Human Computer Studies, 47(1): 97-138, 1997.CrossRefGoogle Scholar
  21. Watt, W.C. “Habitability.” American Documentation, July (1968) 338-51.Google Scholar

Copyright information

© Springer 2008

Authors and Affiliations

  • William Ogden
    • 1
  • James Mcdonald
    • 1
  • Philip Bernick
    • 2
  • Roger Chadwick
    • 1
  1. 1.Computing Research LaboratoryNew Mexico State UniversityLas CrucesUSA
  2. 2.Arizona State UniversityTempeUSA

Personalised recommendations