Abstract
Since the introduction of Semantic Web, the practice of seeking and retrieving documents had been evolved. In this paper, the retrieved documents are ranked based on their annotated documents. We adopt two IR algorithms; Lucene Luke and ComFFICF. In order to verify the generated rankings, we run a user-centered evaluation, where it involved 10 human judges. Then, we assess the performance of ranking using NDCG metric. The assessment shows a ranking by ComFFICF algorithm outperforms a ranking by Lucene Luke. This method is proven to be one of preferable IR algorithms for searching and ranking annotated document.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)
Spink, A., Greisdorf, H., Bateman, J.: From Highly Relevant to Non-Relevant: Examining Different Regions of Relevance. Information Processing and Management 34(5), 599–622 (1998), http://www-staff.lboro.ac.uk/~lsas2/pubs/articles_page6.html (Accessed: 10 Feb.2009)
Berners-Lee, T., James, H., Ora, L.: The Semantic Web. Scientific American Magazine (2001)
World Wide Web Consortium (W3C), http://www.w3.org/standards/semanticweb/
Stoyanovich, J.: Search and Ranking in Semantically Rich Applications. Ph.D. Dissertation, Columbia University (2010)
Davulcu, H., Srinivas, V., Saravanakumar, N.: OntoMiner: Bootstrapping Ontologies from Overlapping Domain Specific Web sites (2004)
Kashyap, V., Ramakrishnan, C., Thomas, C., Sheth, A.: TaxaMiner: An Experimental Framework for Automated Taxonomy Bootstrapping. International Journal of Web and Grid Services 1(2) (2005)
Chung, C.Y., Lieu, R., Liu, J., Luk, A., Mao, J., Raghavan, P.: Thematic Mapping from Unstructured Documents to Taxonomies. In: CIKM (2002)
Kara, S.: An Ontology-Based Retrieval System Using Semantic Indexing. M.Sc. CompEng. Thesis, Middle East Technical University (2010)
Kasneci, G., Suchanek, F.M., Ifrim, G., Ramanath, M., Weikum, G.: NAGA: Searching and Ranking Knowledge, icde. In: 2008 IEEE 24th International Conference on Data Engineering (2008)
Roscoe, H.J.: A Cross-Sectional Test of the Effect and Conceptualization of Service Value. J. of Services Mktg. 11(6), 35–50 (1975); Sekaran, U.: Research Methods for Business: A Skill-Building Approach, 3 edn. John Wiley and Sons, Chichester (2000)
Järvelin, K., Kekäläinen, J.: IR Evaluation Methods for Retrieving Highly Relevant Documents. In: Belkin, N.J., Ingwersen, P., Leong, M.-K. (eds.) Proceedings of the 23rd ACM Sigir Conference on Research and Development of Information Retrieval, Athens, Greece, 2000, pp. 41–48. ACM Press, New York (2000)
Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)
Järvelin, K., Kekäläinen, J.: Cumulated Gain-Based Evaluation of IR Techniques. ACM Trans. Information Systems, pp. 422-446 (2002)
Typke, R., Veltkamp, R.C., Wiering, F.: A Measure for Evaluating Retrieval Techniques based on Partially Ordered Ground Truth Lists. In: ICME, 1793-1796 (2006)
NDCG, http://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG
Lucene Applications (2005), http://wiki.apache.org/lucene-java/PoweredBy
Edgar, M., Maarten, d.R.: Deploying Lucene on the Grid. In: Proceedings SIGIR 2006 workshop on Open Source Information Retrieval, OSIR2006 (2006), http://en.scientificcommons.org/21615939
Venkatachalam, L.: SSP Scalability of Stepping Stones and Pathways. M.Sc in Comp Sc. and App. Thesis, Faculty of the Virginia Polytechnic Institute and State University (2008)
Sartori, G., Gnoato, F., Mariani, I., Prioni, S., Lombardi, L.: Semantic Relevance, Domain Specificity and the Sensory/Functional. Theory of Category Specificity, Neuropsychologia 45, 966–976 (2007)
Dittrich, J.-P., Salles, M.A.V.: iDM: A Unified and Versatile Data. Model for Personal Dataspace Management. In: VLDB (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Rahayu, S.B., Noah, S.A., Wardhana, A.A. (2011). User-Centered Evaluation for IR: Ranking Annotated Document Algorithms. In: Zain, J.M., Wan Mohd, W.M.b., El-Qawasmeh, E. (eds) Software Engineering and Computer Systems. ICSECS 2011. Communications in Computer and Information Science, vol 181. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22203-0_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-22203-0_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22202-3
Online ISBN: 978-3-642-22203-0
eBook Packages: Computer ScienceComputer Science (R0)