Abstract
Over a period of 3 years, RepLab was a CLEF initiative where computer scientists and online reputation experts worked together to identify and formalize the computational challenges in the area of online reputation monitoring. Two main results emerged from RepLab: a community of researchers engaged in the problem, and an extensive Twitter test collection comprising more than half a million expert annotations, which cover many relevant tasks in the field of online reputation: named entity resolution, topic detection and tracking, reputational alerts identification, reputational polarity, author profiling, opinion makers identification and reputational dimension classification. It has probably been one of the CLEF labs with a larger set of expert annotations provided to participants in a single year, and one of the labs where the target user community has been more actively engaged in the evaluation campaign. Here we summarize the design and results of the Replab campaigns, and also report on research that has built on RepLab datasets after completion of the 3-year competition cycle.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Alsop RJ (2006) The 18 immutable laws of corporate reputation: creating, protecting and repairing your most valuable asset. Kogan Page, London
Amigó E, Gonzalo J, Artiles J, Verdejo F (2009) A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf Retr 12(4):461–486
Amigó E, Gonzalo J, Artiles J, Verdejo F (2011) Combining evaluation metrics via the unanimous improvement ratio and its application to clustering tasks. J Artif Intell Res 42(1):689–718
Amigó E, Gonzalo J, Verdejo F (2013) A general evaluation measure for document organization tasks. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’13, pp 643–652. https://doi.org/10.1145/2484028.2484081
Amigó E, Carrillo-de Albornoz J, Almagro-Cádiz M, Gonzalo J, Rodríguez-Vidal J, Verdejo F (2017) Evall: open access evaluation for information access systems. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’17, pp 1301–1304. https://doi.org/10.1145/3077136.3084145
Balahur A, Tanev H (2012) Detecting entity-related events and sentiments from tweets using multilingual resources. In: CLEF (Online Working Notes/Labs/Workshop)
Berrocal A, Luis J, Figuerola CG, Zazo Rodríguez ÁF (2013) Reina at replab2013 topic detection task: community detection. In: CLEF (Working Notes)
Carrillo-de-Albornoz J, Chugur I, Amigó E (2012) Using an emotion-based model and sentiment analysis techniques to classify polarity for reputation. In: CLEF (Online Working Notes/Labs/Workshop)
Carrillo-de Albornoz J, Amigó E, Plaza L, Gonzalo J (2016) Tweet stream summarization for online reputation management. In: Ferro N, Crestani F, Moens MF, Mothe J, Silvestri F, Di Nunzio GM, Hauff C, Silvello G (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 378–389
Chong WN, Tan G (2010) Obtaining intangible and tangible benefits from corporate social responsibility. Int Rev Bus Res Pap 6(4):360
Cossu JV, Bigot B, Bonnefoy L, Morchid M, Bost X, Senay G, Dufour R, Bouvier V, Torres-Moreno JM, El-Bèze M (2013) Lia at replab 2013. In: CLEF (Working Notes)
Cossu JV, Bigot B, Bonnefoy L, Senay G (2014a) Towards the improvement of topic priority assignment using various topic detection methods for e-reputation monitoring on twitter. In: Métais E, Roche M, Teisseire M (eds) Natural language processing and information systems. Springer International Publishing, Cham, pp 154–159
Cossu JV, Janod K, Ferreira E, Gaillard J, El-Bèze M (2014b) Lia at replab 2014: 10 methods for 3 tasks. In: 4th international conference of the CLEF initiative
Doorley J, Garcia HF (2011) Reputation management: the key to successful public relations and corporate communication. Routledge, New York
Filgueiras J, Amir S (2013) Popstar at replab 2013: polarity for reputation classification. In: CLEF (Online Working Notes/Labs/Workshop), vol 60
Gârbacea C, Tsagkias M, de Rijke M (2014) Detecting the reputation polarity of microblog posts. In: Proceedings of the twenty-first european conference on artificial intelligence. IOS Press, Amsterdam, ECAI’14, pp 339–344. https://doi.org/10.3233/978-1-61499-419-0-339
Giachanou A, Gonzalo J, Mele I, Crestani F (2017) Sentiment propagation for predicting reputation polarity. In: Jose JM, Hauff C, Altıngovde IS, Song D, Albakour D, Watt S, Tait J (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 226–238
Glance N, Hurst M, Nigam K, Siegler M, Stockton R, Tomokiyo T (2005) Deriving marketing intelligence from online discussion. In: Proceedings of the eleventh ACM SIGKDD international conference on knowledge discovery in data mining. ACM, New York, NY, KDD ’05, pp 419–428. https://doi.org/10.1145/1081870.1081919
Hangya V, Farkas R (2013) Filtering and polarity detection for reputation management on tweets. In: CLEF (Online Working Notes/Labs/Workshop), vol 60
Hoffman T (2008) Online reputation management is hot – but is it ethical. Computerworld (44). https://www.computerworld.com/article/2537007/networking/online-reputation-management-is-hot----but-is-it-ethical-.html
Jansen B, Zhang M, Sobel K, Chowdury A (2009a) Twitter power: tweets as electronic word of mouth. J Am Soc Inf Sci Technol 60(11):2169–2188
Jansen BJ, Zhang M, Sobel K, Chowdury A (2009b) Twitter power: tweets as electronic word of mouth. J Assoc Inf Sci Technol 60(11):2169–2188
Kaptein R (2012) Learning to analyze relevancy and polarity of tweets. In: CLEF (Online Working Notes/Labs/Workshop), vol 60
Kreps DM, Wilson R (1982) Reputation and imperfect information. J Econ Theory 27(2):253–279
Krishnamurthy B, Gill P, Arlitt M (2008) A few chirps about Twitter. In: Proceedings of the first workshop on online social networks (WOSP’08), pp 19–24
Li YM, Li TY (2013) Deriving market intelligence from microblogs. Decis Support Syst 55(1):206–217. https://doi.org/10.1016/j.dss.2013.01.023. http://www.sciencedirect.com/science/article/pii/S0167923613000511
Mabrouk O, Hlaoua L, Nazih Omri M (2018) Profile categorization system based on features reduction. In: International symposium on artificial intelligence and mathematics, ISAIM 2018, Fort Lauderdale, Florida. http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Mabrouk_etal.pdf
Magdy W, Elsayed T (2016) Unsupervised adaptive microblog filtering for broad dynamic topics. Inf Process Manag 52(4):513–528
Mahalakshmi GS, Koquilamballe K, Sendhilkumar S (2017) Influential detection in twitter using tweet quality analysis. In: 2017 second international conference on recent trends and challenges in computational models (ICRTCCM), pp 315–319. https://doi.org/10.1109/ICRTCCM.2017.62
Martın T, Spina D, Amigó E, Gonzalo J (2012) Uned at replab 2012: monitoring task. In: CLEF 2012 Working Notes, CLEF
Martín-Wanton T, Gonzalo J, Amigó E (2013) An unsupervised transfer learning approach to discover topics for online reputation management. In: Proceedings of the 22nd ACM international conference on conference on information & knowledge management, ACM, New York, NY, CIKM ’13, pp 1565–1568. https://doi.org/10.1145/2505515.2507845
McDonald G, Deveaud R, McCreadie R, Gollins T, Macdonald C, Ounis I (2014) University of glasgow terrier team/project abacá at replab 2014: reputation dimensions task. In: CLEF (Working Notes), pp 1500–1504
McDonald G, Deveaud R, Mccreadie R, Macdonald C, Ounis I (2015) Tweet enrichment for effective dimensions classification in online reputation management. In: 9th international AAAI conference on web and social media, pp 654–657
Nebot V, Rangel F, Berlanga R, Rosso P (2018) Identifying and classifying influencers in twitter only with textual information. In: Proceedings of the NLDB 2018
Panem S, Bansal R, Gupta M, Varma V (2014) Entity tracking in real-time using sub-topic detection on twitter. In: de Rijke M, Kenter T, de Vries AP, Zhai C, de Jong F, Radinsky K, Hofmann K (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 528–533
Peetz MH, de Rijke M, Kaptein R (2016) Estimating reputation polarity on microblog posts. Inf Process Manag 52(2):193–216. https://doi.org/10.1016/j.ipm.2015.07.003. http://www.sciencedirect.com/science/article/pii/S0306457315000874
Qureshi M, Younus A, O’Riordan C, Pasi G (2015) Company name disambiguation in tweets: a two-step filtering approach. In: Information retrieval technology, vol 9460
Qureshi MA, O’Riordan C, Pasi G (2012) Concept term expansion approach for monitoring reputation of companies on twitter. In: CLEF (Online Working Notes/Labs/Workshop)
Qureshi MA, Younus A, O’Riordan C, Pasi G (2017) A wikipedia-based semantic relatedness framework for effective dimensions classification in online reputation management. J Ambient Intell Humaniz Comput 9:1403
Rodriguez J, Carrillo-de Albornoz J, Plaza L, Amigó E, Gonzalo J (2019a) Automatic generation of entity-oriented summaries for reputation management. J Ambient Intell Humaniz Comput:1–15. https://link.springer.com/article/10.1007/s12652-019-01255-9
Rodriguez J, Gonzalo J, Plaza L, Anaya H (2019b) Automatic detection of influencers in social networks: authority versus domain signals. J Assoc Inf Sci Technol 70:7
Spina D, Carrillo-de-Albornoz J, Martín-Wanton T, Amigó E, Gonzalo J, Giner F (2013) Uned online reputation monitoring team at replab 2013. In: CLEF (Working Notes)
Spina D, Gonzalo J, Amigó E (2014) Learning similarity functions for topic detection in online reputation monitoring. In: Proceedings of the 37th international ACM SIGIR conference on research & development in information retrieval. ACM, New York, NY, SIGIR ’14, pp 527–536. https://doi.org/10.1145/2600428.2609621
Spina D, Peetz MH, de Rijke M (2015) Active learning for entity filtering in microblog streams. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’15, pp 975–978. https://doi.org/10.1145/2766462.2767839
Vilares D, Hermo M, Alonso MA, Gómez-Rodríguez C, Vilares J (2014) Lys at clef replab 2014: creating the state of the art in author influence ranking and reputation classification on twitter. In: CLEF (Working Notes), pp 1468–1478
Villena-Román J, Lana-Serrano S, Moreno C, García-Morera J, Cristóbal JCG (2012) Daedalus at replab 2012: polarity classification and filtering on twitter data. In: CLEF (Online Working Notes/Labs/Workshop), vol 60
Acknowledgement
This research was partially supported by the Spanish Ministry of Science and Innovation (Vemodalen Project, TIN2015-71785-R).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Carrillo-de-Albornoz, J., Gonzalo, J., Amigó, E. (2019). RepLab: An Evaluation Campaign for Online Monitoring Systems. In: Ferro, N., Peters, C. (eds) Information Retrieval Evaluation in a Changing World. The Information Retrieval Series, vol 41. Springer, Cham. https://doi.org/10.1007/978-3-030-22948-1_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-22948-1_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-22947-4
Online ISBN: 978-3-030-22948-1
eBook Packages: Computer ScienceComputer Science (R0)