Skip to main content
SpringerLink
Log in

The Scholarly Impact and Strategic Intent of CLEF eHealth Labs from 2012 to 2017

  • Chapter
  • First Online:
Information Retrieval Evaluation in a Changing World

Part of the book series: The Information Retrieval Series ((INRE,volume 41))

Abstract

Since 2012, the CLEF eHealth initiative has aimed to gather researchers working on health text analytics and to provide them with annual shared tasks. This chapter reports on measuring its scholarly impact in 2012–2017 and describing its future objectives. The large number of submissions and citations demonstrate the substantial community interest in the tasks and their resources. Consequently, the initiative continues to run in 2018 and 2019 with its goal to support patients, their family, clinical staff, health scientists, and healthcare policy makers in accessing and authoring health information in a multilingual setting.

Authors’ Contributions: In alphabetical order by forename, HS, LK, and LG co-chaired the CLEF eHealth lab in 2012–2018 and led some of its tasks. To review the lab outcomes, HS first conceptualized the study and then designed and supervised its literature survey and citation content analysis which the co-authors conducted in close collaboration. HS, LK, and LG drafted the manuscript together, with dedicated duties for each coauthor. After this all authors critically commented and revised the manuscript. All authors have read and approved the final version of the chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Afzal Z, Akhondi S, van Haagen H, van Mulligen E, Kors J (2015) Biomedical concept recognition in French text using automatic translation of English terms. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Alharbi A, Stevenson M (2017) Ranking abstracts to identify relevant evidence for systematic reviews: The University of Sheffield’s approach to CLEF eHealth 2017 task 2. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Anagnostou A, Lagopoulos A, Tsoumakas G, Vlahavas I (2017) Combining inter-review learning-to-rank and intra-review incremental training for title and abstract screening in systematic reviews. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Angelini M, Ferro N, Larsen B, Müller H, Santucci G, Silvello G, Tsikrika T (2014) Measuring and analyzing the scholarly impact of experimental evaluation initiatives. Procedia Comput Sci 38(Supplement C):133–137

    Article  Google Scholar 

  • Atemezing G (2017) NoNLP: annotating medical domain by using semantic technologies. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Azzopardi L, Kalphov V, Georgiadis G (2017) SiS at CLEF 2017 eHealth TAR task. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Bedrick S, Sheikshabbafghi G (2013) Lucene, MetaMap, and language modeling: OHSU at CLEF eHealth 2013. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Beloborodov A, Goeuriot L (2016) Improving health consumer search with contextual information. In: Proceedings of the 2nd SIGIR workshop on Medical Information Retrieval (MedIR). Association for Computing Machinery, New York

    Google Scholar 

  • Bodnari A, Deleger L, Lavergne T, Neveol A, Zweigenbaum P (2013) A supervised named-entity extraction system for medical text. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Budaher J, Almasri M, Goeuriot L (2016) Comparison of several word embedding sources for medical information retrieval. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Caballero Barajas K, Akella R (2013) Incorporating statistical topic models in the retrieval of healthcare documents. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Cabot C, Soualmia L, Dahamna B, Darmoni S (2016) SIBM at CLEF eHealth evaluation lab 2016: extracting concepts in French medical texts with ECMT and CIMIND. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Cabot C, Soualmia L, Darmoni S (2017) SIBM at CLEF eHealth evaluation lab 2017: multilingual information extraction with CIM-IND. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Chapman W (2012) Developing resources to assist in development and application of NLP to clinical texts. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Chappell T, Geva S (2013) Working notes for TopSig at ShARe/CLEF eHealth 2013. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Chen J, Chen S, Song Y, Liu H, Wang Y, Hu Q, He L (2017) ECNU at 2017 eHealth task 2: technologically assisted reviews in empirical medicine. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Chernyshevich M, Stankevitch V (2015) IHS-RD-BELARUS: clinical named entities identification in French medical texts. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Choi S, Choi J (2013) SNUMedinfo at CLEFeHealth2013 task 3. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Choi S, Choi J (2014) Exploring effective information retrieval technique for the medical web documents: SNUMedinfo at CLEFeHealth2014 task 3. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Claveau V, Hamon T, Grabar N, Le Maguer S (2014) RePaLi participation to CLEF eHealth IR challenge 2014: leveraging term variation. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Cogley J, Stokes N, Carthy J (2013) Medical disorder recognition with structural support vector machines. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Colineau N, Paris C (2010) Talking about your health to strangers: understanding the use of online social networks by patients. New Rev Hypermedia Multimedia 16(1–2):141–160

    Article  Google Scholar 

  • Cormack G, Grossman M (2017) Technology-assisted review in empirical medicine: Waterloo participation in CLEF eHealth 2017. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Cotik V, Vivaldi J, Rodriguez H (2015) Semantic tagging of French medical entities using distant learning. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • De Vine L, Kholghi M, Zuccon G, Sitbon L, Nguyen A (2015) Patient empowerment via technologies for patient-friendly personalized language. In: Proceedings of the 13th annual workshop of the Australasian Language Technology Association. Australasian Language Technology Association, Sydney, pp 153–164

    Google Scholar 

  • Demner-Fushman D, Elhadad N (2016) Aspiring to unintended consequences of natural language processing: a review of recent developments in clinical and consumer-generated text processing. Yearb Med Inform (1):224–233. https://doi.org/10.15265/IY-2016-017

    Google Scholar 

  • Dermouche M, Looten V, Flicoteaux R, Chevret S, Velcin J, Taright N (2016) ECSTRA-INSERM @ CLEF eHealth2016-task 2: ICD10 code extraction from death certificates. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • D’Hondt E, Grau B, Zweigenbaum P (2015a) LIMSI @ CLEF eHealth 2015 — task 2. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • D’Hondt E, Morlane-Hondére F, Campillos L, Bouamor D, Ribeiro S, Lavergne T (2015b) LIMSI @ CLEF eHealth 2015 — task 1b. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Di Nunzio G, Beghini F, Vezzani F, Henrot G (2017a) An interactive two-dimensional approach to query aspects rewriting in systematic reviews. IMS Unipd at CLEF eHealth task 2. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Di Nunzio G, Beghini F, Vezzani F, Henrot G (2017b) A lexicon based approach to classification of ICD10 codes. IMS Unipd at CLEF eHealth task 1. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Diaz-Galiano M, Martin-Valdivia M, Jiménez–Zafra S, Andreu A, Urena-López L (2017) SINAI at CLEF eHealth 2017 task 3s. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Dramé K, Mougin F, Diallo G (2014) Query expansion using external resources for improving information retrieval in the biomedical domain. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Ebersbach M, Herms R, Lohr C, Eibl M (2016) Wrappers for feature subset selection in CRF-based clinical information extraction. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Ebersbach M, Herms R, Eibl M (2017) Fusion methods for ICD10 code classification of death certificates in multilingual corpora. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Fan J, Sood N, Huang Y (2013) Disorder concept identification from clinical notes: an experience with the ShARe/CLEF 2013 challenge. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Filannino M, Uzuner Ö (2018) Advancing the state of the art in clinical natural language processing through shared tasks. Yearb Med Inform 27(01):184–192. https://doi.org/10.1055/s-0038-1667079

    Article  Google Scholar 

  • Friberg Heppin K, Järvelin A (2012) Towards improving search results for medical experts and laypersons. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Ghoddousi A, Huang J (2015) York University at CLEF 2015 eHealth: medical document retrieval. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Goeuriot L, Jones G, Kelly L, Leveling J, Hanbury A, Müller H, Salanterä S, Suominen H, Zuccon G (2013a) ShARe/CLEF eHealth Evaluation Lab 2013, Task 3: information retrieval to address patients’ questions when reading clinical reports. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Goeuriot L, Kelly L, Jones G, Zuccon G, Suominen H, Hanbury A, Müller H, Leveling J (2013b) Creation of a new evaluation benchmark for information retrieval targeting patient information needs. In: Proceedings of the 5th international workshop on evaluating information access (EVIA), A satellite workshop of the NTCIR-10 conference. National Institute of Informatics/Kijima Printing, Tokyo/Fukuoka

    Google Scholar 

  • Goeuriot L, Kelly L, Leveling J (2014a) An analysis of query difficulty for information retrieval in the medical domain. In: SIGIR’14: proceedings of the 37th international ACM SIGIR conference on research and development in information retrieval. Association for Computer Machinery, New York, pp 1007–1010

    Google Scholar 

  • Goeuriot L, Kelly L, Li W, Palotti J, Pecina P, Zuccon G, Hanbury A, Jones G, Múller H (2014b) ShARe/CLEF eHealth evaluation lab 2014, task 3: user-centred health information retrieval. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Goeuriot L, Kelly L, Suominen H, Hanlen L, Névéol A, Grouin C, Palotti JRM, Zuccon G (2015) Overview of the CLEF eHealth Evaluation Lab 2015. In: Mothe J, Savoy J, Kamps J, Pinel-Sauvagnat K, Jones GJF, SanJuan E, Cappellato L, Ferro N (eds) (2015) Experimental IR meets multilinguality, multimodality, and interaction. In: Proceedings of the sixth international conference of the CLEF association (CLEF 2015). Lecture notes in computer science (LNCS), vol 9283. Springer, Heidelberg, pp 429–443

    Chapter  Google Scholar 

  • Goeuriot L, Kelly L, Zuccon G, Palotti J (2016) Building evaluation datasets for consumer-oriented information retrieval. In: Proceedings of the tenth international conference on language resources and evaluation (LREC 2016), European Language Resources Association (ELRA), Paris

    Google Scholar 

  • Goeuriot L, Kelly L, Suominen H, Névéol A, Robert A, Kanoulas E, Spijker R, Palotti JRM, Zuccon G (2017) CLEF 2017 eHealth evaluation lab overview. In: Jones GJF, Lawless S, Gonzalo J, Kelly L, Goeuriot L, Mandl T, Cappellato L, Ferro N (eds) (2017) Experimental IR meets multilinguality, multimodality, and interaction. In: Proceedings of the eighth international conference of the CLEF association (CLEF 2017). Lecture notes in computer science (LNCS), vol 10456. Springer, Heidelberg, pp 291–303

    Chapter  Google Scholar 

  • Gung J (2013) Using relations for identification and normalization of disorders: team CLEAR in the ShARe/CLEF 2013 eHealth evaluation lab. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Hamon T, Grouin C, Zweigenbaum P (2014) Disease and disorder template filling using rule-based and statistical approaches. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Hanbury A, Müller H (2012) Khresmoi — multimodal multilingual medical information search. In: MIE village of the future

    Google Scholar 

  • Hanlen L (2012) National eHealth living lab: removing the “e” from e-health. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Heimonen J, Salakoski T, Salanterä S (2012) An ontology to improve accessibility and quality of patient instructions. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Herbst K, Fähnrich C, Neves M, Schapranow M (2014) Applying in-memory technology for automatic template filling in the clinical domain. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Herms R, Richter D, Eibl M, Ritter M (2015) Search for clinical speech recognition. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Hervas L, Martinez V, Sanchez I, Diaz A (2013a) UCM at CLEF eHealth 2013 shared task1a. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Hervas L, Martinez V, Sanchez I, Diaz A (2013b) UCM at CLEF eHealth 2013 shared task1b. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Ho-Dac L, Tanguy L, Grauby C, Mby N, Malosse J, Riviére L, Veltz-Mauclair A, Wauquier M (2016) LITL at CLEF eHealth2016: recognizing entities in French biomedical documents. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Ho-Dac L, Fabre C, Birski A, Boudraa I, Bourriot A, Cassier M, Delvenne L, Garcia-Gonzalez C, Kang E, Piccinini E, Rohrbacher C, S/’eguier A (2017) LITL at CLEF eHealth2017: automatic classification of death reports. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Hollmann N, Eickhoff C (2017) Ranking and feedback-based stopping for recall-centric document retrieval. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Huang CC, Lu Z (2016) Community challenges in biomedical text mining over 10 years: success, failure and the future. Brief Bioinform 17(1):132–144

    Article  Google Scholar 

  • Huynh H, Ho S (2014) ShARe/CLEFeHealth: a hybrid approach for task 2. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Huynh N, Nguyen T, Ho Q (2015) TeamHCMUS: a concept-based information retrieval approach for web medical documents. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Hyman H, Fridy W (2014) An eHealth process model of visualization and exploration to support improved patient discharge record understanding and medical knowledge enhancement. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Ilic D (2010) The role of the internet on patient knowledge management, education, and decision-making. Telemed J E Health 16(6):664–669

    Article  Google Scholar 

  • Isenius N, Kvist M, Velupillai S (2012) Initial results in the development of SCAN: a Swedish clinical abbreviation normalizer. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Jagannathan V, Ganesan D, Jagannathan A, Kavi R, Lamb A, Peters F, Seeger S (2013) WVU NLP class participation in ShARe/CLEF challenge. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Jain D (2015) Supervised named entity recognition for clinical data. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Jiang J, YGuan, Zhao C (2015) WI-ENRE in CLEF eHealth evaluation lab 2015: clinical named entity recognition based on CRF. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Jimmy, Zuccon G, Koopman B (2017) QUT ielab at CLEF 2017 e-Health IR task: knowledge base retrieval for consumer health search. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Johri N, Niwa Y, Raghavendra Chikka V (2014) Optimizing Apache cTAKES for disease/disorder template filling: team HITACHI in the ShARe/CLEF 2014 eHealth evaluation lab. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Jones G, Karlgren J, Kreiner K, Müller H, Salanterä S (2012) Panel presentation: towards systematic evaluation of methods, applications, and resources for eHealth document analysis. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Jonnagaddala J, Hu F (2017) Automatic coding of death certificates to ICD-10 terminology. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Kalpathy-Cramer J, Müller H, Bedrick S, Eggel I, Garcia Seco de Herrera A, Tsikrika T (2011) Overview of the CLEF 2011 medical image classification and retrieval tasks. In: Petras V, Forner P, Clough P, Ferro N (eds) CLEF 2011 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1177/

  • Kanhov M, Feng X, Dalianis H (2012) Natural language generation from SNOMED specifications. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Kanoulas E, Li D, Azzopardi L, Spijker R (2017) CLEF 2017 technologically assisted reviews in empirical medicine overview. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Kelly L, Goeuriot L, Jones G, Hanbury A (2012) Considering subjects and scenarios in large-scale user-centered evaluation of a multilingual multimodal medical search system. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Kelly L, Goeuriot L, Suominen H, Schreck T, Leroy G, Mowery DL, Velupillai S, Webber Chapman W, Martínez D, Zuccon G, Palotti JRM (2014) Overview of the ShARe/CLEF eHealth Evaluation Lab 2014. In: Kanoulas E, Lupu M, Clough P, Sanderson M, Hall M, Hanbury A, Toms E (eds) (2014) Information access evaluation – multilinguality, multimodality, and interaction. In: Proceedings of the fifth international conference of the CLEF initiative (CLEF 2014). Lecture notes in computer science (LNCS), vol 8685. Springer, Heidelberg, pp 172–191

    Google Scholar 

  • Kelly L, Goeuriot L, Suominen H, Névéol A, Palotti J, Zuccon G (2016) Overview of the CLEF eHealth Evaluation Lab 2016. In: Fuhr N, Quaresma P, Gonçalves T, Larsen B, Balog K, Macdonald C, Cappellato L, Ferro N (eds) (2016) Experimental IR meets multilinguality, multimodality, and interaction. In: Proceedings of the seventh international conference of the CLEF association (CLEF 2016). Lecture notes in computer science (LNCS), vol 9822. Springer, Heidelberg, pp 255–266

    Chapter  Google Scholar 

  • Kholghi M, Sitbon L, Zuccon G, Nguyen A (2014) Factors influencing robustness and effectiveness of conditional random fields in active learning frameworks. In: AusDM: the twelfth Australasian data mining conference, Queensland University of Technology, Brisbane, pp 1007–1010

    Google Scholar 

  • Kholghi M, Sitbon L, Zuccon G, Nguyen A (2015) External knowledge and query strategies in active learning: a study in clinical information extraction. In: Proceedings of the 24th ACM international on conference on information and knowledge management. Association for Computing Machinery, New York, pp 143–152

    Google Scholar 

  • Kholghi M, Sitbon L, Zuccon G, Nguyen A (2016) Active learning: a step towards automating medical concept extraction. J Am Med Inform Assoc 23(2):289–296

    Article  Google Scholar 

  • Koopman B, Zuccon G (2014) Relevation!: An open source system for information retrieval relevance assessment. In: SIGIR’14: proceedings of the 37th international ACM SIGIR conference on research and development in information retrieval. Association for Computer Machinery, New York, pp 1243–1244

    Google Scholar 

  • Kreiner K, Eckmann H, Hayn D, Kastner P (2012) On the use of text messaging in a diabetes telehealth system results and evaluation of a content analysis. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Ksentini N, Tmar M, Boughanem M, Gargouri F (2015) Miracl at CLEF 2015: user-centred health information retrieval task. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Laippala V, Danielsson-Ojala R, Lundgrén-Laine H, Salanterä S, Salakoski T (2012) Vocabulary in discharge documents: the patient’s perspective. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Leaman R, Khare R, Lu Z (2013) NCBI at 2013 ShARe/CLEF eHealth shared task: disorder normalization in clinical notes with dnorm. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Lee G (2017) A study of convolutional neural networks for clinical document classification in systematic reviews: SysReview at CLEF eHealth 2017. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Limsopatham N, Macdonald C, Ounis I (2013) University of Glasgow at CLEF 2013: experiments in eHealth task 3 with Terrier. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Liu N, Ku L (2014) CLEFeHealth 2014 normalization of information extraction challenge using multi-model method. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Liu X, Nie J (2015) Bridging layperson’s queries with medical concepts — GRIUM@CLEF2015 eHealth task 2. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Liu H, Wagholikar K, Jonnalagadda S, Sohn S (2013) Integrated cTAKES for concept mention detection and normalization. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Liu X, Liu X, Shen W, Nie J (2014) Mining disorder attributes with rules and statistical learning. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Lu F (2015) Employing query expansion models to help patients diagnose themselves. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Luu T, Phan R, Davey R, Chetty G (2015) Automatic clinical speech recognition for CLEF 2015 eHealth challenge. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Malagon J, L/’opez M (2014) LABERINTO at ShARe/CLEF eHealth evaluation lab 2014. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Martinez D, Suominen H, Ananda-Rajah M, Cavedon L (2012) Biosurveillance for invasive fungal infections via text mining. In: CLEF 2012 working notes. CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • McAllister M, Dunn G, Payne K, Davies L, Todd C (2012) Patient empowerment: the need to consider it as a measurable patient-reported outcome for chronic conditions. BMC Health Serv Res 12:157

    Article  Google Scholar 

  • Miftahutdinov Z, Tutubalina E (2017) KFU at CLEF eHealth 2017 task 1: ICD-10 coding of English death certificates with recurrent neural networks. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Mkrtchyan T, Sonntag D (2014) Deep parsing at the CLEF2014 IE task. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Moen H, Marsi E (2012) Towards retrieving and ranking clinical recommendations with cross-lingual random indexing. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Mottin L, Gobeill J, Mottaz A, Pasche E, Gaudinat A, Ruch P (2016) BiTeM at CLEF eHealth evaluation lab 2016 task 2: multilingual information extraction. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Mowery D, South B, Christensen L, Murtola L, Salanterä S, Suominen H, Martinez D, Elhadad N, Pradhan S, Savova G, Chapman W (2013) Task 2: ShARe/CLEF eHealth evaluation lab 2013. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Mowery D, Velupillai S, South B, Christensen L, Martinez D, Kelly L, Goeuriot L, Elhadad N, Pradhan S, Savova G, Chapman W (2014) Task 2: ShARe/CLEF eHealth evaluation lab 2014. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Mowery DL, South BR, Christensen L, Leng J, Peltonen LM, Salanterä S, Suominen H, Martinez D, Velupillai S, Elhadad N, Savova G, Pradhan S, Chapman WW (2016) Normalizing acronyms and abbreviations to aid patient understanding of clinical texts: ShARe/CLEF eHealth challenge 2013, task 2. J Biomed Semant 7(1):43

    Article  Google Scholar 

  • Murtola L, Kauhanen L, Moen H, Lundgr/’en-Laine H, Salakoski T, Salanterä S (2013) Using text mining to explore factors associated with acute confusion in cardiac patients documentation. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Nesrine K, Mohamed T, Faiez G (2014) Miracl at CLEF 2014: eHealth information retrieval task. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Névéol A, Grouin C, Leixa J, Rosset S, Zweigenbaum P (2014) The QUAERO French medical corpus: a ressource for medical entity recognition and normalization. In: Proceeding of BioTextMining Work, pp 24–30

    Google Scholar 

  • Névéol A, Grouin C, Tannier X, Hamon T, Kelly L, Goeuriot L, Zweigenbaum P (2015) CLEF eHealth evaluation lab 2015 task 1b: clinical named entity recognition. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Névéol A, Cohen K, Grouin C, Hamon T, Lavergne T, Kelly L, Goeuriot L, Rey G, Robert A, Tannier X, Zweigenbaum P (2016) Clinical information extraction at the CLEF eHealth evaluation lab 2016. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Névéol A, Robert A, Anderson R, Cohen K, Grouin C, Lavergne T, Rey G, Rondet C, Zweigenbaum P (2017) CLEF eHealth 2017 multilingual information extraction task overview: ICD10 coding of death certificates in English and French. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Norman C, Leeflang M, N/’evéol A (2017) LIMSI@CLEF eHealth 2017 task 2: logistic regression for automatic article ranking. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Oh H, Jung Y (2014) Ta multiple-stage approach to re-ranking clinical documents. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Oh H, Jung Y (2016) KISTI at CLEF eHealth 2016 task 3: ranking medical documents using word vectors. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Oh H, Jung Y (2017) KISTI at CLEF eHealth 2017 patient-centered information retrieval task-1: improving medical document retrieval with query expansion. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Oh H, Jung Y, Kim K (2015) KISTI at CLEF eHealth 2015 task 2. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Osborne J (2014) Disease template filling using the CTAKES YTEX Branch for ShareCLEF 2014 task 2a. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Osborne J, Gyawali B, Solorio T (2013) Evaluation of YTEX and MetaMap for clinical concept recognition. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Ozturkmenoglu O, Alpkocak A, Kilinc D (2014) DEMIR at CLEF eHealth: the effects of selective query expansion to information retrieval. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Palotti J, Rekabsaz N (2017) Exploring understandability features to personalize consumer health search. TUW at CLEF 2017 eHealth. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Palotti J, Zuccon G, Goeuriot L, Kelly L, Hanbury A, Jones G, Lupu M, Pecina P (2015a) CLEF eHealth evaluation lab 2015, task 2: retrieving information about medical symptoms. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Palotti J, Zuccon G, Hanbury A (2015b) The influence of pre-processing on the estimation of readability of web documents. In: Proceedings of the 24th ACM international on conference on information and knowledge management. Association for Computing Machinery, New York, pp 1763–1766

    Google Scholar 

  • Palotti J, Goeuriot L, Zuccon G, Hanbury A (2016a) Patient empowerment via technologies for patient-friendly personalized language. In: Proceedings of the 39th international ACM SIGIR conference. Association for Computing Machinery, New York, pp 965–968

    Google Scholar 

  • Palotti J, Zuccon G, Bernhardt J, Hanbury A, Goeuriot L (2016b) Assessors agreement: a case study across assessor type, payment levels, query variations and relevance dimensions. In: Fuhr N, Quaresma P, Gonçalves T, Larsen B, Balog K, Macdonald C, Cappellato L, Ferro N (eds) (2016) Experimental IR meets multilinguality, multimodality, and interaction. In: Proceedings of the seventh international conference of the CLEF association (CLEF 2016). Lecture notes in computer science (LNCS), vol 9822. Springer, Heidelberg, pp 40–53

    Google Scholar 

  • Palotti J, Zuccon G, Jimmy, Pecina P, Lupu M, Goeuriot L, Kelly L, Hanbury A (2017) CLEF 2017 task overview: the IR task at the eHealth evaluation lab — evaluating retrieval methods for consumer health search. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Patrick J, Safari L, Ou Y (2013a) ShARe/CLEF eHealth 2013 named entity recognition and normalization of disorders challenge. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Patrick J, Safari L, Ou Y (2013b) Share/clef ehealth 2013 normalization of acronyms/abbreviations challenge. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Pestian J, Matykiewicz P, Linn-Gust M, South B, Uzuner O, Wiebe J, Cohen K, Hurdle J, Brew C (2011) Sentiment analysis of suicide notes: a shared task. Biomed Inform Insights 5(Suppl. 1):3–16

    Google Scholar 

  • Pradhan S, Elhadad N, South B, Martinez D, Christensen L, Vogel A, Suominen H, Chapman W, Savova G (2013) Task 1: ShARe/CLEF eHealth evaluation lab 2013. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Pradhan S, Elhadad N, South BR, Martinez D, Christensen L, Vogel A, Suominen H, Chapman WW, Savova G (2015) Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. J Am Med Inform Assoc 22(1):143–154. https://doi.org/10.1136/amiajnl-2013-002544

    Article  Google Scholar 

  • Quiroz L, Mennes L, Dehghani M, Kanoulas E (2016) Distributional semantics for medical information extraction. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Ramanan S, Senthil Nathan P (2014) Cocoa: Extending a rule-based system to tag disease attributes in clinical records. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Ramanan S, Broido S, Senthil Nathan P (2013) Performance of a multi-class biomedical tagger on clinical records. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Rekabsaz N, Lupu M, Hanbury A, Zuccon G (2016) Generalizing translation models in the probabilistic relevance framework. In: Proceedings of the 25th ACM international on conference on information and knowledge management. Association for Computing Machinery, New York, pp 711–720

    Google Scholar 

  • Roberts PM, Cohen AM, Hersh WR (2009) Tasks, topics and relevance judging for the TREC genomics track: five years of experience evaluating biomedical text information retrieval systems. Inf Retr 12:81–97

    Article  Google Scholar 

  • Robertson S, Hull D (2000) The TREC-9 filtering track final report. In: NIST Special Publication 500-249: The 9th Text REtrieval Conference (TREC 9). National Institute of Standards and Technology, Gaithersburg, pp 25–40

    Google Scholar 

  • Saleh S, Pecina P (2014) CUNI at the ShARe/CLEF eHealth evaluation lab 2014. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Saleh S, Pecina P (2016) Reranking hypotheses of machine-translated queries for cross-lingual information retrieval. In: Fuhr N, Quaresma P, Gonçalves T, Larsen B, Balog K, Macdonald C, Cappellato L, Ferro N (eds) (2016) Experimental IR meets multilinguality, multimodality, and interaction. In: Proceedings of the seventh international conference of the CLEF association (CLEF 2016). Lecture notes in computer science (LNCS), vol 9822. Springer, Heidelberg, pp 54–68

    Chapter  Google Scholar 

  • Saleh S, Pecina P (2017) Task3 patient-centred information retrieval: team CUNI. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Saleh S, Bibyna F, Pecina P (2015) CUNI at the CLEF 2015 eHealth lab task 2. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Scells H, Zuccon G, Deacon A, Koopman B (2017) QUT ielab at CLEF 2017 technology assisted reviews track: initial experiments with learning to rank. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Sequeira J, Miranda N, Goncalves T, Quaresma P (2014) Team UEvora at CLEF eHealth 2014 task2a. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Seva J, Kittner M, Roller R, Leser U (2017) Multi-lingual ICD-10 coding using a hybrid rule-based and supervised classification approach at CLEF eHealth 2017. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Shenwei W, Nie J, Liu X, Liu X (2014) An investigation of the effectiveness of concept-based approach in medical information retrieval. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Silva R, Lopes C (2016) The effectiveness of query expansion when searching for health related content: InfoLab at CLEF eHealth 2016. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Singh J, Thomas L (2017) IIIT-H at CLEF eHealth 2017 task 2: technologically assisted reviews in empirical medicine. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Singh G, Marshall I, Thomas J, Wallace B (2017) Identifying diagnostic test accuracy publications using a deep model. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Soldaini L, Edman W, Goharian N (2016) Team GU-IRLAB at CLEF eHealth 2016: task 3. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Song Y, He Y, Hu Q, He L, Haacke E (2015) ECNU at 2015 eHealth task 2: user-centred health information retrieval. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Song Y, He Y, Liu H, Hu Q, He L, Wang Y (2016a) ECNU at 2016 eHealth task 1: handover information extraction. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Song Y, He Y, Liu H, Wang Y, Hu Q, He L (2016b) ECNU at 2016 eHealth task 3: patient-centred information retrieval. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Soualmia L, Cabot C, Dahamna B, Darmoni S (2015) SIBM at CLEF e-Health evaluation lab 2015. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • South BR, Mowery D, Suo Y, Leng J, Óscar Ferrández, Meystre SM, Chapman WW (2014) Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text. J Biomed Inform 50:162–172. https://doi.org/10.1016/j.jbi.2014.05.002, special issue on Informatics Methods in Medical Privacy

    Article  Google Scholar 

  • Sun W, Rumshisky A, Uzuner O (2013) Evaluating temporal relations in clinical text: 2012 i2b2 challenge. J Am Med Inform Assoc 20(5):806–813. https://doi.org/10.1136/amiajnl-2013-001628

    Article  Google Scholar 

  • Suominen H (2012a) CLEFeHealth2012 - the CLEF 2012 workshop on cross-language evaluation of methods, applications, and resources for eHealth document analysis. In: Forner P, Karlgren J, Womser-Hacker C, Ferro N (eds) (2012) CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Suominen H (2012b) Towards international privacy-preserving benchmarks of eHealth technologies. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Suominen H (2014) Text mining and information analysis of health documents. Artif Intell Med 61(3):127–130

    Article  Google Scholar 

  • Suominen H, Basilakis J, Johnson M, Dawson L, Hanlen L, Kelly B, Yeo A, Sanchez P (2012a) Clinical speech to text evaluation setting. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Suominen H, Kreiner K, Hanlen L (2012b) Towards ease of building legos in assessing eHealth language technologies: a RESTful laboratory for data and software. In: CLEF 2012 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  • Suominen H, Salanterä S, Velupillai S, Webber Chapman W, Savova GK, Elhadad N, Pradhan S, South BR, Mowery DL, Jones GJF, Leveling J, Kelly L, Goeuriot L, Martínez D, Zuccon G (2013) Overview of the ShARe/CLEF eHealth Evaluation Lab 2013. In: Forner P, Müller H, Paredes R, Rosso P, Stein B (eds) (2013) Information access evaluation meets multilinguality, multimodality, and visualization. In: Proceedings of the fourth international conference of the CLEF initiative (CLEF 2013). Lecture notes in computer science (LNCS), vol 8138. Springer, Heidelberg, pp 212–231

    Google Scholar 

  • Suominen H, Schreck T, Leroy G, Hochheiser H, Goeuriot L, Kelly L, Mowery D, Nualart J, Ferraro G, Keim D (2014) Task 1 of the ShARe/CLEF eHealth evaluation lab 2014. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Suominen H, Hanlen L, Goeuriot L, Kelly L, Jones G (2015a) Task 1a of the CLEF eHealth evaluation lab 2015: clinical speech recognition. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Suominen H, Zhou L, Hanlen L, Ferraro G (2015b) Benchmarking clinical speech recognition and information extraction: new data, methods, and evaluations. JMIR Med Inform 3(2):e19

    Article  Google Scholar 

  • Suominen H, Zhou L, Goeuriot L, Kelly L (2016) Task 1 of the CLEF eHealth evaluation lab 2016: handover information extraction. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Suominen H, Kelly L, Goeuriot L (2018a) Scholarly influence of the Conference and Labs of the Evaluation Forum eHealth initiative: review and bibliometric study of the 2012 to 2017 outcomes. JMIR Res Protoc 7(7):e10961. https://doi.org/10.2196/10961

    Article  Google Scholar 

  • Suominen H, Kelly L, Goeuriot L, Névéol A, Ramadier L, Robert A, Kanoulas E, Spijker R, Azzopardi L, Li D, Jimmy, Palotti J, Zuccon G (2018b) Overview of the CLEF eHealth Evaluation Lab 2018. In: Bellot P, Trabelsi C, Mothe J, Murtagh F, Nie JY, Soulier L, SanJuan E, Cappellato L, Ferro N (eds) Experimental IR meets multilinguality, multimodality, and interaction. Proceedings of the nineth international conference of the CLEF association (CLEF 2018), Lecture notes in computer science (LNCS), vol 11018. Springer, Heidelberg, pp 286–301

    Google Scholar 

  • Tang B, Wu Y, Jiang M, Denny J, Xu H (2013) Recognizing and encoding discorder concepts in clinical text using machine learning and vector space model. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Tchechmedjiev A, Abdaoui A, Emonet V, Jonquet C (2017) ICD10 coding of death certificates with the NCBO and SIFR annotator(s) at CLEF eHealth 2017 task 1. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Thakkar H, Iyer G, Shah K, Majumder P (2014) Team IRLabDAIICT at ShARe/CLEF eHealth 2014 task 3: user-centered information retrieval system for clinical documents. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Thesprasith O, Jaruskulchai C (2014) CSKU GPRF-QE for medical topic web retrieval. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Thesprasith O, Jaruskulchai C (2015) Task 2a: Team KU-CS: query coherence analysis for PRF and genomics expansion. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Thuma E, Anderson G, Mosweunyane G (2015) UBML participation to CLEF eHealth IR challenge 2015: task 2. In: CLEF 2015 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  • Thuma E, Motlogelwa N, Leburu-Dingalo T (2016) Task 3: patient-centered information retrieval, IRTask 1: ad-hoc search — TEAM ub-botswana. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Thuma E, Motlogelwa N, Leburu-Dingalo T (2017) Ub-Botswana participation to clef ehealth IR challenge 2017: task 3 (irtask1: Ad-hoc search). In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Tsikrika T, Larsen B, Müller H, Endrullis S, Rahm E (2013) The scholarly impact of CLEF (2000–2009). In: Forner P, Müller H, Paredes R, Rosso P, Stein B (eds) (2013) Information access evaluation meets multilinguality, multimodality, and visualization. In: Proceedings of the fourth international conference of the CLEF initiative (CLEF 2013). Lecture notes in computer science (LNCS), vol 8138. Springer, Heidelberg, pp 1–12

    Chapter  Google Scholar 

  • Ullah M, Aono M (2016) KDEIR at CLEF eHealth 2016: health documents re-ranking based on query variations. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Uzuner O, South BR, Shen S, DuVall S (2011) 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc 18(5):552–556

    Article  Google Scholar 

  • van Altena A, Delgado Olabarriaga S (2017) Predicting publication inclusion for diagnostic accuracy test reviews using random forests and topic modelling. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • van Mulligen E, Afzal Z, Akhondi S, Vo D, Kors J (2016) Erasmus MC at CLEF eHealth 2016: concept recognition and coding in French texts. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Verberne S (2014) A language-modelling approach to user-centred health information retrieval. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Vivaldi J, Rodriguez H, Cotik V (2016) Semantic tagging and normalization of French medical entities. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Voorhees E, Tong RM (2011) Overview of the TREC 2011 medical records track. In: NIST Special Publication 500-296: the text REtrieval conference 2011. National Institute of Standards and Technology, Gaithersburg

    Google Scholar 

  • Wang C, Akella R (2013) UCSC’s system for CLEF eHealth 2013 task 1. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Wang R, Lu W, Ren K (2016a) WHUIRGroup at the CLEF 2016 eHealth lab task 3. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Wang Y, Wu S, Liu H (2016b) MayoNLPTeam at the 2016 CLEF eHealth information retrieval task 1. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Wu J, Huang J (2014) Task 3a: Team YORKU. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Wu Y, Tang B, Jiang M, Moon S, Denny J, Xu H (2013) Clinical acronym/abbreviation normalization using a hybrid approach. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Xia Y, Zhong X, Liu P, Tan C, Na S, Hu Q, Huang Y (2013a) Combining MetaMap and cTAKES in disorder recognition: THCIB at CLEF eHealth lab 2013 task 1. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Xia Y, Zhong X, Liu P, Tan C, Na S, Hu Q, Huang Y (2013b) Normalization of abbreviations/acronyms: THCIB at CLEF eHealth 2013 task 2. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Yang H, Goncalves T (2017) UEvora at CLEF eHealth 2017 task 3. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Yang C, Bhattacharya S, Srinivasan P (2014) The university of Iowa at CLEF 2014: eHealth task 3. In: CLEF 2014 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  • Yu Z, Menzies T (2017) Data balancing for technologically assisted reviews: undersampling or reweighting. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Zhang G, Ding Y, Milojević S (2013a) Citation content analysis (CCA): a framework for syntactic and semantic analysis of citation content. J Am Soc Inf Sci Technol 64(7):1490–1503

    Article  Google Scholar 

  • Zhang Y, Cohen T, Jiang M, Tang B, Xu H (2013b) Evaluation of vector space models for medical disorders information retrieval. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Zhong X, Xia Y, Xie Z, Na S, Hu Q, Huang Y (2013) Concept-based medical document retrieval: THCIB at CLEF eHealth lab 2013 task 3. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Zhou L, Suominen H (2015) Information extraction to improve standard compliance: the case of clinical handover. In: Proceedings of the 28th Australasian joint conference on artificial intelligence 2015 (AI2015). Lecture notes in computer science, vol 9457. Springer, Heidelberg, pp 644–649

    Chapter  Google Scholar 

  • Zhou L, Suominen H, Hanlen L (2015) Evaluation data and benchmarks for cascaded speech recognition and entity extraction. In: Proceedings of the ACM multimedia 2015 workshop on speech, language and audio in multimedia, vol 10. Association for Computing Machinery, New York, pp 15–18

    Google Scholar 

  • Zhu D, Wu S, Masanz J, Carterette B, Liu H (2013) Using discharge summaries to improve information retrieval in clinical domain. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Zuccon G (2016) Understandability biased evaluation for information retrieval. In: European conference on information retrieval (ECIR 2016). Lecture notes in computer science, vol 9626. Springer, Heidelberg, pp 280–292

    Google Scholar 

  • Zuccon G, Koopman B (2014) Integrating understandability in the evaluation of consumer health search engines. In: Medical information retrieval (MedIR) workshop. Association for Computer Machinery, New York

    Google Scholar 

  • Zuccon G, Holloway A, Koopman B, Nguyen A (2013a) Identify disorders in health records using conditional random fields and Metamap AEHRC at ShARe/CLEF 2013 eHealth evaluation lab task 1. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Zuccon G, Koopman B, Nguyen A (2013b) Retrieval of health advice on the web AEHRC at ShARe/CLEF eHealth evaluation lab task 3. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  • Zuccon G, Koopman B, Palotti J (2015) Diagnose this if you can: on the effectiveness of search engines in finding medical self-diagnosis information. In: European conference on information retrieval (ECIR 2015). Lecture notes in computer science, vol 9022. Springer, Heidelberg, pp 562–567

    Google Scholar 

  • Zuccon G, Palotti J, Goeuriot L, Kelly L, Lupu M, Pecina P, Müller H, Budaher J, Deacon A (2016a) The IR task at the CLEF eHealth evaluation lab 2016: User-centred health information retrieval. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Zuccon G, Palotti J, Hanbury A (2016b) Query variations and their effect on comparing information retrieval systems. In: Proceedings of the 25th ACM international on conference on information and knowledge management. Association for Computing Machinery, New York, pp 691–700

    Google Scholar 

  • Zweigenbaum P, Lavergne T (2016) LIMSI ICD10 coding experiments on C/’epiDC death certificate statements. In: CLEF 2016 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  • Zweigenbaum P, Lavergne T (2017) Multiple methods for multi-class, multi-label ICD-10 coding of multi-granularity, multilingual death certificates. In: CLEF 2017 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  • Zweigenbaum P, Deleger L, Lavergne T, Neveol A, Bodnari A (2013) A supervised abbreviation resolution system for medical text. In: CLEF 2013 working notes, CEUR workshop proceedings (CEUR-WS.org), ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

Download references

Acknowledgements

We gratefully acknowledge the people involved in Conference and Labs of the Evaluation Forum (CLEF) Electronic Health (eHealth) 2012–2017 as participants or organizers. We are also thankful to support by the CLEF Initiative; Data61; European Science Foundation (ESF) Project EvaLuating Information Access Systems (ELIAS) network program Horizon 2020 program (H2020-ICT-2014-1) under grant agreement 644753 (KConnect); French National Research Agency (ANR), under grant CABeRneT ANR-13-JS02-0009-01; Knowledge Helper for Medical and Other Information users (Khresmoi) project, funded by the European Union Seventh Framework Programme (FP7)/2007–2013 under grant agreement No. 257528; Microsoft Azure for Research Award CRM:0518649; Multiparameter Intelligent Monitoring in Intensive Care (MIMIC) II Database; National Information and Communications Technologies Australia (NICTA), funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the Information and Communications Technology (ICT) Centre of Excellence program; Protégé resource, supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States (of America) (US) National Institutes of Health; PhysioNetWorks Workspaces; Shared Annotated Resources (ShARe) project funded by the US National Institutes of Health (R01GM090187); Swedish Research Council (350-2012-6658); Swedish Vårdal Foundation; US Department of Veterans Affairs (VA) Consortium for Healthcare Informatics Research (CHIR); and US Office of the National Coordinator of Healthcare Technology, Strategic Health Information Technology Advanced Research Projects (SHARP) 90TR0002.

Author information

Authors and Affiliations

  1. Research School of Computer Science, The Australian National University (ANU), Canberra, ACT, Australia

    Hanna Suominen

  2. Data61, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Canberra, ACT, Australia

    Hanna Suominen

  3. Faculty of Science and Technology, University of Canberra, Canberra, ACT, Australia

    Hanna Suominen

  4. Department of Future Technologies, University of Turku, Turku, Finland

    Hanna Suominen

  5. Computer Science Department, Maynooth University, Maynooth, Co. Kildare, Ireland

    Liadh Kelly

  6. Grenoble Informatics Laboratory, University Grenoble Alpes, Grenoble INP, LIG, Grenoble, France

    Lorraine Goeuriot

Authors
  1. Hanna Suominen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liadh Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lorraine Goeuriot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanna Suominen .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova , Padova, Italy

    Nicola Ferro

  2. Consiglio Nazionale delle Ricerche, Istituto di Scienza e Tecnologie dell’Informazione, Pisa, Italy

    Carol Peters

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Suominen, H., Kelly, L., Goeuriot, L. (2019). The Scholarly Impact and Strategic Intent of CLEF eHealth Labs from 2012 to 2017. In: Ferro, N., Peters, C. (eds) Information Retrieval Evaluation in a Changing World. The Information Retrieval Series, vol 41. Springer, Cham. https://doi.org/10.1007/978-3-030-22948-1_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-22948-1_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-22947-4

  • Online ISBN: 978-3-030-22948-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation