Abstract
The Winograd Schema Challenge, the task of resolving pronouns in certain carefully-structured sentences, has received considerable interest in the past few years as an alternative to the Turing Test. Systems developed to tackle this challenge have typically been evaluated on a small set of hand-crafted collections of sentences, since the development of new sentences by individuals is itself a rather challenging task, requiring care and creativity. In this paper we approach the problem of developing Winograd schemas via the introduction of WinoFlexi, a flexible online crowdsourcing system. Our empirical evaluation of the system’s performance suggests that WinoFlexi allows crowdworkers to develop Winograd schemas of quality similar to that of most typical existing collections.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Levesque, H., Davis, E., Morgenstern, L.: The Winograd schema challenge. In: Proceedings of the 13th International Conference on the Principles of Knowledge Representation and Reasoning (2012)
Sharma, A., Vo, N.H., Aditya, S., Baral, C.: Towards addressing the Winograd schema challenge - building and using a semantic parser and a knowledge hunting module. In: Proceedings of the 24th International Joint Conference on Artificial Intelligence, pp. 25–31 (2015)
Isaak, N., Michael, L.: A data-driven metric of hardness for WSC sentences. In: Lee, D., Steen, A., Walsh, T. (eds.), Proceedings of the 4th Global Conference on Artificial Intelligence. EPiC Series in Computing, vol. 55, pp. 107–120. EasyChair (2018)
Morgenstern, L., Davis, E., Ortiz, C.L.: Planning, executing, and evaluating the Winograd schema challenge. AI Mag. 37(1), 50–54 (2016)
Isaak, N., Michael, L.: Using the Winograd schema challenge as a CAPTCHA. In: Lee, D., Steen, A., Walsh, T. (eds.), Proceedings of the 4th Global Conference on Artificial Intelligence. EPiC Series in Computing, vol. 55, pp. 93–106. EasyChair (2018)
Rahman, A., Ng, V.: Resolving complex cases of definite pronouns: the Winograd schema challenge. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Stroudsburg, PA, USA, pp. 777–789. Association for Computational Linguistics (2012)
Christoforaki, M., Ipeirotis, P.: Step: a scalable testing and evaluation platform. In: Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (2014)
Hirth, M., Hoßfeld, T., Tran-Gia, P.: Anatomy of a crowdsourcing platform – using the example of microworkers.com. In: Proceedings of the 5th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, pp. 322–329. IEEE (2011)
Peer, E., Samat, S., Brandimarte, L., Acquisti, A.: In: Diehl, K., Yoon, D.C. (eds.), Beyond the Turk: An Empirical Comparison of Alternative Platforms for Crowdsourcing Online Research. NA - Advances in Consumer Research, vol. 43, pp. 18–22. Association for Consumer Research, MN (2015)
Isaak, N., Michael, L.: Tackling the Winograd schema challenge through machine logical inferences. In: Pearce, D., Pinto, H.S. (eds.), STAIRS. Frontiers in Artificial Intelligence and Applications, vol. 284, pp. 75–86. IOS Press (2016)
Acknowledgments
This work was supported by funding from the EU’s Horizon 2020 Research and Innovation Programme under grant agreements no. 739578 and no. 823783, and from the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination, and Development. The authors would like to thank Ernest Davis for sharing his thoughts and suggestions on this line of research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Isaak, N., Michael, L. (2019). WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas. In: Liu, J., Bailey, J. (eds) AI 2019: Advances in Artificial Intelligence. AI 2019. Lecture Notes in Computer Science(), vol 11919. Springer, Cham. https://doi.org/10.1007/978-3-030-35288-2_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-35288-2_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-35287-5
Online ISBN: 978-3-030-35288-2
eBook Packages: Computer ScienceComputer Science (R0)