The Role of Graduality for Referring Expression Generation in Visual Scenes

  • Albert Gatt
  • Nicolás MarínEmail author
  • François Portet
  • Daniel Sánchez
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 610)


Referring Expression Generation (reg) algorithms, a core component of systems that generate text from non-linguistic data, seek to identify domain objects using natural language descriptions. While reg has often been applied to visual domains, very few approaches deal with the problem of fuzziness and gradation. This paper discusses these problems and how they can be accommodated to achieve a more realistic view of the task of referring to objects in visual scenes.


Referring expression Fuzziness Linguistic description Visual scenes 



This work has been partially supported by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (FEDER) under project TIN2014-58227-P.


  1. 1.
    Reiter, E., Dale, R.: Building Natural Language Generation Systems. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  2. 2.
    Kacprzyk, J., Zadrozny, S.: Computing with words is an implementable paradigm: Fuzzy queries, linguistic data summaries, and natural-language generation. IEEE Trans. Fuzzy Syst. 18(3), 461–472 (2010)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Marín, N., Sánchez, D.: On generating linguistic descriptions of time series. Fuzzy Sets Syst. 285, 6–30 (2016)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Ramos-Soto, A., Bugarín, A., Barro, S.: On the role of linguistic descriptions of data in the building of natural language generation systems. Fuzzy Sets Syst. 285, 31–51 (2016)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Han, X., Berg, T., Daume III., H.: Midge: Generating Image Descriptions From Computer Vision Detections. In: EACL 2012, Avignon, France, pp. 747–756. Association for Computational Linguistics (2012)Google Scholar
  6. 6.
    Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: Baby talk: Understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2891–2903 (2013)CrossRefGoogle Scholar
  7. 7.
    Yatskar, M., Galley, M., Vanderwende, L., Zettlemoyer, L.: See No Evil, Say No Evil : Description Generation from Densely Labeled Images. In: Proceedings of the Third Joint Conference on Lexical and Computation Semantics (*SEM) (2014)Google Scholar
  8. 8.
    Reiter, E., Dale, R.: A fast algorithm for the generation of referring expressions. In: COLING 1992, pp. 232–238 (1992)Google Scholar
  9. 9.
    Krahmer, E., van Deemter, K.: Computational generation of referring expressions: A survey. Comput. Linguist. 38(1), 173–218 (2012)CrossRefGoogle Scholar
  10. 10.
    Elsner, M., Rohde, H., Clarke, A.D.F.: Information structure prediction for visual-world referring expressions. In: EACL 2014, Gothenburg, Sweden, pp. 520–529. Association for Computational Linguistics (2014)Google Scholar
  11. 11.
    Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.L.: ReferItGame: referring to objects in photographs of natural scenes. In: EMNLP 2014, Doha, Qatar, pp. 787–798. Association for Computational Linguistics (2014)Google Scholar
  12. 12.
    van Deemter, K.: Generating referring expressions that involve gradable properties. Comput. Linguist. 32(2), 195–222 (2006)CrossRefzbMATHGoogle Scholar
  13. 13.
    Turner, R., Sripada, S., Reiter, E., Davy, I.P.: Selecting the content of textual descriptions of geographically located events in spatio-temporal weather data. In: Ellis, R., Allen, T., Petridis, M. (eds.) Applications and Innovations in Intelligent Systems XV, pp. 75–88. Springer, London (2008)CrossRefGoogle Scholar
  14. 14.
    Reiter, E., Sripada, S., Hunter, J., Yu, J., Davy, I.: Choosing words in computer-generated weather forecasts. Artif. Intell. 167(1–2), 137–169 (2005)CrossRefGoogle Scholar
  15. 15.
    Cadenas, J.T., Marín, N., Vila, M.A.: Context-aware fuzzy databases. Appl. Soft Comput. 25, 215–233 (2014)CrossRefGoogle Scholar
  16. 16.
    Castillo-Ortega, R., Chamorro-Martínez, J., Marín, N., Sánchez, D., Soto-Hidalgo, J.M.: Describing images via linguistic features and hierarchical segmentation. In: FUZZ-IEEE 2010, pp. 1–8 (2010)Google Scholar
  17. 17.
    Soto-Hidalgo, J.M., Chamorro-Martínez, J., Sánchez, D.: A new approach for defining a fuzzy color space. In: FUZZ-IEEE 2010, pp. 1–6 (2010)Google Scholar
  18. 18.
    Chamorro-Martínez, J., Medina, J.M., Barranco, C.D., Galán-Perales, E., Soto-Hidalgo, J.M.: Retrieving images in fuzzy object-relational databases using dominant color descriptors. Fuzzy Sets Syst. 158(3), 312–324 (2007)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Chamorro-Martínez, J., Martínez-Jiménez, P.M., Soto-Hidalgo, J.M., León-Salas, A.: A fuzzy approach for modelling visual texture properties. Inf. Sci. 313, 1–21 (2015)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Chamorro-Martínez, J., Martínez-Jiménez, P.M., Soto-Hidalgo, J.M., Prados-Suárez, B.: Fuzzy sets on 2D spaces for fineness representation. Int. J. Approx. Reasoning 62, 46–60 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Prados-Suárez, B., Chamorro-Martínez, J., Sánchez, D., Abad, J.: Region-based fit of color homogeneity measures for fuzzy image segmentation. Fuzzy Sets Syst. 158(3), 215–229 (2007)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Hudelot, C., Atif, J., Bloch, I.: Fuzzy spatial relation ontology for image interpretation. Fuzzy Sets Syst. 159(15), 1929–1951 (2008)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Buck, A.R., Keller, J.M., Skubic, M.: A memetic algorithm for matching spatial configurations with the histograms of forces. IEEE Trans. Evol. Comput. 17(4), 588–604 (2013)CrossRefGoogle Scholar
  24. 24.
    Dale, R.: Cooking up referring expressions. In: ACL 1989, Vancouver, BC, pp. 68–75. Association for Computational Linguistics (1989)Google Scholar
  25. 25.
    Grice, H.P.: Logic and conversation. In: Cole, P., Morgan, J.L. (eds.) Syntax and Semantics 3: Speech Acts, pp. 41–58. Elsevier, Amsterdam (1975)Google Scholar
  26. 26.
    Dale, R., Reiter, E.: Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Sci. 19(2), 233–263 (1995)CrossRefGoogle Scholar
  27. 27.
    Gatt, A., Belz, A.: Introducing shared tasks to NLG: the TUNA shared task evaluation challenges. In: Krahmer, E., Theune, M. (eds.) Empirical Methods in NLG. LNCS, vol. 5790, pp. 264–293. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  28. 28.
    Jordan, P.W., Walker, M.A.: Learning content selection rules for generating object descriptions in dialogue. J. Artif. Intell. Res. 24, 157–194 (2005)zbMATHGoogle Scholar
  29. 29.
    Janarthanam, S., Lemon, O.: Adaptive generation in dialogue systems using dynamic user modeling. Computat. Linguist. 40(4), 883–920 (2014)CrossRefGoogle Scholar
  30. 30.
    Castillo-Ortega, R., Marín, N., Sánchez, D., Tettamanzi, A.G.B.: Quality assessment in linguistic summaries of data. In: Greco, S., Bouchon-Meunier, B., Coletti, G., Fedrizzi, M., Matarazzo, B., Yager, R.R. (eds.) IPMU 2012, Part II. CCIS, vol. 298, pp. 285–294. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  31. 31.
    Bugarín, A., Marín, N., Sánchez, D., Triviño, G.: Aspects of quality evaluation in linguistic descriptions of data. In: FUZZ-IEEE 2015, pp. 1–8 (2015)Google Scholar
  32. 32.
    Gardent, C.: Generating minimal definite descriptions. In: ACL 2002, pp. 96–103, July 2002Google Scholar
  33. 33.
    Frank, M.C., Goodman, N.D.: Predicting pragmatic reasoning in language games. Science (New York, N.Y.) 336(6084), 998 (2012)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Kelleher, J.D., Kruijff, G.: Incremental generation of spatial referring expressions in situated dialog. In: COLING-ACL 2006, Sydney, Australia, pp. 1041–1048. Association for Computational Linguistics (2006)Google Scholar
  35. 35.
    Pechmann, T.: Incremental speech production and referential overspecification. Linguistics (1989)Google Scholar
  36. 36.
    Belke, E., Meyer, A.S.: Tracking the time course of multidimensional stimulus discrimination: Analyses of viewing patterns and processing times during “same”-“different” decisions. Eur. J. Cognit. Psychol. 14(2), 237–266 (2002)CrossRefGoogle Scholar
  37. 37.
    Viethen, J., Goudbeek, M., Krahmer, E.: The impact of colour difference and colour codability on reference production. In: CogSci 2012, Austin, TX, pp. 1084–1089. Cognitive Science Society (2012)Google Scholar
  38. 38.
    van Gompel, R.P., Gatt, A., Krahmer, E., van Deemter, K.: Overspecification in reference: Modelling size contrast effects. In: AMLAP 2014 (2014)Google Scholar
  39. 39.
    Itti, L., Koch, C.: Computational modelling of visual attention. Nat. Rev. Neurosci. 2(3), 194–203 (2001)CrossRefGoogle Scholar
  40. 40.
    Erdem, E., Erdem, A.: Visual saliency estimation by nonlinearly integrating features using region covariances. J. Vis. 13(4:11), 1–20 (2013)Google Scholar
  41. 41.
    Oliva, A., Torralba, A.: The role of context in object recognition. Trends Cogn. Sci. 11(12), 520–527 (2007)CrossRefGoogle Scholar
  42. 42.
    Torralba, A., Oliva, A., Castelhano, M.S., Henderson, J.M.: Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychol. Rev. 113(4), 766–786 (2006)CrossRefGoogle Scholar
  43. 43.
    Kanan, C., Tong, M.H., Zhang, L., Cottrell, F.W.: SUN: Top-down saliency using natural statistics. Vis. Cogn. 17(6–7), 979–1003 (2009)CrossRefGoogle Scholar
  44. 44.
    Sedivy, J.C.: Pragmatic versus form-based accounts of referential contrast: evidence for effects of informativity expectations. J. Psycholinguist. Res. 32(1), 3–23 (2003)CrossRefGoogle Scholar
  45. 45.
    Westerbeek, H., Koolen, R., Maes, A.: Stored object knowledge and the production of referring expressions: the case of color typicality. Frontiers Psychol. 6(July), 1–12 (2015)Google Scholar
  46. 46.
    Dale, R., Haddock, N.: Generating referring expressions involving relations. In: EACL 1991, Berlin, Germany, pp. 161–166 (1991)Google Scholar
  47. 47.
    Areces, C., Koller, A., Striegnitz, K.: Referring expressions as formulas of description logic. In: INLG 2008, pp. 42–49 (2008)Google Scholar
  48. 48.
    Gatt, A., Portet, F.: Multilingual generation of uncertain temporal expressions from data: A study of a possibilistic formalism and its consistency with human subjective evaluations. Fuzzy Sets Syst. 285, 73–93 (2016)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Albert Gatt
    • 1
  • Nicolás Marín
    • 2
    Email author
  • François Portet
    • 3
  • Daniel Sánchez
    • 2
  1. 1.Institute of LinguisticsUniversity of MaltaMsidaMalta
  2. 2.Department of Computer Science and A.I.University of GranadaGranadaSpain
  3. 3.Laboratoire d’Informatique de GrenobleGrenoble Institute of TechnologyGrenobleFrance

Personalised recommendations