An Intelligent Sketching Interface for Education Using Geographic Information Systems

  • Aqib Niaz BhatEmail author
  • Girish Kasiviswanathan
  • Christy Maria Mathew
  • Seth Polsley
  • Erik Prout
  • Daniel W. Goldberg
  • Tracy Hammond
Part of the Human–Computer Interaction Series book series (HCIS)


Students learning geography aim to be familiar with a variety of geographic features and be able to identify them on maps. Drawing the geographic entities on a map would, ideally, be a better measure of the recall of the characteristics of such entities and comprehension of the various concepts in geography. However, for teachers trying to evaluate the drawings of a large number of students, this can pose a challenge. In this work, we present a sketch recognition system designed for aiding learning in geography, a field mostly unexplored by the expansive body of work in sketch recognition and education. Our web application allows users to draw rivers on a map and uses a similarity measure to evaluate students’ work. Our main idea is to combine shape and location information of a sketch and check this against the shape information from our data set of geographic features. We evaluated the developed system with 10 users across multiple tests, and the findings reinforce our hope of helping students gain geographic knowledge in an intuitive and effective way through sketching.



We would like to thank the Sketch Recognition Lab, the Computer Science and Engineering Department, and the Department of Geography at Texas A&M University for providing valuable feedback.


  1. 1.
    Alvarado, C., Davis, R.: Sketchread: a multi-domain sketch recognition engine. In: Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, pp. 23–32. ACM (2004)Google Scholar
  2. 2.
    Andrienko, G.L., Andrienko, N.V.: Interactive maps for visual data exploration. Int. J. Geogr. Inf. Sci. 13(4), 355–374 (1999)CrossRefzbMATHGoogle Scholar
  3. 3.
    Anthony, L., Brown, Q., Nias, J., Tate, B., Mohan, S.: Interaction and recognition challenges in interpreting children’s touch and gesture input on mobile devices. In: Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, pp. 225–234. ACM (2012)Google Scholar
  4. 4.
    Atilola, O., McTigue, E.M., Hammond, T., Linsey, J.: Mechanix: evaluating the effectiveness of a sketch recognition truss tutorin program against other truss programs. In: 120th American Society for Engineering Education Annual Conference & Exposition (ASEE), ASEE, Atlanta, 23–26 June 2013, 15 pp.Google Scholar
  5. 5.
    Atilola, O., Valentine, S., Kim, H.H., Turner, D., McTigue, E., Hammond, T., Linsey, J.: Mechanix: a natural sketch interface tool for teaching truss analysis and free-body diagrams. Artif. Intell. Eng. Des. Anal. Manuf. 28, 169–192 (2014). doi:10.1017/S0890060414000079.
  6. 6.
    Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: Dbpedia: a nucleus for a web of open data. Springer, Berlin, Heidelberg (2007)Google Scholar
  7. 7.
    Belongie, S., Malik, J., Puzicha, J.: Shape context: a new descriptor for shape matching and object recognition. In: NIPS, vol. 2, p. 3 (2000)Google Scholar
  8. 8.
    Choi, H., Hammond, T.: Sketch recognition based on manifold learning. In: Proceedings of the 23rd National Conference on Artificial Intelligence, AAAI’08, vol. 3, pp. 1786–1787. AAAI Press (2008).
  9. 9.
    Cummings, D., Fymat, S., Hammond, T.: Sketch-based interface for interaction with unmanned air vehicles. In: CHI’12 Extended Abstracts on Human Factors in Computing Systems, CHI EA’12, pp. 1511–1516. ACM, New York (2012). doi:10.1145/2212776.2223664.
  10. 10.
    Cummmings, D., Fymat, S., Hammond, T.: Reddog: a smart sketch interface for autonomous aerial systems. In: Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling, SBIM’12, pp. 21–28. Eurographics Association, Aire-la-Ville (2012).
  11. 11.
    Feng, G., Viard-Gaudin, C., Sun, Z.: On-line hand-drawn electric circuit diagram recognition using 2D dynamic programming. Pattern Recogn. 42(12), 3215–3223 (2009)CrossRefzbMATHGoogle Scholar
  12. 12.
    Field, M., Valentine, S., Linsey, J., Hammond, T.: Sketch recognition algorithms for comparing complex and unpredictable shapes. In: Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI’11, vol. 3, pp. 2436–2441. AAAI Press (2011). doi:10.5591/978-1-57735-516-8/IJCAI11-406.
  13. 13.
    Forbus, K., Usher, J., Lovett, A., Lockwood, K., Wetzel, J.: Cogsketch: sketch understanding for cognitive science research and for education. Top. Cogn. Sci. 3(4), 648–666 (2011)CrossRefGoogle Scholar
  14. 14.
    Godwin, A., Stasko, J.: Drawing Data on Maps: Sketch-Based Spatiotemporal Visualization Accessed 15 Nov 2017
  15. 15.
    Gonzalez, H., Halevy, A.Y., Jensen, C.S., Langen, A., Madhavan, J., Shapley, R., Shen, W., Goldberg-Kidon, J.: Google fusion tables: web-centered data management and collaboration. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 1061–1066. ACM (2010)Google Scholar
  16. 16.
    Hammond, T.: Ladder: a perceptually-based language to simplify sketch recognition user interface development. Ph.D. Doctoral Dissertation, Massachusetts Institute of Technology (MIT), Cambridge (2007). Advisor: Randall Davis, 495 pp.Google Scholar
  17. 17.
    Hammond, T., Davis, R.: Ladder, a sketching language for user interface developers. Comput. Graph. 4(29), 518–532 (2005)CrossRefGoogle Scholar
  18. 18.
    Hammond, T., Logsdon, D., Peschel, J., Johnston, J., Taele, P., Wolin, A., Paulson, B.: A sketch recognition interface that recognizes hundreds of shapes in course-of-action diagrams. In: CHI’10 Extended Abstracts on Human Factors in Computing Systems, CHI EA’10, pp. 4213–4218. ACM, New York (2010). doi:10.1145/1753846.1754128.
  19. 19.
    Hammond, T., Paulson, B.: Recognizing sketched multistroke primitives. ACM Trans. Interact. Intell. Syst. 1(1), 4:1–4:34 (2011). doi:10.1145/2030365.2030369.
  20. 20.
    Hammond, T., Paulson, B.: Recognizing sketched multistroke primitives. ACM Trans. Interact. Intell. Syst. (TiiS) 1(1), 4 (2011)Google Scholar
  21. 21.
    Hammond, T.A., Logsdon, D., Paulson, B., Johnston, J., Peschel, J.M., Wolin, A., Taele, P.: A sketch recognition system for recognizing free-hand course of action diagrams. In: IAAI (2010)Google Scholar
  22. 22.
    Johnston, J., Hammond, T.: Computing confidence values for geometric constraints for use in sketch recognition. In: Proceedings of the Seventh Sketch-Based Interfaces and Modeling Symposium, SBIM’10, pp. 71–78. Eurographics Association, Aire-la-Ville (2010).
  23. 23.
    Kara, L.B., Stahovich, T.F.: An image-based trainable symbol recognizer for sketch-based interfaces. In: AAAI Fall Symposium, pp. 99–105 (2004)Google Scholar
  24. 24.
    Kebodeaux, K., Field, M., Hammond, T.: Defining precise measurements with sketched annotations. In: Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling, SBIM’11, pp. 79–86. ACM, New York (2011). doi:10.1145/2021164.2021179.
  25. 25.
    Kim, H.h., Taele, P., Valentine, S., McTigue, E., Hammond, T.: Kimchi: a sketch-based developmental skill classifier to enhance pen-driven educational interfaces for children. In: Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling, pp. 33–42. ACM (2013)Google Scholar
  26. 26.
    LaViola Jr, J.J., Zeleznik, R.C.: Mathpad 2: a system for the creation and exploration of mathematical sketches. In: ACM SIGGRAPH 2007 Courses, p. 46. ACM (2007)Google Scholar
  27. 27.
    Miller, E.G., Matsakis, N.E., Viola, P., et al.: Learning from one example through shared densities on transforms. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition, 2000. vol. 1, pp. 464–471. IEEE (2000)Google Scholar
  28. 28.
    Oltmans, M.: Envisioning sketch recognition: a local feature based approach to recognizing informal sketches. Ph.D. thesis, Massachusetts Institute of Technology (2007)Google Scholar
  29. 29.
    Patterson, T., Kelso, N.V.: Free vector and raster map data.
  30. 30.
    Paulson, B., Hammond, T.: Paleosketch: accurate primitive sketch recognition and beautification. In: Proceedings of the 13th International Conference on Intelligent User Interfaces, pp. 1–10. ACM (2008)Google Scholar
  31. 31.
    Paulson, B., Rajan, P., Davalos, P., Gutierrez-Osuna, R., Hammond, T.: What!?! No Rubine Features?: using geometric-based features to produce normalized confidence values for sketch recognition. In: HCC Workshop: Sketch Tools for Diagramming (VL/HCC), p. 57–63. VL/HCC, Herrsching am Ammersee (2008)Google Scholar
  32. 32.
    Rubine, D.: Specifying Gestures by Example, vol. 25. ACM, New York (1991)Google Scholar
  33. 33.
    Rucklidge, W.: Efficient Visual Recognition Using the Hausdorff Distance, vol. 1173. Springer, Berlin (1996)zbMATHGoogle Scholar
  34. 34.
    Sezgin, T.M., Stahovich, T., Davis, R.: Sketch based interfaces: early processing for sketch understanding. In: ACM SIGGRAPH 2006 Courses, p. 22. ACM (2006)Google Scholar
  35. 35.
    Sylverberg, T., Kristensson, P., Leifler, O., Berglund, E.: Drawing on paper maps: reliable on-line symbol recognition of handwritten symbols using a digital pen and a mobile phone. In: 2007 2nd International Conference on Pervasive Computing and Applications (2007)Google Scholar
  36. 36.
    Valentine, S., Vides, F., Lucchese, G., Turner, D., Kim, H.h., Li, W., Linsey, J., Hammond, T.: Mechanix: a sketch-based tutoring and grading system for free-body diagrams. AI Mag. 34(1), 55 (2012)Google Scholar
  37. 37.
    Valentine, S., Vides, F., Lucchese, G., Turner, D., Kim, H.h., Li, W., Linsey, J., Hammond, T.: Mechanix: a sketch-based tutoring system for statics courses. In: IAAI (2012)Google Scholar
  38. 38.
    Vatavu, R.D., Anthony, L., Wobbrock, J.O.: Gestures as point clouds: a $p recognizer for user interface prototypes. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, pp. 273–280. ACM (2012)Google Scholar
  39. 39.
    Wobbrock, J.O., Wilson, A.D., Li, Y.: Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 159–168. ACM (2007)Google Scholar
  40. 40.
    Yin, P., Forbus, K.D., Usher, J.M., Sageman, B., Jee, B.D.: Sketch worksheets: a sketch-based educational software system. In: IAAI (2010)Google Scholar
  41. 41.
    Yu, B., Cai, S.: A domain-independent system for sketch recognition. In: Proceedings of the 1st International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, pp. 141–146. ACM (2003)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Aqib Niaz Bhat
    • 1
    Email author
  • Girish Kasiviswanathan
    • 1
  • Christy Maria Mathew
    • 1
  • Seth Polsley
    • 1
  • Erik Prout
    • 2
  • Daniel W. Goldberg
    • 3
  • Tracy Hammond
    • 1
  1. 1.Sketch Recognition Lab, Department of Computer Science & EngineeringTexas A&M UniversityCollege StationUSA
  2. 2.Department of GeographyTexas A&M UniversityCollege StationUSA
  3. 3.Department of Geography, Department of Computer Science & EngineeringTexas A&M UniversityCollege StationUSA

Personalised recommendations