Sketch Based Interaction Techniques for Chart Creation and Manipulation

  • Andrés N. Vargas GonzálezEmail author
  • Eugene M. TarantaII
  • Joseph J. LaViolaJr.
Part of the Human–Computer Interaction Series book series (HCIS)


In a classroom environment, students are often required to extract descriptive statistics from charts, or find an appropriate graphical representation of the data they are given. On paper these drawings are static, but in an interactive learning environment, the exploration of data can be improved. To address this, we present a set of novel interaction techniques for the creation and manipulation of 12 chart types that leverage pen and touch in a prototype application, SketChart. We allow a user to sketch an appropriate chart type to visualize their data, after which, he or she can remove, merge, and filter data through direct manipulations. Data series and points are also managed through a pair of interactive legends. In a formative user study based on a set of typical statistics questions, we find that even with a chart recognition accuracy of only 81%, the overall system was still well received and can be used as a learning tool for descriptive statistics.



This work is supported in part by NSF CAREER award IIS-0845921. The authors thank members of the Interactive Systems and User Experience Research Cluster of Excellence and the CPTTE community for their support and feedback throughout this project.


  1. 1.
    Alvarado, C., Davis, R.: Sketchread: a multi-domain sketch recognition engine. In: Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, pp. 23–32. ACM (2004)Google Scholar
  2. 2.
    Anthony, L., Wobbrock, J.O.: A lightweight multistroke recognizer for user interface prototypes. In: Proceedings of Graphics Interface 2010, pp. 245–252. Canadian Information Processing Society (2010)Google Scholar
  3. 3.
    Baur, D., Lee, B., Carpendale, S.: Touchwave: kinetic multi-touch manipulation for hierarchical stacked graphs. In: Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, pp. 255–264. ACM (2012)Google Scholar
  4. 4.
    Berk, K., Carey, P.: Data Analysis with Microsoft Excel: Updated for Office 2007. Cengage Learning, Boston (2009). Google Scholar
  5. 5.
    Brade, M., Schneider, F., Salmen, A., Groh, R.: Ontosketch: towards digital sketching as a tool for creating and extending ontologies for non-experts. In: Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies, p. 9. ACM (2013)Google Scholar
  6. 6.
    Browne, J., Lee, B., Carpendale, S., Riche, N., Sherwood, T.: Data analysis on interactive whiteboards through sketch-based interaction. In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, pp. 154–157. ACM (2011)Google Scholar
  7. 7.
    Chamberlain, R., Riley, H., McManus, C., Rankin, Q., Brunswick, N.: The perceptual foundations of drawing ability. In: Thinking Through Drawing: Practice into Knowledge. Proceedings of an Interdisciplinary Symposium on Drawing, Cognition and Education, pp. 95–102. Teachers College Columbia University (2011)Google Scholar
  8. 8.
    Cheema, S., LaViola, J.: Physicsbook: a sketch-based interface for animating physics diagrams. In: Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces, pp. 51–60. ACM (2012)Google Scholar
  9. 9.
    Chung, R., Mirica, P., Plimmer, B.: Inkkit: a generic design tool for the tablet pc. In: Proceedings of the 6th ACM SIGCHI New Zealand Chapter’s International Conference on Computer-Human Interaction: Making CHI Natural, pp. 29–30. ACM (2005)Google Scholar
  10. 10.
    Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–340 (1989)CrossRefGoogle Scholar
  11. 11.
    Hammond, T., Davis, R.: Ladder, a sketching language for user interface developers. Comput. Graph. 29(4), 518–532 (2005)CrossRefGoogle Scholar
  12. 12.
    Harris, R.L.: Information Graphics: A Comprehensive Illustrated Reference. Oxford University Press, Oxford (1999)zbMATHGoogle Scholar
  13. 13.
    LaViola, J.: Mathematical sketching: a new approach to creating and exploring dynamic illustrations. Ph.D. thesis, Department of Computer Science, Brown University (2005)Google Scholar
  14. 14.
    LaViola Jr, J.J., Zeleznik, R.C.: Mathpad 2: a system for the creation and exploration of mathematical sketches. In: ACM SIGGRAPH 2007 Courses, p. 46. ACM (2007)Google Scholar
  15. 15.
    Lee, B., Kazi, R.H., Smith, G.: Sketchstory: telling more engaging stories with data through freeform sketching. IEEE Trans. Vis. Comput. Graph. 19(12), 2416–2425 (2013)CrossRefGoogle Scholar
  16. 16.
    Lee, B., Smith, G., Riche, N.H., Karlson, A., Carpendale, S.: Sketchinsight: natural data exploration on interactive whiteboards leveraging pen and touch interaction. In: 2015 IEEE Pacific Visualization Symposium (PacificVis), pp. 199–206. IEEE (2015)Google Scholar
  17. 17.
    Ouyang, T.Y., Davis, R.: Chemink: a natural real-time recognition system for chemical drawings. In: Proceedings of the 16th International Conference on Intelligent User Interfaces, pp. 267–276. ACM (2011)Google Scholar
  18. 18.
    Rzeszotarski, J.M., Kittur, A.: Touchviz: (multi)touching multivariate data. In: CHI’13 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’13, pp. 3119–3122. ACM, New York (2013). doi:10.1145/2468356.2479625,
  19. 19.
    Rzeszotarski, J.M., Kittur, A.: Kinetica: naturalistic multi-touch data visualization. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, CHI’14, pp. 897–906. ACM, New York (2014). doi:10.1145/2556288.2557231,
  20. 20.
    Sadana, R., Stasko, J.: Designing and implementing an interactive scatterplot visualization for a tablet computer. In: Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces, AVI’14, pp. 265–272. ACM, New York (2014). doi:10.1145/2598153.2598163,
  21. 21.
    Schmieder, P., Plimmer, B., Dobbie, G.: Sketching er diagrams. In: Proceedings of the Tenth Australasian Conference on User Interfaces, vol. 93, pp. 53–60. Australian Computer Society, Inc. (2009)Google Scholar
  22. 22.
    Sun, Z., Wang, C., Zhang, L., Zhang, L.: Free hand-drawn sketch segmentation. In: Computer Vision–ECCV 2012, pp. 626–639. Springer (2012)Google Scholar
  23. 23.
    van Dam, A.: Post-wimp user interfaces. Commun. ACM 40(2), 63–67 (1997). doi:10.1145/253671.253708,
  24. 24.
    Walny, J., Lee, B., Johns, P., Riche, N.H., Carpendale, S.: Understanding pen and touch interaction for data exploration on interactive whiteboards. IEEE Trans. Vis. Comput. Graph. 18(12), 2779–2788 (2012)CrossRefGoogle Scholar
  25. 25.
    Xiong, Y., LaViola, J.J., Jr.: Revisiting shortstraw: improving corner finding in sketch-based interfaces. In: Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling, pp. 101–108. ACM (2009)Google Scholar
  26. 26.
    Zgraggen, E., Zeleznik, R., Drucker, S.: Panoramicdata: data analysis through pen and touch. IEEE Trans. Vis. Comput. Graph. 20(12), 2112–2121 (2014). doi:10.1109/TVCG.2014.2346293 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Andrés N. Vargas González
    • 1
    Email author
  • Eugene M. TarantaII
    • 1
  • Joseph J. LaViolaJr.
    • 1
  1. 1.Interactive Systems and User Experience Research Cluster of Excellence, Department of Computer ScienceUniversity of Central FloridaOrlandoUSA

Personalised recommendations