Visual time period analysis: a multimedia analytics application for summarizing and analyzing eye-tracking experiments

  • Vincenzo Del FattoEmail author
  • Anton Dignös
  • Guerriero Raimato
  • Lorenzo Maccioni
  • Yuri Borgianni
  • Johann Gamper


Recently, an increasing need for sophisticated multimedia analytics tools has been observed, which is triggered by a rapid growth of multimedia collections and by an increasing number of scientific fields embedding images in their studies. Although temporal data is ubiquitous and crucial in many applications, such tools typically do not support the analysis of data along the temporal dimension, especially for time periods. An appropriate visualization and comparison of period data associated with multimedia collections would help users to infer new information from such collections. In this paper, we present a novel multimedia analytics application for summarizing and analyzing temporal data from eye-tracking experiments. The application combines three different visual approaches: Timediff, visual-information-seeking mantra, and multi-viewpoint. A qualitative evaluation with domain experts confirmed that our application helps decision makers to summarize and analyze multimedia collections containing period data.


Data visualization Period data Multimedia analytics Multimedia application 



We would like to thank our colleague Prof. Demis Basso, Faculty of Education of the Free University of Bozen-Bolzano, who provided insight and expertise that greatly assisted the research.


  1. 1.
    Aigner W, Miksch S (2006) Carevis: integrated visualization of computerized protocols and temporal patient data. Artif Intell Med 37(3):203–218CrossRefGoogle Scholar
  2. 2.
    Aigner W, Miksch S, Thurnher B, Biffl S (2005) Planninglines: novel glyphs for representing temporal uncertainties and their evaluation. In: IV, pp 457–463Google Scholar
  3. 3.
    Aigner W, Miksch S, Müller W, Schumann H, Tominski C (2008) Visual methods for analyzing time-oriented data. IEEE Trans Vis Comput Graph 14(1):47–60CrossRefGoogle Scholar
  4. 4.
    André P, Wilson M, Russell A, Smith D, Owens A, Schraefel M (2007) Continuum: designing timelines for hierarchies, relationships and scale, pp 101–110Google Scholar
  5. 5.
    Ankerst M, Kao A, Tjoelker R, Wang C (2008) Datajewel: integrating visualization with temporal data mining. In: Visual data mining. Springer, pp 312–330Google Scholar
  6. 6.
    Behrend A, Schmiegelt P, Xie J, Fehling R, Ghoneimy A, Liu ZH, Chan ES, Gawlick D (2014) Temporal state management for supporting the real-time analysis of clinical data. In: ADBIS, pp 159–170Google Scholar
  7. 7.
    Benito A, Losada AG, Therón R, Dorn A, Seltmann M, Wandl-Vogt E (2016) A spatio-temporal visual analysis tool for historical dictionaries. In: Proceedings of the fourth international conference on technological ecosystems for enhancing multiculturality, TEEM ’16. ACM, New York, pp 985–990Google Scholar
  8. 8.
    Böhlen MH, Dignös A, Gamper J, Jensen CS (2018) Database technology for processing temporal data (invited paper). In: TIME, vol 120 of LIPIcs, pp 2:1–2:7. Schloss Dagstuhl - Leibniz-Zentrum fuer InformatikGoogle Scholar
  9. 9.
    Böhlen MH, Dignös A, Gamper J, Jensen CS (2018) Temporal data management – an overview. In: Business intelligence and big data. Springer International Publishing, pp 51–83Google Scholar
  10. 10.
    Burch M, Pompe D, Weiskopf D (2015) An analysis and visualization tool for dblp data. volume 2015-September, pp 163–170 Institute of Electrical and Electronics Engineers Inc.Google Scholar
  11. 11.
    Chen H, Atabakhsh H, Tseng C, Marshall B, Kaza S, Eggers S, Gowda H, Shah A, Petersen T, Violette C (2005) Visualization in law enforcement. In: DG.O, pp 229–230Google Scholar
  12. 12.
    Chinchor N, Thomas J, Wong P, Christel M, Ribarsky W (2010) Multimedia analysis + visual analytics = multimedia analytics. IEEE Comput Graph Appl 30(5):52–60CrossRefGoogle Scholar
  13. 13.
    Chittaro L, Combi C (2003) Visualizing queries on databases of temporal histories: new metaphors and their evaluation. Data Knowl Eng 44(2):239–264CrossRefGoogle Scholar
  14. 14.
    Combi C, Oliboni B (2012) Visually defining and querying consistent multi-granular clinical temporal abstractions. Artif Intell Med 54(2):75–101CrossRefGoogle Scholar
  15. 15.
    De Chiara D, Del Fatto V, Laurini R, Sebillo M, Vitiello G (2011) A chorem-based approach for visually analyzing spatial data. J Vis Lang Comput 22 (3):173–193CrossRefGoogle Scholar
  16. 16.
    De Chiara D, Del Fatto V, Sebillo M (2012) Visualizing geographical information through tag clouds. Springer, New YorkCrossRefGoogle Scholar
  17. 17.
    De Chiara D, Del Fatto V, Sebillo M, Tortora G, Vitiello G (2012) Tag@map: a web-based application for visually analyzing geographic information through georeferenced tag clouds. In: Proceedings of the 11th international conference on web and wireless geographical information systems, w2GIS’12. Springer-Verlag, Berlin, pp 72–81Google Scholar
  18. 18.
    Del Fatto V, Dignös A, Gamper J (2018) Time diff: a visual approach to compare period data. In: iV2018. IEEE Computer Society, p 7Google Scholar
  19. 19.
    Dignös A., Böhlen MH, Gamper J (2012) Temporal alignment. In: SIGMOD, pp 433–444Google Scholar
  20. 20.
    Dignös A., Böhlen MH, Gamper J (2013) Query time scaling of attribute values in interval timestamped databases. In: ICDE, pp 1304–1307Google Scholar
  21. 21.
    Dignös A, Böhlen MH, Gamper J, Jensen CS (2016) Extending the kernel of a relational DBMS with comprehensive support for sequenced temporal queries. ACM Trans Database Syst 41(4):26:1–26:46MathSciNetCrossRefGoogle Scholar
  22. 22.
    Dignös A, Glavic B, Niu X, Böhlen MH, Gamper J (2019) Snapshot semantics for temporal multiset relations. PVLDB 12(6):639–652Google Scholar
  23. 23.
    Fischer F, Fuchs J, Vervier P-A, Mansmann F, Thonnard O (2012) Vistracer: a visual analytics tool to investigate routing anomalies in traceroutes, pp 80–87Google Scholar
  24. 24.
    Gamper J, Böhlen MH, Jensen CS (2009) Temporal aggregation. In: Encyclopedia of database systems, pp 2924–2929. Springer USGoogle Scholar
  25. 25.
    Gregersen H, Jensen CS (1999) Temporal entity-relationship models - a survey. IEEE Trans Knowl Data Eng 11(3):464–497CrossRefGoogle Scholar
  26. 26.
    Hochheiser H, Shneiderman B (2004) Dynamic query tools for time series data sets Timebox widgets for interactive exploration. Inf Vis 3(1):1–18CrossRefGoogle Scholar
  27. 27.
    Jensen CS, Snodgrass RT (1999) Temporal data management. IEEE Trans Knowl Data Eng 11(1):36–44CrossRefGoogle Scholar
  28. 28.
    Jensen M (2003) Visualizing complex semantic timelines. Technical Report MSU-CSE-06-2 NewsBlipGoogle Scholar
  29. 29.
    Kaptelinin V, Czerwinski M (2007) Beyond the desktop metaphor: designing integrated digital work environments. MIT Press, CambridgeCrossRefGoogle Scholar
  30. 30.
    Keim DA, Mansmann F, Schneidewind J, Ziegler H, Thomas J Visual analytics: scope and challenges. December 2008. Visual data mining: theory, techniques and tools for visual analytics, Springer, Lecture Notes in Computer Science (lncs)Google Scholar
  31. 31.
    Keim ED, Kohlhammer J, Ellis G (2010), Mastering the information age: solving problems with visual analytics, eurographics associationGoogle Scholar
  32. 32.
    Kulkarni KG, Michels J-E (2012) Temporal features in sql: 2011. SIGMOD Record 41:34–43CrossRefGoogle Scholar
  33. 33.
    Lee JH, Ostwald MJ (2018) Measuring cognitive complexity in parametric design. International Journal of Design Creativity and Innovation 0(0):1–21Google Scholar
  34. 34.
    Liu T, Bouali F, Venturini G (2014) Technical section Exod: a tool for building and exploring a large graph of open datasets. Comput Graph 39:117–130CrossRefGoogle Scholar
  35. 35.
    Luo X, Tian F, Liu W, Teng D, Dai G, Wang H (2010) Visualizing time-series data in processlines: design and evaluation of a process enterprise application. In: SAC, pp 1165–1172Google Scholar
  36. 36.
    Maccioni L, Borgianni Y, Basso D (2019) Value perception of green products: an exploratory study combining conscious answers and unconscious behavioral aspects. Sustainability 11(5), 1226;
  37. 37.
    Mahlknecht G, Böhlen MH, Dignös A, Gamper J (2017) VISOR: Visualizing summaries of ordered data. In: SSDBM, pp 40:1–40:5Google Scholar
  38. 38.
    Mahlknecht G, Dignös A, Gamper J (2017) A scalable dynamic programming scheme for the computation of optimal k-segments for ordered data. Inf Syst 70:2–17CrossRefGoogle Scholar
  39. 39.
    Meghdadi A, Irani P (2013) Interactive exploration of surveillance video through action shot summarization and trajectory visualization. IEEE Trans Vis Comput Graph 19(12):2119–2128CrossRefGoogle Scholar
  40. 40.
    Melton J (2006) Database language SQL. Springer, Berlin, pp 105–132Google Scholar
  41. 41.
    Olsson J, Boldt M (2009) Computer forensic timeline visualization tool. Digit Investig 6(SUPPL.):S78–S87CrossRefGoogle Scholar
  42. 42.
    Plaisant C, Milash B, Rose A, Widoff S, Shneiderman B (1996) Lifelines: visualizing personal histories. In: CHI, pp 221–227Google Scholar
  43. 43.
    Richter HA, Brotherton JA, Abowd GD, Truong KN (1999) A multi-scale timeline slider for stream visualization and control. Technical Report GIT-GVU TR 99-30 Georgia Institute of TechnologyGoogle Scholar
  44. 44.
    Rooij O, Van Wijk J, Worring M (2010) Mediatable: interactive categorization of multimedia collections. IEEE Comput Graph Appl 30(5):42–51CrossRefGoogle Scholar
  45. 45.
    Schüller G, Schmiegelt P, Behrend A (2015) Air traffic monitoring using datastream analysis techniques. In: 18th International Conference on Information Fusion, FUSION 2015, Washington, DC, USA, July 6-9, 2015. IEEE, pp 1238–1245Google Scholar
  46. 46.
    Seyfert M, Viola I (2017) Dynamic word clouds. In: Proceedings of the 33rd spring conference on computer graphics, SCCG ’17. ACM, New York, pp 7:1–7:8Google Scholar
  47. 47.
    Shmueli G, Jank W, Aris A, Plaisant C, Shneiderman B (2006) Exploring auction databases through interactive visualization. Decis Support Syst 42(3):1521–1538CrossRefGoogle Scholar
  48. 48.
    Shneiderman B (1996) The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings 1996 IEEE symposium on visual languages, pp 336–343Google Scholar
  49. 49.
    Silva SF, Catarci T (2000) Visualization of linear time-oriented data a survey. In: WISE, pp 310–319Google Scholar
  50. 50.
    Steele J, Iliinsky N (2010) Beautiful visualization: looking at data through the eyes of experts. O’Reilly Media, Inc. 1st editionGoogle Scholar
  51. 51.
    Tominski C, Abello J, Schumann H (2004) Axes-based visualizations with radial layouts. In: SAC, pp 1242–1247Google Scholar
  52. 52.
    Weber M, Alexa M, Müller W (2001) Visualizing time-series on spirals. In: INFOVIS, pp 7–14Google Scholar
  53. 53.
    Worring M, Engl A, Smeria C (2012) A multimedia analytics framework for browsing image collections in digital forensics. In: Proceedings of the 20th ACM international conference on multimedia, MM ’12. ACM, New York, pp 289–298Google Scholar
  54. 54.
    Yang J, Luo D, Liu Y (2010) Newdle: interactive visual exploration of large online news collections. IEEE Comput Graph Appl 30(5):32–41CrossRefGoogle Scholar
  55. 55.
    Zahálka J, Worring M (2014) Towards interactive, intelligent, and integrated multimedia analytics. In: 2014 IEEE conference on visual analytics science and technology (VAST), pp 3–12Google Scholar
  56. 56.
    Zhou X, Wang F, Zaniolo C (2006) Efficient temporal coalescing query support in relational database systems. In: DEXA, pp 676–686Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Faculty of Computer ScienceFree University of Bozen-BolzanoBolzanoItaly
  2. 2.Faculty of Science and TechnologyFree University of Bozen-BolzanoBolzanoItaly

Personalised recommendations