Multimodal Web Based Video Annotator with Real-Time Human Pose Estimation

  • Rui RodriguesEmail author
  • Rui Neves Madeira
  • Nuno Correia
  • Carla Fernandes
  • Sara Ribeiro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11872)


This paper presents a multi-platform Web-based video annotator to support multimodal annotation that can be applied to several working areas, such as dance rehearsals, among others. The CultureMoves’ “Motion-Notes” Annotator was designed to assist the creative and exploratory processes of both professional and amateur users, working with a digital device for personal annotations. This prototype is being developed for any device capable of running in a modern Web browser. It is a real-time multimodal video annotator based on keyboard, touch and voice inputs. Five different ways of adding annotations have been already implemented: voice, draw, text, web URL, and mark annotations. Pose estimation functionality uses machine learning techniques to identify a person skeleton in the video frames, which gives the user another resource to identify possible annotations.


Multimodal video annotations Real-time human pose estimation Machine learning for creativity 



This work was supported by the project CultureMoves, Grant Agreement Number: INEA/CEF/ICT/A2017/1568369, Action No: 2017-EU-tA-0171.


  1. 1.
    Cabral, D., Valente, J., Silva, J., Aragão, U., Fernandes, C., Correia, N.: A creation-tool for contemporary dance using multimodal video annotation. In: Proceedings of the 19th ACM International Conference on Multimedia, MM 2011, pp. 905–908. ACM, New York (2011).
  2. 2.
    Silva, J.M.F., Cabral, D., Fernandes, C., Correia, N.: Real-time annotation of video objects on tablet computers. In: MUM 2012, p. 19 (2012)Google Scholar
  3. 3.
    Cabral, D., Valente, J., Aragão, U., Fernandes, C., Correia, N.: Evaluation of a multimodal video annotator for contemporary dance. In: AVI 2012Google Scholar
  4. 4.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. 38(4), 13:1–13:45 (2006)CrossRefGoogle Scholar
  5. 5.
    Han, J., Shao, L., Xu, D., Shotton, J.: Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans. Cybernet. 43(5), 1318–1334 (2013). Scholar
  6. 6.
    Kawana, Y., Ukita, N., Huang, J.-B., Yang, M.-H.: Ensemble convolutional neural networks for pose estimation. Comput. Vis. Image Underst. 169, 62–74 (2018). ISSN 1077-3142CrossRefGoogle Scholar
  7. 7.
  8. 8.
    Cao, Z., Simon, T., Wei, S., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI (2017).
  9. 9.
    Bargeron, D., Gupta, A., Grudin, J., Sanocki, E.: Annotations for streaming video on the Web: system design and usage studies. Comput. Netw. 31(11–16), 1139–1153 (1999). ISSN 1389-1286CrossRefGoogle Scholar
  10. 10.
    Lausberg, H., Sloetjes, H.: Behav. Res. Methods 41, 841 (2009). Scholar
  11. 11.
    Correia, N., Chambel, T.: Active video watching using annotation. In: Proceedings of the Seventh ACM International Conference on Multimedia (Part 2) (MULTIMEDIA 1999), pp. 151–154. ACM, New York (1999)Google Scholar
  12. 12.
    Goldman, D.B., Gonterman, C., Curless, B., Salesin, D., Seitz, S.M.: Video object annotation, navigation, and composition. In: Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST 2008, New York, USA (2008)Google Scholar
  13. 13.
    Marshall, C.C.: Toward an ecology of hypertext annotation. In: Proceedings of the Ninth ACM Conference on Hypertext and Hypermedia, HYPERTEXT 1998. ACM, New York (1998)Google Scholar
  14. 14.
    Europeana. Accessed 31 July 2019
  15. 15.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Rui Rodrigues
    • 1
    • 2
    Email author
  • Rui Neves Madeira
    • 1
    • 2
  • Nuno Correia
    • 2
  • Carla Fernandes
    • 3
  • Sara Ribeiro
    • 3
  1. 1.Sustain.RD Center, ESTSetúbal, Polytechnic InstituteSetúbalPortugal
  2. 2.NOVA LINCS, DI, FCTNOVA University of LisboaLisbonPortugal
  3. 3.ICNOVA, FCSHNOVA University of LisboaLisbonPortugal

Personalised recommendations