Journal of Grid Computing

, Volume 11, Issue 3, pp 523–551 | Cite as

Semantics and Planning Based Workflow Composition for Video Processing

  • Gayathri Nadarajan
  • Yun-Heh Chen-Burger
  • Robert B. Fisher


This work proposes a novel workflow composition approach that hinges upon ontologies and planning as its core technologies within an integrated framework. Video processing problems provide a fitting domain for investigating the effectiveness of this integrated method as tackling such problems have not been fully explored by the workflow, planning and ontological communities despite their combined beneficial traits to confront this known hard problem. In addition, the pervasiveness of video data has proliferated the need for more automated assistance for image processing-naive users, but no adequate support has been provided as of yet. The integrated approach was evaluated on a video set originating from open sea environment of varying quality. Experiments to evaluate the efficiency, adaptability to user’s changing needs and user learnability of this approach were conducted on users who did not possess image processing expertise. The findings indicate that using this integrated workflow composition and execution method: (1) provides a speed up of over 90 % in execution time for video classification tasks using full automatic processing compared to manual methods without loss of accuracy; (2) is more flexible and adaptable in response to changes in user requests than modifying existing image processing programs when the domain descriptions are altered; and (3) assists the user in selecting optimal solutions by providing recommended descriptions.


Automatic workflow composition Intelligent video analysis Ontology-based workflows HTN planning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Andrews, T., Curbera, F., Dholakia, H., Goland, Y., Klein, J., Leymann, F., Liu, K., Roller, D., Smith, D., Thatte, S., Trickovic, I., Weerawarana, S.: Business Process Execution Language for Web Services Version 1.1 (BPEL). IBM, BEA Systems, Microsoft, SAP AG, Siebel Systems (2003). Last accessed 22 April 2013
  2. 2.
    Blythe, J., Deelman, E., Gil, Y.: Automatically composed workflows for Grid environments. IEEE Intell. Syst. 19(4):16–23 (2004)CrossRefGoogle Scholar
  3. 3.
    Callahan, S.P., Freire, J., Santos, E., Scheidegger, C.E., Silva, C.T., Vo, H.T.: Managing the evolution of dataflows with vistrails. In: IEEE Workshop on Workflow and Data Flow for Scientific Applications (Sciflow’06), pp. 71–75 (2006)Google Scholar
  4. 4.
    Chen-Burger, Y.H., Lin, F.P.: A semantic-based workflow choreography for integrated sensing and processing. In: The 9th IEEE International Workshop on Cellular Neural Networks and Their Applications (CNNA’05) (2005)Google Scholar
  5. 5.
    Currie, K., Tate, A.: O-Plan: The open planning architecture. Artif. Intell. 52, 49–86 (1991)CrossRefGoogle Scholar
  6. 6.
    Deelman, E., Gannon, D., Shields, M., Taylor, I.: Workflows and e-Science: An overview of workflow system features and capabilities. Future Gener. Comput. Syst. 25(5), 528–540 (2009)CrossRefGoogle Scholar
  7. 7.
    Deelman, E., Singh, G., Su, M.H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Patil, S., Vahi, K., Berriman, B., Good, J., Laity, A., Jacob, J.C., Katz, D.S.: Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci. Program. J. 13(3), 219–237 (2005)Google Scholar
  8. 8.
    Ecogrid: National Center for High Performance Computing, Hsin-Chu, Taiwan (2006). Last accessed 21 April 2012
  9. 9.
    Foster, I., Kesselman, C. (eds.): The Grid 2: Blueprint for a New Computing Infrastructure, 2nd edn. Morgan Kaufmann (2003)Google Scholar
  10. 10.
    Foster, I., Voeckler, J., Wilde, M., Zhao, Y.: Chimera: A virtual data system for representing, querying, and automating data derivation. In: 14th Conference on Scientific and Statistical Database Management, pp. 37–46 (2002)Google Scholar
  11. 11.
    Ghallab, M., Nau, D., Traverso, P.: Automated Planning: Theory & Practice. Morgan Kaufmann (2004)Google Scholar
  12. 12.
    Gil, Y., Ratnakar, V., Deelman, E., Mehta, G., Kim, J.: Wings for Pegasus: creating large-scale scientific applications using semantic representations of computational workflows. In: Proceedings of the 19th National Conference on Innovative Applications of Artificial Intelligence (IAAI’07), pp. 1767–1774. AAAI Press (2007)Google Scholar
  13. 13.
    Gómez-Pérez, A., Fernández-López, M., Corcho, O.: Ontological Engineering. Springer (2004)Google Scholar
  14. 14.
    Howell, D.C.: Statistical Methods for Psychology, 6th edn. Belmont, CA (2007)Google Scholar
  15. 15.
    Intel: Open Source Computer Vision (OpenCV) Library (2006). Last accessed 22 April 2013
  16. 16.
    Kim, J., Spraragen, M., Gil, Y.: An intelligent assistant for interactive workflow composition. In: Proceedings of the International Conference on Intelligent User Interfaces (IUI’04), pp. 125–131. ACM Press (2004)Google Scholar
  17. 17.
    Ludäscher, B., Altintas, I., Berkley, C., Higgins, D., Jaeger, E., Jones, M., Lee, E.A., Tao, J., Zhao, Y.: Scientific workflow management and the Kepler system. Concurr. Comput. Pract. Exper. 18(10), 1039–1065 (2005)CrossRefGoogle Scholar
  18. 18.
    McCluskey, T.L., Liu, D., Simpson, R.M.: GIPO II: HTN planning in a tool-supported knowledge engineering environment. In: ICAPS’03, pp. 92–101 (2003)Google Scholar
  19. 19.
    Nadarajan, G., Chen-Burger, Y.H.: Goal, video descriptions and capability ontologies for Fish4Knowledge domain. In: Special Session on Intelligent Workflow, Cloud Computing and Systems (KES-AMSTA’12) (2012)Google Scholar
  20. 20.
    Nadarajan, G., Chen-Burger, Y.H., Fisher, R.B.: A knowledge-based planner for processing unconstrained underwater videos. In: IJCAI’09 Workshop on Learning Structural Knowledge from Observations (STRUCK’09) (2009)Google Scholar
  21. 21.
    Nadarajan, G., Chen-Burger, Y.H., Fisher, R.B.: SWAV: Semantics-based workflows for automatic video analysis. In: Special Session on Intelligent Workflow, Cloud Computing and Systems (KES-AMSTA’11) (2011)Google Scholar
  22. 22.
    Nadarajan, G., Renouf, A.: A modular approach for automating video processing. In: 12th International Conference on Computer Analysis of Images and Patterns (CAIP’07) (2007)Google Scholar
  23. 23.
    Nadarajan, G., Spampinato, C., Chen-Burger, Y.H., Fisher, R.B.: A flexible system for automated composition of intelligent video analysis. In: 7th International Symposium on Image and Signal Processing and Analysis (ISPA’11) (2011)Google Scholar
  24. 24.
    Nau, D., Au, T.C., Ilghami, O., Kuter, U., W.Murdock, Wu, D., Yaman, F.: SHOP2: An HTN planning system. J. Artif. Intell. Res. 20, 379–404 (2003)Google Scholar
  25. 25.
    Nau, D.S., Cao, Y., Lotem, A., Muñoz Avila, H.: SHOP: Simple hierarchical ordered planner. In: International Joint Conference on Artificial Intelligence (IJCAI’99), pp. 968–973 (1999)Google Scholar
  26. 26.
    Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M.R., Wipat, A., Li, P.: Taverna: A tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20(17), 3045–3054 (2004)CrossRefGoogle Scholar
  27. 27.
    Oinn, T., Greenwood, M., Addis, M., Alpdemir, N., Ferris, J., Glover, K., Goble, C., Goderis, A., Hull, D., Marvin, D., Li, P., Lord, P., Pocock, M., Senger, M., Stevens, R., Wipat, A., Wroe, C.: Taverna: Lessons in creating a workflow environment for the life sciences. Concurr. Comput. Pract. Exper. 18(10), 1067–1100 (2006)CrossRefGoogle Scholar
  28. 28.
    Renouf, A., Clouard, R.: Hermes—A human-machine interface for the formulation of image processing applications (2007). Last accessed 22 April 2013
  29. 29.
    Tannenbaum, T., Wright, D., Miller, K., Livny, M.: Condor—A distributed job scheduler. In: Sterling, T. (ed.) Beowulf Cluster Computing with Linux. MIT Press (2001)Google Scholar
  30. 30.
    Tate, A.: Intelligible AI planning—generating plans represented as a set of constraints. In: Proceedings of the Twentieth British Computer Society Special Group on Expert Systems International Conference on Knowledge Based Systems and Applied Artificial Intelligence (ES’00), pp. 3–16. Springer (2000)Google Scholar
  31. 31.
    Taylor, I., Deelman, E., Gannon, D., Shields, M.: Workflows for e-Science. Springer, New York (2007)CrossRefGoogle Scholar
  32. 32.
    Taylor, I., Shields, M., Wang, I., Rana, O.: Triana applications within Grid computing and peer to peer environments. J. Grid Computing 1(2), 199–217 (2003)CrossRefGoogle Scholar
  33. 33.
    Taylor, I., Shields, M., Wang, I., Harrison, A.: The Triana workflow environment: architecture and applications. In: Taylor, I., Deelman, E., Gannon, D., Shields, M. (eds.) Workflows for e-Science, pp. 320–339. Springer, New York (2007)CrossRefGoogle Scholar
  34. 34.
    van Splunter, S., Brazier, F.M.T., Padget, J.A., Rana, O.F.: Dynamic service reconfiguration and enactment using an open matching architecture. In: International Conference on Agents and Artificial Intelligence (ICAART’09), pp. 533–539 (2009)Google Scholar
  35. 35.
    von Laszewski, G., Hategan, M.: Java CoG Kit Karajan/Gridant workflow guide. Technical report, Argonne National Laboratory, Argonne, IL, USA (2005)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Gayathri Nadarajan
    • 1
  • Yun-Heh Chen-Burger
    • 1
  • Robert B. Fisher
    • 2
  1. 1.Centre for Intelligent Systems and their Applications, School of InformaticsUniversity of EdinburghEdinburghUK
  2. 2.Institute for Perception, Action and Behaviour, School of InformaticsUniversity of EdinburghEdinburghUK

Personalised recommendations