Abstract
Researchers at the Austrian Institute of Technology are exploring ways for the processing of media archives on large computer infrastructures using data intensive computing methods. The work is motivated by a strong demand for scalable methods that support the processing of media content such as can be found in archives of broadcasting or memory institutions. Data intensive computing frameworks leverage cloud technologies in order to generate and process large data sets on clusters of virtualized computers. MapReduce provides a highly scalable programming model in this context that has proven to be widely applicable for processing structured data. We have developed an approach and implementation that utilizes this model for the processing of audiovisual content. Here, we present an application that is capable of analyzing and modifying large audiovisual files using multiple computer nodes in parallel and thereby able to dramatically reduce processing times. The article provides detailed insights into the developed approach and the corresponding application. We summarize previous work and provide recent results that evaluate the application in a large-scale cloud environment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
The oscillating effect is due to the fact that nodes receive and finish map tasks at the same time throughout the job duration. Between these processing phases efficiency is remarkably low as almost all workers wait for new tasks to be scheduled by the master node.
References
Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51, 107–113 (January 2008)
Dean, J., Ghemawat, S.: Mapreduce: a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)
Ekanayake, J., Pallickara, S., Fox, G.: Mapreduce for data intensive scientific analyses. In: eScience, 2008. eScience ’08. IEEE Fourth International Conference on. pp. 277–284 (2008)
Gunarathne, T., Wu, T.L., Qiu, J., Fox, G.: Cloud computing paradigms for pleasingly parallel biomedical applications. In: Proceedings of the 19th ACM Int. Symposium on High Performance Distributed Computing. pp. 460–469. HPDC ’10 (2010)
Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conf. on Computer Systems 2007. pp. 59–72 (2007)
Moretti, C., Bui, H., Hollingsworth, K., Rich, B., Flynn, P., Thain, D.: All-pairs: An abstraction for data-intensive computing on campus grids. IEEE Transactions on Parallel and Distributed Systems 21, 33–46 (2010)
Pereira, R., Azambuja, M., Breitman, K., Endler, M.: An architecture for distributed high performance video processing in the cloud. In: Proceedings of the 2010 IEEE 3rd International Conference on Cloud Computing. pp. 482–489. CLOUD ’10 (2010)
Schmidt, R., Sadilek, C., King, R.: A service for data-intensive computations on virtual clusters. Intensive Applications and Services, International Conference on 0, 28–33 (2009)
Thusoo, A., Sarma, J., Jain, N., Shao, Z., Chakka, P., Zhang, N., Antony, S., Liu, H., Murthy, R.: Hive - a petabyte scale data warehouse using hadoop. In: Data Engineering (ICDE), 2010 IEEE 26th International Conference on. pp. 996–1005 (2010)
Warneke, D., Kao, O.: Nephele: efficient parallel data processing in the cloud. In: Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers. pp. 8:1–8:10. MTAGS ’09 (2009)
Yang, H.c., Dasdan, A., Hsiao, R.L., Parker, D.S.: Map-reduce-merge: simplified relational data processing on large clusters. In: Proceedings of the 2007 ACM SIGMOD international conference on Management of data. pp. 1029–1040. SIGMOD ’07 (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Schmidt, R., Rella, M. (2011). An Application for Processing Large and Non-Uniform Media Objects on MapReduce-Based Clusters. In: Furht, B., Escalante, A. (eds) Handbook of Data Intensive Computing. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-1415-5_26
Download citation
DOI: https://doi.org/10.1007/978-1-4614-1415-5_26
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-1414-8
Online ISBN: 978-1-4614-1415-5
eBook Packages: Computer ScienceComputer Science (R0)