Abstract
The paper presents a prototype of a distributed storage, analysis and processing system for multimedia (video, audio, image) content based on the Apache Hadoop platform. The core of the system is a distributed file system with advanced topology, replication and balancing capabilities. Metadata for stored media objects is being managed in a separate database in conformance with well-known metadata models and standards. In addition, the system allows implementation of efficient multimedia data analysis (e.g. face recognition) and processing (e.g. transcoding, digital watermarking) algorithms. Interoperability with external systems, such as the currently-developed Infona is provided by means of an OAI-PMH-compliant interface for metadata harvesting. More in-depth integration is possible by making storage, analysis and processing services available to external entities using the SYNAT integration platform. The article evaluates several important aspects of the proposed distributed repository, such as security, performance, extensibility and interoperability.
Work financed by the National Centre for Research and Development (NCBiR) within project no. SP/I/1/77065/10.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Open CV Open Source Computer Vision (2012), http://opencv.willowgarage.com
SYNAT Project (2012), http://www.synat.pl
Apache Software foundation: Hadoop 1.0.2 API Documentation (2012), http://hadoop.apache.org/common/docs/r1.0.2/api/
Borthakur, D.: HDFS Architecture Guide, Version 1.0.2 (2012), http://hadoop.apache.org/common/docs/r1.0.2/hdfs_design.html
Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. In: OSDI 2006: Seventh Symposium on Operating System Design and Implementation (2006)
Dean, J., Ghemawat, S.: Map Reduce: Simplified Data Processing on Large Clusters. Communications of the ACM 51(1), 1–13 (2008), http://www.usenix.org/events/osdi04/tech/full_papers/dean/dean_html/
Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. ACM SIGOPS Operating Systems Review 37(5), 29 (2003), http://portal.acm.org/citation.cfm?doid=1165389.945450
Kim, M., Lee, H., Cui, Y.: Performance Evaluation of Image Conversion Module Based on MapReduce for Transcoding and Transmoding in SMCCSE. In: 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing, pp. 396–403 (2011), http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6118756
Kiran, G.V.R.: Online Video Conversion Using Map Reduce. Tech. rep., International Institute of Information Technology, Hyderabad
Lin, J., Dyer, C.: Data-Intensive Text Processing with MapReduce, 1st edn. Morgan & Claypool Publishers (January 2010)
Sierra, S.: A Million Little Files (2008), http://stuartsierra.com/2008/04/24/a-million-little-files
Simon, B., Massart, D., van Assche, F., Ternier, S., Duval, E., Brantner, S., Olmedilla, D., Miklós, Z.: A simple query interface for interoperable learning repositories. In: Proceedings of the 1st Workshop on Interoperability of Web-Based Educational Systems, pp. 11–18 (2005)
Sweeney, C., Liu, L., Arietta, S., Lawrence, J.: HIPI: A Hadoop Image Processing Interface for Image-based MapReduce Tasks. Master’s thesis, University of Virginia (2011)
White, T.: Hadoop: The Definitive Guide. O’Reilly Series, vol. 54. O’Reilly Media (2009)
White, T.: The Small Files Problem (2009), http://www.cloudera.com/blog/2009/02/the-small-files-problem/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Dziech, A., Glowacz, A., Wszołek, J., Ernst, S., Pawłowski, M. (2013). A Distributed Architecture for Multimedia File Storage, Analysis and Processing. In: Bembenik, R., Skonieczny, L., Rybinski, H., Kryszkiewicz, M., Niezgodka, M. (eds) Intelligent Tools for Building a Scientific Information Platform. Studies in Computational Intelligence, vol 467. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35647-6_26
Download citation
DOI: https://doi.org/10.1007/978-3-642-35647-6_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-35646-9
Online ISBN: 978-3-642-35647-6
eBook Packages: EngineeringEngineering (R0)