Skip to main content
SpringerLink
Log in
Book cover

International Conference on Big Data Computing and Communications

BigCom 2016: Big Data Computing and Communications pp 309–318Cite as

Efficient Online Surveillance Video Processing Based on Spark Framework

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9784))

Abstract

In the current surveillance video processing systems, the video processing algorithms and the physical resources are highly coupled, and a video stream is usually used as a basic task scheduling unit. With the expansion of the scale of the system, the traditional systems will cause the large resource fragments that cannot be utilized adequately. In this paper, we propose a novel online surveillance video processing system architecture that combines the distributed Kafka message queue and Spark computing framework. Our system decouples the video stream collection and the video stream processing, and further decouples the video processing tasks and the physical resources. This loosely coupled architecture can quickly recover the failed tasks without data loss for the large-scale video surveillance, and can provide the more scalable distributed computing ability. In addition, a fine-grained online video task management method, which uses the cached video data blocks as the scheduling units, is proposed to increase the resource utilization. Experimental results show that our system has the higher resource utilization and the higher task capacity compared with the traditional systems.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Zhao, X.M., Ma, H.D., Zhang, H.T., Tang, Y., Fu, G.P.: Metadata extraction and correction for large-scale traffic surveillance videos. In: Proceedings of IEEE Big Data 2014, pp. 412–420 (2014)

    Google Scholar 

  2. Yang, X., Zhang, H., Ma, H., Li, W., Fu, G., Tang, Y.: Multi-resource allocation for virtual machine placement in video surveillance cloud. In: Zu, Q., Hu, B. (eds.) HCC 2016. LNCS, vol. 9567, pp. 544–555. Springer, Heidelberg (2016). doi:10.1007/978-3-319-31854-7_49

    Chapter  Google Scholar 

  3. Zhang, W., Xu, L., Duan, P., Gong, W., Liu, X., Lu, Q.: Towards a high speed video cloud based on batch processing integrated with fast processing. In: Proceedings of IIKI 2014, pp. 28–33 (2014)

    Google Scholar 

  4. Benfold, B., Reid, I.: Stable multi-target tracking in real-time surveillance video. In: Proceedings of CVPR 2011, pp. 3457–3464 (2011)

    Google Scholar 

  5. Chowdhury, A., Tripathy, S.: Detection of human presence in a surveillance video using fuzzy approach. In: Proceedings of SPIN 2014, pp. 216–219 (2014)

    Google Scholar 

  6. Hossain, M., Hassan, M., Qurishi, M., Alghamdi, A.: Resource allocation for service composition in cloud based video surveillance platform. In: Proceedings of IEEE ICME Workshops 2012, pp. 408–412 (2012)

    Google Scholar 

  7. Zhao, X.M., Ma, H.D., Zhang, H.T., Tang, Y., Kou, Y.: HVPI: extending Hadoop to support video analytic applications. In: Proceedings of IEEE Cloud 2015, pp. 789–796 (2015)

    Google Scholar 

  8. Hu, R., Jiang, J., Liu, G., Wang, L.: KSwSVR: a new load forecasting method for efficient resources provisioning in cloud. In: Proceedings of IEEE 10th SCC, pp. 120–127 (2013)

    Google Scholar 

  9. Zhao, H., Pan, M., Liu, X., Li, X., Fang, Y.: Exploring fine-grained resource rental planning in cloud computing. IEEE Trans. Cloud Comput. 3(3), 304–317 (2015)

    Article  Google Scholar 

  10. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  11. Apache Storm. http://storm.apache.org/

  12. Zaharia, M., Chowdhury, M., Franklin, M., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of HotCloud 2010 (2010)

    Google Scholar 

  13. Ryu, C., Lee, D., Jang, M., Kim, C., Seo, E.: Extensible video processing framework in Apache Hadoop. In: Proceedings of IEEE 5th CloudCom 2013, vol. 2, pp. 305–310 (2013)

    Google Scholar 

  14. Wang, H., Zheng, X., Bo, X.: Large-scale human action recognition with Spark. In: Proceedings of MMSP 2015 (2015)

    Google Scholar 

  15. Apache Kafka: A high-throughput distributed messaging system. http://Kafka.apache.org/

  16. Xuggler. http://www.xuggle.com/xuggler/

  17. Zaharia, M., Chowdhury, M., Das, T., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of NSDI 2012 (2012)

    Google Scholar 

  18. ZooKeeper. http://zookeeper.apache.org/

Download references

Acknowledgment

This work is supported by the National High Technology Research and Development Program of China (No. 2014AA015101); National Natural Science Foundation of China (No. 61300013); Doctoral Program Foundation of Institutions of Higher Education of China (No. 20130005120011); Asia Foresight Program under NSFC Grant No. 61411146001.

Author information

Authors and Affiliations

  1. Beijing Key Lab of Intelligent Telecommunication Software and Multimedia, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Haitao Zhang, Jin Yan & Yue Kou

Authors
  1. Haitao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yue Kou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Yan .

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of N. Carolina at Charlotte, Charlotte, North Carolina, USA

    Yu Wang

  2. Northeastern University, College of Information Science and Engineering, Shenyang, Liaoning, China

    Ge Yu

  3. Department of Electrical & Computer Engineering, Rutgers University, Piscataway, New Jersey, USA

    Yanyong Zhang

  4. Department of Electrical and Computer Engineering, University of Houston Department of Engineering, Houston, Texas, USA

    Zhu Han

  5. College of Information Science and Engineering, Northeastern University, Shenyang , Liaoning, China

    Guoren Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Zhang, H., Yan, J., Kou, Y. (2016). Efficient Online Surveillance Video Processing Based on Spark Framework. In: Wang, Y., Yu, G., Zhang, Y., Han, Z., Wang, G. (eds) Big Data Computing and Communications. BigCom 2016. Lecture Notes in Computer Science(), vol 9784. Springer, Cham. https://doi.org/10.1007/978-3-319-42553-5_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-42553-5_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-42552-8

  • Online ISBN: 978-3-319-42553-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation