Encyclopedia of Big Data Technologies

Living Edition
| Editors: Sherif Sakr, Albert Zomaya

Hardware Reliability Requirements

  • Wenhao LiEmail author
Living reference work entry
DOI: https://doi.org/10.1007/978-3-319-63962-8_173-1


The development of big data applications has brought a great amount of opportunities yet challenges to both the IT industry and academia. Supporting platforms and architectures, such as Clouds, supercomputing centers and data centers, must develop corresponding technologies to meet the “4V” requirements of big data applications, which are veracity, velocity, volume, and variety. Serving as the fundamental layer of these platforms and architectures, the hardware, especially storage hardware, are of great importance to the big data applications, where the reliability of storage units not only have significantly influence the reliability of the data, but also have significant impact on the “4V” requirements of the big data applications. In this article, we discuss the hardware reliability requirements, specifically, storage hardware reliability requirements, for current big data platforms and architectures.

Big Data Storage Architecture

A typical big data application contains...

This is a preview of subscription content, log in to check access.


  1. CAI Y et al (2012) Error patterns in MLC NAND flash memory: measurement, characterization, and analysis. In: Design, automation & test in Europe conference & exhibition, DresdenGoogle Scholar
  2. Dabrowski C (2009) Reliability in grid computing systems. Concurr Comput Pract Experience 21(8):927–959CrossRefGoogle Scholar
  3. Furrer S, Lantz MA, Reininger P (2017) 201 Gb/in2 recording areal density on sputtered magnetic tape. IEEE Trans Magn 99:1–1Google Scholar
  4. Gersback T (2012) Technical article: Energy-efficient climate control “keep cool” in the data centre. RITTAL Pty, New South WalesGoogle Scholar
  5. Harris R (2014) Amazon’s Glacier secret: BDXL. [cited 2017]; Available from https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/
  6. Huang C et al (2012) Erasure coding in Windows Azure storage. In: USENIX annual technical conference, BostonGoogle Scholar
  7. IDEMA (1998) R2–98: specification of hard disk drive reliability. IDEMA StandardsGoogle Scholar
  8. Klein A (2017) Hard drive stats for Q2 2017. [cited 2017]; Available from https://www.backblaze.com/blog/hard-drive-failure-stats-q2-2017/
  9. Li W, Yang Y, Yuan D (2015) Ensuring cloud data reliability with minimum replication by proactive replica checking. IEEE Trans Comput 65:1–13MathSciNetzbMATHGoogle Scholar
  10. Loken C et al (2010) SciNet: lessons learned from building a power-efficient top-20 system and data centre. J Phys Conf Ser 256(1):1–35MathSciNetGoogle Scholar
  11. Mielke N et al (2008) Bit error rate in NAND flash memories. In: IEEE international reliability physics symposium, PhoenixGoogle Scholar
  12. Patterson D, Gibson G, Katz R (1988) A case for redundant arrays of inexpensive disks (RAID). In: ACM SIGMOD international conference on the management of data, ChicagoGoogle Scholar
  13. Pinheiro E, Weber W, Barroso LA (2007) Failure trends in a large disk drive population. In: USENIX conference on file and storage technologies, San JoseGoogle Scholar
  14. Rashmi KV et al (2013) A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the Facebook warehouse cluster. In: USENIX HotStorage, San JoseGoogle Scholar
  15. Rashmi KV et al (2014) A hitchhiker’s guide to fast and efficient data reconstruction in erasure-coded data centers. In: SIGCOMM, ChicagoGoogle Scholar
  16. Schroeder B, Gibson G (2007) Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you? In: USENIX conference on file and storage technologies. San JoseGoogle Scholar
  17. Schroeder B, Lagisetty R, Merchant A (2016) Flash reliability in production the expected and the unexpected. In: USENIX conference on file and storage technologies, Santa ClaraGoogle Scholar
  18. Taylor C (2014) SSD vs. HDD: performance and reliability. [cited 2017]; Available from http://www.enterprisestorageforum.com/storage-hardware/ssd-vs.-hdd-performance-and-reliability-1.html
  19. Vishwanath KV, Nagappan N (2010) Characterizing cloud computing hardware reliability. In: ACM symposium on cloud computing, IndianapolisGoogle Scholar
  20. Wood T et al (2010) Disaster recovery as a cloud service: economic benefits & deployment challenges. In: HotCloud, BostonGoogle Scholar
  21. Xin Q, Schwarz TJE, Miller EL (2005) Disk infant mortality in large storage systems. In: IEEE international symposium on modeling, analysis, and simulation of computer and telecommunication systems, AtlantaGoogle Scholar
  22. Zaman R (2016) SSD vs HDD: which one is more reliable? [cited 2017]; Available from https://therevisionist.org/reviews/ssd-vs-hdd-one-reliable/

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Advanced Computing Research CentreData to Decisions Cooperative Research Centre, University of South AustraliaAdelaideAustralia

Section editors and affiliations

  • Bingsheng He
  • Behrooz Parhami
    • 1
  1. 1.Dept. of Electrical and Computer EngineeringUniversity of California, Santa BarbaraSanta BarbaraUSA