Great Expectations: A Critique of Current Approaches to Random Number Generation Testing & Certification

  • Darren Hurley-SmithEmail author
  • Julio Hernandez-Castro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11322)


Random number generators are a critical component of security systems. They also find use in a variety of other applications from lotteries to scientific simulations. Randomness tests, such as the NIST’s STS battery (documented in SP800-22), Marsaglia’s Diehard, and L’Ecuyer et al.’s TestU01 seek to find whether a generator exhibits any signs of non-random behaviour. However, many statistical test batteries are unable to reliably detect certain issues present in poor generators. Severe mistakes when determining whether a given generator passes the tests are common. Irregularities in sample size selection and a lack of granularity in test result interpretation contribute to this. This work provides evidence of these and other issues in several statistical test batteries. We identify problems with current practices and recommend improvements. The novel concept of suitable randomness is presented, precisely defining two bias bounds for a TRNG, instead of a simple binary pass/fail outcome. Randomness naivety is also introduced, outlining how binary pass/fail analysis cannot express the complexities of RNG output in a manner that is useful to determine whether a generator is suitable for a given range of applications.



This project has received funding from Innovate UK, under reference number 102050 (authenticatedSelf) and from the European Union’s Horizon 2020 research and innovation programme, under grant agreement No. 700326 (RAMSES project). This article is based upon work from COST Action IC1403 CRYPTACUS, supported by COST (European Cooperation in Science and Technology). We would like to thank NXP Semiconductors Ltd. for their timely and professional communication following the responsible disclosure of our findings.


  1. 1.
    Schindler, W., Killmann, W.: Evaluation criteria for true (physical) random number generators used in cryptographic applications. In: Kaliski, B.S., Koç, K., Paar, C. (eds.) CHES 2002. LNCS, vol. 2523, pp. 431–449. Springer, Heidelberg (2003). Scholar
  2. 2.
    NXP Semiconductors Ltd. MF1PLUSx0y1 Public Datasheet. NXP Semiconductors, 21 February 2011Google Scholar
  3. 3.
    NXP Semiconductors Ltd. MF3D(H)x2 MIFARE DESFire EV2 contactless multi-application IC, 2 edn. NXP Semiconductors Ltd., February 2016Google Scholar
  4. 4.
    Altus Metrum. ChaosKey True Random Number Generator, June 2008Google Scholar
  5. 5.
    Marsaglia, G., Tsang, W.W., et al.: Some difficult-to-pass tests of randomness. J. Stat. Softw. 7(3), 1–9 (2002)CrossRefGoogle Scholar
  6. 6.
    National Institute of Standards and Technology. NIST SP800-22 Revision 1a A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications. Accessed 21 May 2018
  7. 7.
    Walker, J.: Ent. A pseudo-random number sequence testing program. Accessed 07 Aug 2018
  8. 8.
    L’Ecuyer, P., Simard, R.: TestU01: a C library for empirical testing of random number generators. ACM Trans. Math. Softw. (TOMS) 33(4), 22 (2007)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Dodis, Y., Ong, S.J., Prabhakaran, M., Sahai, A.: On the (im)possibility of cryptography with imperfect randomness. In: Proceedings of 45th Annual IEEE Symposium on Foundations of Computer Science, pp. 196–205. IEEE (2004)Google Scholar
  10. 10.
    Marsaglia, G.: DIEHARD, a battery of tests for random number generators. CD-ROM, Department of Statistics and Supercomputer Computations Research Institute, Florida State University.
  11. 11.
    Hurley-Smith, D., Hernandez-Castro, J.: Bias in the mifare DESFire EV1 TRNG. In: Hancke, G.P., Markantonakis, K. (eds.) RFIDSec 2016. LNCS, vol. 10155, pp. 123–133. Springer, Cham (2017). Scholar
  12. 12.
    Hurley-Smith, D., Hernandez-Castro, J.: Certifiably biased: an in-depth analysis of a common criteria EAL4+ certified TRNG. IEEE Trans. Inf. Forensics Secur. 13(4), 1031–1041 (2018)CrossRefGoogle Scholar
  13. 13.
    Garcia, F.D., et al.: Dismantling MIFARE classic. In: Jajodia, S., Lopez, J. (eds.) ESORICS 2008. LNCS, vol. 5283, pp. 97–114. Springer, Heidelberg (2008). Scholar
  14. 14.
    Garcia, F.D., Van Rossum, P., Verdult, R., Schreur, R.W.: Wirelessly pickpocketing a Mifare Classic card. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 3–15. IEEE (2009)Google Scholar
  15. 15.
    Kasper, T., Silbermann, M., Paar, C.: All you can eat or breaking a real-world contactless payment system. In: Sion, R. (ed.) FC 2010. LNCS, vol. 6052, pp. 343–350. Springer, Heidelberg (2010). Scholar
  16. 16.
    Renesas AE45C1 and Smartcard Integrated Circuit. BSI-DSZ-CC-0212-2004 (2004)Google Scholar
  17. 17.
    Barker, E., Kelsey, J.: Recommendation for the entropy sources used for random bit generation (DRAFT) NIST SP800-90B (2012)Google Scholar
  18. 18.
    Rukhin, A., Soto, J., Nechvatal, J., Smid, M., Barker, E.: A statistical test suite for random and pseudorandom number generators for cryptographic applications. Technical report, DTIC Document (2001)Google Scholar
  19. 19.
    Rukhin, A., Soto, J., Nechvatal, J.: A statistical test suite for random and pseudorandom number generators for cryptographic applications. NIST DTIC Document. NIST SP800-22 (2010)Google Scholar
  20. 20.
    Mellado, D., Fernández-Medina, E., Piattini, M.: A common criteria based security requirements engineering process for the development of secure information systems. Comput. Stan. Interfaces 29(2), 244–253 (2007)CrossRefGoogle Scholar
  21. 21.
    Killmann, W., Schindler, W.: AIS 31: functionality classes and evaluation methodology for true (physical) random number generators, version 3.1. Bundesamt fur Sicherheit in der Informationstechnik (BSI), Bonn (2001)Google Scholar
  22. 22.
    Brown, R.G., Eddelbuettel, D., Bauer, D.: Dieharder: a random number test suite. Open Source software library, under development (2013)Google Scholar
  23. 23.
    Marton, K., Suciu, A.: On the interpretation of results from the NIST statistical test suite. Sci. Technol. 18(1), 18–32 (2015)Google Scholar
  24. 24.
    Hernandez-Castro, J., Barrero, D.F.: Evolutionary generation and degeneration of randomness to assess the indepedence of the Ent test battery. In: 2017 IEEE Congress on Evolutionary Computation (CEC), pp. 1420–1427. IEEE (2017)Google Scholar
  25. 25.
    Soto, J.: Statistical testing of random number generators. In: Proceedings of the 22nd National Information Systems Security Conference, vol. 10, p. 12. NIST, Gaithersburg (1999)Google Scholar
  26. 26.
    Turan, M.S., DoĞanaksoy, A., Boztaş, S.: On independence and sensitivity of statistical randomness tests. In: Golomb, S.W., Parker, M.G., Pott, A., Winterhof, A. (eds.) SETA 2008. LNCS, vol. 5203, pp. 18–29. Springer, Heidelberg (2008). Scholar
  27. 27.
    Georgescu, C., Simion, E., Nita, A.-P., Toma, A.: A view on NIST randomness tests (in)dependence. In: 2017 9th International Conference on Electronics, Computers and Artificial Intelligence (ECAI), pp. 1–4. IEEE (2017)Google Scholar
  28. 28.
    Zhu, S., Ma, Y., Lin, J., Zhuang, J., Jing, J.: More powerful and reliable second-level statistical randomness tests for NIST SP 800-22. In: Cheon, J.H., Takagi, T. (eds.) ASIACRYPT 2016. LNCS, vol. 10031, pp. 307–329. Springer, Heidelberg (2016). Scholar
  29. 29.
    Fan, L., Chen, H., Gao, S.: A general method to evaluate the correlation of randomness tests. In: Kim, Y., Lee, H., Perrig, A. (eds.) WISA 2013. LNCS, vol. 8267, pp. 52–62. Springer, Cham (2014). Scholar
  30. 30.
    Hurley-Smith, D., Hernandez-Castro, J.: Quam Bene Non Quantum: Bias in a Family of Quantum Random Number Generators. Cryptology ePrint Archive, Report 2017/842 (2017).
  31. 31.
    National Institute of Standards and Technology. NIST SP800-90B Reccommendation for the Entropy Sources used for Random Bit Generation. Accessed 21 May 2018
  32. 32.
    Verbauwhede, I., Maes, R.: Physically unclonable functions: manufacturing variability as an unclonable device identifier. In: Proceedings of the 21st edition of the Great Lakes Symposium on VLSI, pp. 455–460. ACM (2011)Google Scholar
  33. 33.
    Altus Metrum. Accessed 11 Sept 2018
  34. 34.
    Langheinrich, M., Marti, R.: Practical minimalist cryptography for RFID privacy. IEEE Syst. J. 1(2), 115–128 (2007)CrossRefGoogle Scholar
  35. 35.
    Burr, W.E.: Selecting the advanced encryption standard. IEEE Secur. Priv. 99(2), 43–52 (2003)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of KentCanterburyUK

Personalised recommendations