Study on upper limit of sample size for a two-level test in NIST SP800-22


NIST SP800-22 is one of the most widely used statistical testing tools for pseudorandom number generators (PRNGs). This tool consists of 15 tests (one-level tests) and two additional tests (two-level tests). Each one-level test provides one or more p-values. The two-level tests measure the uniformity of the obtained p-values for a fixed one-level test. One of the two-level tests categorizes the p-values into ten intervals of equal length, and apply a chi-squared goodness-of-fit test. This two-level test is often more powerful than one-level tests, but sometimes it rejects even good PRNGs when the sample size at the second level is too large, since it detects approximation errors in the computation of p-values. In this paper, we propose a practical upper limit of the sample size in this two-level test, for each of six tests appeared in SP800-22. These upper limits are derived by the chi-squared discrepancy between the distribution of the approximated p-values and the uniform distribution U(0, 1). We also computed a “risky” sample size at the second level for each one-level test. Our experiments show that the two-level test with the proposed upper limit gives appropriate results, while using the risky size often rejects even good PRNGs. We also propose another improvement: to use the exact probability for the ten categories in the computation of goodness-of-fit at the two-level test. This allows us to increase the sample size at the second level, and would make the test more sensitive than the NIST’s recommending usage.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6


  1. 1.

    Bassham III, L.E., et al.: SP 800-22 Rev. 1a. A statistical test suite for random and pseudorandom number generators for cryptographic applications. Technical report, National Institute of Standards & Technology, Gaithersburg, MD, United States. (2010)

  2. 2.

    Doty-Humphrey, C.: Practically random: specific tests in PractRand.

  3. 3.

    Galassi, M., et al.: GNU scientific library reference manual. (2018)

  4. 4.

    Haramoto, H.: Automation of statistical tests on randomness to obtain clearer conclusion. In: Ecuyer, P.L., Owen, A.B. (eds.) Monte Carlo and Quasi-Monte Carlo Methods 2008, pp. 411–421. Springer, Berlin (2009)

    Google Scholar 

  5. 5.

    Haramoto, H., Matsumoto, M.: A method to compute an appropriate sample size of a two-level test for the NIST test suite. In: Owen, A.B., Glynn, P.W. (eds.) Monte Carlo and Quasi-Monte Carlo Methods, pp. 283–294. Springer International Publishing, Cham (2018)

    Google Scholar 

  6. 6.

    Haramoto, H., Matsumoto, M.: Checking the quality of approximation of p-values in statistical tests for random number generators by using a three-level test. Mathematics and computers in simulation 161, 66–75 (2019). issue on the Eleventh International Conference on Monte Carlo Methods and Applications (MCM 2017), held in Montreal, Canada, July 03–07, 2017)

  7. 7.

    Kim, S.J., Umeno, K., Hasegawa, A.: Corrections of the NIST statistical test suite for randomness. Cryptology ePrint Archive, Report 2004/018. (2004)

  8. 8.

    Knuth, D.E.: The Art of Computer Programming, Volume 2 (3rd ed.): Seminumerical Algorithms. Addison-Wesley, Boston (1997)

    Google Scholar 

  9. 9.

    L’Ecuyer, P.: Testing random number generators. In: Proceedings of the 1992 Winter Simulation Conference, pp. 305–313. IEEE Press (1992)

  10. 10.

    L’Ecuyer, P.: Uniform random number generation. Ann. Oper. Res. 53, 77–120 (1994)

    MathSciNet  Article  Google Scholar 

  11. 11.

    L’Ecuyer, P.: Tests based on sum-functions of spacings for uniform random numbers. J. Stat. Comput. Simul. 59(3), 251–269 (1997).

    Article  MATH  Google Scholar 

  12. 12.

    L’Ecuyer, P., Simard, R.: TestU01: a C library for empirical testing of random number generators. ACM Trans. Math. Softw. 33(4), 22:1–22:40 (2007).

  13. 13.

    Matsumoto, M., Nishimura, T.: Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans. Model. Comput. Simul. 8(1), 3–30 (1998)

    Article  Google Scholar 

  14. 14.

    Matsumoto, M., Nishimura, T.: A Nonempirical test on the weight of pseudorandom number generators. Monte Carlo and quasi-Monte Carlo methods. 2000 (Hong Kong), pp. 381–395. Springer, Berlin (2002)

  15. 15.

    Panneton, F., L’Ecuyer, P., Matsumoto, M.: Improved long-period generators based on linear recurrences modulo 2. ACM Trans. Math. Softw. 32(1), 1–16 (2006).

    MathSciNet  Article  MATH  Google Scholar 

  16. 16.

    Pareschi, F., Rovatti, R., Setti, G.: Second-level NIST randomness tests for improving test reliability. In: 2007 IEEE International Symposium on Circuits and Systems, pp. 1437–1440 (2007).

  17. 17.

    Pareschi, F., Rovatti, R., Setti, G.: On statistical tests for randomness included in the NIST SP800-22 test suite and based on the binomial distribution. IEEE Trans. Inf. Forensics Secur. 7(2), 491–505 (2012).

    Article  Google Scholar 

  18. 18.

    Shiryaev, A.N.: Probability, 2nd edn. Springer, Berlin (1995)

    Google Scholar 

  19. 19.

    Tiku, M.: Noncentral chi-square distribution. In: Kotz, S., Johnson, N. (eds.) Encyclopedia of Statistical science, vol. 6, pp. 276–280. Wiley, New York (1981)

    Google Scholar 

  20. 20.

    Yamaguchi, A., Saito, A.: On the statistical test of randomness based on the uniformity of \(p\)-values used in NIST statistical test suite. In: Proceedings of the 2015 JSIAM Annual Meeting, Applied Chaos, pp. 1–2 (2015)

Download references


The author is indebted to Professor Makoto Matsumoto for constant help and encouragements. The author is thankful to the anonymous referees for many valuable comments. This study was carried out under the ISM Cooperative Research Program (2018-ISMCRP-10 and 2019-ISMCRP-05).

Author information



Corresponding author

Correspondence to Hiroshi Haramoto.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The author is supported by Grant-in-aid for Science Research, nos. 16K13750, 17K14234, and 18K03213.

About this article

Verify currency and authenticity via CrossMark

Cite this article

Haramoto, H. Study on upper limit of sample size for a two-level test in NIST SP800-22. Japan J. Indust. Appl. Math. 38, 193–209 (2021).

Download citation


  • Statistical testing
  • Pseudorandom number generators
  • NIST SP800-22
  • Two-level test

Mathematics Subject Classification

  • 65C10
  • 65C60
  • 68N30