Tools for the Analysis of the Accuracy of Software Reliability Predictions

  • B. Littlewood
  • A. A. Abdel Ghaly
  • P. Y. Chan
Part of the NATO ASI Series book series (volume 22)


Different software reliability models can produce very different answers when called upon to predict future reliability in a reliability growth context. Users need to know which, if any, of the competing predictions are trustworthy. Some techniques are presented which form the basis of a partial solution to this problem. In addition, it is shown that this approach can point the way towards more accurate prediction via models which learn from past behaviour.


Prediction System Software Reliability True Distribution Predictive Quality Reliability Growth 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Z. Jelinski and P. B. Moranda, ‘Software reliability research’, in Statistical Computer Performance Evaluation ( W. Freiberger, ed.). New York: Academic Press, 1972, pp. 465–484.Google Scholar
  2. [2]
    M. Shooman, ‘Operational testing and software reliability during program development’, in Record. 1973 IEEE Symp. Computer Software Reliability (New York, NY, 1973, April 30–May 2 ) pp. 51–57.Google Scholar
  3. [3]
    B. Littlewood and J. L. Verrall, ‘A Bayesian reliability growth model for computer software’, J. Royal Statist. Soc., C (Applied Statistics), 22, 1973, pp. 332–346.MathSciNetGoogle Scholar
  4. [4]
    J. D. Musa, ‘A theory of software reliability and its application’, IEEE Trans. Software Engineering, Vol SE-1, 1975 Sept, pp. 312–327.Google Scholar
  5. [5]
    C. J. Dale, ‘Software Reliability Evaluation Methods’, British Aerospace Dynamics Group, ST-26750, 1982.Google Scholar
  6. [6]
    P. A. Keiller, B. Littlewood, D. R. Miller and A. Sofer, ‘Comparison of software reliability predictions’, Digest FTCS 13 (13th International Symposium on Fault-Tolerant Computing) pp. 128–134, 1983.Google Scholar
  7. [7]
    B. Littlewood, ‘How to measure software reliability and how not to’, IEEE Trans. Reliability, Vol R-28, 1979, June, pp. 103–110.Google Scholar
  8. [8]
    J. C. Laprie, ‘Dependability evaluation of software systems in operation’ IEEE Trans. Software Engineering, 1984, December.Google Scholar
  9. [9]
    E. N. Adams, ‘Optimizing preventive service of software products’, IBM Journal of Research and Development, Vol 28, No. 1, 1984.Google Scholar
  10. [10]
    H. Ascher and H. Feingold, Repairable Systems Reliability, Lecture Notes in Statistics, No. 7, Marcel Dekker, New York, 1984.MATHGoogle Scholar
  11. [11]
    J. Aitchison and I. R. Dunsmore, Statistical Prediction Analysis, Cambridge University Press, Cambridge, 1975.CrossRefMATHGoogle Scholar
  12. [12]
    P. M. Nagel and J. A. Skrivan, ‘Software reliability: repetitive run experimentation and modelling’, BCS-40399, Boeing Computer Services Company, Seattle, Washington, 1981, December.Google Scholar
  13. [13]
    E. H. Forman and N. D. Singpurwalla, ‘An empirical stopping rule for debugging and testing computer software’, J. Amer. Statist. Assoc., Vol. 72, 1977, Dec., pp. 750–757.CrossRefGoogle Scholar
  14. [14]
    B. Littlewood and J. L. Verrall, ‘On the likelihood function of a debugging model for computer software reliability’, IEEE Trans. Reliability, Vol. R-30, 1981 June, pp. 145–148.Google Scholar
  15. [15]
    H. Joe and N. Reid, ‘Estimating the number of faults in a system’, J. Amer. Statist. Assoc., Vol. 80, 1985 March, pp. 222–226.CrossRefMathSciNetGoogle Scholar
  16. [16]
    B. Littlewood and A. Sofer, ‘A Bayesian modification to the Jelinski-Moranda software reliability growth model’, available from first author.Google Scholar
  17. [17]
    B. Littlewood, ‘Stochastic reliability growth: a model for fault- removal in computer programs and hardware designs’, IEEE Trans. Reliability, Vol. R-30, 4, Oct. 1981, pp. 313–320.CrossRefMathSciNetGoogle Scholar
  18. [18]
    A. A. Abdel Ghaly, Ph.D. Thesis, City University, London, in preparation.Google Scholar
  19. [19]
    P. A. Keiller, B. Littlewood, D. R. Miller and A. Sofer, ‘On the quality of software reliability predictions’, Proc. NATO AST on Electronic Systems Effectiveness and Life Cycle Costing (Norwich, UK, 1982), Springer, 1983, pp. 441–460.Google Scholar
  20. [20]
    D. R. Miller, ‘Exponential order statistic models of software reliability growth’, Tech. Report, T-496/84, George Washington University, Washington DC, 1984.Google Scholar
  21. [21]
    J. T. Duane, ‘Learning curve approach to reliability monitoring’, IEEE Trans. Aerospace, 2, 1964, pp. 563–566.CrossRefGoogle Scholar
  22. [22]
    L. H. Crow, ‘Confidence interval procedures for reliability growth analysis’, Tech. Report No. 197, US Army Material Systems Analysis Activity, Aberdeen, Md., 1977.Google Scholar
  23. [23]
    H. Akaike, ‘Prediction and Entropy’, MRC Technical Summary Report, Mathematics Research Center, University of Wisconsin-Madison, June, 1982.Google Scholar
  24. [24]
    M. G. Kendall and A. Stuart, The Advanced Theory of Statistics, Griffin, London, 1961.Google Scholar
  25. [25]
    M. Rosenblatt, ‘Remarks on a multivariate transformation’, Ann. Math. Statist., 23, pp. 470–472, 1952.CrossRefMATHMathSciNetGoogle Scholar
  26. [26]
    A. P. Dawid, ‘Statistical theory: the prequential approach’, J. Royal Statist. Soc., A (1984), 147, pp. 278–292.CrossRefMATHMathSciNetGoogle Scholar
  27. [27]
    A. P. Dawid, ‘Calibration-based empirical probability’, Res. Report 36, Department of Statistical Science, University College, London, 1984.Google Scholar
  28. [28]
    D. R. Cox and P. A. W. Lewis, Statistical Analysis of Series of Events, Methuen, London, 1966.MATHGoogle Scholar
  29. [29]
    A. P. Dawid, ‘The well-calibrated Bayesian’, (with discussion) J. Amer. Statist. Assoc., 77, pp. 605–613, 1982.CrossRefMATHMathSciNetGoogle Scholar
  30. [30]
    A. P. Dawid, ‘Probability Forecasting’, Encyclopedia of Statistical Sciences, Vol. 6 (S. Kotz, N. L. Johnson and C. B. Read, eds.), Wiley-Interscience (to appear).Google Scholar
  31. [31]
    D. R. Miller, private communication, 1983.Google Scholar
  32. [32]
    H. Braun and J. M. Paine, ‘A comparative study of models for reliability growth’, Tech. Report, No. 126, Series 2, Department of Statistics, Princeton University, 1977.Google Scholar
  33. [33]
    B. Littlewood and P. A. Keiller, ‘Adaptive software reliability modelling’, Digest FTCS-14 (14th International Conference on Fault-Tolerant Computing), pp. 108–113, 1984.Google Scholar
  34. [34]
    I. Goudie, private communication, 1984.Google Scholar
  35. [35]
    J. D. Musa, ‘Software reliability data’, report available from Data and Analysis Center for Software, Rome Air Development Center, Rome, NY.Google Scholar
  36. [36]
    N. Langberg and N. D. Singpurwalla, ‘A unification of some software reliability models via the Bayesian approach’, Tech. Report, TM-66571, The George Washington University, Washington DC, 1981.Google Scholar
  37. [37]
    A. L. Goel and K. Okumoto, ‘Time-dependent error-detection rate model for software reliability and other performance measures’, IEEE Trans. Reliability, Vol. R-28, pp. 206–211, 1979.CrossRefMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1986

Authors and Affiliations

  • B. Littlewood
    • 1
  • A. A. Abdel Ghaly
    • 1
  • P. Y. Chan
    • 1
  1. 1.Centre for Software ReliabilityThe City UniversityLondonEngland

Personalised recommendations