Skip to main content
SpringerLink
Log in

Origins of Stochastic Computing

  • Chapter
  • First Online:
Stochastic Computing: Techniques and Applications

Abstract

In the early 1960s research groups at the University of Illinois, USA, and Standard Telecommunication Laboratories (STL), UK, each independently conceived of a constructive use of random noise to implement analog computers in which the probability of a pulse in a digital pulse stream represented a continuous variable. The USA group initially termed this a noise computer but shortly adopted the UK terminology of stochastic computer. The target application of the USA group was visual pattern recognition, and that of the UK group was learning machines, and both developed trial hardware implementations. However, as they investigated applications they both came to recognize that the technology of their era did not support stochastic computing systems that could compete with available computational technologies, and they moved on to develop other computing architectures, some of which derived from the stochastic computing concepts. Both groups published expositions of stochastic computing which provided a comprehensive account of the technology, the architecture of its functional modules, its potential applications and its then current limitations. These have become highly cited in recent years as new technologies and issues have made stochastic computing a competitive technology for a number of significant applications. This chapter provides a historical a historical analysis of the motivations of the pioneers and how they arrived at the notion of stochastic computing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 99.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Standard Telecommunications Company (STC), Footscray, Kent.

  2. 2.

    Standard Telecommunication Laboratories (STL), Harlow, Essex.

  3. 3.

    Standard Telecommunication Laboratories Learning Automaton (STeLLA).

  4. 4.

    I became interested in Pontryagin’s work because one of my experiments in Richard’s laboratory was to replicate the results in a memorandum by Bartlett where he had investigated reaction times in a tapping task with variations in target difference. His results were consistent with the hypothesis that people made Pontryagin-type bang-bang movements using maximum acceleration following by maximum deceleration, and I was later able to demonstrate this in my control task[20].

  5. 5.

    Adaptive Digital Data Integrating Element (ADDIE).

  6. 6.

    Earl Hunt was one of the first to cite this chapter (in the context of von Neumann’s book [81]) in his 1971 paper on “what kind of computer is man?” and comes to the conclusion that man is a stochastic computer. Earl unfortunately died in 2016 just before the advent of stochastic deep learning neural networks [45] and the assessment of how the behaviour of deep networks emulated human visual perception [54] that begins to validate his conjecture.

References

  1. Chusin Afuso. “Analog computation with random pulse sequences”. PhD thesis. University of Illinois, 1968.

    Google Scholar 

  2. A. Alaghi and J. P. Hayes. “Exploiting correlation in stochastic circuit design”. IEEE 31st International Conference on Computer Design (ICCD). 2013, pp. 39–46.

    Google Scholar 

  3. A. Alaghi, W. Qian, and J. P. Hayes. “The promise and challenge of stochastic computing”. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 37.8 (2018), pp. 1515–1531.

    Article  Google Scholar 

  4. Armin Alaghi, Cheng Li, and John P. Hayes. “Stochastic circuits for real-time image-processing applications”. Proceedings of the 50th Annual Design Automation Conference. Austin, Texas: ACM, 2013, pp. 1–6.

    Google Scholar 

  5. John H. Andreae. “STeLLA: A scheme for a learning machine”. Proceedings 2nd IFAC Congress: Automation & Remote Control. Ed. by V. Broida. London: Butterworths, 1963, pp. 497–502.

    Google Scholar 

  6. A. Basu et al. “Low-power, adaptive neuromorphic systems: Recent progress and future directions”. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 8.1 (2018), pp. 6–27.

    Article  Google Scholar 

  7. T. E. Bray. “An optoelectronic-magnetic neuron component”. Proceedings National Electronics Conference. 1961, pp. 302–310.

    Google Scholar 

  8. Peter S. Burge et al. “Stochastic bit-stream neural networks”. Pulsed Neural Networks. Ed. by Wolfgang Maass and Christopher M. Bishop. Cambridge, MA: MIT Press, 1999, pp. 337–352.

    Google Scholar 

  9. Gene L. Clapper. “Machine looks, listens, learns”. Electronics (Oct. 30, 1967).

    Google Scholar 

  10. Electronics. “Dropping the guard”. Electronics (Dec. 12, 1966), p. 48.

    Google Scholar 

  11. John W. Esch. “RASCEL - A Programmable Analog Computer Based on a Regular Array of Stochastic Computing Element Logic”. PhD thesis. University of Illinois, 1969.

    Google Scholar 

  12. John W. Esch. “System and method for frame and unit-like symbolic access to knowledge represented by conceptual structures”. Pat. 4964063. Unisys Corporation. Sept. 15, 1988.

    Google Scholar 

  13. John W. Esch and Robert Levinson. “An implementation model for contexts and negation in conceptual graphs”. Conceptual Structures: Applications, Implementation and Theory. Springer Berlin Heidelberg, 1995.

    Google Scholar 

  14. Peter V. Facey and Brian R. Gaines. “Real-time system design under an emulator embedded in a high-level language”. Proceedings DATAFAIR 73. London: British Computer Society, 1973, pp. 285–291.

    Google Scholar 

  15. Brian R. Gaines. “A mixed-code approach to commercial microcomputer applications”. Conference on Microprocessors in Automation and Communications. London: IERE, 1978, pp. 291–301.

    Google Scholar 

  16. Brian R. Gaines. A Stochastic Computer: Some Notes on the Application of Digital Circuits to the Operations of Arithmetic and Differential Calculus by Means of a Probabilistic Representation of Quantities. Tech. rep. Standard Telecommunication Laboratories, Dec. 9, 1965.

    Google Scholar 

  17. Brian R. Gaines. “Adaptive control theory: the structural and behavioural properties of adaptive controllers”. Encyclopaedia of Linguistics, Information & Control. Ed. by A.R. Meetham and R.A. Hudson. London: Pergamon Press, 1969, pp. 1–9.

    Google Scholar 

  18. Brian R. Gaines. “Foundations of stochastic computing systems”. Digest of IEEE International Convention. New York: IEEE, 1968, p. 33.

    Google Scholar 

  19. Brian R. Gaines. “Interpretive kernels for microcomputer software”. Proceedings Symposium Microprocessors at Work. London: Society of Electronic & Radio Technicians, 1976, 56–69.

    Google Scholar 

  20. Brian R. Gaines. “Linear and nonlinear models of the human controller”. International Journal of Man-Machine Studies 1.4 (1969), pp. 333–360.

    Article  Google Scholar 

  21. Brian R. Gaines. MINIC I Manual. Colchester, UK: Department of Electrical Engineering Science, 1969.

    Google Scholar 

  22. Brian R. Gaines. MINSYS Manual. Colchester, Essex: Department Electrical Engineering Science, 1974.

    Google Scholar 

  23. Brian R. Gaines. “R68-18 Random Pulse Machines”. IEEE Transactions on Computers C-17.4 (1968), pp. 410–410.

    Article  Google Scholar 

  24. Brian R. Gaines. “Stochastic computer thrives on noise”. Electronics (July 10, 1967), pp. 72–79.

    Google Scholar 

  25. Brian R. Gaines. “Stochastic computers”. Encyclopaedia of Linguistics, Information & Control. Ed. by A.R. Meetham and R.A. Hudson. London: Pergamon Press, 1969, pp. 66–76.

    Google Scholar 

  26. Brian R. Gaines. “Stochastic computing”. Spring Joint Computer Conference. Vol. 30. Atlantic City: AFIPS, 1967, pp. 149–156.

    Google Scholar 

  27. Brian R. Gaines. “Stochastic Computing Arrangement”. British pat. 184652. Standard Telephones & Cables Ltd. Mar. 7, 1966.

    Google Scholar 

  28. Brian R. Gaines. “Stochastic computing systems”. Advances in Information Systems Science, 2. Ed. by J. Tou. New York: Plenum Press, 1969, pp. 37–172.

    Google Scholar 

  29. Brian R. Gaines. “Teaching machines for perceptual-motor skills”. Aspects of Educational Technology. Ed. by D. Unwin and J. Leedham. London: Methuen, 1967, pp. 337–358.

    Google Scholar 

  30. Brian R. Gaines. “Techniques of identification with the stochastic computer”. Proceedings IFAC Symposium on The Problems of Identification in Automatic Control Systems. 1967, pp. 1–10.

    Google Scholar 

  31. Brian R. Gaines. “Training the human adaptive controller”. Proceedings Institution Electrical Engineers 115.8 (1968), pp. 1183–1189.

    Article  Google Scholar 

  32. Brian R. Gaines. “Trends in stochastic computing”. Colloquium on Parallel Digital Computing Methods—DDA’s and Stochastic Computing. Vol. 30. London: IEE, 1976, pp. 1–2.

    Google Scholar 

  33. Brian R. Gaines. “Uncertainty as a foundation of computational power in neural networks”. Proceedings of IEEE First International Conference on Neural Networks. Vol.3. Ed. by M. Caudhill and C. Butler. 1987, pp. 51–57.

    Google Scholar 

  34. Brian R. Gaines. “Varieties of computer —– their applications and inter-relationships”. Proceedings of IFAC Symposium on Pulse Rate and Pulse Number Signals in Automatic Control, Budapest: IFAC, 1968, pp. 1–16.

    Google Scholar 

  35. Brian R. Gaines and John H. Andreae. “A learning machine in the context of the general control problem”. Proceedings of the 3rd Congress of the International Federation for Automatic Control. London: Butterworths, 1966, 342–348 (discussion, session 14, p.93).

    Google Scholar 

  36. Brian R. Gaines, M. Haynes, and D. Hill. “Integration of protection and procedures in a high-level minicomputer”. Proceedings IEE 1974 Computer Systems and Technology Conference. London: IEE, 1974.

    Google Scholar 

  37. Brian R. Gaines and Peter L. Joyce. “Phase computers”. Proceedings of 5th Congress of International Association for Analog Computation. 1967, pp. 48–57.

    Google Scholar 

  38. Brian R. Gaines and Ray A. Shemer. “Fitting Control Mathematics to Control Hardware: An Aspect of the 1968 IFAC Pulse-Symposium”. Automatica 5 (1969), pp. 37–40.

    Article  Google Scholar 

  39. Brian R. Gaines et al. “Design objectives for a descriptor-organized minicomputer”. European Computing Congress Conference Proceedings, EUROCOMP 74. London: Online, 1974, pp. 29–45.

    Google Scholar 

  40. F. Jenik. “Electronic neuron models as an aid to neurophysiological research”. Ergebnisse der Biologie 25 (1962), pp. 206–245.

    Article  Google Scholar 

  41. F. Jenik. “Pulse processing by neuron models”. Neural Theory and Modeling: Proceedings of the 1962 Ojai Symposium. Ed. by Richard F. Reiss. Stanford, CA: Stanford University Press, 1964, pp. 190–212.

    Google Scholar 

  42. K. S. Lashley. “The problem of cerebral organization in vision”. Visual Mechanisms. Ed. by H. Kluüver. Oxford: Cattell, 1942.

    Google Scholar 

  43. Robert Legenstein and Wolfgang Maass. “Ensembles of spiking neurons with noise support optimal probabilistic inference in a dynamically changing environment”. PLoS Computational Biology 10.10 (2014), e1003859 1–27.

    Google Scholar 

  44. E. R. Lewis. “The locus concept and its application to neural analogs”. IEEE Transactions on Bio-medical Electronics 10.4 (1963), pp. 130–137.

    Article  Google Scholar 

  45. Y. Liu et al. “An energy-efficient online-learning stochastic computational deep belief network”. IEEE Journal on Emerging and Selected Topics in Circuits and Systems (2018), pp. 1–1.

    Google Scholar 

  46. A.R. Luria. The Role of Speech in the Regulation of Normal and Abnormal Behavior. Oxford: Pergamon Press, 1961.

    Google Scholar 

  47. P. Mars and W. J. Poppelbaum. Stochastic and Deterministic Averaging Processors. Stevenage: IEE/Peregrinus, 1981.

    MATH  Google Scholar 

  48. T. B. Martin. “Analog signed processing by neural networks”. Proceedings National Electronics Conference. 1961, pp. 317–321.

    Google Scholar 

  49. Robert Massen. Stochastische Rechentechnik: Eine Einfuührung in die Informationsverarbeitung mit zurfaälligen Pulsfolgen. Munich: Hanser, 1977.

    MATH  Google Scholar 

  50. E. P. McGrogan. “Improved transistor neural models”. Proceedings National Electronics Conference. 1961, pp. 302–310.

    Google Scholar 

  51. Robert King Merton. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press, 1973.

    Google Scholar 

  52. M. Hassan Najafi, David J. Lilja, and Marc Riedel. “Deterministic methods for stochastic computing using low-discrepancy sequences”. IEEE/ACM International Conference On Computer-Aided Design (ICCAD ’18). New York: ACM, 2018.

    Google Scholar 

  53. Albert B. Novikoff. On Convergence Proofs for Perceptrons. Tech. rep. SRI Project 3605. Menlo Park, CA: Stanford Research Institute, 1963.

    Google Scholar 

  54. Joshua C. Peterson, Joshua T. Abbott, and Thomas L. Griffiths. “Evaluating (and improving) the correspondence between deep neural networks and human representations”. Cognitive Science (2018), 42.8, 2648–2699.

    Article  Google Scholar 

  55. L. S. Pontryagin et al. The Mathematical Theory of Optimal Processes. Oxford, England: Pergamon Press, 1962.

    Google Scholar 

  56. W. J. Poppelbaum. Computer Hardware Theory. New York, Macmillan, 1972.

    Google Scholar 

  57. W. J. Poppelbaum. Record of Achievements and Plans of the Information Engineering Laboratory. Champaign, Urbana, IL: Department of Computer Science, University of Illinois, 1973.

    Book  Google Scholar 

  58. W. J. Poppelbaum. “Statistical processors”. Advances in Computers. Ed. by Rubinoff Morris and C. Yovits Marshall. Vol. Volume 14. Elsevier, 1976, pp. 187–230.

    Google Scholar 

  59. W. J. Poppelbaum. “Transistor Flip-Flop Circuit”. Pat. 2933621. University of Illinois Foundation. Aug. 2, 1956.

    Google Scholar 

  60. W. J. Poppelbaum. “What next in computer technology?” Advances in Computers. Ed. by L. Alt Franz and Rubinoff Morris. Vol. Volume 9. Elsevier, 1969, pp. 1–21.

    Google Scholar 

  61. W. J. Poppelbaum, C. Afuso, and J.W. Esch. “Stochastic computing elements and systems”. Fall Joint Computer Conference. Vol. 31. New York: Books, Inc, 1967, pp. 635–644.

    Google Scholar 

  62. W. J. Poppelbaum et al. “Unary Processing”. Advances in Computers. Ed. by C. Yovits Marshall. Vol. Volume 26. Elsevier, 1987, pp. 47–92.

    Google Scholar 

  63. W. Qian et al. “An architecture for fault-tolerant computation with stochastic logic”. IEEE Transactions on Computers 60.1 (2011), pp. 93–105.

    Article  MathSciNet  Google Scholar 

  64. Werner Reichardt. “Evaluation of optical motion information by movement detectors”. Journal of Comparative Physiology A 161 (1987), pp. 533–547.

    Article  Google Scholar 

  65. S. T. Ribeiro and G. K. Ujhelyi. “Electro-Optical Modulation of Radiation Pattern Using Curved Electrodes”. U.S. pat. 3433554. Secretary of the Navy. May 1, 1964.

    Google Scholar 

  66. Sergio Telles Ribeiro. “Comments on Pulsed-Data Hybrid Computers”. IEEE Transactions on Electronic Computers EC-13.5 (1964), pp. 640–642.

    Article  Google Scholar 

  67. Sergio Telles Ribeiro. “Phase Plane Theory of Transistor Bistable Circuits”. PhD thesis. University of Illinois, 1963.

    Google Scholar 

  68. Sergio Telles Ribeiro. “Random pulse machines”. IEEE Trans. Electronic Computers EC-16.6 (1967), pp. 261–276.

    Article  Google Scholar 

  69. Frank Rosenblatt. “The perceptron: A probabilistic model for information storage and organization in the brain”. Psychological Review 65.6 (1958), pp. 386–408.

    Article  Google Scholar 

  70. Sayed Ahmad Salehi et al. “Computing mathematical functions using DNA via fractional coding”. Scientific Reports 8.1 (2018), p. 8312.

    Google Scholar 

  71. Hermann Schmid. “An operational hybrid computing system provides analog-type computation with digital elements”. IEEE Transactions on Electronic Computers EC-12.6 (1963), pp. 715–732.

    Article  Google Scholar 

  72. Ray A. Shemer. “A hybrid-mode modular computing system”. Proceedings of IFAC Symposium on Pulse Rate and Pulse Number Signals in Automatic Control. Budapest, 1968.

    Google Scholar 

  73. Ray A. Shemer. “A Hybrid-Mode Modular Computing System”. PhD thesis. 1970.

    Google Scholar 

  74. I. W. Smith, D. A. Hearn, and P. Williamson. “Software development for Batchmatic computer numerical control system”. Proceedings of the Fourteenth International Machine Tool Design and Research Conference. Ed. by F. Koenigsberger and S. A. Tobias. London: Macmillan, 1974, pp. 381–389.

    Chapter  Google Scholar 

  75. Mandyam V. Srinivasan and Gary D. Bernard. “A proposed mechanism for multiplication of neural signals”. Biological Cybernetics 21.4 (1976), pp. 227–236.

    Article  Google Scholar 

  76. William G. Thistle. A Novel Special Purpose Computer. Tech. rep. CADRE Technical Note 1460. Valcartier, Queébec: Canadian Armament Research and Development Establishment, 1962.

    Google Scholar 

  77. William G. Thistle. “Integrating Apparatus”. Canadian pat. 721406. Ministry of National Defence. Nov. 30, 1962.

    Google Scholar 

  78. Financial Times “George Kent backs a micro-computer venture”. Financial Times (Feb. 5, 1970), p. 11.

    Google Scholar 

  79. G. K. Ujhelyi and S. T. Ribeiro. “An electro-optical light intensity modulator”. Proceedings of the IEEE 52.7 (1964), pp. 845–845.

    Article  Google Scholar 

  80. Gabor K. Ujhelyi, Sergio T. Ribeiro, and Andras M. Bardos. “Data Display Device”. U.S. pat. 3508821. Carson Laboratories. Aug. 11, 1966.

    Google Scholar 

  81. John Von Neumann. The Computer and the Brain. New Haven, Yale University Press, 1958.

    MATH  Google Scholar 

  82. Computer Weekly. “MINIC system is bought by George Kent”. Computer Weekly (Feb. 12, 1970).

    Google Scholar 

  83. Wikipedia. List of multiple discoveries. 2018. https://en.wikipedia.org/wiki/List_of_multiple_discoveries.

  84. F.K. Williamson et al. “A high-level minicomputer”. Information Processing 74. Amsterdam: Noth-Holland, 1974, pp. 44–48.

    Google Scholar 

  85. Yiu Kwan Wo. “APE machine: A novel stochastic computer based on a set of autonomous processing elements”. PhD thesis. 1973.

    Google Scholar 

  86. S. Wolff, J. Thomas, and T. Williams. “The polarity-coincidence correlator: A nonparametric detection device”. IRE Transactions on Information Theory 8.1 (1962), pp. 5–9.

    Article  Google Scholar 

Download references

Acknowledgements

I am grateful to John Esch for his help in verifying my commentary on the research at the University of Illinois and for providing the photograph of his RASCEL stochastic computer. I am grateful to David Hill for prompting my memory of certain dates and events and for providing the material by Thistle documenting his early stochastic computer. I would also like to thank the editors of this volume for providing the opportunity to contribute this account of the origins of stochastic computing knowing that there are very few of us still alive to do so. I hope it will be of interest to the stochastic computing research community of this era, and wish them well.

Author information

Authors and Affiliations

  1. University of Victoria, Victoria, BC, Canada

    Brian R. Gaines

  2. University of Calgary, Calgary, AB, Canada

    Brian R. Gaines

Authors
  1. Brian R. Gaines
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Brian R. Gaines .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering, McGill University, Montréal, QC, Canada

    Warren J. Gross

  2. ECE Department, University of Waterloo, Waterloo, ON, Canada

    Vincent C. Gaudet

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gaines, B.R. (2019). Origins of Stochastic Computing. In: Gross, W., Gaudet, V. (eds) Stochastic Computing: Techniques and Applications. Springer, Cham. https://doi.org/10.1007/978-3-030-03730-7_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-03730-7_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-03729-1

  • Online ISBN: 978-3-030-03730-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation