Abstract
Recent research in Artificial Neural Networks (ANN’s) has shown that ANN’s will play an important role in solving many signal processing problems. To fully capture the potential that this new computational paradigm possesses, ANN models will have to be implemented in hardware. Initially, attempts were made to simulate ANN’s on serial computers. These software simulations were too slow to be of any practical significance and it was realized that ANN’s will have to be implemented on parallel machines that can exploit the parallelism inherent in ANN’s. In this chapter, we investigate how sparse Neural Networks can be implemented on a fixed size mesh of processors. A number of currently available machines make available a computing environment based on this architecture and this architecture is also amenable to VLSI implementation. We show how one iteration of activation value updates for a sparse neural network with n neurons and e non-zero connections is simulated on a p × p array of processors in O((n + e)/p) time. The efficiency of the algorithm is partly due to the fact that preprocessing is done on the connection matrix. This makes the algorithm efficient carrying out many iterations of the search phase computation with the same connection structure. Although not described here, learning algorithms like the Delta rule, which are based on the computation of a weighted sum, can also be run using a modified version of the algorithm.
This research was supported in part by the National Science Foundation under grant IRI-8905929.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
R. Altes. Unconstrained minimum mean-square error parameter estimation with Hopfield networks. In International Conference on Neural Networks, volume II, pages 541–548, 1988.
V. E. Benes. On Rearrangeable Three Stage Connecting Networks. B.S.T.J., 41:117–125, Sept. 1962.
C. Clos. A Study of Non-Blocking Switching Networks. B.S.T.J., 32:406–424, 1953.
K. I. Diamantara, D. L. Heine, and I. D. Scherson. Implementation of neural network algorithms on the P 3 parallel associative processor. In International Conference on Parallel Processing, volume I, pages 247–250, 1990.
L. Fu. Adaptive signal detection in noisy environments. The Journal of Neural Network Computing, Spring Issue, pages 42–50, 1990.
R. Gorman and T. Sejnowski. Analysis of hidden units in a layered network trained to classify sonar targets. Neural Networks, 1:75–89, 1988.
R. Gorman and T. Sejnowski. Learned classification of sonar targets using a massively parallel network. In IEEE Transactions on Acoustics, Speech, and Signal Processing, volume ASSP-36, pages 1135–1140, 1988.
Dan Hammerstrom. A VLSI architecture for high-performance, low-cost, on-chip learning. In International Joint Conference on Neural Networks, volume II, pages 537–544, 1990.
D. O. Hebb. The Organization of Behavior. Wiley, New York, 1949.
R. Hecht-Nielsen. Neural network nearest matched filter classificaton of spatio-temporal patterns. Applied Optics, 26:1892–1899, 1987.
J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Science, U.S.A., 79:2554–2558, 1982.
J. J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Science, U.S.A., 81:3088–3092, 1984.
S. Jha, C. Chapman, and T. Durrani. Investigation into neural networks for bearing estimation. In J. Lacoume, A. Chehikean, N. Martyin, and J. Malbos, editors, Signal Processing IV: Theories and Applications. Elsevier Science Publishers, London, 1988.
M. S. Tomlinson Jr., D. J. Walker, and M. A. Sivilotti. A digital neural network architecture for VLSI. In International Joint Conference on Neural Networks, volume II, pages 545–550, 1990.
A. Khotanzad, J. Lu, and M. Srinath. Target detection using a neural network based passive sonar system. In International Joint Conference on Neural Networks, volume I, pages 335–340, 1988.
H. T. Kung, D. A. Pomerleau, G. L. Gusciora, and D. S. Touretzky. How we got 17 million connections per second. In International Conference on Neural Networks, volume 2, pages 143–150, 1988.
S. Y. Kung. Parallel Architectures for Artificial Neural Nets. In International Conf. on Systolic Arrays, pages 163–174, 1988.
S. Y. Kung and J. N. Hwang. A Unified Systolic Architecture for Artificial Neural Nets. Journal of Parallel and Distributed Computing, 6:358–387, 1989.
G. Lev, N. Pippenger, and L. Valiant. A fast parallel algorithm for routing in permutation networks. IEEE Transactions on Computers, C-30(2):93–100, Feb. 1981.
R. Lippman and B. Gold. Neural classifiers useful for speech recognition. In International Conference on Neural Networks, volume IV, pages 417–426, 1987.
R. P. Lippman. An Introduction to Computing with Neural Nets. IEEE ASSP Magazine, pages 4–22, April 1987.
A. Maccato and R. de Figueiredo. A neural network based framework for classification of oceanic acoustic signals. In Proceedings of Oceans’ 89, Seattle, pages 1118–1123, 1990.
W. S. McCulloch and W. H. Pitts. A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys., 5:115–133, 1943.
Manavendra Misra and V. K. Prasanna Kumar. Efficient VLSI Implementation of Iterative Solutions to Sparse Linear Systems. In J. McCanny, J. McWhirter, and E. Swartzlander Jr., editors, Systolic Array Processors, pages 52–61. Prentice Hall, 1989. Proceedings of the 3rd Int. Conf. on Systolic Arrays.
Manavendra Misra and V. K. Prasanna Kumar. Massive Memory Organizations for Implementing Neural Networks. In International Conference on Pattern Recognition, June 1990.
Manavendra Misra and V. K. Prasanna Kumar. Neural network simulation on a Reduced Mesh of Trees organization. In SPIE/SPSE Symposium on Electronic Imaging, Feb. 1990.
David Nassimi and Sartaj Sahni. Parallel Algorithms to set up the Benes Permutation Network. IEEE Transactions on Computers, C-31(2):148–154, Feb. 1982.
K. W. Przytula and V. K. Prasanna Kumar. Algorithmic mapping of neural networks models on parallel SIMD machines. In International Conference on Application Specific Array Processing, 1990.
K. W. Przytula, W-M. Lin, and V. K. Prasanna Kumar. Partitioned implementation of neural networks on mesh connected array processors. In Workshop on VLSI Signal Processing, 1990.
C. S. Raghavendra and V. K. Prasanna Kumar. Permutations on ILLIACIV Type Networks. IEEE Transactions on Computers, C-37(7):622–629, July 1986.
U. Ramacher and J. Beichter. Systolic Architectures for Fast Emulation of Artificial Neural Networks. In J. McCanny, J. McWhirter, and E. Swartzlander Jr., editors, Systolic Array Processors, pages 277–286. Prentice Hall, 1989. Proceedings of the 3rd Int. Conf. on Systolic Arrays.
Sanjay Ranka. A distributed implementation of backpropagation. Manuscript, Department of Computer Science, Syracuse University, 1990.
Sanjay Ranka, N. Asokan, R. Shankar, C. K. Mohan, and K. Mehrotra. A neural network simulator on the Connection Machine. In Fifth IEEE International Symposium on Intelligent Control, 1990.
R. Rastogi, P. Gupta, and R. Kumeresan. Array signal processing with inter-connected neuron-like elements. In International Conference on Acoustics, Speech, and Signal Processing, pages 54.8.1-4, 1987.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In Parallel Distributed Processing: Exploration in the Microstructure of Cognition, volume 1, chapter 8, pages 318–362. MIT Press, Cambridge, Massachusetts, 1986.
D. E. Rumelhart, J. L. McClelland, and the PDP Research Group. Parallel Distributed Processing: Exploration in the Microstructure of Cognition, volume 1. MIT Press, Cambridge, Massachusetts, 1986.
Soheil Shams and K. W. Przytula. Mapping of neural networks onto programmable parallel machines. In International Symposium on Circuits and Systems, May 1990.
P. Simpson. Artificial Neural Systems: Foundations, Paradigms, Applications and Implementations. Elmsford Press: Pergamon Press, 1990.
Planning Systems. Sonar classification with neural networks. Neural Ware Connections, 1(1), 1989.
S. Tomboulian. Introduction to a system for implementing Neural Net connections on SIMD architectures. Technical Report ICASE No. 88-3, Institute for Computer Applications in Science and Engineering, NASA Langley Research Center, January 1988.
B. W. Wah and L-C. Chu. Efficient mapping of neural networks on multicomputers. In International Conference on Parallel Processing, volume I, pages 234–238, 1990.
T. Watanabe, Y. Sugiyama, T. Kondo, and Y. Kitamura. Neural Network Simulation on a Massively Parallel Cellular Array Processor: AAP-2. In International Joint Conference on Neural Networks, June 1989.
Stephen S. Wilson. Neural computing on a one dimensional SIMD array. In International Joint Conference on Artificial Intelligence, pages 206–211, 1989.
X. Zhang, M. McKenna, J. P. Mesirov, and D. Waltz. An Efficient Implementation of the Backpropagation Algorithm on the Connection Machine CM-2. In NIPS, 1989.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1991 Springer Science+Business Media New York
About this chapter
Cite this chapter
Misra, M., Prasanna Kumar, V.K. (1991). Implementation of Sparse Neural Networks on Fixed Size Arrays. In: Bayoumi, M.A. (eds) Parallel Algorithms and Architectures for DSP Applications. The Springer International Series in Engineering and Computer Science, vol 149. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-3996-4_10
Download citation
DOI: https://doi.org/10.1007/978-1-4615-3996-4_10
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-6786-4
Online ISBN: 978-1-4615-3996-4
eBook Packages: Springer Book Archive