An Approximate Algorithm for Reverse Engineering of Multi-layer Perceptrons
We present an approximate algorithm for reconstructing internals of multi-layer perceptrons from membership queries. The key component of the algorithm is a procedure for reconstructing weights of a single linear threshold unit. We prove that the approximation error, measured as the distance between the original and the reconstructed weights, is dropping exponentially fast with the number of queries. The procedure is combined with a labelling strategy that involves solving multiple Linear Programming problems. This combination results in an algorithm that extracts internals of multi-layer per ceptrons: the number of units in the first hidden layer, their weights, and a boolean funct ion that is computed by the remaining nodes. In practice, networks that compute boolean combinations of 10–15 hyperplanes can be reconst ructe d in several minutes.
KeywordsHide Layer Approximate Algorithm Labelling Function Labelling Procedure Boolean Combination
Unable to display preview. Download preview PDF.
- D. Angluin. Queries revisited. In T. Z. N. Abe, R. Khardon, editor, Algorithmic Learning Theory, 12th International Conference, ALT 2001, Washington, DC, USA, November 25-28, 2001, 12–31. Springer, 2001.Google Scholar
- C. Bishop. Neural Newtorks for Pattern Recognition. Oxford University Press, 1995.Google Scholar
- P. Goldberg and S. Kwek. The precision of query points as a resource for learning convex polytopes with membership queries. In Proc. 13th Annu. Conference on Comput. Learning Theory, pages 225–235. Morgan Kaufmann, San Francisco, 2000.Google Scholar
- M. Hasenjager and H. Ritter. Active learning in neural networks.Google Scholar
- V. Shevchenko. On deciphering a threshold function of many-values logic. Gorkü State University, pages 155–166, 1987.Google Scholar
- G. Strang. Linear algebra and its applications. Third Edition. Harcourt Brace Jovanovich, Publishers, 1988.Google Scholar
- Y. Zhang. User’s guide to LIPSOL, Rice University, 1998.Google Scholar