Neural Networks: Tricks of the Trade

Second Edition

  • Grégoire Montavon
  • Geneviève B. Orr
  • Klaus-Robert Müller

Part of the Lecture Notes in Computer Science book series (LNCS, volume 7700)

Table of contents

  1. Front Matter
  2. Introduction

    1. Klaus-Robert Müller
      Pages 1-5
  3. Speeding Learning

    1. Klaus-Robert Müller
      Pages 7-8
    2. Yann A. LeCun, Léon Bottou, Genevieve B. Orr, Klaus-Robert Müller
      Pages 9-48
  4. Regularization Techniques to Improve Generalization

    1. Klaus-Robert Müller
      Pages 49-51
    2. Lutz Prechelt
      Pages 53-67
    3. Thorsteinn S. Rögnvaldsson
      Pages 69-89
    4. Jan Larsen, Claus Svarer, Lars Nonboe Andersen, Lars Kai Hansen
      Pages 111-130
    5. David Horn, Ury Naftaly, Nathan Intrator
      Pages 131-137
  5. Improving Network Models and Algorithmic Tricks

    1. Klaus-Robert Müller
      Pages 139-141
    2. Rich Caruana
      Pages 163-189
    3. Patrick van der Smagt, Gerd Hirzinger
      Pages 191-203
    4. Nicol N. Schraudolph
      Pages 205-223
  6. Representing and Incorporating Prior Knowledge in Neural Network Training

    1. Patrice Y. Simard, Yann A. LeCun, John S. Denker, Bernard Victorri
      Pages 235-269
    2. Steve Lawrence, Ian Burns, Andrew Back, Ah Chung Tsoi, C. Lee Giles
      Pages 295-309
    3. Jürgen Fritsch, Michael Finke
      Pages 311-338
  7. Tricks for Time Series

    1. Klaus-Robert Müller
      Pages 339-341
    2. Ralph Neuneier, Hans Georg Zimmermann
      Pages 369-418
  8. Big Learning in Deep Neural Networks

    1. Grégoire Montavon, Klaus-Robert Müller
      Pages 419-420
    2. Léon Bottou
      Pages 421-436
    3. James Martens, Ilya Sutskever
      Pages 479-535
    4. Ronan Collobert, Koray Kavukcuoglu, Clément Farabet
      Pages 537-557
  9. Better Representations: Invariant, Disentangled and Reusable

    1. Grégoire Montavon, Klaus-Robert Müller
      Pages 559-560
    2. Adam Coates, Andrew Y. Ng
      Pages 561-580
    3. Dan Claudiu Cireşan, Ueli Meier, Luca Maria Gambardella, Jürgen Schmidhuber
      Pages 581-598
    4. Grégoire Montavon, Klaus-Robert Müller
      Pages 621-637
    5. Jason Weston, Frédéric Ratle, Hossein Mobahi, Ronan Collobert
      Pages 639-655
  10. Identifying Dynamical Systems for Forecasting and Control

    1. Grégoire Montavon, Klaus-Robert Müller
      Pages 657-658
    2. Mantas Lukoševičius
      Pages 659-686
    3. Hans-Georg Zimmermann, Christoph Tietz, Ralph Grothmann
      Pages 687-707
    4. Siegmund Duell, Steffen Udluft, Volkmar Sterzing
      Pages 709-733
  11. Back Matter

About this book


The twenty last years have been marked by an increase in available data and computing power. In parallel to this trend, the focus of neural network research and the practice of training neural networks has undergone a number of important changes, for example, use of deep learning machines.

The second edition of the book augments the first edition with more tricks, which have resulted from 14 years of theory and experimentation by some of the world's most prominent neural network researchers. These tricks can make a substantial difference (in terms of speed, ease of implementation, and accuracy) when it comes to putting algorithms to work on real problems.


back-propagation graphics processing unit multilayer perceptron neural reinforcement learning optimization

Editors and affiliations

  • Grégoire Montavon
    • 1
  • Geneviève B. Orr
    • 2
  • Klaus-Robert Müller
    • 1
  1. 1.Dept. of Computer ScienceTechnische Universität BerlinBerlinGermany
  2. 2.Dept. of computer ScienceWillamette UniversitySalemUSA

Bibliographic information

  • DOI
  • Copyright Information Springer-Verlag Berlin Heidelberg 2012
  • Publisher Name Springer, Berlin, Heidelberg
  • eBook Packages Computer Science Computer Science (R0)
  • Print ISBN 978-3-642-35288-1
  • Online ISBN 978-3-642-35289-8
  • Series Print ISSN 0302-9743
  • Series Online ISSN 1611-3349
  • Buy this book on publisher's site
Industry Sectors
IT & Software
Consumer Packaged Goods
Energy, Utilities & Environment