article

Asymptotic convergence of backpropagation

Authors:

Gerald Tesauro,

Yu He, and

Subutai AhmadAuthors Info & Claims

Neural Computation, Volume 1, Issue 3

Pages 382 - 391

https://doi.org/10.1162/neco.1989.1.3.382

Published: 01 September 1989 Publication History

Abstract

We calculate analytically the rate of convergence at long times in the backpropagation learning algorithm for networks with and without hidden units. For networks without hidden units using the standard quadratic error function and a sigmoidal transfer function, we find that the error decreases as 1/t for large t, and the output states approach their target values as 1/t. It is possible to obtain a different convergence rate for certain error and transfer functions, but the convergence can never be faster than 1/t. These results are unaffected by a momentum term in the learning algorithm, but convergence can be substantially improved by an adaptive learning rate scheme.

For networks with hidden units, we generally expect the same rate of convergence to be obtained as in the single-layer case; however, under certain circumstances one can obtain a polynomial speed-up for non sigmoidal units, or a logarithmic speed-up for sigmoidal units. Our analytic results are confirmed by empirical measurements of the convergence rate in numerical simulations.

References

[1]

Ahmad, S. 1988. A study of scaling and generalization in neural networks. Master's Thesis, University of Illinois at Urbana-Champaign.

Google Scholar

[2]

Ahmad, S., and Tesauro, G. 1988. Scaling and generalization in neural networks: A case study. In Proceedings of the 1988 Connectionist Models Summer School, D. Touretzky et al., eds. Morgan Kaufman, San Mateo, CA. 3-10.

Google Scholar

[3]

Hinton, G.E. 1987. Connectionist learning procedures. Tech. Rep. CMU-CS-87- 115, Department of Computer Science, Carnegie-Mellon University.

Google Scholar

[4]

Jacobs, R.A. 1988. Increased rates of convergence through learning rate adaptation. Neural Networks 1, 295-307.

Crossref

Google Scholar

[5]

Le Cun, Y. 1985. A learning procedure for asymmetric network. Proc. Cognit. (paris), 85, 599-604.

Google Scholar

[6]

Minsky, M., and Papert, S. 1969. Perceptrons. MIT Press, Cambridge, MA.

Google Scholar

[7]

Parker, D.B. 1985. Learning-logic. Tech. Rep. TR-47, MIT Center for Computational Research in Economics and Management Science.

Google Scholar

[8]

Rumelhart, D.E., Hinton, G.E., and Williams, R.J. 1986. Learning representations by back-propagating errors. Nature (London) 323, 533-536.

Crossref

Google Scholar

[9]

Werbos, P. 1974. Ph.D. Thesis, Harvard University.

Google Scholar

[10]

Widrow, B., and Hoff, M.E. 1960. Adaptive switching circuits. Institute of Radio Engineers, Western Electronic Show and Convention, Convention Record, Part 4, 96-104.

Google Scholar

Cited By

View all

Meulemans ACarzaniga FSuykens JSacramento JGrewe BLarochelle HRanzato MHadsell RBalcan MLin H(2020)A theoretical framework for target propagationProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3497405(20024-20036)Online publication date: 6-Dec-2020
https://dl.acm.org/doi/10.5555/3495724.3497405
Bablani AEdla DTripathi DCheruku R(2019)Survey on Brain-Computer InterfaceACM Computing Surveys10.1145/329771352:1(1-32)Online publication date: 13-Feb-2019
https://dl.acm.org/doi/10.1145/3297713
Ahmad STesauro GHe Y(1989)Asymptotic convergence of backpropagationProceedings of the 2nd International Conference on Neural Information Processing Systems10.5555/2969830.2969904(606-613)Online publication date: 1-Jan-1989
https://dl.acm.org/doi/10.5555/2969830.2969904

Index Terms

Asymptotic convergence of backpropagation
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Backpropagation convergence via deterministic nonmonotone perturbed minimization
NIPS'93: Proceedings of the 6th International Conference on Neural Information Processing Systems

The fundamental backpropagation (BP) algorithm for training artificial neural networks is cast as a deterministic nonmonotone perturbed gradient method. Under certain natural assumptions, such as the series of learning rates diverging while the series ...
Read More
Asymptotic convergence of backpropagation: numerical experiments
NIPS'89: Proceedings of the 2nd International Conference on Neural Information Processing Systems

We have calculated, both analytically and in simulations, the rate of convergence at long times in the backpropagation learning algorithm for networks with and without hidden units. Our basic finding for units using the standard sigmoid transfer ...
Read More
Non-asymptotic superlinear convergence of standard quasi-Newton methods
Abstract
In this paper, we study and prove the non-asymptotic superlinear convergence rate of the Broyden class of quasi-Newton algorithms which includes the Davidon–Fletcher–Powell (DFP) method and the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method. The ... $^{}$
Read More

Comments

Information & Contributors

Information

Published In

Neural Computation Volume 1, Issue 3

Fall 1989

129 pages

ISSN:0899-7667

Issue’s Table of Contents

Publisher

MIT Press

Cambridge, MA, United States

Publication History

Published: 01 September 1989

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

View all

Meulemans ACarzaniga FSuykens JSacramento JGrewe BLarochelle HRanzato MHadsell RBalcan MLin H(2020)A theoretical framework for target propagationProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3497405(20024-20036)Online publication date: 6-Dec-2020
https://dl.acm.org/doi/10.5555/3495724.3497405
Bablani AEdla DTripathi DCheruku R(2019)Survey on Brain-Computer InterfaceACM Computing Surveys10.1145/329771352:1(1-32)Online publication date: 13-Feb-2019
https://dl.acm.org/doi/10.1145/3297713
Ahmad STesauro GHe Y(1989)Asymptotic convergence of backpropagationProceedings of the 2nd International Conference on Neural Information Processing Systems10.5555/2969830.2969904(606-613)Online publication date: 1-Jan-1989
https://dl.acm.org/doi/10.5555/2969830.2969904

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Backpropagation convergence via deterministic nonmonotone perturbed minimization