Skip to content

Learning to Forget: Continual Prediction with LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins

Neural Computation · 2000 · 6,144 citations

No retraction on record in OpenAlex.Check Crossref now

Cite this paper

Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10), 2451–2471. https://doi.org/10.1162/089976600300015015

In text: (Gers et al., 2000)

Ask a question about this research

One question, free, no account. Anchorcite finds the papers, checks each against Crossref and links every sentence of the answer to the passage it comes from.

Abstract

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.

  1. Long Short-Term Memory1997
  2. Gradient-based learning applied to document recognition1998
  3. Particle swarm optimization2002
  4. Greedy function approximation: A gradient boosting machine.2001
  5. Statistical Learning Theory1999

Metadata from OpenAlex (CC0). Citations are generated from the published record.