Skip to content

Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey E. Hinton

Neural Computation · 2002 · 5,034 citations

No retraction on record in OpenAlex.Check Crossref now

Cite this paper

Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771–1800. https://doi.org/10.1162/089976602760128018

In text: (Hinton, 2002)

Ask a question about this research

One question, free, no account. Anchorcite finds the papers, checks each against Crossref and links every sentence of the answer to the passage it comes from.

Abstract

It is possible to combine multiple latent-variable models of the same data by multiplying their probability distributions together and then renormalizing. This way of combining individual "expert" models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert, because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data. A product of experts (PoE) is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary. Training a PoE by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule. Fortunately, a PoE can be trained using a different objective function called "contrastive divergence" whose derivatives with regard to the parameters can be approximated accurately and efficiently. Examples are presented of contrastive divergence learning using several types of expert on several types of data.

  1. Long Short-Term Memory1997
  2. Gradient-based learning applied to document recognition1998
  3. Particle swarm optimization2002
  4. Greedy function approximation: A gradient boosting machine.2001
  5. Statistical Learning Theory1999

Metadata from OpenAlex (CC0). Citations are generated from the published record.