Neural Network Theories

A summary and Python TensorFlow implementation of theories described in Learning Internal Representations by Error Propagation by David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.

A Brief HIstory

In their book, Perceptrons (1969), Marvin Minsky and Seymour Papert state:

The perceptron has shown itself worthy of study despite (and even because of!) its severe limitations. It has many features that attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. There is no reason to suppose that any of these virtues carry over to the many-layered version. Nevertheless, we consider it to be an important research problem to elucidate (or reject) our intuitive judgement that the extension is sterile. Perhaps some powerful convergence theorem will be discovered, or some profound reason for the failure to produce an interesting "learning theorem" for the multilayered machine will be found. (pp. 231-232)

Such pessimistic view presented by Minsky and Papert (1969) arguably brought about the so-called AI winter which lasted throughout the following decade.

In this report published in 1985, Rumelhart, Hinton, and Williams conclude:

Although our learning results do not guarantee that we can find a solution for all solvable problems, our analyses and results have shown that as a practical matter, the error propagation scheme leads to solutions in virtually every case.

Through persistence in the shadow of doubt, the trio of academics answered Minsky and Papert's challenge, and ignited a renewed interest in deep learning. Today, these very models are ubiquitous. From computer processor optimization (e.g. AMD SenseMI) to impeccable text-to-speech synthesis (e.g. Google Tacotron, Google Duplex), neural networks are now the face of artificial intelligence.

Required Python Libraries

matplotlib
numpy
tensorflow

jaewan-yun / neural-network-theories Goto Github PK