Neural Networks: Theory and Applications

Course Outline:

1. Review of essential properties of the biological neuron and the nervous system: neuron, synapse and axon; The Hudgkin-Huxley and Fitzhugh-Nagumo equations; neuron network, generic features, neuron models.
2. Essentials of nonlinear dynamical system theory: state-space representation, stability, attractors, basins of attraction, bifurcation diagram, Poincare sections, Lyapanov exponents, chaos; The pendulum, the logistic map, and the circle map as examples of simple systems that exhibit complex dynamical behavior.
3. The Hopfield Model and Spin Glasses: energy functions, convergence theorem, associative memory, storage capacity, heteroassociative and bi-directional associative memory, N-P hard problems, solution of optimization problems in combinatorial optimization and image processing; Hardware implementation both electronic and photonic. Design of a photonic neural network for rapid solution of the travelling salesman problem.
4. Stochastic Neural Networks and the Boltzmann Machine: The Boltzmann-Gibbs distribution, free energy, entropy, stochastic dynamics. The Metropolis-Kirkpatric algorithm, simulated annealing, stochastic neural nets, noisy thresholding, the Boltzmann machine and examples of its use in the solution of problems in combinatorial optimization and machine vision, (e.g., wire routing and component placement in VLSI, stereo vision, image smoothing, etc...); Stochastic learning. Digital vs. analog implementation of stochastic networks.
5. Multilayer Feedforward Networks For Supervised Learning: The perceptron, convergence theorem, limitations, the X0R, the role of hidden units; Back-propagation, applications (NET talk, sonar target recognition, Image compression, prediction and forecasting, handwritten ZIP-code recognition, optimal path finding).
6. Unsupervised and Competitive Learning Algorithms: Winner-take-all networks, Kohonen's self-organizing feature maps, adaptive resonance networks.
7. Bifurcating Neural Networks: The Bifurcating neuron, chaos as a mechanism for annealing and directed search, Bifurcating neuron networks for cognition and higher level processing. Solution of the TSP with bifurcating neural networks.

Text:

• J. Hertz, A. Krogh and R.G. Palmer, Introduction to the Theory of Neural Computation, Addison Wesley, Redwood City, CA, 1991.

Reference Texts:

• J. Dayhoff, Neural Network Architectures, Van Nostrand Reinhold, New York, 1990.
• Philip D. Wasserman, Neural Computing: Theory and Practice, Van Nostrand Reinhold, 1989.
• S. Haykins, Neural Networks (2nd Ed.), Prentice Hall, 1999.

Class and Laboratory Demonstrations:

The impossible circuit, the double and the triple pendulum, The Hughes LCLV programmable spatial light modulator method of forming programmable connection weights in photonic neural networks, sigmoidal, logistic, and bifurcating photonic neural networks.

Grading:

• 1/3 homework
• 1/3 midterm
• 1/3 final

NB: Code of Academic Integrity

Using or attempting to use unauthorized assistance, material, lab results, or solutions (in part or whole) is a violation of the Code of Academic Integrity and will result in a zero grade for the course.