Exponential-capacity Associative Memory

This is a collaboration between Prof. Amin Shokrollahi, Dr. Amin Karbasi (Yale), Dr. Raj K. Kumar, Dr. Lav R. Varshney (UIUC) and Prof. Wulfram Gerstner.

In this project, we focus on improving the performance of artificial neural associative memories by using techniques from graph-based error correcting codes (such as LDPC) and exploiting the inherent structure of the input patterns in order to increase the pattern retrieval capacity from O(n) to O(a^n), where a > 1. The main idea is that it is much easier to memorize more patterns that have some redundancy, like natural scenes, to memorize the more random patterns.

We propose an online learning algorithm to learn the neural graph from examples and recall algorithms that use iterative message passing over the learned graph to eliminate noise during the recall phase. We gradually improve the proposed neural model to achieve the ability to correct a linear number of errors in the recall phase. We also propose a simple trick to extend the model from linear to nonlinear regimes as well.

We will also show how a neural network with noisy neurons rather counter-intuitively–achieves a better performance in the recall phase. The results of this approach is also used in graph-based error correcting codes to improve the short-length performance of such codes and achieve their asymptotic performance much earlier.

Related Papers
Noise facilitation in associative memories of exponential capacity
A. Karbasi, A. H. Salavati, A. Shokrollahi, L. R. Varshney		 2014
Nonbinary Associative Memory With Exponential Pattern Retrieval Capacity and Iterative Learning
A. H. Salavati, R.K. Kumar, A. Shokrollahi		2014
Coupled neural associative memories
A. Karbasi, A. H. Salavati, A. Shokrollahi		 2013
Iterative learning and denoising in convolutional neural associative memories
A. Karbasi, A. H. Salavati, A. Shokrollahi		 2013
Noise-enhanced associative memories
A. Karbasi, A. H. Salavati, A. Shokrollahi, L. R. Varshney		 2013
Multi-Level Error-Resilient Neural Networks
A. H. Salavati, A. Karbasi		 2012
Exponential pattern retrieval capacity with non-binary associative memory
R. K. Kumar, A. H. Salavati, A. Shokrollahi		 2011
Neural pre-coding increases the pattern retrieval capacity of Hopfield and Bidirectional Associative Memories
A. H. Salavati, R. K. Kumar, A. Shokrollahi, W. Gerstner		 2011

Some Useful Resources