[back]

Dynamic Recurrent Neural Networks for Pattern Recognition
Alex Holub, Gilles Laurent, Pietro Perona

We are investigating the computational properties of recurrent neural networks of binary artificial neurons. Our investigations are guided by recent work performed in the laboratory of Gilles Laurent which involves elucidating the underlying processing mechanisms in early olfactory processing. These physiological investigations indicate that the initial olfactory processing layer (in the locust the antennal lobe) consists of a dynamic recurrent neural network of excitatory and inhibitory units. The presentation of stimuli to the network results in stereotyped spatio-temporal neural firing patterns, with each unique stimulus presentation invoking a unique temporally-varying pattern of activity within the population of neurons. We have approximated the biological networks using recurrent networks with discrete binary neural elements. These non-linear networks exhibit chaotic behavior such that similar input patterns obtain very dissimilar network representations through the network dynamics. Similar pattern spreading characteristics have been observed in the initial processing networks of fish by members of the Laurent laboratory and it has been hypothesized that pattern spreading may be one computational benefit which the initial processing layer provides.

We are currently interested in further exploring the potential computational benefits of dynamical neural networks for the pattern recognition problem. Specifically, biology indicates that the processing layer directly post-synaptic to the recurrent network consists of neurons which act as linear threshold units reading snap-shots of the dynamical activity within the antennal lobe. The biological system thus shares some homology with support vector machines (SVMs). An SVM classifier maps input patterns to a high-dimensional space in a manner specified by the SVM kernel. Patterns corresponding to different classes are then separated in this high-dimensional space using a linear hyperplane (specified by a linear threshold unit) which maximizes the margins between the classes of interest. In the case of olfaction, inputs are mapped to the antennal lobe and the dynamics within the antennal lobe act as the kernel. We are interested in exploring what computational benefits the dynamical kernel of the antennal lobe provides. Of particular interest to us are increases in classification ability from the dynamics, decreases in the computational complexity of the dynamical system, and increases in noise-robustness.

Of added interest is how the connectivity of the recurrent network influences the efficacy of the dynamic kernel. Our initial experiments were all performed using sparse, randomly connected, networks of artificial units. These networks exhibit well-documented chaotic effects such that two similar pattern inputs to the system will inevitably result in disparate representation within the network. By changing the connectivity to follow local rules, such that the probability of two neighboring units being connected is greater than units farther apart, we can influence the global properties of the dynamical system. In particular we observe both a slower rate and smaller absolute divergence between two similar pattern inputs. Both of these properties are useful for the dynamic kernel, as similar patterns will retain similar representations over time -- thereby making them easier to group together and classify by a classifier. Further explorations of how pattern recognition ability changes as a function of network connectivity may provide additional insights into both the computational capabilities of the system as well as the function of its biological counterpart.