Inductive Modelling of Temporal Sequences by Means of Self-organization. Jan Koutnik. IWIM, Prague, 2007.

 Article (in pdf)

Abstract. In this paper we present a new self-organizing neural network, which builds a spatiotemporal model of an input temporal sequence inductively. The network is an extension of Kohonen’s Self-organizing Map with a modified Hebb’s rule for update of temporal synapses. The model building behavior is shown on inductive learning of a transition matrix from a data generated by a Markov Chain.

Keywords. Inductive modelling, self-organization, temporal sequences.

References.

1. Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179–211, 1990.
2. Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
3. Sepp Hochreiter and J¨urgen Schmidhuber. LSTM can solve hard long time lag problems. In Michael C. Mozer, Michael I. Jordan, and Thomas Petsche, editors, Advances in Neural Information Processing Systems, volume 9, page 473. The MIT Press, 1997.
4. Teuvo Kohonen. Self-Organizing Maps. Springer-Verlag, 3rd edition, 2001.
5. J. Koutnik and M. Snorek. Self-organizing neural networks for signal recognition. In 16th International Conference on Artificial Neural Networks Proceedings (ICANN 2006), Part I, volume 4131/2006, pages 406–414. Springer Berlin / Heidelberg, 2006.
6. A. Nyamapfene and K. Ahmad. Unsupervised multi-net simulation: An application to child development. In Proceedings of the IJCNN ’06. International Joint Conference on Neural Networks, pages 1427–1433, 2006.
7. Lawrence R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257– 286, 77(2):257–286, 1989.
8. M. Strickert and B. Hammer. Neural gas for sequences. In T. Yamakawa, editor, Proceedings of the Workshop on Self-Organizing Networks (WSOM), pages 53–58. Kyushu Institute of Technology, 2003.
9. J. Heikkonen K. Kaski T. Koskela, M. Varsta. Temporal sequence processing using recurrent SOM. In Proceedings of the 2nd International Conference on Knowledge-Based Intelligent Engineering Systems, volume 1, Adelaide, Australia, 1998.
10. M. Varsta, J. Heikkonen, and J. Lampinen. Analytical comparison of the temporal kohonen map and the recurrent self organizing map. In M. Verleysen, editor, European Symposium on Artificial Neural Networks (ESANN), pages 273–280. D-facto Publications, 2000.
11. Markus Varsta. Self-organizing maps in sequence processing, 2002.
12. T. Voegtlin. Context quantization and contextual self-organizing maps. In Proc. Int. Joint Conf. on Neural Networks, volume 5, pages 20–25, 2000.
13. Thomas Voegtlin. Recursive self-organizing maps. Neural Networks, 15(8-9):979–991, 2002.
14. P. J.Werbos. Back propagation through time: What it does and how to do it. In Proceedings of the IEEE, volume 78, pages 1550–1560, 1990.