NEuroNet Roadmap
[Preliminary Draft]

First Report on the Use of Neural Networks and Other Computational Intelligence Techniques in Intelligent Multimedia Systems

                           S Kollias, F Piat and A Drossopoulos, ICCS-NTUA, 3rd June 1999

Being involved in the European TMR project Physta, whose goal is to develop an intelligent system to recogniseemotions based on the integration of facial and vocal expressions [4,5], we are particularly aware of the potential of Neural Networks (NNs) for intelligent Human-Computer Interaction (HCI). Making HCI more natural to humans and more intelligent is one of the greatest, fundamental challenges of multimedia applications. Humans naturally communicate through a variety of channels and intelligence is required in order to interpret humans’ messages, at both within- and between-channel levels of processing. The voice and the face are the two privileged channels of communication, and therefore applications to face and voice processing are emphasised in what follows. On another hand, the proliferation of data available to users is so important that information retrieval, data transmission and browsing efficiency are crucial issues of today’s computer use. These applications also represent a great potential for the use of NNs, as mentioned below.
 

Face processing:

• Face recognition (identification)

• Emotion/expression recognition

• HCI: on-line help, teaching
• Detection of emergencies
• Recruitment/business assistant
• Automatic customer satisfaction evaluation
• Compression / Decompression to improve transmission rates of images

• Security applications

• Voice recognition (identification)
• Password identification

• Natural Language Understanding

• Voice-controlled applications
• Interactive foreign language learning

• Search / Indexing / Content-based retrieval: to go beyond simple searches by composer or by piece, search based on tonal 'trace' or 'signature'
• Composition of new variations / new pieces (given one or several themes). Interactive improvisation system.
• Dance and music: Choreography, evaluation

• Audio/Visual (disambiguating messages by using a second channel: e.g., Lip/Speech synchronisation for videoconferencing or improvement of speech recogniser) [6]
• Visuo/Visual (e.g., Face/Gestures)

• Content Based Retrieval/ Scene Analysis: Quick browsing (VICAR system).

• Object (image/video sequence) classification based on content of visual features
• Visual features classification of colour/brightness, texture, shapes

• Agents: User profile/classification (limit search into subsets of the database) using Fuzzy NNs [7]

References

[1] Special Issue on Artificial Neural Network Applications, Proceedings of the IEEE, vol. 84, no. 10, Oct. 1996.

[2] Special Issue Part One: Multimedia Signal Processing, Proceedings of the IEEE, vol. 86, no. 5, May 1998.

[3] Special Issue Part Two: Multimedia Signal Processing, Proceedings of the IEEE, vol. 86, no. 6, June 1998.

[4] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz and J. Taylor, “Emotion recognition in human-computer inter-action,” submitted for publication to the IEEE Signal Processing Magazine, January 1999.

[5] Development of Feature Representations from Emotionally coded Facial Signals and Speech, Report for TMR Physta project, Research contract FMRX – CT97 – 0098 (DG12-BDCN)

[6] S.Y. Kung and J.N. Hwang, “Neural Networks for Intelligent Multimedia Processing”, Special Issue Part Two: Multimedia Signal Processing, Proceedings of the IEEE, vol. 86, no. 6, June 1998.

[7] Modulate Project Report, Education Multimedia Project, 1999-2001.