Network Acronym: PHYSTA


Title: Principled Hybrid Systems: Theory and Applications


Contract Number: ERB4061PL970238


Contractual Period: Start date: 1998-01-09
End date: 2001-01-08
Duration: 3 years


Coordinator: Prof. Stefanos Kollias
Institute of Communications and Computer Systems
National Technical University of Athens
Electrical and Computer Engineering
9 Heroon Polytechniou Str.
Zographou 15773, Athens
Tel: +30-1-772 2488
Fax: +30-1-772 2492


Other Participants: Prof. J. Taylor, King’s College London (KCL),
London, UK
Prof. C. Gielen, Katholieke Universiteit Nijmegen (KUN),
Nijmegen, NL
Ass. Prof. B. Apolloni, University of Milan (UM),
Milan, IT
Dr. E. Douglas-Cowie, Queen’s University of Belfast (QUB), Belfast, Northern Ireland


Objectives of the Network: Systematic principles for integrating symbolic and subsymbolic processing will be developed in the project. Key aims are to ensure that the resulting total hybrid system retains desirable properties of both processing levels. On the one side the signal processing abilities, robustness and learning capability of neural networks should be preserved. On the other side advantage should be taken of the ability of rule-based systems to exploit high level knowledge and existing algorithms and to explain (to a user) why conclusions were reached in particular cases. The methodologies to be developed in the project will be tested in a challenging application related to human computer interaction, which is recognition of emotion based on both voice and visual cues. Low level features will be extracted from signals using neural networks and subsequent formulation of rules will provide a conceptual framework, substantial for emotion analysis.


Partnership: The collaboration of the partners will be achieved in the following ways: through the exchange of young researchers, who will carry and absorb the expertise of different partners; by transferring to each other through e-mail the results of weekly meetings to be held in each of the labs; through audio-conferences which will be held in cases where specific problems need to be discussed between different groups in a round-table manner; through the two yearly meetings of all the participants in the project.


Applications: The developed neural networks and rule-based systems will be tested for the detection of emotion in human-computer interaction (HCI) systems; new rules will be derived or known rules will be enriched based on the analysis of vocal and facial input signals. The inverse HCI problems of emotionally appropriate 3-D face modelling and speech synthesis will also take advantage of the advances generated by the project. The work of the project will be highly relevant for several industrial applications. British Telecom (BT) and Intracom S.A. have expressed their interest in following the results of the project.


Training Aspects: The expertise in the various groups is complementary in several aspects: connectionist versus knowledge-based techniques (UM, KUN, NTUA), artificial versus biological networks (KCL, KUN), software versus hardware implementation (UM, KCL, NTUA), psychological and linguistic versus mathematical analyses (QUB, KUN), image and speech analysis and engineering applications versus theoretical computer science (NTUA, QUB, UM). The (young) researchers (Ph.D. students and postdocs) from all groups will work in each other's laboratories in order to learn and transfer new expertise.


Results and Achievements: In the first year of the project there has been significant progress in both aspects of the project, i.e. subsymbolic neural network based feature extraction and analysis as well as symbol generation on one hand, and analysis of the emotion understanding application on the other. In the former aspect, a state-of-the-art review has been performed and various extensions and theories are being developed; in the latter, a first systematic exploration of the problem has been performed, which includes many novel ideas and specific ways for the emotion understanding system implementation. This lead in the second year to the development of two systems that complete each other for a full, comprehensive description of emotions. "Feeltrace", a software for the emotional labelling of any material (audio and/or visual) has been implemented correspondingly. The description systems also provided the guidelines needed for the collection of an audio-visual database covering all aspects of emotions, of better quality and more realistic than could be found anywhere. The collection of this database is now almost complete. Many methods to recognise emotions from static images were tried, all meeting limited success. This points out the limits inherent to that static approach, and the audio-visual database collected enables us to move towards the more promising dynamic approach. On the more theoretical side, first versions of a theory of hybrid systems have been developed, and procedures for the confidence estimation of neural networks' performances, as well as for their on-line retraining, have been implemented. Altogether, this resulted in the production of 6 main reports:

-“Review of Artificial Intelligence and Neural Network Techniques for Mapping Signals to Symbol”

-“Review of Existing Techniques for Human Emotion Understanding and Applications in Human-Computer Interaction”

-“Development of Feature Representations from Emotionally coded Facial Signals and Speech”

-“Test Material: Format and Availability”

-“Confidence estimation and on line retraining of neural networks”

-“Neural networks for mapping features to symbols”


Keywords: Neural Networks, Artificial intelligence, Multimedia Processing


Network Home Page: