Open Access

Tobias Baur

• The research area of Social Signal Processing paves the way for conversational companions, such as virtual agents or social robots, to become aware of nuances in our behaviours and implicit messages that come along with them. For machines to understand and interpret such behavioural cues, the state-of-the-art procedure is the application of various machine learning techniques. In many ML tasks, statistical models are trained on a large amount of annotated samples and an algorithm aims to match patterns that represent specific classes or values. ML tehniques, such as artificial neural networks, nowadays do pretty well in mapping and even identifying low level features to a specific recognition problem. A large drawback here is that the decisions they are making are not comprehensible and understandable to humans and that their assumptions are often wrong in changing contexts. Therefore a new research direction -"eXplainable Artificial Intelligence" (XAI)- identified the need of AIThe research area of Social Signal Processing paves the way for conversational companions, such as virtual agents or social robots, to become aware of nuances in our behaviours and implicit messages that come along with them. For machines to understand and interpret such behavioural cues, the state-of-the-art procedure is the application of various machine learning techniques. In many ML tasks, statistical models are trained on a large amount of annotated samples and an algorithm aims to match patterns that represent specific classes or values. ML tehniques, such as artificial neural networks, nowadays do pretty well in mapping and even identifying low level features to a specific recognition problem. A large drawback here is that the decisions they are making are not comprehensible and understandable to humans and that their assumptions are often wrong in changing contexts. Therefore a new research direction -"eXplainable Artificial Intelligence" (XAI)- identified the need of AI systems to be able to explain their decisions. In this thesis we investigate strategies to make the recognition and interpretation of complex social signals more transparent and explore ways to empower the human in the machine learning loop. To gain a better understanding of how humans interpret social cues, we first introduce an overview on results of behavioural psychology. We then describe the creation of various multi-person and muli-modal corpora in varying contexts that aim to induce multiple aspects of such behaviours. Next, we briefly introduce common techniques used in the area of social signal processing and machine learning. To successfully annotate and manage large continuous databases, a novel tool, named NOVA is presented. It allows to distribute the annotation task on multiple labellers and supports various types of annotations. NOVA further allows to take advantage of ML techniques already during the annotation process (a concept named cooperative machine learning). By employing CML, data is annotated simultaneously with the machine, which speeds up the annotation process and gives a more transparent idea of a machine's decisions. For inferring more complex behaviours, such as a person's conversational engagement or emotion regulation strategies, an approach is introduced that considers the predictions of multiple social cue recognisers and various types of context information. Finally, an outlook on future research directions is given.…