Prosody has many functions in speech; e.g., cueing information structure (“Max bought a HOUSE.” vs. “MAX bought a house.”), sentence type (“Max bought a house?”), or communicative functions such as turn management (do I want to continue telling you about Max’s new house or am I done talking). This thesis investigates the prosody of yet another kind of communicative function, the expression of attitude (also called stance-taking, evaluation).

Few topics in science and philosophy have been as controversial as the nature of causality. Interestingly, the discussion becomes relatively benign, from a philosophical perspective, as soon as one agrees on a well-defined mathematical model of causality, such as Pearl’s structural causal model (SCM). Assuming that the data comes from some model within a considered class of SCMs, causal questions reduce, in principle, to epistemic questions, i.e., questions about what and how much is known about the model.

This dissertation project explores the cochlea’s intricate and nonlinear mechanisms, a crucial component of the human auditory system. The goal is to develop advanced models that represent these biological processes with greater precision and enhance our understanding of their complexities. The human auditory system exhibits notable nonlinear characteristics in various dimensions, including temporal resolution, frequency resolution, and dynamic amplification. Despite its significance, the underlying nature of this nonlinearity remains poorly understood, which has resulted in models that only capture these features qualitatively. By delving deeper into this area, the research aims to bridge the knowledge gap and contribute to creating more accurate and comprehensive representations of the cochlear function.

When speaking spontaneously, we often reduce articulatory precision, put less effort into producing flawless sentences, or utter disfluent structures. As humans, we are usually still able to decode (understand) such imperfect utterances. One reason for that is that we have been learning to deal with spoken language during our lifetime which provides us with powerful speech processing models. An automatic speech recognition (ASR) system, in contrast, is much more limited to the (finite amount of) data it was trained on. Another reason is that humans can fall back on context and the history of a conversation which helps them to evaluate the plausibility of (sequences of) words in given surroundings and thus untangle probable disambiguations. ASR systems achieve accuracies close to human performance for read speech. When it comes to recognising spontaneous speech, however, systems perform much worse. The last notable advancement in ASR has been the introduction of transformer-based systems, which deliver impressive results on established conversational speech databases. These systems were trained on huge amounts of data and make use of a wider context; therefore, one could expect that they should be able to solve the last remaining challenges on conversational speech recognition. When it comes to recognising spontaneous, unscripted conversations, however, they still yield surprisingly high word error rates. This thesis analyses the similarities and differences between transcription errors made by ASR systems and those made by human transcribers. It aims at finding where we can still learn from human performance and linguistic knowledge and encourages further research on speech from unscripted, spontaneous conversations.

Upon asking what kind of problems hearing aid users have when listening to music, most of the answers will be that some instruments are too loud, some too soft, or that it is all one big mush. The field of musical scene analysis (MSA) investigates the human perceptual ability to organize complex musical structures, such as the sound mixtures of an orchestra, into meaningful lines or streams from its individual instruments or sections. Many studies have already been performed on various MSA-tasks for humans as it bears the key to better understand music perception and help improve the enjoyment of music in hearing impaired people. For example, Siedenburg et al. (2020, 2021) demonstrated the effect of instrumentation on the ability to track instruments in artificial and natural musical mixtures. Bürgel et al. (2021) showed that lead vocals in pop music especially attract attention of the listener. Furthermore, Hake et al. (2023) presented results of MSA tests that differed depending on the participant’s level of hearing loss. However, there are still many open questions. One key question concerns the acoustical features underpinning MSA in natural music the human ear and brain use to selectively filter out single instruments/voices from sound mixtures. The goal of my PhD research is to create a signal-based model that accounts for response behavior of human listeners when asked if a sound from a target instrument can be heard in a musical mixture. I seek to analyze methods of the auditory apparatus to process music and study the ways in which they are hindered by sensorineural hearing loss. As a starting point I will be using existing models for speech perception and audio quality that use features such as linear and non-linear filterbanks, modulation filterbanks, and envelope extraction that simulate the auditory processing. Drawing from previous experiments, model performance is assessed by evaluating fit to human performance. The resulting model might then be used to test algorithms to improve selective hearing in music and provide a detailed picture on how humans perceive music.

Successful medical conversations need practice. In medical education, students practice medical conversations with trained actors, who learn their role with the help of scripts with realistic yet not real patient histories. Given restricted resources, the number of training opportunities available during medical education is limited. This PhD thesis explores how social robots can support the training of students in conducting medical conversations. Specifically, it explores how automatic speech processing technologies can identify and model aspects of effective medical interactions. One such core aspect is to identify whether there is mutual understanding between the patient and the medical professional, a not trivial task in medical conversations. For model development, this thesis uses data from interactions between students and trained actors recorded at the Medical University of Graz. These recordings, along with accompanying scripts for the actors, structured questionnaires from observing students, and metadata (e.g., illness type, age), provide rich data for analysis. One example application of this work is enabling social robots to simulate patient conversational behaviors accurately, providing realistic interaction experiences and natural feedback to medical trainees.

Voice plays a fundamental role in human communication, not only serving a functional purpose but also shaping personal identity and social interaction. Voice disorders, such as dysphonia or conditions resulting from laryngeal cancer, can severely impact the ability to communicate, often leading to social isolation and psychological burdens. In cases requiring a laryngectomy, patients rely on electro-larynx (EL) devices, which generate unnatural, robotic speech that hinders effective interaction. This research explores the potential of voice conversion (VC) models to enhance speech quality for individuals with pathological voices, bridging the gap between assistive technology and natural communication. While state-of-the-art VC models exist, few are optimized for medical applications, particularly in real-time streaming scenarios. A key focus of this work is developing low-latency, high-quality VC models tailored for pathological speech, including EL voice conversion. By improving the efficiency and adaptability of VC systems, this research aims to push the boundaries of speech synthesis and enable real-world applications that enhance communication for individuals with voice disorders.