MIR – Speech emotion visualization

Related work: Lee, C. Lui, S. 2014. “Visualization of time-varying joint development of pitch and dynamics for speech emotion recognition” 167th meeting of the Acoustical Society of America (ASA), Providence, Rhode Island. [url] [pdf to appear soon]


In this paper, we propose a new approach for visualizing the time-varying acoustic features for speech emotion recognition. Although the emotional state does not carry any linguistic information, it is a crucial factor that offers sentiment feedback to the listener. We propose to extract the two most prevalent acoustic features: pitch and dynamics, to identify the speech emotion of the speaker. We represent the time-varying pitch and dynamics as a trajectory in a two-dimensional feature space. Multiple trajectories are then segmented and clustered into signature patterns. This technique was successful in identifying and retargeting expressive musical performance styles. In evaluation, we use the German emotion language database. The database was created with ten professional actors (five males and five females) of ten emotionally unbiased sentences performed in six target emotions (Angry, Happy, Fear, Boredom, Sad, and Disgust). Results showed that the speech samples from the same actor of the same sentence but different emotions have dramatically different trajectory patterns. On the other hand, obvious common patterns were found among low valence emotions like Boredom and Sadness. The current study also opens future research opportunities for applying advanced pattern recognition techniques (e.g., Support Vector Machine and Neural Network) for better emotion identification.

Brief Description

We used a pitch dynamic trajectory to analysis speech emotion.

Figure 1. Angry


Figure 2. Happy


Figure 3. Neutral