Related work: Lui, S. 2013. “A Compact Spectrum-Assisted Human Beatboxing Reinforcement Learning Tool On Smartphone”, International Conference on New Interfaces for Musical Expression (NIME), Daejeon. [25% acceptance rate]
Music is expressive and hard to be described by words. Learning music is therefore not a straightforward task especially for vocal music such as human beatboxing. People usually learn beatboxing in the traditional way of imitating audio sample without steps and instructions. Spectrogram contains a lot of information about audio, but it is too complicated to be understood in real-time. Reinforcement learning is a psychological method, which makes use of reward and/or punishment as stimulus to train the decision-making process of human. We propose a novel music learning approach based on the reinforcement learning method, which makes use of compact and easy-to-read spectrum information as visual clue to assist human beatboxing learning on smartphone. Experimental result shows that the visual information is easy to understand in real-time, which improves the effectiveness of beatboxing self-learning.
Figure 2. Game mode of the app.
We propose to use this simplified spectrogram design to learn pronunciation visually. It can be applied to learning beatbox, musical instrument playing, or speech.
We also extended this work to help people with hearing impairment to “look at” their voice, so that they can speak naturally. The paper is under review.
We are now undergoing IRB (human subject test approval) process for the experiment with people with hearing impairment. We will update this page again after IRB is approved.