Multimodal Datasets for Studying Expert Performances of Musical Scores

Abstract

In many musical styles, performing a piece of music means to produce an 'interpretation' of a score. This interpretation involves performers manipulating musical parameters such as timing, dynamics, timbre, and pitch to communicate their artistic conception of the piece, often to an audience. Much previous research into musical interpretations has examined aspects of expressive performance strategies. However, these studies have largely focused solely on the sounds produced in the performance, investigating players' manipulation of musical parameters but little of the performance's broader context and impact. Multimodal datasets, which contain multiple diverse data types offering distinct perspectives on the musical performance (e.g. audio, Musical Instrument Digital Interface, video, motion capture, physiological data), can support more holistic cross- and interdisciplinary study of performers' interpretative decision-making and its effects on audiences. We propose a taxonomy of modalities relevant to study of musicians' interpretations of musical scores. These modalities are distinct facets of the performance or its context through which the performance and musical interpretation can be analysed (e.g. 'venue acoustics', 'performer movements', 'performance sound'). We use this taxonomy to systematically review relevant open-access multimodal datasets and the modalities they support. Underrepresented modalities are then highlighted, along with practical suggestions for including data that support these modalities in future datasets. We next examine key challenges of reporting and working with multimodal datasets, emphasising the need for standardisation of data reporting and reliable options for data storage and access. Finally, we summarise the broader interdisciplinary applications of these datasets in artificial intelligence and performance research.

Authors

• Katelyn Emerson