Designing NIME for Novice Musicians
NIME, doctoral, consortium, interface, linear, logarithmic, transposition
The aim of this proposed study is to understand
whether a particular arrangement of a musical interface has an impact on skill acquisition;
if so what kind of musical interface design can facilitate faster and smoother skill acquisition for non-professionals and
what aspect of a particular design of a musical interface improves or diminishes the efficiency of learning.
This involves investigating different learning curves for different musical interface designs, both traditional and novel. How long does a player need to acquire a specific skill? How fast and how much does a specific skill fade if not regularly practiced? I will investigate, catalogue, and apply the taxonomy of difficulties encountered when learning a musical interface at different stages of skill acquisition(Dreyfus and Dreyfus, 1980 , Fitts and Posner, 1967 , Tadlock and Stone, 2005 ).
There are many aspects to learning a musical instrument for example
background in music education,
the preferred music genre and
personalities and commitment.
To confine the scope of the study, I am mainly going to focus on the acquisition of mechanical skills and especially in transposition and improvisation. For my interest in the democratization of the making of music, the study will exclude professionally trained musicians. Professionally trained musicians usually spend so much effort in the musical instrument that no matter how the musical interface is designed, they achieve virtuosity. The focus of the study will be on amateur musicians on their ability to jam or compose with their friends in a casual or informal setting.
Learning a musical instrument is not an easy task, this holds especially true for people without childhood musical education. This study comprises skill acquisition and musical interface design, especially with regards to complexity. The literature review examines the existing model of skill acquisition and existing design and evaluations of both traditional and altered/augmented/novel musical interface.
The effort to make musical instruments for amateur or novice musicians has been passionately pursued by instrument makers through the community of New Interfaces for Musical Expression(NIME), Human Computer Interaction(HCI) and Crowdfunding (McPherson et al., 2019 ). This review finds that although many instruments were built, most evaluations were quite mundane. For the ones with evaluations, the evaluations were not made against alternative forms of interface.
Take some examples through crowdfunding, most of the novel interfaces made are MIDI controllers (McPherson et al., 2019 ). From this group, most of the novel interfaces, although claimed to be designed for novice musicians, are quite complex. For the few designs that have simplified the interface, the designs took a lot away from the musical expression meaning they are more like a musical toy than full instruments with comprehensive expression. None of the researchers have conducted an extensive evaluation of the efficiency of the interfaces in comparison to their traditional counterpart.
NIME have been trying to design novel interface since the beginning of the community (Blaine and Fels, 2003  , Miletto et al., 2011 , McPherson et al., 2019 ).Most of these works focus on talking about the design, only a few of the works have evaluation. Among these works, an augmented flute for novice musicians (Heller et al. ) was evaluated with expert panels. However, the evaluation is not done to compare the augmented interface with the original interface, but rather to calibrate the interface. A novel actuated percussion system was evaluated only in performance (van Troyer, 2017 ), meaning it is used by professional musicians. A novel digital interface where two novice musicians take turns to generate music (Hansen et al. ) did a comparative evaluation but only with two alternative novel interface designs. The novel musical interface is rhythmic rather than melodic which means the interface overall is not as complex as traditional acoustic musical instruments.
Novel musical interface design in HCI community mainly focus on gesture (Lyu et al., 2017 , Chuang et al., 2015 ), interactivity (Schnell et al., 2015 ), and collaboration (Bengler and Bryan-Kinns, 2013, , Zamorano, 2013, ). All of these works use a digital interface, and the evaluation mainly focuses on how the users interact with the designed interface without comparing them to another existing interface design.
Accessible Digital Musical Instruments (ADMIs) uses multimodal input and feedbacks to accommodate musicians with special needs (Frid, 2019 , Frid and Ilsar, 2021). This body of works focus on the accessibility of the interface and in exchange these designs excessively diminish the artistic expression of the instrument which make these interfaces more like a musical toys than musical instruments.
Lamb and Robertson classified musical interface into discrete and continuous, and designed a novel musical instrument and they tried to combine the two type of musical interface into a novel musical interface named seaboard (Lamb and Robertson 2011 ). This work also focuses on the richness of musical expression but not the ease of adoption like ease of transposition or improvisation.
In the context of design theory, Janlert and Stolterman identified and proposed the sacrifice, eliminate, confine, hide, dilute and shape framework for dealing with complexity in interaction and interactivity(Janlert and Stolterman, 2017  6:78-81). In the comparison of different musical interface designs, this framework could provide identification and speculation of why a certain design feature of an instrument has a different level of complexity than the alternative design.
So to summarize, new musical interface designs either mainly focus on artistic expressions which are more targeting professional musicians, or excessively diminishes the expressions to focus on education use or accessibility. There are great insights for interface design to be summarized and generalized from existing traditional and novel musical interfaces. There are also great design opportunities in adjustments to existing musical interface design which facilitates informal learning that is more expressive than a music toy but not targeting professional musician training.
To understand the complexity associated with different stages of skill acquisition, I need to investigate various difficulties an instrument player faces in learning the instrument. I need to:
Identify how specific organizations of musical interfaces influence the complexity of learning.
Understand the cognitive load in each stage of skill acquisition for players without childhood musical education.
Identify intervention/alterations that may reduce the complexity of a certain learning task.
So a mixed method will be employed through the whole process of the doctoral research.
For identifying the organization of the existing musical instrument, both tradition and novel, similar to Lamb and Robertson’s taxonomy of discrete and continuous. I am going to identify more taxonomies of musical interface. As an example, in analysis of traditional keyboard interface, it can be categorized to “regular”(Isomorphic keyboard like Janko) and “irregular“(Traditional piano keyboard)
It can also be categorized to “linear”(Keyboard) and “logarithmic”(String/Theremin)
For understanding cognitive load and interventions possible, Research through Design (RtD) (Zimmerman et al., 2007, Zimmerman et al., 2010, Zimmerman and Forlizzi, 2014 ) is going to be used. Modifications on existing musical interfaces or novel interfaces will be prototyped and tested on players without prior experience in the genre of instrument. For example, I have attempted to propose an altered interface of Bohem for electric wind instruments(EWI), for easing the learning of transposition.
Although there’s a similar design of splitting keys — the Yewi 3.0 by Prof. Yann LeCun. However the design focuses more on the richness of expression(note bending, and chromatic scale) rather than the ease of transposing.
Human-centered methods of co-design workshops will be used for ideation of musical interface for amateurs. Usability workshops will also be employed to evaluate the artifacts designed using the RtD method. These workshops can identify the effectiveness, generate insights for new iterations, and ultimately identify the guidelines of what are the characteristics that benefit amateur musicians.
There are two major parts of research outcomes that I expect to get.
By analyzing existing musical interfaces, taxonomies of musical interfaces focused on different aspects will be listed. Based on where a certain kind of musical interface sits in these different categories, I will identify the characteristics that make certain acquisition of mechanical skills difficult and why the characteristics make it difficult.
By exploring a modified or novel musical interface through design artifact and user testing, I expect to define design guidelines for designing musical interfaces that enables a flatter learning curve and yet does not go in the realm of musical toy.
From the Doctoral Consortium, I expect to widen my view by looking at peer’s work on novel musical interfaces. I also expect to get professional feedback and critiques on the method I propose.
I am doing my PhD in the context of design oriented HCI program. With advice from Professors both in music school and from school of informatics. I hope to make sure that my work is informed by the NIME community in the service of transdisciplinary gaze.
 Stuart E Dreyfus and Hubert L Dreyfus. 1980. A five-stage model of the mental activities involved in directed skill acquisition. California Univ Berkeley Operations Research Center.
 Paul M Fitts and Michael I Posner. 1967. Human performance. Brooks/Cole.
 D. Tadlock and R. Stone. 2005. Read Right: Coaching Your Child to Excellence in Reading. McGraw-Hill Education. Retrieved from https://books.google.com/books?id=tuE_1BuSws0C
 Andrew McPherson, Fabio Morreale, and Jacob Harrison. 2019. Musical Instruments for Novices: Comparing NIME, HCI and Crowdfunding Approaches. In New Directions in Music and Human-Computer Interaction, Simon Holland, Tom Mudd, Katie Wilkie-McKenna, Andrew McPherson and Marcelo M. Wanderley (eds.). Springer International Publishing, Cham, 179–212. DOI:https://doi.org/10.1007/978-3-319-92069-6_12
 Tina Blaine and Sidney Fels. 2003. 2003: Contexts of Collaborative Musical Experiences. In A NIME Reader, Alexander Refsum Jensenius and Michael J. Lyons (eds.). Springer International Publishing, Cham, 71–87. DOI:https://doi.org/10.1007/978-3-319-47214-0_6
 Dr Evandro Manara Miletto, Marcelo Soares Pimenta, François Bouchet, Jean-Paul Sansonnet, and Damián Keller. 2011. Principles for Music Creation by Novices in Networked Music Environments. Journal of New Music Research 40, 3 (2011), 205–216. DOI:https://doi.org/10.1080/09298215.2011.603832
 Florian Heller, Irene Meying Cheung Ruiz, and Jan Borchers. An Augmented Flute for Beginners. 4.
 Akito van Troyer. 2017. MM-RT: A Tabletop Musical Instrument for Musical Wonderers. In Proceedings of the International Conference on New Interfaces for Musical Expression, Aalborg University Copenhagen, Copenhagen, Denmark, 186–191. DOI:https://doi.org/10.5281/zenodo.1176215
 Anne-Marie Skriver Hansen, Hans Jørgen Andersen, and Pirkko Raudaskoski. Two Shared Rapid Turn Taking Sound Interfaces for Novices. 4.
 Fei Lyu, Feng Tian, Wenxin Feng, Xiang Cao, Xiaolong (Luke) Zhang, Guozhong Dai, and Hongan Wang. 2017. EnseWing: Creating an Instrumental Ensemble Playing Experience for Children with Limited Music Training. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), Association for Computing Machinery, New York, NY, USA, 4326–4330. DOI:https://doi.org/10.1145/3025453.3025583
 Galen Chuang, Shelley Wang, Sara Burns, and Orit Shaer. 2015. EmotiSphere: From Emotion to Music. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’15), Association for Computing Machinery, New York, NY, USA, 599–602. DOI:https://doi.org/10.1145/2677199.2688809
 Norbert Schnell, Sébastien Robaszkiewicz, Frederic Bevilacqua, and Diemo Schwarz. 2015. Collective Sound Checks: Exploring Intertwined Sonic and Social Affordances of Mobile Web Applications. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, ACM, Stanford California USA, 685–690. DOI:https://doi.org/10.1145/2677199.2688808
 Ben Bengler and Nick Bryan-Kinns. 2013. Designing collaborative musical experiences for broad audiences. In Proceedings of the 9th ACM Conference on Creativity & Cognition, ACM, Sydney Australia, 234–242. DOI:https://doi.org/10.1145/2466627.2466633
 Francisco Zamorano. 2013. SimpleTones: a collaborative sound controller system for non-musicians. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA ’13, ACM Press, Paris, France, 3155. DOI:https://doi.org/10.1145/2468356.2479634
 Emma Frid. 2019. Accessible Digital Musical Instruments—A Review of Musical Interfaces in Inclusive Music Practice. MTI 3, 3 (July 2019), 57. DOI:https://doi.org/10.3390/mti3030057
 Emma Frid and Alon Ilsar. 2021. Reimagining (Accessible) Digital Musical Instruments: A Survey on Electronic Music-Making Tools. In International Conference on New Interfaces for Musical Expression (NIME), Shanghai, China. Retrieved from https://hal.archives-ouvertes.fr/hal-03250922
 Roland Lamb and Andrew Robertson. 2011. Seaboard: A New Piano Keyboard-related Interface Combining Discrete and Continuous Control. In NIME, Citeseer, 503–506.
 Lars-Erik Janlert and Erik Stolterman. 2017. Things that keep us busy: The elements of interaction. MIT Press.
 John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, San Jose California USA, 493–502. DOI:https://doi.org/10.1145/1240624.1240704
 John Zimmerman, Erik Stolterman, and Jodi Forlizzi. An Analysis and Critique of Research through Design: towards a formalization of a research approach. 10.
 John Zimmerman and Jodi Forlizzi. 2014. Research Through Design in HCI. In Ways of Knowing in HCI, Judith S. Olson and Wendy A. Kellogg (eds.). Springer New York, New York, NY, 167–189. DOI:https://doi.org/10.1007/978-1-4939-0378-8_8