Abstract

The NIME community has proposed a variety of interfaces that connect making music and education. This paper reviews current literature, proposes a method for developing educational NIMEs, and reflects on a way to manifest computational thinking through music computing. A case study is presented and discussed in which a programmable mechatronics educational NIME and a virtual simulation of the NIME offered as a web application were developed.

Author Keywords

Computational Thinking, Web Application, Educational NIMEs

CCS Concepts

•Applied Computing Arts and Humanities Sound and Music Computing

•Applied Computing Education Interactive Learning Environments

Introduction

Computational thinking is a construct that has been introduced as a potential foundation for formal education in the public school system. However, for the construct to be effective in this setting, it needs to be supported by well-defined curricula [1]. Wing [2] presents computational thinking as a blend of mathematical and engineering thought processes that drive the design of physical or cyber-physical solutions. This paper defines computational thinking as problem-solving using computers.

Educational NIMEs have have a history of addressing the lack of appropriate curriculum with two distinct design approaches. One approach is to design a learning environment for the development of electronic musical instruments that use custom hardware [3][4][5]. Another approach is to design virtual musical instruments that run on common hardware [6][7][8]. Educational NIMEs that depend on custom hardware constrain the ability for distance learning and internet interventions, unless learners have the necessary hardware platform. The ability for practicing electronic system design with virtual educational NIMEs is constrained by learners’ access to physical computer platforms and controllers that can be integrated in customized hardware designs.

A blend of these two design approaches can reduce each other’s constraints. Therefore, by developing and documenting the design of physical NIMEs, while also developing a virtual model of the NIME as a web service, learners could have access to the web service, learning resources and design documentation online, as well as to the custom hardware during in-person workshops. Such a design approach can promote distance learning [9] and internet interventions, while musical expression adds a personalized experience through self-expression. For example, during the COVID-19 pandemic there was an increased need for distance learning. Learning resources for in-person teaching that were ready to be used in distance learning mattered for teaching continuity across educational tiers. It is not a matter of developing virtual rather than physical NIMEs for educational purposes. Developing learning resources available online and virtual simulations of physical NIMEs as web services can make a difference in reaching a wider audience than by in-person learning activities with any kind of educational NIMEs.

Educational NIMEs

The existing research suggests that the practice of computational thinking depends on engineering limitations. Leeuw and Tamminga report on pedagogical practices of their school, which includes system design as an introduction to electronics; however, they point out the lack of organized laboratories for electronics development [10]. Xambó et al. discuss a curriculum for physical computing, which provides physical constraints and reveals that custom hardware and design complexity is a limiting factor in designing NIME-related curricula appropriate for distance learning [11].

The prior work also offers examples of using programming for sound production in order to learn through practice. Kaneko discusses the computational thinking behind turning a touchscreen into an interface for musical expression that makes it easy for its user to navigate modes and scales during sound production [12]. Ford and Nash discuss an iterative design method for creating music using a visual programming language. The creative process is broken down to visual components, so that learners can practice computational thinking through their experimentation with the available components [13].

It is common for educational NIMEs to focus on playfulness, which has been suggested as an alternative to performing and with the purpose of learning through enjoyable practice. Marquez-Borbon discusses the development of environments for collaborative learning based on group playing with NIMEs [14]. Tomás describes a number of different pedagogical methods for learning based on playing with or designing NIMEs [15]. The discussed literature seems to reinforce the previously stated points on educational NIMEs:

• The development of custom and potentially fragile hardware limits the widespread use of NIMEs, yet it promotes computational thinking for the development of physical interfaces, and

• In contrast, the development of virtual NIMEs promotes the widespread use of NIMEs, yet it demotes computational thinking with constraints in developing physical interfaces.

Case Study

The following case study focuses on the advantages of combining both physical and virtual NIMEs. A method for educational NIMEs is proposed for the development of learning resources that can be shared between in-person and remote learning. A programmable mechanical glockenspiel was developed on a Raspberry Pi platform with an Analog-to-Digital Converter (ADC) add-on to read analog sensors and drive a Direct Current (DC) motor using a Pulse-Width-Modulation (PWM) electrical control signal and a solenoid electromagnet. The educational NIME focuses on playfulness (experimentation) rather than performance (optimization). It is programmable via the Pd-L2Ork software (https://github.com/pd-l2ork/) [16] and its goal is to practice computational thinking by encouraging users to control the actuators with a goal of sound and music production [17]. Additionally, as a response to the COVID-19 pandemic, a virtual glockienspiel was developed to simulate the functionality of the mechanical glockenspiel. The virtual model is accessible via a web browser and communicates with a server over a web socket connection to send and receive data. Figure 1 shows both the physical and virtual glockenspiels.

The physical educational instrument was initially developed for in-person workshops with children in secondary education. The goal was to get children exposed to musical computing and electronics on a visual programming platform which promotes easy prototyping via programmable virtual objects. However, during the COVID-19 pandemic distance learning and online research were more feasible than in-person learning and research. The virtual simulation of the physical instrument was developed as a programmable resource accessible via web browser to close the gap of having to have the custom hardware in order to use the available learning resources.

Figure 1: Educational NIME: Physical NIME (top), Virtual model (bottom)

The Internet of Things (IoT) can enable physical and virtual NIMEs to exchange data and allows for multiple NIMEs to be networked so that one NIME can control another. Figure 2 shows the proposed design method for educational NIMEs and Figure 3 shows an example of a learning resource for the practice of computational thinking on the topic of types of measurement (nominal, ordinal, interval, and ratio).

Figure 2: Proposed Design Method for Educational NIMEs.

Figure 3: Types of Measurement: An Example using Scales of Sound-Pitch.

A study for the evaluation of differences between using the physical and the virtual instrument for the practice of computational thinking has not yet been conducted. An ongoing relevant study is looking at the differences in participants’ affective states of Enjoyment, Excitement [18][19], and Motivation, [20][21], between 3 different learning modalities for the practice of computational thinking through visual programming: 1) Visual programming; 2) Visual programming for sound production, and 3) Visual programming for sound production including the virtual educational instrument (Figure 1, top-right). It is hypothesized that when the modality of the learning activity requires more cognitive effort, the participants’ affective states will decrease.

Discussion

Experiential learning promotes positive affective states during instructional activities [22]. Educational NIMEs can be used for experiential learning due to their performative character that enables creative scenarios with immediate multisensory feedback. Virtual educational NIMEs that can be accessed via web browser or offered as mobile applications for common platforms can simulate physical models in order to increase access, including learning scenarios over distance.

An advantage of developing virtual simulation models for physical NIMEs is that virtual models can be presented to learners with variable degrees of simplification and in juxtaposition to models with different design principles. Virtual model complexity variability can be used to target different age levels of learners, while the juxtaposition of virtual models with different design principles can enable a comparison of different designs and their efficacy in delivering desired learning outcomes.

The ability of sharing learning resources between physical and virtual NIMEs for in-person and distance learning activities respectively, can enable the development of cohesive groups of learning activities for learners to choose from. More importantly, cognitively effortful learning activities decrease motivation [23], thus it is important for learners to be able to find activities that they enjoy and are more excited from in order to maintain higher levels of academic motivation.

Conclusion

A design method has been proposed for the development of educational NIMEs to broaden the use of educational NIMEs and to pursue both in-person and distance learning using virtual counterparts to the physical NIMEs. Physical computing is a way to provide physical constraints for the practice of computational thinking. However, educational activities that require custom hardware are less suitable for distance learning activities. A method for increasing the suitability of learning activities based on physical NIMEs for distance learning is to develop a virtual simulation model that shares learning resources with the physical NIME. The development of virtual models that simulate physical educational NIMEs alleviates the requirement of having access to custom platforms for the delivery of learning activities. In the context of computational thinking practice, programmable NIMEs allow for testing, observation, reflection on their operation, and imagination of alternative designs. Physical educational NIMEs can benefit from programming frameworks that run on web browser by sharing learning resources with virtual simulation models that are more suitable for distance or other kind of learning activities.

Acknowledgments

The Institute for Creativity, Arts, and Technology (ICAT) at Virginia Tech for partial support.

Compliance with Ethical Standards

No potential conflict of interest is known. Partial funding for the fabrication of mechatronic units was received. This paper does not report any data from human subject research. No animals were harmed.