Doctoral consortium proposal
Entangled instruments: exploring participatory digital music making environments
Enaction, participatory sense-making, participatory design
Musical instruments have traditionally been played by a single person, with individual musicians then working together to create ensembles. A collaborative dialogue emerges between the ensembled performers as they adapt their musical output in response to that of their co-players (Linson, Dobbyn and Laney 2012). Recent research has used technology to directly connect musicians in various ways, creating a coupled network of musicians, to enhance the collaborative aspect of music making whilst also retaining individualistic instrumentation.
This practice research PhD proposes to extend this research further by exploring a series of music making systems that require a high degree of direct inter-player engagement in order to produce a meaningful experience.
Early research suggests two possible methods of facilitating this level of interdependence.
-a system whose interface either directly couples players’ actions in a way that necessitates all performers to co-act in order to generate a meaningful experience, or,
-a collection of individually controlled audio processing sub-systems directly coupled through shared audio input and output.
Calling these Entangled Instruments, I envisage such systems will create an intimate moment of interaction for the players, whether experienced musicians or not.
Key research questions
What new taxonomies are needed to describe and understand participants’ actions in entangled instruments?
What are the pertinent factors in building co-operative, multiplayer, instruments that enable particular balances of entangled and individualised playing modes?
What kind of musical interactions occur in variations of participative/entangled instruments and how do the participants’ perceptions of playing differ in each?
How does the experience change when participating with another human vs complex system/synthetic agent?
Where there is a growing body of research questioning the interdependencies and social-political spaces between people (Barad 2007), the musicians and music technologists who are making collaborative musical instruments are yet to fully explore such ideas and still follow individualistic models of understanding. Whilst a few technologically mediated collaborative instruments are documented in the NIME conference proceedings , I seek to interrogate their apparent tendency to be based on individual instrumentation with the emerging interaction revolving around players being given the ability to affect another musician’s instrument whilst retaining their own. This individuality is not that surprising considering that, except for a few notable exceptions such as the, technology mediated, Tooka (Fels and Vogt 2002) historical concepts of musical instrumentation and musicianship are predominately individualistic. Thus collaborative musical instrument design has emerged from notions of playing individual instruments in a band or group, on which there is a good deal of existing research exploring its breadth, experience and benefits, rather than a mutually regulating process between coupled performers.
A musical instrument that involves inter-connected activity between several people requires a shared space for the interaction to occur. This has led to several table-top instrument projects (Barbosa, 2003), including Tonetable (Bowers, 2001) and the Reactable (Jorda 2005). The former bases the interaction around users influencing objects “floating” within a fluid mechanics simulation, where certain modes of sound and visual only emerge when more than one of the four users work in a coordinated manner. Despite the successful collaborative outcome of the project, the Tonetable creators highlight what they see as pitfalls of doing so via complex systems with users reporting confusion and the makers noting that such features could simply emerge from experimental user play rather than through considered cooperative action.
The Reactable takes a more open approach to creating a shared environment. The performer places tangible user interface objects, in the form of pictorial markers attached to wooden blocks, onto a camera and projector augmented tabletop. Individual blocks activate different sound making modules which link and affect the resulting audio path (Jorda, 2005) as they are moved around the shared table-space. This Tangible User Interface (TUI) offers an incredibly wide range of permutations for creating music, allows for more participants and crucially allows for individuals to directly change or delete another’s work. Such editability is a key feature of mutual engagement (Bryan-Kinns et al, 2007), in fact Bryan-Kinns suggests that this indicates a high level of inter-participant connection. Bryan-Kinns et al designed their Daisyphone project to allow for users to edit or delete another’s contribution to the group output, desiring to imply “an egalitarian approach to role assignment within the tool”, citing work by Dourish and Bellotti (1992) on shared workstations for this.
Such projects certainly promote aspects of participative play. They are also examples of task focussed symbolic interfaces (Gritten, 2011) where the operator has an intention (make a noise, discover what an icon ‘sounds’ like) which then transforms into action over time. They are all very much based on the idea of providing an individual with an instrument tool and then exploring how to make these tools shared.
For me it appears that creating a collaborative system by allowing individuals to act on or with other participants’ individual voices, seems to take existing user-computer interface issues and complicates the situation further. Feedback from Daisyphone for example indicates that several users were not that happy to have what they saw as their work deleted by other users, which can be seen as undermining their perceived role. It’s not surprising that participants take such a position when engaging with a collaborative system based on an individualistic metaphor, any emerging co-voice is constrained at a fundamental design level.
Despite such complexities an exciting new dynamic can be seen to emerge between inter-connected performers. I believe we can see a hint of it by returning to the Tonetable for a moment. The fact that the authors report the participants’ learning of the collaborative features through play suggests two things, firstly the collaborative features were designed in as rewards and not as a core functionality (as discussed above) but secondly, if the participants recognise a collaborative feature (which is implied in the paper, we don’t know if they then work together to recreate at collaborative effect mentioned) then there must be something interesting happening to connect them as individuals.
Jorda talks about adding more connections between players to increase collaboration but doesn’t discuss whether it is the manner of these connections that is key. The Brain Dead Ensemble appears to take a unique approach to interconnections by feeding the audio produced back into each other’s resonant instruments causing a group shared interface to emerge alongside and stemming directly from the auto-instruments. This use of the shared feedback of complex information seems an effective way of facilitating co-voice and warrants further study as it may bypass my concern that facilitating the co-voice element by increasing the number connections between performing individuals ties the maker into creating an ever more complex system. I also feel other metaphors exist that place a performative interface designed with multiple partnerships at its core. However co-voice is realised, adding dialogue facilitating inter-connections creates a space for sense-making (De Jaegher and Di Paolo, 2007). I contend that the exploration of this emerging space through an entangled instrument will lead to a differing experience to that of playing an individual instrument alongside others, however in order to consider design narratives to facilitate this I will also need to better understand their effect on the players’ mental state.
Herber (2010) argues that engagement with interactive artwork leads to a new level of consciousness and cites Ascott, who argues that such artworks are something to be experienced and are based upon the 'art' being brought forth (poietic) and so the participant performer should be referred to as the poiesist - the bringer forth. Ascott is specifically referring to the way that the public engaging with interactive artwork shift from a purely reflective role to one of including production, albeit within the narrow constraints designed by the artwork’s instigator. Elements of an entangled instrument reflect an interactive artwork in that the poietic musician is also experiencing the additional collaborative affordances of the interface they are playing. Theorists agree that such experience leads to enhanced levels of consciousness though disagree over specifics that pertain to their area of interest such as mutual engagement (Bryan-Kinns, 2007) and transformative action (John-Steiner 2000) and flow (Csikszentmihalyi, 1990). Comparing frameworks seems to reveal a conflict between the positions that starts with the individual and then looks to facilitate collaboration by adding more connections and systemic functions; and ideas of intra-action (Barad 2021) and performative sense-making (De Jaegher 2007) which start with a fuzzier sense of entanglement.
This research seeks to take the learning from existing collaborative music making systems, and create new distinct environments that require direct mutual action of multiple participants, much in the way several people participate in the process of manoeuvring a piano, say, up a spiral staircase. I am interested in seeking new design metaphors to enable this participation and identifying the nuances of any subsequent effect on the playing experience. I contend that there is a world-view that is implemented into the design of such entangled instruments that in turn affects the players’ perception of each other and the instrument that they are engaging in. Considering the intra-player relationships that entangled instruments afford, I particularly seek to question to what extent their playing is essentially an enactive experience and to what degree the resulting participation refocuses the concept of musical instruments from being a tool for direct effect to that of exploration and understanding and, inevitably, affecting the instrument world that emerges from play.
Practice research lies at the heart of this project, following a process of designing, making, testing and refining. However, the public facing design and test stages are deliberately constructed as a participatory process where participants are invited to envision links between participative actions and what sounds they might associate with the movements.
I feel that engaging with the consortium as an early-stage doctoral researcher would provide me with an excellent opportunity to build professional links that can sustain and deepen my research over the coming years. The NIME conference is perfectly timed for this as it comes at the end of an intensive first year where I have built and tested some prototypes and so am at a reflective point where input and discussion with experienced peers will be very valuable to direct the subsequent research. I am a mature student with 20 years professional experience making and performing NIMEs and creatives technologies I believe that I would bring a sensitivity and enquiring attitude to any collegiate discussions.
Barad, Karen Michelle. 2007. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning. Durham: Duke University Press.
Barbosa, A. 2003. ‘Displaced SoundScapes A Survey of Network Systems for Music and Sonic Art Creation’. Leonardo Music Journal 13.
Blaine, Tina, and Sidney Fels. 2003. ‘Contexts of Collaborative Musical Experiences’. In Proceedings of the 2003 Conference on New Interfaces for Musical Expression, 129–34. NIME ’03. Montreal, Quebec, Canada: National University of Singapore.
Bowers, John, Simon Bowen, and Tim Shaw. 2016. ‘Many Makings: Entangling Publics, Participation and Things in a Complex Collaborative Context’. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems, 1246–57. Brisbane QLD Australia: ACM. https://doi.org/10.1145/2901790.2901883.
Bryan-Kinns, Nick, Patrick GT Healey, and Joe Leach. 2007. ‘Exploring Mutual Engagement in Creative Collaborations’. In Proceedings of the 6th ACM SIGCHI Conference on Creativity & Cognition, 223–32. ACM.
De Jaegher, Hanne, and Ezequiel Di Paolo. 2007. ‘Participatory Sense-Making: An Enactive Approach to Social Cognition’. Phenomenology and the Cognitive Sciences 6 (4): 485–507. https://doi.org/10.1007/s11097-007-9076-9.
Evens, Aden. 2005. Sound Ideas: Music, Machines, and Experience. Minneapolis: University of Minnesota Press.
Fels, Sidney, and Florian Vogt. 2002. ‘Tooka: Explorations of Two Person Instruments’, June. Gritten, Anthony. n.d. ‘Instrumental Technology’. Accessed 18 December 2017. https://www.academia.edu/3320894/Instrumental_Technology.
Herber, Norbert. 2010. ‘Amergent Music:Behaviour and Becomming in Technoetic and Mediat Arts’. PhD, University of Plymouth. http://www.x-tet.com/PDFs/herber-amergent-music.pdf.
Hurtado, Enrique, Thor Magnusson, and Josu Rekalde. 2020. ‘Digitizing the Txalaparta: Computer-Based Study of a Traditional Practice’. Computer Music Journal 43 (2–3): 125–41. https://doi.org/10.1162/comj_a_00522.
Jordà, Sergi. 2005. ‘Multi-User Instruments: Models, Examples and Promises’. In Proceedings of the 2005 Conference on New Interfaces for Musical Expression, 23–26. National University of Singapore.
Kocaballi, A. Baki, Petra Gemeinboeck, and Rob Saunders. 2010. ‘Investigating the Potential for Shared Agency Using Enactive Interfaces’. Sydney, Australia, June 1. https://zenodo.org/record/1177829#.YFR3UEj7SL4.
Weinberg, Gil. 2005. ‘Interconnected Musical Networks: Toward a Theoretical Framework’. Computer Music Journal 29 (2): 23–39. https://doi.org/10.1162/0148926054094350.