Skip to main content
SearchLoginLogin or Signup

NIME Scores: a Systematic Review of How Scores Have Shaped Performance Ecologies in NIME

This paper investigates how scores have been used by the NIME community. In particular, we primarily looked at the role that scores play in performance ecologies, using ARCAA, a recent framework created to investigate artifact ecologies in computer music performances.

Published onApr 29, 2021
NIME Scores: a Systematic Review of How Scores Have Shaped Performance Ecologies in NIME
·

Abstract

This paper investigates how the concept of score has been used in the NIME community. To this end, we performed a systematic literature review of the NIME proceedings, analyzing papers in which scores play a central role. We analyzed the score not as an object per se but in relation to the users and the interactive system(s). In other words, we primarily looked at the role that scores play in the performance ecology. For this reason, to analyze the papers, we relied on ARCAA, a recent framework created to investigate artifact ecologies in computer music performances. Using the framework, we created a scheme for each paper and clustered the papers according to similarities. Our analysis produced five main categories that we present and discuss in relation to literature about musical scores.

Author Keywords

Score, Performance Ecology, Literature Review, Taxonomy

CCS Concepts

Applied computing Arts and humanities →Sound and music computing;

Introduction

Musical scores have impacted electronic musicians' reasoning, at the same time, electronic musicians appropriated the conception of scores, bending it to the new languages and technologies. As we will detail in this paper, such relation reverberated and further evolved in the NIME practice.

Scores can be investigated from many different perspectives; for instance, what type of score it is, what notation it adopts, or which music form it assumes. These perspectives could offer interesting reflections, and scholars have investigated scores from these angles. However, scores are also central in music performance ecologies, which are the complex net of interactions occurring in a performance [1]. In such ecologies, scores act as elements that connect people, music, and instruments; usually, different people: those who write scores (composers) and those who read scores (performers). In this perspective, scores can be understood as an interface between different people and the music; and are pivotal in determining relations within performance ecologies. Within music technology, the idea of performance ecology has proved to be useful to look at the roles and interactions that occur around music technology [1].

Given the within NIME focus on interaction, we decided to investigate scores within NIME from this perspective. Therefore, this paper aims to understand what types of relationships exist among musicians, technology, pieces, and instruments in works presented at NIME in which score was central. We summarized this objective in the following research question: 

What kind of performance ecologies occur when scores have a central role in NIME systems?

To this end, we present a systematic review of papers presented at NIME in which the notion of score was somehow central. We analyzed the ecologies implied by the systems presented in these NIME papers using ARCAA [2], a framework recently developed to support the understanding of ecologies in music performances. We then clustered these articles according to similarities. This analysis produced five categories that we suggest can be used as a taxonomy to classify score adoption in NIME. As a complementary contribution, this analysis also highlights the new different types and conceptions of score that correspond to such ecologies.

Background

Score - an overview

Although the conception of score largely evolved in the last century, especially in NIME, scores are musical tools that emerged in Europe and whose adoption and evolution are intimately connected with a western approach to musical practice. Scores have played a pivotal role in the unfolding of the Western musical tradition. A remarkable exemplification of this fact is provided by the musicologist Taruskin who marked the beginning of his History of Western Music by analyzing the first forms of musical inscriptions [3]. Scores also consolidated their central role in music practice during the Renaissance, thanks to the invention of music printing [4]: by facilitating the circulation of music, scores consolidated the role of composers as authors.

Over the centuries, a standardized form of notation consolidated and allowed for a relatively straightforward interpretation of music; consequently, a piece could be re-performed at any time, and performers could rely on a relatively accurate description of the final musical form. From this angle, we support that a score can be intended both as a communication tool that acts as a human-human interface (composer-performer) and as an artistic object that encodes an artwork's characteristics. The latter aspect is reinforced by how the word score is used in the classic music jargon. For instance, in an orchestral piece, the score is the book that comprises all the individual parts, which normally is read by the conductor, while instrumentalists read from their parts, not from the score.

As composers started to expand their music vocabulary in the last century, the notion of score evolved. In 1963, the composer Cardew wrote [5]:

"a composer who hears sound will try to find a notation for sounds. One who has ideas will find one that expresses his ideas, leaving their interpretation free".

This trend is particularly evident in aleatoric music, where scores no longer encoded the pieces in their final form. Instead, composers created scores that defined sets of possibilities and asked performers to choose among them. Italian philosopher Umberto Eco proposed the term open work to define the indeterminacy in these works [6]. In these pieces, the aesthetic identity defined by the scores does not correspond with the final form of the music; rather, it encodes the rules to create such a form. In many cases, this indeterminacy relied on graphical non-standardized elements [7].

Score and Music Technology

Graphic notations and aleatoric avant-garde practices have substantially impacted the development of music technology and digital systems. For instance, graphical forms of notation have been applied to screen scores as a form of "new media manuscript", as defined by Hope and Vickery [8], in which musical scores are displayed in real-time, often in a dynamic form, on digital screens (e.g. [9],[10],[11]). Similarly, Magnusson argued that live coding is a form of music notation and connected it with the evolution of music scores, including the avant-garde graphic experimentations [12]. Aleatoric scores have also been used to design VR installations [13].

Within the electronic and computer music domain, compositional processes and the creation of music technology have often overlapped. For instance, the composer Lucier, reflecting on some of the work with the collective of composers/makers/performers Sonic Art Union, formulated that "the scores were inherent to the circuits" [14]. This approach was crucial in transforming the conception, role, and function that scores play in the music technology practice. In his book, Magnusson analyzes how instruments often embed theoretical music models, determining the type of music implied by the instrument itself [15].

Within NIME, the overlapping between composition and technology is discussed since the first editions of the conference: in 2002, Schnell and Battier proposed the composed instrument [16]: an interactive musical artifact that embodies the notion of the score. Discussing them, Magnusson argued that "the distinction often blurs between instrument and composition on the one hand, and performance and composition on the other" [17]. This idea can also be traced beyond Cook's guideline "make a piece, not an instrument" [18], as it implies that a new Digital Musical Instrument (DMI) should be designed aiming at a specific piece.

Discussing the overlapping of instruments and scores, Maestri reflected on how this tendency affects the ecology of the various actors involved in music performances [19]:

"the very communicative chain and roles between instrument-makers, composers, performers, and computer-music designers are to be genuinely rethought as cycles of synergy rather than linear models" .

Ecology of Music Performances

In the computer music discourse, the idea of a performance ecology indicates a complex set of agents, including makers, performers, composers, instruments, and the environment. For instance, Gurevich and Trevino, discussing the ecology of music creation, focused on the "relationships between composers, performers, and listeners as a part of a system", also considering history, genre, and context [20]. Similarly, Waters speaks about performance ecosystems to study the interactions between performers, instruments, and environment [1]. This perspective was recently further developed by Masu et al., who, by borrowing the concept of artifact ecology from HCI scholars (developed ARCAA, a framework to analyze digital performance ecologies initially proposed in [2], and further discussed in [21]; see [22] for a reference on artifact ecologies).

ARCAA [2] is based on the structure of MINUET [23], a previous design framework presented at NIME in 2014. The purpose of ARCAA is to help designers, musicians, or composers to understand the relations between the various human actors and artifacts that occur in a performative situation. The framework (Figure 1) aims to provide an overview of a given performance ecology by connecting all the actors (top in the scheme in the image) to all the artifacts (bottom in the scheme in the image) using three levels: Role, Context, Activities. Each level proposes a different question related to the actors: 1) "Who is involved, and in which role?"; 2) "In which context is each actor involved?"; 3) "What kind of activities are the actors performing?" [2]. Given our research objective, we argue that ARCAA is a valid tool for analyzing what types of ecologies are determined by the use of scores in systems presented at NIME.

Figure 1. ARCAA as presented by Masu et al. [2].

Self-reflections within the NIME community

A long tradition of self-reflective research has characterized the NIME community. For instance, researchers reflected upon the value of community in interactive music research [24], creative and technical discourses surrounding DMIs design [25], general research approaches [26], the use of thematic analysis [27], and the identity of NIMEs in relation to the music performed [28].

In this context, systematic analysis of NIME proceedings has been used, for instance, to study the meaning of the word gestures for NIME authors [29], the evaluation strategies used [30], looking for trends in the development of NIMEs [31], and environmental practices [32]. Additionally, Sullivan and Wanderley recently analyzed DMI reliability across the proceedings of NIME/ICMC/SMC [33].

Systematic Literature Review

To perform the systematic literature review, we conducted a search in  Zenodo, the official NIME proceedings repository, using score as a keyword. According to the Zenodo guidelines, results match records of the word in any field of the repository. We performed this research on the week of the 26th of October 2020. The research produced 51 papers. Eight papers were excluded from the analysis, of which five were excluded because they simply mention the word score, but scores are not central in the interaction: [34],[35],[36],[37],[38]One paper used the word score to indicate point counting in a game [39]. Two other papers have been excluded because there is no relation between the users involved in the interaction and the score [40],[41]. Kim et al. developed a system in which a score is used for musical elements that are played by different devices [40]. There is no user-score relationship in this system, as the various devices interact among themselves. Xia and Dannenberg [41] used piano scores to train a system for automatic accompaniment, but there is no real relationship between users and scores in the final interaction. Figure 2 presents the occurrences of the papers for every year after the exclusion.

Figure 2: Number of occurrences per year after the exclusion.

We analyzed the identified 43 papers following a three-step procedure. In the first step, we created an ARCAA representation for each paper. In the second step, we coded the type of actions that users perform regarding the score and the main interactive system, also considering the context as emerged in ARCAA. We acknowledge that the term user can be limiting in the context of music performance, as recently pointed out by Rodger et al. [42], in previous research we argued in favour of the term actor [2]; in the context of this paper, we used user to identify the person who primarily interacts with the score in the system. In the third step, we recursively harmonized the codes according to similarities in the corresponding ARCAA representation and clustering them, following a procedure inspired by thematic analysis [43].

Results

Our analysis produced five main categories, two of them with sub-categories. We could not cluster three papers. An overview of the results is provided in the following table.

Category

Sub-categories

Papers

Scores as instructions (15 papers)

Scores suggesting how to play a particular new DMI (3 papers)

[44][45][46]

Scores suggesting how to play an acoustic instrument for pedagogical purposes (3 papers)

[47][48][49]

Scores suggesting how to play an instrument in non-pedagogical contexts (9 papers)

[50][51][52][53][54][55][56][57][58]

Scores as an interface to play a DMI (10 papers)

Interaction with the score only in real-time (6 papers)

[59][60][61][62][63][64]

Interaction with the score in a two-step process (4 papers)

[65][66][67][68]

Score as synchronization (6 papers)

[69][70][71][72][73][74]

Scores creation (6 papers)

[75][76][77][78][79][80]

Score as a recording (3 papers)

[81][82][83]

Non categorized papers (3 papers)

[84][85][86]

In the following sections, we detail categories and sub-categories, briefly describing the interactions that emerge in the various papers. We present the various papers in paragraphs according to similarities within the categories and sub-categories.

Scores as Instructions

The first category comprises 15 papers in which the score gives instructions in real-time to a performer who is playing an instrument (in most cases, a traditional instrument), like in screen score systems [8]. In these papers, scores are one of the main outputs of the interactive systems, performers are implicitly conceived as the main users, and the scores provide instruction on how to play their instrument. In this category, the interactions primarily occur in a real-time context either for concert or rehearsal. Figure 3 represents the core element of this ecology; some papers have also other actors, but in all cases, this configuration is at the core of the interaction. We realize that these scores have a composer. However, these papers focus on the type of interaction that performers have with the score, not the relation with the composer.

The sub-categories are defined according to similarities in the type of instrument performed (DMI or traditional instrument) or in the specific context (a few papers developed system for pedagogical contexts).

Figure 3. The core interactive ecology of the first category.

Scores Suggesting how to Play a Particular new DMI

In three papers, the score is used to give indications or support performances with a novel musical interface. In two cases, the user reads the score while performing with a DMI. In one case, the DMI is controlled by a Brain-Computer Interface (BCI) [44] and in the second case it is a keyboard that controls a vocal synthesizer [45]. In both these cases, the ecology is relatively simple; there is one user who acts as a performer in real-time and plays a new DMI while reading a score.

The last paper in this sub-category presents MICON [46], an installation where visitors can control an imaginary orchestra while reading a score displayed on a screen. This example is slightly different compared to the other two as the context is not performative, and the notion of the performer is a bit stretched in this case. The user is a visitor rather than an instrumentalist who controls an interface rather than plays an instrument. However, the connections in the ecology create a structure comparable with the other two papers presented in this subsection: the relation that the user has with the digital technology that produces the sound (play/control) and with the score (read in real-time).

Scores Suggesting how to Play an Acoustic Instrument for Pedagogical Purposes

Three papers presented systems designed to support students or novices in learning an instrument. Two papers [47],[48] present a system that listens to the performer/student and shows errors or alignment issues. These two papers present the very same ecology: one actor in his rehearsal/study context plays his instrument while reading a score on a screen, which also provides information about eventual errors.

The third paper in this category has a slightly more complex ecology, as it also includes a maestro who plays the role of the conductor [49]. In this paper, the system takes the time provided by the conductor and displays it on the score to facilitate beat counting to the pupils. We decided to incorporate this paper in the same category as it belongs to the same context, and the primary user (the student) has a similar relationship with the score and the instrument.

Scores Suggesting how to Play an Instrument in Non-Pedagogical Contexts

This last sub-category comprises nine papers in which an interactive system displays a score in real-time, and a performer plays an acoustic instrument. In some cases, the score is directly manipulated by the users. For instance, the performer controls the score (with a pedal)[50], or influences it by her movement [51]. In these two cases, the performer, while performing, reads a score and plays her instrument, but at the same time, her action affects the score. So the instrumentalist acts both in the input and in the output phases of the interactive systems.

In the majority of cases, however, the score is not controlled or manipulated by the instrumentalist. Therefore, from the instrumentalist perspective, the score is only the output of the interactive system. These cases include papers where many instrumentalists read the score on screens in network performances [52],[53] and a case in which a pianist performs a screen score that is influenced by an audience member with a BCI [54]. In another case, the score is delivered to the performer in the form of temperature feedback [55], different keys of a piano keyboard change temperature according to the score's notes. The notion of reading is not literal with this paper, but still, the user has to understand and interpret the information that the score gives him.

In the last three papers belonging to this category, composers are explicitly mentioned and briefly discussed. In one case, the composer adapts the score for a digital visualization [56]; in another case, the composer is the main author of the piece [57]. However, both papers mainly focus on the experience of the performer, who relates to the score and the instrument in the same way as the other papers in this category. The last paper belonging to this category is a system in which the composer manipulates the score in real-time, and a pianist reads it and performs it [58]. We decided to cluster this paper here because the situation is identical to the one described above from the pianist's perspective.

In some cases, the overall ecology varies a bit. However, the core of all this category’ ecologies is the same: the context is performative, and the performer plays her instrument while following the instruction on a digitally modified score.

Scores as an Interface to Play a DMI

The second category comprises ten papers presenting systems in which the score is used as an interface to control a DMI: the score is the input of the interactive systems. In this category, there are two main sub-categories: 1) systems in which all the interactions with the score occur in real-time, and 2) systems in which there is a two-step process; before the performance, users define some element of the score, and during the performance users explore them. The core relationship of all these papers - the users manipulate a score to play an interface - is the same, and for this reason, we categorized all of them together. However, the two sub-categories present slightly different core ecologies, and for this reason, we decided to present the ARCAA representation for each sub-category in the following dedicated two subsections.

Interaction with the Score only in Real-Time

In this sub-category (six papers), the user interacts with the score only during the performance (Figure 4). Examples include systems in which tangible elements are new forms of scores, such as physical objects representing notation that can be placed on a Reactable [59], or tangible plates with graphic elements engraved that can be explored through touch and represent the score as an inherent layer of the instrument [60]. Graphic elements are also used in live coding contexts as a representation of the music; in this context, the performer modifies the score by using code [61]. The DMI Notesatz has a mixed approach between tangible and graphical exploration of scores: a tangible interface is used to explore a graphic score [62]. In some cases, the manipulation of the score is collaborative, and different audience members can manipulate it via web networks [63], [64].

Figure 4. The core interactive ecology of the first sub-category of the second category.

Interaction with the Score in a Two-Step Process

In this sub-category (four papers), the user defines some score elements before the performance and interacts with them during the performance (Figure 5). Some systems use scores as visual elements; for instance, in LUSH, a preloaded score is transformed into a visual space that is explored during the performance [65]. In another example, users can define their rules before the performance and use such rules to draw a score on a whiteboard and scan it to affect the musical output [66]. Similarly, CABOTO [67] takes as input a score created before the performance that can be explored during the performance. In the last example [68], before the performance, the user defines the parameters of the score, and in real-time, she can explore them with a gestural controller.

Figure 5. The core interactive ecology of the second sub-category of the second category.

Overall in this category, the user in a real-time performative interaction creates or manipulates a score as a way to perform with the DMI. The score is the input and the DMI produces the sound. In some of these cases, the interaction with the score always occurs in real-time; in other cases, the user can set up some elements before the performance and then interact with them.

Score as Synchronization

In the third category, five papers describe systems in which a score is used for synchronizing or relating the performer’s actions with other elements [69], [70], [71], [72], [73]. In these papers, the system listens to the musician(s) for synchronization purposes. The score-following technique belongs to this category. Orio et al. [73] presented a meta-review of score following rather than a specific system; however, the relation between interactive technology, score, and performer is aligned with the others papers.

Liang and colleagues presented a framework to cope with those situations where “improvisation prevents traditional score following, but where synchronization is achieved at the level of beats, measures, and cues” and the score becomes dynamically adapted to the performer [74].

Overall, in this category, a performer plays an instrument in a real-time interaction and gives information to the interactive system that checks what the performer is playing against a score, synchronizing it or creating dialogues with other elements (Figure 6).

Figure 6. The core interactive ecology of the third category.

Scores Creation

In the fourth category, six papers present systems designed to support the creation of scores. In all these papers, the actors play the role of the creator/composer, the primary interaction occurs off-line, not on the stage, and the scores created with the system are usually supposed to be performed only in a second moment, which is outside of the focus of the paper. In some cases, there is one composer; in other cases, there are more than one, however, the core ecology is the same (Figure 7).

Two papers present systems to create scores collaboratively via the web. In the first case, the audience creates a new score corresponding to a new version of the piece for each concert [75]. The second case describes an online tool for collaborative composition [76].

Other systems are meant for individual usage. An example is note~, a tool created to help composers to use drawing as part of the score creation [77]. note~ has also been used in a studio context to control virtual instruments [78]. Other examples span from helping composers to use drawing as part of their compositional process [79] to offering gestural control of a scripting language [80].

Figure 7. The core interactive ecology of the fourth category.

Scores as a Recording

The fifth category (three papers) presents systems in which the main user is a musician who improvises and interacts with a digital instrument. Simultaneously, the system records a score without affecting the interaction nor the performance (Figure 8). For instance, Beson and colleagues created a sound spatialization system and a shaping interface that allows users to record scores of what they perform [81]; Liang and colleagues developed a system that analyzes piano pedaling and records a score of it [82]; finally, MuDI allows for real-time creation of scores for films [83].

Figure 8. The core interactive ecology of the fifth category.

Papers that we could not cluster

We could not cluster three papers. The first paper presents a notation system for Gametrak-based computer music instruments [84]. In their paper, the notation process is not automated; rather, it is hand-made in a more classical way. We could not cluster this paper as this paper simply presents a new form of notation.

The second paper that we could not cluster presents a complex system, where a playback system, a live score system, a BCI controlled by an audience member, a conductor, and multiple performers interact to produce a live soundtrack for a film [85]. This system shares characteristics with Scores as Instruction category if we look at the perspective of the instrumentalists. However, the number of different actors did not allow us to select instrumentalists as main users as it would have oversimplified the representation of the interaction.

The last non-categorized paper describes a system to scan scores and optically recognize notation for transcription, this tool was primarily meant for transcribing existing music [86].

Discussion - Toward a Taxonomy of Score Adoption in NIME

Our analysis identified five main tendencies in the use of scores in NIME that we briefly discuss here, contextualizing them within other literature about scores. Using ARCAA [2] helped us to understand the role that the score plays in the entire performative scenario, and we could perform an analysis that goes beyond simply classifying the scores' characteristics. We propose that the five categories could act as a taxonomy indicating possible approaches to adopt scores in future NIME practice, in relation to the users and their roles. In particular, we consider the first two categories to be richer as they are more profoundly linked to performative possibilities of interactive technology.

Scores as Instruction. This first category can be generally ascribed to screen scores. Although not all the systems rely on a screen to communicate the score, the fundamental interaction within this category matches well with the concept of screen scores [8],[10], an interactive digital system that gives instructions to musicians on how to play their instruments. Even the system that delivers the score in the form of temperature feedback [55] falls into this specific type of relationship. From this perspective, we argue that even this form of a score can be considered an extended version of the "new media manuscript" [8]. It is also interesting to see how, concerning the main interactive system, the score is primarily an output; and the main user reads it.

Scores as an Interface to Play a DMI. This category represents the general trend in which the score becomes inherent to the instrument, described, for instance, by Magnusson [15]. A straightforward consequence of the idea that "inherent to the circuits" [14] is creating interfaces to directly interact with such scores. A project like tangible score [60] was created precisely for this purpose: the score becomes a physical layer of the instrument. It is interesting to observe how, in this category, the score is the main input of the system, overall creating an opposite relation with the user when compared to the first category.

Scores as Synchronization. To a certain extent, this conception of scores reflects the idea and function of a score in a classical orchestra (with the distinction between score and parts that we previously described). The score is the tool that shows all the individual parts on the same page, facilitating the understanding of time correspondences and synchronizations among the various instruments. In these cases, the system acts as a coordination tool, partially resembling a conductor.

Scores Creation. This category is the less focused on real-time performative interaction. In this case, the score does not assume a new conception; on the contrary, it is aligned with the traditional meaning of score that consolidated throughout Western music history. Technology supports composers in their compositional process but does not change the overall relationship and function that scores have. Technology has helped composers ever since, being an algorithm that helps to compose, for instance, double counterpoints or canons (see [87]) or more complex systems as the ones analyzed in this paper. In the end, the score produced has the same function and relation with the piece and the performers.

Scores as a Recording. Similar to the Score Creation category, this last category does not change the traditional conception of scores. The score is a representation of a performance. The main difference between the two categories is that in this case, the score is resulting from performance - a recording -, while in the previous category the score is resulting from a process that focuses only on the compositional activity.

Limitations

The choice of the keywords represents one limitation of this paper. We decided to look only at the word score and exclude others with similar meanings, such as notation or inscription, for one main reason: the concept of score (not that of notation) encodes and embeds the idea of a music piece. This element provides the basis for the idea that DMI can embed a specific musical idea and the notion of the score [15], [16].

Another limitation of this paper is that it continues the idea that NIME places its practice in the trajectory of Western European music history [20]. This fact is a truism, and we argue that it is extremely valuable to study how the NIME community appropriated and modified a concept that is so central in western music practice as the score is. However, we also acknowledge that this trend implicitly carries a shade of neocolonialist practice that risks being too narrow. Recently, it has been discussed how neocolonialism can be a problem within NIME (e.g. [88], [89]). The first author of this paper comes from a western music background, and, therefore, the angle of this paper reflects his personal biographic history and perspective. However, we acknowledge that this paper can contribute to a NIME perspective overly focused on the western legacy, and we wanted to highlight this issue.

Conclusion

In this paper, we analyzed how the concept of score has been appropriated and modified within NIME practice and performance ecologies. We hope that this analysis can contribute to a better understanding of the relationship between score and performance ecology, NIMEs and scores, and, in general, NIME and western legacy. Even more, we hope that, in the future, other authors coming from different backgrounds will analyze how non-western concepts have been appropriated by the NIME community.

Acknowledgments

The paper is part of the Ph.D. research of the first author. We wish to acknowledge the feedback received by the Thesis Plann evaluation committee, in particular for the idea of using ARCAA to study research conducted by other researchers. The first author acknowledges ARDITI -Agencia Regionalpara o Desenvolvimento e Tecnologia under the scope of the Project M1420-09-5369-FSE-000002 - PhD Studentship. We acknowledge the support of LARSyS to this research (Projeto - UIDB/50009/2020). This work is also co-funded by FCT/MCTES NOVA LINCS PEst UID/CEC/04516/2019.

Compliance with Ethical Standards

This paper complies with the NIME ethical standards. The only ethical issue with this research is the western perspective discussed in the Limitations section.

Comments
0
comment
No comments here
Why not start the discussion?