NIME 2022 Doctoral Consortium Submission
Human—computer interaction, movement-based interaction, dance, movement, sonification, documentation
Dance is an embodied practice that is challenging to capture, document and archive because dance movement encompasses complex somatic and also cultural knowledge. Up to date, dance is mainly recorded through audio visual media on either institutional databases or streaming websites such as YouTube or Vimeo. In training environments, dancers often resort to video to learn and share their knowledge on their practices and guide each other visually . However, mere video recordings do not inform on the cultural context, the movement qualities nor the kinaesthetic sensations in dance, among others.
This PhD aims at designing interactive systems for dance documentation and transmission. The goal is to capture traces of dance practices and cultural experiences.
The first year of this PhD includes a project grounded in a field work engaging with the modern dance repertoire by Isadora Duncan, as thought by a Duncanian dancer and teacher. The project aims to explore ways in which sound can be used as a tool and material when designing for documentation and transmission of the movement qualities and kinaesthetic sensations — or experiential qualities — that the Isadora Duncan repertoire embodies.
Throughout the creative process, some dance practitioners invent personal ways of representing their dance practices using heterogeneous artefacts to document it such as pictures, videos, drawings, annotations  but also using objects that carry cultural and personal values. Moreover, dance practice generates a diversity of embodied sensations and experiences that are difficult to articulate and transmit for practitioners . Despite the diversity and complexity of dancers’ embodied experiences and the heterogeneity of their documentation practice, the literature in HCI focuses mostly on providing tools supporting sketching and video annotation of movement .
Music can be understood as movement , and from a phenomenological perspective, how our bodies move in the world defines how we experience the world. Body movement is essential in both musical performance, perception, as well as our experience of music . Arising from this idea that music is movement, within the context of the Isadora Duncan project, we aim to address movement through sound, where sound and music acts as a carrier of movement qualities and experiential qualities .
This PhD proposal aims at designing interactive systems for dance documentation and transmission, with the goal of capturing traces of dance practices and cultural experiences. The specificity of this thesis is to go beyond video annotation tools. To do so, we will follow an ethnographic design approach  in order to develop movement-based and somaesthetic interaction design applications . In addition to sound, tools such as soft robotics, prothesis and wearables will be considered. Such tools would be capable of both capturing the dancers’ movements but also influencing, changing and disrupting it. To our knowledge, such ethnographic design approaches have never been applied to the context of dance archives and training. The PhD is also be committed to a design through practice  and user-centred design methods which would place the users in the centre of the design of the technology emphasising a first-person perspective and minimising imposed choices from the academic researchers.
Specifically for the Isadora Duncan project, the PhD candidate is sensitising their own body to the Isadora Duncan repertoire through attending dance lessons over a period of 7 months, in parallel with close collaboration and exchange with the Duncanian dance teacher. In these contexts, sound and music stood out an important mode of transmission for the teacher, actively drawing from musical terms articulate movement qualities and consistently using sound and music during lessons. Hence, sound and music stands out as central to the design process through the use of methods such as vocal sketching, movement sonification, and now moving into the design of sounding objects with a variety of mappings. A sounding scarf is in the process of making, as silk scarfs play an important role in the taught choreographies, and are used in dance lessons to demonstrate movement quality (see Image 1).
The silk scarf is currently being developed using an IMU (internal measurement unit) connected to Wifi and conductive material, in order to sonify both the temporal movement of the scarf as well as the dancers touching the scarf and passing it in-between each other. Various approaches to sonification are being explored, using material from vocal sketching of the Isadora Duncan movement qualities by the Duncanian dancer and fluid sounds, referring to Isadora Duncan’s attachment to the water and the ocean.
As a participant in the NIME doctoral consortium, I hope to contribute by bringing forth a discussion on music-related movement, and further developments of methods and theory regarding the role of movement in systems for musical interaction, and the role of music in movement-based interaction. Additionally, I hope to be able to bring interesting perspectives from my field work and research through practice from an HCI (human—computer interaction) perspective.
Through presenting my research and design work, I hope for the NIME community to be able to provide further insights into the theoretical grounding of my work. I hope to participate in an exchange on system development, in particular in regards to the use of IMUs and conductivity as input modalities for sonic output.
Furthermore, I hope to have deep and engaging discussions regarding mapping strategies and the use of sound as a reward vs sound as an incentive when moving with the desired movement qualities. I would be thrilled to take part in a forum that would show interest in discussing questions such as: how can we design music-movement interactions for a given movement quality, while at the same time staying true to designing for rich experiential qualities? How do we as NIME and HCI practitioners discern movement qualities and experiential qualities in our work and creative processes? And can more lucid articulation of both (movement qualities and experiential qualities), contribute to creating interfaces that manage to document and transmit a more in-depth trace of dance and other movement practices?
 Rivière, J.-P., Alaoui, S. F., Caramiaux, B., & Mackay, W. E. (2019). Capturing Movement Decomposition to Support Learning and Teaching in Contemporary Dance. Proc. ACM Hum.-Comput. Interact., 3(CSCW). https://doi.org/10.1145/3359188
 Ciolfi Felice, M., Alaoui, S. F., & Mackay, W. E. (2016). How Do Choreographers Craft Dance? Designing for a Choreographer-Technology Partnership. In Proceedings of the 3rd International Symposium on Movement and Computing. Thessaloniki, GA, Greece: Association for Computing Machinery. https://doi.org/10.1145/2948910.2948941
 Fdili Alaoui, S., Schiphorst, T., Cuykendall, S., Carlson, K., Studd, K., & Bradley, K. (2015). Strategies for Embodied Design: The Value and Challenges of Observing Movement. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (pp. 121–130). Glasgow, United Kingdom: Association for Computing Machinery. https://doi.org/10.1145/2757226.2757238
 Hsueh, S., Alaoui, S. F., & Mackay, W. E. (2019). Deconstructing Creativity: Non-Linear Processes and Fluid Roles in Contemporary Music and Dance. Proc. ACM Hum.-Comput. Interact., 3(CSCW). https://doi.org/10.1145/3359305
 Carroll, E. A., Lottridge, D., Latulipe, C., Singh, V., & Word, M. (2012). Bodies in Critique: A Technological Intervention in the Dance Production Process. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (pp. 705–714). Seattle, Washington, USA: Association for Computing Machinery. https://doi.org/10.1145/2145204.2145311
 Singh, V., Latulipe, C., Carroll, E., & Lottridge, D. (2011). The Choreographer’s Notebook: A Video Annotation System for Dancers and Choreographers. In Proceedings of the 8th ACM Conference on Creativity and Cognition (pp. 197–206). Atlanta, Georgia, USA: Association for Computing Machinery. https://doi.org/10.1145/2069618.2069653
 El Raheb, K., Kasomoulis, A., Katifori, A., Rezkalla, M., & Ioannidis, Y. (2018). A Web-Based System for Annotation of Dance Multimodal Recordings by Dance Practitioners and Experts. In Proceedings of the 5th International Conference on Movement and Computing. Genoa, Italy: Association for Computing Machinery. https://doi.org/10.1145/3212721.3212722
 Godøy, R. I., & Leman, M. (2010). Musical gestures: Sound, movement, and meaning. Routledge.
 Jensenius, A. R. (2007). Action-sound: Developing methods and tools to study music-related body movement.
 Ley-Flores, J., Bevilacqua, F., Bianchi-Berthouze, N., & Taiadura-Jimenez, A. (2019). Altering body perception and emotion in physically inactive people through movement sonification. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE. https://doi.org/10.1109/acii.2019.8925432
 Caramiaux, B., Bevilacqua, F., Bianco, T., Schnell, N., Houix, O., & Susini, P. (2014). The Role of Sound Source Perception in Gestural Sound Description. ACM Transactions on Applied Perception, 11(1), 1–19. https://doi.org/10.1145/2536811
 Dunne, A., & Raby, F. (2013). Speculative everything: design, fiction, and social dreaming. MIT press.
 Hook, K. (2018). Designing with the body: Somaesthetic interaction design. MIT Press.
 St\aahl, A., Tsaknaki, V., & Balaam, M. (2021). Validity and Rigour in Soma Design-Sketching with the Soma. ACM Transactions on Computer-Human Interaction (TOCHI), 28(6), 1–36.
 Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through Design as a Method for Interaction Design Research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 493–502). San Jose, California, USA: Association for Computing Machinery. https://doi.org/10.1145/1240624.1240704
Tove is in her first year of the PhD (3 years funding), and is working towards the goal of capturing traces of dance practices and cultural experiences, with an emphasis of the practitioners’ first-person experiences. In order to work towards this goal, her PhD will consist of the following steps/methods/fieldwork :
To run ethnographic field studies where she articulates dance artists’ experiences by interviewing them, observing and probing their documentation practice.
To design interactive applications that allow dance artists to trace, capture and share their dance practices from their first-person perspective.
To deploy the technology in real-world situations.
Tove is currently working on the Isadora Duncan project where she collaborates with a connoisseur in Duncanian dance to design an interactive sound object that allows to transmit the Duncanian style of dance. In this project she is engaged personally in learning the repertoire and understanding from a first person perspective the principles of Duncan that she will design for.
Tove is working within the ex)situ group under my supervision, in the LISN and interaction department at Université Paris-Saclay. She participates in the group’s research activities and teaches human computer interaction in the computer science department. Tove also participates to the research group Element in IRCAM, a consortium working on AI in musical interfaces. She will benefit from participating to the doctoral consortium early into her PhD in order to have feedback on the design of the sonic object she is working on.
Sarah Fdili Alaoui
Associate Professor in Human-Computer Interaction + Dance Artist
Head of the HCID Masters
LISN - Université Paris Saclay