Skip to main content
SearchLogin or Signup

FaMuRa: Music Generation System using Unconstrained Body Motion

NIME2021 Work-In-Progress Showcase

Published onJun 01, 2021
FaMuRa: Music Generation System using Unconstrained Body Motion
·

FaMuRa: Music Generation System using Unconstrained Body Motion

Yuta Shimomichi (The University of Tokyo), Hidetsugu Irie (The University of Tokyo), Shuichi Sakai (The University of Tokyo)


1.Pub Pub Link

https://nime.pubpub.org/pub/uki5a875/draft?access=8u5402uu

2.ABSTRACT

We propose a music generation system named FaMuRa (Fairy Music orchestRa) that generates diverse music from the user's body motion.

This system has three features.

First, when a user uses this system, no specific gestures are required. The user can move his or her body as he or she wants to play music. Second, no matter how the user moves their body, the generated sound will be musical. In this discussion, the term "musical" means that the generated sounds contain rhythm, melody, and harmony. Third, when the user interacts with the system with intent, the degree of music excitement changes.

To achieve these features, we propose three main ideas.

The first idea is that the system only detects whether the user's body is moving or not. It generates sound only when the user moves their body. When the user's body is still, there is no sound.

The second idea is that the system overlaps some loop playings of a simple motif to build music. When the user moves their body, the system starts to play a simple motif repeatedly in quantized timing and tempo. The way to determine the motif is as follows.

(a) Determine the scale.

(b) Determine the outline of the motif.

(c) Determine the first note of the motif.

The user determines the scale and the outline of the motif in advance. After starting to play, the system determines the first note of the motif depending on the position the user moves their hands. For example, when the user moves their hands in a high position, the motif starts from a high-pitched tone. Besides, the loop playing will stop probabilistically. In this way, as long as the user moves their body, loop playings of the motif that start from various pitches will start and overlap each other, coming together to form music.

The third idea is that the system connects the user's body motion and the state of the generated music indirectly by using intermediate variables. We represent the intermediate variables as a point that moves on a two-dimensional plane. It moves on that plane according to the magnitude of the user's body motion. And the state of the music changes according to the position of the point on the plane. In this way, the user can interact with the system by unconstrained body motion. And the generated music changes its state naturally without being overly influenced by noise in skeleton estimation and small unintended body movements.

These features make it possible for a wide range of users, from beginners to experienced users, to have a new experience of generating music through unconstrained body motion without any training.

3.Program Notes

FaMuRa (Fairy Music orchestRa) is an interactive music generation system that generates diverse music from the user's body motion. This system overlaps some loop playings of a simple motif to build music. When a user uses this system, no specific gestures are required. The user can move their body as he or she wants to play music. In addition, when the user interacts with the system with intent, the degree of music excitement changes. These features make it possible for a wide range of users, from beginners to experienced users, to have a new experience of generating music through unconstrained body motion without any training.

4.Requirements

- equipment: a computer powerful enough to run skeletal estimation, a web camera, an audio interface (UR44), stereo speakers.

- OS: ubuntu 18.04

- software: openpose(https://github.com/CMU-Perceptual-Computing-Lab/openpose), Super Collider, Open Sound Control.

- space: This system is suitable for a small room or public space.

4. Media



Comments
0
comment

No comments here