NIME 2021 paper "Human noise at the fingertip: Positional (non)control under varying haptic × musical conditions"
As technologies and interfaces for the instrumental control of musical sound get ever better at tracking aspects of human position and motion in space, a fundamental problem emerges: Unintended or even counter-intentional control may result when humans themselves become a source of positional noise. A clear case of what is meant by this, is the “stillness movement” of a body part, occurring despite the simultaneous explicit intention for that body part to remain still.
In this paper, we present the results of a randomized, controlled experiment investigating this phenomenon along a vertical axis relative to the human fingertip. The results include characterizations of both the spatial distribution and frequency distribution of the stillness movement observed.
Also included are results indicating a possible role for constant forces and viscosities in reducing stillness movement amplitude, thereby potentially enabling the implementation of more positional control of musical sound within the same available spatial range.
Importantly, the above is summarized in a form that is directly interpretable for anyone designing technologies, interactions, or performances that involve fingertip control of musical sound.
Also, a complete data set of the experimental results is included in the separate Appendices to this paper, again in a format that is directly interpretable.
•Applied computing → Sound and music computing; Performing arts;
•Hardware~Emerging technologies~Biology-related information processing
•Human-centered computing → Haptic devices
Often when learning to play an instrument, old or new, the goal will be to obtain some level of intentional control over musical sound. Even where the ultimate goal is to (partially) relinquish control, as in the uncontrollability being developed in feedback instruments (see e.g. [4] and [2]), the necessity of mastering a level of intentional control still holds. Often, too, whether in traditional instruments or novel interfaces, such control will be directly related to the position of (parts of) the human body in space.
Here, however, a fundamental problem comes into focus, as technologies and interfaces get ever better at tracking aspects of human position and motion in space: Movement will occur, even during episodes in which a person has no intention for the part of their anatomy in question to move – or, more strongly, even when the explicit intention for that body part is to remain still. Here, we will call such movement stillness movement.
For new interfaces for musical expression, stillness movement is potentially problematic: It may lead to unintended control, where control changes occur which do not correspond to any user intention (e.g. “I have no intention of moving right now”). Stillness movement may also lead to counter-intentional control, where control changes occur which go against user intention (e.g. “I want to remain completely still right now”).
In short, from the perspective of intentional control, human stillness movement introduces noise.
To properly inform control implementations in which control should be intentional, it is therefore useful to experimentally obtain data sets which after analysis will enable statements that, first of all, quantitatively characterize stillness movement noise.
Additionally, it is useful to extend such experimentation to include musical haptics (see e.g. [3]): This simply because in haptic systems with precise position inputs, this kind of noise will be present; also, however, because haptic devices and algorithms with appropriate force outputs may play a role in dealing with human noise where it is a problem, due to their inherent ability to influence the mechanics of movement and touch. A useful extension of experimentation, therefore, would be to also quantify the effects of different haptic conditions on stillness movement, and on the simultaneously resulting musical control.
Below, we present the execution and results of this kind of experimentation, when focusing on a specific type of anatomical movement involving the fingertip: vertical fingertip movement, which is here defined as movement that is approximately orthogonal to both the surface of the human fingerpad itself and to some second surface of known position. This anatomical movement is of course very important in human manipulation and control, and integrated in the critical path of many forms of instrumental control of musical sound, from ancient to recent.
To be sure, however, such experimentation will not occur in a contextual vacuum in more ways than this. Hand and finger tremors have been long studied in the medical literature, and this also in the absence of pathology. For example, based on [5], in healthy individuals, we may reasonably expect such finger movement to be mainly occurring in the 1 - 30 Hz frequency range, and to be largely determined by the mechanical properties of the finger itself. There also already has been considerable research into involuntary micromotion in a musical context, for example in [6], where, during standstill, the effect of listening to different types of musical sound on head motion was tracked and studied. A different example is [7], where involuntary micromotion elsewhere in the human body was used to generate auditory feedback, with the aim of inducing an awareness of the related kinaesthetic phenomena themselves. Below, however, we will focus on fingertip movement in the context of the control of musical sound.
The randomized, controlled experiment was executed using a Ghostfinger system for computed fingertip haptics (see [1]). This system provides haptic programming primitives for up/down fingertip movement which are simultaneously visualized as stereoscopic holograms, together with a floating cursor tracking the user's fingertip position (see Figure 1). As is illustrated, the system both tracks vertical fingertip movement (in mm) and applies vertical forces (in N) along a z axis above the device surface, via the fingerpad transducer shown.
The weight of the fingerpad transducer was 10 g, with applied forces added to or subtracted from this weight. Tracking of transducer position was based on pulsed reflective infrared sensing, while forces were applied electromagnetically. Peak-to-peak positional amplitude noise was less than 0.2 mm (signal recordings are in the Appendices). The total haptic roundtrip latency from input to output was 4.0 ms, when measured using strict magnetic field strength thresholds of ≥ 99% of target level. Force output had a resolution of ± 0.003 N, this with noise performance aided by the technology completely avoiding the use of connected mechanical parts moving relative to the target anatomical site, and with an underlying capability to produce accurate wave output up to 1000 Hz. For more information, including on how handling the occurrence of lateral fingertip slippage was incorporated in the basic design of the technology, please see [1] and its references.
Correspondingly, the raw measurements made were fingerpad z position (mm) and z speed (mm/s) time series, sampled at 4000 Hz. Simultaneously, the system's outputs were used to apply a target z force (N) to the fingerpad at a sample rate of 1000 Hz, and produce headphone audio at 96 kHz. As regards proper periodic sampling of the z (mm) signal, it was experimentally double-checked that any aliased additions (i.e. at frequencies > 2 kHz) added along the various analog and digital stages of the sensing chain stayed below the peak-to-peak threshold used for amplitude noise in general.
A shared property of all experimental conditions, however, was that subjects did not receive (computed) visual output during the actual measurement of stillness movement. The idea here was that facilitating an attentional focus on haptic and audio output might help in measuring effects specific to these modalities.
Another property shared across conditions was that fingerpad z position always stayed between 0.0 and 10.0 mm above device surface. This was the range where the haptic and musical-control conditions that were used existed, and where measurement took place. This also implied that a subject's fingertip was always held suspended, and did not rest on the device surface.
A third shared property was that, as part of the experimental task, subjects moved their fingertip both up and down, doing so before each measurement but with its condition already activated. The idea here was that subjects could perceive the current haptic and musical-control response to movement before measurement of stillness movement took place; and that this measurement was then part of an episode of continuous positional control.
A final shared property across all conditions was that during the measurement itself, subjects explicitly focused on holding their fingertip as still as possible, and this for a duration of 4 seconds, chosen as a reasonable duration for a short real-life control episode.
The properties that were specific to each of the experimental conditions are then described in Table 1.
Regarding the haptic component (“n”) of the conditions listed, zero force was intended as part of the control condition. The specific parametrizations shown for positive force, negative force, viscosity, anti-viscosity, and positional markers were then intended to help preserve the subjects' ability to exercise continuous positional control while testing these variations.
Here, the amplitude for positional markers was based on a trade-off between easy perceptibility and limited infraction on continuous positional control during their occurrence. Anti-viscosity was limited in amplitude so as to avoid the occurrence of wild positional oscillations having a negative impact on both people and technology.
Regarding the musical-control component (“m”) of the conditions in Table 1, no musical control, too, was intended as part of the control condition. The parametrization of no musical control versus musical control was then intended to have this component vary not the presence, but the control of sound.
In addition to this, the parametrization of musical control specifically was also intended to aim for both ease and sensitivity during the control of sound: After limited dry runs, control had to be executable also by subjects with little to no musical training; and very small fingertip movement was still to result in audibly perceivable changes during stillness movement.
Further details about the experimental conditions can be found by looking at the code archived as part of the separate Appendices to this paper.
Details about the experimental protocol followed also can be found by looking at the code in the Appendices, e.g. regarding the specific dry runs used. Overall, the protocol can be summarized by describing three repeatedly occurring phases:
• Pre-measurement phase: The subject rests the palmar side of the right hand on the device surface, while keeping the right index fingerpad above surface. Across the spatial measurement range, the current haptic × musical-control condition is activated. The subject is asked and observed to move their fingerpad up and down, ending up mid-range (i.e. near 5.0 mm at a position of their own choosing); and to close their eyes.
• Measurement phase: The subject is now focusing on holding as still as possible both their fingerpad and the output of musical control, if the latter is present. (For all experimental runs, finger posture was observed to be as illustrated in Figure 1.)
• Post-measurement phase: The subject is asked and observed to open their eyes; haptic output transitions to 0 N, and audio output to silence.
In the above way, each condition was tested 3 times with each subject, for a total of 36 randomized experimental runs per subject.
There were 8 test subjects, 3 male, 5 female, 15-19 years old, with 2 individuals being left-handed. The test subjects followed a university course explicitly designed to combine research education with simultaneous participation in actual experimental research. Their age range was considered a suitable one, for being a typical age at which people (first) study the mastery of musical expression. 7 out of 8 subjects reported having had between 1.5 and 10 (average: 6.2) years of lessons on a finger-operated musical instrument. 6 out of 8 reported regularly making music by playing their finger-operated musical instrument (being either an acoustic, electric, or bass guitar, or a cello or violin), and this on both amateur (4×) and semi-professional (2×) levels. No finger length measurements were taken.
In general, all test subjects reported having neither hearing problems nor problems with manual control. However, during the experiment 1 subject reported not being able to concentrate well; 1 reported being not well-rested; 2 reported being a bit nervous; and 2 reported having cold hands (room temperature throughout the experiment was 18 to 21 degrees Celsius).
In Figure 2, a visualization and quantitative analysis is shown of the signal I/O during a single experimental run recording 4 seconds of stillness movement by a fingertip. This format was used to analyze and archive all experimental runs.
The starting point in this format is the graph on the left that is second from top: It shows the fingertip z (mm) time series. Directly below it, along the same time scale, but of course using different physical amplitude scales, are the simultaneous fingertip speed (mm/s) and target force (N) time series.
Directly to the right of the z (mm) time series is a histogram view of its amplitude distribution, using an identical vertical scale, for easy visual comparison, and with submillimeter bins.
Directly above the z (mm) time series is a spectrogram view of it, again placed along the same time scale, but intended only to provide visual clues about changes over time in the frequency domain.
To its right, however, is a graph showing a custom Discrete Fourier Transform (DFT) of the recorded segment of stillness movement as a whole. For easy visual comparison, this is again plotted using an identical vertical scale (now, in Hz). To support interpretability, the black dots (with lines inbetween) here show either the z signal average (at 0 Hz) or the estimated peak-to-peak amplitude (above 0 Hz) directly in millimeters.
So, for reasons of direct interpretability, the DFT analysis here (and below) intentionally was carefully implemented and tested to produce its results in terms of real-world units, measured along linear measurement axes: This enables easy comparison to real-world fingertip movements, e.g. when using the results during the design of fingertip-based musical control.
Finally, directly below the histogram in Figure 2 is an identifier of the experimental run and its source recording. The printed statistics below this identifier as well as the DFT will be further discussed in the next sections.
When looking at the spatial distributions of fingertip stillness movement recorded under conditions where a zero force was applied (see Appendices), it often would seem clearly incorrect to summarize what is visible in the time series and histograms as movement around a single mean, with variation distributed symmetrically.
This was confirmed by performing a Jarque-Bera statistical test on each of the z (mm) time series: The p values computed for the null hypothesis of normality were always < 0.005.
As another means, then, to quantitatively summarize observed stillness movement, z travel amplitude (mm) was used: i.e. the difference between the highest and lowest z positions recorded during a single run. This derived measure has the possible disadvantage of being sensitive to outliers in the underlying time series. An advantage, however, is the direct interpretability when designing fingertip interactions.
Figure 3 gives an overview of z travel amplitude as observed across experimental runs: The dot plot shows the individual values, and, introducing more structure, a visually matching histogram shows the distribution of these travel amplitudes. Overall, the observed segments of stillness movement had travel amplitudes between 0.27 and 3.52 mm. As can be seen in the histogram, within this general range, travel amplitude most often was between 0.5 and 1.5 mm.
When now moving to the frequency characteristics of observed stillness movement, it is important to first briefly discuss the interpretability of the z (mm) DFTs introduced previously.
The DFT used was verified to correctly measure peak-to-peak movement amplitudes at frequencies matching the underlying process of cosine correlation. However, just as for DFT in general, for cosine signal input with periods that did not exactly match test periods, DFT yielded false-positive, non-zero amplitudes at other frequencies; a highest DFT amplitude that could be lower than the wave amplitude in the time series; and addition of frequency-adjacent DFT amplitudes also not accurately yielding the wave amplitude.
Importantly, and more domain-specifically, the spread of the “false positive” frequencies and the corresponding amplitude levels seemed to be non-negligible when analysing the type of fingertip distance time series acquired in this experiment. Therefore, in per-run visualization and interpretation, DFT peak-to-peak amplitude in mm was regarded only as an estimate, not an exact measurement.
Having said this, under the conditions where zero force was applied, the peak-to-peak amplitude of fingertip movement was always measured as less than 0.01 mm at frequencies above 30.0 Hz.
For an example view of what then was happening between 30 Hz and 0 Hz, Figure 4a plots the DFTs of two experimental runs, one with and one without musical control (and with the frequency axis now oriented horizontally).
As can be seen, for both examples, the estimated peak-to-peak amplitude increases from values below 0.1 mm to values above 0.5 mm when going toward 0 Hz, doing so steeply, but irregularly. When averaging DFTs across all zero-force experimental runs, however, a less random picture of fingertip stillness movement emerges: As is shown in Figure 4b, both for the case with and the case without musical control, the incline has become smoother, and can be approximated by a 3/17 · 1/f model.
The observed stillness movement was also compared across haptic conditions, using the same z travel amplitude (mm) measure that was introduced in the previous section. In Figure 5a, the results of this are first summarized for the experimental runs with no musical control: For each haptic condition, the observed minimum, maximum, and mean z travel amplitude are shown, and also the observed standard deviation.
As can be seen, under conditions positive force and viscosity, fingertip stillness movement on average had a smaller travel amplitude than under the zero force control condition.
Of course, chance plays a role in such relative outcomes, and the worry would be that on repeated execution the result, by chance, would become different. Consider, however, the results for the experimental runs with musical control, shown in Figure 5b.
Here too, positive force and viscosity on average yielded a smaller travel amplitude than zero force.
As part of the a priori fixed experimental design, an analysis of variance (ANOVA) across the experimental conditions was initiated – now of course hopefully also to further investigate the reliability of this specific difference. As its first step, a preliminary test of normality was performed, to ascertain that both group and population measurements were approximately normally distributed. As there were < 50 measurements per group, a Shapiro-Wilk statistical test was used for this. Its input and outcomes can be found in the Appendices.
In summary, however, the within-group measurements did not show a statistically significant likeness to the normal distribution, as for both the experimental conditions with and without musical control, 4 out of 6 yielded a p < 0.05. The ANOVA therefore could not be meaningfully executed, only yielding the result that future tests based on the travel amplitude measure should be of a different type.
It was hoped that positional markers would help in reducing stillness travel amplitude relative to the control condition of not applying a force to the fingertip, by haptically signaling small movements to the user. As can be seen in Figures 5a and 5b, however, this did not happen.
This was also reflected within the results of a subjective questionnaire, filled in before and after the measurement runs by each test subject: 3 out of 8 mentioned positional markers as explicitly unhelpful in performing the experimental task, writing down “spatial markers [...] made keeping still harder”; or even, “spatial markers made executing the task hardest”; and finally, “did not prefer spatial markers, as these amplified small movements”.
Indeed, when we observe the reflection of ftarget in z in the experimental run recorded in Figure 2, the sentence quoted last seems an astute observation. In general, parametrizing positional markers to produce force step changes of 0.044 N amplitude (at each 0.2 mm positional transition) may have been too large a value to achieve the desired effect.
Above, we have described the motivation, execution, and results of a randomized controlled experiment in which 8 test subjects repeatedly performed the task of holding their right index fingertip as still as possible, during a series of 4-second episodes, and under varying conditions of fingertip haptics and musical control.
The vertical stillness movement that was experimentally observed while not applying a force to the fingertip can be characterized by the following main points:
• spatially, it did not follow a normal distribution;
• its peak-to-peak amplitude lay between 0.2 and 3.6 mm;
• most often, between 0.5 and 1.5 mm (see Figure 3 for the distribution
profile);
• in the frequency region above 30 Hz, peak-to-peak amplitude was always
measured as less than 0.01 mm;
• between 0 and 30 Hz, the peak-to-peak amplitude on average could be
approximated by a 3/17 · 1/f model (see Figure 4b);
• the average peak-to-peak amplitude decreased when a constant upward force
to the fingertip, or constant viscosity was activated – both when musical
control was present and when it was absent (see Figures 5a and 5b).
The final point listed above, although without claims of statistical significance, can be seen as an empirical hint that applying constant upward forces and viscosities on average may yield smaller fingertip stillness movement amplitudes, which in turn may enable implementing more positional control within the same available spatial range.
Also more generally, the above points can be used as a directly interpretable reference when designing technologies, interactions, or performances that involve the fingertip control of musical sound.
Finally, in the spirit of open data, the data set consisting of the complete per-run experimental results – archived in the separate Appendices to this paper – is also presented as a research outcome. And, notwithstanding that this data set is directly representative of the (larger-size) raw signal recordings, the author of course can also be contacted for the latter.
I would like to thank Annebeth Simonsz and the Honours Academy for supporting the development and execution of the course (of which this research was an integral part) every step of the way.
My heartfelt thanks also to René Overgauw and Arno van Amersfoort, at the Electronics Department at LION, for their important and continued support.
Sincere thanks also to Anne Ouweneel, Aquilla van den Berg, Bardia Nikookar, Dave Aulia, Dennis Vianen, Julia Wesseling, Kiki Abels, Laurentine Rietvelt, Simone Flipse, Sraman Chatterjee, Stella Berbiers, Stijn Huijgen, Timo Alderliesten, and Wouter Bakker, for their assistance while executing this research.
Last but not least, I would like to thank the anonymous reviewers and meta-reviewer, for their valuable comments and guidance in processing these.
[1] De Jong S, 2017 Ghostfinger: a novel platform for fully computational fingertip controllers. In Proceedings of NIME 2017, Copenhagen, Denmark.
[2] Kiefer C, Overholt D, Eldridge A, 2020 Shaping the behaviour of feedback instruments with complexity-controlled gain dynamics. In Proceedings of NIME 2020, Birmingham, UK.
[3] Papetti S, Saitis C (eds.) 2018 Musical haptics, Springer Series on Touch and haptic systems.
[4] Úlfarsson H, 2019 Feedback mayhem: Compositional affordances of the halldorophone discussed by its users. In Proceedings of the ICMC 2019, New York, USA.
[5] Raethjen J, Pawlas F, Lindemann M, Wenzelburger R, Deuschl G, 2000 Determinants of physiologic tremor in a large normal population. Clinical Neurophysiology, 1 October 2000.
[6] Gonzalez-Sanchez V E, Zelechowska A, Jensenius A R, 2018 Correspondences between music and involuntary human micromotion during standstill. Frontiers in Psychology, 7 August 2018.
[7] Candau Y, Françoise J, Alaoui S F, Schiphorst T, 2017 Cultivating kinaesthetic awareness through interaction: Perspectives from somatic practices and embodied cognition. In Proceedings of MOCO 2017, London, UK.
For purposes of reference, reproducibility, and transparency, the Appendices, in PDF format, also contain the complete experimental results. Their visualization has been carefully designed so that one experimental run can still be inspected while fitting on a single (A4-size or larger) electronic screen. This supports the quick and easy comparison, across subjects and conditions, of multiple individual runs.