Abstract

Audio and haptic sensations have previously been linked in the development of NIMEs and in other domains like human-computer interaction. Most efforts to work with these modalities together tend to either treat haptics as secondary to audio, or conversely, audio as secondary to haptics, and design sensations in each modality separately. In this paper, we investigate the possibility of designing audio and vibrotactile effects simultaneously by interpolating audio-haptic control spaces. An inverse radial basis function method is used to dynamically create a mapping from a two-dimensional space to a many-dimensional control space for multimodal effects based on user-specified control points. Two proofs of concept were developed focusing on modifying the same structure across modalities and parallel structures.

Introduction

Audio and haptic sensory experiences appear in various devices and interactions with distinctions between the tactile and kinesthetic submodalities. Tactile sensations, like vibration, occur in the skin while kinesthetic sensations are caused by movement and force. Acoustic musical instruments necessarily produce both tactile and kinesthetic feedback through intrinsic physical features, but so too is intentional tactile feedback important in the design and use of digital musical instruments (DMIs) [1][2]. In the broader realm of NIMEs, interfaces with a vibrotactile component exist and have reported a different experience compared to audio alone [3][4][5][6][7]. Translation algorithms have been developed that process an audio file to create an accompanying vibrotactile component [8][9][10].

This paper investigates simultaneous design of audio and vibrotactile effects through the use of radial basis function interpolation as a means of treating the modalities equally. This is done to better understand the role of a designer’s approach to multimodal effect and experience creation. Two proofs of concept were implemented to treat audio and vibrotactile effect synthesis as independent and equal sensory components. One system is used to control the same feature multimodally while the other uses different features connected through metaphor.

Background

Haptic feedback in the form of vibration can be constructed like audio, but the tactile sense differs from the auditory sense, for example in how frequency is perceived. [1][11][12][13]. However as with audio, illusions are often used to design more complex effects. For example, the common “sensory funnelling illusion” or “phantom actuator illusion” results in separate stimuli being perceived as occurring at a point between them [14][15][16][17]. DMIs can benefit from the addition of haptic feedback to provide information to a performer that cannot be gained through audition, vision, and proprioception alone [1]. Vibrotactile feedback is common in contactless DMIs, however it is also important in DMIs that require physical contact — and thus kinesthetic feedback — for normal use [8][18]. While some vibrotactile tools support the inclusion of audio, these expect audio to be completed and primarily aid in synchronization tasks [19][20][21]. This is part of a common trend among existing approaches to design separate audio and tactile design.

In musical applications, design of the audio component typically precedes the tactile component. The tactile component is then designed independently or through a causal link in the case of translation algorithms. They are then combined after the fact with revision when necessary. This may differ in a few cases, as projects such as CORDIS-ANIMA [18] and DIMPLE [22] simultaneously consider haptics and audio and the recent MSCI Platform actually computes both audio and haptics synchronously [23].

With respect to control, the use of interpolation from a two- to a many-dimensional space has already been used successfully in music [24]. The space metaphor, with closeness representing similarity between points, has shown itself to be flexible in use and easy for users to understand.

Methodology

Radial Basis Function Interpolation

In the approach described by Amorim et al. [25], set of $m$-dimensional points are used to characterize the resulting interpolation function. Each point $x_i$ has a corresponding location $y_i \in \mathbb{R}^2$ where closeness indicates similarity. A radial basis function, in our case the Gaussian function $\phi(r) = e^{-\epsilon r}$ where $\epsilon$ is shape, is used to define a mapping $s : \mathbb{R}^2 \rightarrow \mathbb{R}^m$ where for any point $p \in \mathbb{R}^2$ its corresponding point in the control space can be calculated as:

where the $\lambda_{i,j} \in \mathbb{R}^m$ are weights for the $j$th index of the vector given by the $i$th control point. The weights for each function $s_i(p)$ can be determined by solving

where $z_i$ are the $i$th index of the control points. As such, the weights necessary for $s(p)$ can be found by solving $m$ systems of equations.

System Architecture

A library1 to solve the instances of Equation 2 and calculate $s(p)$ was written in C++ using the Eigen2 library. The library uses two namespaces for its contents: Interpol for generic interpolation resources and Interpol::Gaussian for resources specific to use with a Gaussian radial basis function to permit future support of other radial basis functions. Vectors and two-dimensional locations were paired in the ControlPoint structure and functions were written to calculate the interpolation matrix, weight parameters and determine $s(p)$. These were wrapped into an external for Pure Data written using Flext3 where the interpolate object, initialized with the shape parameter $\epsilon$, performs the entire process. The object has two inlets and one outlet where the first expects a list of floating-point numbers representing coordinates of the interpolation point. The second expects a similar list representing the control points that define the interpolation. These are unpacked and transformed into an output of the interpolated vector where weight parameters are stored to prevent unnecessary recalculation.

Proof of Concept Patches

Two patches in Pure Data were created to try out the system described in Section 3.2. For these, audio and vibrotactile hardware was selected and used. Audio was simply delivered over wired headphones. Two Haptuator voice-coil actuators [26] were used for these patches and driven through a Sure Electronics AA-AB3213 class D amplifier. They were strapped to the inner wrist and forearm as in Figure 1. When only one Haptuator is in use, it is the one on the wrist.

Each patch permits a user to follow the same workflow, as illustrated in Figure 2. It progresses from specifying control vectors for the points that categorize the space, positioning the control and interpolation points, and then running the interpolation function to obtain a new control value based on the location of the interpolation point. While the patches control different parameters and work with different sized dimensional spaces, the external itself adjusts to the correct number of dimensions at runtime. Only parts of the patch for specifying and using the control space $\mathbb{R}^m$ need to be updated.

The first patch4 is used to control separate attack-decay-sustain-release ADSR envelopes for the audio and vibrotactile signals. Envelope was selected as it is a common parameter across both signals and is easy to visualize. The second patch5 controls continuous but different qualities of the signals. For the audio signal, fundamental frequency and relative amplitude of six harmonics are controlled by interpolation. For the vibrotactile signal, frequency, amplitude, and the position of the a phantom actual illusion between the two Haptuators are controlled. The method of creating this illusion permits separately controllable location and intensity [15].

Discussion and Future Work

Proof of concept testing showed a notable trend in design. Control over different features across modalities as in the second patch seemed more engaging to the authors than the same feature as in the first patch. While the first patch seemed to present two different effects together, the second moved towards a sense of a shared experience where the audio tone changed while the vibrotactile changed moved along the arm. This seems to suggest more perceptually interesting effects build upon cross-modal analogies, like the movement of the tone in both the audio and vibotactile modalities.

The Pd interface is currently fairly difficult to use. While functional for the purposes of testing, it would need refinement before more extensive use or distribution (for example online through WebPd [27]). The use of a keyboard and pointer as the only input devices is also potentially worth changing, for instance by developing haptic instruments [28]. Additionally, supporting unimodal control points for interpolation may prove interesting in certain applications.

Conclusion

A reverse radial basis function interpolation algorithm was implemented in C++ and used in a Pure Data external for the purpose of exploring simultaneous authoring of audio and vibrotactile effects. Two proofs of concept were developed to test this system focusing on separate design approaches. Limited testing of these prototypes emphasized the importance of cross-modal analogy in multimodal design. Experiencing effects that make use of these analogies through exploration rather than meticulous planning appeared interesting to the authors and seems encouraging of future investigation. Issues with the user interface and hardware require consideration before this, however, to ensure an environment conducive to authoring.

Acknowledgment

We would like to thank Jeremy R. Cooperstock and the Shared Reality Lab for providing the materials used in this project.