Breakthrough new technology lets people listen to music with touch

rainbow of notes

The prototype consists of an audio-haptic algorithm that uses a “tactile illusion” to convert monophonic music into tangible vibration-based stimuli.

An audio haptic algorithm created by scientists at the University of Malaga conveys melodic information through vibrations.

A breakthrough prototype developed by experts from the Department of Electronics of the University of Malaga and members of the research and development group “Instrument and System Electronics” will allow hearing-impaired people to listen to music through the sense of touch.

It consists of an audio-haptic algorithm that uses a “tactile illusion” to transform monophonic music into tangible vibration-based stimuli. According to the researchers, “it’s like ‘hacking’ the nervous system to receive different responses to real stimuli sent.”

“In the long run, what we want to achieve is to enable music to be ‘heard’ by those who cannot hear it,” assures the paper’s lead author, researcher Paul Remache, who insists that Music has the power to affect emotions, as well as its potential as a treatment for mental disorders and pain therapy.

The researchers predict that this will lead to portable terminals that can be brought to concerts, as the prototype can be easily transferred to technological devices such as smartphones.


Researchers in the Department of Electronics have developed an audio haptic algorithm that conveys melodic information through vibrations.Image credit: University of Malaga

map music

In collaboration with UMA professors Andrés Trujillo and Fernando Vidal, the young researcher developed an algorithm that converts musical characteristics and structures taken from MIDI files (Musical Instrument Digital Interface) into “vibrotactile stimuli”.

Prototype music by touch

Initial experiments proved that the “tactile illusion” perceived through the vibrations evoked a different emotional response—mainly positive—from those who hadn’t heard the original music.Image credit: University of Malaga

“It’s similar to mapping music,” explains Remache, who adds that it’s possible because this type of file not only plays and generates sound, but also provides a “symbolic representation.”

control vibration

Current models cannot guarantee a correspondence between an emotional response to music and its vibro-haptic version. With this in mind, these engineers at UMA came up with an arrangement of “tactile illusions” to improve and expand the spectrum of musical signatures, adding dynamics to vibrations in the form of changes in motion, direction and position.

“This is a challenging procedure because the perceptible frequency range of the skin is lower than that of the auditory system, which may result in the loss of certain musical characteristics”, they explain.

Department of Electronics, University of Malaga

People with hearing loss will be able to hear music through the sense of touch, thanks to a pioneering prototype designed by researchers at the Department of Electronics at the University of Malaga.Image credit: University of Malaga

different emotional responses

The results of the first experiment involving more than 50 volunteers showed that the “tactile illusion” arrangement elicited more positive emotions than negative ones. They are also considered more enjoyable and stimulating than audio, eliciting a different emotional response than the original music.

Smart Instrumentation and Its Application in Healthcare

The first prototype was presented at the 11th International Symposium on Haptic and Audio Interaction Design (UK) – the largest international event dedicated to these areas of research – after publication in a scientific journal LNCS. Currently, researchers at UMA are working on a second model and continuing their experiments.

The research is the result of Paul Remache’s doctoral dissertation as part of the national program project “Smart Instruments and Their Applications in Healthcare”.

Reference: “Using the Haptic Illusion to Map Monophonic MIDI Tracks to Vibrations” by Byron Remache-Vinueza, Andrés Trujillo-León, Maria-Alena Clim, Fabián Sarmiento-Ortiz, Liliana Topon-Visarrea, Alexander Refsum Jensenius, and Fernando Vidal-Verdú Tactile Stimulation”, 18 August 2022, Computer Science Lecture Notes.
DOI: 10.1007/978-3-031-15019-7_11



Source link