Researchers have shown that humans can detect the temperature of water using sound, an ability that likely develops as a result of exposure, pointing to the potential for creating new sensory maps of the brain and improving perceptual skills.

Researchers at the Ivcher Institute for Brain, Cognition and Technology (BCT Institute) at Reichmann University (IDC Herzliya) have used machine learning to shed light on the dynamics of cross-modal perception (the interaction between different senses), revealing a largely overlooked perceptual ability. In a recent study, the team investigated whether a person can determine thermal properties, such as the temperature of water, from their voice, and whether this perception occurs consciously.

Using the principles of multisensory integration (the way the brain combines information from different sensory modalities to create a unified perception of the environment), the team investigated the potential of multisensory thermal perception. They also used a pre-trained deep neural network (DNN) and a classification algorithm (support vector machine) to test whether machine learning could successfully and consistently classify sound recordings of pouring water at varying temperatures and create a map of these thermal properties physically encoded in the audio.

“The perception of temperature is quite unique compared to other sensory experiences,” says Dr. Adi Snir, a postdoctoral researcher at the BCT Institute and co-author of the study. “We have special sensory ‘organs’ such as the eyes and ears for vision and hearing; for temperature, we rely on special receptors in the skin that respond to different temperature ranges that we experience as warm and cool, but in the animal world we know that snakes, for example, can actually ‘see’ body heat, which allows them to identify prey.”

The question of whether multisensory temperature perception extends to humans has been raised before. “Previous studies have looked at this at a behavioural level,” says Professor Amir Amedi, founding director of the BCT Institute. “These studies showed that people can hear the difference between hot and cold liquids, but they didn’t understand how or why this was possible,” he explains.

Examination of multisensory perception

The researchers aimed first and foremost to replicate previous results and confirm this surprising perceptual ability, and also to find out whether this ability is innate or acquired – a question that has long been the subject of much debate. “We also wanted to investigate whether people are aware of these differences in the sound properties of thermal differences,” says Sneer, “and which features of the sounds allow them to differentiate their perception.”

To do this, the team used a pre-trained deep neural network (DNN) to characterize recordings of different temperatures of gushing water, a machine learning algorithm to classify the thermal properties of the water, and computational analysis of the auditory features of each recording. “We found that participants were consistently able to detect the temperature of the water from its sound, even if they didn’t believe they could do so, suggesting to us that this is likely an implicit skill acquired through exposure to auditory cues throughout life,” Amedi explains — at the same time, a machine learning model trained on recordings of hot and cold water showed high accuracy in sound classification.

The findings suggest that humans can learn complex sensory maps from everyday experiences, and that machine learning could help clarify subtle perceptual phenomena. “The next step is to see if humans create new sensory maps in the brain for this experience, just as they do for vision, touch and hearing,” Amedi says. “Theoretically, Elon Musk’s recent claims that Neuralink can create superhuman abilities could become reality if this method were combined with brain stimulation,” he adds.