Looking at images usually linked to a sound activates the superior temporal auditory cortex in a tenth of a second. The study carried out by the University of Milan-Bicocca and Ibfm-CNR reveals, besides other neurophysiological mechanisms, why seeing the labial fosters language comprehension and why the off-sync distorts it so heavily.
The auditory cortex is activated by vision too. Looking at pictures linked to a sound, for instance a saxophonist with his bulging cheeks blowing into his instrument, activates in only 110 milliseconds the superior temporal gyrus (Brodmann Area (BA) 38), the brain region devoted to auditory perception, and also involved in auditory hallucinations. This does not occur when the pictures lack any sound reference, such as the picture of a flock of fishes.
The close association of visual information with auditory information to which they are usually bound, as film-makers from the silent movies era knew even too well, has been discovered by Alice Mado Proverbio, a professor of Psychobiology at the University of Milan-Bicocca, and her research group, including Roberta Adorni, and Guido Edoardo D'Aniello, in close collaboration with Alberto Zani, a researcher of the Institute of Molecular Bioimaging and Physiology (IBFM). The findings of this study have been published in ScientificReports of Nature Group (see http://www.nature.com/srep/2011/110804/srep00054/full/srep00054. html).
This mechanism is based on audiovisual mirror neurons and allows our brain, for instance, to draw the image of a cat while listening to its mewing or the voice of a known person looking at his/her picture. Audiovisual neurons are also involved in perceptual phenomena like auditory hallucinations, they being directly activated by specific emotional states, like fear. Think, for instance, of when, biased by darkness, we believe to sense noises that we are afraid of, such as, for instance, creaking sounds, or foot step sounds, notwithstanding the still silence.
To compare brain activity related to a condition of pure visual perception with that related to the representation of an auditory image, 15 volunteers have been recruited, both males and females, without any neurological or psychiatric disorders, and not making use of any medicines or drugs.
To make the study objectively effective, this sample of volunteers was asked to perform a secondary task, such as pushing a button to infrequent images of a bicycle race, while on a TV screen a sequence of 300 colour photographs was presented, each one lasting a second and with an off-set of 1500-1900 milliseconds from each other. Despite the images were similar in luminance, size, affective values, and portrayed subjects, only one half elicited specific sounds, such as the crying of a child, a pneumatic hammer, a church bell ringing, or an opera singer.
The investigation has been carried out recording brain electrical activity by means of a 128 sensors high density montage placed all over the scalp. This methodology, called "event-related potentials" or ERPs, allows the measurement of intracranial neurons activation, changing as a function of visual stimulation.
Thanks to the use of ERPs intracranial source-reconstruction method called Loreta, the acronym for Low resolution electromagnetic tomography, based on the superimposition of mathematically computed cerebral sources onto three-dimensional magnetic resonance images (3D-MRI) of the brain, the study provided on-line information about the (millisecond-by-millisecond) temporal unfolding of activation of brain areas, while an individual is looking at a visual image.
Experimental data showed that the brain is able to extract sound information usually bound to visual images just starting one tenth of a second after their presentation, activating the superior temporal cortex, the inferior and middle temporal gyri, and, shortly after, the primary auditory cortex (BA41) too, in the same way as what occurs with real sounds perception, or with auditory hallucinations.
The existence of audiovisual synesthetic experiences in this region explains why the sight of the labial fosters language comprehension, and why talking on the phone may be very difficult in a noisy environment.
This is the first study offering evidence of the existence in the human being of audiovisual mirror neurons previously identified in the monkey brain.
Proverbio AM, D'Aniello GE, Adorni R, Zani A (2011). When a photograph can be heard: Vision activates the auditory cortex within 110 msec. www.Nature.com/ScientificReports. 1:54.DOI: 10.1038/srep00054.
Focus