Multisensory Perception

Alisa Mandrigin is a post-doctoral research­er on the AHRC Science in Culture Theme Large Grant, Rethinking the Senses: Uniting the Neuroscience and Philosophy of Perception. She is based in the Department of Philosophy at the University of Warwick.

We see, hear, touch, taste and smell. This is what com­mon sense tells us about per­cep­tion. The view that the nature and breadth of our per­cep­tu­al exper­i­ence is accur­ately cap­tured by those five verbs and that the sens­ory sys­tems are dis­crete and isol­ated from one anoth­er has gov­erned much of the philo­soph­ic­al research into per­cep­tion. Recently, though, research­ers in psy­cho­logy and neur­os­cience have star­ted to focus on the myri­ad inter­ac­tions between the senses and the many oth­er kinds of sens­ory inform­a­tion avail­able to the nervous system.

How do we know that the sens­ory sys­tems inter­act with one anoth­er? Some inter­ac­tions res­ult in effects that fea­ture in every­day exper­i­ence. For example, if we are presen­ted with a light flash and a beep at dif­fer­ent loc­a­tions but at the same time and then asked to loc­ate the beep, we judge it to be at, or at least close to, the loc­a­tion of the light flash (Bertelson, 1999). Judgements about loc­a­tion are one of the meas­ures of the Ventriloquism effect: the mis-location in per­cep­tu­al exper­i­ence of aud­it­ory objects or events as a res­ult of see­ing some­thing at a dif­fer­ent location.

We can meas­ure the effect in the labor­at­ory with visu­al and aud­it­ory stim­uli, but in every­day life we are often in situ­ations in which we acquire spa­tially dis­crep­ant inform­a­tion about what is appar­ently the same source. Ventriloquists make use of the effect in their acts, hence the name giv­en to the effect. At the cinema the film’s soundtrack is played through speak­ers spread out around the screen­ing room, and not from behind the parts of the screen on which visu­al images of mov­ing lips, explo­sions, and so on are presented.

Another visual-auditory inter­ac­tion gives rise to the McGurk effect. If we see a video clip of lip move­ments that should pro­duce the phon­eme /ga/ with an audio record­ing of the phon­eme /ba/ dubbed over the top, the res­ult is per­cep­tion of the phon­eme /da/ (McGurk & MacDonald, 1976). Again, it seems that pro­cessing in the visu­al sys­tem influ­ences pro­cessing in the aud­it­ory sys­tem. We can appre­ci­ate this by listen­ing to the same aud­it­ory stim­u­lus with our eyes closed: without the visu­al stim­u­lus there is no effect. You can try it for your­self by view­ing this video:

Multisensory inter­ac­tions are not lim­ited to vis­ion and audi­tion. We have evid­ence of inter­ac­tions between all five of the sens­ory sys­tems, as tra­di­tion­ally con­ceived. It’s only now, though, that philo­soph­ers are begin­ning to take prop­er notice of the implic­a­tions of these dis­cov­er­ies for per­cep­tu­al exper­i­ence. This brings us to a ques­tion that seems to be import­ant if we are to make sense of these inter­ac­tions and their con­sequences for per­cep­tu­al exper­i­ence: how do we dis­tin­guish the senses? Can we dis­tin­guish the senses on the basis of the dif­fer­ent kinds of exper­i­ence pro­duced, or should we dis­tin­guish them on the basis of dis­tinct sens­ory pro­cessing sys­tems in the brain, or by means of the nature of the prox­im­al stim­u­lus of the exper­i­ence, or by some­thing else entirely (Macpherson, 2011)? There’s an ana­log­ous ques­tion about kinds of exper­i­ence: how can we dis­tin­guish exper­i­ences from one anoth­er as being, for example, visu­al or auditory?

Settling on an answer to these ques­tions seems to be neces­sary if we are to make any head­way in clas­si­fy­ing inter­ac­tions as multi­s­ens­ory and decid­ing wheth­er these inter­ac­tions res­ult in mul­timod­al per­cep­tu­al experiences.

For example, our per­cep­tu­al exper­i­ence when we eat and drink involves retro-nasal smell—the sens­ing of odours when we breathe out—as well as taste. When you’ve had a blocked nose you’ve prob­ably noticed this, find­ing food to be tem­por­ar­ily fla­vor­less and insip­id. One response to this has been to claim that we have a dis­tinct kind of fla­vor exper­i­ence, res­ult­ing from inter­ac­tions between the olfact­ory and taste sys­tems (Smith, 2013). This view con­ceives of the exper­i­ence as multi­s­ens­ory in so far as it involves pro­cessing in two dis­tinct sens­ory sys­tems, but the exper­i­ence itself is not taken to be mul­timod­al since it is thought of as being a kind of exper­i­ence in it’s own right, dis­tinct from either smelling or tast­ing. The mat­ter is com­plic­ated fur­ther by evid­ence that what we see and hear, and tact­ile sen­sa­tions with­in the mouth also con­trib­ute to our per­cep­tu­al exper­i­ence when we eat (Auvray & Spence, 2008).

Even if we can settle on a way of dis­tin­guish­ing the senses, there are fur­ther ques­tions about the kinds of inter­ac­tions that take place between the sens­ory sys­tems. One kind of inter­ac­tion might involve mere mod­u­la­tion of pro­cessing in one sens­ory sys­tem by pro­cessing in the oth­er. Another kind of inter­ac­tion might involve the integ­ra­tion of redund­ant inform­a­tion across the senses. A fur­ther kind of inter­ac­tion might involve the bind­ing togeth­er of inform­a­tion about dif­fer­ent prop­er­ties of the same object. For example, when you look at a key that you hold in your hand, visu­al inform­a­tion about col­our might be bound togeth­er with tact­ile inform­a­tion about tex­ture, gen­er­at­ing a multi­s­ens­ory rep­res­ent­a­tion of the key as smooth and sil­ver (O’Callaghan, 2014). These dif­fer­ent kinds of inter­ac­tion may have dif­fer­ent kinds of impact on per­cep­tu­al exper­i­ence. What, for example, is the nature of the inter­ac­tion between vis­ion and audi­tion in vent­ri­lo­quism and how does it impact per­cep­tu­al experience?

One approach to vent­ri­lo­quism explains the effect in terms of the mod­u­la­tion of inform­a­tion in audi­tion by inform­a­tion in vis­ion. Ventriloquism is often meas­ured by sub­jects’ point­ing responses to the aud­it­ory stim­u­lus. Subjects point to a pos­i­tion in between the actu­al loc­a­tions of the aud­it­ory and the visu­al stim­uli. How can we explain this in terms of mod­u­la­tion? We can say that the con­flict­ing visu­al inform­a­tion about loc­a­tion mod­i­fies the aud­it­ory inform­a­tion about loc­a­tion (and vice versa). The res­ult is that sub­jects hear the aud­it­ory stim­u­lus as being in between the actu­al pos­i­tion of the aud­it­ory and the visu­al stim­uli. This explan­a­tion of vent­ri­lo­quism is con­sist­ent with per­cep­tu­al exper­i­ences remain­ing modality-specific throughout.

There is, how­ever, an altern­at­ive explan­a­tion of the mis-localisation of aud­it­ory stim­uli in vent­ri­lo­quism. This altern­at­ive explains sub­jects’ point­ing beha­vi­or in terms of the integ­ra­tion of con­flict­ing spa­tial inform­a­tion. If sens­ory inform­a­tion is integ­rated, it seems pos­sible that this integ­ra­tion will res­ult in a single mul­timod­al exper­i­ence of an object at a loc­a­tion in space, in this case an audio-visual exper­i­ence. If there is integ­ra­tion (or bind­ing) of inform­a­tion across the senses, then we need to give some account of how the sens­ory sys­tems determ­ine that inform­a­tion belongs together.

The issues I’ve men­tioned here offer just one aven­ue that we can pur­sue in rethink­ing and revis­ing our views of per­cep­tu­al exper­i­ence in light of empir­ic­al dis­cov­er­ies about multi­s­ens­ory pro­cessing. Another aven­ue con­cerns cross­mod­al cor­res­pond­ences. We reli­ably match, for example, high pitch sounds with bright lights, high spa­tial elev­a­tions or small objects (Spence, 2011). How, though, are these asso­ci­ations between what seem to be dif­fer­ent kinds of prop­er­ties estab­lished? Are pairs or groups of appar­ently unre­lated fea­tures of objects, or dimen­sions of stim­uli, encoded in the brain in the same way?

A fur­ther line of research con­cerns syn­aes­thesia. In some cases of syn­aes­thesia an exper­i­ence in one sens­ory mod­al­ity seems to induce an exper­i­ence in anoth­er, non-stimulated sens­ory mod­al­ity. For instance, for some syn­aes­thetes, hear­ing sounds causes them to have col­our exper­i­ences. Franz Liszt and Olivier Messiaen reportedly exper­i­enced col­ours when they heard par­tic­u­lar tones in this way. As with cross­mod­al cor­res­pond­ences, syn­aes­thet­ic exper­i­ence is reli­able and robust: hear­ing par­tic­u­lar tones con­sist­ently induces exper­i­ences of par­tic­u­lar col­our hues. How do we explain the phe­nomen­on? Do syn­aes­thetes have two dis­tinct modality-specific experiences—an aud­it­ory exper­i­ence and a col­our exper­i­ence, for example—or are their exper­i­ences alto­geth­er dif­fer­ent, exper­i­ences of col­oured sounds, for instance (Deroy, in press)?

We are just now start­ing to under­stand the many and var­ied inter­ac­tions that occur across the sens­ory sys­tems and their impact on per­cep­tu­al exper­i­ence. What is clear, though, is that the multi­s­ens­ory nature of per­cep­tu­al exper­i­ence is rel­ev­ant to all us, not just to those who work on the philo­sophy of per­cep­tion or in psy­cho­logy, or to those who work in the arts or in mar­ket­ing, but to all of us, simply because we are perceivers.




Auvray, M. & Spence, C. (2008). The multi­s­ens­ory per­cep­tion of fla­vour. Consciousness & Cognition. 17. p. 1016–1031. doi:10.1016/j.concog.2007.06.005

Bertelson, P. (1999). Ventriloquism: a case of cross-modal per­cep­tu­al group­ing. In Aschersleben, G., Bachmann, T., & Müsseler, J. (eds.). Cognitive con­tri­bu­tions to the per­cep­tion of spa­tial and tem­por­al events. Amsterdam: Elsevier.

Deroy, O. (in press). Can sounds be red? A new account of syn­aes­thesia as enriched exper­i­ence. In Coates, P. & Coleman, S. (eds.). Phenomenal qual­it­ies. Oxford: Oxford University Press.

Macpherson, F. (2011). Cross-modal exper­i­ences. Proceedings of the Aristotelian Society. 111 (3). p. 429 – 468. doi:10.1111/j.1467–9264.2011.00317.x

McGurk, H. & MacDonald, J. (1976). Hearing lips and see­ing voices. Nature. 264. p. 746 – 748. doi:10.1038/264746a0

O’Callaghan, C. (2014). Not all per­cep­tu­al exper­i­ence is mod­al­ity spe­cif­ic. In Stokes, D., Matthen, M. & Biggs, S. (eds.) Perception and Its Modalities. Oxford: OUP.

Smith, B. C. (2013). Philosophical Perspectives on Taste. In Pashler, H. (ed.). The Encyclopaedia of Mind. Newbury Park, CA.: Sage.

Spence, C. (2011). Crossmodal cor­res­pond­ences: A tutori­al review. Attention, Perception, & Psychophysics. 73. p. 971–995. doi:10.3758/s13414-010‑0073‑7

One thought on “Multisensory Perception”

Comments are closed.