Does Action-oriented Predictive Processing offer an enactive account of sensory substitution?

Krzysztof Dołęga — PhD stu­dent — Ruhr-Universität Bochum, Institut für Philosophie

Action-oriented Predictive Processing (PP for short, also known as Prediction Error Minimization or Predictive Coding) is an excit­ing con­cep­tu­al frame­work emer­ging at the cross­roads of cog­nit­ive sci­ence, stat­ist­ic­al mod­el­ing, inform­a­tion the­ory, and philo­sophy of mind. Aimed at obtain­ing a uni­fied explan­a­tion of the pro­cesses respons­ible for cog­ni­tion, per­cep­tion and action, it is based on the hypo­thes­is that the brain’s archi­tec­ture con­sists of hier­arch­ic­ally organ­ized neur­al pop­u­la­tions per­form­ing stat­ist­ic­al infer­ence. Rather than accu­mu­lat­ing and com­pound­ing incom­ing inform­a­tion, the neur­al hier­arch­ies con­tinu­ously form hypo­theses about their future input. Thus, the tra­di­tion­al bottom-up approach to explain­ing cog­ni­tion is sub­sumed by a top-down organ­iz­a­tion in which only sens­ory inform­a­tion diver­ging from the pre­dicted pat­terns of activ­a­tion is propag­ated up the cor­tic­al hier­arch­ies. Due to its diver­gence from the ‘pre­dicted’ pat­terns, this inform­a­tion is often referred to as “pre­dic­tion error” (Clark, 2013). Minimizing error is pos­tu­lated to be the main func­tion of the brain; by accom­mod­at­ing unex­pec­ted inform­a­tion in its pre­dic­tions the brain can fine-tune its future hypo­theses regard­ing sens­ory input and track the states of the world caus­ing this input more accurately.

However, most of the frame­work’s appeal lies with the rel­at­ively recent pro­pos­al that the brain can min­im­ize error not only by revis­ing and con­struct­ing new hypo­theses about the input pat­terns, but also by inter­act­ing with the envir­on­ment in order to erase the source of mis­match between the best hypo­thes­is and pat­terns of sens­ory activ­a­tion. This fea­ture, referred to as “active-inference” (e.g. Hohwy, 2012), is primar­ily respons­ible for much of the frame­work’s appeal and its prom­ise of a uni­fied the­ory of brain organ­iz­a­tion, explain­ing how inform­a­tion about many dif­fer­ent cog­nit­ive func­tions is encoded and pro­cessed in the brain (Friston, 2010).

In my poster present­a­tion at the first iCog con­fer­ence, I tried to draw sim­il­ar­it­ies between Predictive Processing and anoth­er rad­ic­al pro­pos­al about the nature of per­cep­tion and cog­ni­tion – enact­iv­ism. Because of the rel­at­ive nov­elty of the PP frame­work, its rela­tion­ship to the embod­ied and sen­sor­imo­tor approaches to cog­ni­tion and per­cep­tion has not been well defined (at least at the time of the con­fer­ence, see the bib­li­o­graphy below for sev­er­al recent art­icles tack­ling these issues). What struck me as an inter­est­ing aven­ue for research was the sim­il­ar­ity between the notion of active-inference on the PP frame­work and the enact­ive focus on the role sen­sor­imo­tor con­tin­gen­cies and pos­sib­il­it­ies for action play in shap­ing per­cep­tion and phe­nomen­o­logy. Pursuing this cor­res­pond­ence is espe­cially valu­able for PP, as it does not yet offer a clear account of how phe­nomen­o­logy fits with­in its prob­ab­il­ist­ic archi­tec­ture. Due to perception’s breadth as a top­ic, I decided to focus on a very par­tic­u­lar case of Sensory Substitution Devices.

Sensory sub­sti­tu­tion devices emerged from the labor­at­ory led by Paul Bach-y-Rita in the ’60s. Bach-y-Rita (1983) set out to prove the extent of lifelong brain plas­ti­city (a highly con­tested thes­is at the time) by devis­ing gad­gets that would help han­di­capped people restore lost senses by sub­sti­tut­ing them with inputs com­ing from dif­fer­ent sens­ory mod­al­it­ies. The idea behind the pro­ject was to use the brain’s nat­ur­al abil­ity to adapt to the inputs it receives from dif­fer­ent sens­ory chan­nels in order to train it to recog­nize inform­a­tion spe­cif­ic to the lost mod­al­ity in pat­terns delivered through a dif­fer­ent sens­ory mod­al­ity. Bach-y-Rita’s work focused on tactile-visual sens­ory sub­sti­tu­tion (TVSS for short), in which visu­al inform­a­tion from a video cam­era was trans­lated into vibro-tactile input on the sub­jects skin. Despite its lim­it­a­tions, this meth­od proved to be a huge suc­cess, as sub­jects were able to learn to extract vision-like inform­a­tion (e.g. a pres­ence of a white X in front of the cam­era) from tact­ile stim­u­la­tion after a sur­pris­ingly short adapt­a­tion time. This dis­cov­ery jump­star­ted a whole new field of research. Below is a video of a recent TVSS device:

TVSS proved to be a fer­tile study ground for enact­iv­ism due to lim­it­a­tions and prob­lems inher­ent to the pro­ject of sens­ory sub­sti­tu­tion. Very early into his research, Bach-y-Rita dis­covered that sub­sti­tu­tion is mostly unsuc­cess­ful when sub­jects do not have con­trol over the cam­era move­ments. The crit­ic­al import­ance of explor­a­tion and act­ive sampling for TVSS fits well with the core enact­ive claim that per­cep­tion con­sists in ‘exer­cising a mas­tery of sen­sor­imo­tor con­tin­gen­cies’ (O’Regan & Noë 2001: 85), under­stood as prac­tic­al know-how about the pos­sible changes in per­cep­tion of objects caused by our actions (O’Regan & Noë 2001: 99). Moreover, O’Regan & Noë have argued that the con­tin­gen­cies of how our sens­ory mod­al­it­ies sample and inter­act with par­tic­u­lar objects explain cer­tain fea­tures of the brain’s plas­ti­city. For example, it is because of the dynam­ics par­tic­u­lar to our visu­al involve­ment with the world that the blind TVSS sub­jects show increased activ­ity in the visu­al cor­tex and can be said to genu­inely see (although in some impov­er­ished way, TVSS does not allow for col­or per­cep­tion). To sup­port this con­tro­ver­sial claim they point to neur­o­lo­gic­al data, as well as exper­i­ments demon­strat­ing sub­jects’ gull­ib­il­ity to dis­tinct­ively visu­al illu­sions exploit­ing the basic prop­er­ties of visu­al engage­ment with the envir­on­ment, such as mak­ing per­cep­tu­al con­tact only with the sur­faces facing the observ­er (Hurley & Noë 2003: 143).

Let us now return to Action-oriented Predictive Processing and how the frame­work can accom­mod­ate sens­ory sub­sti­tu­tion. PP assumes that the main func­tion of the brain is min­im­iz­a­tion of pre­dic­tion error res­ult­ing from com­par­ing actu­al sens­ory inputs with pre­dicted pat­terns of activ­a­tions gen­er­ated by the sys­tem. To pre­dict the sens­ory states effi­ciently, the brain tracks pat­terns of stat­ist­ic­al reg­u­lar­it­ies present in the incom­ing sig­nals and tries to infer the caus­al struc­ture of the world respons­ible for these reg­u­lar­it­ies. Thus the brain con­structs and main­tains a mod­el of the world, which it uses to pre­dict (i.e. gen­er­ate) its own sens­ory states. The par­tic­u­lars of this pro­pos­al are much more com­plex (I recom­mend Jakob Hohwy’s 2013 mono­graph for details), but these core ideas are suf­fi­cient to under­stand how PP can explain sens­ory substitution.

On Action-oriented Predictive Processing, what hap­pens dur­ing TVSS is a res­ult of the hier­arch­ic­al sys­tem recog­niz­ing and sub­sequently track­ing a set of stat­ist­ic­al reg­u­lar­it­ies spe­cif­ic to the visu­al mod­al­ity in sens­ory pat­terns delivered through tact­ile stim­u­la­tion. The sub­jects need to have con­trol over the input device (here a cam­era) in order to learn how the reg­u­lar­it­ies in the sens­ory stim­u­lus change with the sampling of the envir­on­ment. Having done this, the brain can pre­dict how the stim­u­lus will change in response to par­tic­u­lar actions, updat­ing its gen­er­at­ive mod­el accordingly.

The core of the PP explan­a­tion of TVSS is thus very sim­il­ar to the enact­ive treat­ment. In both cases it is the sys­tem’s abil­ity to recog­nize pos­sib­il­it­ies for action and the abil­ity to pre­dict how these actions will change the states of the sens­ory input that make sens­ory sub­sti­tu­tion pos­sible. One could try and push this sim­il­ar­ity fur­ther by say­ing that in PP, just like in enact­iv­ism, it is the sen­sor­imo­tor con­tin­gen­cies that shape the phe­nom­en­al qual­ity of the sub­sti­tu­tion. After all, the sys­tem tracks dis­tinct­ively visu­al reg­u­lar­it­ies obtain­ing between the body and the world. From the per­spect­ive of the brain, the man­ner of their present­a­tion (via tact­ile stim­u­lus vs ocu­lar nerves) plays a sec­ond­ary role to their con­tents (in both cases the brain has to infer the causes behind the pat­terns of sens­ory activations).

Despite these sim­il­ar­it­ies, one should be care­ful about cast­ing PP as sub­scrib­ing to an enact­ive under­stand­ing of per­cep­tion and sens­ory sub­sti­tu­tion. Though the views in ques­tion do over­lap in their explan­at­ory ambi­tions, they are built on dia­met­ric­ally oppos­ing assump­tions. In the pre­vi­ous para­graph I tried to speak about ‘the sys­tem’ rather than the brain or agent as a whole. This is because PP is usu­ally under­stood as a neuro­centric view (Hohwy, 2014), while enact­iv­ism instead stresses the situ­ated and embod­ied nature of cog­ni­tion (Noë, 2005). Moreover, PP is based on an infer­en­tial archi­tec­ture, often asso­ci­ated with rich rep­res­ent­a­tion­al con­tents – some­thing widely eschewed by enactivists.

The divide between the two pos­i­tions is not unsur­mount­able and much of present work by Andy Clark is focused on bridging the gap between these rad­ic­al views (see Clark’s forth­com­ing book). This post does not allow me to dive into the nuances of both views and how sim­il­arly they treat TVSS and ana­log­ous cases; how­ever, I hope I man­aged to spark some interest in these rad­ic­al views about per­cep­tion and how they may be related. Below is a list of ref­er­ences, some of which were unavail­able at the time of the ori­gin­al present­a­tion of this material.



Bach-y-Rita, P. (1983). Tactile Vision Substitution: Past and Future. International Journal of       Neuroscience 19: 29–36.

Briscoe, R. (forth.). Bodily Action and Distal Attribution in Sensory Substitution. [online] Available from: [Retrived: 21, Nov. 2013].

Clark, A. (2013). Whatever next? Predictive brains, situ­ated agents, and the future of cog­nit­ive sci­ence. Behavioral and Brain Science 36(3): 181–204.

Friston, K. (2010). The free-energy prin­ciple: A uni­fied brain the­ory?. National Review of Neuroscience 11(2):127–138.

Friston, K. (2008). Hierarchical Models in the Brain, PLoS Computational Biology 4(11)      doi:10.1371/journal.pcbi.1000211.

Hohwy, J. (2014). Self-evidencing Brain. Nous48(1). doi: 10.1111/nous.12062

Hohwy, J. (2013). The Predictive Mind. Oxford: Oxford University Press.

Hohwy, J. (2012). Attention and con­scious per­cep­tion in the hypo­thes­is test­ing brain. Frontiers in         Psychology 3(96). doi: 10.3389/fpsyg.2012.00096.

Hurley, S. & Noë, A. (2003). Neural Plasticity and Consciousness. Biology and Philosophy 18: 131–168.

O’Regan, J.K. & Noë, A. (2001). What is it like to see: A sen­sor­imo­tor the­ory of per­cep­tu­al     exper­i­ence. Synthese 129(1): 79–103 .

Noë, A. (2005). Action in Perception. Cambridge, MA: MIT Press.

Pepper, K. (2013). Do Sensorimotor Dynamics Extend The Conscious Mind?. Adaptive Behavior           22(2): 99–108.

Pickering, M., Clark, A. (2014). Getting Ahead: Forward Models and their place in Cognitive Architecture. Trends in Cognitive Sciences 18(9): 451–456.

Prinz, J. (2009). Is Consciousness Embodied? in Robbins, P. & Aydede, M. (Eds.) Cambridge   Handbook of Situated Cognition. Cambridge: Cambridge University Press.

Rietveld, E., Bruineberg, J. (2014). Self-organization, free energy min­im­iz­a­tion, and optim­al grip           on a filed of afford­ances. Front. Hum. Neurosci 8(599). doi: 10.3389/fnhum.2014.00599.

Ward, D. (2012). Enjoying the Spread: Conscious Externalism Reconsidered. Mind 121(483):731- 751.