Can a visual experience be biased?

by Jessie Munton — Junior Research Fellow at St John’s College, Cambridge

Beliefs and judge­ments can be biased: my expect­a­tions of someone with a London accent might be biased by my pre­vi­ous expos­ure to Londoners or ste­reo­types about them; my con­fid­ence that my friend will get the job she is inter­view­ing for may be biased by my loy­alty; and my sus­pi­cion that it will rain tomor­row may be biased by my expos­ure to weath­er in Cambridge over the past few days. What about visu­al exper­i­ences? Can visu­al exper­i­ences be biased?

That’s the ques­tion I explore in this blog post. In par­tic­u­lar, I’ll ask wheth­er a visu­al exper­i­ence could be biased, in the sense of exem­pli­fy­ing forms of racial pre­ju­dice. I’ll sug­gest that the answer to this ques­tion is a tent­at­ive “yes”, and that that presents some nov­el chal­lenges to how we think of both bias and visu­al per­cep­tion.

According to a very simplist­ic way of think­ing about visu­al per­cep­tion, it presents the world to us just as it is: it puts us dir­ectly in touch with our envir­on­ment, in a man­ner that allows it to play a unique, pos­sibly found­a­tion­al epi­stem­ic role. Perception in gen­er­al, and visu­al exper­i­ence with it, is some­times treated as a kind of giv­en: a sourceof evid­ence that is immune to the sorts of ration­al flaws that beset our cog­nit­ive responses to evid­ence. This approach encour­ages us to think of visu­al exper­i­ence as a neut­ral cor­rect­ive to the kinds of flaws that can arise in belief, such as bias or pre­ju­dice: there is no room­in the pro­cesses that gen­er­ate visu­al exper­i­ence for the kinds of influ­ence that cause belief to be biased or pre­ju­diced.

But there is a ten­sion between that view and cer­tain facts about the sub­per­son­al pro­cesses that sup­port visu­al per­cep­tion in creatures like ourselves. In par­tic­u­lar, our visu­al sys­tem faces an under­de­termin­a­tion chal­lenge: the light sig­nals received by the ret­ina fail, on their own, to determ­ine a unique extern­al stim­u­lus (Scholl 2005). To resolve the res­ult­ing ambi­gu­ity, the visu­al sys­tem must rely on pri­or inform­a­tion about the envir­on­ment, and likely stim­uli with­in it. But those pri­ors are not fixed and immut­able: the visu­al sys­tem updates them in light of pre­vi­ous exper­i­ence (Chalk et al 2010, Chun &Turk-Browne 2008). In this way, the visu­al sys­tem learns from the idio­syn­crat­ic course that the indi­vidu­al takes through the world.

Equally, the visu­al sys­tem is over­whelmed with pos­sible input: the inform­a­tion avail­able from the envir­on­ment at any one moment far sur­passes what the brain can pro­cess (Summerfield & Egner 2009). It must select­ively attend to cer­tain objects or areas with­in the visu­al field, in order to pri­or­it­ise the highest value inform­a­tion. Preexisting expect­a­tions and pri­or­it­ies determ­ine the sali­ence of inform­a­tion with­in a giv­en scene. The nature and con­tent of the visu­al exper­i­ence you are hav­ing at any moment in part depends on the rel­at­ive value you place on the inform­a­tion in your envir­on­ment.

We per­ceive the world, then, in light of our pri­or expect­a­tions, and past expos­ure to it. Those pro­cesses of learn­ing and adapt­a­tion, of devel­op­ing skills that fit a par­tic­u­lar envir­on­ment­al con­text, leave visu­al per­cep­tion vul­ner­able to a kind of visu­al coun­ter­part to bias: we do not come to the world each time with fresh eyes. If we did, we would see less accur­ately and effi­ciently than we do.

Cognitive biases often emerge as a response to par­tic­u­lar envir­on­ment­al pres­sures: they per­sist because they lend some advant­age in cer­tain cir­cum­stances, but come at the expense of sens­it­iv­ity to cer­tain oth­er inform­a­tion (Kahneman & Tversky 1973). Similarly, the capa­city of the visu­al sys­tem to devel­op an expert­ise with­in a par­tic­u­lar con­text can restrict its sens­it­iv­ity to cer­tain sorts of inform­a­tion. We can see this kind of struc­ture in the spe­cial­ist abil­it­ies we devel­op to see faces.

You might nat­ur­ally think that we per­ceive high-level fea­tures of faces, such as the emo­tion they dis­play or the racial cat­egory they belong to, not dir­ectly, but only in vir­tue of, or per­haps via some kind of sub­per­son­al infer­ence from, their lower-level fea­tures: the arrange­ment of facial fea­tures, for instance, or the col­or and shad­ing that let us pick out those fea­tures. In fact, there’s good evid­ence that we per­ceive the social cat­egory of a face, or the emo­tion it dis­plays dir­ectly. For instance, we demon­strate “visu­al adapt­a­tion” to facial emo­tion: after see­ing a series of angry faces, a neut­ral face appears happy. And those adapt­a­tion effects are spe­cif­ic to the gender and race of the face,suggesting that these cat­egor­ies of faces may be coded for my dif­fer­ent neur­al pop­u­la­tions (Jaquet, Rhodes, & Hayward 2007, 2008; Jacquet & Rodes 2005, Little, DeBruine, & Jones 2005).

Moreover, our skills at face per­cep­tion seem to be sys­tem­at­ic­ally arranged along racial lines: most people are bet­ter at recog­niz­ing own-race and dominant-race faces, (Meissner & Brigham 2001), the res­ult of a pro­cess of spe­cial­isa­tion that emerges over the first 9 months of life as infants gradu­ally lose the capa­city to recog­nize faces of dif­fer­ent or non-dominant races (Kelly et al. 2007). A White adult in a major­ity white soci­ety will gen­er­ally be bet­ter at recog­niz­ing oth­er white faces than Black or Asian faces, for instance, where­as a Black per­son liv­ing in a major­ity Black soci­ety will con­versely be less good at recog­niz­ing White than Black faces.  This extends to the iden­ti­fic­a­tion of emo­tion from faces, as well as their recog­ni­tion: sub­jects are more accur­ate at identi­fy­ing the emo­tion dis­played on dom­in­ant or same-race faces than other-race faces (Elfenbeim & Ambady 2002).

One way of under­stand­ing this pro­file of skills is to think of faces as arranged with­in a mul­ti­di­men­sion­al “face space” depend­ing on their sim­il­ar­ity to one anoth­er. We hone our per­cep­tu­al capa­cit­ies with­in that area of face space to which we have most expos­ure. That area of face space becomes, in effect, stretched, allow­ing for finer grained dis­tinc­tions between faces. (Valentine 1991; Valentine, Lewis and Hills 2016). The great­er “dis­tance” between faces in the area of face space in which we are most spe­cial­ized renders those faces more mem­or­able and easi­er to dis­tin­guish from one anoth­er. Another way of think­ing of this is in terms of “norm-based cod­ing” (Rhodes and Leopold 2011): faces are encoded rel­at­ive to the aver­age face encountered. Faces fur­ther from the norm suf­fer in terms of our visu­al sens­it­iv­ity to the inform­a­tion they carry.

On the one hand, it isn’t hard to see how this kind of facial expert­ise could help us extract max­im­al inform­a­tion from the faces we most fre­quently encounter.  But the impact of this “same-race face effect” more gen­er­ally is poten­tially highly prob­lem­at­ic: a White per­son in a major­ity White soci­ety will be less likely to accur­ately recog­nise a Black indi­vidu­al, and less able to accur­ately per­ceive their emo­tions from their face. That diminu­tion of sens­it­iv­ity to faces of dif­fer­ent races paves the way for a range of down­stream impacts. Since the visu­al sys­tem fails to advert­ise this dif­fer­en­tial sens­it­iv­ity, the indi­vidu­al is liable to reas­on as though they have read their emo­tions with equal per­spicu­ity, and to draw con­clu­sions on that basis (that the indi­vidu­al feels less per­haps, when the emo­tion in ques­tion is simply visu­ally obscure to them). Relatedly, the lack of inform­a­tion extrac­ted per­cep­tu­ally from the face makes it more likely that the indi­vidu­al will fill that short­fall of inform­a­tion by draw­ing on ste­reo­types about the rel­ev­ant group: that Black people are aggress­ive, for instance, (Shapiro et al. 2009; Brooks and Freeman 2017). And restric­tions on the abil­ity to accur­ately recall cer­tain faces will bring with them social costs for those indi­vidu­als.

Compare this visu­al bias to someone writ­ing a report about two indi­vidu­als, one White and one Black. The report about the White per­son is detailed and accur­ate, whilst the report on the Black per­son is much spars­er, lack­ing inform­a­tion rel­ev­ant to down­stream tasks. In such a case, we would reas­on­ably regard the report writer as biased, par­tic­u­larly if their report writ­ing reflec­ted this kind of dis­crep­ancy between White and Black tar­gets more gen­er­ally. If the visu­al sys­tem dis­plays a struc­tur­ally sim­il­ar bias in the inform­a­tion it provides us with, should we regard it, too, as biased?

To answer that ques­tion, we need to have an account of what it is for any­thingto be biased, be it a visu­al exper­i­ence, a belief, or a dis­pos­i­tion to behave or reas­on in some way or oth­er. We use ‘bias’ in many dif­fer­ent ways. In par­tic­u­lar, we need to dis­tin­guish here what I call form­al bias from pre­ju­di­cial bias. In cer­tain con­texts, a bias may be rel­at­ively neut­ral. A ship might be delib­er­ately giv­en a bias to list towards the port side, for instance, by uneven dis­tri­bu­tion of bal­last. Similarly, any sys­tem that resolves ambi­gu­ity in incom­ing sig­nal on the basis of inform­a­tion it has encountered in the past is biased by that pri­or inform­a­tion. But that’s a bias that, for the most part, enhances rather than detracts from the accur­acy of the res­ult­ing judge­ments or rep­res­ent­a­tions.  We could call biases of this kind form­al biases.

Bias also has anoth­er, more col­lo­qui­al usage, accord­ing to which it picks out some­thing dis­tinct­ively neg­at­ive, because it indic­ates an unfairor dis­pro­por­tion­atejudge­ment, a judge­ment sub­ject to an influ­ence that is dis­tinct­ively ille­git­im­ate in some way. Bias in this sense often involves undue influ­ence by demo­graph­ic cat­egor­ies, for instance. We might describe an admis­sions pro­cess as biased in this way if it dis­pro­por­tion­ately excludes working-class can­did­ates, or women, or people with red hair. We can call bias of this kind pre­ju­di­cial bias.

The visu­al sys­tem is clearly cap­able of exhib­it­ing the first kind of bias. As a sys­tem that sys­tem­at­ic­ally learns from past exper­i­ences in order to effect­ively pri­or­it­ise and pro­cess new inform­a­tion, it is a form­ally biased sys­tem. Similarly, the same-race face effect in face per­cep­tion involves the sys­tem­at­ic neg­lect of cer­tain inform­a­tion as the res­ult of task-specific expert­ise. That renders it an instance of form­al bias.

To decide wheth­er this also con­sti­tutes an instance of pre­ju­di­cial bias, we need to ask: is that neg­lect of inform­a­tion ille­git­im­ate? And if so, on what grounds? Two dif­fi­culties present them­selves at this junc­ture. The first is that we are, for the most part, not used to assess­ing the pro­cesses involved in visu­al per­cep­tion as legit­im­ate, or ille­git­im­ate (though that has come under increas­ing pres­sure recently, in par­tic­u­lar in Siegel (2017).) We need to devel­op a new set of tools for this kind of cri­tique. The second dif­fi­culty is the way in which form­al bias, includ­ing the devel­op­ment of per­cep­tu­al expert­ise of the kind demon­strated in the same race face effect, is a vir­tue of visu­al per­cep­tion. It makes visu­al per­cep­tion not just effi­cient, but pos­sible. Acknowledging that can seem to restrict our abil­ity to con­demn the bias in ques­tion as not just form­al, but pre­ju­di­cial.

This throws us up against the ques­tion: what is the rela­tion­ship between form­al and pre­ju­di­cial bias? Formal bias is often a vir­tue: it allows for the more effi­cient extrac­tion of inform­a­tion, by draw­ing on rel­ev­ant post inform­a­tion. Prejudicial bias on the oth­er hand is a vice: it lim­its the sub­jects’ sens­it­iv­ity to rel­ev­ant inform­a­tion in a way that seems intu­it­ively prob­lem­at­ic. What are the cir­cum­stances under which the vir­tue of form­al bias becomes the vice of pre­ju­di­cial bias?

In part, this seems to depend on the con­text in which the pro­cess in ques­tion is deployed, and the task at hand. The vir­tues of form­al biases rely on sta­bil­ity in both the individual’s envir­on­ment and goals: that’s when reli­ance on past inform­a­tion and expert­ise developed via con­sist­ent expos­ure to cer­tain stim­uli is help­ful. The same-race face effect devel­ops as the visu­al sys­tem learns to extract inform­a­tion from those faces it most fre­quently encoun­ters. The res­ult­ing expert­ise can­not adapt at the same pace as our chan­ging, com­plex social goals across a range of con­texts. As a res­ult, this kind of form­al per­cep­tu­al expert­ise res­ults in a loss of import­ant inform­a­tion in cer­tain con­texts: an instance of pre­ju­di­cial bias. If that’s right, then the dis­tinc­tion between form­al and pre­ju­di­cial bias isn’t one that can be iden­ti­fied just by look­ing at a par­tic­u­lar cog­nit­ive pro­cess in isol­a­tion, but only by look­ing at that pro­cess across a dynam­ic set of con­texts and tasks.



Brooks, J. A., & Freeman, J. B. (2017). Neuroimaging of per­son per­cep­tion: A social-visual inter­face. Neuroscience Letters.

Chalk, M. S., & Series, A. R. (2010). Rapidly learned stim­u­lus expect­a­tions alter per­cep­tion of motion.Journal of Vision, 108(2), 1–18.

Chun, M. M., & Turk-Browne, N. B. (2008). Associative Learning Mechanisms in Vision. Oxford University Press. Retrieved from‑7

Elfenbein, H. A., & Ambady, N. (2003). When famili­ar­ity breeds accur­acy: cul­tur­al expos­ure and facial emo­tion recog­ni­tion. Journal of Personality and Social Psychology,85(2), 276.

Jaquet, E., & Rhodes, G. (2008). Face afteref­fects indic­ate dis­so­ci­able, but not dis­tinct, cod­ing of male and female faces. Journal of Experimental Psychology. Human Perception and Performance, 34(1), 101–112.–1523.34.1.101

Jaquet, E., Rhodes, G., & Hayward, W. G. (2007). Opposite afteref­fects for Chinese and Caucasian faces are select­ive for social cat­egory inform­a­tion and not just phys­ic­al face dif­fer­ences. The Quarterly Journal of Experimental Psychology, 60(11), 1457–1467.

Jaquet, E., Rhodes, G., & Hayward, W. G. (2008). Race-contingent afteref­fects sug­gest dis­tinct per­cep­tu­al norms for dif­fer­ent race faces. Visual Cognition, 16(6), 734–753.

Kahneman, D., & Tversky, A. (1973). On the psy­cho­logy of pre­dic­tion. Psychological Review,80, 237–251.

Kelly, D. J., Quinn, P. C., Slater, A. M., Lee, K., Ge, L., & Pascalis, O. (2007). The other-race effect devel­ops dur­ing infancy: Evidence of per­cep­tu­al nar­row­ing. Psychological Science, 18(12), 1084–1089.

Little, A. C., DeBruine, L. M., & Jones, B. C. (2005). Sex-contingent face after-effects sug­gest dis­tinct neur­al pop­u­la­tions code male and female faces. Proceedings of the Royal Society B: Biological Sciences, 272(1578), 2283–2287.

Meissner, C. A., & Brigham, J. C. (2001). Thirty years of invest­ig­at­ing the own-race bias in memory for faces: A meta-analytic review. Pyschology, Public Policy and Law,7(1), 3–35.

Rhodes, G., & Leopold, D. A. (2011). Adaptive Norm-Based Coding of Face Identity.

Scholl, B. J. (2005). Innateness and (Bayesian) visu­al per­cep­tion. In P. Carruthers (Ed.), The Innate Mind: Structure and Contents(p. 34). New York: Oxford University Press.

Shapiro, J. R., Ackerman, J. M., Neuberg, S. L., Maner, J. K., Becker, D. V., & Kenrick, D. T. (2009). Following in the Wake of Anger: When Not Discriminating Is Discriminating. Personality & Social Psychology Bulletin, 35(10), 1356–1367.

Siegel, S. (2017). The Rationality of Perception.Oxford University Press

Summerfield, C., & Egner, T. (2009). Expectation (and atten­tion) in visu­al cog­ni­tion. Trends in Cognitive Science, 13(9), 403–409.

Valentine, T. (1991). A uni­fied account of the effects of dis­tinct­ive­ness, inver­sion, and race in face recog­ni­tion. The Quarterly Journal of Experimental Psychology, 43(2), 161–204.

Valentine, T., Lewis, M. B., & Hills, P. J. (2016). Face-space: A uni­fy­ing concept in face recog­ni­tion research. The Quarterly Journal of Experimental Psychology, 69(10), 1996–2019.