iCog research grants selection process

iCog had some leg­acy funds to dis­burse in the 2019/20 aca­dem­ic year. In keep­ing with iCog’s remit to sup­port and encour­age inter­dis­cip­lin­ary research, the iCog steer­ing com­mit­tee decided that the best use for these funds would be to award grants for the par­ti­cipant costs of empir­ic­al research by juni­or research­ers doing inter­dis­cip­lin­ary work in cog­nit­ive sci­ence. Projects which were par­tic­u­larly likely to fall through the gaps of the fund­ing remit of oth­er fund­ing bod­ies due to their inter­dis­cip­lin­ary nature would be giv­en priority.

We received 33 applic­a­tions: 9 from the UK, 8 from the US, 4 from Germany, 3 from Switzerland, 2 from Finland, and 1 each from Australia, Austria, Belgium, France, Italy, the Netherlands, and Turkey.

The selection process

Usually all that an applic­ant for research fund­ing is told is that his or her applic­a­tion was or (more often – in this case about ten times more often) was not suc­cess­ful, while the pro­cess lead­ing to that res­ult remains opaque. With this post, we wish to bring some trans­par­ency to the selec­tion pro­cess we used. By explain­ing in some detail here how we pro­ceeded and why, we hope to show that the pro­cess was fair and systematic.

No selec­tion pro­cess is per­fect. But we also hope that our way of pro­ceed­ing with this selec­tion will find imit­at­ors across the academy, in par­tic­u­lar in its strict com­mit­ment to blind review­ing, as detailed below. Preparing this can be a little labour-intensive. But, in the interest of fair­ness to all applic­ants, blind­ing applic­a­tions as needed was a labour worth doing.

Of the total of 33 applic­a­tions received, one applic­a­tion was ineligible as its pur­pose fell out­side the remit of the grants, i.e. par­ti­cipant costs in empir­ic­al stud­ies. The 32 eli­gible applic­a­tions were blind reviewed, ini­tially by two aca­dem­ic review­ers (drawn from a grants review sub­com­mit­tee of five) with cross-disciplinary expert­ise but from two dif­fer­ent ‘home’ dis­cip­lines, who assessed and scored their qual­ity, inter­dis­cip­lin­ar­ity, and feasibility. 

From this pro­cess four research pro­pos­als emerged with very high aggreg­ate scores from both review­ers. These were auto­mat­ic­ally short­l­is­ted. A fur­ther five applic­a­tions had suf­fi­ciently high scores from at least one review­er to be con­sidered fur­ther; these pro­pos­als were then each reviewed by a third review­er. Two of these five pro­pos­als thereby reached an aggreg­ate score com­par­able to that of the four pro­pos­als already short­l­is­ted, lead­ing to a final short­l­ist of six. (The length of the short­l­ist was dic­tated by the need to keep it man­age­able for the grants sub­com­mit­tee to review all short­l­is­ted proposals.)

Common reas­ons why pro­pos­als were rejec­ted at this stage were that oth­er pro­pos­als were qual­it­at­ively super­i­or, insuf­fi­cient detail on the meth­od or feas­ib­il­ity of the pro­posed stud­ies, or – in some cases – lack of interdisciplinarity.

The six short­l­is­ted research pro­pos­als, along with their applic­ants’ CVs and the two or three ini­tial reviewer’s reports and scores, were then sub­mit­ted to the full grants sub­com­mit­tee for final delib­er­a­tion. This was done by a com­bin­a­tion of dis­cus­sion and, where this proved incon­clus­ive, votes. Taking into account avail­able funds and the amount of fund­ing reques­ted by applic­ants, the subcommittee’s decision was to award full fund­ing to two research pro­pos­als and par­tial fund­ing to a third. The three short­l­is­ted applic­ants who were ulti­mately unsuc­cess­ful were offered feed­back on their applic­a­tions; all of them took up that offer.

Blind review

The entire review pro­cess up to and includ­ing the final decisions on fund­ing was ‘blind’. In the stages up to short­l­ist­ing, review­ers were not shown the names or affil­i­ations of the applic­ants. For the short­l­is­ted applic­a­tions, affil­i­ations were then vis­ible inso­far as they appeared on applic­ants’ CVs, but applic­ants’ names were still redac­ted from these. Why and how did we do that?

The rationale for blind review­ing is simple: it is to avoid review­ers hav­ing impli­cit biases on the grounds of an applicant’s gender, pre­sumed back­ground, or loc­a­tion. Personal names often reveal a person’s gender and eth­nic ori­gin, so these were with­held from review­ers through­out the pro­cess. The decision also to with­hold applic­ants’ insti­tu­tion­al affil­i­ations dur­ing the first stages of the review pro­cess, up to short­l­ist­ing, was like­wise to pre­vent review­ers from being influ­enced by favour­able or unfa­vour­able biases they might have about an applicant’s insti­tu­tion or depart­ment or its loc­a­tion. (For the short­l­ist, it would not have been prac­tic­able to con­tin­ue with­hold­ing affil­i­ations, since applic­ants’ CVs were also being con­sidered at that stage.)

As to the meth­od of ensur­ing blind review­ing, this too was rel­at­ively straight­for­ward. The applic­a­tion form was designed in such a way that applic­ants’ research pro­pos­als could eas­ily be pulled out of the data sub­mit­ted without the accom­pa­ny­ing per­son­al or insti­tu­tion­al details (and just as eas­ily matched again at the end of the pro­cess, by means of a unique ID alloc­ated to each application). 

We had asked for the research pro­pos­als sub­mit­ted to be ‘suit­able for blind review­ing’. Many applic­ants were com­mend­ably assidu­ous in com­ply­ing with this request, though some were not – we ought per­haps to have been more expli­cit by e.g. put­ting a note on the applic­a­tion form request­ing ‘No names or affil­i­ations in the text of the pro­pos­al, please’. We had like­wise asked for uploaded CVs to con­tain the applicant’s name only once, to speed up the blind­ing process.

To make sure that the mater­i­al passed to review­ers was indeed ‘suit­able for blind review­ing’, it was neces­sary to redact any men­tion of applic­ants’ names or gendered per­son­al pro­nouns and insti­tu­tion­al affil­i­ations from research pro­pos­als and, in the CVs of short­l­is­ted applic­ants, redact the one occur­rence of their name from the doc­u­ment. We were able to do this because the iCog com­mit­tee had des­ig­nated an admin­is­trat­or for this pur­pose, as well as for organ­ising the review pro­cess gen­er­ally, without him­self being a mem­ber of the review subcommittee.