We've updated our attention and comprehension check guidance!

Hi everyone :wave:

We have just released our updated attention and comprehension check guidance and wanted to flag it on here so you are all aware of the changes.

For the specific details of the new policy please read the help centre article linked above, but the two major changes are:

  • Nonsensical items/questions/statements are now a valid form of attention check on Prolific

  • Comprehension checks are now covered by our guidance

Both types of checks have some specific guidelines regarding how to make them valid, so we encourage you to read the guidance to ensure that your checks are in line with our policy. If you have any questions about the guidance, or if you would like to have your check reviewed by us please contact the support team.

Thank you Andrew.

I just noticed. Wow.

Nonsensical checks are now allowed as a type of attention check and also basis for rejection such as (Prolific’s example)

I swim across the Atlantic Ocean to get to work every day : Strongly disagree, Disagree, Agree, Strongly agree.
Agree and Strongly agree are a basis for rejection.

“Participants should not be given a ‘neutral’ response option (i.e., no middle scale value)” (I am not sure why the middle term is simply not allowed as a basis for rejection).

I am very grateful of the inclusion of nonsensical attention checks but at the same time I think it will open the proverbial can of worms and be tough for support who have my sympathy.

E.g. There may be argued to be a continuum in these
I have a really tough commute.
I have a devil of a job getting to work.
My commute is like going through a war zone.
My commute is like crossing the Atlantic Ocean.
My commute is like swimming the Atlantic Ocean.
My commute is through a war zone.
I have to swim the Atlantic Ocean to get to work.
I swim across the Atlantic Ocean to get to work every day.

So there are going to be participants claiming that they understood any and all “nonsense” metaphorically. There are already lots of gripes on reddit.

Similarly
I sleep in a thimble. (My bedroom is small)
Colorless green ideas sleep furiously (famous example of “nonsense” but to which applicable situations have been suggested)
I am from another planet (used to express weirdness).
I have seventeen fingers (is used already apparently, but might indicate dexterity)
I drive a combine harvester in my kitchen. (I use all manner of kitchen appliances)

All the same, I think nonsense should stay and I am grateful.

Tim

Hey Tim,

Thanks for the feedback on this, you raise some interesting points!

I must admit that inclusion of nonsensical items was a tough decision to make, however we have seen that (a) many researchers were already using these types of checks, and (b) there is quite a lot of precedent for such checks in the literature.

I understand the concern about how some participants may judge these checks on a continuum such as the ones you have suggested, however we believe that these will only be the very fringe cases. In each case, if a dispute is raised we will review the check ourselves and come to a decision on whether there is room for misinterpretation, but if we believe there is not then the check will count.

Similarly, regarding the ‘neutral’ value, the rationale for not allowing this was that it forces a participant to either agree or disagree, therefore creating a binary decision and making the participant weigh up whether the evidence is really there for any more outlandish answer they could make.

It will be interesting to see how this plays out over the coming months. If we do see significant numbers of complaints or misunderstandings we may of course take another look at this policy, however I’m hopeful this will not be the case (the support team are prepped for the onslaught :slight_smile:)

Best,
Andrew

1 Like