Community Attention Check Library - Add Your Own!

šŸŒ The Co-Lab Community Libraries
1

All details below :slight_smile:

2

Fair Attention Check Library

Attention checks are an important tool in the fight against bad data quality. But, we also want to make sure that any checks that are used are fair.

We know that you face two key problems in this area:

  1. Itā€™s not clear what a ā€˜fairā€™ attention check looks like in some scenarios
  2. Some researchers are great at making fair attention checks, but others need more guidance.

How are we going to solve these problems?

  1. Weā€™ll soon be releasing brand-new guidance that aims to be clearer.
  2. Weā€™re launching a Community Attention Check Library

The Library

This will be a resource created by the community, for the community! Youā€™ll be able to submit attention checks which can be used by other members. The process is simple:

  1. Use this template
  2. Submit your attention checks here
  3. Our team will review it within the next few weeks, and if it needs to be made more ā€˜fairā€™, weā€™ll contact you with our recommended changes.

Note: This is just an initial test of this idea, so our release of the approved checks may take longer than expected, and we may not be able to fully launch this for some time.

Looking forward to your submissions!

1 Like
3

Hi Josh, When Iā€™m clicking on ā€œSubmit your attention checks hereā€, I get an error message saying that I need permission

image
image1702Ɨ655 40.9 KB

1 Like
4

My apologies! Iā€™ve just removed that restriction.

Thanks so much for submitting!

1 Like
5

Attention Check Library Submissions & Feedback

Thanks to everyone who submitted a check to our library! Our team have reviewed them all. Below are the submissions with our comments on why we approved or did not approve them.

Although some of these checks are not currently valid, we want to thank you for suggesting them to us! Our attention check policy is an area we are keen to constantly evolve in response to our researchers needs while keeping the best interests of our participants in mind.

Please continue to submit your ideas for checks, and we will review every case!

:white_check_mark: Approved Checks

Rating Scale - here :white_check_mark: (message me for password)

Submitted By: @Magdalena_Kachlicka

The file is password protected to avoid it being used to train bots to bypass checks :slight_smile:

We are happy to accept this type of check as a valid attention check on Prolific as long as the question clearly states the number the participant is supposed to select - e.g., ā€œHow did you like the picture you saw? Regardless of how much you liked it please select 7ā€.

This check meets the following criteria in our guidance:

  • Checks whether a participant has paid attention to the question, not so much to the instructions above it.
  • Explicitly instructs the participant to complete a task in a certain way

:x: Not Approved Checks

These are reference only, and should not be used as part of your studies to check for attention. You can use them for other purposes.

Weā€™re publishing them, with our comments, to give you guidance on what you should avoid :slight_smile:

Dogs & Cats - here :x:

Please note: we have not approved this as an attention check. You may wish to use it for another purpose.

While the specific check example provided seems clear, there are two main reasons why we would not allow this as a valid attention check on Prolific:

  1. It relies on logic and interpretation of the question on the part of the participant. For instance, if the instructions were less specific on the type of image to click (e.g., referring to the cat as a ā€˜felineā€™) then participants may either (a) not understand the direction (due to not understanding the wording), or (b) interpret the direction differently (e.g., the term ā€˜felineā€™ can be used to describe more than just a cat). Because we cannot check the wording of every check like this used, we cannot accept these types of checks as valid on Prolific.

  2. Secondly, and more importantly, while dogs and cats generally have sufficient discriminability, this may not be the case with other pairs of image subjects. This makes it impossible for us to accept these types of checks as a whole as valid due to (a) the subjective nature of what is and isnā€™t discriminable, and (b) not having control over what types of images are used.

One alternative to the provided image-based version of this check may be to use simple black and white squares and ask participants to select all squares of one colour. As black and white are not open to interpretation, and are also readily discriminable (even to those with vision issues), this provides a fairer test of attention.

Language - here :x:

Please note: we have not approved this as an attention check. You may wish to use it for another purpose.

This check would not be considered valid because it relies on participants maths ability for them to answer it correctly. While the check provided is straightforward, to comply with our guidance a check needs to only check for attention, not another skill. This check could be made valid by explicitly instructing the participant to select a certain number instead.

Internal Consistency - here :x:

Please note: we have not approved this as an attention check. You may wish to use it for another purpose.

Although the idea behind these internal consistency checks is interesting (and something we would like to include in our policy in the future), they do not currently meet our criteria for attention checks on Prolific. The reason for this is that some of the provided examples come very close to relying on participants maths skills, and, as noted in response to check 3, to comply with our guidance a check needs to only check for attention, not another skill.

Furthermore, for a type of check to be valid on Prolific, we need to be able to provide objective and clear guidance on what is and isnā€™t allowed within the check. In the case of these checks we do not feel that we could provide clear enough guidance to our researchers that would stop invalid checks of this type being used.

Add to the Library

Please continue to submit your ideas for checks, and we will review every case! :heart:

2 Likes
6

Hi Josh,

We have a bunch of interesting attention, engagement and bot checks for Gorilla users in our samples here:

Gorilla studies are often testing a range of cognitive skills, so these do not necessarily only test attention. But are more useful for researchers deciding whether to include the participantsā€™ data in their analysis.

Anagram task: Click the scrambled letters in the right order to make a word.
Naming Task: Name the animal in the picture! Participants are presented with the silhouette of a common animal. They have to type the name of the animal.
Real Effort Task: Count how many zeroes appear in a grid. This is a classic real effort task. A grid of 1s and 0s is shown to the participant. They have to respond with the number of 0s. The grid is shown as an image, to make it more bot proof.
Sentence Unscrambling Task: Click the scrambled words in the right order to make a sentence.
Visual Search: Can you find the cat among the dogs?

1 Like
7

These tools are great @JoEvershed! Thanks for sharing :slight_smile:

1 Like
8

Hello, how do I obtain the password to see the library of attention check questions?

9

Hi Josh,

I canā€™t open the library, I see a message that the files have been deleted. Can you please help with this?

Thank you,

Giada

10

Hello Josh.

When i click on "use this template " it says it has been deleted. I cannot get through it.

11

Hello, the links to templates or submissions appear not to work.

12

I also find these approved attention checks do not exist. Could you just pop them into the chat rather than use a google doc? Its a very important resource.

13

Hi Josh,

I canā€™t open the library either. I also see a message that the files have been deleted.

Can you please help with this?

Thank you and kind regards,
Mathom