Not an attention check

Hi Team,
I’m starting a study (my first on Prolific!) in which participants are asked to complete trials as quickly as possible (we have a small bonus for each trial completed). They have an overall time limit (~30 minutes) and “as much time as they need” to plan during each trial. During piloting, I discovered that some participants might not care about their performance, despite the bonus – one took about 6 minutes to plan a single trial (the average seems to be around 2-10 seconds).

I don’t think an attention check would be possible in this scenario, but I’m not sure how to handle a participant who obviously just let the countdown timer go and did not attempt to work quickly. We thought about something like a pop-up saying “Hey are you still working?” with a three strikes and you’re out or something like that, but due to the constraints of the task, that won’t be possible. Do y’all have any ideas about what to do in this situation?

I can provide more information if that would be helpful. Thanks,P.

Hi @P.P , thanks for your question, and very exciting that you are about to start your first study with us!

Unfortunately, we currently do not have any written guidance on ‘low-effort’ responses such as this. With such a vast array of studies being run on the platform the definition of low-effort will changes considerably across experiments, making it difficult for us to provide any baseline criteria that must be met.

For your particular study, I would imagine that any participant who is not attempting to work quickly and just letting the timer run out is going to show pretty clear evidence of low-effort at the end of the study because they will have answered far fewer trials? If this is true, and the evidence is clear, then I would suggest that you contact our researcher support team at that point and we can provide further guidance. We like to take each case on its merits, so if the evidence is clear we will always be in favour of allowing rejections for the affected participant.

Sorry to not have more concrete guidance for you in this case, but I hope this helps.

2 Likes

Hi Andrew_Gordon, thanks for the response! I didn’t think there would be a simple solution, and I appreciate the guidance.

I think we’ll add some instructions intended for the least motivated participants like “This task is meant to be completed in one sitting. Please participate actively in the task at all times.”, and then if/when we run into these cases we can contact the researcher support team. It would be clear in the data - both the timing and the number of trials completed - that these participants are different than the others.

2 Likes