As mentioned in another post, I have been trying to run online multiplayer experiments with big groups (at least 8 at a time). However, there are a lot of complications and moving parts involved in doing so. The main aim is to get a lot of people to come in at the same time so that people don’t have to wait for the groups to slowly fill up.
It was recommended to me that I follow a process similar to that of video/audio interview. This involves creating a study, where participants are directed to a scheduling tool to schedule when they will participate. In the meantime, they have to be given a code so that they can submit the study and do other studies between now and then.
I decided to do without the scheduling tool, they can have privacy issues (even Doodle encourages - although does not force - participants to put in some sensitive information such as names and emails). Instead, I do everything in one link with my coding. Participants are sent to a page that tells them that the study will start in X amount of time and to come back here on time before it starts. It informs them about the code to be able to continue and reminds them that they have to actually complete the study to receive payment.
Nevertheless, we are left with quite a few moving parts and complications:
- Participants might lose the link or not get a reminder to come back to the study in time. (sending a reminder message 15 mins before might not reach them as they won’t get a notification in the browser add-on).
- Participants might not return their participation if they didn’t complete the study. It’s never nice to have to chase participants to return their study, nor report them if they refuse to do so.
- This can still seem weird to participants.
- They might not plan well and start another study just before ours.
- They might over-plan and wait in front of our study for a whole chunk of time before it starts.
I was wondering if Prolific might consider removing all these moving parts by creating an option do to studies where participants register for a later start. The experimenter would set when to publish the study and when to release the actual link to the study. The participants would register for it. No need to do stuff by providing a code and it wouldn’t block them from doing other studies. They get a reminder a few minutes before the start, where they can click and get to the actual study. This would make things a lot simpler for experimenters and a lot clearer for participants.
(bonus points if you can replace the scheduling process for the video and audio interviews, but I am just interested in this fix for multiplayer experiments, I don’t need people to choose amongst different time slots, just one time to come back to later in the day.)
Thank you for time
This is a really great idea @Samuel_Dupret! We know how fiddly the current video/focus group interview process is. Any ideas on how to improve it are very welcome!
I’m working on a process to ensure that great community ideas get in front of the right team who can work on implementing them. I’ll keep you updated on where this idea could fit into our future product roadmap
I decided to do without the scheduling tool, they can have privacy issues (even Doodle encourages - although does not force - participants to put in some sensitive information such as names and emails).
Just a note on this: our policy on collecting sensitive data when arranging these types of studies
Collection of personal information is not usually permitted on Prolific, so participants would need to be assured that any personal details if at all collected (e.g. Skype usernames) will only be used to contact them for the interview, and deleted as soon as this purpose has been fulfilled.
Let me know if you have any questions!
I have been doing some testing with multiplayer scheduled experiments. I stand by my ask for a system handled by Prolific. Here are a few new things I have discovered (or rediscovered).
One that’s just general to multiplayer studies where we try to create groups (e.g., 10 participants) and we can’t run groups with fewer participants. If we cannot form a group, we send a message saying we are sorry to participants and offering a small bonus in exchange for them returning their participation. Some participants are annoyed by this and demand that they get to participate.
Some of my comments refer to stuff that a Prolific internal system will solve however:
- allowing participants to enter the code early so that they can do other studies whilst waiting creates weird overall pay/time scores. Some participants only give in the code at the end, so they count as having participated for a lot longer than the actual study (say 30min wait before, and 30 more minutes of the study, instead of just 30 minutes for the study).
- Despite providing the code for them to enter early, some participants refer to the waiting time as time that counts. Sometimes indirectly, such as “I waiting X amount of time, and didn’t get to participate”, etc. The fact that the countdown/time wait happens on the experimenter’s side instead of through an official Prolific system means that participants might perceive the waiting time as ‘time they worked for’.
Also, the auto approval feature messes up this whole system.
I’m piloting a group Zoom-call study. In case it’s helpful to you, here is my current design: In my Study 1, I have participants indicate their availability for pre-set appointment times for the upcoming two weeks (this is done via Qualtrics survey). Afterwards, I send them a message via Prolific to announce the appointment time I have chosen for them (based on when most respondents are available). Then, I publish Study 2 (with Custom Allowlist) for which the study URL is the actual Zoom URL. Toward the end of our group Zoom session, I send everyone (via the Zoom chat box) a URL to another Qualtrics survey which automatically routes them to the Prolific Study Complete page when they complete that survey.
However, I’m running into the issue of participants not showing up for their session; up to 30% of participants don’t show up. (I am forming groups of 3; if I schedule 15 participants, I should have 5 groups. But if one person doesn’t show up, I have to send two people home with a prorated payment sent via the “Bonus” feature after they Return the study in Prolific.)
Do you have any thoughts on how I can decrease my rate of no-shows? Currently, the day after they complete Study 1, I send an announcement of their appointment date/time. I also send a reminder message the day before their session, the morning of their session, and 1 hour before their session (all messages via Prolific). Also, I publish Study 2 immediately after announcing it (the day after Study 1). That way, Study 2 stays visible in their queue as a visual reminder (the first line of the description is an instruction to not start the study until their appointment time).
Hi Moses - This seems like a smart way to design a scheduled experiment. However, I was wondering if you had found any further improvements to get around the 30% attrition rate you experienced?
Also, in reference to Samuel’s original post in this thread, has Prolific introduced any kind of proper scheduling feature onto their platform yet?
I’m glad you asked, otherwise I would have forgotten to follow up on this.
I’m not aware of any kind of scheduling feature built into Prolific yet, but the Prolific team seems really good about incorporating user-feedback into their updates. Anyway, I eventually realized I didn’t have to schedule people in advance because there are thousands of Prolific users online at any given moment. At that point, my new plan was to just launch my studies on Prolific without any scheduling (the studies had pre-screeners built into them). The whole idea of scheduling the participants was a cognitive artifact of on-campus data-collection, which naturally would require scheduling. But online, there’s no need for that because there are thousands of users at any given moment, waiting “at the door” to enter my online “lab”.
However, when I tried that new method, I noticed I was still only getting less than a handful of participants joining per hour, and they were all spread out over the hour (my task required teams of 3 members). Eventually I discovered this page on Prolific’s Help Centre, which mentions a rate-limiting mechanism that is built into Prolific (I imagine it’s to help make everyone’s sample more representative, i.e., to prevent the top percentile of Prolific users from getting all of the studies). I was able to email Prolific’s support team to request the rate-limiting mechanism be removed from my study, because I needed multiple people to sign up simultaneously. Once I did that, I was able to activate my studies and get as many participants as I wanted (typically I only ever needed between 6 and 15 participants per session) within 5-7 minutes of activating the study.
In short, my routine was to activate a study 10 minutes before the start-time advertised in the study title/description (after making sure to update my custom block list to disallow any prior participants from participating again) while I was waiting in my Zoom room (the “start” button of the study linked to the Zoom room). Participants would join the Zoom waiting room and I would message everyone in the waiting room (Zoom allows me to do that) and tell them we would start shortly (of course, I had to make sure the participants’ pay rate included the time spent in the waiting room). Once I had a sufficient number of participants (this generally happened a couple minutes before the scheduled start time), I would admit people into the Zoom room and immediately change everyone’s name in Zoom to something anonymous (per my protocol) and start my session.
Hope this is helpful!
We’ve not implemented it yet, but if you check out our new Apps & Integrations page, you’ll see that we’re laying the foundations to make it easier for you to use tools like Calendly etc, within your Prolific workflow
Thank you, Moses, that’s really helpful!
I attempted to run a live Public Goods Game with 18 participants on M-Turk, previously. I ran into a similar problem as you, whereby some participants joined but not enough for the game to run. I was then left with disgruntled participants who were left waiting while the game waited in vain for enough others to join.
I’ll definitely look into seeing if I can use Prolific if they are able to remove this rate-limiting function you mention. Thanks again for all of your useful advice and insight; it’s been really useful!
Hi Josh - thanks for getting back to me with this! I will take a look.
Reach out here, and we can turn off the rate limiting for your account