How is the estimated completion time calculated?

Hello,
I have a question about estimated time to complete a study. I know we as researchers provide an approximate value, but that then Prolific calculates a value and adjusts the predicted pay/hour accordingly.

Could you say a bit more about how this is calculated? Are any ‘waiting for approval’ submission counted or are only ‘approved’ submissions used to calculate this? Are participants who timeout affecting this calculation?

Thank you,
Sam

7 Likes

Hey Samuel, welcome to the community!

  • We use the ‘median’ to calculate the average study completion time, so that extremely short/long times are excluded
  • The median is calculated using the ‘approved’ AND ‘awaiting review’ submission times
  • ‘Returned’ and ‘timed out’ submissions are excluded from the calculation

I hope that helps! Let me know if anything is unclear :slight_smile:

More info on how the reward per hour fluctuates can be found here.

5 Likes

Hi @Josh, I have a question related to Samuel’s: Is there any functionality on Prolific that would allow the participant to participate as much or as little as they want–that is, perform as many tasks as they want until they get bored, then opt out, then get paid a per hour amount commensurate with the time they spent? Whereas another user who completed fewer tasks / spent less time would be paid less?

1 Like

Hey @CHRISTIAN_WILLIAMS, welcome to the community! Glad you’re here :tada:

get paid a per hour amount commensurate with the time they spent?

Unfortunately, participant rewards are fixed as soon as you publish a study. It’s not possible to automatically pay people different amounts.

But what you can do, is pay everyone the minimum amount (£5/hour), then offer bonus payments according to how many tasks a participant completes.

Let me know if you have any other questions! :slight_smile:

Oh, interesting solution. Thank you, @Josh.

As far as implementing this, what’s the simplest route do you think? Would we need to manually identify how many tasks each participant performed and then provide them with that bonus? Or, are we able to pass a URL parameter back to Prolific that includes the number of tasks completed, and that being used to automatically calculate bonus? I assume not possible but wanted to check.

1 Like

Are we able to pass a URL parameter back to Prolific that includes the number of tasks completed, and that being used to automatically calculate bonus?

Unfortunately, this won’t be possible. But, to make the manual proces easier, you can:

  • Using your external survey software, track how many tasks each participant completes
  • Group participants according to the number they’ve completed
  • Export the IDs of the participants in the various groups, and ‘bulk pay’ those IDs with their relevant bonus

Does that make sense? :slight_smile:

1 Like

Thanks Josh, much appreciated.

1 Like

Hi @Josh , we’re trying to test our study end to end. Our site allows users in by looking at the parameters that we expect to be passed on by Prolific. So https://www.oursite.com/studies/study1/?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

It expects some string after ?PROLIFIC_PID= the problem is, we aren’t getting that parameter passed along to us by Prolific, and so participants are unable to access our site. Do you have any idea why the parameters would not be passed along?

I set myself up as a participant so I could troubleshoot this, but it’s really difficult, and my study isn’t appearing as an option to my “Participant” account, despite my About Me and the study’s prescreener matching me perfectly. What’s the best way for me to do a true test of our study, end to end, so I can troubleshoot why the Prolific parameters are being lost? Thanks!

1 Like

Dear Christian

The Session ID is way above my head. I am very sorry I can’t help you there. I am also stumped as to why your prolificsb account can’t access the survey.

But may I ask why you ate not carrying out the sudies sequentially? That would be a simple way of running two studies with diffeerant criteria. There is a function for blacklisting particioants of other studies. If you copy a study then participants of the copied study are automatically excluded from the copy (ah, could this be why your prolificsb account is unable to participate?).

You could I think increase the places on each study alternately to ensure that both studies are taking place at almost the same time. (In that case however I think you would have to use the manual blacklist to prevent participants from taking both studies.)

Just a suggestion,

Tim

1 Like

The best way is to use a custom allowlist which only has your participants account ID on it.

Studies aren’t distributed by our system only on the basis of eligibility. We also distribute them according to how many studies a participant has done previously to ensure we maintain naivety. So, it’s not guaranteed that you’d receive a study just because you’re eligible.

Hi @Josh , we’re trying to test our study end to end. Our site allows users in by looking at the parameters that we expect to be passed on by Prolific. So https://www.oursite.com/studies/study1/?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

I think you’ve got an extra ‘/’ before ‘?PROLIFIC_PID’. Try this link:

https://www.oursite.com/studies/study1?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

Lmk if that works! :slight_smile:

Thanks, this is a great response @Josh. I’ll check the URL fix you suggest on our next deploy.

In the meantime, I’m still unable to access my study from my participant account, even after making a custom allowlist. (Pictures attached.) Is this because of your comment on how Studies aren’t distributed by the system only on the basis of eligibility? I really need to be able to test our study end to end; it seems like there’s got to be a workaround here.



1 Like

@Josh searches don’t bring anything up, but I’m looking for any information on implementing a non-disclosure agreement into a study. Is there a means for adding this?

1 Like

That’s strange. It should work. I’ve DM’d you with my participant ID, so we can see if it works with my account :slight_smile:

We have a prescreening filter in place to help you recruit participants willing to sign up to a confidentiality agreement:

But, in practical terms it may be difficult (or impossible) to enforce these sorts of restrictions against anonymous participants in your study.

The only requirement that we have is that participation on Prolific must remain anonymous. Beyond this, how exactly you manage the agreement within your study is up to you as data controller (see Data Protection and Privacy). :slight_smile:

Tim,

Are you able to view my prolific set up just to make sure it is correct?

Thanks

Hi Josh,

is it possible to request an exception to the anonymity rule? We’re working with a startup and they’re - understandably - highly concerned about participants leaking their idea.

Of course, we’d clearly communicate this exception to potential participants, and only those willing to sign the NDA would be a part of the study.

Singing the NDA would be done through a secured digital signing service such as DocuSign or Hellosign. We’d need the participant’s name, surname, and address for NDA to be binding.

Other than signing the NDA, the rest of the video survey would be per the usual Prolific process.

Not being able to get participants to sign NDA prior to seeing the prototype would mean we’d need to look elsewhere for recruiting participants.

2 Likes

Hey @Infinum_d.o.o, welcome to the community! :tada:

Unfortunately, we don’t make any exceptions to the anonymity rule. We do have participants who are willing to sign NDAs, and studies have been run successfully in the past using them. But we do appreciate it would be difficult to enforce such an agreement with anonymous individuals.

We’re sorry that we can’t support your study.

Thanks for the answer and clarification :+1:

1 Like

Hi,

Could you tell me how the “maximum time allowed” is computed prior to collecting data?
It says this is “automatically calculated based on your estimated completion time.”, but how is this done precisely?

Thanks!
Andre

1 Like

I looked everywhere and could not find an answer to this.

Two short is three standard deviations less than the mean but too long is set before standard deviation and mean is known so it is just some function of the time estimated by the researcher, I presume, but if we are to know (and it may be a secret) Josh or someone from Prolific will have to enlighten us.

Tim

2 Likes