#TipTuesdays - Our Research Best Practice Guide

Hey Profs! :sunny:

From now on, our #TipTuesday posts are going to pivot slightly into giving you tips on best practice in research.

All the content that will be shared is readily available in our Best Practice Guide. But on the forum, we’ll be going through a different aspect each week. And we’d love to have a discussion about what you think about our advice!

If you like what you see, leave us a :heart:

So, let’s begin…

Prolific’s Best Practice Guide :memo:

Phase 1: Questions to ask before creating a study :thinking:

What research question do I want to answer? And why?

Asking yourself these two questions will not only influence the questions you ask and the answer formats you choose, but also the population(s) you target.

Let’s say you are trying to answer the question: How does leadership style influence burnout? You may be interested in this because burnout is a common phenomenon in many organizations, or because there are currently inconsistent findings on the relationship between these two concepts. In this example, your target group will most likely include employees of a company, and perhaps even employees in a certain type of organization (e.g., IT consultancy firms) in a certain country.

You may want to specify what type of relationship exists between leadership style and
burnout (e.g., predictive, mediating, moderating) and you’ll have to decide on how to measure these variables (e.g., using a psychometrically validated questionnaire).

Ideally, your research question will be grounded in current scientific theory: It may address a knowledge gap, or aim to replicate an existing finding in a different population to examine how generalizable it is. If you’re curious about theory development, please check out our section on theory building in this guide, where we briefly explain why theory development is a critical cornerstone of scientific research, and research in general.

That’s all for this week folks! Next week we’ll be asking the questions:

  • Am I doing exploratory or confirmatory research?
  • Why does it matter?

Am I doing exploratory or confirmatory research? Why does it matter?

Although both types of research aim for findings that are reliable and valid, there are important differences between them.

Exploratory research (sometimes called hypothesis-generating research) aims to uncover possible relationships between variables. In this approach, the researcher does not have any prior assumptions or hypotheses.

In confirmatory (also called hypothesis-testing) research, the researcher has a pretty specific idea about the relationship between the variables under investigation. In this approach, the researcher is trying to see if a theory, specified as hypotheses, is supported by data.

Depending on the type of research you are doing, your approach and research design will be different. Imagine you are doing a confirmatory study to test the hypothesis that playing table tennis at work increases the employees’ creativity. In this case, you will have precise ideas about which measurements to use (e.g., measure the time played table tennis and count the ideas generated in a subsequent brainstorming session or the scores in a creativity test). You will (hopefully) also have a good theoretical foundation to justify why there should be a connection between playing games at work and creativity in the first place, and (hopefully) also a good rationale for measuring “playing games” and “creativity” in a certain way.

If your research was exploratory, however, you would not have any hypotheses in advance, but you would still be interested in finding out what may increase employees’ creativity. You might collect extensive qualitative data through interviews with employees. When analysing these interviews, you may notice that the topics of leisure activities and games come up quite frequently. Based on this insight, you could then develop the post-hoc hypothesis that playing games in the office (and, more specifically, table tennis) might have a positive impact on creativity.

Source:

Why does it matter?

It’s dangerous to confuse these two types of research. All too often researchers treat exploratory results as confirmatory, and this hindsight bias (also known as the ‘I-knew-it-all-along effect’) can make us feel as though we had a prediction all along, even though we didn’t. For example, you might unexpectedly discover that playing table tennis at work decreases creativity, and then find post-hoc reasons for why this is plausible after all. This is called HARKing (Hypothesizing After the Results are Known) and it’s a poor research practice, for several reasons:

  • HARKing can make exploratory findings more publishable by falsely giving the impression that an unanticipated result was expected. This may lead fellow researchers to believe that your finding has been empirically tested more times than it actually has (and that it is more robust than it actually is), thus creating unwarranted confidence in a result and ultimately reducing reproducibility.
  • Valuable information about your original hypothesis might be lost
  • HARKing may promote (conscious or inadvertent) fudging of statistical analyses
  • It presents a distorted, inaccurate model of science
  • It violates a fundamental ethical principle of science: to communicate one’s work honestly and completely
  • HARKing promotes narrow (i.e. context and paradigm bound) new theory (rather than powerful, general theory)

One way to tackle HARKing is to preregister your research. Preregistration helps to distinguish analyses and outcomes that result from prediction (i.e., confirmatory, hypothesis-testing research) from those that result from postdiction (i.e., exploratory, hypothesis-generating research). At the end of the day, we need both exploratory and confirmatory research to do good science. For more information on how to preregister, check out the preregistration article in this guide.

What is my target population?

When you know what research question you want to answer and why, it’s time to think about your target population.

When you know what research question you want to answer and why, it’s time to think about your target population. Are they of a specific age? From a certain country? Only women? Or have a certain political attitude? When you’ve decided on the demographic details of your target group, you can select it by using Prolific’s pre-screening filters. There are more than 200 pre-screeners available, but if you can’t find one you want, just contact our Support team and we will do our best to help you out!

Getting a representative sample

Generalizing your findings to a national population can be difficult, especially if your sample doesn’t match the population very well! We now offer the technology to support representative sampling on Prolific, and can provide you with representative UK/US samples for your study. With this feature, you will be able to easily collect a sample with strata that reflect the general population in the US or the UK.

So what exactly does this mean? Let’s say you want to generalize your findings to the UK population. We will then take your intended sample size and stratify it across three demographics: age, sex, and ethnicity, based on the UK Office of National Statistics data. For example, your sample will have a similar proportion of 30–40 year old Asian women to the general population.

1 Like

What type of data do I need? Quantitative or Qualitative?

Catch up on the most important differences between these types of data to decide what approach fits best for your research.

When it comes to designing your study, you’ll need to make another important decision: Do you want quantitative or qualitative data, or a combination of both? It’s worth looking into the differences, advantages and disadvantages of each before you make your decision.

Quantitative data are numeric and are typically obtained en masse using tests or questionnaires delivered to a large number of people. Results are usually reported in the form of statistical analyses, which are then used to draw statistical inferences. The greatest advantage of quantitative data is that it allows the capture of a diversity of responses, at scale. Hopefully, this best practice guide will help shed light on some of the most important statistical concepts! If you’re really keen on improving your understanding of how to conduct empirical research and use statistics, we highly recommend the free online course by Daniel Läkens on Coursera called “Improving Your Statistical Inferences”.

Qualitative data are non-numeric. They is usually exploratory in nature and aims to explain a phenomenon in terms of ‘how’ and ‘why’. Methods can be anything from open-ended questions and interviews to diary studies. Qualitative data can be analyzed and interpreted in a variety of ways, for example, by letting independent raters assign numerical values to the collected documents, or by coding words and extracting themes until saturation.

Qualitative data can have some advantages over quantitative data. For example, qualitative research may allow researchers to gain unique insights into participants’ feelings, thoughts, and behaviours, thereby detecting areas and issues that may otherwise be missed in a purely quantitative approach. Qualitative data can also be complementary to quantitative data, especially when there are contradictory or ambiguous results. In conjunction, they can hint at potential (causal) relationships, and thus point to future research directions more accurately than a solely quantitative measure could.

In conclusion, when deciding which type of data to collect, you should refer back to your research question and identify the type of information needed to answer the question.

How large should my sample be?

More about statistical power, sample size, and how to determine it.

The method used to determine your sample size depends on whether you’re doing quantitative or qualitative research. With qualitative research, you want your sample to be big enough to draw valid conclusions about the population in question. Otherwise, the data you collect may not provide much evidential value (unless it’s just a pilot).

In quantitative research, your approach depends on whether your study is observational or experimental. If observational, then you need to decide on an acceptable margin of error for your primary variable of interest, in addition to the level of confidence you want in your finding. For a quick sample size determination, you can use a table like this one.

If experimental research, on the other hand, it is important to understand the concept of statistical power in order to calculate the required sample size. In brief, higher statistical power means that you’re more likely to detect an effect, if that effect actually exists. In more formal terms, power is the probability that you correctly reject the false null hypothesis when a specific alternative hypothesis is true. Consequently, an experiment with more statistical power has a better chance of detecting a true effect.

The question is, how much power do you need to detect the effect you are investigating? And consequently, how many participants do you need to recruit for your study? First, you need to turn to the existing literature or run pilot experiments to decide which effect size you expect. You should then do a power calculation to determine the sample size required to detect that effect, given the p -value threshold you intend to use.

Whilst a sample can certainly be too small (or “underpowered”), can your sample size also be too large (that is, “oversampled”)? The short answer is: no. The simple reasoning behind this is that increasing your number of participants also increases the likelihood to find the true effect size. After all, the reason you investigate only a certain amount of people is because you cannot investigate the whole population. But you still want to be confident about the conclusions you draw from your sample to the larger population. So the closer your sample size gets to the population size, the more confident you can be in the effect size you find.

Read more about the question of oversampling in our blog!

2 Likes

Preregistering your Research

If you are studying social sciences, you might be especially interested in reading about how to preregister your research.

What is a preregistration?

Preregistering a study involves writing a plan of how you intend to carry out your research, and then uploading that plan to a preregistration archive (e.g., OSF) before you begin data collection. Your plan should include as much detail as possible, such as:

  • Your hypotheses
  • Study design
  • Inclusion and exclusion criteria for participants
  • Intended sample size (and how you chose that number)
  • Study materials
  • Dependent variables (and how you measure/calculate them)
  • Procedure
  • Randomization approach
  • How you’ll clean up your dataset so it will be ready for analysis
  • How you’ll define outliers in your dataset
  • Analysis plan
  • Data management plan

This might seem like a lot! But not every aspect of the research project has to be detailed beforehand. Deviations from the initial plan are acceptable as long as you can justify them and transparently communicate them. For example, you may want to use non-parametric (rather than parametric) statistical tests because you might find out after data collection that your data are not normally distributed.

Once your plan is written, you are ready to begin your preregistration! On the OSF, you can create a project record for your upcoming study, invite collaborators and upload your protocol. When you are happy and everything is in place, you can create a preregistration - a timestamped, frozen version of your project. If you are worried about your ideas being stolen, you can embargo the registration for up to four years, but if not, you can make your project public right away.

Fundamentally, preregistration lends credibility to your scientific findings. When you make a prediction and find results to match it, your conclusions have more weight. Preregistration lets you prove that your hypotheses are ‘predictions’ and not ‘postdictions’. It lets you prove that you haven’t p-hacked: Since your analysis plan, sample size and exclusion criteria were specified in advance, you can’t fudge your analysis to achieve statistically significant results. Accordingly, preregistration helps fight publication bias because your research article is more likely to be accepted based on its methodology, and not because you found statistically significant results. On top of that, writing a thorough plan before you begin collecting data helps you to spot flaws in your methodology, and makes writing up the paper easier once the data is all in. Perhaps most importantly, it demonstrates openness, honesty, and transparency - three key characteristics of any good scientist.

Even if you are doing exploratory research, preregistration is still worth it. It indicates to others that you are interested in producing credible scientific results that you (and others) can have confidence in.

In psychology, some have called these steps towards improved science Psychology’s Renaissance. Exciting times, right? :slight_smile:

And what are Registered Reports?

There is a second approach to preregistration called the registered report (or reviewed preregistration). In registered reports, your research question and methodology are peer-reviewed prior to your data collection. Based on this review, journals agree to accept articles for publication if the authors follow the methodology specified in the preregistration (this is called ‘in principle acceptance’). Registered reports provide you with expert input on your methodology before you begin data collection. It might sting, but your study will be much better off for it! And, let’s be honest: It’s even more upsetting if you put all your effort in conducting a study and writing a paper that then gets rejected because of a small detail in the design that you overlooked, right?

Still have questions? You can find more answers to questions about preregistration here.

Sources:

2 Likes

Good topic! A site people might find a very user-friendly home for pre-registration is aspredicted.org

It’s light-weight, has nice templates and guides, supports grouping registrations into folders to gather the registrations for a paper or project together, make it public, output as pdf etc.

It’s also easy to use, and facilitates communicate between team members (you get mailed when a registration is saved and can sign off on it or go for another round of edits before sealing it). Very goal-oriented.

A second comment is that thinking “How can I use pre-reg to make my project run more smoothly and get an even better paper?” is a nice mindset - Not a chore but a chance to avoid hassles by thinking through choices and ensuring that the data collected support a great outcome.

2 Likes

Thanks for your thoughts @Tim_Bates!

Defo agree re aspredicted.org! It’s very good. We actually recommended it as the preregistration service of choice for our £10k Grant Competition.

Not a chore but a chance to avoid hassles by thinking through choices and ensuring that the data collected support a great outcome.

Agreed! I’d rather have the extra hassle of preregistering than potentially carrying out a dud of a study

Thanks! I’m definitely a fan of aspredicted.org and that’s what I tend to use.

I also wholeheartedly agree with the point about the indirect benefits that come from being forced to really think through what you’re doing before you collect data. Having to write out my main hypotheses and planned analyses early on has really saved me from some costly mistakes at least once or twice.

2 Likes

Phase 2: Things to keep in mind when designing a study

How to phrase questions and items

Some useful advice on how to formulate the questions you want to ask:

The phrasing of questions is very important when it comes to constructing your study. You should think carefully about how to phrase your questions and items in order to collect results that are as unbiased as possible. Here are some simple rules to follow:

  • Make sure that participants understand your question/item. A good way to ensure this is to pilot test your questions.

  • Avoid negative or double negative phrasing (e.g., use“I am often happy” instead of“I am not often sad”)

  • Avoid ambiguous questions. This means that you ask only one question per item. Also, make sure that your question is phrased in a neutral way. For example, the adjectives “fast ”or “nice” have a subjective meaning to every individual. Give examples for clarification when necessary.

  • Avoid conditional sentences (e.g., “I feel good if I play the piano”)

  • Avoid overly general expressions (e.g., “All children are noisy”)

  • Make sure that your question/item matches the response format. For example, if you ask an open-ended question, you cannot choose a rating scale as the response format.

Source:
https://opentextbc.ca/researchmethods/chapter/constructing-survey-questionnaires/
Peterson, R. A.(2000).Constructing effective questionnaires. Thousand Oaks, CA: Sage

1 Like

These tips are great, thanks!

For surveys, I think brevity is key. My own experience taking surveys is that it’s harder to maintain attention if you’re met with a wall of text, so, I try to have at most 3-4 sentences per paragraph and 1-2 paragraphs per survey page.

Another thing is that the above tips seem mostly aimed at reducing noise in your data, i.e. random variation with a mean of zero. But you also want make sure to phrase your questions in a way that avoid bias, i.e. variation that tends in a specific direction. I don’t have a very principled approach to this, but some things I consider are:

  • Avoiding leading questions
  • Avoiding phrases that may prime participants one way or the other
  • Randomising the order of questions whenever possible to avoid any order effects
  • Making sure that questions in different survey conditions are as similar as possible and only differ with respect to the particular thing you’re experimentally manipulating.

Curious to hear advice from others!

1 Like

Special focus: Sensitive questions

Some advice to increase the likelihood that you will get honest answers to sensitive questions from your participants.

Getting honest answers to sensitive questions from your participants is a tricky issue. There are, however, a few steps you can take to ensure that your results are less biased by participants’ social desirability.

First, make sure to mention in the briefing that participants’ responses are fully anonymous. This is especially important for questions that might ask for illegal behavior such as drug abuse. The good news is: People already feel more anonymous when filling out an online questionnaire compared to face-to-face interviews.

Second, pay special attention to how you phrase your question. You can either emphasize that the phenomenon you are asking about is perfectly normal or widely spread (e.g., by referring to a newspaper or research article about it), or by pointing out at the beginning of the question that you are asking the participants for their personal opinion.

Lastly, it might be a good idea to pilot test these sensitive questions in order to see if people would actually answer them, or if you need to rephrase them.

Depending on the type of sensitive questions, it can be useful to provide contact details of certain helplines in your debriefing. For example, if your study investigated drug abuse, providing respective helpline details can make sense.

Source: https://dism.ssri.duke.edu/survey-help/tipsheets/tipsheet-sensitive-questions

1 Like

The Order of Your Questions

Question order can have a big influence on your participants’ response behavior - here is what you should know before you start.

When constructing a study, you’ll need to decide on an order for your questions. You might think this doesn’t matter too much, but question order can make quite a difference to how your participants respond.

You should start with general questions that are easy to answer and not too sensitive or overly personal. There is a debate about whether sensitive questions should go near the start or nearer to the end of your questionnaire. On the one hand, asking sensitive questions early allows participants to drop out of the study quickly if they feel uncomfortable answering them. On the other hand, asking sensitive questions late means you can use the preceding questions to get the participant into the right mindset to answer more sensitive questions.

It is also preferable to keep questions about the same topic or area together. If you have different response formats, it might make sense to group your questions by both response format and topic area. Indicate the topic area by headings and separate the different groups of questions by page breaks.

Again, a pilot test can be very helpful to detect if the order that you have chosen is introducing any kind of confusion or bias to the data you obtain.

If you are trying to avoid biases introduced by the order of the questions, or if you simply want to test your questions, it may be a good idea to randomize the questions, or blocks of questions.

Sources:

https://surveymethods.com/blog/how-survey-question-randomization-can-be-used-for-question-testing/

https://www.qualtrics.com/support/survey-platform/survey-module/block-options/question-randomization/

1 Like

Tips for the visual design of your survey

We present three basic design tips to make your survey look great.

Did you know that visual design may influence your participants’ responses? Regardless of the tool you use to create the study you launch on Prolific, there are some basic design tips that you should follow in order to keep your participants’ motivation high throughout the study.

First, make sure that the goal of your study matches the design. Bright colours may be suitable for a study about creativity, but probably not if you are asking about alcohol consumption.

Second, ensure that your questions and instructions are visually easy to read. For example, you could use a sans-serif font like Arial, which is easy to read. However, if your participants are supposed to read a longer text, you should choose a serif font. Regarding the choice of colour, make sure you always maximise the contrast between the text and your background to enable easy reading.

Third, we recommend you have a progress bar! This is especially important for longer or monotonous studies. The progress bar can be in percent or give the exact amount of remaining questions. Don’t make the pages of your questionnaire too long, but at the same time, try to group questions about the same topic together.

Sources: