How can we improve the language screeners on Prolific?

Hey Prolificos ,

We are currently doing a review of the language screeners on Prolific and would love your feedback on how we can improve them for you:

  • Are you missing some screeners?
  • Could some screeners be improved? ( can be done by improving the questions, definitions…)

Specifically, we would love to get insights from those of you working in @Linguistics as to which aspect of language matter most to you when screening participants (e.g. competence, fluency, location, type or chronology of exposure).

Looking forward to chatting!

2 Likes

Hello,
I think it could be a good idea to define “first language”, as this led to a misunderstanding with one of my participants. I pre-screened for first language = French, and in my instructions I wrote, in French, that participants needed to have French as their mother tongue (langue maternelle), which is the most common expression used in French. In the experiment, I also included one question asking the participants (in French) “is French (one of your) your mother tongues?”, and a few participants answered “no”. I discussed this with one of them, and they said that they were raised by a French-speaking person during their first years (so French was their first language), but they lost French when the French-speaking person did no longer raise them. So, English is their primary language but French was their first one.

3 Likes

The questions are fine except the last one about time spent before 18. Why the UK? Why not have it open ended or with more choices? For example, I’d want to know the country for my research between, say Argentina and Colombia. That’s impossible here. On a different topic, the lack of Latin American countries among the pool is highly unfortunate. I can only get Mexicans, really.

Thanks

Michael Newman
Professor of Linguistics and Chair

Department of Linguistics and Communication Disorders
Queens College/CUNY

mnewman@qc.cuny.edu

1 Like

Thank you for reviewing these! I agree with @Julia_Tibblin that it would be good to define “first language” because, as she said, the language someone learns chronologically first does not necessarily remain their primary language. Maybe they should be two questions - What language(s) were you exposed to before age 5? and What language(s) do you consider your primary language at this time? (I would also note that someone could have more than one language in response to each of these questions, so the response options should allow for that.)

I would say that “bilingual” and “fluent” could use more precise wording, too. For my research, I would mostly be interested in current use, so instead of asking if someone is bilingual, I would ask something like, Do you use more than one language on a regular basis? or What language(s) do you use on a regular basis? “Bilingual” can be a charged term and someone who is insecure about their proficiency might say that they are not bilingual, when maybe the researcher would consider them to be so. For that same reason, I don’t think “fluently” should be part of the description of being bilingual (or the response options for the question about being monolingual).

If you want to include screening options that take level of proficiency into account (to replace the idea of fluency), then the ideal would be to have questions about different language skills, like rating your skill on a scale of 1-5 for speaking/signing, reading, writing, and understanding spoken/signed language, because someone can be very skilled at speaking but struggle with writing if it’s a language they didn’t use in school, or vice versa if it’s a language that they learned in a classroom but don’t use much outside of the classroom setting. Of course, there are issues with self-ratings too, but it would be a start.

3 Likes

Thank you for asking.

If I want to screen for:

I agree with Julia. A short definition of what “first language” is would be helpful. Apart from a couple of participants who lost their first language or were raised bilingual and were confused about which language to choose, I also encountered people who understood “first language” as a “first language they use in communication”. So they indicated English as the first language in the Prolific questionnaire. Still, when asked in my demographics questionnaire about their native language (and follow up questions about the language of their education, languages spoken at home when they were growing up, etc), they appeared to be second-language speakers of English who happen to live abroad now and use English on a daily basis.

2 Likes

That’s a good point! A more general question about “Place of most time spent before 18” should include various countries because there are many participants on Prolific who currently live in the UK, but they were not born or raised here. That will be relevant for studies interested in how much immersion in L2 environment people had etc.

1 Like

So many important points! :heart_eyes: Yeah, we could also talk about “early bilinguals” and “late bilinguals” and discuss what the cutoff would be.

1 Like

I wonder if the distinction between British English and Amarican English (and perhaps other English varieties or varieties of different languages) would matter for other researchers?

1 Like

Yes,

And prolific is used outside of the UK. My prolific participants have been in Spain and Mexico.

In our own studies, we usually ask about three aspects of language use:

  1. Age of acquisition (AoA) – for each language you speak, when did you acquire it?
  2. Proficiency-- in your own estimation, how good are you at using this language?
  3. Current usage-- what percentage of the time do you use this language in your life right now?

Some more detailed questionnaires will break up proficiency and usage into smaller categories, like reading versus speaking proficiency, usage at work versus usage at home, etc.

Another new and interesting feature is language entropy, or how mixed a person’s language experience is. Are there conversations where they switch fluidly between two languages, or are languages always used in separate context?

For example, take a look at the list of language features here from a recent paper:

Gullifer, J. W., Kousaie, S., Gilbert, A. C., Grant, A., Giroud, N., Coulter, K., Klein, D., Baum, S., Phillips, N., & Titone, D. (2021). Bilingual language experience as a multidimensional spectrum: Associations with objective and subjective language proficiency. Applied Psycholinguistics , 42 (2), 245–278. Bilingual language experience as a multidimensional spectrum: Associations with objective and subjective language proficiency | Applied Psycholinguistics | Cambridge Core

Gullifer, J. W., & Titone, D. (2019). Characterizing the social diversity of bilingualism using language entropy. Bilingualism . Characterizing the social diversity of bilingualism using language entropy | Bilingualism: Language and Cognition | Cambridge Core

1 Like

In terms of early versus late bilinguals, I think different researchers will want the option of having different cutoff points; so I’d rather ask for age of acquisition than have a pre-determined early/late bilingual filter.

1 Like

Another term we could bring into the conversation is a person’s “dominant” language. Even cradle-bilinguals who have grown up with two different languages from birth can have one language that comes to them easier than the other. Not sure how we could phrase a question around this, but it could be informative.

Hi, I want to screen for political orientation (left-right) in Spanish speakers from Mexico and Spain, for example.

In my research focused on bilinguals, I am often interested in ages when people learn languages, which could make for useful screeners. For example, in some studies, I want only participants who were exposed to both languages in early childhood (say, before age 6). In another, I might want only those who were definitely not exposed to the second language until later (say, after age 14).

On research with Spanish speakers in the US, I am sometimes interested in what variety of Spanish they were most exposed to, so it could be helpful to screen for something like their parents birthplaces - someone with both parents from Mexico and someone with both parents from Puerto Rico speak different dialects of Spanish, and that can be something good to control.

Finally, this may already be a screener (I can’t recall), but knowing where someone was educated can tell you a lot. So, for example, being able to screen for what country someone did their primary and secondary schooling in would be useful.

Hope those ideas help!

1 Like

Thank you all for your feedback and great suggestions, that is really appreciated.
We’ve been summarising your input, and will review and take a number of actions to address it in the coming weeks.
I’ll get back to you once those changes and additions to the language screeners have been implemented.
Many thanks again to you all!

2 Likes

Thank you so much for your input here. We’ve found it all to be super valuable. For now, here are the steps we’ll be taking:

  • Create a primary language spoken screener - this would be to capture which language participant speak most currently

  • Create a native language screener, this would be to capture the first language learned in life by participant

  • Expire the first language screener when both of the above have been taken up

  • Expand the list of countries to the time spend before 18 screener so that it’s not only UK centric

We’re still discussing your other points, and let you know if we’re able to action them.

@Lucie_Barouillet will give a summary of the exact changes we’ve made once they’re live on the platform :slight_smile:

@Brad_Hoot @Alejandro_Rosas @aisa2 @Michael_Newman @Magdalena_Kachlicka @AKronrod @Jessica @Julia_Tibblin

2 Likes

Primary language screener should be a single-answer option. Currently many people reproduce “fluent” languages in the “primary” language category- when there should only be on “primary” language by definition.