An Experimental Study to Increase Consensus on Polarized Topics

An Experimental Study to Increase Consensus on Polarized Topics

Research Team

Stefano Balietti, University of Heidelberg and University of Mannheim
Samuel Fraiberger, MIT, NYU, World Bank
Amir Tohidi, MIT
Manuel Tonneau, NYU, World Bank, Centre Marc Bloch

State of research

Partisan bias has become a new social norm. Survey respondents openly disclose negative sentiments towards outgroup political members (Lelkes, 2016), and parents reveal that they would be unhappy if their child were to marry someone supporting another political party (Iyengar et al., 2012). It is therefore not so surprising if less than 15% of the US population recognize core moral traits in members of the outgroup party (Miller and Conover, 2015), who are also perceived to be less physically attractive (Klar and Krupnikov, 2016; Nicholson et al., 2016), less worthy of academic scholarships (Iyengar and Westwood, 2015), and receive fewer job interview callbacks (Gift and Gift, 2015).

Scholars unanimously agree that since the 1970s there has been an increase in political polarization (Poole and Rosenthal, 1997; McCarty, 2019), even if the causes of this trend are still debated (Winkler, 2019). The Internet and social networks have often come under fire for promoting either voluntary (Sunstein, 2001;) or algorithmic (Adamic and Glance, 2005; Pariser, 2011) attitudinal-consistent readership, and, more recently, for allowing ad microtargeting (Settle, 2019; Kim et al. 2018), and the spread of misinformation campaigns (Benkler et al., 2018).

In addition, there is also a debate about the actual types of polarization that are in place (Hetherington, 2009; Abramowitz, 2010; Mason, 2015; Fiorina, 2017). In particular, about the mass public, Fiorina et al. (2005) talk about “The Myth of Polarization,” while Mason (2015) coined the expression “I disrespectfully agree,” to emphasize how the actual issue positions of the mass public are not so far away or even reconcilable, despite very strong ideological feelings of identity to one’s own party.

A large body of research has highlighted how people often do not understand the complexity of the issues at stake (Norton and Ariely, 2011; Fernbach et al., 2013; Hauser and Norton, 2017), but providing additional information or corrections may backfire and increase polarization instead (Nyhan and J. Reifer, 2010; Alesina et al., 2018; Mosleh et al., 2021). In this line of research additional information comes usually in the form of news articles manually selected or ad-hoc crafted by the experimenter (Sunstein et al. 2016). In this project we take a novel and interdisciplinary approach, leveraging massive datasets of political news in conjunction with state-of-the-art deep learning classifiers to feed data to a series of online survey experiments, as described below.

Research strategy

This proposal aims at measuring how selective media exposure might have contributed to the polarizing trend of the mass public. To do so, we have collected real-world news from the news provider Factiva (Factiva - Global News Monitoring & Search Engine | Dow Jones) in collaboration with the World Bank about two controversial and politicized topics: climate change (3mln news), and inequality and wealth redistribution (30k news). T

  1. Following Budak et al. (2016)’s crowdsourcing approach to news classification, news articles are labelled for their stance on the topic using the nodeGame software (Balietti, 2017). In this step, we also collect information about both the labeler and the news:
  • info about labeler: socio-demographic indicators, as well as his or her issue position on the controversial political topic;
  • labels about news: stance, policies, style; in addition: the expected belief of the labeler of how such a news will impact the attitudes towards the controversial political topic of other readers with a given issue position.

This step has been piloted for the topic of inequality and wealth redistribution.

  1. Using a state-of-the-art pre-trained language model tailored for long text sequences (Beltagy et al., 2020) and the set of labels from step 1, we will predict the stance position for the full set of articles. We will distinguish between absolute vs perceived stance. For computing the perceived stance, classifiers will include features of the recipients, such as their own stance, and of the news source (Pennycook and Rand, 2019). To classify news sources, existing algorithms to compute media slant could be used (Groseclose and Milyo, 2005; Gentzkow and Shapiro, 2010).

  2. For any given controversial political topic, we define the two opposite stance positions as L and R. Then, using the output of the classifier at Step 2, we determine the news articles that, according to labelers with position L, have the greatest potential to change the attitude of L (same-stance) and R (cross-stance) readers (likewise for predictions by labelers with position R). The total number of treatment arms is 30 (see preregistration).

  3. We run a number of survey-experiments on the topic of wealth redistribution in which:

  • We collect socio-demographic information and stance position of respondents
  • We assign one news article conditionally to the stance position of each respondent
  • For the control group, we assign a set of neutral articles
  • We re-measure the stance position of the respondent
  • We collect a behavioral outcome, i.e., foregoing part of bonus earnings to a campaign or an organization in favor or against the topic at stake.
  • We perform a comprehension check on the news article similar to the labeling task in Step 1, to feed back data into the classifier.

We can then answer the following research questions:

  • Can classifiers correctly identify news increasing (decreasing) polarization?
  • Are same-stance or cross-stance predictions more accurate?
  • Are behavioral outcomes, i.e., donations of earnings, in line with self-reported opinion shifts?

How or why you chose your sample size

Our sample will be adults in the United States. We will not use a representative sample, but we will manually stratify age (2 classes), income (2 classes), sex (3 classes), and political leaning (3 classes).

Given that we plan to use LMM, an accurate power calculation (e.g., with simr package) can be done only after a pilot. However, based on the results of a similar study by Balietti et al. (2021), we estimated the low-end effect size as Cohen’s D ~ 0.25. This requires 243 obs. per treatment to detect a null effect with a power of 80% using t-test. Under this assumption we can run 15 treatment arms. At minimum, we will run 9 arms (4 cross-stance L/R + 4 same-stance L/R + 1 control).

Total Participants: ~3645.

Description of the study costs

-Total Requested 10,000£

Collection of labels: ~£1810

Pilot: £900

Experiments: (£1 Fee + £0.5 Bonus) x 3,645 = £7,290

Bonus is guaranteed, but participants may decide to forego part of it as a donation. Notice: if a participant donates 50% of the bonus, for a 10 minute survey earnings are £7.5/h.

Explain how you’ll make your findings, study materials, analysis code and data openly available

  • Findings will be published in international peer-review journals;
  • Analysis code and aggregated research data will be published in a public online repository (e.g., The Dataverse https://dataverse.org/ );
  • The nodeGame experimental code as well as a live survey experiment will be available at: https://nodegamedemo.herokuapp.com/.
  • The classifiers will be open sourced in the Hugging Face model hub (https://huggingface.co/models)

Evidence that you’ve preregistered key aspects of your study.

Preregistration link: OSF | Predicting Polarization: Perceived vs Actual Opinion Shifts via Exposure to News Articles

References

Abramowitz, A., 2010. The disappearing center: Engaged citizens, polarization, and American democracy. Yale University Press.

Adamic, L.A. and Glance, N., 2005. The political blogosphere and the 2004 US election: divided they blog. In Proceedings of the 3rd international workshop on Link discovery (pp. 36-43). ACM.

Alesina, A. et al. 2018. Intergenerational mobility and preferences for redistribution". American Economic Review 108.2 , pp. 521-554.

Balietti, S. et al., 2021. Reducing opinion polarization through exposure to the views of selected peers. Submitted.

Balietti, S., 2017. nodeGame: Real-time, synchronous, online experiments in the browser. Behavior research methods, 49(5), pp.1696-1715.

Beltagy, I., Peters, M. E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

Benkler, Y., Faris, R. and Roberts, H., 2018. Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press.

Budak, C., Goel, S. and Rao, J.M., 2016. Fair and balanced? quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly, 80(S1), pp.250-271.

Drouvelis, M. and Isen, A. and Marx, B., 2019. The Bonus-income donation norm. CESifo Working Paper No. 7961

Fernbach, P.M., Rogers, T. and Fox., C.R., 2013. Political extremism is supported by an illusion of understanding. Psychological Science 24(6), pp. 939-946.

Fiorina, M.P., Abrams, S.J. and Pope, J.C., 2005. Culture war. The myth of a polarized America.

Fiorina, M.P., 2017. Unstable Majorities: Polarization, Party Sorting, and Political Stalemate. Hoover.

Gentzkow, M. and Shapiro, J.M., 2010. What drives media slant? Evidence from US daily newspapers. Econometrica, 78(1), pp.35-71.

Gift, K. and Gift, T., 2015. Does politics influence hiring? Evidence from a randomized experiment. Political Behavior, 37(3), pp.653-675.

Groseclose, T. and Milyo, J., 2005. A measure of media bias. The Quarterly Journal of Economics, 120(4), pp.1191-1237.

Hetherington, M.J., 2009. Putting polarization in perspective. Brit. Journal Pol Sci., 39(2), pp.413-448.

Iyengar, S., Sood, G. and Lelkes, Y., 2012. Affect, not ideology: A social identity perspective on polarization. Public opinion quarterly, 76(3), pp.405-431.

Iyengar, S. and Westwood, S.J., 2015. Fear and loathing across party lines: New evidence on group polarization. American Journal of Political Science, 59(3), pp.690-707.

Klar, S. and Krupnikov, Y., 2016. Independent politics. Cambridge University Press.

Lelkes, Y., 2016. Mass polarization: Manifestations and measurements. Public Opinion Quarterly, 80(S1), pp.392-410.

Mason, L., 2015. “I disrespectfully agree”: The differential effects of partisan sorting on social and issue polarization. American Journal of Political Science, 59(1), pp.128-145.

McCarty, N., 2019. Polarization: What Everyone Needs to Know®. Oxford University Press.

Miller, P.R. and Conover, P.J., 2015. Red and blue states of mind: Partisan hostility and voting in the United States. Political Research Quarterly, 68(2), pp.225-239.

Nicholson, S.P., Coe, C.M., Emory, J. and Song, A.V., 2016. The politics of beauty: The effects of partisan bias on physical attractiveness. Political Behavior, 38(4), pp.883-898.

Nyhan, B. and Reifer, J., 2010. When corrections fail: The persistence of political misperceptions. Political Behavior 32 (2) , pp. 303-330.

Norton, M.I. and Ariely, D., 2011. Building a better America one wealth quintile at a time". Perspectives on Psychological Science 6(1), pp. 9-12.

Hauser, O.P, and Norton, M.I., 2017. (Mis)perceptions of inequality". Current Opinion in Psychology 18 , pp. 21-25.

Mosleh, M. et al., 2021. Perverse Downstream Consequences of Debunking: Being Cor-rected by Another User for Posting False Political News Increases Subsequent Sharing of Low Quality, Partisan, and Toxic Content in a Twitter Field Experiment Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–13.

Pariser, E., 2011. The filter bubble: What the Internet is hiding from you. Penguin UK.

Pennycook, G. and Rand, D.G., 2019. Fighting misinformation on social media using crowdsourced judgments of news source quality. PNAS, 116(7), pp.2521-2526.

Poole, K. T. and Rosenthal, H., 1997. Ideology and Congress. Transaction Publishers.

Settle, J.E., 2018. Frenemies: How social media polarizes America. Cambridge University Press.

Sunstein, C.R., 2001. Republic. com. Princeton university press.

Sunstein, C.R., Bobadilla-Suarez, S., Lazzaro, S.C. and Sharot, T., 2016. How people update beliefs about climate change: Good news and bad news. Cornell L. Rev., 102

Winkler, H., 2019. The effect of income inequality on political polarization: Evidence from European regions, 2002–2014. Economics & Politics.

Forgot icon attribution link:

1 Like