Nyhan, Brendan and Jason Reifler. 2014. "The Effect of Fact-checking on Elites: A field experiment on U.S. state legislators". American Journal of Political Science.
Abstract
Research suggests fact-checking may be ineffective at reducing public misperceptions about controversial issues.
H1: fact checking might instead help improve political discourse by increasing reputational costs or risks of spreading misinformation for political elites.
H1: fact checking ⬆ -> reputation costs ⬆ -> political discourse quality ⬆
The effect of fact-checking on politicians
Experimental design
- IV: assignation of treatment condition i.e. assignation of letter emphasizing the risks of having misleading or inaccurate statements exposed by fact checkers.
- Placebo-IV: assignation of Hawthrone (placebo) condition i.e. assignation of letter alerting legislators that a study was being conducted of the accuracy of statements made by politicians (excluding language about fact-checking or consequences of inaccurate statements)
- Control group: no treatment
- DVs:
1) negative rating from PolitiFact
2) media coverage questioning statement accuracy of legislators
3) combination of 1) & 2)
- Control variables: state, political party, legislative chamber, wether or not a legislator had previously received a PolitiFact rating, previous vote share, and fundraising.
(! - that's us)
(! - that's what we want to achieve)
It would be worth it asking for the Supporting Materials of this article, there are probably many useful things, among others the details of the procedures followed to ensure intercoder reliability in their experiment using Krippendorff's alpha. nyhan@dartmouth.edu / jreifler@exeter.ac.uk
Results
State legislators who were sent letters about the threat posed by fact-checkers (IV) were less likely to have their claims questioned as misleading or inaccurate (DV) during the fall campaign.
- Treatment effect for DV1: not significant at the p < .05 level (one tailed), but expected direction.
- Treatment effect for DV2: significant at the p < .05 level.
- Treatment effect for DV3: highly significant at the p < .02 level.
Literature worth checking
- Amazeen, Michelle. 2012. "Blind spots; Examining political advertising misinformation and how the U.S. news media hold political actors accountable." Ph.D. dissertation, Temple University.
- Nyhan, Brendan. 2012. "Another fact checking fiasco: Journalistic failure in coverage of Harry Reid and his mysterious source." Columbia Journalism Review. United States Project.
- Nyhan, Brendan. 2013. "That's not a factcheck! How punditry undermines the mission of journalistic watchdogs."Columbia Journalism Review. United States Project.
- Those are the highlights, but there are more valuable topics in the References.
Abstract
Research suggests fact-checking may be ineffective at reducing public misperceptions about controversial issues.
H1: fact checking might instead help improve political discourse by increasing reputational costs or risks of spreading misinformation for political elites.
H1: fact checking ⬆ -> reputation costs ⬆ -> political discourse quality ⬆
The effect of fact-checking on politicians
[...] major news organizations [...] frequently refrain from questioning the accuracy of contested claims made by public figures even when the statements are verifiable
[fact checking] represents a potentially radical change in how journalism is practiced with significant consequences for political accountability and democratic discourse [... it] has the potential to create career risks for politicians by generating negative coverage [...] that could damage their reputation and credibility.
- IV: assignation of treatment condition i.e. assignation of letter emphasizing the risks of having misleading or inaccurate statements exposed by fact checkers.
- Placebo-IV: assignation of Hawthrone (placebo) condition i.e. assignation of letter alerting legislators that a study was being conducted of the accuracy of statements made by politicians (excluding language about fact-checking or consequences of inaccurate statements)
- Control group: no treatment
- DVs:
1) negative rating from PolitiFact
2) media coverage questioning statement accuracy of legislators
3) combination of 1) & 2)
- Control variables: state, political party, legislative chamber, wether or not a legislator had previously received a PolitiFact rating, previous vote share, and fundraising.
Due to the complexities of language and politics, no perfectly objective measure of statement accuracy has yet been created.
One potential concern is whether PolitiFact truth ratings are consistent and accurate.
It would be worth it asking for the Supporting Materials of this article, there are probably many useful things, among others the details of the procedures followed to ensure intercoder reliability in their experiment using Krippendorff's alpha. nyhan@dartmouth.edu / jreifler@exeter.ac.uk
Results
State legislators who were sent letters about the threat posed by fact-checkers (IV) were less likely to have their claims questioned as misleading or inaccurate (DV) during the fall campaign.
- Treatment effect for DV1: not significant at the p < .05 level (one tailed), but expected direction.
- Treatment effect for DV2: significant at the p < .05 level.
- Treatment effect for DV3: highly significant at the p < .02 level.
Literature worth checking
- Amazeen, Michelle. 2012. "Blind spots; Examining political advertising misinformation and how the U.S. news media hold political actors accountable." Ph.D. dissertation, Temple University.
- Nyhan, Brendan. 2012. "Another fact checking fiasco: Journalistic failure in coverage of Harry Reid and his mysterious source." Columbia Journalism Review. United States Project.
- Nyhan, Brendan. 2013. "That's not a factcheck! How punditry undermines the mission of journalistic watchdogs."Columbia Journalism Review. United States Project.
- Those are the highlights, but there are more valuable topics in the References.
Last edited: