The Politician Truth Ratings Project

 

Project Summary

The goal of this project is to raise political truth literacy enough so that lying is no longer the winning strategy, as it is now for the wave of populist and authoritarian politicians we are seeing around the globe. Instead, telling the truth (about what's best for the common good) becomes the winning strategy.

Once that happens everything changes because the human system has undergone a mode change. Instead of the system serving powerful special interests, which is who populists and authoritarians are really serving, the system now "wants" to optimize the common good of all.

The strategy is to raise politician truth literacy by a combination of:

1. Education on how to spot commonly used fallacies. Lying politicians use the same fallacies over and over, like false dilemma, ad hominem attack, cherry picking, and strawman. They also use the same fallacious strategies over and over, like false promise and false common enemy.

2. How to use Politician Truth Ratings to make better decisions. Ratings help people make complex decisions more correctly. Ratings work so well they are everywhere, like product ratings, bond ratings, wine ratings, and restaurant ratings. But where are the ratings citizens need to make the very complex decision of who to vote for? They are nowhere to be found. Politician Truth Ratings fills that gap.

The prototype is here.

Overview and videos

Thwink.org is implementing the Politician Truth Ratings solution element by building the the Truth Ratings System (TRS). A rating will look something like this:

Politician Truth Rating mockup

For an introduction to why we need Politician Truth Ratings, how they work, and how this is a new form of journalism, please see this video. Watch in full screen mode so you can see the detail. Use the best speakers or headphones you have since there's a movie-quality soundtrack.

 

After you've watched the first video you may be interested in this second video:

 

 

(NOTE - The version in the videos is old compared to the current version. These are Vimeo videos. Some browsers, such as Firefox, may have trouble running the videos. If that occurs try Chrome.)

These videos are undergoing development. If you have suggestions for improvement or questions, please contact us.

Below is a frame from the above video at 17:33 illustrating how Structured Argument Analysis can someday (with considerable automation) be applied to all important knowledge. This would be a new form of knowledge that could be called Structured Argument Knowledge. At 16:00 the video says:

"There's some pretty important implications here. Structured Argument Analysis can be used on knowledge of any kind. Whatever it's applied to becomes trustworthy, because you know its level of truth. At first we plan to apply it to politicians so we can have trustworthy politicians. ''...

But that's just a start. The real potential is to gradually apply Structured Argument Analysis to all knowledge, to the entire world-wide-web of knowledge that we depend on for a high quality of life. (This will require a high degree of automation, which will eventually come, but we don't know when.)

"Imagine what it would be like to read any news article and see its truth rating, which would be the average of all the rated claims in the article. And imagine doing the same with Wikipedia entries, magazine articles, talk show transcriptions.... (and so on)"

WWW of knowledge ratings

For a thorough introduction of the analysis behind the Politician Truth Ratings solution element, see the Democracy in Crisis film series. Note that you can examine a graphic summarizing key points in each film by studying the images on this page. These will be hard to fully understand, however, without watching the films.

The first generation of this project was called the Political Persuasion Knowledge Base (PPKB).

One long-term plan we have is a browser add-on that shows the Truth Rating for news sources, if they have been rated. It will take a long time to reach this point but if we can do it, the add-on will go a long way toward combating fake news, propaganda, and political deception in general. The Truth Rating System can be used to develop ratings for any kind of source, not just politicians.

Prototype

The prototype is here. Please note it's an early version and currently runs only in Chrome on Windows and MacOS, thought we've not yet tested it on Linux.

We have recently produced an improved version with the info and articles panel moved to the left. The improved version differs quite a bit from the ones shown in the videos. The largest difference is it has so many features that it's ready to produce actual claim-check articles.

However, because there is no server database it's essentially a single-user application. To simulate a multi-user application, users producing claim-checks need to export their work and email it to Thwink.org so we can update the simulated master database. How this can be done is briefly described in the Help documentation. See the section on Prototype limitations and testing. Please be sure to work closely with Jack Harich at Thwink.org if you are considering testing or using the tool to produce claim-checks on a limited basis.

What we're actually building is the Truth Ratings System. It can be used to rate politicians, pundits, news organizations, or any source of "truth." But our first priority is rating politicians.

This entire project is 100% open source.

If you have comments, questions, or suggestions, please contact us. Thanks!

Risk Management

We foresee two main risks to the success of this project:

Risk 1. Claims cannot be analyzed accurately.

The book Deciding What’s True: The Rise of Political Fact-Checking in American Journalism, by Lucas Graves, 2016, states that:

"... the question of rating claims [is] the most visible and contentious dividing line in the fact-checking world." (page 41)

"For instance, FactCheck.org vocally rejects the ratings systems developed by its newer rivals. Brooks Jackson [the founder of FactCheck.org] has said there's no 'academically respectable way' to rate the truth of political claims." (page 39)

In short, the largest controversy among fact-checkers is whether the probability of truth of fact-checks can be accurately determined. We strongly think it can if the right tools are used. Politician Truth Ratings depends on this assumption. But what if we’re wrong? What if a rating cannot be accurately determined?

To us accuracy means two things:

1. A claim can be objectively analyzed, meaning it will not be biased.

2. A claim can be analyzed in a repeatable manner, meaning different analysts will all arrive at about the same truth confidence level.

If an analysis is both objective and repeatable, the analysis closely measures reality and is therefore accurate and may be considered a scientific measure of something.

To make analyses objective and repeatable, and thus accurate, analysts follow strict protocols. A protocol specifies how a particular step is to be performed. This is a widely used practice in many industries, like medical procedures, aircraft flight and maintenance, and science procedures.

We don’t yet know what the protocols will be. Fact-checkers follow established protocols and are trained in their use. We expect the TRS protocols will use these as a starting point and evolve the rest, based on study of tool use.

Once TRS can produce claim-checks, we can begin experimentation on claim analysis accuracy. A long series of experiments will allow us to converge on protocols that produce accurate results within a certain range, called the analysis accuracy error. Once we know this range, for a Politician Truth Rating it will be added to the margin of sampling error to give the margin of rating error. For a claim-check, the margin of error is the analysis accuracy error.

We expect that the simpler an argument is, the lower the analysis accuracy error. This may give rise to levels of claim complexity. The more complex, the lower the accuracy and the larger the error. There’s a lot of interesting research ahead in this project. How we plan to get started is briefly described in the Help documentation in the section on Prototype limitations and testing.

Risk 2. The cost of analyzing one claim is too high

If we cannot bring the cost of analyzing one claim down to a very low amount, organizations will find it too expensive to produce Truth Ratings. The cost is mostly labor, i.e. how long it takes a person or team to analyze a claim.

Calculating a Politician Truth Rating requires averaging hundreds of randomly selected claims. Suppose total claims for a politician is 2,000. For a margin of sampling error of +/- 3%, which is what polls use, we need a sample size of 696 claims. That a lot of claims!

Here are some very rough estimates. Currently it takes fact-checkers several hours (?) to prepare a fact-check article for a new claim. Let’s estimate this is two hours at a minimum. 2 hours per claim times 696 claims equals 1,392 hours of labor. The median journalist salary in the US is $36,000 per year. Analyzing claims is skilled work, plus there’s organization overhead, so let’s up the salary to $50,000 per year. At 2,000 hours per year, 1,392 / 2000 times $50,000 equals $34,800 to create one rating. That’s unaffordable for all but the largest well-funded organizations, because there are so many candidates to rate during election season. Therefore, TRS must cut this time by an order of magnitude, and thus cut the cost to about $3,480 per claim. That’s affordable.

The text editor and argument map are designed to let users work as fast as they can think, type, and use the mouse to mark nodes and create a map. We expect that with practice the tool will allow users to analyze claims at almost the speed of thought. Like a good word processor, the tool will not be in their way.

We think achieving this level of productivity can be achieved by:

1. Eventually, after many claims are entered, 90% or more of the Rules, Facts, and Reusable Claims needed to analyze a claim will already be in the database, because what politicians say contains a high amount of repetition.

2. Extremely fast lookup of a needed Rule, Fact, or Reusable Claim from the database and use of that node in the text. For each of the three database node types, users can search node summary and description, and for rules, by exploring a tree of nodes.

3. A smoothly working, highly intuitive, easy to learn text editor and argument mapper.

To speed up managing this risk the prototype doesn't produce ratings. It only analyzes claims. This allows us to address these two risks simultaneously as we proceed with the work described in the Help documentation in the section on Prototype limitations and testing.