Rating sources: NewsGuard

#1
Recently, Jack made me aware of this article by WikiTribune, where they talk about NewsGuard, a plug-in by Microsoft that rates news websites trustworthiness. Part of the TRS is calculating what we call a "Source Credibility Rating" (CR), which makes it interesting taking a look at similar efforts by others.

These are some of my first impressions, and questions that could be worth asking, if we contacted them in the future.

Comments
  • The binary rating of green vs red is better than nothing, but actually pretty simplistic. We are offering way more precision than that.
  • One of the measures that I see they are taking to ensure reliability is that the ratings are set by more than one analyst. This is good but may not be enough. Critics could say that they are a cluster of like-minded people.
  • Their highest weighted criterion (22/100 points) is that the site "does not reportedly publish false content". In order to determine if this is the case or not, they rely on the assessments of "journalists of NewsGuard or elsewhere". That of course can be (1) too subjective if using solely their journalists' assessments, or (2) fall in a circular trap of using source A to determine the truthfulness of information published in source B, source B to determine the truthfulness of source C, and source C to determine the truthfulness of source A.
  • Their criterion "handles the difference between news and opinion responsibly" is the only one that includes some of what we call Patterns of Deception. The patterns that they mention in the description seem like nothing more than a couple illustrative examples, but even if they had a more comprehensive list of PoDs, to me they are missing the central concept of a Deception Impact Density (DID). Without that, it is hard to tell how significant the impact is. They weighting of this criterion gives it 12.5/100 points, not too high for being the one identifying explicit deception patterns.
Questions
- How often do you update the rating of the websites?
- What analytical procedure did they follow to set the criteria and criteria weighting

I hope this is interesting, and feel free to add any thoughts on the matter to this thread!
 
Last edited:

Jack Harich

Administrator
Staff member
#2
Hey Montserrat, these are insightful notes. I like the way they pinpoint what matters.

That sites like these keep appearing indicates a large public thirst for knowing how trustworthy a news site is. That's a large market. We are approaching that market from a public service point of view. Others approach it with a profit motive, such as News Guard. They can maximize their market share by appearing to offer a green seal of approval for trustworthy news sites, when in fact that seal is not very accurate. It only ball parks a site's level of deception. Two levels, red and green, also greatly reduce the cost of determining a site's level of deception, which maximizes profits.

By contrast, our approach was designed to be as accurate and reliable as possible, with no thought about the cost of analyzing a news source.

I hope that we will be able to make setting a source's credibility rating fast without losing quality. If we can develop that technology, then it will probably be widely adopted. We are pioneers here, serving the common good by inventing a method for measuring the truth.

Another point. Doing a good job of setting a source's credibility rating will be somewhat expensive but not prohibitively so. It might average $1,000 per source. Hundreds or thousands of sources need to be rated per nation, and then re-rated every few years. The ratings need to be audited. They need to be 100% transparent and so well-written they are highly educational. All that points to the need for a single (?) large independent non-profit organization per nation or area to efficiently centralize this work. Their output would be shared by many types of customers: companies like News Guard, news article authors, solution element like Politician Truth Ratings, Wikipedia for use in determining if a source was good enough to use or not, and so on. Such an organization would be as fundamentally vital to the health of a nation as its police or judicial system. They would be "policing the truth."

News source credibility ratings would soon be seen as an epic change to the human system. It would be revolutionary, because at last Homo sapiens knows how to measure the truth!
 
#3
Thanks for the reply Jack!

An International Truth Policing Organization! I love it! That would be a completely new type of institution, and the total opposite of the "Ministry of Truth" in Orwell's 1984.