September 2018 Core Group Meetings

Jack Harich

Administrator
Staff member
#1
For the September 2 meeting at 8:00 pm EST.

Overall the research project is going very well, considering its difficulty.

Previously we've found some happy little bumps in the road. But recently we've discovered a large serious bump in the road, as described in the thread on Solving the problem of how to measure the truth. It's so large we need to all focus our efforts on solving it. Otherwise our project will probably fail.

This meeting is organized differently so we can focus our attention on the problem. Here's the agenda:

A. Discussion of solving the problem of how to measure the truth, with the goal of developing a plan of action.
  1. Please study the problem thread before the meeting, take notes, and develop your own thoughts on the problem.
  2. Do we all have the same definition of the problem? We take turns defining the problem in our own words.
  3. Is the problem well-structured? A well-structured problem is decomposed into the pieces that make up the problem in such a manner that makes getting started on analyzing the problem much easier. For example, SIP does this with a formal problem definition and by suggesting identifying the subproblems as the first step. Root cause analysis does this by suggesting a series of WHY questions as the first step. What I've done is structure the problem into subproblems to solve. These are the 6 categories of decisions. But is this good structure? Are other approaches possible?
  4. I'm beginning to suspect we have a missing abstraction(s). Tell story of what that term means, why so critical. What's missing is an abstraction that would tie our difficulties together with an insight explaining why we are having trouble and how it would be best to proceed, in a simple, elegant direction. One possibility for the missing abstraction is, for rule CLs between 0 and 100%, we are not using unified models of inductive logic. An example is the models discussed in this article on Inductive Logic. We don't need one model for everything. But we probably do need one model for each Spectrum Set type, as explained next.
  5. The crux of solving the truth measurement problem appears to be selecting the right rule with the right confidence level.
    • Each rule should specify what inputs are needed.
    • The inputs are a much easier part of the problem to solve, like setting fact CLs and selecting the right facts and reusable claims.
    • Presently the rules tree has 4 main container folders: Fallacies, Non-fallacies, Opposite Pairs, and Spectrum Sets. The first 3 have zero or 100% CLS in their rules. These are not a big problem. The problem lies in the Spectrum Sets that do not deal with statistical certainty. These are the inconsistent and consistent evidence leaf folders. These contain rule CLs between 0 and 100% , like 5% and 20%.
    • Click on the Evidence - Inconsistent folder. Study its description. That's a start on how to make selecting the right rule with the right CL easy and correct. The 0% CL rule selection criteria looks okay. (?) But the others look weak. The result will be low precision, which means high variability across claim-checks. How can we improve the criteria? This is exactly where I'm stuck right now. It may be this needs a formal model behind it.
    • Once we have solved the weak selection problem of the Evidence - Inconsistent folder, we can probably use that solution on the Evidence - Consistent folder.
    • Once we've solved the above two problems, we can then look at the 6 categories of decisions and attempt to solve all of them. I suspect that task will be much easier once we've solved the "selecting the right rule with the right confidence level" problem.
  6. This problem is an area Scott Collison may be able to help in.
  7. Any other thoughts on solving the problem of how to measure the truth?

B. Montserrat
  1. Research project report. This is the 6th report. The areas of the report are:
    • Risk status.
      • Is it possible we are too focused on details, and are missing some higher level risks or priorities?
    • Schedule status.
      • What about the impact of the above problem on the schedule?
    • Help needed in upcoming week.
    • Work focus and central challenges for the upcoming week.
    • Discussion as needed.
  2. Comments on her first claim check.
  3. Comments on her trip to take the scholarship test.
  4. Anything else?

C. Scott
  1. Any topics here?

D. Jack
  1. Good news. The software is stable, complete enough, and is now taking only about 20% of my time.
  2. I'm expecting to be working with Scott Collison soon.
  3. I'm focusing on the problem of how to measure the truth.
 
Last edited:
#2
After the long meeting we had last week, apparently this week we can expect a shorter, but still very interesting one!

Raise Truth Literacy Project

A. Project Reports
  1. Weekly report on Montserrat's Research Plan. This is the 7th report, available on the weekly report's thread after the meeting.
  2. Report on Jack's role in the research.
- On Friday I created a document on Solving the Problem of How to Measure the Truth and posted it to the forum. This summarizes my work during the week and has the beginning of an Implementation Plan for adding Bayes Rule to Structured Argument Analysis. By carefully putting these thoughts in writing, others can gain the benefit of my work and apply it. We can work better as a team.

- Now I'm working on the Implementation Plan steps. No progress yet. This is very difficult work. One thing I'm trying to do is take a unified approach by using Bayes Rule for all rules.

- I thwink this portion of our work is going to be a fine example of it takes a team instead of an individual to solve a difficult problem.
- Newest discovery on Adding Bayes Rule to SAA

Team Member Items

B. Montserrat
  1. Last week I went to Mexico to take the test for the scholarship, being away didn't let me be too productive last week, but I think the exam went well, and it was worth the short trip, now it's time to wait for the results!
  2. This week I'll resume my work on the R course, now that I don't have to study for the test anymore!

C. Scott
  1. General observations

D. Jack
  1. Any topics here?
 
Last edited:
#3
This is the agenda for the September 16 meeting (the Mexican Independence Day! :D)

Raise Truth Literacy Project

A. Project Reports

1. Weekly report on Montserrat's Research Plan. This is the 8th report. From this week on, a summary of my report will be included in the agenda.
Risk status
  • Since the beginning of this month we identified the main risk the project is facing: Solving the problem on how to measure the truth. This has been the main and only focus for us during the last weeks.
  • Last week I focused on studying the literature of Bayesian Networks and its applications in Legal Arguments (Fenton et al. 2012). Based on that, I started my own document on Solving the Problem of How to Measure the Truth. I tried to adopt the same structure of Jack's document, to be able to compare our approaches and conclusions 1:1. The document is a work in process, but I may be able to upload my first finished version of it to the forum as soon as tomorrow.
  • From summarizing the problem for myself, I realized there may be one way to narrow the knowledge gap even more. There is a step prior to setting the CLs of facts and rules, and the weights of rule inputs, namely deciding what the correct rule is, and what the necessary inputs for that rule are. It occurred to me that it can be worth analyzing the structure of the claims themselves, maybe we can find a way to categorize claims, in such a way that helps identifying more precisely what type of rule is going to be required. I have created a spreadsheet to analyze the structure of claims, and I've started working with the claims we have already checked, including the reusable claims. I have noticed a couple interesting patterns, but nothing too groundbreaking yet. The reason may be that (1) the sample of claims we have checked is still small, and most importantly (2) we have exclusively been working with claims coming from fact-checks, who are limited to work with a certain type of claims they call "empirical claims", a somewhat misleading term that refers to claims that include information such as numbers and figures, that can be compared to the source of the data. Again, this is not directly related of the question of how to set the CLs and weights, but I do think it is helpful solving all the things around that that we can.
Schedule status
  • As decided last week. We're working out of schedule until we have solve this problem. The main focus currently is figuring out a way to add Bayes Rule to SAA, to solve the problem of how to measure the truth.
Help needed
  • Review of types of claims with Jack
  • Review of our conclusions from the Bayes Theorem as needed
Goals for next week
  • Further review of types of claims
  • Continue studying Bayes Theorem
  • Work on the Bases Rule applications on R
2. Report on Jack's role in the research.

- I spent the week focusing on a review of the Bayes Rule literature related to the problem of how to measure the truth. The field is very fragmented and immature. Each article and book covers Bayes Rule and its use differently. Terms, symbols, equation names, and standard equation forms differ. A lot of effort is spent defending the Bayesian viewpoint versus the frequentist (traditional statistics) viewpoint. From the viewpoint of The Kuhn Cycle, logic/statistics is in the Model Revolution step.

- Despite the foggy literature, I was able to read enough to see approximately where the field is today. Bayes Rule is now accepted in statistics and machine reasoning where appropriate. Most of the money and applications relate to AI. The further you get from machine reasoning and large sample sizes, the more immature and uncertain the field is. Our application, how to measure the truth with a very small number of rule inputs (2 to 5), lies on the opposite end of the spectrum from machine reasoning and large sample sizes. Thus, it's no surprise that the knowledge we need is simply not there. It's so fragmented and immature that we will have to do some serious synthesis. But that's normal for breakthrough research.

- The good news is that now that I somewhat understand what Bayes Rules is, what its foundation is, and how it's been applied, I can see a path to solving our problem. The synthesis revolves around two foundation elements. The first is we must define the axioms for the Structured Argument Analysis system, since that foundation defines the "truth" as this system sees it. The second foundation element is rule and factual types. I can see 5 rule types clearly. There will be more. I can see at least 2 factual types.

- Thus our next step is for Jack and Montserrat to have a series of synthesis meetings. This is a wonderful and rare step in research. Before this step, you don't know if a problem is solvable and you have no clear vision for a solution. After this step you know it's solvable and you have the foundational elements that define the solution vision. Subsequent work is filling in the details, executing the plan, evolving the foundation, and so on.

- This bring to mind a quote from Thomas Edison: "None of my inventions came by accident. I see a worthwhile need to be met and I make trial after trial until it comes. What it boils down to is one per cent inspiration and ninety-nine percent perspiration." Synthesis is the 1%. I am deliberately postponing doing this step myself so Montserrat and I can do it together. Note that there could be more foundational elements, or the two described above could be wrong.
Team Member Items

B. Montserrat
  1. News from Slack interactions

C. Scott
  1. General observations

D. Jack
- Learning about Bayesian logic has been exhausting. Most articles, papers, and books have vast sections that, for our problem, are irrelevant, confusing, immature, poorly written, etc. So much text is impossible, at least for me, to clearly understand. This is to be expected in a field in the Model Revolution step, but still, it make one weary.

- This work has also been discouraging. It appears that we are attempting to solve a VERY big problem that others have not been able to solve. Research solutions, except for the AI end of the spectrum, are weak and are not being adopted. Realistically, we cannot expect to solve a problem that dozens of people/teams far more qualified than we are have failed to solve.

- But fortunately, my readings and notes in the last two days have been encouraging. As I explained above in my report, it appears there is practical path to solution. It involves not solving ALL of the problem, just enough for our tool to handle most public discourse claims. The two foundational element of the synthesis appear to be enough to get started successfully. This is similar to early work on SIP and much work in the history of science.
 
Last edited:

Jack Harich

Administrator
Staff member
#4
Scott and Montserrat,

Another great meeting. Regarding the document on Structure Argument Analysis synthesis, see Plausible Reasoning with Bayes Rule. Page 4 has Jaynes' list of axioms. Very, very insightful. I've not gone much further in this document. So far I'm not focusing on the math, but the bigger picture. However, page 3 describes decision making chains, something we may be able to use.

This is the first chapter of Jaynes book. Page 112 has his first axiom. Page 113 had the second axiom. Page 114 has 3, 4, and 5. I've not read this chapter, but skimmed it.

Enjoy!
 

Jack Harich

Administrator
Staff member
#6
Sorry, but not much. We don't yet have much to analyze for claim types. Our emphasis is more on the hardest part of solving the problem, which is how to implement Bayes Rule.

I just made this post. You can see our latest work on what we discussed yesterday there.
 
#7
This is the agenda for the September 23 meeting. Today we'll be trying a new format: first, we will test having one section per person, so that we can report on the different tasks we're working on together, and after that, we will move to a section of group topics, and further discussion about the covered topics as necessary.

A. Jack's responsibility areas:

1. Report on Solving the problem of how to measure the truth - The problem was identified and a thread begun on September 1. The opening post structured the problem by decomposing it into 6 subproblems. The first was how to set the confidence level of facts, which Montserrat is working on.

Subproblems 2 and 4 can be combined into how to set the confidence level of rules and select the correct rule. Part of this is easily solved. The rest becomes the problem of how to implement Bayes Rule. Jack is working on this problem.

When these two problems are solved we expect the entire problem will be solved, not perfectly, but good enough to proceed.


2. Report on How to implement Bayes Rule - This is really hard. The problem has been structured into rule types as described in the September 22 version of the document on How to Measure the Truth. On page 15 lists the 4 rule types:
1. Deductive fallacy
2. Deductive non-fallacy
3. Inductive with independent inputs
4. Inductive with dependent inputs

The main work for the week was figuring out the math for the third type, inductive with independent inputs. This is complete. It was done with decision chaining using Bayes Rule.

The fourth rule type, inductive with dependent inputs, is unsolved. It will require some form of Bayes Rule. I thought I'd find the needed equations in research on using Bayes Rule in legal arguments, but in this paper the equations are given for only one input, which is useless. One input is neither independent or dependent. It just "is". For one input we can use the inductive with independent inputs rule.

The above paper led to the book Risk Assessment and Decision Analysis with Bayesian Networks, by Fenton and Neil, 2013. This describes an example of two dependent inputs to a rule. But it asks the user to fill in a "node probability table" for the various conditional probabilities. In the table is the answer, ie the value we want to calculate. Thus, this example (and possible the entire book) is useless.

Behind the example is a tool called AgenaRisk. The book promises to explain all the math behind the tool, which is where we would find the equations we need. But as far as I can tell, the book doesn't actually provide the necessary equations. Why? I thwink it's because the authors are trying to use the book to sell the tool, which costs $2,600 per user.

My next step is to begin studying discussion forums for how to derive the equations we need. For example, here is a discussion on Mathematics Stack Exchange on Bayes' Theorem with multiple random variables. This was found using a google search on "bayes rule multiple".


4. Martha says "I can tell it's a hard problem. You can tell it's a hard problem by the number of books you get to solve it." :)


B. Montserrat's weekly report:

1. Report on how to set the confidence level of facts.
- I'm just getting started on this, and I have been investigating how other organizations already work on tackling this problem.

2. Report on How to implement Bayes Rule.
- Even tough Jack has taken the lead on this matter, I am making an effort to follow through on all the details, and be able to reach the point of also understanding Bayes Theorem very intuitively. This will prove to be necessary sooner or later.
- Last week stumbled upon what I think is quite significant, but is not very clear in the literature, namely, that a statistical application of the Bayes Rule won't solve our problem. To do that, I've been working through Jack's document on Solving the problem of how to measure the truth, and all of its updates. As a result I have my own version of that document. Last week I sent the first part to Jack for review, that version still didn't go into the application of Bayes Rule, but my newest version does. I want to review it with Jack again, to make sure I'm not having any thinking mistakes. In this mathematical exercises, it's easy to assume you understand something by just reading about it, but the reality is, you never truly understand it, until you've derived it for yourself.

3. Report on collaboration.
- My goal for next week is to contact organizations that have some experience on ranking the quality of sources to advance the goal of solving how to set the confidence level of facts


C. Scott

1. General observations

D. Group Topics

1. We had a terrific six hour long intellectual conversation at the Tower yesterday, with me, William Kurkian, and Montserrat. Lots of heavy lifting, high quality thwinking, pizza, brownies, and merriment! Scott, I wish you had been there!!!

2. Montse's update on a possible short trip to South Korea

3. Anything else here?
 
Last edited:

Jack Harich

Administrator
Staff member
#8
Special update - This meeting has been canceled. Scott Booher cannot make it, as he is in Seattle taking care of his mom after an operation. That leaves only Jack and William, who is just getting up to speed on our current topics. So we will move the special topic, how to calculation the confidence level of facts, to next week. Thanks!
Oh what a busy week! Montserrat flew off to South Korea Wednesday night and is returning on Monday. She will be flying Sunday evening and so will miss the meeting. But to offset that loss, William Kurkian plans on attending, plus Scott Booher and Jack Harich as usual.

Here's the agenda for the September 30 meeting at 8:00PM EST. Again, this is organized by one section per person, so each of us can report on our individual responsibilities or areas of interest, as well as pose penetrating questions and observations about anything. The group topic section is last. This week it contains the topic we will spend the most time on.

A. Jack
  1. Report on How to implement Bayes Rule - This problem is my central focus until solved. It has proven to be extremely difficult. No progress this week other than understanding more about the Bayes Rule literature and our particular application of Bayes Rule. I did start a thread on the math section of Stack Exchange on Friday. Just formulating the question was helpful. The replies are in earnest and welcome, but they haven't helped yet. Meanwhile, I'm going over and over the fundamentals of the problem.
  2. Report on Michael Hoefer, a potential new thwinker - I forwarded an email conversation on this. This is a fine example of a large investment in the quality of the material on Thwink.org. I suspect that due to the global rise of totalitarianism and the destructive actions of Trump/Republicans in the US, thoughtful people have grown more aware of society's large unsolved social problems. No credible solutions are in sight. So they look around for alternative approaches to solution and what do some of them find? Thwink. Also, Michael is a great fellow. We three really hit it off.

B. Montserrat
  1. (Not in attendance) Montserrat's main current task is How to set the confidence level of facts. In order to help her along, this task will be covered in the group topic.

C. Scott
  1. General questions and observations.

D. William
  1. Review of his work and how he may be able to play a sort of board-of-directors advisory role, like Scott. What areas might he have relevant expertise and interest in?
  2. General questions and observations.

E. Group topics - Discussion of A system for setting probabilities. How can we solve this problem?
  1. For description of this problem see the latest version of the How to measure the truth document. Please read the section on page 9, Element 4. A system for setting probabilities before the meeting and consider how we can approach solving it. Checkout the NewsCracker link and read that page closely, since it's the closest application to our problem we have found.
  2. The main challenge here is How to set the truth confidence level of facts. Unless we do this well, we will have a garbage-in, garbage-out problem. Structured Argument Analysis relies on good fact truth levels.
  3. The problem is already well-structured into information types. Find your probability type and then set it.
  4. Is this structure appropriate? Do the five steps (or something like them) offer a standard process for setting probabilities that reflects how people actually think?
  5. What other ways could we structure the problem that might help?
  6. Have you heard of any other approaches like NewsCracker that we might learn from?
  7. NewsCracker rated 56 news sources for their level of truth. The details on how they did this are not published. Nor are the results. Montserrat will be contacting them on this and the possibility of working together, such as on centralization.
  8. We will need to do the same or work with them or others with a centralized database of rated news sources. Any ideas on this centralization?
  9. Any ideas for how to actually truth rate news sources that go beyond what NewsCracker does?
  10. Anything else?
 
Last edited: