August 2018 Core Group Meetings

[Jack Harich]

Things are getting downright exciting! We are closing in on having the Truth Ratings System tool ready for running well-designed claim-check experiments. If our central hypothesis, that it's possible to accurately measure the truth of claims is not proven false, then we have taken the first giant step on the royal road to project success!

Scott can't make the August 5 meeting. Montserrat and Jack will meet at her attic apartment at 6:00 PM. Our agenda is:

A. Project reports

1. Weekly report on Montserrat's Research Plan for the Rest of 2018. This is the second such report. The report is structured and includes these areas. The bulk of the meeting will be on this all-important agenda item.
   1. Risk status.
   2. Schedule status.
   3. Help needed (including working together on specific tasks) in the upcoming week.
   4. Work focus and central challenges for the upcoming week.
   5. Discussion as needed. All this structure is designed to maximize the probability of project success!
2. Report on Jack's software development project. Going better than expected lately, in terms of quality and speed of work. It's so nice to be well past the very difficult work I had to do to learn database programming and implement a framework suitable for our application. The major work accomplished in the last week was combining rules and factuals into one table, getting automatic updates to reusable claims to occur when rules, facts, and reusable claims they used were changed, and adding a hierarchical folder tree for rule classification. Next step is to allow online editing of protocols and help doc. Then I may be ready to change gears and start doing claim-checks.

Next come team member items:

B. Montserrat

1. Any topics here?

C. Jack

1. Really looking forward to getting past full time on software development and on to using the tool to actually do claim-checks. There will be lots of surprises, insights, and discoveries here. We need to get to this stage ASAP, so we have a somewhat smoothly running system for the experiments.
2. Will get to adding Montserrat to the Thwinker's page soon. Thanks for the bio!
3. Will get to reading Montserrat's forum posts soon.
4. Work on trouble calling Montserrat on my phone. Can text her but not call her.
5. When should we take the fact-check training with the Poynter Institute?

 

[Jack Harich]

We're now well into the research project a LOTS is happening. It's almost TOO exciting !

My apologies, but all of a sudden Martha and Jack were invited to dinner with family we have not seen in years. We thus need to reschedule the meeting. However, here is the tentative agenda:

A. Project reports

1. Weekly report on Montserrat's Research Plan for the Rest of 2018. This is the third such report, covering these areas:
   1. Risk status.
   2. Schedule status.
   3. Help needed (including working together on specific tasks) in the upcoming week.
   4. Work focus and central challenges for the upcoming week.
   5. Discussion as needed. All this structure is designed to maximize the probability of project success!
2. Report on Jack's getting the Truth Rating System (TRS) ready for the first claim-check experiments. I completed several software features, including the basic rules tree and search factuals. Now it's on to the goal of making TRS ready for users. Two requirements are: (1) Make it easy for users to find factuals (fact and reusable claims) and (2) Make it easy for users to find the correct rules. Search factuals should achieve the first requirement. But finding the rule you need easily is not yet possible because the rules need to be well-organized, well-described, and entered. To make rapid high-quality progress I will be studying the Hurley book on logic, looking over the material Scott sent me and other material, improving the rules tree folder descriptions quite a bit, and entering enough rules so that the pattern for now to enter them and where is clear. Then I can shift into entering many rules, hopefully enough for claim-check experiments and soon, general use.

Next come team member items:

B. Montserrat

1. Any topics here?

C. Scott

1. Any topics here?

D. Jack

1. Montserrat and Jack created Thwink pages for Montserrat. Now we just need to complete Montse's bio page, the comments.
2. In case this is not covered above, we have a big risk in the claim-check accuracy experiment design. It appears that by having facts already added for the fact-check we are claim-checking, we're making it too easy and will not produce realistic variation results. We need to have facts NOT already entered. But what happens when one person enters a fact they need, and five minutes later another person goes looking for a fact and finds the one just entered. What if it's low quality, ie the wrong confidence level or so brieflly described it doesn't appear in search results, and so on? Then we have an experiment contamination problem. One subject affects the work of another. New reusables should go through review. Perhaps if we have review implemented, this will fix the problem. But what if the review takes so long that another duplicate fact is added for review before the first review is done? Plus the software does not yet support review. Wow, this is tricky....
3. I've started wrestling with creating claim-checks. My first one is completing the one I created earlier, The False Dilemma that Took a Country to a False War. Editing it to use high-quality rules is surprisingly difficult. Unless it's a well-known fallacy, so far the hard part is creating the perfect rule for the situation. What we're doing here is way beyond the simple examples found in textbooks and other material. This reveals how much pioneering work we have ahead of us. It's going to take some time to bust through into the new way of thwinking required to create arguments using the tool. I hope this risk doesn't impact our schedule.

 

[Jack Harich]

Another great meeting! Scott and Montserrat, the logic book I have on order is Inductive Reasoning: Experimental, Developmental, and Computational Approaches, by Aidan Feeney.

After the meeting Jack and Montse talked about a number of things. One was Jack's agenda item 3. We talked about a new idea that could be the solution to this risk is that in the future, 5 to 10 years from now, nearly all facts will already be in the database. New ones will be rapidly entered by news organizations and organizations like Wikipedia and government agencies. Automation will be doing quite a bit of the work. Thus we can resolve the risk by NOT letting experiment participants enter new facts. That's a clean solution, hopefully.

 

[Jack Harich]

We're starting to encounter a few small bumps in the road on the research project, but this is normal. Overall, things are going very well.

Montserrat will be out of town on Sunday, so we've rescheduled the meeting to Saturday, August 18, 8:00PM as usual. Here's the agenda:

A. Project reports

1. Weekly report on Montserrat's Research Plan for the Rest of 2018. This is the fourth such report, covering these areas:
   1. Risk status.
   2. Schedule status. (There has been a one week slide. What will each of us do with that extra week?)
   3. Help needed in the upcoming week.
   4. Work focus and central challenges for the upcoming week.
   5. Discussion as needed. All this structure is designed to maximize the probability of project success!
2. Report on Jack's getting the Truth Rating System (TRS) ready for the first claim-check experiments. I am behind schedule, so in a meeting with Montserrat we slid the beginning of experiment one a week. The cause is the tool is not ready to smoothly do claim-checks. The barrier was the Rules page was not ready and I was nowhere near the important goal of having done 10 to 20 practice claim-checks and reached the point where claim-checks were fairly easy to do. They just take thoughtful time. There was plenty of software still to develop and bugs to fix. The biggest task for the week was reorganizing the rule tree into a truth spectrum. This makes it MUCH easier to quickly find the right rule. On Friday I finally started my first claim-check of a fact-check with the current online multi-user software, using a very short fact-check from WikiTribune. Despite its brevity, it took about 6 hours to think through how to map it and conceive of the new rules required. The claim-check is now done, though to avoid cluttering up the database I've not entered the upper four facts. After doing this there were still bugs. They had a common root cause, so I implemented a new version of caching database records on the client. So far so good on stomping out those bugs.

Next come team member items:

B. Montserrat

1. Any topics here?

C. Scott

1. Any topics here?

D. Jack

1. This may come up in Montserrat's report. From what I've seen so far of fact-check articles and the three claim-checks I've now done, I thwink we will have to confine experiment one, on measuring claim-check accuracy and bias, to a narrow subset of all possible claims. I remain astounded at the explosion of complexity that happens when trying to analyze all but the simplest claim. Over time we must be able to handle any claim. It's normal when pioneering a new type of measuring tool for it to not work too well at first. But it's so much better than no way to measure something at all that people don't mind. They are glad to see progress and use the tool. We have already talked about this a little.
2. I'm very excited about the upcoming experiment and paper. Things are looking very solid on the way Montserrat is taking the time to design high quality hypotheses and methods for measuring what Structured Argument Analysis does. Some cool stuff here !!!

 

[Montserrat Koloffon]

Time flies when you're having fun, and one week more has passed!
Jack has been extra busy this past week, still working on details of the software, but also doing claim-checks, and being there for Martha, who got sick, so I'm for the first time gladly putting the agenda together for this week's meeting!

Raise Truth Literacy Project

A. Project reports

1. Weekly report on Montserrat's Research Plan. This is the 5th version of the reports, and all the past versions are now available on a new thread on the forum!! The report includes the usual sections, and this week's report will be uploaded to the new thread I just mentioned after the meeting.
2. Report on Jack's role in the research. The software side of the Truth Ratings System (TRS) is done for now. Small improvements and bug fixes will be made as needed. Some features were added.
   - The lists of rules and factuals now have a UC (use count) column. This shows the number of times the reusable has been used.
   - On the rules page, there is a new toggle button. "Showing Definition" means that when you click on a rule its definition is shown. "Showing Uses" means clicking on a rule shows its uses.
   
   - Double clicking on a rule uses rows goes to the reususable claim where the rule is used. This is very handy for learning how to use rules.
   The database now has 6 completed claim-checks. 4 were done with the new software. Producing claim-checks is getting smoother, but it's not yet completely smooth, ie you can't work at the speed of thought and how fast you can move the mouse and type. That's primarily because we've not yet created the rules needed, with good examples and a description of each one. Jack considers the work goal hasn't been met yet, and we have schedule slippage again. We need to discuss how to handle this.

Team Member Items

B. Montserrat

1. I think I've found the Argument Mapping Project from Georgia Tech that Scott told us about. It's here. Michael Hoffmann is the Director, and he together with other people involved in that project may be interested in what we're doing. I see a lot of potential for interesting collaborations and mutual learning!
2. I (finally!) published my observations on Jack's questionnaires. It was an extremely interesting exercise, and I hope my notes are helpful at some point. Thank you very much for sharing it Jack!
3. I have some minor changes I'd like to do to my bio on the website. I'll send Jack my (minor) revisions.
4. I haven't been working on the R course at all, because I've been studying for the scholarship exam. It takes place in 2 weeks, and I'm going to be away for some days (still don't know exactly how many). We need to consider that for the schedule. Those days are going to be good for tasks that I can work on by myself, but not on collaborative tasks.
5. I also started preparing another application for yet another scholarship. This is taking some time out of my day, but it's necessary for ensuring the perfect place where I can continue working on the UN research project after I leave Atlanta.

C. Scott

1. Any topics here?

D. Jack

1. We need to followup on working with Scott Collison and Jess Bernhart. What a good start! Let's talk about how to best do this. Potential areas of collaboration are rules, overall flaws, finding people to test the tool, finding people who would like to be study participants, and study/paper design on difficult areas, like what is the "truth"?
2. The critical path for Study 1 seems to be getting creating claim-checks to go smoothly enough, so that when we begin the study we are focusing on the study, rather than the hard awkward aspects of creating claim-checks. I thwink we need to work together very closely for a few days and get past this barrier. It will be challenging, but I thwink we can do it. Also, but thwinking together through the tough discoveries and decisions, we can arrive at better results.

 

[Jack Harich]

Very productive meeting.

Scott, you reported that when not logged on you could not see the Factuals tab. I logged off and I can see it.

I tested a url like https://www.truthratings.org/#factuals/id=53 when logged off. Sure enough, found a bug. Fixed it.

A potential problem is people clicking on a link to a reusable and not using Chrome. That's why I bolded the info about Chrome in my reply to Montserrat in the letter to Jimmy Wales.