The First Experiment:
A Brief Introduction to the Truth Test
Overview
This experiment consists of a 5 page handout testing
a single factor: whether or not a brief one page introduction
to the Truth Test makes a difference. Subjects read actual
political statements and then respond with multiple choice
questions about how they feel about the politicians who
made the statements.
The experiment was tested, but not
really run, on April 30, 2006. While the sample size was
only 4 people and 2 of them were biased, the results strongly
supported the hypotheses. There was also some very lively
discussion after the actual experiment was run. Later runs have shown various results.
The Hypothesis
Even a very brief exposure to the Truth Test
can raise a person's ability to detect political deception.
How to Run the Experiment
The independent variable is whether or not
a subject is exposed to the Truth Test. This is accomplished
by using different second pages in the handouts. The
experimental group receives a page introducing them to
the Truth Test and how to use it to spot fallacious arguments.
The control group receives a page about a neutral subject
related to political statements. The reason for the "dummy
page" is so that both groups take about the same
time to complete the questionnaire.
The dependent variable is the effect on how a subject responds
to various political statements. This is measured by multiple choice
questions that follow each statement.
Special Note on Group Size - You do not need to run the experiment only on a group. Individuals or ultra small groups of 2 or 3 will work just as well. For them to produce valid results, the subjects need to be picked somewhat randomly, and the handouts absolutely need to be given out randomly. After you've accumulated over 30 subjects (ideally) worth of data, write up your results just as if you used a single group.
The experiment is run by:
Step 1. Print out half of the handout without the
Truth Test and half with it. Staple each set together,
so that there are 5 pages in each set. The all important
page 2 is not visible. Shuffle the sets of pages to be
handed out ahead of time. Hand them out in any manner
desired. If the experimenter is not the one who shuffled
them, or shuffled them in a manner such that they lost
track of which was which, then we have a double blind
experiment and randomly selected different groups. This
is important. It
is crucial that the group be unaware of the two types
of handouts and the existence of the Truth Test, so do not discuss
this beforehand. Just tell everyone you will be running
an interesting social experiment concerning a mock election.
Step 2. The group silently reads the questionnaire
and answers the questions. This takes about 30 minutes or less.
Step 3. When all are done, everyone hands in their
answer pages and keeps the rest of the pages. The experimenter
then hands out the additional material pages, which reveal
who said what and discuss what the experiment is trying
to do. While the group is reading this, the experimenter
enters the answer sheet data into a spreadsheet, which
then calculates the results.
Step 4. After everyone has read the material, the
experimenter leads a discussion. This can be very insightful and educational.
Emphasis should be placed on how it was a true double blind, randomly
assigned groups experiment. This is the gold standard of experimentation.
But the results only hold for the total population if the full group
was randomly selected, which is usually not the case in small social
group settings. The hypothesis is that those subjects with the packets
educating them about how to detect fallacious arguments will be less
easily deceived.
After people understand the experimental design the results of the
experiment should be presented. How they support or do not support the
hypothesis should be discussed. If the results support the hypothesis,
which is usually the case, then how this new scientific knowledge can
be used to improve the way the human system is engineered is a lively
topic, and a possible good note to end the formal discussion on.
You may also want to discuss the dueling loops and the high leverage
point the experiment is testing. If so, please study The
Dueling Loops of the Political Powerplace.
At some point you may want to discuss what Logical
Fallacies are. A well run experiment should itself result in a
small bit of training the population to be more immune to the immense
power of political deception.
Step 5. Report the results to Thwink.org. Include
a writeup on the 5 to 10 most intriguing things the participants
wrote on their questionnaires or said in the discussion.
This helps us generate insights.
Files Needed to Run the Experiment
1.
Handout with Truth Test
PDF -
Print 5 of these for every 10 people.
2.
Handout without Truth Test
PDF -
Print 5 of these for every 10 people.
3.
Additional Material
PDF -
Print 10 of these for every 10 people.
4. Experiment
Data and Results XLS -
Use this Excel file to enter your data. The data on
the WOTT and WTT pages is empty. After you enter your
data, the results will be calculated automatically
on the Results page. EMail a copy of the completed
file to Thwink when done, along with your comments on how
the experiment went. These spreadsheets require Microsoft
Excel 2003 or later.
5. Hollcraft
Data and Results XLS -
Before you run the experiment, study and play with
this file. It contains the actual data from Michael
Hollcraft's running of the experiment on February 16,
2007. You may also want to show this file to your group(s).
Here are the four pages in this and the above Excel
file:
Intro - This explains how to enter the data and see the results.
WOTT - This page is for the WITHOUT
Truth Test data.
WTT - This page is for the WITH
Truth Test data.
Results - This displays the
results. Please
call us up to discuss
how to interpret this page and use the scenario cell,
unless you are already familiar with these things.
We want to help you run great experiments.
Experimental Results to Date
March 29, 2007 - 19 subjects, results.
Run by Philip Bangerter. Contrary to
expectations, exposure to the Truth Test seemed to increase susceptibility
to political deception. See this forum
thread.
February 17, 2007 - 17 subjects, results.
Run by Michael Hollcraft. The hypothesis was supported,
with a 44% decrease in susceptibility due to
political deception. However, due to small group size,
this is not valid at the 95% level of confidence. See this
forum thread.
April 30, 2006 - 4 subjects. (This
was not a bona fide run, but just a shake down test run
by Jack.) Summary - Using a point system where strongly
disagree was a 1 and strongly agree was a 5, those with
no Truth Test averaged 2.5 on statement agreement, while
those with the Truth Test averaged 1.6. Because all four
statements contained fallacious reasoning, brief exposure
to the Truth test appeared to decrease susceptibility
to political deception by 35%.
Using a point system of 1 point for choosing to vote for a politician
and 0 points for choosing neither, the group with no Truth Test averaged
.75 while the other group averaged .5. Because all four politicians
based their appeals on fallacious reasoning, brief exposure to the Truth
test appeared to decrease susceptibility to political deception by 33%
when it comes to voting decisions. This is enough to change election
outcomes, if it is a lasting change.
However, the sample size is to small to be useful. Furthermore, 2 of
the people in the group, Jack and Martha, knew all about the experiment.
Thus even though the results look promising, we should not yet assume
we really have found a way to push on this high leverage point.
The real purpose of the April 30 experiment was preliminary testing
of the experiment itself. No problems were found.
We hope to run the experiment many more times. The results will be
published here.
