(10 pts.)
Suppose an experimenter randomly assigns six participants to treatment
A and six partipants to treatment B. She runs the experiment in four
rooms that have been carefully constructed to be as indentical as
possible, which hold just three people each. She believes that the
rooms have essentially no effect whatsoever on the score that is
measured, but just for administrative purposes she keeps track of what
room each participant was in. Here are the results of the experiment:
Treatment Room Score
A 1 1.5
A 1 3.0
A 1 4.5
A 2 1.8
A 2 3.0
A 2 4.2
B 3 3.8
B 3 5.0
B 3 6.2
B 4 3.4
B 4 5.0
B 4 6.6
She hands off the data to a research assistant, saying "run an ANOVA
and let me know if there is a significant difference between groups."
(A) The research assistant doesn't know the experiment design and
just sees the coding in the data file. So he thinks, Well, looks like
there were four groups of subjects, so I'd better run an ANOVA with
three planned contrasts:
1 -1 0 0 (room 1 vs room 2, both treatment A)
0 0 1 -1 (room 3 vs room 4, both treatment B)
1 1 -1 -1 (treatment A versus treatment B)
Conduct those contrasts in SPSS. Include the output tables. Using
the Bonferroni correction [show your work!] for multiple planned
comparisons, what does the research assistant conclude about the
contrast of treatment A versus treatment B?
(B) The research assistant reports back to the experimenter,
who says, "Oh, I never planned to compare the rooms with each other. I
included the room codes merely to keep track of data files, not
because they were part of the design." So the research assistant
jettisons consideration of his first two contrasts, and only considers
the one last contrast of treatment A versus treatment B. What does he
conclude? Explain what this does to the Bonferroni correction.
(C)
The research assistant reports back to the experimenter, who says,
"Oh, you shouldn't do an ANOVA contrast on the four rooms like that,
instead, just do a two-group ANOVA, coding the two treatments as 1 and
2." Conduct the analysis in SPSS. Include the output table. What does
the research assistant conclude?
(D) A few days later the experimenter comes back to the
research assistant, saying: "Well, it's a good thing I kept track of
the rooms each subject was in. I was looking at the data and noticed
small differences between room scores, and that got me thinking about
the rooms. So I looked at the rooms again and noticed that they felt
different. I checked with building maintenance, and, sure enough,
there's a problem with the air conditioning in those rooms. The rooms
might have had different temperatures and air flow. So, we need to
consider each room as a different group, just like you did in the
first place." So, the researcher digs out his original set of
contrasts. He realizes, though, that the contrast is now post-hoc, so
he uses the Scheffe method to set the critical value. Show your
work. What does he conclude?
The moral to the story: To me, this illustrates
the logical quagmire that null hypothesis significance testing leads
to. For each contrast, the meaning of the p value is clear: It is
the probability of getting that F, or greater, when sampled from the
null hypothesis population. But interpreting that p as
"significant" (and in turn establishing confidence intervals) is a
matter of where you set your cutoff. Different considerations
prescribe different settings of the cutoff, and hence the messiness
above. All this nonsense is avoided with Bayesian methods!