• Some new notation
• Computing the f-ratio
• computing degrees of freedom
• Using the ANOVA table

t-tests and z-scores are great but they are limited, they can't be used for more than 2 groups. Instead what we need to do is use a new inferential statistical procedure: Analysis of Variance (ANOVA).

Why do we have to use variance?

Because we have more than two groups.



How would we compute a single score that would describe the difference between these distributions? Difference just doesn't cut it, but variance does (recall it is a measure of how these much these three distributions differ from one another).

Now let's consider the sources of variance (that is, what causes the variability that we observe in our dependent variable).

What kinds of things might cause the samples to be different from one another (between treatments variability)?
• Treatment/group effects - the treatment caused the differences
• Individual difference effects - differences due to differences in individuals
• Random (experimental) error - just what it sounds like, random error

What about the variability within each sample (within treatment variability)?
• Individual difference effects
• Random (experimental) error
• BUT NOT treatment/group effects - this is the key

This is how ANOVA gets its name. Analysis of Variance.

So when is an ANOVA the appropriate analysis? Check the decision tree.

Find the string of decisions that lead to a 1-way between groups Analysis of Variance.

Hypothesis testing with ANOVA

We'll be using the same hypothesis testing logic that we used in previous chapters, but the details will change (as they did from chapter to chapter).

step 1: state your H₀ (and H₁ ??) & figure out your critera: a = ?
the ANOVA test is always one-tailed
step 2: figure out the df for your test (there will be two dfs)
step 3: compute your f-statistic for your samples
step 4: compare your f-statistic with the critical f-statistic found in the ANOVA table
step 5: make your conclusions about the H₀

Okay, let's start by considering a new example that is a single-factor, independent-measures research design (more complex designs will flow in the future weeks).

Example research project

Suppose that for your senior research project you decide to test the effectiveness of three different studying methods on learning. Method A, is to have students only read the textbook, but not go to class. Students assigned to Method B, go to class and take notes, but don't read the textbook. Students the the Method C group don't read the textbook or go to class, they just get to look at another student's class notes

Step 1: setting your alpha level is the same as we've worried about in the past. Because we're dealing with variances the F-distribution really only has a positive tail that we worry about, so consider all tests to be 1-tailed.

The big difference with ANOVA is that we've got more than one possible alternative hypothesis. However, the basic ANOVA is still only testing the null hypothesis. Selecting between different alternative hypotheses require further tests (we'll discuss these a couple of lectures from now).

Null hypothesis: still is basically a statement that "there are no differences between the groups"
m₁ = m₂ = m₃

The basic alternative hypothesis is that "not all groups are the same." However there are a number of ways that this could be statisfied.

H_a: m₁ not equal to m₂ not equal to m₃
H_a: m₁ = m₂ not equal to m₃
H_a: m₁ not equal to m₂ = m₃
H_a: m₁ = m₃ not equal to m₂

Step 2: Compute our degrees of freedom

There are a number of different degrees of freedom that we need to compute, one for the between group variance and one for the within groups variance (and a third for total df).

df_total = total number of scores - 1

df_within = total number of scores - number of groups

df_between = number of groups - 1

These degrees of freedom are all related to one another by the following formula:
df_total = df_within + df_between

Step 3: The F-ratio is conceptually similar to the tests that we've already examined, but the number of computations that we have to do increase (note: they don't get harder, just more numerous).

Okay, here is where things will start to get complex. For now, we'll stick with talking about things at the conceptual level (no computations yet). In many ways the ANOVA is very similar to a two independent smaples t-test (in fact we'll later we'll talk about how they are nearly identical under certain circumstances).

t_obs=

Remember the pop mean part = 0. Also recall that the standard error is the standard deviation of the distribution of sample means. A standard deviation is the average difference from the standard (in this case the mean of the sample means). So the standard error is basically the average difference that we'd expect by chance. So the formula basically can be restated conceptually as:

                tobs  = obtained difference between sample means  
                                difference expected by chance

For ANOVA the test statistic (called the F-ratio) is similar except that we use variance rather than just difference.

                F =  variance (differences) between sample means
                     variance (differences) expected by chance (error)
                        

Think back to the sources of variance.

Between treatments variability is made up of three sources:
• Treatment/group effects - the treatment caused the differences
• Individual difference effects - differences due to differences in individuals
• Random (experimental) error - just what it sounds like, random error

Within treatment variability is made up of two sources:
• Individual difference effects
• Random (experimental) error
• BUT NOT treatment/group effects - this is the key

  So the F-ratio can be expressed as:

          F-ratio =    variance (differences) between sample means
                          variance (differences) expected by chance (error)
  
  
          F-ratio = variance between treatments
                          variance within treatments
  
  
          F-ratio = treatment effect + individual differences + random error
                          individual differences + random error
  
  

So the short story is that ANOVA is conceptually very similar to the t-tests and z-tests. However, as you might expect, the details and the computations (the long story) do get a bit more complex.

Some new notation

The experimental situations that we're dealing with are more complex than those we've dealt with so far. We aren't going to need to use any new kinds of math (still just adding, subtracting, multiplying, and dividing), but we do need some new notation. The computations that we'll go through are pretty much the same things that we've done in the past (e.g. computing sums of squares, means, etc.), but there are more of them.

K = # of treatment conditions (or groups), each of which is called a level of the factor. ni = # of observations in the ith group (if they are not equal) N = Sni = total sample size Ti = SXij (where Xij is the jth observation in the ith group) G = SXij = the sum of all the X's = grand sum = G / N = grand mean SSi = the sum of squares for each group = S(Xij - i)2

Basically the new notation is takes into account the need to know means and standard deviations for each group alone and for all of the data together (collapsed across the different groups).

Example:

Suppose that for your senior research project you decide to test the effectiveness of three different studying methods on learning. Method A, is to have students only read the textbook, but not go to class. Students assigned to Method B, go to class and take notes, but don't read the textbook. Students the the Method C group don't read the textbook or go to class, they just get to look at another student's class notes

Study method
Method A book alone	Method B taking notes	Method C borrowing notes
0	4	1
1	3	2
3	6	2
1	3	0
0	4	0
T1 = 5	T2 = 20	T3 = 5
SS1 = 6	SS2 = 6	SS3 = 4
n1 = 5	n2 = 5	n3 = 5
1 = 1	2 = 4	3 = 1

SX² = 106
G = 30 = grand sum
N = 15 = total sample size
= 30/15 = 2 = grand mean
K = 3 = # of treatment conditions (or groups), each of which is called a level of the factor

recall that what we're after is:

                F-ratio =       variance between treatments
                                 variance within treatments

in the past we've used s² = SS/df. That's essentially what we're going to be doing here, but things will look a bit more complex.

SS_total = S(X - )²
SS = (0-2)² + (1-2)² + ... + (0-2)² = 46

There is a shorter way to do the computation:

SS_total = SX² - (G²/N)

-- same as our old formula, but with Grand total instead of SX

SS_total = 106 - (30²/15) =106 - 60 = 46

But it isn't the SS_total that we really need. Remember that what we really need are the within and between groups variabilities.

SS_total = SS_within + SS_between
So how do we get the SS_within?

we add together all of the SS estimates for each group

SS_within = SSS inside each treatment = SSS_i

= 6 + 6 + 4 = 16

So how do we get the SS_between?
The quick way is:
SS_total - SS_within

But, there are two drawbacks of doing it this way:

• if you make a mistake in computing any of you SS_i's then you'll get the wrong answer.
• it doesn't give you the sense of what SS_between is made up of.
I'll give two formulas for the direct computation of SS_between, a definitional and a computational equation

definitional	computational
SSbetween = S[ni( - )2]	SSbetween = S(T2/ni) - G2/N
= 5(1 - 2) 2 + 5(4 - 2) 2 + 5(1 - 2)2 = 5 + 20 + 5 = 30	= 52/5 + 202/5 + 52/5 - 302/15 = 5 + 80 + 5 - 60 = 30

Let's check out math.

SS_total = SS_within + SS_between = 16 + 30 = 46 (and that's the number we got before)

Okay, let's return to what we're interested in: Variances.

s² = SS/df.

We've got our SS, now we need to figure out our df. There are going to be two (or three depending on how you look at it) degrees of freedom, one for the between group variance and one for the within groups variance (and a third for total df).

df_total = N - 1

df_within = N - K

df_between = K - 1

df_total = df_within + df_between

for our example:
df_within = 15 - 3 = 12

df_between = 3 - 1 = 2

df_total = 15 - 1 = 14, which is also = 12 + 2

Alright, now we are ready to calculate our variances. However, here is a quirk that we'll just have to live with, we don't call it variance but rather Mean Square.
variance = Mean Square = MS = SS/df

MS_between = SS_between/df_between

for our example = 30/2 = 15

Note: sometimes the MS_within is called the mean square error.

MS_within = MS_error = Mean Square Error = SS_within/df_within

--> for our example = 16/12 = 1.33

Almost done. Let's look back at what we're after:

F-ratio =  variance between treatments      =    MSbetween
           variance within treatments           MSwithin

So the F-ratio for our example is: 15/1.33 = 11.28

Often you'll see this presented in a table.

Source                  SS      df      MS              
Between treatments      30      2       15.0    F = 11.28
Within treatments       16      12      1.33                    
Total                   46      14

Using the ANOVA table (the F-table)

So what's the next step?

Look at the F-table and determine the F_crit and then decide what to conclude about our hypotheses
df_between = df in the numerator
df_within = df in the denominator (the error term)

So for our example:

df_within = 12

df_between = 2

	df in the numerator
df in denominator	1	2	3	4	5
1	161 4052	200 4999	216 5403	225 5625	230 5764
2	18.51 98.49	19.00 99.00	19.16 99.17	19.25 99.25	19.30 99.30
3	10.13 34.12	9.55 30.92	9.28 29.46	9.12 28.71	9.01 28.24
: :	: :	: :	: :	: :	: :
12	4.75 9.33	3.88 6.93	3.49 5.95	3.26 5.41	3.11 5.06
13	4.67 9.07	3.80 6.70	3.41 5.74	3.18 5.20	3.02 4.86
: :	: :	: :	: :	: :	: :

If we selected an a = .05, our F_crit = 3.88
If we selected an a = .01, our F_crit = 6.93

In either case, our F-ratio of 11.28 is greater than the F_crit., so we would reject the H₀ (m₁ = m₂ = m₃).

How would one report this?

Some ANOVA terms

In analysis of variance, a factor is an independent variable. A study that invloves only one independent variable is called a single-factor design. A study with more than one independent variable is called a factorial design. The individual treatment conditions that make up a factor are called levels of the factor.

APA style reporting

F(df_between,df_within) = F_obs, p-value

for example:

A one-way ANOVA yeilded a significant effect of study method, F(2,12) = 11.28, p < 0.01.

Another way that ANOVA reports are presented is as an ANOVA table:

Source	SS	df	MS
Between treatments	30	2	15.0	F = 11.28
Withing treatments	16	12	1.33
Total	46	14

---

Between subjects designs.
• Advantages:
  • The biggest advantage is that exposure to different levels of the independent variable(s) cannot "contaminate" the dependent variable ("transfer" or "carry over" effects)
  • Sometimes this is a 'must,' because you can't reverse the effects of prior exposure to other levels of the IV
• Counterbalancing is not required
• Matching can reduce variability between groups
  • Systematically matching participants in each group for all the important extraneous variables that you think are important
  • Trying to reduce the variability from of individual differences
• Random assignment of participants to groups eliminates bias
• Disadvantages
  • Potentially differences between groups
    • Random assignment to groups - for between subjects designs eliminates biases
  • More resources are needed (participants, time)
  • Less statistical power (the ability to detect an effect) because of individual difference variance
  • Matching takes time and effort and assumes no transfer from matching operation

  Note: in this day and age, computers actually have the family of F distributions, so your data output may actually give you your actual p-value. Rememeber that the logic of the test is such that you must specify your a level ahead of time. If you select 0.01, then that's the level that you are using for all of your tests. So if you do two experiments and your computer stats program tells you that in experiment 1, your p-value = .001, and in experiment 2 your p-value is .01 they are both equally statistically significant. The H₀ is a yes/no decision. In this example the answer to both is YES. The results in Experiment 1 are NOT "more significant" than Experiment 2.

  ---

Using SPSS to do single factor ANOVA

To set up a paired samples t-test you will need two columns of data, one for each sample (related samples) or one for each meansurement (repeated measures).

Note: To do One-way ANOVA you'll need to have two variables (columns) in your data file (this is just like with 2 independent samples t-test except now your independent variable will have more than two categories). One column will contain the data (your dependent measure). The other column will be an independent variable that specifies which group the subject belongs to (e.g., 1 for group 1, 2 for group 2).
Go to the Analyze menu and select the submenu Compare Means. In this submenu you'll see several tests. The one that we're interested in today is One-way ANOVA.
After selecting One-way ANOVA you'll get a window that looks like this. Here you should select the variables that you are testing. Your test variable is your dependent variable. Your group variable is the independent variable that assigns each subject to a group.
Here is what the output will look like.
Notice that the output is given in the standard ANOVA table output. SPSS doesn't tell you to reject or fail to reject the H0, nor does it give you the Fcrit. To make your decision about the H0 you must compare the p-value with your a-level. If the p-value is equal to or smaller than the your a-level, then you should reject the H0, otherwise you should fail to reject H0.

One factor independent samples ANOVA F-ratio = variance (differences) between sample means variance (differences) expected by chance (error) F-ratio = treatment effect + individual differences + random error individual differences + random error Relevant formulas K = # of treatment conditions (or groups), each of which is called a level of the factor. n = # of observations in each group (if they are equal) ni= # of observations in the ith group (if they are not equal) N =  = total sample size Ti = G =  = grand sum SSi =  = grand mean SStotal = SSwithin = SSSinside each treatment        = SStotal = SSwithin + SSbetween dftotal = dfwithin + dfbetween SSbetween =               = SSbetween =               = dftotal = N - 1 dfwithin = = N - K dfbetween = K - 1 MSbetween = MSwithin = MSerror = F-ratio = MSbetween               MSwithin Single variable – one Factor ·      Two levels (t-test) o      Basically you want to compare two groups o      The statistics are pretty easy, a t-test     Disadvantages: ·      “True” shape of the function is hard to see ·      interpolation and extrapolation are not a good idea ·      more complex theories typically need more complex designs (more than two levels of one IV)   ·      More than two-levels (ANOVA)   o      Gives a better picture of the relationship (function) o      Requires more complex statistical analysis (analysis of variance and pairwise-comparisions) o      Needs more resources (participants and/or stimuli)

Let's finish up with the situation that we started with in the fisrt two ANOVA lectures.

Suppose that for your senior research project you decide to test the effectiveness of three different studying methods on learning. Method A, is to have students only read the textbook, but not go to class. Students assigned to Method B, go to class and take notes, but don't read the textbook. Students the the Method C group don't read the textbook or go to class, they just get to look at another student's class notes

Here was the data for this study.

Study method
Method A book alone	Method B taking notes	Method C borrowing notes
0	4	1
1	3	2
3	6	2
1	3	0
0	4	0
n1 = 5	n2 = 5	n3 = 5
1 = 1	2 = 4	3 = 1

And here were the results.

Source                       SS      df      MS                      

Between treatments      30      2       15.0    F = 11.28
Within treatments    16      12      1.33                    
Total                   46      14

With a = 0.05, our F_crit(2,12) = 3.88. So we rejected the H₀.

Recall what the H₀ is:

m₁ = m₂ = m₃

If we were to write out our alternative hypotheses what would we write?

There are several possibilities:

m1 not equal to m2 not equal to m3	m1 not equal to m2 = m3
m1 = m2 not equal to m3	m1 = m3 not equal to m2

More generally, the alternative hypothesis is not all the groups are equal.

All that our ANOVA analysis has told us is that we reject H₀. However, we may want to know which of the groups are different (in other words, which alternative hypotheses can we reject). In our current example this is pretty easy to answer just by looking at the means.

1 = 1

2 = 4

3 = 1

Groups 1 and 3 clearly aren't different (1 - 1 = 0). Groups 1 and 2 differ by 3 (4 - 1) as do groups 2 and 3. So here the difference that the ANOVA yeilds as being statistically different must be 1&2 and 2&3. However it is not always this easy to tell just by looking at the means. Usually what we have to do are some additional statistical tests.

There are two sets of tests that are used to determine which groups are different: planned means comparisons and post hoc tests.

• Planned means comparisons are theoretically motivated comparisons that you decide that you'll do, before you perform your ANOVA. In other words, you have some specific predictions in advance that certain groups should be different than other groups. However, you are limited to testing for only these differences (and it should be a small set, typically no more than K - 1 comparisons).
• Post hoc tests are comparisons that you make after you've performed your ANOVA and found that you have rejected the H₀. The post hoc tests aren't theoretically motivated searches for group differences, so they typically test all of the possible pairwise differences (by pairwise I mean all the possible pairings, e.g. 1 vs 2, 1 vs 3, and 2 vs 3). There are a number of different kinds of post hoc tests, which we use depends on the experimental situation, your underlying assumptions, and how conservative you want to be.

The downside of making many comparisons (whether planned or post hoc) is that you increase the chance of making a Type I error. Recall that the a-level that we set is the probability of rejecting the H₀ when there really isn't a difference. This is true for each comparison. However, when we start testing a whole set of comparisions, then the overall the a-level increases. In other words, the more comparisions that we make, the bigger the chance of making a Type I error. The combined chance of making a Type I error is refered to as Experimentwise error.

The reason that we limit the number of planned comparisons (typically to no more than K - 1) is to keep our Experiment wise error levels low. Post hoc tests are designed to take experimentwise error into account, although depending on which test you use, the way in which it is taken into account varies.

Each of these comparisons is a separate hypothesis test, each one has a risk of making a Type I error. So, the more comparisions that you make, the higher the risk of concluding that there is a difference when there really isn't one. This is called experimentwise alpha level (or familywise error)

aEW = 1 - (1 - a)c c = # of comparisons

so for our example, if we chose a = 0.05 and make 3 comparisons

aEW = 1 - (1 - a)c = 1 - (.95)3 = 1 - .857 = .143

our chance of making a Type I error when making the comparisions is now 1 in 7 rather than 1 in 20

Most post hoc tests have been designed to control the experimentwise error.

Rather than doing planned comparisons or post hoc tests by hand, we'll focus on how to do them and interpret them using SPSS.

Using SPSS to do single factor ANOVA: Planned comparisons and Post hoc tests

Reminder: To do One-way ANOVA you'll need to have two variables (columns) in your data file (this is just like with 2 independent samples t-test except now your independent variable will have more than two categories). One column will contain the data (your dependent measure). The other column will be an independent variable that specifies which group the subject belongs to (e.g., 1 for group 1, 2 for group 2, 3 for group 3).

Go to the Analyze menu and select the submenu Compare Means. In this submenu you'll see several tests. The one that we're interested in today is One-way ANOVA.

After selecting One-way ANOVA you'll get a window that looks like this. Here you should select the variables that you are testing. Your test variable is your dependent variable. Your group variable is the independent variable that assigns each subject to a group.

Click on the "contrasts" button to do planned comparisons. Click on the "post hoc" button to do post hocs.

• To do planned means comparisons you need to set up orthogonal comparisions. Notice that the SPSS box makes you enter some "coefficients". This is where the "orthogonal" part will come in. Basically it means that the coefficients that you enter have to sum up to zero. The coeffiecients are how you tell SPSS which groups to compare. You assign positive numbers for one (or more, but we won't worry about these more advanced comparisions) group and negative numbers to the group you want to compare it to. Then you assign zeros to the groups that you want to ignore. Consider the following set of coefficents.

  Coefficents Comparison 1, -1, 0 group 1 vs. group 2 1, 0, -1 group 1 vs. group 3 0, 1, -1 group 2 vs. group 3 To the right is what the output from SPSS looks like. The results of Comparision 1 (group 1 vs 2) is significant (p = 0.001) Comparison 2 (group 1 vs 3) is not significant (p > 0.05) Comparison 3 (group 2 vs 3) is significant (p = 0.001)

• To do post hoc tests is even easier. All you need to do is to click on the box of the kind of post hoc test that you want to do. Some of the most common are Tukey's HSD, Fisher's LSD, and Scheffe (a very conservative post hoc test). Notice that to do these tests you need to specify what level of a you want to use.

  The output for these three tests is presented below. For each, you will see the results of each pairwise comparision. For example, the Tukey HSD test, book alone vs. notes alone, is significant (p = 0.004), while book alone vs. borrowed notes is not significant (p > 0.05). The next set is notes alone vs book alone and notes alone vs borrowed, and the set after that is borrowed against book alone and borrowed against notes alone. The results for the other post hoc tests are aranged in the same way.

  

  Based on these results (either the planned comparisons, if we had some reason in advance to test the groups against one another, or the post hocs, if we found rejected the H₀ based on our ANOVA first) we can reject several of the alternative hypotheses:

  m1 not equal to m2 not equal to m3	REJECT
  m1 not equal to m2 = m3	REJECT
  m1 = m2 not equal to m3	REJECT
  m1 = m3 not equal to m2	FAIL TO REJECT

An example

A drug company is developing several new pain killers. It wants two test the effectiveness of the drugs compared to a placebo. They give 4 groups of participants one of 4 drugs, A, B, C, and Placebo and then measure their pain tolerance. Consider the following set of data. Use SPSS to perform the One-way ANOVA.

Drug type
Placebo	Drug A	Drug B	Drug C
0	0	3	8
0	1	4	5
3	2	5	5

• What is your computed F-ratio?
• What is your H₀?
• If we assume a = 0.05, what conclusion will we make about our H₀?

  Source SS df MS F p Between treatments 54 3 18.0 9.0 0.006 Within treatments 16 8 2.0 Total 70 11 H0: m1 = m2 = m3 = m4 Looking at the p-value, we should reject H0.

• Which groups are different from the placebo?
  • Assume that you had this question in mind in advance and do planned comparisons.

    Comparison t p 1 (1, -1, 0, 0) 0.0 > 0.05 2 (1, 0, -1, 0) -2.6 0.032 3 (1, 0, 0, -1) -4.3 0.003 Looking at the p-values, we should conclude that drugs B and C differ from the placebo, but drug A does not.

  • Assume that you didn't think about this in advance, so you do a Tukey HSD test.

    Comparison difference p Placebo drug A 0.0 > 0.05 drug B -3.0 0.117 drug C -5.0 0.011 Looking at the p-values, we should conclude that C differs from the placebo, but A and B don't. Notice that this differs from the results of the planned comparisons. Planned comparisons are more statistically powerful than post hoc tests.