Behavior

Hattie has divided this into the separate categories of:

Decreasing Disruptive Behavior - d = 0.34 (Rank = 80)
Classroom Management - d = 0.52 (Rank = 42)
Classroom Cohesion - d = 0.53 (Rank = 39)
Classroom Behavioral - d = 0.80 (Rank = 6)

There is a lot of overlap in these categories which is a problem concerning confounding variables. Teacher experience would suggest that improving behavior would increase achievement and some of these results verify this.

However, using Hattie's interpretation, d = 0.40, as the 'hinge' point' for identifying what is and what is not effective; 'decreasing disruptive behavior' would appear not to be as effective as other influences.

Decreasing Disruptive Behavior:

Hattie uses 3 meta-analyses to get his average, d = 0.34.


Authors/Disruptive BehaviourYear#studiesstudentsMean (d)CLEVariable
Skiba & Casey1985418830.9366%disruptive behaviour
Stage & Quiroz19979950570.7855%decreasing disruptions
Reid, Gonzlez, Nordness, et al2004252486-0.69-49%behavioural disturbance

Reid et al (2004) seem to contradict the other 2 studies by indicating that decreasing disruptive behavior decreases student achievement. How can reducing disruptive behavior decrease achievement to that extent?

The negative probability value (CLE) of -49% is also another major mistake by Hattie.

Reid, et al., compared the achievement of students labelled with 'emotional/behavioral' disturbance (EBD) with a 'normative' group. They used a range of measures to determine EBD, e.g., students who are currently in programs for severe behavior problems e.g., psychiatric hospitals (p132).

The effect size was calculated by using (EBD achievement) - (Normative achievement) / SD (p133).

The negative effect size indicates the EBD group performed well below the normative group. The authors conclude: "students with EBD performed at a significantly lower level than did students without those disabilities across academic subjects and settings" (p130).

Hattie clearly misrepresents this study, as it is not investigating 'decreasing disruptive behavior' as a teaching strategy. Also, the sample is from a population of 'abnormal' students. Therefore this meta-analysis should not be included in Hattie's work.

If this study was removed, the average effect size would then be, d = 0.86 which would rocket its ranking up to #6.

Hattie's lack of consistency in interpreting effect sizes is a major problem, e.g., using Frazier et al (2007) (see below) students were identified as having ADHD and proxies for achievement were used: GPA, class and parent rating, etc (p51).

The ADHD group was the control group and the normative group was the experimental group. So effect size was calculated using (Normative - ADHD) getting a large positive result.


This is the REVERSE of Reid et al (2004) study who calculated effect size using (EBD - Normative) getting a large negative result.


Skiba, et al (1985) investigated reinforcement and feedback as strategies for reducing behavioral problems. They measured indicators of behavior, like noncompliance, off task and withdrawal; they did not measure achievement.

They conclude,

"Results indicated that both reinforcement and feedback type procedures are highly effective in the remediation of classroom behavior problems across a variety of behaviors, settings, and administrative arrangements" (p472).

So the effect size is not an indicator of achievement but of behavior.

Stage, et al (1997) also only measured disruptive behavior, not achievement "the treated students reduced their disruptive behavior compared to nontreated students" (from abstract).

Classroom Management:

Hattie uses 1 study Marzano (2003) 'Classroom management that works', to get d = 0.52.

Note: an excellent blog detailing the issues with Marzano's research can be found here.

The study does measure achievement and aims to have randomised control and experimental groups, so the result is worthwhile. However, there are only 553 students used (p10) - Hattie does not mention this.


Classroom Cohesion:

Hattie uses 3 meta-analyses to get his average, d = 0.53.


AuthorsYear#studiesstudentsMean (d)CLEVariable
Haertel et al198012178050.1712%classroom climate
Evans & Dion1991270.9265%group cohesion
Mullen & Copper19944987020.5136%group cohesion

Haertel et al (1980) state, "The socio-psychological environment is typically measured by asking students to agree on a three-to-five point scale to such items as 'The students enjoy their work in the  class' and 'The goals of the class are clear'. The purpose of the study is to estimate the magnitude of this relationship between learning and the environment of the classroom, and the relation of its variability across grades, subject areas and aspects of the learning environment" (p113).

"Learning outcomes and gains, including student achievement, performance and self-concept, were found to be positively associated with student perceived Cohesiveness, Satisfaction, Task Difficulty, Formality, Goal Direction, Democracy and Material Environment. Negative associations were found with Friction, Cliqueness, Apathy and Dis-organisation" (p114).

They warn, "given the correlational basis of much of the research, however rigorously controlled by conventional statistical methods, the next steps in the research should emphasise continued true experimentation" (p114).

Evans & Dion (1991) found a large effect size relating cohesion to group performance. However, the studies were mostly on small sports teams and military units. Also, they state performance criteria for these types of groups are simple, e.g., win/loss record of a sports team. In contrast, the performance criteria for normal work groups are not easily identified (p696).

They summarise, 'Given the nature of the studies used here, caution is suggested in generalising these results to "real" work groups' (p690).

Mullen & Copper (1994) Once again studies were mostly on very small military groups and sports teams. They used many of the studies that Evans & Dion used, thus introducing bias into Hattie's work. They isolated the factor of 'commitment to task' as the most influential aspect of cohesion rather than interpersonal attraction or group pride (p210).

They conclude, the effect was larger in small groups compared to large groups and was larger in correlation studies compared to experimental studies (p210).

One wonders how relevant these are to the classroom given they are about small sports and military groups.

Classroom Behavioral:

Hattie uses 3 meta-analyses to get his average, d = 0.80.

Mullen & Copper (1994) from 'classroom cohesion' state the general rule for meta-analyses - subjects should not be sampled from abnormal populations (p215). Yet, all of the following studies are from abnormal populations -students diagnosed with ADHD or a learning disability.


Authors/Classroom behavioralYear#studiesstudentsMean (d)CLEVariable
Bender & Smith199025.1.10178%disabled behavior
Dupaul & Eckart1997630.5841%ADHD behavior
Frazier et al2007720.7150%ADHD behavior

Bender & Smith (1990) This meta-analysis did not measure classroom behaviour as an influence on achievement, as Hattie implies. But rather, it compares learning disabled students with non-disabled students on a number of behavioural measures. They conclude, 

"Results showed that both methodologically strong and weak studies demonstrated significant behavioural deficits of children with learning disabilities compared to their non-disabled peers in each of five overall areas: on-task behaviour, off-task behaviour, conduct disorders, distractibility, and shy/withdrawn behaviour. Both observational and teacher rating data demonstrated these differences. Effect sizes for both groups of studies seemed to cluster around 1 standard deviation, suggesting noticeable and educationally significant impairment in the behaviour of children with disabilities" (p298).

They caution, "that drastic increases of handicapped students in certain mainstream classes may result in a more negative classroom climate, as mainstream teachers attempt to deal with increased behaviour problems" (p305).

Below is the summary table of their results (p301), I can not find any achievement measures:



DuPaul & Eckert (2012) [note: I have not been able to find their original 1997 study but have found this updated study] The authors state, "The purpose of the present meta-analysis is to provide a quantitative review of school-based intervention studies for students with ADHD" (p389).

They conclude, "The results of this meta-analysis indicate that school-based interventions for students with ADHD yield moderate to large effects for both behavioural and academic outcomes" (p401).

DuPaul & Eckert comment on the broader issues of meta-analysis that have been covered in other sections. Regarding proper experiments:  'randomised control trials are considered the scientific "gold standard" for evaluating treatment effects ... the lack of such studies in the school-based intervention literature is a significant concern' (p408).

Regarding comparing different studies: "It is difficult to compare effect size estimates across research design types. Not only are effect size estimates calculated differently for each research design, but there appear to be differences in the types of outcome measures used across designs" (p408).

Frazier et al (2007) Once again participants were identified as having ADHD and proxies for achievement were used: GPA, class ranking, parent/teacher rating, etc (p51).

The ADHD group was the control group and the normative group was the experimental group. This was done to get positive effect sizes (p51). So the result d = 0.71 means the ADHD group underperformed the normal group. Note: Reid et al (2004) above reversed the calculation getting a negative result.

They summarise, there is a moderate to large discrepancy in academic achievement between individuals with ADHD and those without ADHD (p59).

These 3 studies simply compared students with ADHD or a learning disability with normal students and found that they performed less in achievement.