Hattie's Rankings

From Visible Learning 2009 - based on the effect size statistic "d"

Note: in Hattie’s 2017 publication, Learning strategies: a synthesis and conceptual model - he argues against ranking (p9)!
'There is much debate about the optimal strategies of learning, and indeed we identified >400 terms used to describe these strategies. Our initial aim was to rank the various strategies in terms of their effectiveness but this soon was abandoned. There was too much variability in the effectiveness of most strategies depending on when they were used during the learning process.'
Wecker et al (2016, p34) also question these rankings:
'Hattie's ranking suggests that the 138 influencing factors are to be seen as basically alternative options for action, from which the most effective one is to be selected, for example, during lesson design. However, this would require that the influencing factors are in the first place real options for action and that their effect sizes were determined in relation to the same "baseline", ie. that the control conditions in the primary studies in all meta-analyzes included in the first step are to be regarded as an equivalent benchmark. A look at the ranking shows that this is neither the case nor is it possible at all, since the influencing factors belong to completely different types: On the one hand, one finds not only institutional framework conditions such as school size. As well as personal traits such as the self-concept. Such as those classified under "direct instruction". Effect sizes usually relate to a comparison with some form of "traditional" instruction... 
It is doubtful that teachers would ever usefully choose between direct instruction instruction and distributed practice rather than the optimal combination of both to decide. If this is the case, the assessment of the relative effectiveness of these options cannot be based in one case on a comparison with an instructional "standard condition" and on the other hand on a comparison with another design variant of the same instruction form.'
Prof Adrian Simpson is similarly critical of rankings in his detailed analysis, 'The misdirection of public policy: comparing and combining standardised effect sizes' (p451),
'The numerical summaries used to develop the toolkit (or the alternative ‘barometer of influences’: Hattie 2009) are not a measure of educational impact because larger numbers produced from this process are not indicative of larger educational impact. 
Instead, areas which rank highly in Marzano (1998), Hattie (2009) and Higgins et al. (2013) are those in which researchers can design more sensitive experiments. 
As such, using these ranked meta-meta-analyses to drive educational policy is misguided.'
Schulmeister & Loviscach (2014) Errors in John Hattie’s “Visible Learning”.
'If one corrects the errors mentioned above, list positions take big leaps up or down.Even more concerning is the absurd precision this ranking conveys. It only shows the averages of effect sizes but not their considerable variation within every group formed by Hattie and even more so within every individual meta-analysis.'
'To think that didactics can be presented as a clear ranking order of effect sizes. It is a dangerous illusion. To an extreme degree, the effect of a specific intervention depends on the circumstances. Focusing on the mean effect sizes and ignoring their considerable variations and condensing the data to a seeming exact ranking order, Hattie pulls the wool over his audience’s eyes.'

1StudentSelf-report grades1.44
2StudentPiagetian programs1.28
3TeachingProviding formative evaluation0.90
4TeacherMicro teaching0.88
6SchoolClassroom behavioral0.80
7TeachingComprehensive interventions for learning disabled students0.77
8TeacherTeacher clarity0.75
9TeachingReciprocal teaching0.74
11TeacherTeacher-student relationships0.72
12TeachingSpaced vs. mass practice0.71
13TeachingMeta-cognitive strategies0.69
14StudentPrior achievement0.67
15CurriculaVocabulary programs0.67
16CurriculaRepeated reading programs0.67
17CurriculaCreativity programs0.65
19TeacherProfessional development0.62
20TeachingProblem-solving teaching0.61
21TeacherNot Labeling students0.61
22CurriculaPhonics instruction0.60
23TeachingTeaching strategies0.60
24TeachingCooperative vs. individualistic learning0.59
25TeachingStudy skills0.59
26TeachingDirect Instruction0.59
27CurriculaTactile stimulation programs0.58
28CurriculaComprehension programs0.58
29TeachingMastery learning0.58
30TeachingWorked examples0.57
31HomeHome environment0.57
32HomeSocioeconomic status0.57
33TeachingConcept mapping0.57
35CurriculaVisual-perception programs0.55
36TeachingPeer tutoring0.55
37TeachingCooperative vs. competitive learning0.54
38StudentPre-term birth weight0.54
39SchoolClassroom cohesion0.53
40TeachingKeller's PIS0.53
41SchoolPeer influences0.53
42SchoolClassroom management0.52
43CurriculaOutdoor/adventure Programs0.52
44TeachingInteractive video methods0.52
45HomeParental involvement0.51
46CurriculaPlay programs0.50
47CurriculaSecond/third chance programs0.50
48SchoolSmall group learning0.49
50SchoolSchool effects0.48
52StudentEarly intervention0.47
55StudentPreschool programs0.45
56TeacherQuality ofTeaching0.44
57CurriculaWriting Programs0.44
59SchoolSchool size0.43
61TeachingBehavioral organizers/Adjunct questions0.41
62TeachingMatching style of learning0.41
63TeachingCooperative learning0.41
65CurriculaSocial skills programs0.40
66StudentReducing anxiety0.40
67CurriculaIntegrated Curriculum Programs0.39
69CurriculaCareer Interventions0.38
70TeachingTime on Task0.38
71TeachingComputer assisted instruction0.37
72TeachingAdjunct aids0.37
73CurriculaBilingual programs0.37
74SchoolPrincipals/ school leaders0.36
75StudentAttitude to mathematics/science0.36
76CurriculaExposure to reading0.36
77CurriculaDrama/Arts programs0.35
79TeachingFrequent/effects of testing0.34
80SchoolDecreasing disruptive behavior0.34
83TeachingInductive teaching0.33
84StudentPositive view of own ethnicity0.32
85TeacherTeacher effects0.32
86TeachingInquiry based teaching0.31
87SchoolAbility grouping for gifted Students0.30
89HomeHome visiting0.29
93CurriculaUse of calculators0.27
94CurriculaValues/moral education programs0.24
95TeachingProgrammed instruction0.24
96TeachingSpecial college programs0.24
97TeachingCompetitive vs. individualistic learning0.24
98SchoolSummer school0.23
100TeachingIndividualized instruction0.23
101SchoolReligious Schools0.23
102StudentLack of Illness0.23
103TeachingTeaching test taking0.22
104TeachingVisual/audio-visual methods0.22
105TeachingComprehensive teaching reforms0.22
106SchoolClass size0.21
107SchoolCharter Schools0.20
108TeachingAptitude/treatment interactions0.19
110TeachingLearning hierarchies0.19
111TeachingCo-team teaching0.19
112TeachingWeb-based learning0.18
113HomeFamily structure0.17
114CurriculaExtra-curricular programs0.17
115TeachingTeacher immediacy0.16
116SchoolWithin class grouping0.16
117TeachingHome-school programs0.16
118TeachingProblem-based learning0.15
119CurriculaSentence combining programs0.15
121SchoolAbility grouping0.12
124TeacherTeacher training0.11
125TeacherTeacher subject matter knowledge0.09
126TeachingDistance Education0.09
127SchoolOut of school curricula experiences0.09
128CurriculaPerceptual-Motor programs0.08
129CurriculaWhole language0.06
130SchoolCollege halls of residence0.05
131SchoolMulti-grade/age classes0.04
132TeachingStudent control over learning0.04
133SchoolOpen vs. traditional0.01
134SchoolSummer vacation-0.09
135HomeWelfare policies-0.12

No comments:

Post a Comment