There is a very interesting new paper out from C. Kirabo Jackson of Northwestern University: Non-Cognitive Ability, Test Scores, and Teacher Quality: Evidence from 9th Grade Teachers in North Carolina.

He looked at LOTS of data and isolated two factors–which he refers to as cognitive and non-cognitive skill–that each seem to explain much of the variation in particular stats for a given student: algebra I and English standardized test scores, attendance, discipline, GPA, on-time grade promotion, etc.

He was able to statistically isolate these two factors from other demographic factors and was able to track how these changed over time for particular students.

In particular, he looked at the effect that teachers had on both of these factors.

He built a very sophisticated model designed to separate out the effects strictly associated with particular teachers (as opposed to effects due to other courses students might be taking simultaneously, students/teachers not being randomly assigned to courses/levels, school-wide effects, etc.). Students were effectively only compared with other students who took the same classes and went to the same school. He worked through very detailed justifications for why this model is indeed valid (and better than alternatives).

Here are the results:

•  These two statistically-meaningful measures (cognitive and non-cognitive skill) can indeed be identified for each student. The factor called cognitive skill is most-closely related to standardized test scores, while the factor called non-cognitive skill is most-closely related to the measures of attendance, behavior, GPA, etc. These factors are indeed distinct: students with high cognitive skill may or may not have high non-cognitive skill.
• BOTH of these can be used to predict future outcomes (college attendance, arrest rates, future wages).
• Individual teachers do indeed play a large role in changing student cognitive and non-cognitive skill.
• The effect of teachers on cognitive skill is NOT correlated with those same teachers’ effect on non-cognitive skill. Thus, for teachers who are above average at improving cognitive skill, about half will be above average at improving non-cognitive skill while about half will be BELOW average at improving non-cognitive skill. (If teachers who are the best at improving non-cognitive skill are also the best at improving non-cognitive skill, we’d expect all teachers who are “above average” at for cognitive to also be “above average” for non-cognitive. However, this study, surprisingly, says that a teacher’s ability to improve cognitive skill IS NOT linked with their ability to improve non-cognitive skill.)

Thus, since test scores are most-closely associated statistically with the cognitive skill factor, not all teachers who are particularly good at raising test scores are particularly good at improving non-cognitive skills.

Yet, the non-cognitive skills are strong predictors of future outcomes (particularly for students closer to the lower end of the income spectrum, interestingly). Thus, test-scores themselves miss out on this important factor AND a teacher who is good at increasing test scores might fall in ANY percentile of teachers with regard to how well (s)he increases  non-cognitive skills.

While measuring teachers’ ability to increase test-scores seems to be a measure of the effect they have had on students’ cognitive skills, this entirely misses the (equally important) effect the teachers may or may not be having on non-cognitive skills:

Because unexplained variability in outcomes associated with individual teachers is not just noise, but is systematically associated with their ability to improve unmeasured noncognitive skills, classifying teachers based on their test score value-added will likely lead to large shares of excellent teachers being deemed poor and vice versa.

Jackson goes on to point out the possible frightening implications of incentivizing teachers JUST to raise scores:

[…] if teachers must expend less effort improving non-cognitive ability in order to improve cognitive ability, regimes that increase the external rewards for test scores (such as paying teachers for test score performance or test-based accountability) may undermine the creation of students’ non-cognitive skills.

I’m curious to see how this will change the debate about teacher assessment (let alone about STUDENT assessment)!

HT: Larry Ferlazzo