It is not news that, in the last decade, the use of data (often based on standardized test scores) has exploded. In addition to being used to assess how well students have learned the material for a particular class, student test scores are increasingly being used to assess teachers and schools.

Proponents of assessment vs. Opponents of hyper-testing: a false distinction

Proponents of this testing argue that students shouldn’t be able to be promoted to the next grade if they haven’t mastered the previous grade (they’re right). They argue that teachers and schools should have a way of knowing how well their methods are actually working (they’re right, too). They argue that this data can be used to pinpoint areas of student weakness, so that these can be addressed (they’re right). Some also argue that students and parents should know how well their schools are doing (they might be right). Let’s say that  people who agree with most or all of these points are called “group 1.”

Opponents of this testing argue that important things often can’t be measured by standardized tests (they’re right). They argue that testing turns schools into test-prep factories for months and even sometimes entire school years (they’re right). They argue that students start thinking of success as simply a matter of getting a high score on a test and lose sight of what actual human development and success could look like (they’re right). They argue that this will lead to schools cutting all non-tested subjects and eliminating major projects from classes and eliminating field trips and eliminating other non-test-related activities (they’re right). They argue that in the eyes of the school, students will lose all humanity and be reduced to a set of numbers that characterize how well they can repeat what their teachers have told them (they might be right about that, too). Let’s say that people who agree with most or all of these points are called “group 2.”

More and more people are starting to realize that proponents of assessment AND opponents of hyper-testing are BOTH actually making some pretty reasonable assertions. Many people are even starting to realize that group 1 and group 2 need not be mutually exclusive. That is, people might simultaneously consider themselves members of BOTH group 1 and group 2 at the same time. My guess is there are an increasing number of people who would categorize their views in that way.

This strikes me as a positive development. People who are in both groups would probably agree that feedback, assessment, and data are important parts of a student’s education, but that standardized tests don’t really provide all the information that is needed or that they might cause too much harm to other beneficial aspects of education.

How to please group 1 and group 2

Lisa Nielsen has an excellent post in which she provides some strategies for actually collecting and using data on things that actually matter: Transforming education by measuring what matters. Hint: It’s not test scores.

Here are a few things she wants schools to collect data on:

• Which students have a plan to develop their passions?
• Which students have strong advisers and advocates at school?
• How satisfied are students with the support they receive at school?

She lists some more things school should be tracking and offers some models for how this could actually look when implemented in a school. I encourage you to read the full article.

At the beginning of this school year, I was thrilled to discover that the school I’m working in does actually try to track which students have strong relationships with teachers–as reported by the teachers–to try to find students who don’t (yet!) have these connections. I’m interested to see how well this system actually works this year.

Data can only be used to “measure success”–an unnecessary assumption

Those are all EXCELLENT things to be tracking and I am thrilled that there is a growing push to be focusing on these measures as opposed to just standardized test scores.

However, I’d like for people in group 1, group 2, or both to step back a bit from their assumption that the only use of data is to “measure success.”  Sure, this data that is being collected–particularly the data mentioned above–can and should be used to measure success and then to directly try to increase success. However, that is not the only use of data. Data can also be used to find patterns that wouldn’t otherwise have become clear while providing new insights into how good teaching and learning happens.

Certainly, those who use data to measure success do inherently do some pattern-seeking. When their system is working, if they see a student’s scores going down, they rightly try to intervene to get her back on track. If a certain teacher’s students show less improvement across the board than otherwise might be expected, the teacher (hopefully) gets some additional support. If a teacher sees that many of his students did uncharacteristically poorly on an assignment covering some particular topic, he knows to reteach that topic (hopefully in a different way the second time).

However, all of these examples simply take on the following format: “scores are dropping, let’s try to fix it so the scores go back up.” Besides the fact that scores don’t necessarily correlate to actual learning, there is not necessarily a problem with this model. In fact, when using some of the more meaningful types of data listed above–beyond just test scores–this is actually a pretty good way to operate a school: use changes in this data to signal that there is some problem with the school’s effectiveness and then try to explore ways of increasing that effectiveness, with the hope that these interventions will improve the data (and the hope that the improved data actually indicates improved learning and growth among the students).

Use data for pattern-seeking!

Besides using trends in the data to directly try to find ways to create better trends in the data, the collected information can also be used to search for patterns, which can provide some surprising and useful insight about the students, the teachers, and the school…..which can then be used to help teachers to teach better and students to learn better.

Here’s a hypothetical example:  Teachers certainly have access to grades and attendance information for students in their own classes, but these are rarely linked with each other, let alone with similar data from other teachers across the school. Maybe, as it turns out, students in this particular school who are absent are substantially less likely to turn in assignments that were due the day they missed. If this is the case, the school can then explore how to streamline the process of finding a missed assignment after an absence and also work with students to help them stay motivated and responsible enough to follow through with this process once they return to school.

Yeah, you could look at attendance data, and try to improve it (“95% of students were here this week…much better than 92% last week!”)….or you could look at students grades and try to improve them (possibly by encouraging them to turn in missing work), but having access to this information in a useful way–in this case, linked together–helps the school develop a deeper understanding of the problem and more insight into how it might be solved.

Maybe, at another school, there are two teachers: Teacher A and Teacher B. Every year, the students in these two teachers’ classes score around the school average on most assessments. In a few years, once the students graduate high school, the students have GPAs comparable to all of their classmates, have taken similar numbers of AP tests (and gotten similar scores), and gotten into colleges at the same rate as their classmates (and are going to colleges of comparable quality), on average. Maybe the school is even tracking some of the more important measures such as Lisa Nielsen’s above, and students who had Teacher A or Teacher B are still very normal compared to their peers when looked at through those lenses as well.

However, maybe, for reasons that aren’t entirely understood, it is discovered that students who had Ms. A in 7th grade and Mr. B in 8th grade do substantially better than all of their peers, including those who had only had one of those teachers. Let’s say this effect even occurs over multiple years of data over an extended period of time. This would be pretty interesting!

Why might this be? Maybe Ms. A provides students a super strong foundation in something that puts kids on track to get the most out of their year with Mr. B, who is particularly effective at building on Ms. A’s foundation. Maybe the teaching styles of Ms. A and Mr. B complement each other in a perfect way that sets students on an ideal track towards future success. Maybe Ms. A teaches something from one perspective, while Mr. B teaches it from a different perspective, and students benefit from seeing both perspectives so much, and in a such way, that they are able to excel in future classes.

The observation that students who had BOTH of those teachers are more successful in the long run–even more so than students who only had one or the other–leads to a deeper exploration of WHY that might be. The school would need to look into which of the possibilities above (or none of them) was actually taking place. Maybe whatever is discovered could then inform improvements in the practice of other teachers in the school (or even at other schools).

Noticing this pattern allows the school to figure out what can be learned from these teachers (even if they didn’t realize it themselves). It’s not a matter of saying “Mr. B., let’s figure out how to increase your scores,” but rather, “wow, this is pretty cool, lets figure out why this particular effect is happening and see if we can help other students get these same benefits.”

Finale–“Data: not just for assessment, anymore!”

Measure things that matter and then try to find deeper patterns in this information to better inform what works and what doesn’t (and even come up with some guesses as to WHY), and then try to test some of these ideas. In addition to the valid and justified desire to use data to see how you are doing and try to improve, this pattern-seeking perspective can reveal some deeper insights from your data and lead to some longer-term benefits. Collecting data is NOT just a tool for assessment!

Additional thoughts/caveats:

1. Just because some pattern is observed, does NOT necessarily tell you what is causing that pattern. For example, maybe there is an assistant principal at Ms. A and Mr. B’s school who really likes those two teachers, and takes some of the top kids every year and puts them in Ms. A’s class in 7th grade and then in Mr. B’s class in 8th grade. This may indeed be enough to explain the observed pattern of achievement for students who have had both teachers. Whenever some kind of interesting pattern emerges, more work is needed to be able to determine how interesting the root cause of that pattern might be–maybe some great new insight about how people learn, or it may be nothing of note.

2. Just like not all observed patterns are interesting, it is also not possible to observe every interesting pattern in this way–professional teachers in classrooms can pick up on things that are not represented in the data, but which have huge impacts on kids’ lives. This is meant to supplement the work of those teachers, not to replace it.

3. There is some overlap between this pattern-seeking and the concept of formative assessment. Teachers who make excellent use of formative assessment in their classrooms are probably already pretty adept at finding patterns in student information for a particular student over time or for a whole class over one lesson. The pattern-seeking mindset discussed here would seek to enhance teachers’ ability to do this well and to help them pick up on things a normal human might not naturally notice.

4. To actually start doing a better job of this pattern-seeking, school districts need better software that is designed, first of all, to have ALL of student data in one place–not spread across different software, and, secondly, to be designed to actually explore the data in this way, both by curious humans who want to play around with the data or test their guesses about patterns that may exist AND by statistical software on computers designed for this purpose.