Education Cities has released a new report and interactive data on achievement gaps nationally. They have collected a large national data set and have created a new metric, which they are calling the Education Equality Index (EEI) in order to compare changes in achievement gaps in schools, cities, and states over time. Their calculations are based on pass-rates on state tests for students who are eligible for free or reduced lunch (FRL).

It is exciting that they are attempting to collect, organize, and interpret achievement gap data nationally, and the website provides great interactive graphics to dig into the data.

However, the EEI metric they have created is extremely problematic, to the extent of being nearly un-interpretable.

Strangely-triangular “normal” distribution from the methodology page

The methodology they released is a bit vague, but, essentially, to calculate a school’s EEI, they create a pseudo-school made up of just the school’s FRL students, calculate the performance of this pseudo-school as percentile compared to all of the other (real) schools in the state, and then do some adjustments. These school scores are combined to calculate city and state EEIs.

EEI doesn’t really measure equality or size of the achievement gap

Based on the information that has been released about their methodology, here are a few of the big concerns I have about the metric: things that are true about EEI, but that shouldn’t be true of any good measure of achievement gaps and academic equality.

1. It is possible for two schools with identical EEI scores to have massively different internal achievement gaps.
   Assume that both schools enroll the same proportions of FRL students.
   At School A, the same percentage of FRL and non-FRL (say 70%) pass the state test–there is no gap.
   At School B, 70% of FRL students pass the test (just like School A), but, here, 100% of the non-FRL students pass–the gap is 30%. 
   However, these two schools would have the same EEI score! EEI is publicized as a measure of equality (“Education Equality Index”), but a measure of equality is invalid if it gives the same rating to these two schools, which clearly have very different achievement gaps! 
2. It is possible for the gap to be WIDENING in a school or small city, and yet that school or city shows improvement on the EEI. If a city’s non-FRL students increase by more than the FRL students, then the gap is widening, by definition. However, if the FRL students increased by more than other schools in the state, the EEI will improve. A metric publicized as being a measure of equality is not valid if it can improve when scores become less equal.
3. A school or small city’s EEI (and the direction the EEI is moving–which supposedly indicates widening or shrinking achievement gaps) depends on arbitrary proficiency cut-scores.  Basing calculations on percentages of students who pass as opposed to students’ average scores has been well-documented to distort achievement gaps for several mathematical reasons. See this article, based on work by Andrew Ho, for more details.
   In addition to the problems described in that article, using such a metric as the raw data to construct the EEI leads to an unacceptable characteristic of EEI–that the EEI (and subsequent conclusions about existence of a gap, size of the gap, and change in the size of the gap) all depend not just on students’ actual test scores, but also depend on an arbitrarily chosen number, the cut-score. It is absurd that a school or city could be praised for closing an achievement gap (if the cut-score is set at 350), but that same school or city could be criticized for NOT closing the achievement gap (if the cut-score were to be set at 370)–even though the students’ actual test scores are identical in both situations. Whether or not an achievement gap exists is not dependant on an arbitrarily chosen number, so a metric that claims to quantify an achievement gap should not be, either. 

   Here are some more details about how that could happen (optional): 
   As a thought experiment, imagine a state’s cut-score was 0 points–anyone who writes their name passes. In this case 100% of non-FRL students would pass and 100% of FRL students would pass: there is no achievement gap. Now, assume the highest possible test score on the state test is 500 points and imagine that the cut-score was 501–so high that no one can pass. Since 0% of FRL and 0% of non-FRL students pass,  there is also no achievement gap. Now, assume all of student tests have already been graded and each student has a particular score. Now, it just remains to determine if they passed. Imagine slowly sliding a hypothetical cut score from 0 points (where everyone passes) to above 500 points (where no one passes). As you slide, the percentage of students who pass decreases for FRL and non-FRL students in the school or city AND at the state level. As these numbers change, each school’s EEI will go up and down in a complex way that depends on the score distributions of each of these groups of students at all schools across the state, the proportions of FRL students at each school, and the total number of students at each school. In some cases, there can be particular values of the cut-score at which a school’s EEI may be higher than last year’s EEI (indicating a shrinking achievement gap) and at other particular values of the cut-score the EEI may be lower than last year’s (indicating a widening achievement gap). A state’s choice of a cut score shouldn’t be able to determine whether or not an achievement gap exists!
   In Education Cities’ defense, they were basing their analysis on these kinds of scores (reported as “% who exceeded a cutoff”) since that is what they had access to from states, as they apparently didn’t have student-level data. While this is understandable, it makes the analysis susceptible to these problems.

EEI is impossible to interpret (and designed that way)

It is hard to tell from just looking at EEI what each point on EEI actually represents. Moving from an EEI of a 32 to 33 means what exactly?

Unfortunately, although this metric is quite abstract, Education Cities has chosen to scale it onto a range of 0 to 100, making it very easy to misinterpret as some kind of percentage (which it is isn’t) or some kind of percentile, which it only kind-of is.

This 100-based scale also makes lay readers assume that 100 is a perfect score, but 100 may not actually be achievable, in most cases. Additionally, any EEI of 100 (or near 100) indicates that FRL students are doing substantially better than their non-FRL peers, which is actually a very non-equal situation and one that a valid measure of “equality” shouldn’t favor.

However, because EEI is designed to be on this 0-100 scale, it makes it very easy for people to misinterpret EEI data. For example,  Chalkbeat TN:

While Memphis’ gap is larger than 70 percent of major U.S. cities, it narrowed the gap by a whopping 19 percent between 2011 and 2014, one of the fastest rates in the nation, the study says.

Narrowing the achievement gap by 19% is NOT a valid interpretation of an EEI change from 23.7 in 2011 to 28.3 in 2014. Yes, the EEI has increased by 19%, but EEI isn’t directly a measure of points of achievement gap decline. It would be correct to, instead, say something like:

An abstract metric that can’t be directly interpreted due to its design but that is somewhat related to how Memphis’ FRL students performed relative to all students in the state has increased by 19%.