Post originally appeared in the Thomas B. Fordham Flypaper blog, republished here with slight modifications.
Education Cities and Great Schools recently released a useful new educational data tool called the Education Equality Index (EEI), which allows users to compare cities and states across the nation that are “closing the achievement gap.” The tool compiles school-level low-income student achievement data (2011–2014), compares it to state average proficiency rates for all students (by test and grade), and adjusts the school’s score based on the population it serves. The EEI then rolls up these school scores into city- and state-wide scores and quantifies the size of this gap.
Education Cities should be applauded for helping raise the issue around our nation’s huge achievement gap. We need to pay more attention to disparities by race, income, and gender in our schools. We need to apply even more resources to understanding how schools are narrowing these gaps and devote greater attention to those efforts that are actually getting students to achieve at higher levels. EEI also has some great data visualizations from which the National Center on Educational Statistics could take a few tips.
There are, however, a number of significant problems with the EEI tool—not least of which is the inherent limitation of using any single measure to evaluate complex systems like districts or schools. Here are my top five concerns.
1. Defining “the gap” matters: EEI’s achievement gap is based on a comparison between the proficiency rate of low-income students in a given school and the same rate for all students in the state; it is not a direct comparison between low-income students and their better-off peers in the same school (or district or state). As a result of the EEI comparison bar, there can be some very misleading conclusions about gap-closing. According to the EEI measure, for example, Denver has one of the highest achievement gaps in the country (this echoes findings from lots of other research). The EEI metric also rates Denver as the number-two city (behind Omaha) in terms of “closing the gap.” Denver looks like it is closing the gap quickly, in large part because the state as a whole has made so little progress improving proficiency rates.
Looking within the district, however, Denver’s non-low-income students are improving at a very similar rate their low-income peers (reading and writing), or a greater rate (math). In a school (or city) where low-income students outperform the state average, therefore, they will have “closed the gap” by the EEI measure—even if large gaps remain between low-income and non-low-income students. The gap within Denver is not actually closing, in other words. That’s an important point that could be easily lost on some readers. Denver should be celebrated for driving improvements for all kids, but not for closing gaps. We suspect that this may be the case in other cities that EEI lauded, such as Washington, D.C.
2. The EEI ignores absolute performance: It appears that it is easier to narrow the EEI gap in states with lower overall performance. For example, let’s take the case of Boston, which ranks sixty-ninth on the EEI. If we look at performance on an easily comparable test (TUDA), we see that 28 percent of students eligible for free or reduced-price lunch (FRL) in Boston reached proficiency on the 2015 eighth-grade reading test. Massachusetts has the highest performance of any state on that test (46 percent of eighth graders scored proficient or above), meaning that Boston has a large gap by this measure. Taken by itself, one could interpret that Boston is doing one of the worst jobs educating low-income kids. But compare Boston’s performance to Chicago (ranked nineteenth on EEI; 19 percent of FRL students scored proficient) or Tampa (ranked twenty-second on EEI; 15 percent of FRL students scored proficient). Presenting gap data in a vacuum omits an important part of the story—we should be asking where outcomes are best for students. Which leads me to my next point.
3. Be careful making comparisons on one measure: While I love having a new large data set, it is still a fairly limited snapshot of most schools and cities. And though I appreciate the effort to make cross-city comparisons, some of those comparisons are problematic viewed through this single lens. This is particularly true for comparing states or cities across state boundaries to one another with this data set. Should New Mexico be celebrated for having a low-achievement gap using this method? Is Miami really six times better than Denver at closing the achievement gap, as determined by the portion of its students attending gap-closing schools? Given the constraints I’ve discussed, we do not think so without a much better understanding of other data about these places.
4. Messaging and Spin: Education Cities (and much of the press) has lauded schools and cities that are closing the achievement gap by income. For example, Education Cities notes that D.C.’s achievement gap is lower than 90 percent of other cities. But if we look at TUDA results, we see that 8 percent of the District’s low-income students meet proficiency, compared to 53 percent of more affluent students. There is no question that many of the schools and cities recognized by the EEI should be celebrated for improving achievement for low-income students; but in reality, there are no cities and very few schools that have closed the achievement gap by income. This is the most vexing problem in schools and American society today. Some the most exalted schools on the Education Cities lists are doing great relative to most schools. But if schools and cities were truly closing the achievement gap, we would see low-income and non-low-income students matriculating to and completing college at similar rates, and scoring the same on AP, ACT, and SAT tests. That’s simply not happening.
5. Myths and misunderstandings: I get the need to venerate schools and districts that are making progress. In addition to doling out praise, we need to shed more light on exactly what these schools are doing so that others might learn (as well as encourage these schools to do even better). Let’s not hang “Mission Accomplished” banners when we know even the best gap-closing schools are often far from fully achieving their aim. This methodology hides the work we have left to do. It is not just misleading, but also dangerous to suggest to policymakers and educators that all reforms have worked—some have, some have not. We need to be clear that we are making progress in some places—not in all—and by any measure, we still have a long way to go.
Even with all of these concerns, I’d still say that, on balance, the EEI tool is a valuable addition to the array of available information about American schools for an informed user. While I applaud Education Cities for quickly addressing the problem with their state ranking methodology, I hope that they can also constructively respond to these other concerns while building in more local data so that users don’t come to the wrong conclusions about particular places. Not only could this be misleading for funders, who need to be adequately informed to make the tough calls about how to improve school systems; it is even more problematic for families, who need to choose schools that work best for their kids. Brightening our students’ academic prospects won’t be the work of three years, or five, or ten. It will require decades to realize significant improvement, and we need to get away from thinking that it can be accomplished in the next grant cycle. We need to act with the “fierce urgency of now,” as Dr. Martin Luther King said, while being more thoughtful about how we measure success and finish the work we have left before us.