The Test Score Tango.
Preface: You may already know how I feel about Big Testing and test scores. If not and you're interested, I've written about it here, and here, and here, and here. That said, the least terrible of the Big Tests is probably the National Assessment of Educational Progress, or NAEP, if only because you don't have to shut down an entire school every year to give it, and because it doesn't sort the high-score schools from "failing" ones.
Yet, when the latest NAEP scores dropped a couple weeks ago the usual suspects were predictably aghast. Hilariously, some of the same dopes that declared the teacher shortage a fake now credulously announced the loss of twenty years of progress based on one set of scores on one test given to nine-year-olds in the third year of a fucking pandemic.
Kids, there are demons out there bent on the destruction of public schools, and they'll do whatever it takes to make it happen. When I was in the classroom, it felt like I was in the middle of one big chaotic shitstorm. It may feel that way to you, too. But it is not. It is strategy. It's a program. Think of it that way and it's easier to recognize. You just have to know what to look for. Then maybe we can fight back.
And now, the Test Score Tango.
A lot of the early NAEP score headlines were overwrought and eerily similar. The usual suspects, of course. The New York Times had "The Pandemic Erased Two Decades of Progress in Math and Reading." But also from outlets that should know better. From Chalkbeat: "Math and reading scores plummet on national test, erasing 20 years of progress." And of course piling on from The74: "‘Nation’s Report Card’: Two Decades of Growth Wiped Out by Two Years of Pandemic" (all emphasis mine). A little later The New Yorker had this terrible "interview" full of leading questions and reformster* rhetoric.
Getting out early with sensational claims helps shape the narrative. Fortunately, several observers responded quickly with a more measured, nuanced analysis than the doomsayers. This article from Education Week does a good job explaining what the scores actually mean. Jan Resseger situates the test results within the broader context of the trauma and disruption caused by the pandemic, and suggests the limits on conclusions that may be drawn from them. And this thoughtful article from Jill Barshay titled, "6 Questions to Better Understand Math and Reading Scores" asks really good questions and doesn't pretend to have all the answers. In advance.
Leonie Haimson of classsizematters.org stresses the lack of clarity in the data and pushes back on the popular assertion that closing schools and the consequent remote learning were a key driver in the score decline.
Haimson also challenges the conventional proposals for what administrators and education Big Thinkers call "learning recovery." Addressing "negative impacts on learning and test scores," Haimson writes, will require not simply "longer days and years." Instead, she advocates for smaller class sizes rather than what she calls "false and damaging policy prescriptions."
It's difficult to know whether all the hyperventilation comes from genuine misunderstanding, or underlying partisanship, or if it's just for clicks, but it is surely... overblown.
Many of the agitated commenters were no doubt expecting--and knew their readers (and funders) were expecting--the pandemic to have a severe impact on student learning. And many remain particularly focused on the closing of school buildings and the shift to remote learning. Many expected those effects to have been disastrous, and they are heavily invested in blaming teachers and their unions for the damage. When the scores came out, it was easy to read them as confirmation of their expectations.
But the truth is, in the best of times the NAEP scores don't mean what a lot of people think they mean or would like them to mean. And in pandemic times, it's even harder to draw conclusions.
One of the things about NAEP that doesn't mean what people think it means is "grade level." In 2016's "The NAEP proficiency myth" from Brookings, Tom Loveless demolishes the notion that NAEP proficiency is "synonymous with grade level" and urges states not to use NAEP’s proficient level as a basis for education policy.
Some interested parties--including organizations promoting a "failing schools" narrative to advocate for school choice--continue to deliberately misinterpret the NAEP levels and conflate "proficient" with grade level. By this definition, then, in a given year two-thirds! of test takers might fall short of "grade level."
The truth is, however, that what most of us think of as "grade level" more closely aligns to NAEP's Basic level. By simply being honest about the meaning of the NAEP scores, instead of two-thirds below grade level we in fact have two-thirds at or above. The sky may be a bit cloudy, but it's not falling.
The Covid pandemic has obviously had a profound impact on students, and the new NAEP scores clearly reflect that. However, in an example of the scores not meaning what some people would like them to mean, several partisan observers have used the scores to criticize the ways schooling was conducted during the pandemic. In doing so, they either misconstrue or ignore the data completely.
At the center of the project to blame teachers and unions for the drop in scores is the issue of school closings and the reliance on remote learning. Lots of interested parties have passed around the same study to show a link to learning loss. The NAEP scores themselves reveal a more complicated picture.
For example, as Haimson and others have noted, the evidence for a correlation between school closings/remote learning and the drop in test scores is decidedly inconclusive. And even the NCES itself warns us that "Users are cautioned against interpreting NAEP results as implying causal relations." As Peter Greene put it in 2019: "Everyone ignores that advice, but NAEP clearly acknowledges that there are too many factors at play here to focus on any single one."
What can we know based on this limited set of data? As author and former writing teacher John Warner describes it in a NAEP discussion with Nick Covington of the Human Restoration Project: We know "a big thing happened." And "not only was school disrupted, the entire lives of everybody in the country and the world was disrupted" and "the notion that we could have avoided this with some kind of different school or different choices, going back in person sooner... it's not reflected in any of the data." (7:50)
Although the impact of the pandemic and the disruption and trauma that have accompanied it are clear and universal, the impact on student learning of going remote is, as yet, unclear. Facts are stubborn things.
Another thing we can learn from NAEP concerns the testing regime spawned by No Child Left Behind. If you assume that the promoters of NCLB were acting in good faith and not simply trying to inject an air bubble into the bloodstream of public schooling, the testing regime it initiated (not NAEP, but the hundreds of other tests kids have to take every year) hasn't succeeded on its own terms. Rather than more and more testing leading to a rise in scores, with the exception of a limited burst in the early 2000s, the NAEP scores have flattened. Big Testing hasn't worked.
Those are things we can legitimately infer from the limited NAEP data we have right now. But what about "Two Decades of Growth Wiped Out!"? It would be easy to dismiss it as professional alarmism, but in this case the stakes are too high to simply chalk it up to selling papers.
There are demons out there, and they are perched, waiting to jump to conclusions and manufacture analyses that serve the project to dismantle authentic** public schooling. They can't wait even a few weeks until more complete data is available.
As Jill Barshay writes, "A more detailed report from the Department of Education on student achievement during the pandemic is expected in October. It will list state achievement scores for fourth and eighth graders on another NAEP test. Hopefully, we can unravel more of these knots together." Maybe that's what the vultures are afraid of.
In this environment, information like the NAEP scores is routinely politicized. The scores are used to sell programs and products from organizations that profit from brokenness. And of course, the scores are cynically weaponized by the school choicers and their enablers in the press in order to promote a failing schools narrative and advance the choice and anti-union agenda.
But the scores do not mean what the screamers say they mean. In order to turn the volume up to 11, they compare the raw scores to previous scores, find a match twenty years ago, and announce twenty years of progress loss. The thing is, it's twenty years of test score progress, not twenty years of student progress.
In an interview conducted by Liz Mineo for the Harvard Gazette, Dr. Andrew Ho, professor at the Harvard Graduate School of Education and former member of the NAEP governing board, explains:
[E]ach point on the NAEP scale is roughly three weeks of an academic year, and the overall decline was seven points, which is roughly five months in terms of an academic school year of learning that this cohort of students is relatively behind.
Five months. Not twenty years. But the average overall decline doesn't tell the whole story, of course. Predictably, students who were most challenged before the pandemic were most challenged by the pandemic. For students who were already struggling, who were already coping with insufficient resources--including educational resources--and whose families were likely to be most severely impacted by the pandemic, results were more alarming. From Dr. Ho:
But this average betrays the inequality where the decline for the higher-scoring students was only three points, or nine weeks, whereas the decline for the lowest-scoring students was 12 points, which is 36 weeks, which is almost approaching an entire academic year.
Terrible. Unjust. Unacceptable. We need to address this, but this is not just a test score. This is a reflection of the racism and resource inequality that permeates our system. And still, the most impacted students lost "almost approaching an entire academic year." NOT twenty.
The kids being told right now that the scores are down and the pandemic has cost them twenty years are being crushed for the objectives of the crushers.
Dr. Ho goes on to discuss the broader impact of the pandemic on students' lives apart from one set of test scores and reminds us what returning to "normal" means for students getting back to school. He also suggests the need for a "learning infrastructure," perhaps in the form of a "National Assessment of Educational Equity." In his words:
I’m not just worried about academic learning. I’m worried about the structures of educational opportunity and inequality that have increased over the pandemic; this is just the tip of the iceberg. I hope we can pay attention to the entire structure of educational opportunity. Academic learning outcomes are easy to measure, but also easy to overemphasize, as if students are only trying to get better at math and reading and not also trying to reconnect with their classmates and their teachers and remember how to sit still, listen, learn, and enjoy learning and playing with others.
Again, even among the most severely impacted, most vulnerable students, the setback was 12 points, 36 weeks, "approaching an entire school year." Which, after what they've gone through, seems predictable.
Not good, but not twenty years. All the screaming is premature. Before the additional scores (8th grade) and details in October. Before students have a chance to fully readjust to in-person schooling and reinvigorate their intellectual lives. Before students have had the chance to right themselves before we've given them a chance to catch up. For many of them, before they've had a chance to fully mourn their personal losses.
Yet the drumbeat goes on. In the newspapers. Online. On the TV news. Among administrators. Sometimes friends and family shaking their heads and tsk tsking.
What can we do? I'm no longer on the job but when I was, I tried to follow this first rule of survival:
Do not fall for their bullshit. Because they tell you something is bad doesn't mean it is. Because they tell you scores are everything doesn't mean they are. Because they say your scores are down and you failed doesn't mean it's true.
First of all, they are not your scores. They are your students' scores. And it's only a test. You know your students, and you know how little of their intelligence and talent and humor is reflected in any single test score.
Nod your head, pretend to care, ignore the bs. Remember that most of the world doesn't understand your world, and 99% of the press reports and all the talk on the morning news is from people who either don't know anything or are deliberately lying for their own purposes.
That goes for the bosses, too. Don't expect your admin to honestly tell you why you're doing this test or that test or why you're spending so much time prepping for them. Don't break your head trying to figure them out. Remember that they have a different agenda. They are speaking a language different from yours. Your concerns are not their concerns. Their only honest answer to "why" is the same as yours, the same as your students': Because we're told to.
If you're in a position to resist, do so. Ask other, "non-why" questions--they hate that. Challenge their analysis of the scores. Their interpretation. With the rest of the world who are not your bosses, push back. At parties, with family, on social media. And write letters to editors; that's still a thing. Write a blog. You're an educator, let's educate the ones who don't know better and confront the ones who do.
You are right and they are wrong. Don't let the bastards grind you down.
*h/t Peter Greene
**h/t Steven Singer
