Listen to "The Sky is Falling. Again. Test Score Lunacy. Again."

The Sky is Falling. Again. Test Score Lunacy. Again.

More comprehensive NAEP data dropped over the weekend and it gave the haters and weak-minded another chance to gasp and point fingers. In Los Angeles, as in other places I'm sure, it was also an opportunity to amplify the recently-released state scores on the mordantly-named Smarter Balanced assessments. Never ones to waste an opportunity to advance the "failing schools" narrative, the usual suspects piled on. 

I've already done the NAEP thing. Here are a few words on the SBAC scores. 

The rhetoric has been typically overheated. The Los Angeles Times predictably howled over "deep pandemic setbacks" and, credited LAUSD Superintendent and world-class suit-wearer Alberto Carvalho with promoting the tired canard that "five years of gradual academic progress in the nation’s second-largest school district have been reversed." 

Newsflash: Nothing has been reversed. Not one student knows less today than they knew five years ago. Not one. Test scores are lower than they were a minute ago (see: Pandemic!). That's it.

The release of NAEP data gave the Times a chance to return to SBAC and turn up the scare quotient, as "setbacks" became "drops," and results were described as "devastating" and "dismal." As with most things LA Timesian, this particular article settles comfortably into anxiety over how much this is all going to cost, reminding readers of "the multibillion-dollar investments in public education" California is making in the effort to get everybody back up to speed, and presumably warning them to be sure and get their money's worth.

Other outlets had similarly breathless takes. EdSource had "2022 California standardized test results wipe out years of steady progress." CBS News Sacramento had the mystically incongruous "Smarter Balanced test scores reveal California students doing worse in school." Even the freakin New York Times got into the California bashing biz, but with extra spice. With a headline that could have come direct via @DougJBalloon, they weighed in with the following:

I went to the state website to see what all the noise was about. I spent about a half hour on it and found it to be confusing and missing important features that could have made comparisons easy. I guess the point is to make interpretation of the scores difficult and time-consuming. That way, interpretation becomes the province of interpreters as they tell us what the scores mean.

Even so, from what I could gather, the number of eighth grade students either meeting or exceeding the ELA "standard" in 2018-2019 was 49.41%. In 2021-2022 it was 46.64%. That is to say, a drop of less than 3% of students. Math is certainly a challenge, as 2018-2019 meet+exceed was 36.63% and in 2021-2022 it was 29.24%. However, digging in a bit further, we find that in all three categories of math: Concepts & Procedures, Problem Solving and Modeling & Data Analysis, and Communicating Reasoning, the combined percentages of students achieving scores of "Near" and "Above Standard" actually went UP(!) anywhere from 3-5 % in each category. 

That's correct. Even though the percentage of students scoring "Standard Not Met" for overall Achievement Level went from 40.78% to 48.02% (resulting in the big drop), student performance in the three domains actually improved. The only explanation I can come up with--other than the tests are bogus--is that the low scores were significantly lower than in 2019.

I've written about test score "data" before and, because I can't help myself, I'm sure I will again. But it almost feels like the things I'm writing about everyone knows already. Or would, if they cared to. It's almost as if these media outlets and think tanks and advocacy organizations are not interpreting the scores in good faith. As if they have a whole different agenda. 

How many different ways can someone write "Test scores went down during the pandemic"? Is there anyone who doesn't know this already? Is anyone surprised by this? We don't have data yet for test scores and school closings. We don't have data for test score (I won't write learning loss) recovery. We really don't know anything except that test scores when down. 

Pandemic!

But that won't stop the interpreters from spinning the results to support their preferred assumptions, and it won't stop the haters from making shit up to serve their purposes. That's okay. Just don’t listen to them if you can help it, and whatever you do, don't believe them.

You all are a tough bunch. Don't let the bastards grind you down.

Listen to "NAEP? Nope." The Test Score Tango.

NAEP? NOPE. - Seemed like a good day to replay this one. From Sept. 19 this year.

 The Test Score Tango.

Preface: You may already know how I feel about Big Testing and test scores. If not and you're interested, I've written about it here, and here, and here, and here. That said, the least terrible of the Big Tests is probably the National Assessment of Educational Progress, or NAEP, if only because you don't have to shut down an entire school every year to give it, and because it doesn't sort the high-score schools from "failing" ones. 

Yet, when the latest NAEP scores dropped a couple weeks ago the usual suspects were predictably aghast. Hilariously, some of the same dopes that declared the teacher shortage a fake now credulously announced the loss of twenty years of progress based on one set of scores on one test given to nine-year-olds in the third year of a fucking pandemic. 

Kids, there are demons out there bent on the destruction of public schools, and they'll do whatever it takes to make it happen. When I was in the classroom, it felt like I was in the middle of one big chaotic shitstorm. It may feel that way to you, too. But it is not. It is strategy. It's a program. Think of it that way and it's easier to recognize. You just have to know what to look for. Then maybe we can fight back. 

And now, the Test Score Tango.

A lot of the early NAEP score headlines were overwrought and eerily similar. The usual suspects, of course. The New York Times had "The Pandemic Erased Two Decades of Progress in Math and Reading." But also from outlets that should know better. From Chalkbeat: "Math and reading scores plummet on national test, erasing 20 years of progress." And of course piling on from The74: "‘Nation’s Report Card’: Two Decades of Growth Wiped Out by Two Years of Pandemic" (all emphasis mine). A little later The New Yorker had this terrible "interview" full of leading questions and reformster* rhetoric.

Getting out early with sensational claims helps shape the narrative. Fortunately, several observers responded quickly with a more measured, nuanced analysis than the doomsayers. This article from Education Week does a good job explaining what the scores actually mean. Jan Resseger    situates the test results within the broader context of the trauma and disruption caused by the pandemic, and suggests the limits on conclusions that may be drawn from them. And this thoughtful article from Jill Barshay titled, "6 Questions to Better Understand Math and Reading Scores" asks really good questions and doesn't pretend to have all the answers. In advance. 

Leonie Haimson of classsizematters.org stresses the lack of clarity in the data and pushes back on the popular assertion that closing schools and the consequent remote learning were a key driver in the score decline. 

Haimson also challenges the conventional proposals for what administrators and education Big Thinkers call "learning recovery." Addressing "negative impacts on learning and test scores," Haimson writes, will require not simply "longer days and years." Instead, she advocates for smaller class sizes rather than what she calls "false and damaging policy prescriptions."

It's difficult to know whether all the hyperventilation comes from genuine misunderstanding, or underlying partisanship, or if it's just for clicks, but it is surely... overblown.

Many of the agitated commenters were no doubt expecting--and knew their readers (and funders) were expecting--the pandemic to have a severe impact on student learning. And many remain particularly focused on the closing of school buildings and the shift to remote learning. Many expected those effects to have been disastrous, and they are heavily invested in blaming teachers and their unions for the damage. When the scores came out, it was easy to read them as confirmation of their expectations. 

But the truth is, in the best of times the NAEP scores don't mean what a lot of people think they mean or would like them to mean. And in pandemic times, it's even harder to draw conclusions.  

One of the things about NAEP that doesn't mean what people think it means is "grade level." In 2016's "The NAEP proficiency myth" from Brookings, Tom Loveless demolishes the notion that NAEP proficiency is "synonymous with grade level" and urges states not to use NAEP’s proficient level as a basis for education policy. 

Some interested parties--including organizations promoting a "failing schools" narrative to advocate for school choice--continue to deliberately misinterpret the NAEP levels and conflate "proficient" with grade level. By this definition, then, in a given year two-thirds! of test takers might fall short of "grade level." 

The truth is, however, that what most of us think of as "grade level" more closely aligns to NAEP's Basic level. By simply being honest about the meaning of the NAEP scores, instead of two-thirds below grade level we in fact have two-thirds at or above. The sky may be a bit cloudy, but it's not falling.

The Covid pandemic has obviously had a profound impact on students, and the new NAEP scores clearly reflect that.    However, in an example of the scores not meaning what some people would like them to mean, several partisan observers have used the scores to criticize the ways schooling was conducted during the pandemic. In doing so, they either misconstrue or ignore the data completely.

At the center of the project to blame teachers and unions for the drop in scores is the issue of school closings and the reliance on remote learning. Lots of interested parties have passed around the same study to show a link to learning loss. The NAEP scores themselves reveal a more complicated picture.  

For example, as Haimson and others have noted, the evidence for a correlation between school closings/remote learning and the drop in test scores is decidedly inconclusive. And even the NCES itself warns us that "Users are cautioned against interpreting NAEP results as implying causal relations." As Peter Greene put it in 2019: "Everyone ignores that advice, but NAEP clearly acknowledges that there are too many factors at play here to focus on any single one."

What can we know based on this limited set of data? As author and former writing teacher John Warner describes it in a NAEP discussion with Nick Covington of the Human Restoration Project: We know "a big thing happened." And "not only was school disrupted, the entire lives of everybody in the country and the world was disrupted" and "the notion that we could have avoided this with some kind of different school or different choices, going back in person sooner... it's not reflected in any of the data." (7:50)

Although the impact of the pandemic and the disruption and trauma that have accompanied it are clear and universal, the impact on student learning of going remote is, as yet, unclear. Facts are stubborn things.

Another thing we can learn from NAEP concerns the testing regime spawned by No Child Left Behind. If you assume that the promoters of NCLB were acting in good faith and not simply trying to inject an air bubble into the bloodstream of public schooling, the testing regime it initiated (not NAEP, but the hundreds of other tests kids have to take every year) hasn't succeeded on its own terms. Rather than more and more testing leading to a rise in scores, with the exception of a limited burst in the early 2000s, the NAEP scores have flattened. Big Testing hasn't worked. 

Those are things we can legitimately infer from the limited NAEP data we have right now. But what about "Two Decades of Growth Wiped Out!"? It would be easy to dismiss it as professional alarmism, but in this case the stakes are too high to simply chalk it up to selling papers. 

There are demons out there, and they are perched, waiting to jump to conclusions and manufacture analyses that serve the project to dismantle authentic** public schooling. They can't wait even a few weeks until more complete data is available. 

As Jill Barshay writes, "A more detailed report from the Department of Education on student achievement during the pandemic is expected in October. It will list state achievement scores for fourth and eighth graders on another NAEP test. Hopefully, we can unravel more of these knots together."    Maybe that's what the vultures are afraid of.

In this environment, information like the NAEP scores is routinely politicized. The scores are used to sell programs and products from organizations that profit from brokenness. And of course, the scores are cynically weaponized by the school choicers and their enablers in the press in order to promote a failing schools narrative and advance the choice and anti-union agenda.  

But the scores do not mean what the screamers say they mean. In order to turn the volume up to 11, they compare the raw scores to previous scores, find a match twenty years ago, and announce twenty years of progress loss. The thing is, it's twenty years of test score progress, not twenty years of student progress. 

In an interview conducted by Liz Mineo for the Harvard Gazette, Dr. Andrew Ho, professor at the Harvard Graduate School of Education and former member of the NAEP governing board, explains:

[E]ach point on the NAEP scale is roughly three weeks of an academic year, and the overall decline was seven points, which is roughly five months in terms of an academic school year of learning that this cohort of students is relatively behind.

Five months. Not twenty years. But the average overall decline doesn't tell the whole story, of course. Predictably, students who were most challenged before the pandemic were most challenged by the pandemic. For students who were already struggling, who were already coping with insufficient resources--including educational resources--and whose families were likely to be most severely impacted by the pandemic, results were more alarming. From Dr. Ho:

But this average betrays the inequality where the decline for the higher-scoring students was only three points, or nine weeks, whereas the decline for the lowest-scoring students was 12 points, which is 36 weeks, which is almost approaching an entire academic year. 

Terrible. Unjust. Unacceptable. We need to address this, but this is not just a test score. This is a reflection of the racism and resource inequality that permeates our system. And still, the most impacted students lost "almost approaching an entire academic year." NOT twenty. 

The kids being told right now that the scores are down and the pandemic has cost them twenty years are being crushed for the objectives of the crushers.

Dr. Ho goes on to discuss the broader impact of the pandemic on students' lives apart from one set of test scores and reminds us what returning to "normal" means for students getting back to school. He also suggests the need for a "learning infrastructure," perhaps in the form of a "National Assessment of Educational Equity." In his words:

I’m not just worried about academic learning. I’m worried about the structures of educational opportunity and inequality that have increased over the pandemic; this is just the tip of the iceberg. I hope we can pay attention to the entire structure of educational opportunity. Academic learning outcomes are easy to measure, but also easy to overemphasize, as if students are only trying to get better at math and reading and not also trying to reconnect with their classmates and their teachers and remember how to sit still, listen, learn, and enjoy learning and playing with others.

Again, even among the most severely impacted, most vulnerable students, the setback was 12 points, 36 weeks, "approaching an entire school year." Which, after what they've gone through, seems predictable.

Not good, but not twenty years. All the screaming is premature. Before the additional scores (8th grade) and details in October. Before students have a chance to fully readjust to in-person schooling and reinvigorate their intellectual lives. Before students have had the chance to right themselves before we've given them a chance to catch up. For many of them, before they've had a chance to fully mourn their personal losses. 

Yet the drumbeat goes on. In the newspapers. Online. On the TV news. Among administrators. Sometimes friends and family shaking their heads and tsk tsking.

What can we do? I'm no longer on the job but when I was, I tried to follow this first rule of survival:

Do not fall for their bullshit. Because they tell you something is bad doesn't mean it is. Because they tell you scores are everything doesn't mean they are. Because they say your scores are down and you failed doesn't mean it's true. 

First of all, they are not your scores. They are your students' scores. And it's only a test. You know your students, and you know how little of their intelligence and talent and humor is reflected in any single test score. 

Nod your head, pretend to care, ignore the bs. Remember that most of the world doesn't understand your world, and 99% of the press reports and all the talk on the morning news is from people who either don't know anything or are deliberately lying for their own purposes. 

That goes for the bosses, too. Don't expect your admin to honestly tell you why you're doing this test or that test or why you're spending so much time prepping for them. Don't break your head trying to figure them out. Remember that they have a different agenda. They are speaking a language different from yours. Your concerns are not their concerns. Their only honest answer to "why" is the same as yours, the same as your students': Because we're told to. 

If you're in a position to resist, do so. Ask other,  "non-why" questions--they hate that. Challenge their analysis of the scores. Their interpretation. With the rest of the world who are not your bosses, push back. At parties, with family, on social media. And write letters to editors; that's still a thing. Write a blog. You're an educator, let's educate the ones who don't know better and confront the ones who do.

You are right and they are wrong. Don't let the bastards grind you down.

*h/t Peter Greene

**h/t Steven Singer

 

On a related note, the vilification of school teachers by conservatives and some centrists, the same people who not only teach and educate our kids, but also now lay down their lives for our kids!, is a truly sickening and despicable development.

— Mehdi Hasan (@mehdirhasan) October 25, 2022

One-Room Schoolhouse

One-Room Schoolhouse

“Fernando.”

    Fernando looks at his friend Dahlia, who nods. He then rises grinning from his seat and bounces toward the teacher table at the front of the room. I hand him his book and, after giving it a cursory glance, Fernando twirls and retraces his steps. His friends giggle, and so does he.
    Between handing out books 17 and 18, I take a look around the room. Fernando is talking with his friends Irma and Jasmine. He has already put the book away in his backpack, not moved by the fact that there are still eighty-five minutes left in our block period. His friends don’t stop him and don’t ask to see the book. Unlike Fernando, they can read, but not in English. The same is true of Song, who arrived from Korea four months ago and seems most interested in the size of the book. He pretends to weigh it in his hands as he says something to Kim, a friend he has made in the class. They both laugh broadly.
    That’s the predictable, even easy part of the story. Now I look to my right and see Tina, focused and already reading ardently. Behind her, Juan is also eagerly chewing through hunks of text, only he’s reading out loud to his friends Milton and Rudy. His mouth seems to taste the words as it forms them, measured, deliberate. In the front corner of the room, Jose sits and stares, first at the cover of the book as he traces the drawing with his finger, then out the window.
    The thirty-two students gathered together in this ninth grade English class are so diverse an observer might fairly ask if there’s been a mistake. I have students from fourteen years old to eighteen, some brand new to the country, some born within blocks of the school. They have English reading levels from below the first grade to above the eleventh. To friends and family I’ve described it as anything from a balancing act to a grand pageant, from a juggling act to just plain impossible. A hundred years ago, students of all ages and skill levels in a community might be taught in one room by a single teacher. They called it a one-room schoolhouse.

I wrote that ten years ago. 

I had just moved from the school where I had taught for fifteen years to a brand new "pilot school" campus and we were creating a school where none had been. When we arrived we had to invent systems and structures that would serve our students and remain true to our mission of social justice and environmental responsibility. And because we were a k-12 span school, we had to do it for elementary, middle, and high school kids from 5 to 19 years old.

For those of you who don't know, pilot schools, established in 2007 through a side agreement between the Los Angeles Unified School District and United Teachers Los Angeles, are essentially the district's answer to the charter schools that have been siphoning off students and resources from authentic public schools for years. 

There is a much more fulsome interrogation of charter schools in a near-future post, but briefly: From the CDE website, "In 1992, California became the second state in the nation to adopt public charter school legislation." That measure was followed with additional legislation dissolving limits on charter schools and forcing districts and even existing neighborhood schools to divert more and more resources--up to and including giving away the public school's actual facilities--to support the charters with which they were now forced to compete for students. 

The district's responses to this existential threat have been reliably hapless and politically self-serving. Of course there was the capitulation to the testing-industrial complex that grew out of No Child Left Behind and its evil progeny. Once fully addicted to the poison pill of competition and synthetic accountability, the district resorted to mimicking its tormentors, adopting the assumptions and solutions of the school choicers and internalizing them.

So, rather than focus on the contributions of authentic neighborhood schools, their value to their communites, and the importance of strengthening them, the district instead opted for sham self-determination (school-based management, expanded school-based management, something called an iDesign Schools Division), or offered its own recipe for school choice (Public School School Choice and Open Enrollment).

In fact, LAUSD itself  got on the charter train almost immediately, with one superintendent even floating the idea of a charter district of LAUSD's very own. 

The pilot school model is simply one more answer to the wrong question. Not "How do we best serve our students, our communities, and our world?" But rather, "How do we  compete?" 

Because they are advertised as having several "autonomies" that allow them to operate independent of the school district, today's pilot schools are peddled as alternatives both to charter schools and to traditional schools starved of resources and trapped in a bureaucracy more concerned with scores than students. 

These autonomies include decisions with respect to testing and curriculum, budget, governance, and the school calendar. Most critically, the school--and by school, I mean principal--maintains control over staffing. The stated objective is to bring together like-minded individuals committed to a shared mission. One Elect-to-Work agreement reads: "The Principal may release staff members that are unwilling to support and implement the vision and mission" of the school. 

Theoretically, this independence facilitates the innovation promised by charter schools concerned with being more efficient--and more segregated--versions of public schools.

We're a pilot school! We're not like those other public schools. We have autonomies! We have a theme! We're small for personalized instruction! We can get rid of those troublesome teachers who won't get with the program or who--gasp!--don't believe in your kids.  

Sadly, in our school we quickly found limits we didn't anticipate. In spite of our outwardly posh surroundings, we were basically a bare-bones operation. We shared the campus with five other schools and both our staffing and our physical space were severely limited. Our middle/high school counselor had to keep track of about six hundred students. There were classes we couldn't offer because there were not enough classrooms. We were denied access to certain spaces--the freakin brand new  theatre, for example--and even our lunch times were subject to complicated negotiations. 

We had no teachers or classrooms to create an infrastructure for language learners, so even newcomers were programmed into traditional English classes, often with no additional support. And on the first day of school we found out we were a "full inclusion" school and students with special needs were immersed in the standard program, even if their previous experience was in a special day class. We were told this was in keeping with our "social justice" mission. 

In both these cases, it was clear to me that our model was not a sufficient response to the needs of our students. I didn't really understand the scope of the war being waged against public schools. As I've written, I was siloed in the day to day struggle of trying to serve my students and stay healthy. 

Even then, though, I could see that what we were doing--the posing, the authoritarian professional developments and prescriptive procedures, the minimal resources and skeleton crew, the disappearing of teachers who spoke out, and the compulsive testing--was not school. It was, rather, a reaction to the Walmartization* of schooling driven by the privatizers and school choicers in general and the charter school movement in particular. 

*Found here, and later here, and by Diane Ravitch here.

Charter schools have long benefited from a gigantic public relations/advocacy machine that has been very successful at shaping public opinion, intimidating dissenters, and influencing public officials--including district officials--to support them. Unfortunately, this pilot school pretend-charter model was a symptom, not a solution. 

However, saying so, and speaking up about the many ways we were falling short, did not make me a favorite with administration.

Pro tip: When your principal is assigned and paid by the district and exercises total authority over staffing (there's autonomy, for you), there's no such thing as independence from the district. 

The district's response to the enemies of public schooling was anemic and wrong-headed, but it was not entirely cynical. We wanted to be better. But we should not have wanted to be like them. 

So it was in this context that I was sitting at a table in my one-room schoolhouse, handing out books to the most eclectic group of students I had faced in my career. I was also trying to understand my school and district and the many ways forces outside my classroom were becoming more and more powerfully influential in the way I conducted my interactions with kids. I was very frustrated.

More of that frustration in the next post when I'll share what I thought about it all then, what I think about it now, and some of what I've been able to see from outside the silo. 

See you next time.

Jack Schneider

In place of a post, I'm sharing these two fantastic twitter threads, one on the "unstated theory of change" underlying recent "dominant approaches to education reform" and one pulling back the curtain on vouchers.

If you are not familiar with the work of Jack Schneider    @Edu_Historian, do yourself a favor. This is his website at the University of Massachusetts Lowell which has links to the awesome podcast he does with Jennifer Berkshire @BisforBerkshire. 

Together they co-authored A Wolf at the Schoolhouse Door: The Dismantling of Public Education and the Future of School. 

You can also get to the podcast here.

The three dominant approaches to education reform of the past quarter-century were all rooted in a flawed theory.

Here's what that tells us about supporting school improvement in the years to come...

🧵

— Jack Schneider (@Edu_Historian) October 19, 2022

And because this one is great, too:

 A thought about vouchers and what they're really about...

1/

— Jack Schneider (@Edu_Historian) October 13, 2022 

Listen to "LAUSD Up -- Periodic Reporting on the Los Angeles Unified School District"