It. Doesn't. Work.*

*Unless by "work" you mean "advance The Big Project to Kill Public Education"

The road to public school privatization is paved with test scores. It's an industry hijacked-- some convincingly argue manufactured and developed--for the purpose of discrediting public schools so they lose students, become nonviable, and can be stripped down and sold for parts. 

<brief aside>

(At last report, Pio Pico Middle School here in Los Angeles was being "dissolved" to make way for a nearby charter school--one already co-located on the campus of Los Angeles High School but which apparently needs additional room to grow. Information has been hard to come by--looking at you, UTLA!--so if anybody out there knows what's going on over there on Arlington, please share in the chat. In the meantime, there's a petition at change.org if you're so inclined.)

Testing, particularly the state-sponsored standardized that is being carpet bombed onto schools all over Los Angeles, is the hired assassin of Privatization (alias "school choice"). The thing is, it's bullshit. The data produced by state-sponsored standardized testing--which is to say the test scores on which "school choice" depends--it's all bullshit. As @Liat_RO says in her retweet referring to a working paper from the Annenberg Institute at Brown University, the testing fails even on its own terms.  

“NO measurable impact”

Any teacher could have told you this, but still https://t.co/T3W91pRdU8

— Liat in BK (@Liat_RO) January 24, 2022

More on the researchers' findings later. First, some scenes from previous episodes. 

When it comes to The Big Project To Kill Public Education, whether it's the organizations that benefit directly--like the testing corporations, consulting firms, and charter school management companies--or it's the broader freemarket  community that has invested heavily in the demolition of a public institution so that corporations might live! (more on ALEC and their interest in the Project in a future post. For a taste, go here), underneath nearly all of the destruction is testing. 

The notion that every student has to be tested every year using state-sponsored standardized tests is relatively new--I used to peg it at the arrival of No Child Left Behind in 2002, but friends remind me that, as bad as George W. Bush was at everything, about this one thing he comes in second at least. Winner of the presidential "Terrible on Public Education" championship is none other than President Barack Obama, whose diabolically named "Race to the Top" extortion racket brought truckloads of cash to desperate states with desperate school districts all over the country. 

The nearly four-and-a-half billion dollar pot of gold seems quaint now, but it was a lot of money in those days. And all the districts had to do--all we had to do--was sell our souls, nod our heads, give a few tests, raise a few scores, and base personnel evaluations--and the paychecks that go along with them--on  those scores. Hatchet man Arne "never met a charter he didn't love" Duncan, was tasked with delivering the bad good news, and is reported to have been the happiest former Secretary of E____ (I can't even write it) when one Betsy DeVos replaced him on the "Worst Ever" list. 

In any event, this "Are we getting our money's worth?" commercial--dare I say capitalist--model of education probably really goes back at least to Ronald Reagan's "A Nation at Risk," because most of today's awful shit does, and it has grown and evolved and metastasized until it resembles the blob who ate the world. 

Well, all that growing and evolving doesn't happen on an empty stomach, so test scores. Production for use, as the old socialists use to say (ironic much?), and once you get test scores you have to use test scores. Where do you think "failing schools" come from, anyhoo?

The incessant state-sponsored standardized testing that compels schools to compete for resources and students, narrows the curriculum, powers the "failing schools" narrative, falsely encourages parents to seek alternative "choices," and demoralizes students and staff is a crucial weapon of public school mass destruction.

And for what?  Lots of different justifications are given for the testing mania.  "Identifying holes in instruction" is one. And then there's "promoting racial equality and ensuring support for all traditionally marginalized groups of students." I like that one and wish it were true. But truly bad ideas like today's testing boondoggle require swanky suites of jargon for protection.  

Meet the overarching rationales for the state tests: "objectivity," "comparability," and "accountability." It's nice that they rhyme, but let's take a closer look. 

We'll start with objectivity and its explicit distrust of educational professionals. Not satisfied that training and experience might prepare a teacher to evaluate a student's learning in good faith, proponents of these exams declare that they prevent biased grade distortion which could redound to the teacher's benefit. It's code for "teachers can't be trusted" and is an important tool in school destabilization. The punch line? Testing, of course, is itself not objective, as it privileges certain kinds of knowledge and narrow, prescribed modes of demonstrating "proficiency." (You might be surprised how the "proficiency" cut-off point was determined. Story for another time.)

Comparability. Another of the main rationales for the testing regime is to use the "objective" metric of test scores to compare schools across a district, a state, or the country. Set aside for a moment the fact that the test scores are not valid measures of learning but simply a snapshot of how well a student can recall a favored subset of "knowledge" under a particular set of circumstances. The real question is: Why compare them at all? How does it help a school or its students to know their scores are lower or higher than the scores of another school across town, or across the state or country? The answer is: It doesn't. We compare schools so parents will have a way to choose when it comes to school choice. You need a mechanism for elevating certain schools and discrediting others. This is it.

Of course, although the hucksters talk objectivity and comparability, these testing "ideas" are only meaningful as tools to achieve the principal goal of testing. Coming in at Number One on the sounds-like-it-ought-to-be good-but-it's-evil hit parade is everyone's favorite: Accountability, which is to say, evaluating and punishing schools and everyone in them. 

This kind of testing is terrible. It degrades and destabilizes public schools, putting them under constant stress with its voracious appetite for resources, its capacity to place administrators and teachers in public jeopardy and to demoralize students and staff, and its power to distort the curriculum. The Big Project to Kill Public Education depends on it.

That's true in general, but it's stupendously true when teachers are evaluated based on test scores. And I'm not even talking about the art teachers evaluated based on math and English scores, or teachers evaluated based on students they never had. Oh yeah, it happens. Teacher evaluation systems. Value-Added. Merit pay, hirings, and firings have all been based on the test scores of individual teachers. Which are not their test scores. They are their students' scores.

Today's focus is teacher evaluation and the premise that all this testing is necessary because you need a metric to compare schools and teachers. That way you can separate the good ones from the bad and incentivize the bad ones to get better or get out. 

And to some people who have no idea what teachers do or what schools are like, that can make a kind of soggy-brained sense. I mean, if teachers are the selfish, cynical half-humans they are advertised to be, then just base their evaluations and paychecks on their test scores (by which we mean their students' scores) and watch them jump! All you have to do is threaten them a little (or a lot) and they'll change their attitudes, get off their asses, and raise those test scores! By which we mean their students' scores. It just stands to reason! 

It. Doesn't. Work. High-stakes reforms tying teacher evaluations to their students' test scores do not raise test scores. Usually bullshit is detectable by the smell, but thanks to a group of researchers we have more objective evidence.  Introducing the star of today's episode:

From the Annenberg Institute at Brown University, this report from December of last year details the findings of Joshua Bleiberg, Eric Brunner, Erica Harbatkin, Matthew A. Kraft, and Matthew Springer as they examined, just like the title says: "The Effect of Teacher Evaluation on Achievement and Attainment: Evidence from Statewide Reforms."

Now, cutting to the chase for those of you so busy you can't believe you've read this far, from the Abstract:

We find that, on average, state teacher evaluation reforms had no discernable effect on student achievement in math or ELA.

And just in case you missed it, from the paper's Conclusion: 

We find that, on average, teacher evaluation reforms had no detectable effect on student achievement or attainment.

To quote Liat in BK, "Any teacher could have told you this...." To which I say, "Yes, teachers have been saying this for freakin ever, but who listens to us?" The paper is a pretty good one and I'll discuss it further below for those of you so inclined. But regardless, it gives us something tangible to point to and say, "See?" 

Whether that matters to anyone whose paycheck depends on evaluating teachers using their students' test scores is an open question. 

Continuing on. Some background from the paper:

Between 2009 and 2017, 44 states and Washington, D.C. implemented major reforms to their teacher evaluation systems. Prior to the reforms, teacher evaluation was largely a perfunctory exercise that resulted in nearly all teachers receiving satisfactory ratings (Weisberg et al. 2009). Strong incentives by the federal government helped spur the widespread reforms. The $4.35 billion federal Race to the Top (RTTT) grant competition incentivized states to reform evaluation systems by regularly evaluating teachers based on multiple measures (including student academic growth) and using performance ratings to inform personnel decisions.

This evaluation reform wasn't just a suggestion. 

The rapid uptake of teacher evaluation reforms came, in part, as a response to President Obama’s RTTT program and its offer of large competitive grants to states that were struggling during the Great Recession (Bleiberg and Harbatkin 2020; Howell and Magazinnik 2017). In particular, the application rubric for RTTT rewarded states for using student outcomes to evaluate teachers and inform personnel decisions with evaluation ratings. (emphasis mine)

So the new study shows "null effects on achievement." Case closed, right? Not so fast, my friends. See, as we've discussed, the objective isn't to identify and nurture an army of fabulous instructors but rather to enfeeble the system to the point of defenselessness. And the Privatizers and Profiteers who promote this stuff and profit from it will not take "no" for an answer. 

So they look for the answers they want. They think they've found one in the infamous Michelle Rhee D.C. Public Schools brainchild IMPACT, and in fact, there have been some desired outcomes, by which I mean test scores. Test scores did move but, and there's always a but, the changes were generally attributed to the release or resignation of teachers deemed to be "ineffective" and replacement with new, presumably more "effective" teachers. Or at least teachers who are more test-focused. 

Easy, right? Disturbingly, from the same paper:

Similarly, evidence from a national study of teacher evaluation reforms found that these reforms increased the number of new teaching candidates who had attended more competitive undergraduate institutions but also decreased the overall supply of teaching candidates (Kraft et al. 2020). (emphasis mine)

This is not good. If there's one thing we know about schooling in a pandemic, it's that staffing is going to be a problem. Thanks to decades of depredation and character assassination, people are leaving in numbers, and replacing experienced teachers with bus drivers and the National Guard is unlikely to have positive effects on student outcomes. We'll have to wait for that study. 

In the meantime, IMPACT aside, the paper warns that the results of studies looking at teacher evaluation system reforms similar to those that have been "adopted at scale nationally" are "decidedly mixed."

Furthermore, even in districts that claim to have had success with "evaluation reform" (and when you hear the word "reform," you should always take a look over your shoulder), have done so with programs that offer much more than threats based on insufficient "value added."

Again, from Bleiberg, Brunner, Harbatkin, Kraft, and Springer:

Several quasi-experimental and experimental studies in large urban school districts point to the potential for evaluation systems to serve as engines for professional growth. Taylor and Tyler (2012) studied Cincinnati Public School’s peer evaluation and feedback system. They found that being observed and evaluated by experienced, expert teachers and school principals improved teachers’ ability to raise student achievement in math but did not affect ELA achievement. A similar study of France's national teacher evaluation system found that high-stakes observation and feedback by certified pedagogical inspectors improved teachers’ contributions to student achievement (Briole and Maurin 2020). (emphasis mine)

For evaluation reform to be authentically effective and not just bogus advertising, it takes tons and tons of resources. Lots of observations, lots of meetings about observations, lots of feedback about observations, all done by trained, certified experts in teaching. No teacher who has been evaluated in the last thirty years would waste ten minutes of their lives waiting for that to happen. 

More from the same paper:

The evaluation process itself may support ongoing improvements in teachers’ practice if evaluators provide feedback and coaching, prompt teachers to reflect on their practices, or provide data that allow districts to match teachers with targeted professional development (Donaldson 2020; Donaldson and Firestone 2021; Galey-Horn and Woulfin 2021; Mintrop and Trujillo 2007; Springer 2010; Woulfin and Rigby 2017).

No shit. Observation and feedback. As Liat in BK might say, any teacher could have told you that. 

And one more thing. "Evaluation Reform" also requires lots and lots of money for bribes incentives. Because that has worked out so well in Georgia, Indiana, Massachusetts, Nevada, and Virginia. At least. It's not that hard to raise test scores, if all you want is to raise test scores.

There are some problems with the paper. Start with the fact that, aside from a brief nod to graduation rates, "student outcomes" means what it always means: test scores. Until we force a broader, more humane set of metrics, the battle against the commercial model of schooling cannot be won. This paper, though, is a significant addition to our arsenal.  

The report's most serious failing, however, is that there are thousands of words about teachers and test scores, but very few about actual students. Maybe the authors figured that those considerations don't belong here. But then where do they belong?

I've been thinking a lot about that, so I decided to include here a short excerpt from my unpublished book, Answer Key. Based, as they say, on too many true stories:

Just imagine being one of those students--one of the ones you know quite well because, you know, you teach them every day. And imagine that you have been taking tests for what seems like your entire school life and that these tests are calibrated to yield a certain number of “not proficient”s every year, and that for as long as you can remember you have been one of them.

And even though you are “not proficient,” you have been going through school and learning stuff and passing your classes and doing pretty well, but every time you think you know something the test gets harder and you are once again one of the “not proficient”s. And imagine you don’t even find that out until the next year when there’s nothing you can do about it.

It must really suck, right? And so after having been battered and beaten with a number two pencil for long enough, a lot of those kids get discouraged and give up. Not all of them, but certainly enough for us to ask, “What significant and actionable information are we collecting by putting kids and parents and teachers and schools and districts through all this every year and sometimes every month? Why are we even doing this?”

As I've written before and will definitely be writing again, the scores on state-sponsored standardized tests are not meaningful for students or for teachers. The testing causes tremendous disruption--not in a good way--and it doesn't even deliver on its own terms. And the arguments for the testing regime are specious, because the people and organizations making those arguments know the data is not meaningful. 

Finally, a note regarding new LAUSD Superintendent Alberto Carvalho. In support of his school choice agenda, Mr. Carvalho is extremely fond of repeating: "One size fits none." It will be interesting to see if he applies that same principle to the district's standardized testing obsession.