Okay. One more.

Test Scores are bullshit. Secret data is evil bullshit.

Peter Greene @palan57,  Senior Education Contributor for Forbes and author of the excellent CURMUDGIFICATION CURMUDGUCATION (updated and corrected) blog, has a new article in Forbes that describes legislation now before Congress. The bill, whose title is long enough to look like satire but just long enough to be acronymically shortened to "the America COMPETES Act," contains a "College Transparency" amendment introduced by Rep. Andy Levin (D-MI). 

Aimed at supporting "'the next generation of American workers,'" according to Levin, the amendment means to help "'our young people transition seamlessly into good-paying careers is to make sure that high-quality education and job training is affordable and accessible to all.'" 

What does the bill do? From Greene's article:

It requires the government to collect student-level data reporting enrollment, persistence, retention, transfer, and completion measures for all credentials, plus any other data the advisory committee deems necessary.

The bill also requires the collection of “post-completion outcomes for all postsecondary students, including earnings, employment, and further education” both immediately after leaving postsecondary education and “at time intervals appropriate to the credential sought and earned.” That data collection is to be periodically checked against data held by other agencies, such as Defense and the Social Security Administration. But this calls for lifetime data collection of anyone who attended a postsecondary institution.

Dressed up as a way to help the U.S. compete economically around the world, the legislation, the article notes that the law ensures that if you "Go to college or any other post-high school institution... the federal government will watch you for the rest of your life." It is, as Greene puts it, "a plan for lifetime data tracking of college students."

Is this what we want? To cavalierly feed students to the machine?  Affected students might be forgiven for one day shaking their heads and lamenting, like the miners in Matewan, "We've had about all the help we can stand." (h/t John Sayles) Better yet, "We asked for help, and this is what you give us?"

The truth is, that's not the half of it. Data collection and student tracking is being done right now in public schools all the way down into elementary. It's secret--or at least not discussed in polite company among educators who don't want to acknowledge the truth. And it's done by the government and government-sponsored entities. 

My experience, from Answer Key:

This is so bad I suppressed my own awareness of it and my own complicity in its collection. Thank you to my friend Mike for reminding me. 

In today’s Los Angeles Times there is an Op-Ed written by Caleb Scharf, Columbia University’s director of astrobiology. Just the title creeps me out: “We are our data, our data are us.” In the article, Scharf compares the “informational ocean” we’re all swimming in to a living system, and even though he asserts that this “externally held information” (not in our DNA) is nothing new, I still find myself terrified.

But that’s not the problem. We are all undeniably swimming in and under this Panthalassa of bytes of data, and some of that data will be harvested and used for purposes good and ill. However, schools should not be collecting student information for use by other entities. And they are.

Attendance rates and test scores are easily identifiable examples of student information collected by schools. However, thanks in part to stressed resources, lots of schools have accepted the help of outside vendors in order to provide services that in a different universe might have been understood to be the natural province of the schools themselves. Not-for-profit organizations and specialists have set up shop on campuses all over the country and collect student information in a variety of ways including surveys and having kids fill out forms in order to participate in the vendor’s program. Much of this data is then shared under the rubric of “scholarship program” or “college outreach.” Don’t forget to check the box!

Even the laptops I passed out in class for our research project were little data dynamos. Of course, the district monitored every website visited by every student, but so did The Google. And no doubt so did Microsoft and Amazon. Every Yahoo News click was another bitbyte in some kid’s digital profile. I suppose someone could argue that everybody knows that everybody knows everything anyway, and that kids are savvy and experienced digital consumers and fully informed of the risks and exposure, but nobody says that about kids and drugs, and nobody who knows anything about kids would say that anyway.

No matter what, by permitting ancillary entities to occupy campuses and operate this way, and by distributing and utilizing our own technology without sufficient regard to the implications, schools effectively sanction these practices. And whether it’s browsing histories from school laptops, or household education, family size, and even income information though a “We’re here to help” not-for-profit, this covert data collection should be illegal and is just another example of why data is evil bullshit.

What am I supposed to do about it? Resist. Like with many of the institutional outrages described in this book, a teacher’s capacity to curb this abuse is limited.  However, I urge you to resist. Make sure your students are thoroughly informed of their options “Don’t check that box!” including their option to withhold their participation. Remind them that sharing their information, even for school, even on a school computer, has risks and help them protect themselves from those risks. Don’t be thinking years from now that you could and should have done more. Don’t be me.

We are collecting data on students, sometimes little kids, right now--data that will be attached to them forever. We are allowing district-affiliated tech platforms like Schoology and Naviance, and teacher favorites such as Google Classroom and ClassDojo and Kahoot!--and about a bazillion others--to collect student data. We utilize public platforms like Zoom and YouTube and TikTok, and every one of them tracks kids. Every administrator, every superintendent, every teacher knows it. We are complicit.

What choice do we have? we say to ourselves. Public schools are starved for resources. Education Technology companies offer "solutions." And besides, the kids like it. Yes, for the children. And finally: It doesn't matter anyway. The world is a digital matrix and we are our data. That may very well be true. But do schools have to be instruments of collection?  

Everyone knows that when it comes to Big Data, the consumer is the product. In schools, without asking or even honestly discussing it, we are selling our students. 

Data (scores from state-sponsored standardized tests) is bullshit. Part Three, part two

Are we there yet?

So, like I said, in my continuing education I've discovered a lot of people who can and have discussed and argued the pointlessness and destructiveness of the present testing regime. Their outstanding work includes academic studies and personal narratives and lots of stuff in between. I continue to work my way through them.

In the meantime, what can I offer? I can share my own experience and let readers decide what is similar and what is different to their own. I can talk to you and to other teachers like I've been there because I have. I know what you're going through because I went through it, and I came out the other side healthy, happy, and retired.

So here's one final, personal rant on data. I have other stuff to say, after all.

This, from Answer Key:

Data is bullshit, and professional developments about data are nuclear powered bullshit and tough to take without going completely insane. Besides all the philosophical reasons for opposing the toxic testing-industrial complex, and all the moral reasons for fighting against the reduction of human beings to data points, the numbers are phony. It’s all a fraud. The data do not mean what they say the data mean.

Now if you are tasked with administering these tests and you happen to be arithmophobic (an arithmophobiac?), you may not notice or want to, but the tests do not measure what they purport to. They are not reliable. They are not valid. First of all, if you’ve ever given a ten question multiple choice quiz to your students, you will know that even if you test exactly the same material again a week later, sometimes the same student will score higher and sometimes their score will go down. Shit, if you've ever taken an "ideal mate" quiz online--I almost wrote in a magazine, which tells you how old I am--you know that one day you'll wind up with Barack Obama and the next week you'll be yearning for Courtney Love.

And before you start sputtering about the difference between questions of preference vs. matters of fact, take a look at the questions students are asked on these exams--not that you're allowed to talk about them. There may be clear answers in algebra, but the relative strength or weakness of evidence, or of an opening sentence, or the interpretation of an extended metaphor are subjective and open to...well...interpretation.

Performance on a task including an examination depends on any number of factors including most recent experience, blood sugar level, and the quantity and intensity of available distractions. That is why no teacher worth the name would ever rely on a single assessment given a single time to evaluate a student in any significant way. We give tests, but we also give projects and writing assignments short and long, formal and informal. We assign debates, we listen to conversations and questions and we try to get a comprehensive, meaningful picture of what each student knows and can do along with some idea of what they need--particularly from us--in order to be able to know and do more.

In other words, it's the opposite of the testing-industrial imperative. And the opposite of what most people who are not teachers mean when they say "data."

And it’s even worse than that. In addition to not measuring what a student has learned, the tests don’t deliver even within their own narrow objectives. They do not measure what they say they do.

I’ll give you an example: My school hired a group of education consultants to boost our test scores. (Hilariously, we were not allowed to acknowledge that’s what we were doing.) We had days and days and days of meetings with these “experts”--only the math and English departments of course, because (sotto voce) That’s what we test. Meanwhile, everyone else could work on what they needed to do to prepare for actual teaching.

And when we were through with these literal weeks of PDs, we had created five exams to give in each class, one every five weeks and each test would assess five distinct standards representing five discrete skills. That’s five, then five weeks later a different five, then a different five, until we get to twenty-five. Why twenty-five? They’re the ones that occur most frequently on the Big Test in the spring.

And so, after each test we would look at how dismal our scores were and compare them to the previous test to see if they went up or down. In a PD about *data* we would assemble in the elementary cafeteria and show each other our dismal scores (our students’ scores) and make up a story about why our scores might have gone down or up, and we’d make posters and cut out student names and put up the posters around the cafeteria and go one-by-one around the room and tell our stories and answer for our sins. That is, those of us in the math and English departments. All others adjourn to your rooms to work on teaching stuff.

The tests were all different and tested different standards and skills. We were told we could not test the same things twice to see if we had made progress. We compared scores from different kinds of tests and had to pretend that the improvement or decline from one test to the next meant something. “Wow, those kids really got it this time.” “Oh (downward inflection), that’s disappointing. What did you do differently?” “I taught different stuff!” And we did this for years.

I tried several times during the early days of this catastrophe to point out that we were not actually measuring progress or the lack of it. “It’s apples to orangutans!” “Five answers to five questions is not a valid measure of proficiency!” “Who chose four correct answers as the benchmark for proficient? And why?” “Skills? Standards? I looked at the data and my student scores precisely tracked their reading levels. Aren’t we really just assessing their reading?” I was... unheard.

And there’s more. We were required to give the exam--in English--to every single student. Boost scores? Shhh! Yet we were required to prep for and administer the exams to seniors who would never be taking another test in high school. Our ELD teacher was forced to give the exam--in English--to newcomers to the U.S. who had been in country for a week and as yet spoke zero English. For these students and others, as for the teachers in their classrooms, the tests were a deliberate insult.

<snip>

To defend standardized testing, you are likely to hear some version of business management guru Peter Drucker’s assertion that “If you can’t measure it, you can’t improve it.” I once heard “We measure what we value,” which I thought was ridiculous because my students and I valued collaboration, humor, persistence, flexibility, and lots of other things depending on the day. None of these ever showed up on the standardized exams--not that I’m discussing what did show up because, of course, that would be breaking the oath they make us all sign to protect the market share and business model of testing companies. You might ask yourself why we are not allowed to discuss these top secret contents. Wouldn’t that help us better prepare our students? Wouldn’t it lead to better exams? Good questions all, but beside the point. And the point is profit.

What I found is that, far from measuring what we value, we instead assign value to the measurements themselves, as long as they can be readily obtained and easily expressed. The measurements are the point, not what they measure. The numbers are vital to a system that employs thousands and thousands of people who interpret the numbers, explain the numbers, compare the numbers, and thousands more who come to your school to tell you how bad your numbers are and what you should do to make them marginally less bad. All the while, mind you, not measuring what you value. They don’t care about what teachers value or what students value. This is what they value.

Do not fall for it. You may have to give the tests, but you do not have to believe in them. Do not adopt their self-serving framing that everything can and should be quantified. It can not. But when you are a hammer, everything looks like a nail. Big money in nails.

I know you know this. I know you have a million stories just like this and much, much worse. I know that many of you are tortured by the knowledge that the relentless testing is preventing you from serving your students better. I know I was. You and I know that, regardless of advertising, testing is an insidious form of control that shifts power away from the classroom and limits what students learn and teachers teach. Testing is a crucial component in a weapons system armed and aimed directly at public education.  

But, and I know this is pretty rich coming from a retired guy, please please please please please please please don't give up. We all know that what is happening is wrong, but we don't yet know how to stop it. Still, your students need you, and I'm living proof that you can survive--at least for twenty-five years. If you are in a position to speak up, resist and push back, I urge you to do so. If you are not in that position yet, you will be if you stay long enough. 

I'm going to keep fighting and looking for ways to win. I'm going to shift gears from ranting to strategizing. If you have any brilliant ideas, please read this blog and comment here. Or contact me directly at nowwaid@gmail.com. 

For the time being, I'll be shifting another kind of gear as well. I began this joyride by announcing that I was retired and loving it. I think I even wrote that the best retirement advice you'll ever get is "As soon as possible." Well that's all still true, but it's a year now and it's time for me to do some real retirement stuff, so I'll be taking a road trip across the country and back. By camper van. Because that's what you do. 

I think I'm going to post about the trip a little, just to give a glimpse of the delight that waits for you on the other side. Over the rainbow. Please stay tuned and stay in the fight.

Hey! It's the weekend! Have a cocktail--or several. Turn your clocks forward and, when you do, remember that there will never be enough time to do everything you need to do. Start with the things you really want to do, and go from there. 

Data (scores from state-sponsored standardized tests) is bullshit. Part Three, part one

I've been advised that I need to broaden my vocabulary when it comes to data. "Bullshit" is imprecise and banal. I will strive to be better.

But first, two things: I have used "bullshit" because it perfectly reflects the bad faith with which test scores are put forward as accurate measures of student learning, and because I like the percussive "b."  It helps me forcefully convey my frustration and anger at this bad-faith waste of time and students' lives.  

Also, when I quote from Answer Key, I'll keep the bullshit in its original form.

When I talk about data I'm really talking about two categories of information. The first category is useful information. It's the kind that promotes rational decision-making and leads to effective action. It includes all kinds of stuff from miles-per-gallon to blood sugar levels to price per square foot. In schools, attendance, reading levels, numeracy, class size, number of credentialed teachers, these are useful data.

On the other hand, there's a lot of information out there that is not useful, either because it's inaccurate or because it doesn't mean what it pretends to mean, or because it doesn't lead to any action. This information is useless. It is often used to pretend something is true. I will call this fake information. 

For example, the drive from Los Angeles to New York is about six hundred miles and takes about two days. You don't want to start a trip thinking this is accurate or you might find yourself in Gallup, New Mexico, looking for the Empire State Building. Also, this:

Captain Renault: What in heaven's name brought you to Casablanca?

Rick: My health. I came to Casablanca for the waters.
Captain Renault: The waters? What waters? We're in the desert.
Rick: I was misinformed.

Some inaccurate information is deliberately inaccurate. A lot of COVID disinformation is this and it's worse than not accurate. It's cynical, strategic, and potentially deadly. 

Second, sometimes information doesn't mean what you think it means. A lot of this is advertising, and a lot of it is deliberate sleight of hand. Colgate's famous 2007 ad "stating that 80% of dentists recommend their product" is one example. A reporter selecting the most expensive gas station on the planet to dramatically represent the price of gas is another. In some schools, a 100% graduation rate means not counting students who were "counseled into alternative settings." Charters are notorious for this. 

The third kind of fake information is useless because it either is not or cannot be used. One country has launched a nuclear strike which will destroy your city in eighteen minutes. There are things you can do with your eighteen minutes, but none of them will stop the missiles. 

Undoubtedly the worst, most evil type of fake information is that which is collected or generated with no intention of acting on it. Often it's generated for a reason other than the one advertised. This category was invented for state-sponsored standardized testing to which millions of k-12 students are subjected every year. 

Ask most people not in education why students are tested so much, and the first thing they're likely to say is either "To find out what they know" or "They're tested how much?" A lot of people think of testing like it's the good old days, you know, when you took a final to pass your class and an SAT to get into college. Sometimes, somewhere in there was a PSAT for practice. Each one of these tests had real stakes for the students taking them, and the big standardized ones were voluntary. It was a long time ago, but I think I recall taking my SAT on a Saturday. And I had to pay for it. 

That is no longer the testing environment in which we live. There are still final exams and final projects, and students usually feel connected to them because they are based on the work they've been doing in class and because the results will be reflected in their final grade.

In my school, the district required that we turn over school days and school staff to the administration of both the SAT, which was paid for by the district, and the PSAT, which was also paid for by the district and which we started administering in the eighth grade. There is a powerful equity argument for making the tests available, but eighth grade is just ridiculous. The school didn't know how to make a schedule when whole segments were testing in classrooms (we didn't have a common area big enough to hold the testers) while other students were changing classes.

That's just a start. Now come the state-sponsored standardized tests. I know I've posted this chart before, but it illustrates the second semester experience in California schools.

And for dessert, there are extra tests that schools are required to develop in order to practice for the other tests. In a "nice career, be a shame if something happened to it" way, the district offers administrators consultants and materials and programs to help raise test scores. The equation looks something like this:

stagnant test scores + consultants, materials, programs = more practice tests = stagnant test scores - time, resources

And then districts punish schools, their students, teachers, and administrators for "failure" to raise test scores by offering--you guessed it--more consultants, materials, and programs until finally one day, lots of people quit or get fired and the dance begins anew. It's the circle of Life in Schools.

And for what? If the point of all this was to improve instruction, this is certainly not the system we would construct. With state-sponsored standardized tests, there is no opportunity for meaningful analysis by classroom teachers and no comprehensive plan for addressing the issues raised (and I don't mean the predictable default position of "Fuck the teachers, let them figure it out.") However, and this is important, not only is the information produced by these state-sponsored exams effectively useless, the tests do not even measure what they say they measure. They do not show what the profiteers and hucksters and fanatical true-believers say they show.

In an earlier draft of this post I included a lengthy excerpt here from Answer Key talking about reliability and validity, but I don't feel like doing that anymore. As I've been educating myself on this issue--now that I'm no longer consumed by the day-to-day battle for classroom survival--I have discovered how many people have been fighting this fight for years, and it feels like anything I have to say has been said much better than I can, and said over and over.

My latest wake-up call in an ongoing series comes from Dr. Mitchell Robinson, associate professor and chair of music education, and coordinator of the music student teaching program at Michigan State University. And it comes via a paper he wrote in 2014 in which he writes powerfully and eloquently much of what I've been trying to say. A few relevant bits:

Perhaps the most pernicious trend that drives current education reform initiatives is the singular and sole reliance on data as evidence of student learning and teacher effectiveness (Berliner and Biddle 1995).

And, regarding "stack ranking," a system of "dividing employees into arbitrarily predetermined 'effectiveness categories,' based on ratings by managers" (sound familiar?):

The main problem with using stack-ranking systems in teacher evaluation is that the approach is predicated on a series of faulty assumptions. Proponents of this approach believe that:

• teachers are the most important factors influencing student achievement,
• effective teaching can be measured by student test scores and is devoid of context,
• large numbers of American teachers are unqualified, lazy, or simply ineffective, and
• if we can remove these individuals from the work-force, student test scores will improve. (Amrein-Beardsley 2014, 84–88)

The argument is seductively convincing. There is, however, little to no research-based evidence to support any of these assumptions.

Robinson goes on to dispute each one of these assumptions, and I encourage you to read the whole paper, but I do want to include one example, as I don't think I have yet ranted about this specific defect in the current testing/accountability model. On child poverty, from the paper:

Similarly, much of the rhetoric surrounding the so-called “teacher effect” not only ignores the role of poverty in student learning, but actively dismisses this concern.

<snip>

Again, the research here is abundantly clear. When results are controlled for the influences of poverty, nearly every international test of student learning shows that American students score at the top of the rankings. For example, when test scores for U..S students on the 2009 Program for International Assessment (PISA) exams were disaggregated by poverty levels, American children from middle– and upper–socioeconomic status families per-formed as well or better than students from the top three nations in the rankings: Canada, Finland, and South Korea(Walker 2013a).

It occurs to me that I have very little to add to what Dr. Robinson, Diane Ravitch, Peter Greene at Forbes, Bob Shepherd (and here), Jan Resseger, Dennis Shirley, the people at FairTest, and many, many others have said better than I ever could, and they've been saying it for a long time. I want to acknowledge authors whose books I haven't even had time to read yet because I was, you know, teaching--authors such as Alfie Kohn; Anya Kamenetz; Daniel Koretz; Peter Sacks; Harris, Smith, and Harris; David Hursh; and Arlo Kempf, and others who have done the hard work and heavy lifting to make the case against testing in the face of intense opposition. In fact, the argument has been made, and the argument has been won. So what's the trouble?

It's impossible to get someone to understand something 
if their paycheck depends on their not understanding it. 

The trouble isn't that we don't know. The trouble is that we won't do. 

Time for a change. As I continue my education, I'm going to try and shift from ranting about something we all know to talking about what we can do about it. We just got a new boss in LAUSD, and I don't think the best argument is going to be enough. What are some real strategies for putting an end to this madness?

First, however, I do have one other thing to offer: my own experience. Next time I'll do a last post on data and then let it go for a while. I have other stuff to say, after all.

Next time:  Data (scores from state-sponsored standardized tests) is bullshit. Part Three, part two