A reason to cheat

By Dave Anderson:

What do steroid popping baseball players, blood-doping cyclists, Michelle Rhee's school reforms in Washington D.C. and the Houston "Drop-out miracle" have in common?

They are all situations where the incentives point towards cheating because of structural design. Each instance has a high performance threshold where failure to meet those thresholds have severe negative financial consequences (loss of job, loss of sponsorship, loss of future earnings potential) and even individuals who want to play cleanly are at a severe disadvantage because there are a sufficent number of other competing individuals who are not playing cleanly.

The Houston drop-out reduction has the data being faked at worse, or deliberately and systemically mis-coded at best (60 Minutes, Jan. 1 2004)

It was called the "Texas Miracle..."

It was an approach to education that was showing amazing results, particularly in Houston, where dropout rates plunged and test scores soared.

Houston School Superintendent Rod Paige was given credit for the schools' success, by making principals and administrators accountable for how well their students did.

Once he was elected president, Mr. Bush named Paige as secretary of education. And Houston became the model for the president's "No Child Left Behind" education reform act.

Now, as Correspondent Dan Rather reported last winter, it turns out that some of those miraculous claims which Houston made were wrong.

And it all came to light when one assistant principal took a close look at his school's phenomenally low dropout rates - and found that they were just too good to be true.

"I was shocked. I said, 'How can that be,'" says Robert Kimball, an assistant principal at Sharpstown High School, on Houston's West Side. His school claimed that no students - not a single one - had dropped out in 2001-2002.

But that's not what Kimball saw: "I had been at the high school for three years, and I had seen many, many students, several hundred a year, go out the door. And I knew that they were quitting. They told me they were quitting."

All in all, 463 kids left Sharpstown High School that year, for a variety of reasons. The school reported zero dropouts, but dozens of the students did just that. School officials hid that fact by classifying, or coding, them as leaving for acceptable reasons: transferring to another school, or returning to their native country.

"That's how you get to zero dropouts. By assigning codes that say, 'Well, this student, you know, went to another school. He did this or that.' And basically, all 463 students disappeared. And the school reported zero dropouts for the year," says Kimball. "They were not counted as dropouts, so the school had an outstanding record."

USA Today on Washington D.C.'s "accountability-based" school reform's shining star:

Michelle Rhee, then chancellor of D.C. schools, took a special interest in Noyes. She touted the school, which now serves preschoolers through eighth-graders, as an example of how the sweeping changes she championed could transform even the lowest-performing Washington schools. Twice in three years, she rewarded Noyes' staff for boosting scores: In 2008 and again in 2010, each teacher won an $8,000 bonus, and the principal won $10,000....

USA TODAY investigation, based on documents and data secured under D.C.'s Freedom of Information Act, found that for the past three school years most of Noyes' classrooms had extraordinarily high numbers of erasures on standardized tests. The consistent pattern was that wrong answers were erased and changed to right ones.

Erasures are detected by the same electronic scanners that CTB/McGraw-Hill, D.C.'s testing company, uses to score the tests. When test-takers change answers, they erase penciled-in bubble marks that leave behind a smudge; the machines tally the erasures as well as the new answers for each student.

In 2007-08, six classrooms out of the eight taking tests at Noyes were flagged by McGraw-Hill because of high wrong-to-right erasure rates. The pattern was repeated in the 2008-09 and 2009-10 school years, when 80% of Noyes classrooms were flagged by McGraw-Hill....

To be flagged, a classroom had to have so many wrong-to-right erasures that the average for each student was 4 standard deviations higher than the average for all D.C. students in that grade on that test.

Speaking as a professional program and policy evaluator, extreme changes and lottery-type oddities are massive red flags for me. There may, repeat, may, be innocous reasons. However when oddities are wide-spread, systemic and bizarre, my first instinct is to think that the data is not reflecting reality. A 2 standard deviation difference is often "statistically significant" and publishable in a peer review journal while 3 and 4 standard deviations are extremely rare.

In both situations the "accountability" metrics drive everything despite the fact that the current level of "accountability" metrics are amazingly bad at producing a decent explanation of change responsiblity due to a variety of factors. Science Blog's Mike the Mad Biologist looks at New York City's teacher "accountability" system:

Moreover, as the city indicates on the data reports, there is a large margin of error. So Ms. Isaacson's 7th percentile could actually be as low as zero or as high as the 52nd percentile -- a score that could have earned her tenure.

So, basically, we have 'progressives' lauding a teacher evaluation method ("measurable differences") that can't really measure if a teacher is doing a good job. The method works great, except that it can't tell if you're succeeding or failing.

As long as we are determined to impose "accountability" with horrendously imprecise and easily gamed metrics, we should expect cheating to occur because the stakes are too high and the ease of cheating outweighs the consequences. Actually doing in-depth, comprehensive statistical quality management of education would require a very different testing regime. Instead of having every student take a lowest common denominator/cheap enough to adminster to millions of individuals standardized exam, a small randomly selected group of students from each district would be subjected to several days of very probing exams in order to actually get useful results. The same expenditures may be used for testing, but it would be concentrated on 3% to 5% of a districts' population instead of everyone.