Teacher Ratings Get New Look, Pushed by a Rich Watcher

NY Times
December 3, 2010

PRINCETON, N.J. — In most American schools, teachers are evaluated by principals or other administrators who drop in for occasional classroom visits and fill out forms to rate their performance.

The result? More than 9 out of 10 teachers get top marks, according to a prominent study last year by the New Teacher Project, a nonprofit group focusing on improving teacher quality.

Now Bill Gates, who in recent years has turned his attention and considerable fortune to improving American education, is investing $335 million through his foundation to overhaul the personnel departments of several big school systems. A big chunk of that money is financing research by dozens of social scientists and thousands of teachers to develop a better system for evaluating classroom instruction.

The effort will have enormous consequences for the movement to hold schools and educators more accountable for student achievement.

Twenty states are overhauling their teacher-evaluation systems, partly to fulfill plans set in motion by a $4 billion federal grant competition, and they are eagerly awaiting the research results.

For teachers, the findings could mean more scrutiny. But they may also provide more specific guidance about what is expected of the teachers in the classroom if new experiments with other measures are adopted — including tests that gauge teachers’ mastery of their subjects, surveys that ask students about the learning environments in their classes and digital videos of teachers’ lessons, scored by experts.

“It’s huge,” said Deborah Loewenberg Ball, dean of the University of Michigan School of Education. “They’re trying to do something nobody’s done before, and do it very quickly.”

The Gates research is by no means the first effort of its kind. Economists have already developed a statistical method called value-added modeling that calculates how much teachers help their students learn, based on changes in test scores from year to year. The method allows districts to rank teachers from best to worst.

Value-added modeling is used in hundreds of districts. But teachers complain that boiling down all they do into a single statistic offers an incomplete picture; they want more measures of their performance taken into account.

The Gates research uses value added as a starting point, but aims to develop other measures that can not only rate teachers but also help educators understand why one is more successful than another.

Researchers and educators involved in the project described it as maddeningly complex in its effort to separate the attributes of good teaching from the idiosyncrasies of individual teachers.

Mr. Gates is tracking the research closely. The use of digital video in particular has caught his attention. In an interview, he cited its potential for evaluating teachers and for helping them learn from talented colleagues.

“Some teachers are extremely good,” Mr. Gates said. “And one of the goals is to say, you know, ‘Let’s go look at those teachers.’ What’s unbelievable is how little the exemplars have been studied. And then saying, ‘O.K., How do you take a math teacher who’s in the third quartile and teach them how to get kids interested — get the kid who’s smart to pay attention, a kid who’s behind to pay attention?’ Teaching a teacher to do that — you have to follow the exemplars.”

The meticulous scoring of videotaped lessons for this project is unfolding on a scale never undertaken in educational research, said Catherine A. McClellan, a director for the Educational Testing Service who is overseeing the process.

By next June, researchers will have about 24,000 videotaped lessons. Because some must be scored using more than one protocol, the research will eventually involve reviewing some 64,000 hours of classroom video. Early next year, Dr. McClellan expects to recruit hundreds of educators and train them to score lessons.

The goal is to help researchers look for possible correlations between certain teaching practices and high student achievement, measured by value-added scores. Thomas J. Kane, a Harvard economist who is leading the research, is scheduled to announce some preliminary results in Washington next Friday. More definitive conclusions are expected in about a year.

The effort has also become a large-scale field trial of using classroom video, to help teachers improve and to evaluate them remotely.

“Video lasts,” Dr. McClellan said, creating possibilities for dialogue among teachers about improving classroom techniques. “Colleagues can watch your video and say, ‘Right here — where you did that — try this next time.’ So the teacher learns a new skill.”

There are advantages for teacher evaluations, too, Dr. Kane said.

With videos, for instance, several professionals, rather than just one principal, could rate the same classroom performance, making ratings less subjective, he said.

“It potentially creates a cottage industry for retired principals, or even expert teachers, to moonlight on weekends scoring classroom observations,” he said.

An Internet-based approach to teacher evaluation could also alleviate some pressures on school districts. New laws in many states, after all, are requiring more frequent observations of teachers.

A new evaluation system in Washington, D.C., for example, requires five observations each year, compared with the previous systems that required one or two at most, and in many cases none at all. Starting next fall, a Tennessee law will require at least four observations a year, rather than one every five years.

In some districts, the increased pace is straining the workload of administrators. Memphis officials realized that under the new rules, their district would need to conduct more than 28,000 classroom observations annually, a task that could overwhelm the city’s school principals.

“This technology can help us face the logistical challenge of being in so many places at the same time,” said John Barker, who leads the district’s research and evaluation office.

The district still intends to have principals visit classrooms, but in January will start a pilot program to use videotaped observations, he said.

Dr. Kane said the foundation hoped more school districts would start using classroom videos, for training and for evaluations, and has worked to keep costs down.

Teachscape, a contractor providing cameras, software, and other services for the research, estimated first-year startup costs of about $1.5 million for a district with 140 schools and 7,000 teachers to buy one camera per school and lease the software to carry out classroom observations using digital video. After that, annual costs would drop to about $800,000, said Mark Atkinson, the chief executive of Teachscape, which is based in San Francisco.

In addition to the cost — which many struggling districts may consider too high — another barrier could be teacher opposition. The Memphis teachers union, an affiliate of the National Education Association, has partnered with the foundation for the project. But Keith Harris, its president, said the use of videotaped observations in evaluations raised troubling questions.

“Whose eyes would see these videos?” Mr. Harris asked. “Who would own them? This seems like an ‘I gotcha’ kind of thing. We think these observations deserve a human being.”

Randi Weingarten, president of the American Federation of Teachers, which has several affiliates participating in the research, also expressed reservations. “Videotaped observations have their role but shouldn’t be used to substitute for in-person observations to evaluate teachers,” Ms. Weingarten said. “It would be hard to justify ratings by outsiders watching videotapes at a remote location who never visited the classroom and couldn’t see for themselves a teacher’s interaction and relationship with students.”

Dr. Kane said doubts may disappear with time. “We’re not naïve,” he said. “We realize that most principals and teachers imagine an in-person visit from a human being when they think of classroom observations. But that could rapidly change. It’s not out of the realm of possibility that millions of classrooms could be using this technology within four or five years.”