Tuesday, August 13, 2013

Lost on the Test: Opening Post

Lost on the Test

The mind is not a vessel to be filled but a fire to be kindled.
On Listening to Lectures, Plutarch

The ARRA provides $4.35 billion for the Race to the Top Fund, a competitive grant program designed to encourage and reward States that are ... achieving significant improvement in student outcomes, including making substantial gains in student achievement...
Race to the Top Legislation, 2009[1]

We have debated the goals of education for over two thousand years, but in America today we have come up with a simple answer: The goal of educational is student achievement— the now common euphemism for "higher test scores." As Plutarch's quote illustrates, student achievement has not always been the answer, nor is it typical of the sophisticated answers from some of our great philosophers and intellects—Plato, Aristotle, Rousseau, Locke, etc.  This small essay is an attempt to examine the consequences of making student achievement our goal. It contains neither new research nor profound new insights. It is primarily a synthesis of observations about how our choice affects schools, teachers, and students. It is not a screed against tests, which I emphasize are valuable tools when properly used. It is rather a commentary on the changes in our education system that occur when we make student achievement our goal—when we substitute something mundane (test scores) for something profound (educated students).

The prominent role of student achievement is so firmly and unconsciously accepted that it guides all school reform, forms the basis for most research, and determines countless major decisions, from closing schools to firing teachers. Student achievement is part of most current education legislation, illustrated by the quote above from the Race to the Top. To most people, this focus on tests seems natural. After all, tests are commonplace. We use them to assign grades, decide who graduates, determine who enters which college, and certify professionals—doctors, nurses, engineers, and lawyers. People are comfortable using tests for everyday tasks like issuing drivers licenses. Educators routinely use tests to communicate with students—you learned this but not that, you have to work on such-and-such, or you need to master this technique. Tests allow teachers to set short-term expectations for students, judge whether those expectations are met, and give feedback about what more needs to be done. Tests are tools that measure a certain kind of learning.

In America today, however, we clearly use tests as goals rather than tools. Our language describing education revolves around tests—student achievement, academic progress, academic achievement, low-performing, high-performing. The rationale and the measure of success for all education reform are provided by tests. When schools are closed, they are closed because of low test scores. When we evaluate teachers, we are told we must use tests as an integral part of the process. When reporters write about education, the data they reference are test scores. When politicians or policy makers promise improvement, they promise higher test scores—they promise student achievement.

Confusing tools for goals sounds academic, like one of those punctilious distinctions philosophers make. But the confusion has profound consequences in modern life—good government is supplanted by polls and surveys, high-quality research by page and citation counts, and good business practice by a focus only on quarterly financial reports. But education provides a particularly dramatic example. It substitutes small for large, modest for lofty, shallow for profound. It devalues things like creativity, curiosity, and a passion for knowledge, and places a premium on skills and facts. In short, confusing tools with goals corrupts education by driving out more worthy goals for education.

This idea of corruption was famously formulated by the sociologist Donald Campbell in 1976:
"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."[2]

The most obvious corruption is outright cheating. The recent headline in the New York Times seemed sensational: "Ex-Schools Chief in Atlanta Is Indicted in Testing Scandal." [3]  In fact, the allegations of wide-spread cheating in Atlanta were not new; for over a decade rumors about cheating had circulated and been reported by journalists. And Atlanta was not alone in hosting such scandals. In the past few years, large-scale cheating has been reported in more than 37 states and the District of Columbia.[4]

Cheating is not always blatant. A teacher may drill students on a question he or she knows will appear on tomorrow's test, or walk past students and suggest they think harder about an answer, or give them "a little extra time" on some pretext. Principals may suggest that teachers encourage marginal students to stay home on the day of an important test. They may shift students into special classes for the same purpose. Administrators sometimes manipulate proficiency cutoff scores or even lower standards in order to show “improvement."  While these things may not represent outright dishonesty, they diminish integrity. Of course, student achievement does not cause cheating and it is not an excuse for dishonesty, but Campbell's Law says we should not be surprised when it happens.

Cheating is not the only area changed by student achievement, however.

Curriculum: This is what people refer to as "teaching to the test," that is, narrowing the curriculum to focus mainly, or sometimes exclusively, on what is tested. Tests cover only a portion of what students learn, and if the stakes are high, teachers and administrators will discern that portion and focus on it. Over time, the curriculum narrows, shedding key topics that are not easily tested. Teaching to the test sounds innocent enough, but when a curriculum is closely "aligned" with a test, student achievement may go up while actual learning goes down.

Teaching: Teaching is complicated; there is no single style perfect for every circumstance. For centuries, the Socratic Method, a dialogue between teacher and student, was held in high esteem. Student-centered teaching has been promoted for the past couple decades, in contrast to teacher-centered lecturing. Recently, especially in the hard sciences, some educators have promoted learning in which students are actively constructing their own understanding. This is a small sampling of a very complicated topic. When student achievement is the goal, the range of styles narrows. It becomes harder to try out novel approaches or to cater to the needs of individual students; education evolves into test preparation.

Teachers: Recently, newspapers in some major cities published valued-added statistics that purport to rate teachers on their contributions to raising student test scores, separate from all other contributions. Some teachers, who were experts in their discipline, craftsmen in the classroom, and greatly admired by their students, were humiliated because they were rated low. In interviews, they wondered why they had not recognized their deficiencies earlier. It is astounding that we allow a statistic, mysterious and highly variable, to supersede the judgment of a teacher’s supervisors, peers, and students.

Teachers choose their career for many reasons. Most teachers are motivated by idealism and believe their job is to educate and inspire their students, preparing them for life. No one should be surprised therefore that many teachers are repelled by the culture of student achievement; teaching to the test is not high-quality teaching.  Over time, teachers with the most idealism will become demoralized and move on; those who view teaching as test prep will stay and be joined by like-minded colleagues.

Research: In 1945, George Pólya, one of the preeminent mathematics educators of the 20th century, wrote a book filled with advice for teachers of mathematics[5]. The advice was gleaned from his experience, and by the standards of the time it was based on "education research". Pólya's work was carried out during many years of observation and analysis. At the time, there was a field known as philosophy of education, and people wrote about principles, collective experience, and derived wisdom. Research in education was applied scholarship.

Here are samples of current research based on student achievement.
On average, a one standard deviation improvement in teacher VA in a single grade raises earnings by about 1% at age 28. Replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students’ lifetime income by more than $250,000 for the average class- room in our sample.
Raj Chetty et al, 2011[6]

In comparison to classrooms of students elsewhere with similar baseline achievement and demographics, a teacher’s achievement gain in one year is correlated at a rate of .48 in math and .36 in English language arts (ELA), with the average growth of students in another year. Such volatility notwithstanding, a track record of achievement gains is a more reliable predictor of the gains of future students than classroom observations or student surveys.
Thomas Kane, 2012[7]

Of course, these are statements about test scores, not about education. Student achievement has made education research the domain of economists and statisticians. Economics departments have spawned education institutes. Much education research is published in economics journals. Research used to be shaped by analysis and philosophy. Today, it deals with data and statistics. It draws conclusions often with absurd levels of precision (a raise in earnings of 1% at age 28!). It ignores and undermines what at one time was deep scholarship carried out by great intellects.

Reform: It's possible to improve education without immediately improving test scores. Fostering creativity and imagination may not change scores. Cultivating better attitudes about a subject may have no effect on scores. Inspiring students to be life-long learners will pay off, but perhaps not this year or even next, and it won’t necessarily change scores. Reform that focuses on student achievement has limited objectives and a narrow focus.

Student achievement also misdirects reform, short-changing those who most need our help. The "achievement gap" refers to the difference in education for advantaged and disadvantaged children. It is a tragedy, not uniquely American but particularly acute here. If we measure the "gap" only by student achievement, we "close it" merely by raising test scores. Some reformers promote every technique known to do that—a narrow curriculum, continuous test preparation, and endless drill. When we emphasize test prep and drill, we de-emphasize things like critical thinking, curiosity, and perseverance. We may be “closing the gap” for these students without actually providing them what they need most for long-term success.

Aspirations: The greatest corruption from student achievement may be the most abstract. When test scores are the focus, aspirations are quickly lowered. Should we build character? Create good citizens? Cultivate an ability to innovate? Perhaps. But if our goal is only student achievement, our primary goal is to raise scores. Some argue that test scores serve as proxies for other goals, but test scores serve as poor proxies for the most important goals. In his recent book, the reporter Paul Tough writes about the work of educators, psychologists, and neuroscientists describing qualities that underlie both early school success and lifelong learning.
What matters most in a child's development ... is not how much information we can stuff into her brain in the first few years. What matters, instead, is whether we are able to help her develop a very different set of qualities, a list that includes persistence, self-control, curiosity, conscientiousness, grit, and self-confidence.
Paul Tough, 2012[8]

Surely instilling these qualities ought to be among our most important aspirations, yet none is easily measured by standardized tests.

Einstein is famous for promoting curiosity combined with critical thinking—what he refers to as "imagination."  
Imagination is more important than knowledge. For knowledge is limited, whereas imagination embraces the entire world, stimulating progress, giving birth to evolution. It is, strictly speaking, a real factor in scientific research.
Albert Einstein, 1931[9]

Compare this to a quote by Michelle Rhee as related in a story from Time magazine:
People say, ‘Well, you know, test scores don’t take into account creativity and the love of learning,’ ... I’m like, ‘You know what? I don’t give a crap.’ Don’t get me wrong. Creativity is good and whatever. But if the children don’t know how to read, I don’t care how creative you are. You’re not doing your job.
Amanda Ripley, 2008[10]

And here we see the most troubling consequence of student achievement: We have made our aspirations for education shallow, dull, and uninspiring. High aspirations for children should not be after-thoughts. Contrary to the assertions of some reformers, having high aspirations (to make them curious, foster creativity, and inspire them to be life-long learners) does not mean we must shortchange fundamental skills.


In spite of its corrupting effect, student achievement dominates education policy in this country. Policy-makers say we face a crisis; we must demand accountability; we must focus on student achievement. For several decades, the triumvirate of crisis, accountability, and testing has been the basis for the education agenda of both major political parties. It was the impetus for No Child Left Behind; it is why we are racing to the top.

The assertion that accountability was absent before the advent of student achievement is ridiculous. There are tools to judge high-quality teachers and teaching.  We should have well-articulated expectations for teachers. They should know the subject they teach with sufficient depth not only to be competent but also to be creative. They should have passion for what they teach so they can inspire their students and engage them. They should have deep pedagogical knowledge that makes them craftsmen at the intricate task of explaining, motivating, and cultivating a thirst for knowledge. These things can and should be judged; they are part of a much larger set of expectations defining an accomplished teacher. Student test scores for accomplished teachers are likely to rise as well, but this is a consequence of being accomplished, not its definition.

It has become fashionable to dismiss the preceding description of accountability as a focus on inputs rather than outputs. This is a strange point of view that replaces accountability for true professionalism, such as knowing your subject and inspiring your students, with test prep, which is something far easier. In any case, if we want to hold teachers and schools accountable for outputs, we ought to hold them accountable for outputs of real value—their students' eventual success as citizens, leadership in science and business, major prizes and awards, or the intellectual vitality of our public life—and not for something as trivial as test scores. Relying on outputs alone encourages people to find ways to manipulate the outputs to their advantage, distorting the real goals of education.

Student achievement proponents say the old accountability tools were ineffective, and indeed accountability has sometimes failed on multiple levels. Schools hired teachers who were unprepared to teach the classes to which they were assigned; principals made cursory observations of teachers or, more importantly, neglected to engage the teachers in the evaluation process in a meaningful way; districts or schools chose textbooks or curricula for convenience rather than efficacy. But when the accountability process fails, we should fix the process, not redefine our goals to create faux accountability. Other countries have successful achieved accountability without debasing the purpose of education. We can learn from them.

It is almost thirty years since the publication of the famous report, A Nation at Risk, which was the first among many reports promoting student achievement. During these past three decades, Americans have enacted major new education legislation, created a system of testing unparalleled in the rest of the world, nearly doubled the spending per pupil (adjusted for inflation), opened and closed schools, fired and recruited thousands of teachers, and embarked on countless experiments—charter schools, extended days, performance pay, vouchers, value-added models, and national standards (soon to be superseded by national assessments). Yet year-by-year we become ever more despondent about the state of American education.

We are justified in feeling despondent because things are indeed getting worse. But this is largely our own doing. We are obsessed with tests and testing as a way to hold each part of the system accountable, forgetting that the true measure of success is what we achieve not merely whether we achieve it. We are dissipating our most precious resource, our outstanding teachers, who are increasingly demoralized and driven from a profession that is no longer treated as one. Our system of education is more fragmented, less cohesive, and more unpredictable than ever before.

We are truly lost on the test.
John Ewing
August 2013

[2] Donald T. Campbell, "Assessing the Impact of Planned Social Change," The Public Affairs Center, Dartmouth College, Hanover New Hampshire, 1976.
[3] New York Times, March 29, 2013.
[5] George Pólya, How to Solve It, (Princeton: Princeton University Press 1945).
[6] Raj Chetty, John N. Friedman, and Jonah E. Rockoff, "The long-term impacts of teachers: Teacher value-added and student outcomes in adulthood," National Bureau of Economic Research, working paper 17699, Dec 2011.
[7] Thomas Kane, "Capturing the Dimensions of Effective Teaching," Educationnext, Vol 12, no. 4, Fall 2012.
[8] Paul Tough, How Children Succeed: Grit, Curiosity, and the Hidden Power of Character, (New York: Houghton Mifflin Harcourt Publishing, 2012)
[9] Albert Einstein, Cosmic Religion: With Other Opinions and Aphorisms (Dover Publications, 1931), p. 97
[10] Amanda Ripley, "Rhee Tackles Classroom Challenge," Time Magazine, Nov 26, 2008.