Steven Weinberg, a retired Oakland teacher and regular Education Report blogger, tells us what standardized tests can’t measure, in his view, and why. -Katy
In my last 10 years working for Oakland Unified School District, I spent considerable time investigating the California Standards Tests and their results to help my school make sense of the data the tests generated. During that time I became aware of a number of myths have been built up about these tests, many propagated by the state or the test makers themselves.
Knowing the facts about these tests is important for drawing reasonable conclusions from their results and for making sound educational decisions for the future.
I know that most readers of this blog are already fairly sophisticated about the nature of standardized testing, but the results of these tests are so often misused, it is worth taking some time to review these misconceptions.
Myth 1: The California Standards Tests (CSTs) measure what teachers are supposed to teach.
Fact: Not everything measured on the CSTs is included in the standards. While the CST questions are linked to standards that they purportedly cover, they actually often require students to know information that is not included in the standards. Many test questions include above grade level vocabulary. Others require background knowledge, not included in the standards, that only some students will be familiar with because of their family backgrounds and life experiences.
In addition, important sections of the California Standards (e.g. oral language, writing at most grade levels, critical thinking, being able to conduct experiments) do not lend themselves to standardized testing, so those standards are never tested at all.
Myth 2: The CSTs were designed and scored by the State of California Department of Education.
Fact: The CSTs are written by out-of-state private testing companies under contract with the State Department of Education. The tests are even shipped out-of-state to be graded. Furthermore, these companies have often helped write the laws which control the bidding by which the contracts were awarded, thus helping them maintain a monopoly on test preparation.
Myth 3: The CSTs are carefully prepared.
Fact: The CSTs, like most of the standardized tests created in recent years, were constructed under timelines enacted by politicians who did not understand what was required to produce a quality test. The New York Times investigated this problem and concluded the tests were thrown together to meet political demands. The California High School Exit Exam, for example, had “impossible, unrealistic time lines” according to a testing company executive familiar with the process quoted in a New York Times story.
Myth 4: The CSTs are carefully examined to avoid errors.
Fact: Every year the CSTs contain mistakes, but the confidentiality rules required of anyone involved with the testing process prevent those mistakes from being publicized, or even reported. I have personally tried on 5 occasions to notify the state of errors on the test and have been told each time that I could not report them even to the State Department of Education itself, without violating the confidentiality rules. Since the state refused to let me tell them where the mistakes were, some of those errors were repeated the following years.
Myth 5: Individual student scores accurately measure a student’s achievement.
Fact: The tests are designed to give a broad picture of student achievement across a large number of students, not to be an accurate measure of any individual student’s achievement. Test makers expect that all students will guess at the answers to some questions and the scores of those who make lucky guesses will be balanced out by the scores of students who make unlucky guesses, but this only holds true if the group is large enough. That is why the test scores for groups of students under 50 and schools smaller than 100 are not given the same weight as the scores for larger groups. The state and the testing companies state that no educational decisions should be based on a single test score. The CDE information booklet for school districts and staffs says although the tests may be used to “help make decisions about student placement, promotion, retention or other considerations related to student achievement. These test scores should never be used by themselves to make…important decisions.” (p. 13, emphasis added)
Myth 6: Measuring the change in test scores of a group of students as they move from grade to grade is a good way to measure teacher or school quality.
Fact: The CSTs were not designed to allow for accurate comparisons from one grade level to another. The state itself says: “The CSTs should never be used for the following purposes: To monitor the progress of cohorts of students as they move through the grades. (Differences in state content standards tested between the grades, differences in performance level setting, and other factors prohibit cohort tracking with CST results.)”
Over the years, a consistent statewide dip in test scores in certain grades indicate that some tests (eighth grade English for example) are significantly harder than the tests taken the year before or the year after, so teachers of that grade will always appear to be doing a poor job. In spite of this fact, Oakland Unified has included these types of comparisons in their school rating procedure. Even worse, performance-pay advocates are advancing plans that ignore the limitations of our present tests and attempt to tie teacher pay or job retention to changes in student test scores from one grade to another.
I will have more to say about the limits and defects of these tests in future postings. Meanwhile, others might have their own comments to add.