A CONCEPTUAL FRAMEWORK FOR DATA-DRIVEN DECISION MAKING IN EDUCATION MATHEMATICA POLICY RESEARCH
8
More generally, reliability tends to be a bigger challenge for measures that focus on changes
or differences in other underlying measures (for instance, achievement gains versus achievement
levels). Subtracting one result from another makes the random variation in each of the two
measures a larger proportion of what is left. For example, even if a student assessment produces
a reliable measure of a student’s current achievement level, a measure of the change in the
student’s achievement from one test to the next may be unreliable. This can be especially
challenging for teacher-developed assessments such as those that are sometimes used for
“student learning objectives.” Teachers should be very cautious of overinterpreting the apparent
change in achievement from one test to the next for any individual student. In general, educators
and policymakers should try to understand the reliability of data before using them to make
decisions, especially if those decisions involve high stakes.
Even when data are reliable, they may not be valid for informing the decision at hand. Data
that are improperly analyzed or interpreted can lead to invalid inferences that are biased, that is,
that cause decision makers to draw exactly the wrong conclusions. Using student achievement
data to assess the effectiveness of teachers, principals, schools, or interventions is especially
susceptible to biased (invalid) inference because student achievement can be affected by many
factors that are unrelated to the effectiveness of staff, schools, or programs. For example, judging
a teacher’s effectiveness based on the achievement of her students without accounting for the
prior achievement of those students would lead to many exemplary teachers being labeled as
ineffective simply because they serve disadvantaged students. This is not to say that student
outcomes data should not be used to evaluate effectiveness, but care in the analysis of such data
is critical for avoiding faulty inferences. In many cases—notably the measurement of teacher and
school value-added—the analysis needed to produce valid inferences is complicated, and a
district or state may want to consult outside experts to conduct it.
As Figure 2b suggests, the difficulty of analyzing data to draw valid inferences for decisions
tends to increase with the decision maker’s level in the structure. Teachers need to know, for
example, the academic strengths and weaknesses of their students, which typically can be
directly identified using assessments. Principals need to know what individual teachers are
contributing to student achievement growth—a more difficult concept to measure. Raw student
achievement data may be valid for purposes of understanding the skills of individual students but
invalid for understanding teachers’ contributions. District officials need to be able to assess the
effectiveness of each principal, which is even more difficult than assessing the effectiveness of a
teacher because principals’ effects on student achievement may take considerable time to
become apparent and because principals’ professional practice is difficult to observe. Similarly,
state officials need to know whether the takeover of a school or district is called for and what
kinds of interventions are most likely to improve the performance of the school or district—
decisions that ideally require data on the effectiveness of alternative interventions, the capacity
of local staff, the pool of human capital from which to draw, and various aspects of the local
context.
In sum, being driven by data requires much more than the existence of an effective data
infrastructure, the accessibility of the data, and a culture of data use. It also requires careful
attention to ensuring that data are both relevant and diagnostic for each decision maker and
decision. Otherwise, there is a high risk that decision maker will either drive in the wrong
direction or drown in the data.