Calls for research on curriculum in mathematics education have included the recommendation that studies include systematic data collection on the fidelity of curriculum implementation (Confrey & Stohl, 2004). At the same time, there is little consensus on what fidelity of curricular implementation is and how to operationalize the construct for data collection (Chval, et al., 2008; Huntley, 2009). The objective of this research symposium is to explore different methods used to study curriculum implementation and the implications of these choices across three major research projects that examine the connections between implementation and student achievement. The research projects both explore methodological questions of how to measure fidelity of implementation and what are the implications of different measurement choices for explaining student achievement.

The first set of studies frame curriculum implementation from the perspective of teacher choice in instructional materials use. This research group takes the view that the instructional materials represent cultural tools for teachers to use as they deem appropriate for their students’ learning needs (Cole 1996; Wertsch 1985). As a cultural tool, the textbook is not an intervention itself, but a tool that teachers use to work with students to enact a lesson leading to student learning (Remillard, 2005). From this perspective, we seek to understand when teachers use and adapt this tool for their purposes, and the impact of this use and adaptation on the desired outcome, student learning of mathematics.

To explore this, we surveyed all middle grades mathematics teachers in an urban district that had adopted the Connected Mathematics Projects (CMP) instructional materials. We used two different surveys: the Surveys of Enacted Curriculum (SEC; Porter, 2002, 2011), which focused on the content and cognitive demand of instruction; and the CMP Implementation Survey (Authors, 2009) focused on the use of particular lessons and units in the CMP and the decisions for that use. Using HLM, the analysis explores the ways in which these two surveys allow for similar and differing explanations of teachers’ enacted curriculum. The SEC-based analysis allows for explanations that focus on content coverage, cognitive demand, and curricular and assessment alignment. The CMP Implementation survey-based analysis frames the explanations in terms of use and adaptation of particular curricular lessons and units. While both analyses present statistically-significant variables related to student achievement, they present different images of teachers’ fidelity of implementation to the CMP instructional materials with implications for the definition of fidelity of implementation.

In the second set of studies, Authors [in press] describe how fidelity of implementation was conceptualized and measured in a large-scale study of curricular effectiveness at the secondary level. Specifically, they sought to ascertain relationships between curriculum implementation and student learning in the context of high schools that offer parallel integrated and subject-specific programs. In this study, fidelity of implementation is distinguished from an arguably related construct, opportunity to learn. A unique contribution of this research stems from the project’s deliberate effort to take into account authors’ intent; in project interviews, textbook authors espoused observable characteristics of “faithful” implementation that informed the design of classroom visit protocols. Individual attributes of curriculum implementation enabled researchers to render global content fidelity and presentation fidelity ratings along a continuum from low to moderate to high fidelity. Multiple instruments were developed to gauge implementation and opportunity to learn and principle components analyses were employed to responsibly reduce the data. Student learning was assessed each year using three dependent measures and using hierarchical linear modeling, student outcomes were examined in relation to numerous student- and teacher-level variables. In year 1, results indicate curriculum type was a significant predictor of student learning as was opportunity to learn; implementation fidelity did not hold significant explanatory power. A similar pattern of results was obtained in year 2 but a curriculum effect was detected on only one of three measures. The studies highlight the complexities of measuring curricular effectiveness in valid and reliable ways that are essential in establishing causal relationships.

In the third set of studies, Authors [2011, in press] will present findings from a longitudinal research project, which followed 1,300 students from Grades 6-8, and then is following 1000 of the 1300 students from Grades 9-12 to examine the effect of Connected Mathematics Program (CMP) on students’ learning.  The overall goal of the longitudinal research project is to provide scientific evidence about the effectiveness of curricula on students’ learning, but also to provide venues to test fundamental hypotheses about curriculum and teaching variables. A quasi-experimental design with mixed methods has been used in the project and multiple outcome measures were used for assessing student learning.  In this presentation, the authors will first present the conceptualization and specific measures of curriculum implementation by examining the nature and quality of classroom instruction.  And then authors will present results from analysis using HLM Growth Curve Modeling to longitudinally investigate the relationship between curriculum implementation and student learning.  It is found that the nature and quality of classroom instruction are significant predictors for students’ achievement gains over the three middle school years.  Now, these CMP and non-CMP students are in high schools, and they are mixed together in the same classes with the same teachers using the same curricula.  The uniqueness of this project is to investigate the curriculum implementation and its effect on students’ learning beyond grade bands.  

The session will begin with a brief 5-minute introduction to the three projects, followed by 20-minute presentations of the projects’ differing perspectives. A discussant will react to the presentations for 15 minutes. Thirty minutes will be devoted to audience discussion of the following set of questions:

1. What are the affordances and constraints of different operational definitions of fidelity of implementation of curriculum materials?
2. What are the affordances and constraints of different data collection methods for studying fidelity of implementation, particularly the distinctions between teacher self-report and observational methods?
3. What are the implications of different definitions of fidelity of implementation for recommendations for instructional practice and support in districts that adopt particular curricular materials?

References

Chval, K. B., Chavez, O., Reys, B., & Tarr, J. E. (2009). Considerations and limitations related to conceptualizing and measuring textbook integrity. In J. T. Remillard, B. A. Herbel-Eisenmann & G. M. Lloyd (Eds.), Mathematics teachers at work: Connecting curriculum materials and classroom instruction (pp. 70-84). New York: Routledge.

Cole, M. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Belknap Press of Harvard University Press.

Confrey, J., & Stohl, V. (Eds.). (2004). On evaluating curricular effectiveness: Judging the quality of K-12 mathematics evaluations. Washington, DC: National Academy Press.

Huntley, M. A. (2009). Measuring curriculum implementation. Journal for Research in Mathematics Education, 40(4), 8.

Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3-14.

Porter, A. C. (2011). Surveys of enacted curriculum and the council of chief state school officers collaborative. In W. F. Tate, K. D. King & C. R. Anderson (Eds.), Disrupting tradition: Research and practice pathways in mathematics education (pp. 21-31). Reston, VA: National Council of Teachers of Mathematics.

Remillard, J. T. (2005). Examining key concepts in research on teachers' use of mathematics curricula. Review of Educational Research, 75(2), 211-246.

Wertsch, J. V. (1985). Vygotsky and the social formation of mind. Cambridge, Mass.: Cambridge, Mass. : Harvard University Press.