Assessment is “the process of gathering evidence about a student’s knowledge of, ability to use, and dispositions toward, mathematics and making inferences from that evidence for a variety of purposes” (NCTM, 1995, p. 3). Any method used to assess children’s mathematics learning should reflect meaningful goals and objectives (Thompson & Briars, 1989) and help educators identify ways to improve mathematics teaching and learning (NCTM, 1989). Mathematics educators have indicated that how students learn (i.e., the mathematical processes through which they learn) is as important as what they learn (i.e., the specific mathematical content) (Ma, 1999).  To further instruction, teachers need to gain insight into their students’ thinking, both to understand what their students have learned as well as how students have made connections between concepts. Assessment, both formal and informal, is one strategy by which teachers gain these insights.

The goal of this work was to develop, test, and refine a framework to study the mathematical processes embedded in the assessed curriculum of chapter tests accompanying elementary textbooks. The research questions were: 

1.   To what extent can a framework to analyze mathematical processes and proficiencies be developed and reliably applied to the mathematics assessments accompanying published elementary curricula? 

2.   To what extent can the framework illuminate differences among publishers and content strands in the level of mathematical processes and proficiencies embodied in the tests that accompany published elementary mathematics curricula?

Although the tests that accompany published curricula do not likely constitute teachers’ entire classroom-based assessment program, we believe they are worthy of analysis because evidence exists that teachers depend on them as a primary form of assessment, especially at the elementary level (Blok, Otter, & Roeleveld, 2002; Delandshere & Jones, 1999). We believe much can be learned about students’ opportunities to demonstrate their thinking by undertaking a detailed analysis of the assessments accompanying published curricula. Similar to the suggestion by Begle (1973) that textbooks are a variable to be manipulated to enhance learning, we believe the nature of tests can be manipulated so that assessments embody the spirit of recommended reforms in mathematics and provide insight into student thinking and reasoning.

Over the course of eight years, through four major phases, we developed and validated the Mathematical Processes Assessment Coding Framework (MPAC Framework) that includes ten criterion, each with two to seven indicators, to:

• identify the mathematical processes and proficiencies embodied in assessments that accompany published curricula;
• be applicable across all content strands in elementary school mathematics;
• be able to capture nuances among test items with regard to the processes students are expected to employ while answering the items;
• be reliably applied by classroom teachers, curriculum specialists, or researchers to provide information that could be used to enhance assessments to gain information about students’ mathematical thinking;
• be applicable to assessments accompanying a range of curricula from typical publisher-generated curricula to more alternative curricula developed by curriculum projects in response to the Standards; and
• build upon existing related research.

For data sources, we focused on mainline curricula that were on the textbook adoption list in the researchers’ state and were likely to be widely used throughout the country. Samples for coding were fifteen chapter tests drawn from all five content strands across the grades 3-5 published curricula from: Go Math! from Houghton Mifflin Harcourt (Adams, Dixon, Larson, McLeod, & Leiva, 2011), Math Connects from Macmillan/McGraw-Hill (Altieri et al., 2011), and enVisionMATH  from Scott Foresman-Addison Wesley (Charles et al., 2011).  All curricula presented their unit/chapter assessments in ancillaries included with the teachers’ materials, and all three included multiple versions of tests for each chapter.

We focused on free-response assessments from all three publishers because we believed some mathematical processes, such as reasoning and communication, are difficult to evoke with a multiple-choice format. Because the focus of the work was to develop a framework and ensure its ability to highlight differences within and between publishers and across content domains, we used a stratified selection of tests for coding. Nine tests across the three publishers, three grade levels, and five content domains were independently coded to check for reliability. Cohen’s Kappa was run for each criterion, with Kappa greater than 0.90 for all criteria except Depth of Knowledge (0.867) and Conceptual vs. Procedural (percent agreement was 0.72). These results suggest that, with appropriate training, the MPAC Framework can be reliably applied.

The item-level results were compiled by test, and the ratio of items on each test that were coded for each indicator was calculated. The ratios were divided into five ranges from “0” (no items on the test were coded with that indicator) to “1” (all items on the test were coded with that indicator), to provide a mechanism to assess the relative emphasis of each criterion and highlight differences among publishers and content strands in levels of engagement with mathematical processes.

The poster for this session will present the MPAC Framework and graphical results of the emphases across processes. Some highlights of our results were that in one curriculum, over three-fourths of the items in number and operations engaged students in mathematical communication yet another curriculum had no opportunities for communication in that same content strand. The tests from one curriculum focused mainly on procedural understanding whereas the other two curricula’s tests concentrated on conceptual understanding. The graphical presentation of results will allow participants to observe differences among and within publishers, and sample test items and their codes will aid participants in their engagement with the framework. The researchers will use input from interacting with visitors to the poster to inform their future work in applying the MPAC Framework to a more extensive set of chapter assessments from a variety of curriculum publishers to determine the extent to which the assessments give students opportunities to engage in important mathematical processes.

References

Adams, T. L., Dixon, J. K., Larson, M., McLeod, J. C., & Leiva, M. A. (2011). Houghton Mifflin Harcourt Go Math! Florida. Grades 3-5. Orlando, FL: Houghton Mifflin Harcourt.

Begle, E. G. (1973). Some lessons learned by SMSG. Mathematics Teacher, 66, 207-214.

Blok, H., Otter, M. E., & Roeleveld, J. (2002). Coping with conflicting demands: Student assessment in Dutch primary schools. Studies in Educational Evaluation, 28, 177-188.  

Delandshere, G., & Jones, J. (1999). Elementary teachers’ beliefs about assessment in mathematics:  A Case of assessment paralysis. Journal of Curriculum and Supervision, 14, 216-240.

Ma, L. (1999). Knowing and teaching elementary mathematics. Mahwah, NJ: Lawrence Erlbaum.

National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.

National Council of Teachers of Mathematics. (1995). Assessment standards for school mathematics. Reston, VA: Author.

Thompson, A. G., & Briars, D. J. (1989). Assessing students’ learning to inform teaching: The message in NCTM’s evaluation standards. Arithmetic Teacher, 37(4), 22-26.