The Validation Process of Proportional Reasoning Attributes

Description:

The speakers will discuss the process of verifying whether the required psychometric properties, commonly known as attributes, were indeed being used by students in solving proportional reasoning problems. It is part of a larger research project on the cognitive diagnosis modeling in the mathematics subject area of proportional reasoning.

1.      Introduction

Proportional reasoning maintains a critical role in structuring the development of students' mathematics learning as a bridge from elementary mathematics concepts to more advanced ones (Lesh et al., 1988). This paper is part of a larger research project on the cognitive diagnosis assessment in the mathematics subject area of proportional reasoning. Previous studies have presented first steps of the process of reviewing the literature, attribute identification, and task development (xxx, 2010; xxx, in preparation). The current study describes the validation process of the psychometric properties in the assessment of students' mastery of proportional reasoning.

2.      Proportional Reasoning and Cognitive Diagnosis Assessment

Numerous other research studies in mathematics education during the 1980's were also dedicated to lay the foundation of the study in proportional reasoning. Given that there were many interpretations of proportional reasoning in the literature review (Tourniaire & Pulos, 1985; xxx, in preparation), a group of mathematics researchers, mathematics educators, and mathematics practitioners were consulted in search for a list of even finer grained skills, cognitive processes or problem-solving steps (xxx, 2010). This list of psychometric properties, commonly referred to as attributes, provides information about the necessary skills that a student needs to master in order to be deemed proficient in proportional reasoning. After several meetings with the group of consultants, the list of attributes was agreed to include: (1) prerequisite skills; (2a) comparing fractions; (2b) ordering fractions; (3a) constructing ratios; (3b) constructing proportions; (4) multiplicative relationship; (5) proportional relationship; and (6) applying algorithm.

These six attributes were aimed to serve as measurable competence in the assessment of proportional reasoning through the lens of cognitive diagnostic models (CDMs). Unlike traditional psychometric models such as the item response theory, CDMs view students' specific strengths and weaknesses in a particular subject area of interest at a more detailed level so as to optimize the classroom feedback mechanism (xxx, 2004). Based on the six attributes, 13 items were developed in preparation for the validation process of the attributes as to be administered in form of a test to middle school and college students. The study described in this paper was aimed to continue the work in validating the required attributes as the corresponding items were solved by middle school and college students.

3.      Attribute Validation Process: Methodology

3.1.   Subjects and Instruments

Nineteen subjects in this study can be categorized into two different groups. The first group consisted of eight seventh grade students, and the second group consisted of 11 college students. All 19 students were interviewed individually using the think aloud protocol. The middle school group was asked to solve 10 out of the 13 items, whereas the college group was asked to solve all 13 items. The first 10 items tested out to the college group were the exact same 10 items tested out to the middle school group. The three additional items for the college group were deemed more appropriate for college students in the proportional reasoning context at a higher level mathematics with applications in chemistry and physics.

3.2.   Coding Procedure

These interviews were audio-taped and transcribed. Along with students' written solutions, the transcripts were coded in order to detect any presence of students' use of any of the six attributes. The coding scheme used was detailed by xxx (in preparation). It should be made clear that the coding scheme used was not intended to score students' responses or mastery of any of the six attributes, but rather it was aimed to examine which attributes that students used in solving the 13 items in the context of proportional reasoning. The following three tables summarize the result of coding analysis. (Note: required attributes were colored in light blue.)

Table 1. Result of Coding Analysis for the Middle School Student Group

Table 2. Result of Coding Analysis for the College Student Group

Table 3. Result of Coding Analysis for both the Middle School and the College Student Groups

4.      Discussions on the Findings

The findings revealed relatively poor performance in proportional reasoning problem solving. Only 34% of the 10 items and 65% of the 13 items were answered correctly by the middle school and college groups respectively. Nonetheless, those who did answer the items correctly demonstrated, to some extent, an agreeable pattern in the use of required attributes. This evidence was more apparent in the case of college students than that of middle school students. Seven out of all 13 items were considered “on-target” in general. As many as three items (i.e. Items 6, 11, and 13) might be considered “on-target” in general if it were not for the poor student performance on these items.

Even so, one might have a different interpretation of the more complete picture. Although required attributes appeared relatively significant for most of the items, they were often accompanied with other attributes that were not intended to be assessed for those particular items. On the whole, the discrepancy was mainly due to the fact that students tended to perform unnecessary work than required. In the end, one might not be completely assured that some of the developed items with the required attributes were indeed “on-target” in general because of the noise from the non-required attributes.

5.      Other Thoughts and Implications on Future Research

The authors would like to point out some thoughts in regards to Attributes 1 and 5 in the coding procedure. Attribute 1 is very pervasive. As predicted (xxx, 2010), this attribute appeared in all 13 items consistently with at least three-fifths of a likelihood.  One explanation is, as discussed earlier, perhaps due to the fact that students often performed unnecessary arithmetic calculations for even a very straightforward problem.  A question remains: Should this attribute be eliminated for the sake of simplicity? Attribute 5 has proven to offer its unique challenge to identify during the coding process.  In spite of this difficulty, students' mastery of this Attribute 5 can be maintained to be critical in their general understanding of proportional reasoning.

References

Lesh, R., Post, T., & Behr, M. (1988). Proportional reasoning. In J. Hiebert & M. Behr (Eds.), Number concepts and operations in the middle grades (pp. 93-118). Reston, VA: National Council of Teachers of Mathematics.

Tourniaire, F., & Pulos, S. (1985). Proportional reasoning: A review of the literature. Educational Studies in Mathematics, 16 (12), 181-204.

xxx. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69 (3), 333-353.

xxx. (2009). A cognitive diagnosis model for cognitively-based multiple-choice options. Applied Psychological Measurement, 33, 163-183.

xxx. (2010). Measuring grade 8 proportional reasoning: The process of attribute identification and task development and validation. Paper presented at the Annual Meeting of the American Educational Research Association, Denver, CO.

xxx. (in preparation). Conceptual and theoretical issues in proportional reasoning.