Research Overview
This research study generated likely explanations why students selected incorrect answer choices on the Virginia Standards of Learning (SOL) mathematics assessments and what if any differences existed among students at different pass levels. The 2007 and 2008 middle grade multiple-choice assessments were used for this analysis. Each assessment contained 50 items spread across five Reporting Categories.
The Virginia Department of Education (VDOE) provided frequencies for each distractor by pass level. Qualitative data was generated by Document Analysis Teams (DATs) who generated a list of possible explanations for each distractor. Each grade level DAT consisted of a mathematics teacher, a special education teacher, and a mathematics educator.
The DAT explanations were categorized based on the Categorization Framework for Distractor Explanations (see Figure 1). This structure resulted from a merging of Schoenfeld’s Knowledge Base (1992), the framework used for reporting results on the NAEP by Kloosterman et al (2007, 2004), and the VDOE Reporting Categories.
Figure 1. Categorization Framework for Distractor Explanations
Excel was then used to record the categorizations for each distractor alongside the frequencies by pass level so that Pivot tables could be used to interpret the data. The Data Analysis Process is summarized in Figure 2.
Figure 2. Data Analysis Process
Educational Significance
No Child Left Behind (NCLB) greatly increased the high-stakes nature of state assessments by requiring all students reach at least the proficient level in mathematics by 2014 (NCLB, 2002, 115 STAT. 1447). Teachers and principals need specific information about why student select incorrect responses. “The assumption is that armed with quantitative data, local leadership – teachers, mathematics coordinators and supervisors, principals, assistant superintendents, superintendents, and school board members – will proactively respond to data” (Tate & Rousseau, 2007, p. 1211). However, the reality is that “many schools lack an understanding of the changes that are needed and lack the capacity to make them” (NRC, 1999, p. 5).
Research Questions and Results
Research Question #1
What are the most likely explanations of middle school students’ incorrect distractor choices when responding to multiple choice items on the Mathematics Virginia Standards of Learning (SOL) assessment?
Conceptual Errors and Problem Solving Errors were the primary categories of error among middle school students (see Table 1).
Table 1
Percentage of All Student Responses
Response Categorization |
6th |
7th |
8th |
Correct Responses |
|
|
|
2007 |
71 |
66 |
76 |
2008 |
75 |
68 |
78 |
Conceptual Errors |
|
|
|
2007 |
12 |
20 |
12 |
2008 |
11 |
14 |
9 |
Procedural Errors |
|
|
|
2007 |
4 |
3 |
3 |
2008 |
3 |
5 |
3 |
Problem Solving Errors |
|
|
|
2007 |
12 |
10 |
8 |
2008 |
11 |
12 |
9 |
Unknown Errors |
|
|
|
2007 |
1 |
1 |
1 |
2008 |
0 |
1 |
1 |
Total |
|
|
|
2007 |
100 |
100 |
100 |
2008 |
100 |
100 |
100 |
Conceptual errors accounted for more of the incorrect responses than Problem Solving errors. However, there were differences in the most common Conceptual error for each grade level and year (see Table 2).
Table 2
Summary of Most Common Category of Conceptual errors: All Students
Year |
6th |
7th |
8th |
2007 |
NS |
NS |
PFA
|
2008 |
PFA |
G |
PFA |
Note: NS – Number Sense; G-Geometry; PFA – Patterns, Functions, Algebra
Research Question #2
How do the incorrect distractor choices differ among students who pass advanced, pass, fail basic, and fail below basic on the Mathematics SOL?
Although the percentage of responses which are incorrect was different for each pass level, the percentage of incorrect responses due to Problem Solving Errors was similar regardless of grade level and pass level (see Table 3).
Table 3
Percentage of Incorrect Student Responses due to Problem Solving Errors
Grade Results by Pass Level |
|
2007 |
2008 |
Below Basic |
|
|
|
6th |
|
43 |
42 |
7th |
|
28 |
37 |
8th |
|
33 |
34 |
Basic |
|
|
|
6th |
|
42 |
43 |
7th |
|
29 |
37 |
8th |
|
35 |
38 |
Proficient |
|
|
|
6th |
|
40 |
42 |
7th |
|
27 |
36 |
8th |
|
34 |
41 |
Advanced |
|
|
|
6th |
|
41 |
42 |
7th |
|
25 |
35 |
8th |
|
32 |
42 |
The percentage of students who chose incorrect responses due to Conceptual errors was similar across all grade levels and pass levels. However there were some differences in the Conceptual Error Content categories. As shown in Table 4, the most common Conceptual errors were in the areas of Number Sense and Pattern, Functions, and Algebra.
Table 4
Summary of Most Prominent Conceptual Error Content Area by Pass Level
Year |
6th |
7th |
8th |
Below Basic |
|
|
|
2007 |
NS |
NS |
NS |
2008 |
PFA |
G |
NS |
Basic |
|
|
|
2007 |
NS |
NS |
PFA |
2008 |
PFA |
NS G |
NS |
Proficient |
|
|
|
2007 |
NS |
NS |
PFA |
2008 |
PFA |
G |
PFA |
Advanced |
|
|
|
2007 |
M |
NS |
PFA |
2008 |
PFA |
G |
PFA |
Note: NS – Number Sense; G-Geometry; PFA – Patterns, Functions, Algebra;
M-Measurement
Discussion of Results
This study revealed that conceptual errors were a major issue among students of all ability levels and grade levels. Students do not know the meaning of various mathematical terms. Students lack clear and accurate concept images (Tall and Vinner, 1981). This study also provides evidence that students are unable to properly interpret or use mathematical symbols and lack rational number sense (Lamon, 2007).
This study provides evidence that many students are not successfully solving mathematics problems. Students taking the Virginia SOL assessments are selecting incorrect responses because they incorrectly analyze the conditions of the problem. Students are also not exercising self-monitoring and control. They are not looking back over the conditions of the problem, their solution path, and their result, to see if they have successfully found a solution that solves the problem given the conditions and the question that was asked.
Timeline
The presentation time will include a brief discussion of the research and results discussed in this paper along with sharing some items from the SOL assessments. During the roundtable discussion the audience will share what information is available in their states about student performance on state assessments and how the results are currently used to improve instruction. A standardized assessment for the Common Core Standards is currently in progress. The audience will discuss what information mathematics educators need from this assessment for it to be a catalyst for effective change at the state and national levels.
REFERENCES
Hill, H. C., Ball, D. B., & Schilling, S. G. (2008). Unpacking Pedagogical Content Knowledge: Conceptualizing and Measuring Teachers’ Topic-Specific Knowledge of Students. Journal for Research in Mathematics Education, 39(4), 372-400.
Kloosterman, P., & Lester, F.K., Jr. (Eds.). (2004). Results and Interpretations of the 1990-2000 Mathematics Assessment of the National Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics.
Kloosterman, P., & Lester, F.K., Jr. (Eds.). (2007). Results and Interpretations of the 2003 Mathematics Assessment of the National Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics.
Lamon, S. (2007). Rational Numbers and Proportional Reasoning: Toward a Theoretical Framework for Research. In Frank K. Lester, Jr. (Ed.), Second Handbook of Research on Mathematics Teaching and Learning, (pp. 629-667).New York, NY: Macmillan Publishing Company.
National Research Council. (1999). Testing, teaching, and learning. Washington, DC: Author.
No Child Left Behind Act of 2001, § 101, Pub. L. No. 107-110, 115 Stat. 1447 (2002).
Schoenfeld, Alan H. (1992). Learning to Think Mathematically: Problem Solving, Metacognition, and Sense Making in Mathematics. In Douglas A. Grouws (Ed.), Handbook of Research on Mathematics Teaching and Learning, (pp. 334-370). New York, NY: Macmillan Publishing Company.
Schoenfeld, Alan H. (1989). Problem Solving in Context(s). In R. Charles & E.Silver (Eds.), The Teaching and Assessing of Mathematical Problem Solving, (pp. 82– 92). Reston, VA: National Council of Teachers of Mathematics.
Tall, David., & Vinner, Shlomo. (1981). Concept Image and Concept Definition in Mathematics with Particular Reference to Limits and Continuity. Educational Studies in Mathematics, 12(2), 151-169.
Tate, William F., & Rousseau, C. (2007). Engineering Change in Mathematics Education. In F.K. Lester, Jr (Ed.), Second Handbook of Research on Mathematics Teaching and Learning (pp. 1209-1246). Charlotte, NC: Information Age Publishing.
Virginia Department of Education. (2001). Mathematics Standards of Learning for Virginia Public Schools. Retrieved from http://www.doe.virginia.gov/testing/sol/standards_docs/mathematics/index.shtml#previous_sol
Virginia Department of Education. (2005a). Virginia SOL Assessment: Grade 6 Mathematics Test Blueprint. Retrieved from http://www.doe.virginia.gov/testing/sol/blueprints/mathematics_blueprints/2001/blueprint_math6.pdf
Virginia Department of Education. (2005b). Virginia SOL Assessment: Grade 7 Mathematics Test Blueprint. Retrieved from http://www.doe.virginia.gov/testing/sol/blueprints/mathematics_blueprints/2001/blueprint_math7.pdf
Virginia Department of Education. (2005c). Virginia SOL Assessment: Grade 8 Mathematics Test Blueprint. Retrieved from http://www.doe.virginia.gov/testing/sol/blueprints/mathematics_blueprints/2001/blueprint_math8.pdf
Virginia Department of Education. (2007). Virginia SOL Assessments: Technical Report 2006- 2007 Administration Cycle. Richmond, VA: Author.
Virginia Department of Education. (2008). Virginia SOL Assessments: Technical Report 2007- 2008 Administration Cycle. Richmond, VA: Author.
Virginia Department of Education. (2011). Regulations Establishing Standards for Accrediting Public Schools in Virginia. Retrieved from http://www.doe.virginia.gov/boe/accreditation/stds_archive/soa_2010.pdf
Analysis generated explanations for why students selected incorrect answers on Virginia Standards of Learning mathematics assessments. These explanations, combined with answer-choice frequencies, revealed students primarily selected incorrectly due to a failure to analyze problem conditions, ineffective self-monitoring, or conceptual errors.
Session Type: Poster Session