Research Overview

This research study generated likely explanations why students selected incorrect answer choices on the Virginia Standards of Learning (SOL) mathematics assessments and what if any differences existed among students at different pass levels.  The 2007 and 2008 middle grade multiple-choice assessments were used for this analysis.  Each assessment contained 50 items spread across five Reporting Categories. 

The Virginia Department of Education (VDOE) provided frequencies for each distractor by pass level.  Qualitative data was generated by Document Analysis Teams (DATs) who generated a list of possible explanations for each distractor.  Each grade level DAT consisted of a mathematics teacher, a special education teacher, and a mathematics educator. 

The DAT explanations were categorized based on the Categorization Framework for Distractor Explanations (see Figure 1).  This structure resulted from a merging of Schoenfeld’s Knowledge Base (1992), the framework used for reporting results on the NAEP by Kloosterman et al (2007, 2004), and the VDOE Reporting Categories.

Figure 1. Categorization Framework for Distractor Explanations

 

Excel was then used to record the categorizations for each distractor alongside the frequencies by pass level so that Pivot tables could be used to interpret the data.  The Data Analysis Process is summarized in Figure 2.

 

Figure 2.  Data Analysis Process

 

                                                   Educational Significance                                                  

            No Child Left Behind (NCLB) greatly increased the high-stakes nature of state assessments by requiring all students reach at least the proficient level in mathematics by 2014 (NCLB, 2002, 115 STAT. 1447).  Teachers and principals need specific information about why student select incorrect responses.  “The assumption is that armed with quantitative data, local leadership – teachers, mathematics coordinators and supervisors, principals, assistant superintendents, superintendents, and school board members – will proactively respond to data” (Tate & Rousseau, 2007, p. 1211).  However, the reality is that “many schools lack an understanding of the changes that are needed and lack the capacity to make them” (NRC, 1999, p. 5).

 

 

Research Questions and Results

Research Question #1

What are the most likely explanations of middle school students’ incorrect distractor choices when responding to multiple choice items on the Mathematics Virginia Standards of Learning (SOL) assessment? 

            Conceptual Errors and Problem Solving Errors were the primary categories of error among middle school students (see Table 1). 

Table 1

Percentage of All Student Responses

Response Categorization	6th	7th	8th
Correct Responses
2007	71	66	76
2008	75	68	78
Conceptual Errors
2007	12	20	12
2008	11	14	9
Procedural Errors
2007	4	3	3
2008	3	5	3
Problem Solving Errors
2007	12	10	8
2008	11	12	9
Unknown Errors
2007	1	1	1
2008	0	1	1
Total
2007	100	100	100
2008	100	100	100

           

            Conceptual errors accounted for more of the incorrect responses than Problem Solving errors.   However, there were differences in the most common Conceptual error for each grade level and year (see Table 2). 

 

Table 2

Summary of Most Common Category of Conceptual errors:  All Students

Year	6th	7th	8th
2007	NS	NS	PFA
2008	PFA	G	PFA

Note:  NS – Number Sense; G-Geometry; PFA – Patterns, Functions, Algebra

Research Question #2

How do the incorrect distractor choices differ among students who pass advanced, pass, fail basic, and fail below basic on the Mathematics SOL?

            Although the percentage of responses which are incorrect was different for each pass level, the percentage of incorrect responses due to Problem Solving Errors was similar regardless of grade level and pass level (see Table 3). 

 

 

 

 

 

 

 

 

Table 3

Percentage of Incorrect Student Responses due to Problem Solving Errors

Grade Results by Pass Level		2007	2008
Below Basic
6th		43	42
7th		28	37
8th		33	34
Basic
6th		42	43
7th		29	37
8th		35	38
Proficient
6th		40	42
7th		27	36
8th		34	41
Advanced
6th		41	42
7th		25	35
8th		32	42

 

The percentage of students who chose incorrect responses due to Conceptual errors was similar across all grade levels and pass levels.  However there were some differences in the Conceptual Error Content categories.  As shown in Table 4, the most common Conceptual errors were in the areas of Number Sense and Pattern, Functions, and Algebra. 

 

 

 

 

 

Table 4

Summary of Most Prominent Conceptual Error Content Area by Pass Level

Year	6th	7th	8th
Below Basic
2007	NS	NS	NS
2008	PFA	G	NS
Basic
2007	NS	NS	PFA
2008	PFA	NS G	NS
Proficient
2007	NS	NS	PFA
2008	PFA	G	PFA
Advanced
2007	M	NS	PFA
2008	PFA	G	PFA

Note:  NS – Number Sense; G-Geometry; PFA – Patterns, Functions, Algebra;

M-Measurement

 

Discussion of Results

This study revealed that conceptual errors were a major issue among students of all ability levels and grade levels.  Students do not know the meaning of various mathematical terms.  Students lack clear and accurate concept images (Tall and Vinner, 1981).   This study also provides evidence that students are unable to properly interpret or use mathematical symbols and lack rational number sense (Lamon, 2007). 

This study provides evidence that many students are not successfully solving mathematics problems.  Students taking the Virginia SOL assessments are selecting incorrect responses because they incorrectly analyze the conditions of the problem.  Students are also not exercising self-monitoring and control.  They are not looking back over the conditions of the problem, their solution path, and their result, to see if they have successfully found a solution that solves the problem given the conditions and the question that was asked. 

Timeline

            The presentation time will include a brief discussion of the research and results discussed in this paper along with sharing some items from the SOL assessments.  During the roundtable discussion the audience will share what information is available in their states about student performance on state assessments and how the results are currently used to improve instruction.  A standardized assessment for the Common Core Standards is currently in progress.  The audience will discuss what information mathematics educators need from this assessment for it to be a catalyst for effective change at the state and national levels.

                                                                                                           

 

 

 

 

 

 

 

 

REFERENCES

 

Hill, H. C., Ball, D. B., & Schilling, S. G. (2008). Unpacking Pedagogical Content Knowledge: Conceptualizing and Measuring Teachers’ Topic-Specific Knowledge of Students. Journal for Research in Mathematics Education, 39(4), 372-400.

 

Kloosterman, P., & Lester, F.K., Jr. (Eds.). (2004). Results and Interpretations of the 1990-2000 Mathematics Assessment of the National Assessment of Educational Progress.   Reston, VA:  National Council of Teachers of Mathematics.

 

Kloosterman, P., & Lester, F.K., Jr. (Eds.). (2007). Results and Interpretations of the 2003 Mathematics Assessment of the National Assessment of Educational Progress.   Reston, VA:  National Council of Teachers of Mathematics.

 

Lamon, S. (2007). Rational Numbers and Proportional Reasoning: Toward a Theoretical Framework for Research.  In Frank K. Lester, Jr. (Ed.), Second Handbook of Research on Mathematics Teaching and Learning, (pp. 629-667).New York, NY: Macmillan Publishing Company.

 

National Research Council. (1999). Testing, teaching, and learning. Washington, DC: Author.

 

No Child Left Behind Act of 2001, § 101, Pub. L. No. 107-110, 115 Stat. 1447 (2002).

 

Schoenfeld, Alan H. (1992). Learning to Think Mathematically:  Problem Solving, Metacognition, and Sense Making in Mathematics.  In Douglas A. Grouws (Ed.), Handbook of Research on Mathematics Teaching and Learning, (pp. 334-370). New York, NY: Macmillan Publishing Company.

 

Schoenfeld, Alan H. (1989).  Problem Solving in Context(s). In R. Charles & E.Silver (Eds.), The Teaching and Assessing of Mathematical Problem Solving, (pp. 82– 92).  Reston, VA: National Council of Teachers of Mathematics.

 

Tall, David., & Vinner, Shlomo. (1981). Concept Image and Concept Definition in Mathematics with Particular Reference to Limits and Continuity.  Educational Studies in Mathematics, 12(2), 151-169.

 

Tate, William F., & Rousseau, C. (2007). Engineering Change in Mathematics Education.  In F.K. Lester, Jr (Ed.), Second Handbook of Research on Mathematics Teaching and Learning (pp. 1209-1246). Charlotte, NC: Information Age Publishing.

 

 

 

Virginia Department of Education. (2001). Mathematics Standards of Learning for Virginia Public Schools. Retrieved from http://www.doe.virginia.gov/testing/sol/standards_docs/mathematics/index.shtml#previous_sol

 

Virginia Department of Education. (2005a). Virginia SOL Assessment: Grade 6 Mathematics Test Blueprint.  Retrieved from http://www.doe.virginia.gov/testing/sol/blueprints/mathematics_blueprints/2001/blueprint_math6.pdf

 

Virginia Department of Education. (2005b). Virginia SOL Assessment: Grade 7 Mathematics Test Blueprint.  Retrieved from http://www.doe.virginia.gov/testing/sol/blueprints/mathematics_blueprints/2001/blueprint_math7.pdf

 

Virginia Department of Education. (2005c). Virginia SOL Assessment: Grade 8 Mathematics Test Blueprint.  Retrieved from http://www.doe.virginia.gov/testing/sol/blueprints/mathematics_blueprints/2001/blueprint_math8.pdf

 

Virginia Department of Education. (2007). Virginia SOL Assessments: Technical Report 2006- 2007 Administration Cycle.  Richmond, VA: Author.

 

Virginia Department of Education. (2008). Virginia SOL Assessments: Technical Report 2007- 2008 Administration Cycle.  Richmond, VA: Author.

 

Virginia Department of Education. (2011). Regulations Establishing Standards for Accrediting Public Schools in Virginia. Retrieved from http://www.doe.virginia.gov/boe/accreditation/stds_archive/soa_2010.pdf