Potential Impact of the XYZ Program on Student Achievement

Theoretical Framework

Research has shown that when today's eighth-grade students make the transition into high school, they confront a number of issues that can affect their later academic performance (Neild, Stoner-Eby, & Furstenberg, 2008). In addition to external factors, students' internal theories about the nature of intelligence—linked to their academic confidence, motivation, and willingness to pursue various problem-solving strategies—can have a positive or negative effect on later high school success (Blackwell, Trzesniewski, & Dweck, 2007; Roderick, 2003; Schullo & Alperson, 1998).

In recent years, several interventions have demonstrated success in increasing student achievement by targeting students' internal beliefs. For example, Blackwell, Trzesniewski, and Dweck (2007) found that middle school students' mathematics achievement increased dramatically after receiving 8 sessions of instruction concerning the direct effect of effort on brain growth as compared to the control group who received instruction solely on study skills. In a similar study involving college-aged students, Walton and Cohen (2007, 2011) found that, after a 1-hour laboratory session targeting students' sense of belonging, students earned higher grade point averages in future years than did the control group that received information not related to belonging.

These interventions aimed to influence students' attributions or, in other words, the ways students explain the causes of their successes or failures (Kelley, 1973; Weiner, 1986). For example, students can attribute academic setbacks as a result of a permanent lack of ability and may not work to improve their performance as a result. These types of attributions may contribute to a cycle of negative attributions and poor performance (cf. Storms & Nisbett, 1970). Conversely, students can view setbacks as temporary and can work to improve their performance.

The XYZ program was designed to encourage positive attributions by students in an effort to impact their academic performance. Unlike the studies mentioned above, however, XYZ targeted students that were making the critical transition from middle school to high school Algebra I. XYZ is a yearlong program that involves both a summer and academic year component. XYZ students attend a 3-week summer program co-taught by two Algebra I teachers; XYZ students are then assigned to one of the two co-teachers for their Algebra I course during the academic year. During the summer program, the XYZ program delivers a unique curricular integration of Algebra I and youth development concepts as well as research-based lessons on the nature of learning itself; these concepts are revisited periodically during the academic year.

The purpose of this study was to determine (a) if, after participating in the program, XYZ students made gains in their Algebra I achievement and (b) if these gains were greater than gains seen by students that did not participate in the program during the 2010-2011 academic year.

Method

            The current study used a quasi-experimental pre-post design. The treatment group was comprised of students involved in the XYZ program; the control group represented students that were not in the XYZ summer program, but may or may not have been in an XYZ teacher's classroom during the academic year. Because students elected whether or not to enroll in the XYZ program, we were limited to use of convenience sample for analysis. We expect that the convenience sample could cause some bias in the data, but results may still offer a reasonable approximation of effect.

Data Source

            Achievement data were collected from students in a rural northern Texas school district via a pre- and post-test of Algebra I content. The 24-item assessment was created by  Center staff using items that had previously been validated with high school students. Item analyses from the current data were consistent with those from prior administrations. The pre-test was administered to the treatment group at the beginning of the summer XYZ program. The same pre-test was administered to the treatment group as a post-test measure and the control group as a pre-test at the beginning of the academic year. Both the treatment and control groups then took a delayed post-test at the end of the academic year.

Results

            To determine changes in the achievement of the XYZ students, we ran a repeated measures analysis of variance (RM ANOVA) across three time points – (1) pre-test (beginning of summer program), (2) post-test (beginning of academic year), and (3) delayed post-test (end of academic year). Results showed statistically significant gains, Wilks' l=.35, F (2,15) = 14.15, p<.001, partial h²=.65. Overall, XYZ students (n=17) saw an average of a 1.08 point (M_pre=8.65, SD_pre=2.40; M_post=9.71, SD_post=3.57)  gain out of 24 total points from pre-test to post-test and even sharper increases from post-test  to delayed post-test (M_post=9.71, SD_post=3.57; M_delayed=14.06, SD_delayed=3.25).

            We sought to determine if these gains were greater or less than those observed with the control group whose members did not participate in the XYZ program. A RM ANOVA was again used, but this time with a between-subjects factor of treatment/control. The interaction effect of pre-post by treatment/control was not statistically significant, Wilks' l=.97, F (1,37) = 1.025, p=.32, partial h²=.03. As examination of the effect size demonstrates, the effect was quite small. However, this result could be an artifact of the small sample size (n_treatment=17, n_control=22) and should not be disregarded as inconsequential. As can be seen in Figure 1, the average treatment gains were descriptively greater than the gains seen by the control group. At the very least, the result warrants further investigation, preferably with a larger sample size.

Figure 1.  Average Achievement Scores for Treatment and Control Groups at Time 1 (Pre-test) to Time 2 (Delayed Post-test) (n=38)

Scientific Importance of Research

            These results are encouraging as to the impact that social-emotional interventions can have on students' achievement. While the sample is somewhat small, the results nonetheless indicate that students in the XYZ program made significant gains in their mathematics achievement from when they began the XYZ program to the end of the following academic year. While our analyses did not show a statistically significantly different growth rate between treatment and control groups from pre- to delayed-post, the result should not be disregarded. Descriptively, these results are promising and warrant further analysis to investigate the differences in growth between groups.

References

Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of

intelligence predict achievement across an adolescent transition: A

longitudinal study and an intervention. Child Development, 78(1), 246-263.

Kelley, H.H. (1973). The process of causal attribution. American Psychologist, 28, 107-128.

Neild, R., Stoner-Eby, S., & Furstenberg, F. (2008). Connecting entrance and departure:

The transition to ninth grade and high school dropout. Education and Urban

Society, 40(5), 543.

Roderick, M. (2003). Aligning dropout prevention approaches with efforts to raise achievement and improve high school performance: Evidence from the Chicago Public Schools. Chicago: The Consortium on Chicago School Research.

Schullo, S., & Alperson, B. (1998, April). Low SES Algebra 1 students and their

teachers: Individual and a bi-directional investigation of their relationship and

implicit beliefs of ability with final grades. Paper presented at the Annual

Meeting of the American Educational Research Association. San Diego, CA.

Storms, M.D., & Nisbett, R.E. (1970). Insomnia and the attribution process. Journal of Personality and Social Psychology, 16, 319-328.

Walton, G.M., & Cohen, G.L. (2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92, 82-96.

Walton, G.M., & Cohen, G.L. (2011).  A brief social-belonging intervention improves academic and health outcomes among minority students. Science, 331, 1447-1451.

Weiner, B. (1986). An Attribution Theory of Emotion and Motivation. New York: Springer-Verlag.