Paper

A Summary of Project Efforts to Examine the Impact of the LSC on Student Achievement

Project 3 (6- 12 Mathematics)

Serving schools in several states in its geographic region, the project finds it difficult to collect common data from the schools it serves. Thus, the project has encouraged each school in their LSC to undertake its own study. The project provided results from five of the thirty- three participating schools ³ each compared students in classes using LSC designated materials (IMP)

to students in traditional mathematics classes. The schools used a variety of instruments including the mathematics portion of the Scholastic Achievement Test (SAT), a state mandated mathematics assessment, the New Standards Reference Exam (NSRE), and the Terra Nova. One school also used a self- developed problem- solving test.

The first school categorized students into four groups, advanced IMP, regular IMP, advanced traditional, and regular traditional, though there was no information given as to how the advanced/ regular distinction was made. While this distinction may have been intended to control for initial differences in student ability levels, not enough information is provided to judge whether this goal was accomplished. The school found that, at both levels, students in IMP classes had higher mathematics SAT scores than students in non- IMP classes (see Table 3).

The study also shows that IMP students had higher verbal SAT scores than non- IMP students (see Table 4). While the school may have provided these data to show impact on verbal SAT (given the writing intensive nature of IMP), it raises the question about initial differences between the two groups of students, as there was only a marginal control for initial differences in student abilities. No significance testing was reported.

The second school compared IMP and traditional students using the mathematics portion of the SAT and the state mathematics assessment. As can be seen in Table 5, students in IMP classes outperformed students in basic and standard level mathematics courses, but not those in honors courses (the IMP students were roughly 75% standard level students and 12.5 % each of basic and honors, though no information was provided as to how this categorization was made). They also showed that IMP juniors outperformed non- IMP juniors on a school- developed problem solving test (an average score of 3.4 out of 5 for IMP students compared to 2.5 for non- IMP students). No significance testing was reported.

The next school administered the Terra Nova test to all 9^th graders in the spring of 1996 (the end of the first year of IMP implementation). The trend was for IMP students to score higher than non- IMP students in all nine strands, with the differences statistically significant in five of the nine. There were no controls for initial differences between the two groups of students.

The fourth school compared mathematics SAT scores of IMP and non- IMP students, grouping the students in two levels - honors and regular. IMP students scored higher at both levels than did non- IMP students (see Table 7). No significance testing was reported and not enough information is provided to judge whether IMP and non- IMP students were initially equivalent.

The final school compared IMP students to non- IMP students using scores from the NSRE administered at the 10^th grade. It is important to note that, with three exceptions for highly

motivated students, the IMP course was offered only to students who had a raw SAT score of 40 or higher (though they could choose a traditional mathematics class). To compensate for initial differences between IMP students (high achievers and self- selected) and non- IMP students, the school constructed three matched- pair samples to use as comparison groups. They then used an ANOVA to show that there were no significant differences on the 8^th grade SAT scores among the four groups, allowing them to combine the three control groups into one large control group.

The results of the school's comparisons of IMP and non- IMP students can be seen in Table 8. On the 10^th grade NSRE, the 33 IMP students scored significantly higher than the 99 students in the

control group on the 10^th grade Math Scale Score (a difference of .57 standard deviations). They

also found that the IMP students scored higher than the control group on the 10^th grade SAT- 9

Scale Score (an effect of .54 standard deviations). There were no differences among the groups on the Math Concepts scale, but IMP students scored higher on the Math Skills and Math Problem Solving scales (effect sizes of .75 and .56 respectively).