Paper

A Summary of Project Efforts to Examine the Impact of the LSC on Student Achievement

Table 12 shows that 7^th grade students who had LSC trained teachers for two or three years scored about 3 percentage points (. 14 standard deviations) higher than 7^th graders who had a LSC teacher for one year or less.

Project 8 (K- 8 Science)

This study compared results on the SAT- 9 Science Open- ended assessment for two matchedpairs of schools (two schools that had participated in LSC professional development and two schools that had not participated). It is unclear on which variables the control schools were selected. Slightly more than 100 4^th grade students were tested at each school; twice as many 5^th graders were tested. Participation was defined as having a school- wide average of at least three kit trainings per teache r. Overall, there were few differences detected between students at the LSC treated schools and the non- treated schools, with the two exceptions being at the 4^th grade level. One was that 4^th grade students at one treated school outperformed the students of the matching school (see Table 13). The other exception was that 4^th grade students at both treated schools outperformed the untreated schools' students on the Problem- Solving and Decision-Making sub- scale (see Table 14). There were no differences on the other five sub- scales.

While this study used a matched sample to control for initial differences in student ability, the relatively small sample sizes reduce the study's chances of detecting differences between the control and experimental groups with this design. Further information regarding how control schools were selected and how initial equivalency of students was determined would strengthen this study.

Project 9 (K- 8 Science)

The school district administered the Stanford Achievement Test - 9^th edition, Form T to all 4^th and 6^th grade students. They analyzed only the scores of students who had been enrolled in the district for the past four years (around 630 at each grade level), allowing the m to compare students who had and had not been exposed to their LSC science program. Mean percentile rankings were presented for each grade level. Neither standard deviations nor standard errors were included in the report and no statistical tests were used to compare group means.

The data appear to show, at both grade levels, that students of teachers who participated in the LSC professional development and used the LSC designated instructional materials during the 1998- 99 school year scored higher on the SAT- 9 than did students of teachers who did not participate. The data were then further disaggregated by the number of years (from zero to four) students had teachers who participated in the district's LSC. The data appear to show a stair- step increase in student performance on the SAT- 9. As can be seen in Table 15, the mean score increases with each additional year of having a LSC trained teacher.

The project also examined pass rates on the 6^th grade writing proficiency test with the hypothesis that the writing- intensive nature of the science program would improve students' writing abilities. Student scores were disaggregated in the same two ways as with the science scores, with similar results found (see Table 16). Students of teachers who participated in the LSC during the 1998- 99 school year had a higher passing rate than stud ents of teachers not participating. Further, a similar stair- step pattern emerges, as with the science data, when the data are broken down by the number of years the students had a LSC participating teacher, although there is no difference in pass rates for students in the 3 and 4 years groups. HRI was able to run significance tests on these differences ⁵ and found that students of participating teachers did pass the writing proficiency test at a higher rate than did students of nonparticipating teachers.

While these data appear promising, there are some dangers to drawing conclusions regarding the impact of the LSC in the district. First, with no standard deviations reported, it is impossible to judge the magnitude of the differences. Second, questions remain regarding how teachers were selected to participate in the LSC's professional development. Were schools targeted on a cohort basis? If so, were the original targeted schools high performing than schools targeted in later years? Or were participating teachers volunteers, and perhaps more enthusiastic about teaching science? Finally, while the results of the writing proficiency test are reported to show a crossover effect, such assessments are commonly used as ability measures to control for initial differences. Hence, the trend of higher SAT- 9 scores with increased years of participation could be explained by initial differences in student ability levels as measured by the writing proficiency test.

Conclusions and Recommendations

Given the limited information provided in many of these studies, one must interpret their results with extreme care. Table 17 summarizes the results of these studies and provides a rough measure of each study's internal validity. At first glance, it appears that in both mathematics and science, the LSCs are having a positive impact on student achievement. All of the mathematics and three of the four science projects show increases in student performance. However, many of the studies do not present enough information to build a convincing case that the LSC was responsible for improved student achievement. Because of this, it impossible to judge with any certainty whether the results from these studies are real or spurious, due to factors other than the LSC or perhaps simply artifacts of the study's methodology. The most common threats to internal validity in these studies were:

• Lack of a control group - for example, the study reported gain scores for schools in the LSC, but not for schools outside of the LSC.
• Failure to account for initial differences between control and experimental groups - while the study may have reported that LSC students scored higher than non- LSC students, it was unclear as to whether the two groups started at the same achievement level.
• Sample selection bias - the study did not address how teachers were selected for participation in LSC training and whether this may have affected the study's results.

While it is impossible to generalize to the LSC program as a whole given the small number of studies made available to HRI, it is encouraging that all five studies rated as solid or strong found positive impacts on student achievement. Each of these five studies makes a defensible case that the gains are attributable to their LSC. As more of the individual LSCs undertake and complete studies of quality comparable to these, the stronger the case can be made as to the LSC program's impact on students.

Given that 8 out of the 13 studies, either through omission of data or poor research design, did not present enough evidence to make their results credible, NSF may want to consider offering the LSCs additional support for conducting studies of student outcomes. For projects that have research and evaluation experts on staff, a set of criteria or guidelines that communicate NSF's information needs should be sufficient. Other projects will require some form of technical assistance, ranging from small doses to help refine research plans to extensive assistance in design research studies and analyzing data. NSF may want to consider offering a conference to help projects articulate their information needs, raise their awareness of key issues in research design, and become savvier consumers of technical assistance.

¹ When comparing percents, the effect size is calculated using the difference between the arcsine transformation of the percents of the two groups. For means, the effect size is calculated as the difference between the group means, divided by the standard deviation of the population. Following standard conventions, effect sizes of .2 are considered small effects, .5 medium effects, and .8 large effects (Jacob Cohen, Statistical Power Analysis for the Behavior Sciences, Hillsdale, NJ: Lawrence Erlbaum Associates, 1988).

² Statistically significant differences (p < .05) are noted with an asterisk.

³ HRI does not know whether these were the only schools conducting studies or if the project chose to send only these results to HRI.

⁴ Data for the other sub- scales were not included in the report sent to HRI.

⁵ To statistically test for differences in percent passing the writing test, all that is required is the number in each group and the percent passing, both of which were provided in the report sent to HRI. To test for differences in the mean national percentile rankings, the standard deviation or standard error of the mean is required. This information was not included in the project's report.

⁶ A strong study controls for most threats to internal validity and provides enough evidence for the results to be compelling. A solid study controls for many threats to internal validity, and although some flaws in methodology or analysis remain, the results are credible. Studies categorized as "indeterminate" did not provide enough information to make a persuasive argument as to the credibility of the results

⁷ Project 9 provided HRI with a preliminary write- up of their results. While the study design appears solid, HRI does not have enough information to make a fully- informed judgement on the study's validity.