Paper

LSC Year Three Cross-Site Report

Teachers' perceptions of impact come from two sources-- teacher questionnaires and interviews-- the most comprehensive of which is the teacher questionnaire. As noted earlier, many of the teacher questionnaire items have been combined to form composites.¹ These composites group together responses to related questions. For example, the "Attitudes Toward Teaching" composite combines questionnaire data on how teachers feel about reform- minded instructional strategies. This grouping of responses into composites allows the data to be analyzed thematically, rather than item-by-item.

Most of the LSC projects have identified a subset of the targeted teacher population as lead teachers. These teachers are likely to be in the group who report most intensive involvement in the LSC, but it is likely that many of them were already fairly aligned with the LSC vision prior to their participation in the project. To avoid biasing the results, teachers who indicated they have been designated as lead teachers in the LSC have been omitted from the comparisons of teacher attitudes and classroom practices by level of participation in the LSC.

This section shows the results by level of participation for each of the composites relevant to classroom impact. As noted in Chapter Two, K- 8 teachers who have and have not participated in the LSC professional development are similar both demographically and in terms of course background. Accordingly, comparisons of the results between these two groups are used as an indication of the effect of the LSC projects on teachers and teaching.²

Composite: Attitudes Toward Teaching

The "Attitudes Toward Teaching" composite describes participants' beliefs about a number of reform-oriented strategies. It includes items such as the perceived importance of:

• Providing concrete experience before abstract concepts;
• Developing students' conceptual understanding of mathematics/science;
• Having students participate in appropriate hands- on activities; and
• Engaging students in inquiry- oriented activities.

As can be seen in Figure 28, teacher beliefs are quite well- aligned with the tenets of mathematics and science reform, and they remain fairly constant regardless of hours of LSC professional development. While the results show a small positive effect for K- 8 science and a small negative effect for K- 8 mathematics, LSC involvement appears to have relatively little impact on teachers' already positive attitudes toward the reform strategies.

Composite: Pedagogical Preparedness

While the previous composite focused on the perceived importance of various reform strategies, this composite reflects how prepared teachers feel to implement reform strategies. These include instructional strategies such as cooperative learning and use of manipulatives, as well as recognizing and responding to student diversity.

Comparisons by treatment level, shown in Figure 29, indicate that teachers who participated in LSC professional development report increasingly high levels of pedagogical preparedness. The 3 percentage points difference between untreated and highly treated K- 8 mathematics teachers constitutes a small effect (. 23 standard deviations), while the 8 percent difference for K- 8 science (. 60 standard deviations) is a medium effect.

Composite: Content Preparedness

The mathematics and science versions of this composite reflect how well prepared teachers feel to teach a number of topics within their discipline. As can be seen in Figures 30 and 31, teachers who have participated in LSC professional development are more likely to report feeling well prepared in the various content areas; however, due to fairly large standard deviations for these variables, the effect sizes are small.

Composite: Investigative Culture

The "Investigative Culture" composite examines strategies used by teachers to facilitate exploration and investigation on the part of students, including such practices as:

• Arranging seating to facilitate student discussion;
• Using open- ended questions;
• Requiring students to supply evidence to support their claims; and
• Encouraging students to consider alternative explanations.

Again, K- 8 science teachers show a steady increase in this area with level of treatment. (See Figure 32.) The nine- point difference between the untreated and highly treated K- 8 science teachers constitutes a medium effect, while the one point difference for K- 8 mathematics represents a very small effect size.

Composite: Investigative Practices

The "Investigative Practices" composite looks more closely at what students actually do in the classroom. It includes such instructional strategies as having students:

• Engage in hands- on mathematics/ science activities;
• Work on models or simulations;
• Work on extended investigations; and
• Write reflections in a notebook or journal.

As can be seen in Figure 33, the means for this composite tend to be lower than in other areas, likely reflecting the fact that while the classroom may always be set up to facilitate group work, even the most highly investigative classes do not do field work or design their own investigations on a daily basis. Participation in the LSC appears to have a fairly substantial impact on classroom practice in K- 8 science (. 69 standard deviations), and a smaller impact in K- 8 mathematics (. 32 standard deviations).

Composite: Investigative Practice, Use of Calculators and Computers

The factor analysis yielded a composite on the frequency of use of calculators and computers in mathematics instruction:

• For learning or practicing skills;
• To develop conceptual understanding; and
• As a tool, e. g., spreadsheets, data analysis.

Two results in Figure 34 are noteworthy. First, comparing only mathematics teachers who have not yet participated in the LSC, calculator and computer usage is much higher in grades 7- 12 (69 percent of total possible points) than in grades K- 8 (49 percent). Second, K- 8 mathematics teachers who have participated in at least 40 hours of LSC professional development are more likely to use calculators and computers in their instruction than are their untreated peers. (The 10 percentage point difference between the two groups is .47 standard deviations, categorized as a medium effect.)

Table 20 summarizes the effect sizes for the classroom impact composites, using 0 for no effect, a single "+" to indicate a small positive effect, and so on. Note that the LSC professional development appears to have had a marked impact on K- 8 science teachers, with small effects on 2 of the composites, and medium effects on the other 3, including both establishing an investigative culture and implementing investigative practices in the classroom. The impact on K- 8 mathematics teachers appears less substantial, with small effects on four composites, a medium effect only on use of calculators and computers, and a small negative effect on teacher attitudes towards reform.

Teacher Interviews

As described earlier, the LSC core evaluation uses interviews to collect more in- depth information than provided in the teacher questionnaires, but from a smaller number of people. Approximately 10 teachers from each Cohort 1 and Cohort 2 project were interviewed about their beliefs, level of confidence, and the way they approached teaching science or mathematics prior to the LSC, as well as the impact of the LSC in these areas.

When asked about the impact of the LSC, most of the teachers were able to provide specific examples of ways in which the program had affected their teaching. Interviewers interpreted these comments and rated the impact of the LSC on a five- point scale from "1, little or no impact" to "5, extensive impact." Table 21 shows the percentage of teachers with varying levels of involvement in the LSC describing extensive positive impact (rated level 4 or 5 by the interviewer).

Two facts about this table are noteworthy. First, the majority of teachers interviewed indicated that the LSC had had considerable impact, particularly in the areas of curriculum and instruction.

Second, the amount of impact generally increases with increasing time spent involved in LSC activities. It is also interesting to note that impact on leadership ability seems to depend on having a high level of involvement in the LSC.

Teachers' descriptions of their former and current practices highlight the impact LSC projects are having on many teachers.

I just taught multiplication to my second graders and rather than teach them the multiplication formulas, I let them explore multiplication first.

* * *

These activities have provided me ongoing science instruction and really heightened my interest in science. I had no clear direction before. I feel more knowledgeable and secure. I know what to teach and there is a plan.

* * *

I have learned that I should not automatically give answers. I now do more questioning. I used to teach for the right answer!

However, not all respondents considered the LSC impact to be a positive one. The following comments illustrate some of the difficulties a number of teachers are experiencing.

I teach it because it is the mandated curriculum. The kids in my class are not getting it. Problem solving is new and hard for them.... I'd like to see more of the basics.

* * *

The workshops give teachers strategies for implementing inquiry science in the classrooms, but it's in the classroom during actual implementation where the weaknesses of the training are seen and need to be addressed. I realize that a lot of strategies and techniques discussed in the workshops don't work in the classroom.

Principals' Impressions of Impact

All principals in schools targeted by LSC projects were asked to complete questionnaires. Among other items, they were asked to rate: (1) the level of their school's current involvement in the LSC, (2) the extent to which their teachers were using the LSC- specified instructional materials, and (3) the extent to which teachers were using the LSC- advocated instructional strategies with their students. When either the materials or strategies variable is plotted against level of involvement, the results are seen to be highly correlated: the higher the level of the school's involvement in the LSC, the more likely the teachers will be using exemplary materials and teaching strategies.³

Principals were also asked to describe their school's progress in moving toward excellence in mathematics and science education. Slightly more than half of the principals involved in K- 8 mathematics and science projects in Cohort 1 and Cohort 2 felt that their schools were at least "well along in improving" their science education.

Impact as Measured by Classroom Observers

By looking at the synthesis and capsule ratings of the three different cohorts, it is possible to get an indication of impact of the LSC programs. While Cohort 1 and Cohort 2 contained teachers that have received at least 20 hours of LSC professional development, Cohort 3 teachers have not yet received 20 hours of professional development. As can be seen in Figure 35, lessons of teachers in all cohorts were rated similarly for the quality of their implementation. Cohort 1 and Cohort 2 teachers were rated higher in all other aspects than were Cohort 3 teachers.

Lead Evaluators' Analyses of Impact

Each project's lead evaluator submits a core Evaluation Report to Horizon Research, Inc. every year. These reports make clear that different projects are dealing with distinct populations of teachers and are beginning at different points. To quote from one report:

[In many ways,] teachers are just like students, all over the place in their abilities and attitudes. What, of course, reform efforts of all kinds have substantiated in recent years is that teachers are not "like" students-- they are students, adult students and what is known about learning applies to them as well as to young people. No longer would anyone expect all children of a certain age to learn to read or to add fractions in exactly the same way at the same time, in lock- step fashion.... Both teachers' attitudes and beliefs, and their behaviors, are not going to be uniform at this or any point in the project.

Although teachers enter the LSC projects with a wide range of abilities and knowledge, evaluators frequently commented on participants' growing self confidence as they engage in LSC activities. Most Cohort 1 and Cohort 2 evaluators also reported that there had been impact on teachers' classroom practices, pointing out that they noticed frequent usage of the designated materials in both mathematics and science. However, they often observed a rather mechanical level of implementation of activities. An evaluator of a science project described the situation as follows:

[Teachers] are using the kits but are not yet at the point of selecting and tailoring appropriate kit activities to meet their goals or being able to extend beyond the preplanned suggestions that come with the kits. In a sense some teachers are using the kits with a "recipe" approach.... One positive effect of the program is that teachers recognize the difference between hands- on activity in the science classroom and authentic, rigorous investigation. They realize that they are becoming more capable in the hands- on part of science exploration, but they also recognize that they are less fluent with the minds- on part of exploration.

Another project evaluator described teachers as being at the "getting- it- going" level of classroom implementation, and not yet to the next level of "getting- it- good." Similarly, a project evaluator commented:

It appeared to us that many teachers were working through the [mathematics] lessons carefully and thoroughly, step by step.

These comments emphasize that many teachers are conscientiously trying the new materials and strategies but have yet to grasp the larger conceptual picture. This situation points to the need for continued mentoring and support to encourage teachers to see beyond the mechanics of implementation.

Specific areas of classroom practice mentioned by evaluators as being in need of attention include lesson closure or "sense- making," reflection by students, portraying science as a dynamic discipline, and making connections to other disciplines. One evaluator commented that although science was portrayed as dynamic and investigative in the professional development sessions, this conception was not passed on to students. This "dilution effect" was mentioned by several other evaluators as well, including comments that the professional development experience affected the lead teachers' classroom practices much more than that of the targeted teachers.

A concern noted by several evaluators as limiting the impact of the LSC was testing policies at both state and district levels, particularly in mathematics. Evaluators reported that many teachers felt the need to revert to more traditional teaching practices to prepare their students to do well on these standardized tests. This tension pulls at teachers: they are held accountable, in many cases, for how well their students perform in relation to others on these norm- referenced exams. Yet covering the curriculum to be tested may not support the more in- depth investigations advocated by the LSC.

Comparison of Impact Data

Data from a wide variety of sources-- at the individual teacher level, at the school level, and at the project level-- all suggest that the LSC projects are having a positive impact on both teachers' feelings of competence to teach mathematics/ science and their ability to actually do so at the classroom level. LSC- supported curricula and methods are being implemented across a wide spectrum of classrooms, schools, and districts. Principals, teachers, and evaluators agree on this. Differences of opinion occur at a finer level and center around the nature of the impact.

As noted earlier, the majority of principals view their schools as being well along in reform. This contrasts with the view of the evaluators who generally see the teachers as just beginning to change. How can these differences in opinion be reconciled assuming that the principals and the evaluators see the same types of activities occurring in the same types of classrooms.

The difference may originate in the level of subject matter expertise the two groups bring to the observation. Evaluators are generally knowledgeable about mathematics and science content. Furthermore, they have spent considerable time thinking about and discussing with PIs and other evaluators during LSC meetings just what exemplary mathematics or science instruction might look like. They have examined videos for benchmarking purposes and considered indicators. Because of this reflection, evaluators are apt to see exemplary instruction as more that just implementation of curriculum. As one evaluator noted, "getting- it- going" is not the same as "getting- it- good." Principals, on the other hand, are responsible for all academic subjects; their knowledge of mathematics/ science instruction may be less deep. To them, adoption of a standards- based curriculum and seeing these materials used on a regular basis in the classroom with students may represent not just a first step, but being well along on the road to reform.

What implications does this have for the LSC projects? If principals are to support teachers in their reform efforts beyond the level of following the directions that come with the modules, they will need to become aware of what might occur beyond this level. Like classroom teachers, they may need to develop images of exemplary instruction that go beyond theory. They need opportunities to observe and discuss with thoughtful, knowledgeable persons a variety of strong mathematics/ science lessons being taught in the context of classrooms similar to those in their schools. As projects struggle with means to move teachers to the next level of expertise, they will need to include the principals in developing a shared vision of exemplary instruction.

Summary

• LSC is having an overall positive effect on teachers' beliefs, feelings of preparedness, and classroom practices, with these effects being greater among K- 8 science teachers than K- 8 mathematics teachers. The impact of LSC professional development on their beliefs is relatively small, in part because most teachers report positive beliefs prior to the LSC.

• Teachers participating in LSC activities are more likely to be using LSC recommended materials and strategies than their peers; many have yet to grasp the "big picture," and need mentoring and support to see beyond the mechanics of implementation.

• The level of impact increases with increased hours of participation in LSC activities. Leadership appears to require more hours of professional development for impact to occur than other areas.

• Most principals perceive their schools as being well along the road to reform, whereas most evaluators see change as being in the beginning stages. the support of principals might be enhanced by encouraging positive views of reform, while improving their understanding of exemplary mathematics and science instruction.

1 See pages 7 and 8 for a description of how composite scores were calculated.

2 Results by treatment level are presented separately for K- 8 science and K- 8 mathematics teachers. The effect size is calculated as the difference between the "0 hours" and "40 or more hours" group means, divided by the standard deviation of the population. Following standard conventions, effect sizes of .2 are considered small effects, .5 medium effects, and .8 large effects (Jacob Cohen, Statistical Power Analysis for the Behavior Sciences, Hillsdale, NJ: Lawrence Erlbaum Associates, 1988). Since all of the 7- 12 mathematics projects included in this report were funded as part of Cohort 3, the number of teachers participating in LSC professional development at the time of data collection was too small for separate analysis by treatment level; consequently for 7- 12 mathematics, results are presented for untreated teachers only.

3 Spearman' rho correlation of .67 and .64, respectively.