Paper

Highlights of the Year Three LSC Cross-Site Report

Current Status of Teachers' Beliefs and Practices

As part of the core evaluation process, considerable data were collected about mathematics and science instruction in the LSC districts. From the teacher questionnaires, teacher interviews, and classroom observations, information was obtained concerning teacher beliefs and practices.

Description of Targeted Teachers' Beliefs and Practices

The cross- site data analyzed for this report support many of the findings of the previous year. As can be seen in Table 4, teachers generally report positive attitudes toward teaching mathematics and science, and they feel supported by colleagues to try out new ideas. Although they frequently share materials and ideas with their colleagues, there is little time set aside during the school week to work with peers.

Figure 3 shows the percentage of teachers in each group indicating that they are "well informed" about the National Science Education Standards/ National Council of Teachers of Mathematics (NCTM) Standards. Secondary mathematics teachers were most likely, and elementary science teachers least likely, to report being familiar with national standards.

While they may not be familiar with the standards documents per se, most LSC elementary teachers, report attitudes and beliefs that are in fairly close alignment with standards- based instruction. Overall, more than two- thirds of the teachers indicated it was very important for students to participate in hands- on, inquiry- oriented mathematics and science instruction, and to make connections between these and other disciplines. Interestingly, secondary teachers were less likely than elementary teachers to consider these strategies to be of great importance. It is also interesting to note that substantial numbers of teachers do not feel well prepared to implement some of the strategies that they consider important.

Elementary teachers also reported on the number of the last five school days they had taught each of several subjects. Due to the emphasis on reading and mathematics in the elementary curriculum, and perhaps also due in part to teachers' feelings of preparedness, science is taught less frequently than either reading or mathematics. On the average, reading and language arts were taught 4.7 of the 5 most recent days school was in session, while mathematics was taught on an average of 4.6 days, science 2.8 days, and social studies 2.7 days.

Observers' Descriptions of Classroom Practice

Trained observers in each project were to observe 10 randomly selected classrooms. For Cohorts 1 and 2, the teachers to be observed had already participated in at least 20 hours of professional development provided by the project. A total of 519 classes were observed, including 292 science classrooms; 198 K- 8 mathematics classrooms; and 29 7- 12 mathematics classrooms. These observations provided information on student demographics, the physical environment, purposes and disciplinary content of the lessons, class activities, and the quality of the lesson in relation to the LSC vision for exemplary instruction.

The observer talked with the teacher before and after observing each class to find out about both the intended purpose of the lesson and whether these purposes shifted during the lesson for any reason. As can be seen in Figure 4, roughly half of the K- 8 mathematics and science teachers, but only 23 percent of the 7- 12 mathematics teachers, reported a major focus on developing conceptual understanding.

Observers documented the use of instructional materials from a variety of sources, including traditional textbooks or worksheets, materials developed by another school district, or teacher developed materials. However, the majority of the observed lessons made use of LSC- designated materials.

Evaluators were given a checklist of possible class activities and asked to indicate the ones that were included in the observed lesson. Elementary mathematics and science lessons were more likely to include hands- on activities and class discussions than formal presentations by the teachers. In contrast, grade 7- 12 mathematics teachers were as likely to use investigative activities as they were to use formal presentations. (See Table 5.)

Observers' Ratings of Lessons

For Cohorts 1 and 2, evaluators observed lessons taught by teachers who had already participated in LSC professional development. Many of these lessons were quite strong (ratings of 4 or 5 on a five- point scale) in the extent to which:

• Active student participation was encouraged and valued (74 percent);

• The content was appropriate for the purposes of the lesson and developmental level of the class (68 percent);

• The mathematics/ science content was significant and worthwhile (67 percent);

• Teacher- presented information was accurate (67 percent);

• The teacher was confident in his- her ability to teach mathematics/ science (67 percent);

• The teacher was cognizant of prior knowledge of students (66 percent); and

• There was a climate of respect for students' ideas, questions, and contributions (65 percent).

In contrast, relatively few of these classes were highly rated for the extent to which:

• Adequate time and structure were provided for reflection (37 percent);

• Appropriate connections were made to other areas of mathematics/ science, to other disciplines, and/ or to real- world contexts (37 percent);

• Intellectual rigor, constructive criticism, and the challenging of ideas were valued (36 percent);

• The lesson was modified as needed, based on teacher questioning or other student assessments (33 percent);

• The teacher's questioning strategies were likely to enhance the development of student conceptual understanding (31 percent); and

• The degree of closure or resolution of conceptual understanding was appropriate for the developmental levels/ needs of the student and the purposes of the lesson (22 percent).

Impact of LSC Activities

Two of the 6 core evaluation questions address the impact of the LSC on teachers and teaching. One focuses on the impact on teacher preparation, attitudes, and beliefs; and the other on the impact on classroom practices.

Impact From the Teachers' Point of View

To facilitate the reporting of large amounts of survey data, and because individual questionnaire items are potentially unreliable, HRI used factor analysis to identify survey questions that could be combined into "composites." Each composite represents an important construct related to one of the key evaluation questions.¹

Results showed that teachers who had participated in 40 or more hours of LSC professional development were more likely than non- participants to:

• Feel well prepared in both content and pedagogy;
• Establish an investigative culture in the classroom;
• Use investigative instructional strategies; and
• Use calculators and computers in mathematics instruction.

In interviews, teachers indicated that the LSC had had considerable impact, particularly in the areas of curriculum and instruction. Also, the amount of impact generally increased with time spent in LSC activities. (See Table 6.)

Principals' Impressions of Impact

Principals in schools targeted by LSC projects were asked to complete questionnaires. Among other items, they were asked to rate: (1) the level of their school's current involvement in the LSC, (2) the extent to which their teachers were using the LSC- specified instructional materials, and (3) the extent to which teachers were using the LSC- advocated instructional strategies with their students. Fifty- six percent of the principals involved in K- 8 science projects in Cohort 1 and Cohort 2, and 59 percent of those in K- 8 mathematics projects felt that their schools were at least "well along in improving" their science education.

Impact as Measured by LSC Evaluators

By looking at the synthesis and capsule ratings of the three different cohorts, it is possible to get an indication of impact of the LSC programs. While Cohort 1 and Cohort 2 contained teachers that have received at least 20 hours of LSC professional development, Cohort 3 teachers have not yet received 20 hours of professional development. As can be seen in Figure 5, lessons of teachers in all cohorts were rated similarly for the quality of their implementation. Cohort 1 and Cohort 2 teachers were rated higher in all other aspects than were Cohort 3 teachers.

Although teachers enter the LSC projects with a wide range of abilities and knowledge, evaluators frequently commented on participants' growing self confidence as they engage in LSC activities. Most Cohort 1 and Cohort 2 evaluators also reported that there had been impact on teachers' classroom practices, pointing out that they noticed frequent use of the designated materials in both mathematics and science.

Supportiveness of Context

Systemic reform theory underlying the LSC initiative calls for aligning policies and practices in support of standards- based mathematics and science education. LSC project evaluators were asked to assess the supportiveness of the policy environment in a number of areas including the curriculum scope and sequence, student assessment, and teacher evaluation policies. They were also asked to gauge the extent of stakeholder support for mathematics/ science reform.

Strategies to Increase Stakeholder Support

Evaluators described some of the mechanisms used by the LSC projects to engender support among key stakeholders. The most common strategy reported by the evaluators was the involvement and active participation of principals (and, to a lesser degree, central office staff) in professional development activities. For the most part, these kinds of professional development activities were designed specifically for administrators, but projects also included principals in teacher professional development activities as a way to increase both their support and their capacity for instructional leadership in mathematics and science.

LSC evaluators also reported a variety of strategies used by projects to increase the support of community stakeholders, such as hosting family mathematics/ science nights and involving local organizations and businesses.

When asked to reflect on the impact of LSC activities on stakeholder support for reform, evaluators most frequently identified increased collegiality among LSC participants. As one evaluator put it, the "sheer number of hours teachers spend in professional development in mathematics and science sets the stage for the development of collegial relationships."

Other evaluators noted that the various ways projects had involved principals and other administrators had led to a better understanding of the support teachers need and a greater willingness to provide them with the necessary resources.

District Policies and Practices

Teachers, principals, and evaluators were all asked to rate the extent to which specific policies and practices hinder or facilitate mathematics and science reform processes. Areas most frequently reported to facilitate reform included quality of instructional materials, curriculum scope and sequence, systems for purchasing and managing materials and supplies, and the importance placed on mathematics and science education. In contrast, policies for teacher recruitment, teacher evaluation, and student assessment were judged to be impeding reform in quite a few districts.

Inadequate time for teachers to plan, both individually and with their colleagues, appears to be particularly problematic. While the precise percentages differ among teachers of the various subjects, and between teachers and principals, all groups agreed that the factors that most impede effective instruction are: time available for teachers to plan and prepare lessons; opportunities for teachers to work with other teachers; and funds for purchasing equipment and supplies.

For example, 44 percent of targeted teachers indicated that the lack of time available for teachers to plan and prepare lessons inhibited effective instruction, roughly twice as many as cited such

factors as inadequate opportunities for professional development, quality of instructional materials, state or district testing policies, or public attitudes toward reform.

Overall Ratings of Supportiveness

Evaluators were asked to use all of the information available to them to rate the extent to which the district contexts support LSC reform. Figure 6 provides a summary of the district continuum ratings by cohort for 82 districts across the 46 LSC projects. Note that there appears to be a trend toward a supportive context from projects in their earliest stages (Cohort 3) to the most veteran of the LSCs (Cohort 1). The modal rating for Cohort 1 districts is 4, which indicates "emerging support" in terms of policy alignment and commitment of diverse stakeholders. By the same measure, Cohort 3 districts are predominantly "in transition"-- modifying policies that hinder effective mathematics/ science education and grappling with uneven support among stakeholders. The distribution of ratings for districts in Cohort 2 projects shows them to be shifting from "in transition" to "emerging support."

Sustainability of Reform

The LSC awards are intended to provide support for professional development during the grant period and to develop strategies to ensure that the districts will continue to support reform after the funded period. As a prelude to rating participating LSC districts on the sustainability of mathematics and science reform, evaluators were asked to review their notes of interviews with PIs and district personnel, project documents, observations of professional development activities and district meetings, and district documents. Based on these data, evaluators were asked to reflect on the extent to which the LSC has enhanced both the capacity and will of the districts to provide high- quality mathematics and science professional development.

While evaluators' information on sustainability was limited, they did note areas that might contribute toward maintaining high- quality professional development programs. Most LSC districts are working to build the capacity of teacher leaders and involving these teachers in facilitating professional development activities. Some districts are changing their policies to increase teacher participation in high- quality professional development. Districts are being encouraged to increase their investments in mathematics and science education.

In addition to noting specific ways in which the districts were, and were not, creating mechanisms to sustain the LSC reform process, evaluators assigned each district a "sustainability rating" on a five- point continuum from 1, "predominately ineffective system" to 5, "institutionalization of a high- quality professional development system."

Continuum ratings across cohorts once again suggest some trend toward increased sustainability as projects (and district involvement) mature; however, this pattern is less striking for sustainability than it is for supportiveness. (See Figure 7.)

Summary and Recommendations

In its third year of implementation, the Local Systemic Change Initiative included 46 projects in 263 districts throughout the United States. The 46 LSC projects plan to involve a total of approximately 40,000 teachers in more than 2,000 schools; by the completion of these projects, an estimated 1,356,000 students will receive instruction from LSC- treated teachers each year.

Each targeted K- 8 teacher is to participate in a minimum of 100 hours of professional development; at the secondary level, the minimum is 130 hours, over the course of the project. A serious concern is the fact that a few projects appear to have redefined targeted teacher population to mean those who are willing to participate, which is clearly inconsistent with the intent of the LSC initiative. NSF may need to re- emphasize to PIs the requirement of reaching all teachers in the participating districts in order to receive funding under the LSC initiative.

Quality of Professional Development

Evaluators observed a total of 276 professional development sessions during the 1996- 97 data collection year. A cross- site analysis provided an overview of the key purposes and activities that characterized most sessions, as well as insight into major strengths and areas in need of further attention.

Sessions were much more likely to be led by teacher leaders and other district personnel than by university faculty or other professionals. Only 13 percent of presenters/ facilitators were members of minority groups, which is reflective of neither the targeted teacher population (25 percent minority) nor the targeted student population (51 percent minority). This points out the need for the LSC projects, and NSF, to pay particular attention to increasing the pool of minorities prepared to serve in leadership roles in mathematics and science professional development.

Professional development sessions were most likely to emphasize pedagogy, although quite a few of the observed sessions focused both on increasing teachers' mathematics/ science content knowledge and addressing classroom pedagogy issues. A smaller number of observed sessions addressed preparing lead teachers to serve in leadership roles in the LSC.

As was the case in previous years of the core evaluation, evaluators noted a number of key strengths of LSC professional development. Observers found that the majority of LSC sessions provided high- quality professional development experiences that were likely to enhance the capacity of teachers to implement exemplary instruction in their classrooms. Of special note was the collegial and engaging culture established among participants and facilitators. Observers found that most sessions were well- facilitated, and targeted mathematics/ science content that was both sound and appropriate for the purposes of the session and background of participants.

Both participating teachers and project evaluators indicated that LSC projects are providing fairly high- quality professional development. Forty percent of participating teachers rated the LSC professional development excellent or very good, with those that had participated for more hours more likely to rate it highly. Teachers were most likely to give LSC professional development programs high marks for providing a wealth of opportunities for mathematics/ science related professional development and for providing support as they implement what they have learned. In each of those areas, teachers rated LSC professional development much higher than professional development prior to the LSC. In contrast, there were only small differences between the LSC and "prior" professional development in the extent to which teachers were given time to work with other teachers, or to reflect on how to apply what they have learned to the classroom.

In addition to its focus on involving all teachers in a targeted district, the LSC initiative is distinguished from previous teacher enhancement efforts by its emphasis on preparing teachers to implement designated exemplary mathematics and science instructional materials in their classrooms. Linking professional development to exemplary curriculum materials has proven to be an effective way to simultaneously model inquiry- based strategies and address teacher content needs. While links to instructional materials were clearly beneficial in numerous ways, evaluators cautioned that in focusing on the use of module activities, projects risk losing the emphasis on key mathematics and science concepts, pointing out the need to keep the "big picture" in mind.

Areas that observers identified as frequently problematic in professional development sessions were attention to "sense- making" and closure at appropriate points in the sequence, and providing adequate time and structure for teachers to consider how to apply what they were learning to their instruction. LSC project staff may need to pay special attention to addressing these challenges in planning future professional development programs.

LSC projects are confronted with the dilemma that while 100- 130 hours of professional development in mathematics/ science education over five years is substantially more than most inservice teachers receive, it is still a tremendous challenge to "cover" all the areas in which teachers need assistance. Add to that, the need to address the necessary disciplinary and pedagogical content in a manner that models effective practice, and the magnitude of the challenge becomes clear.

Unfortunately, the theory of effective professional development-- and the research base that underlies that theory-- is rather thin. While general principles can be derived from the research on adult learners and on in- service education generally, the current knowledge base provides little guidance on how to approach specific content areas. For example, if teachers are expected to use inquiry- based instruction in their classes, is it important that they use such strategies in learning the content addressed in the professional development? As illustrations or all of the time? If the goal is improved classroom practice for multiple units or "kits," is it more effective to delve into one in great depth, or to distribute the available time more evenly across the kits? Finally, are there differences in optimal time allocations-- or instructional strategies-- based on the difficulty of the particular content area for teachers and/ or students?

As professional development service delivery projects, the LSCs are typically not set up to do the kinds of focused research necessary to answer questions such as these. At the same time, providing PIs and other project staff opportunities to share experiences and lessons learned about these issues and others would enable NSF to capture the "wisdom of practice" that will help move the field forward.

Leadership Development

Although most LSC projects include a leadership development component in their design, and a substantial portion of professional development sessions were facilitated by and included teacher leaders as participants, very few of the observed sessions actually targeted leadership content, such as planning and implementing high- quality professional development.

Evaluators identified a number of elements that were important for effective professional development for teacher leaders, including: clear communication of expectations; balancing attention to disciplinary, pedagogical and leadership content; providing opportunities for practice; ongoing administrative and technical support; and broadening their professional experiences.

While leadership content was rated fairly high when it was included in observed professional development sessions, relatively few of the evaluators give leadership content high marks in the overall LSC programs. It appears that this rating was more reflective of the lack of attention to leadership content than the quality of the content when it was addressed. Leadership content is an area of the LSC professional development that clearly needs additional attention, including providing opportunities for PIs to share ideas and discuss strategies with each other.

Impact on Teachers and Teaching

LSC projects are having a positive impact on both teachers' feelings of competence to teach mathematics/ science and their ability to actually do so at the classroom level.

Classroom observations provided insight into areas of strength of LSC teachers and areas of particular difficulty. Lessons taught by teachers who had participated in at least 20 hours of LSC professional development tended to focus on significant content that was at an appropriate level for their students; the teachers seemed to have a good understanding of their students' prior knowledge and teacher- presented information was generally accurate. Moreover, teachers were able to establish a classroom culture of active participation and respect for students' ideas.

Areas that proved to be problematic mirror some of the same ones reported in quite a few of the LSC professional development activities: adequate time and structure for reflection, and providing an appropriate degree of closure. In addition, evaluators found the following areas to be especially challenging for teachers: using questioning strategies that are likely to enhance the development of conceptual understanding; making appropriate connections to real- life applications and to other disciplines; and valuing intellectual rigor and the challenging of ideas. Project PIs and staff may need support in exploring ways to improve these aspects of classroom practice.

Supporting and Sustaining Local Systemic Reform

In addition to improving classroom instruction through the professional development of teachers, projects are expected to garner support for exemplary mathematics and science educational practices. LSC projects are grappling with ways to not only provide this supportive context, but also to sustain changes that have taken place in the years after NSF funding is terminated.

Evaluators reported a variety of strategies used by the LSC projects to involve key stakeholders, most notably principals, but also central office staff, parents, and other community members. Based on questionnaire data from both teachers and principals, there has in fact been an increase in principal support for mathematics and science education reform. However, evaluators noted that, as projects struggle with means to move teachers to the next level of expertise, it will be important to continue to include principals in developing a shared vision of exemplary instruction.

Many of the LSC projects have garnered the active support of institutions for higher education, business/ industry, museums, and other science- rich institutions. In contrast, evaluators typically reported that parents, non- LSC teachers, and teacher unions were not actively involved in supporting the LSC reforms.

Relatively few evaluators discussed specific strategies developed by projects to influence district policies that would encourage sustainability. The problem may be that project staff, who have expertise in professional development, are not as skilled in strategic planning and systemic reform. Or perhaps evaluators are less attuned to the nuances of policy alignment and therefore less likely to focus on this area in their reports. In any event, NSF should consider providing technical assistance to LSC projects in understanding the importance of the policy domain in systemic reform and in developing strategies to increase alignment of district policy with the LSC vision.

On the other hand, some evaluators did relate a number of ways in which districts are building upon the LSC efforts in order to institutionalize the reform process, including the convergence of resources in support of the LSC vision, the development of incentives for continued professional development, and plans to maintain the involvement of LSC teachers in key leadership positions in the districts.

Most LSC districts are working to build the capacity of teacher leaders in the hopes that the district will continue to support these teachers to work in a leadership capacity after the LSC project. Evaluators noted, however, that in many cases districts appear to be dependent on the LSC grant for planning and delivering high- quality professional development, and there is no system in place for ongoing leadership development after the LSC grant period.

Typically those projects that have been in operation longest are described by evaluators as having "an emerging infrastructure" to sustain reform, while the newer projects are "in transition." Clearly, institutionalization issues are key to the long- term impact of the LSC projects. Consequently, it would be helpful to provide PIs and other project staff opportunities to learn from each other as well as from external experts about strategies for increasing the likelihood that the LSC reform process will be sustained.

^{1 For a full definition of how composites are calculated, see the technical report.}