Paper

Executive Summary of the Year Three LSC Cross-Site Report

In the spring and summer of 1995, the National Science Foundation (NSF) funded the first cohort of eight projects in a new initiative, the Local Systemic Change through Teacher Enhancement (LSC) program. Eighteen additional projects were funded in 1996 and 20 more in 1997, for a total of 46 projects in Cohorts 1, 2, and 3.

The goal of the LSC program is to improve the teaching of science, mathematics, and technology by focusing on the professional development of teachers within whole schools or school districts.

LSC projects are expected to align policy and practice within the targeted district( s) and to include:

• A shared comprehensive vision of science, mathematics, and technology education;

• Active partnerships and commitments among stakeholders;

• A detailed self- study that provides a realistic assessment of the current system's strengths and needs;

• Strategic planning that incorporates mechanisms for engaging each teacher in intensive professional development activities over the course of the project; and

• A set of clearly defined, measurable outcomes for teaching, and an evaluation plan that provides ongoing feedback to the project.

In its third year of implementation, the Local Systemic Change Initiative included 46 projects in 263 districts throughout the United States. The 46 LSC projects plan to involve a total of approximately 40,000 teachers in more than 2,000 schools; by the completion of these projects, an estimated 1,356,000 students will receive instruction from LSC- treated teachers each year.

Each targeted K- 8 teacher is to participate in a minimum of 100 hours of professional development; at the secondary level, the minimum is 130 hours, over the course of the project. A serious concern is the fact that a few projects appear to have redefined targeted teacher population to mean those who are willing to participate, which is clearly inconsistent with the intent of the LSC initiative. NSF may need to re- emphasize to PIs the requirement of reaching all teachers in the participating districts in order to receive funding under the LSC initiative.

Core Evaluation

The LSC solicitation indicated NSF's plan to "provide a framework for data collection (including a set of instruments and procedures) that will allow the Foundation to evaluate individual projects, aggregate data and information across projects, and produce a cross- project analysis" (NSF 94-73). NSF contracted with Horizon Research, Inc. (HRI) of Chapel Hill, NC to design the data collection framework, provide technical assistance in its implementation, and prepare a cross- site analysis of the evaluation results. ¹

HRI has worked with the National Science Foundation and PIs and evaluators of the LSC projects on the design and implementation of a core evaluation system to allow aggregating information across projects. Evaluators of each LSC project address a common set of evaluation questions (see box) and collect data using designated core evaluation procedure and instruments.

Quality of Professional Development

Evaluators observed a total of 276 professional development sessions during the 1996- 97 data collection year. A cross- site analysis provided an overview of the key purposes and activities that characterized most sessions, as well as insight into major strengths and areas in need of further attention.

Sessions were much more likely to be led by teacher leaders and other district personnel than by university faculty or other professionals. Only 13 percent of presenters/ facilitators were members of minority groups, which is reflective of neither the targeted teacher population (25 percent minority) nor the targeted student population (51 percent minority). This points out the need for the LSC projects, and NSF, to pay particular attention to increasing the pool of minorities prepared to serve in leadership roles in mathematics and science professional development.

Both participating teachers and project evaluators indicated that LSC projects are providing fairly high- quality professional development. Forty percent of participating teachers rated the LSC professional development excellent or very good, with those that had participated for more hours more likely to rate it highly.

Professional development sessions were most likely to emphasize pedagogy, although quite a few of the observed sessions focused both on increasing teachers' mathematics/ science content knowledge and addressing classroom pedagogy issues. A smaller number of observed sessions addressed preparing lead teachers to serve in leadership roles in the LSC.

As was the case in previous years of the core evaluation, evaluators noted a number of key strengths of LSC professional development. Observers found that the majority of LSC sessions (71 percent) provided high- quality professional development experiences that were likely to enhance the capacity of teachers to implement exemplary instruction in their classrooms. Of special note was the collegial and engaging culture established among participants and facilitators. Observers found that most sessions were also well- facilitated, and targeted mathematics/ science content that was both sound and appropriate for the purposes of the session and background of participants.

In addition to its focus on involving all teachers in a targeted district, the LSC initiative is distinguished from previous teacher enhancement efforts by its emphasis on preparing teachers to implement designated exemplary mathematics and science instructional materials in their classrooms. Linking professional development to exemplary curriculum materials has proven to be an effective way to simultaneously model inquiry- based strategies and address teacher content needs. While links to instructional materials were clearly beneficial in numerous ways, evaluators cautioned that in focusing on the use of module activities, projects risk losing the emphasis on key mathematics and science concepts, pointing out the need to keep the "big picture" in mind. Attention to systemic issues and the broad- framework of national standards in mathematics and science has enhanced the quality of the LSC professional development programs.

Areas that observers identified as frequently problematic in professional development sessions were attention to "sense- making" and closure at appropriate points in the sequence, and providing adequate time and structure for teachers to consider how to apply what they were learning to their instruction. LSC project staff may need to pay special attention to addressing these challenges in planning future professional development programs.

Teachers were most likely to give LSC professional development programs high marks for providing a wealth of opportunities for mathematics/ science related professional development and for providing support as they implement what they have learned. In each of those areas, teachers rated LSC professional development much higher than professional development prior to the LSC. In contrast, there were only small differences between the LSC and "prior" professional development in the extent to which teachers were given time to work with other teachers, or to reflect on how to apply what they have learned to the classroom.

Leadership Development

Although most LSC projects include a leadership development component in their design, and a substantial portion of professional development sessions were facilitated by and included teacher leaders as participants, very few of the observed sessions actually targeted leadership content, such as planning and implementing high- quality professional development.

Evaluators identified a number of elements that were important for effective professional development for teacher leaders, including: clear communication of expectations; balancing attention to disciplinary, pedagogical and leadership content; providing opportunities for practice; ongoing administrative and technical support; and broadening their professional experiences.

Although leadership content was rated fairly high when it was included in observed professional development sessions, relatively few (40 percent) of the evaluators give leadership content high marks in the overall LSC programs. It appears that this rating was more reflective of the lack of attention to leadership content than the quality of the content when it was addressed. Leadership content is an area of the LSC professional development that clearly needs additional attention, including providing opportunities for PIs to share ideas and discuss strategies with each other.

Impact on Teachers and Teaching

LSC projects are having a positive impact on both teachers' feelings of competence to teach mathematics/ science and their ability to actually do so at the classroom level.

Classroom observations provided insight into areas of strength of LSC teachers and areas of particular difficulty. Lessons taught by teachers who had participated in at least 20 hours of LSC professional development tended to focus on significant content that was at an appropriate level for their students; the teachers seemed to have a good understanding of their students' prior knowledge and teacher- presented information was generally accurate. Moreover, teachers were able to establish a classroom culture of active participation and respect for students' ideas.

Areas that proved to be problematic mirror some of the same ones reported in quite a few of the LSC professional development activities: adequate time and structure for reflection, and providing an appropriate degree of closure. In addition, evaluators found the following areas to

be especially challenging for teachers: using questioning strategies that are likely to enhance the development of conceptual understanding; making appropriate connections to real- life applications and to other disciplines; and valuing intellectual rigor and the challenging of ideas. Project PIs and staff may need support in exploring ways to improve these aspects of classroom practice.

Supporting and Sustaining Local Systemic Reform

In addition to improving classroom instruction through the professional development of teachers, projects are expected to garner support for exemplary mathematics and science educational practices. LSC projects are grappling with ways to not only provide this supportive context, but also to sustain changes that have taken place in the years after NSF funding is terminated.

Evaluators reported a variety of strategies used by the LSC projects to involve key stakeholders, most notably principals, but also central office staff, parents, and other community members. Based on questionnaire data from both teachers and principals, there has in fact been an increase in principal support for mathematics and science education reform. However, evaluators noted that, as projects struggle with means to move teachers to the next level of expertise, it will be important to continue to include principals in developing a shared vision of exemplary instruction.

Many of the LSC projects have garnered the active support of institutions for higher education, business/ industry, museums, and other science- rich institutions. In contrast, evaluators typically reported that parents, non- LSC teachers, and teacher unions were not actively involved in supporting the LSC reforms.

Relatively few evaluators discussed specific strategies developed by projects to influence district policies that would encourage sustainability. The problem may be that project staff, who have expertise in professional development, are not as skilled in strategic planning and systemic reform. Or perhaps evaluators are less attuned to the nuances of policy alignment and therefore less likely to focus on this area in their reports. In any event, NSF should consider providing technical assistance to LSC projects in understanding the importance of the policy domain in systemic reform and in developing strategies to increase alignment of district policy with the LSC vision.

On the other hand, some evaluators did relate a number of ways in which districts are building upon the LSC efforts in order to institutionalize the reform process, including the convergence of resources in support of the LSC vision, the development of incentives for continued professional development, and plans to maintain the involvement of LSC teachers in key leadership positions in the districts.

Most LSC districts are working to build the capacity of teacher leaders in the hopes that the district will continue to support these teachers to work in a leadership capacity after the LSC project. Evaluators noted, however, that in many cases districts appear to be dependent on the LSC grant for planning and delivering high- quality professional development, and there is no system in place for ongoing leadership development after the LSC grant period.

Typically those projects that have been in operation longest are described by evaluators as having "an emerging infrastructure" to sustain reform, while the newer projects are "in transition toward supporting and sustaining systemic reform." Clearly, institutionalization issues are key to the long- term impact of the LSC projects. Consequently, it would be helpful to provide PIs and other project staff opportunities to learn from each other as well as from external experts about strategies for increasing the likelihood that the LSC reform process will be sustained.

1 Results of the Year Three cross- site evaluation are reported in the "Local Systemic Change through Teacher Enhancement: Year Three Cross-Site Report" and the "Highlights of the Local Systemic Change through Teacher Enhancement: Year Three Cross- Site Report" which can be found at Horizon Research, Inc.'s web site, www.horizon-research.com.