Paper

Highlights of the Local Systemic Change through Teacher Enhancement Year Two Cross-Site Report

Summary

In its second year of implementation, the Local Systemic Change Initiative supported 26 projects involving a total of 121 districts throughout the United States. LSC projects are characterized by their focus on adoption and use of exemplary science and mathematics instructional materials. These projects are also distinguished from teacher enhancement projects of the past by their systemic focus, including targeting all teachers in participating schools rather than volunteer teachers, and reviewing district and school policies and practices to determine the extent of alignment with the reform vision. The LSC Initiative is proving to be an effective mechanism for providing high quality professional development to large numbers of teachers. Moreover, by operating in a systemic context, the LSC projects have the potential for greater and more sustained impact than has typically been the case in teacher enhancement projects involving individual teachers from a large number of schools and districts.

Data on the quality and impact of the LSC projects come from teacher and principal questionnaires, teacher interviews, and observation of both professional development activities and science and mathematics classrooms.

Project evaluators observed a number of professional development sessions in each project using a standardized protocol developed by Horizon Research, Inc. These sessions generally received high marks for the quality of their design and implementation, for the appropriateness of the disciplinary and pedagogical content, and their inclusive, collegial culture. Overall, the LSC programs appear to be well-aligned with national science and mathematics standards, and in particular those for professional development; projects generally attend to important science and mathematics content and pedagogy in a spirit of inquiry, reflection, and continuous improvement.

Evaluators noted a number of ways in which the LSC professional development reflects current standards for best practice in professional development. Among these are the relevancy of professional development activities to teachers' work in classrooms, the way in which professional development typically modeled effective pedagogical strategies; the opportunities for teachers to reflect individually and with their colleagues; and the attention to follow-up support as teachers worked on implementing standards-based curriculum materials in their classrooms.

At the same time, evaluators in some projects have noted a need for greater attention to the balance between content and pedagogy when helping teachers learn to implement exemplary instructional materials; "going through the activities" is valuable, but teachers also need opportunities to explore the conceptual underpinnings of these activities in more depth. Also, evaluators note that projects using inexperienced professional development providers need to pay greater attention to ensuring consistently high quality service delivery; this issue was particularly important in projects where university scientists and/or lead teachers were responsible for conducting professional development sessions.

Teachers who had participated in LSC professional development were generally quite positive about their experiences, indicating that they had become more confident in teaching science as a result. Moreover, teachers who have participated extensively in science and mathematics professional development report higher levels of preparedness and more frequent use of standards-based instructional strategies than do teachers with less professional development involvement.

It is important to note that most teachers in LSC districts reported very positive attitudes towards science and mathematics reform, indicating that they recognize the value of many standards-aligned instructional strategies. However, there is a large disparity between the perceived importance and the preparedness teachers feel for implementing reform strategies in their classrooms. Similarly, sizable proportions of teachers report feeling less than well prepared to teach a number of science and mathematics topics they are expected to teach.

There was a marked consistency in the picture of classroom instruction that emerged from teacher questionnaires and classroom observations. Teacher presentations are quite common in the LSC districts, with most observed lessons including a formal presentation by the teacher. While most observed lessons also included some kind of hands-on/investigative activity, these typically involved students following detailed instructions to answer predetermined questions, rather than allowing the students to pose questions and design their own investigations to address these questions.

In keeping with NSF's focus on system-wide reform, the core evaluation questionnaires and interviews asked teachers about the support they receive from parents, principals, and other teachers in their schools. Teachers in the LSC districts generally feel supported by other teachers in their schools to try out innovative ideas in science and mathematics teaching, but they rarely have time during the regular school week to work with one another. In fact, lack of time to work with other teachers, inadequate funds for purchasing equipment and supplies, lack of access to computers, and the need for more planning time headed the list of problems for science and mathematics instruction reported by both teachers and principals.

Evaluators were also asked to describe the extent to which district policies and resources were aligned in support of the LSC reforms and the likelihood that the reforms would be sustained after the NSF funding period had ended. Most projects were rated at Level 3 (out of a possible 5) in terms of district support, indicating that district policies were in transition toward a more supportive context. Many spoke of commitments the districts had made to the LSC reforms, including purchasing kit-based instructional programs and establishing centers for maintaining and refurbishing the kits.

Finally, while many LSC projects have begun the process of developing lead teachers and otherwise increasing internal capacity to provide high quality professional development, many of the LSC districts do not yet have the mechanisms or resources to sustain these systems.

The core evaluation of the Local Systemic Change Initiative requires the energy, efforts, and insights of a very large number of people.

Conrad Katzenmeyer in the National Science Foundation's (NSF) Division of Research, Evaluation, and Communication; Joyce Evans, Susan Snyder, and Diane Spresser in NSF's Division of Elementary, Secondary, and Informal Education (ESIE); and Joy Frechtling of Westat, Inc. were instrumental in the design and implementation of the core evaluation system. Numerous other NSF/ESIE program officers provided valuable assistance throughout the process, as well.

Within HRI, in addition to the authors of this report, Joan Pasley, Ruth Pershing, Ben Kurgat, Eugene Soar, Scott Hanrath, Jarrod Jenzano, Alison Schaff, Carol McDonald, Susan Hudson, Claudia Templeton, and John Supovitz were responsible for various tasks essential to the production of this report, from instrument design, to data processing and analysis, to report production.

Most notably, this report would not have been possible without the efforts of the LSC project lead evaluators, who persevered as we attempted to fine-tune the core evaluation system. And special thanks are due to the thousands of teachers throughout the 121 participating districts who took time from their busy schedules to provide information about their science and mathematics teaching.

Introduction

LSC projects are expected to align policy and practice within the targeted district(s) and to include:

• A shared comprehensive vision of science, mathematics, and technology education;
• Active partnerships and commitments among stakeholders;
• A detailed self-study that provides a realistic assessment of the current system's strengths and needs;
• Strategic planning that incorporates mechanisms for engaging each teacher in intensive professional development activities over the course of the project; and
• A set of clearly defined, measurable outcomes for teaching, and an evaluation plan that provides on-going feedback for the project.

NSF contracted with Horizon Research, Inc. (HRI) to design and implement an evaluation for collecting data that could be aggregated across projects. The core evaluation system designed by HRI in conjunction with NSF and the project PIs and evaluators includes observations of professional development activities and science and mathematics classrooms; teacher and principal questionnaires; and teacher interviews. Project evaluators use these data, as well as any additional, project-specific data they may have collected, to address the following core evaluation questions:

0. What is the overall quality of the LSC professional development activities?
1. What is the extent of school and teacher involvement in LSC activities?
2. What is the impact of the LSC professional development on teacher preparedness, attitudes, and beliefs about science and mathematics teaching and learning?
3. What is the impact of the LSC professional development on classroom practices in science and mathematics?
4. To what extent are the district and school contexts becoming more supportive of the LSC vision for exemplary science and mathematics education?
5. What is the extent of institutionalization of high quality professional development systems in the LSC districts?

Project data sheets completed by the PIs provide some basic information about the 26 projects included in Cohorts 1 and 2.¹

• Roughly half of the Cohort 1 and Cohort 2 projects are single-district projects; at the other end of the scale, 4 projects involve more than 10 districts, including 1 with 21 districts.
• The 8 Cohort 1 projects and 18 Cohort 2 projects plan to involve a total of 28,239 teachers in 1,312 schools in 121 districts across the United States.
• By the completion of these projects, an estimated 706,000 students will receive instruction from LSC-treated teachers each year.
• 20 projects target K-8 science; 2 projects target K-8 mathematics; and 4 projects target both K-8 science and mathematics.

This report presents highlights from the 1995-96 project year, including results from the 8 Cohort 1 projects that were in their second year of funding and the 18 Cohort 2 projects that were funded in 1996.²

Description of LSC Professional Development

Core evaluation data to assess the quality of LSC professional development activities come from observations of the professional development activities, and to a lesser extent, from teacher questionnaires. Cohort 1 projects augmented these data with teacher interviews. Based on these data, evaluators were asked to rate the quality of the project's professional development in eight component areas and to provide an overall rating of the project's professional development system.

Evaluators observed a total of 180 professional development activities, an average of 7 per project. Using a standard protocol developed by HRI, they described the sessions they observed; they also assessed the appropriateness of the design of each session for accomplishing the stated purposes and the quality of its implementation. As can be seen in Figure 1, nearly half of the observed science sessions included physical science content, 37 percent focused on life science, and 33 percent had an earth-science content focus.

Major Disciplinary Content Areas of Observed Science Professional Development Sessions Figure 1

Major Disciplinary Content Areas of Observed Mathematics Professional Development Sessions Figure 2

Among projects targeting mathematics, the content areas of focus were most often computation and estimation, numeration and number theory, patterns and relationships, and mathematical problem solving. (See Figure 2.) While a few of the science sessions included emphasis on one or more mathematics topics (especially measurement), none of the mathematics sessions included science content as a major focus.

Figure 3 shows evaluators' overall "capsule ratings" of the observed sessions. Very few LSC sessions were rated as ineffective professional development (Level 1), and fewer than 10 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary science/mathematics instruction or be leaders in reform. More than 60 percent of the observed professional development sessions received ratings of 4 or 5, with Cohort 1 sessions somewhat more likely than Cohort 2 sessions to be considered "exemplary professional development."

Figure 3

The sample vignettes that follow illustrate two sessions with similar purposes, but judged to be quite different in quality.

Professional Development Session #2: A professional development session for first-grade teachers was designed to help the teachers learn the content targeted by a kit-based unit on liquids, and to become familiar with the activities included in the kit. Some participants were new to the system; others were veteran teachers who wanted to learn more about the curriculum that they would be implementing in their classes in the coming year. The session was led by two facilitators--a lead teacher in the district and a scientist from a local university. On the previous day, the lead teacher had taken participants through several activities included in the kit. Most of the two-hour session on this day was focused on introducing the "big ideas" of the unit to the whole group, with the scientist presenting content about atoms, measuring, volume, density, and phase changes. While the presented information was accurate, there was little integration of the content with the activities of the previous day. The presentation ended with an activity in which the entire group role-played the movement of molecules at different temperatures. In the last half-hour of the session, the teachers met in small groups to discuss the content covered in the morning presentation and to identify questions they may have about the content or the activities included in the unit. The scientist and lead teacher initially circulated among the groups; however, both of the facilitators spent the majority of the remaining time with one of the four small groups to address questions about one of the activities from the previous day. The session ended with an announcement of when the group would meet next for professional development on a second unit to be implemented in all first-grade classes. This session received ratings of 2 on a five-point scale on its design and implementation; and a rating of 3 on content. Due to the lack of participant engagement and absence of collaborative or participant-centered interactions, the session was rated 2 in professional development culture. Overall, the session received a capsule rating of 2, indicating that there were elements of effective professional development, but that there were major areas in need of improvement.

Evaluators' overall ratings of the LSC professional development indicate that LSC projects are generally providing high-quality staff development. Each of the component areas of professional development--the design, the implementation, the disciplinary and pedagogical content, and the culture of the sessions--received very high ratings, as well. Mean ratings for each of those areas, as well as the likelihood that the project would enhance teachers' instructional capacity were consistently between 3.5 and 4.0 on the five-point scale. Projects that focused on developing the leadership capacity of teachers in the district were rated somewhat lower in that area, with a mean rating of approximately 3.3.

Each project evaluator was also asked to rate the project's professional development system holistically, considering all of the data available, and place their LSC project on a five-level continuum. All evaluators judged the LSC projects to be at Level 3 or above on a five-level continuum, indicating that the projects are either transitioning toward, or have already achieved, a level of professional development reflective of current standards for best practice.

Evaluators found that the professional development sessions generally incorporated state-of-the-art strategies, including activities that were relevant to the teachers' work in classrooms, modeling of effective pedagogical strategies, opportunities for teachers to reflect and talk with peers, strong components of follow-up and support, and active involvement of teachers as key players in their own professional development. The sessions were generally led by competent and skilled facilitators and characterized by very positive cultures that encouraged the engagement of teachers as professionals in learning communities.

Areas of strength for some projects were also described as areas in need of improvement for others. For example, some evaluators commented on the need for participants to have more time for reflection, increased attention to issues of classroom application. Others pointed out a need for a better balance in attention to content and pedagogical strategies, or greater emphasis on the content addressed by particular kits or modules. Finally, evaluators noted a need for more consistency in the skills and expertise of facilitators.

Teacher Preparedness in Science and Mathematics Education

The core evaluation design calls for a random sample of teachers in each project to complete questionnaires about their attitudes and instructional practices. In Cohort 2, these teachers are providing baseline data about the status of science and mathematics education. In Cohort 1, the sample includes teachers who have begun to participate in the LSC professional development, but in most projects only a very small proportion of the teachers had received in-depth assistance at the time of questionnaire administration.

Data collected from teachers in the LSC districts suggest that it will take a long and concerted effort to prepare these teachers to teach science and mathematics in a standards-based fashion. While the vast majority of teachers in LSC districts described attitudes in keeping with the standards (e.g., noting that hands-on activities, inquiry-based instruction, providing concrete experiences, and connections with other disciplines are very important for effective science and mathematics instructions), relatively few teachers feel very well prepared to implement these instructional strategies. For example, more than 80 percent of teachers in the LSC districts indicated that it was very important to have students participate in hands-on science activities, but only about 30 percent felt very well prepared in this area. (See Figure 4.)

Figure 4

Teachers are even more likely to recognize inadequacies in their content preparation. While all but a handful of elementary science teachers feel at least somewhat prepared to teach such topics as the human body and ecology, 20 percent or more indicated they were not adequately prepared to teach sound, astronomy, forces and motion, mixtures and solutions, machines, processes of change over time, and electricity, all areas that are integral to the elementary science curriculum.

Elementary mathematics teachers in LSC districts were likely to express confidence in their ability to teach traditional mathematics topics such as computation, numeration and number theory, measurement, and estimation. However, substantial proportions of teachers noted that they were only "somewhat prepared" to teach some of the topics advocated for inclusion in standards-based instruction in the elementary grades, including geometry and spatial sense, data analysis and probability, and algebraic concepts.

	Percent of Teachers
	Not Adequately Prepared	Somewhat Prepared	Fairly Well Prepared	Very Well Prepared
Computation	0	2	26	72
Patterns and relationships	1	8	36	55
Numeration and number theory	1	9	35	55
Estimation	1	9	38	53
Measurement	1	9	38	53
Geometry and spatial sense	2	16	41	40
Data analysis and probability	7	27	40	27
Technology in support of mathematics	11	32	37	20
Pre-algebra/algebra	17	25	32	26

Classroom Practice

A number of core evaluation data sources contributed information about classroom practices. The teacher questionnaire data provide self-report information from a sample of all LSC-targeted teachers about the frequency with which they use various instructional strategies in their classes. The classroom observations provide an external look at the quality of classroom instruction, including information related to equity, classroom culture, and student engagement that are difficult to measure with questionnaire items.

As is the case nationally, elementary teachers in LSC districts report that they teach science less frequently than they teach reading/language arts or mathematics, and about as often as they teach social studies. For example, only about 25 percent of teachers reported teaching science (or social studies) on all 5 of the last 5 days instruction, compared to more than 80 percent for mathematics and nearly 90 percent for reading/language arts. Figure 5 shows the purposes of the observed science and mathematics lessons. Note that teachers were most likely to cite developing conceptual understanding as a major purpose of science lessons and reviewing concepts as a major purpose of mathematics lessons.

Figure 5

Observers noted that science lessons were much more likely to focus on life science than on either physical or earth/space science and that mathematics lessons most frequently focused on computation, followed by measurement, numeration and number theory, and mathematical patterns and relationships. (See Figures 6 and 7.)

Figure 6

Figure 7

Most observed lessons included both a formal presentation by the teacher and some kind of hands-on/investigative activity by the students, typically ones in which students followed a detailed set of instructions rather than designing and implementing their own investigations. Cohort 1 science classes taught by teachers who had already participated in LSC professional development were more likely than others to be involved in writing reflections in a notebook or journal and less likely to be working on answering textbook/worksheet questions. (See Table 4).

	Percent of Classes
	Cohort 1 Science	Cohort 2 Science	Cohort 2 Mathematics
Formal presentation by teacher	71	73	56
Hands-on/investigative activity	44	19	7
Following detailed instructions	2	1	45
Record/analyze/represent data	38	12	3
Work on models/simulations	65	22	17
Design/implement own investigations	12	11	1
Play game to develop/renew knowledge/skills	1	78	60
Field work/field trip	44	33	14
Class discussion	2	11	3
Whole class, teacher led	40	37	10
Small groups	1	59	8
Whole class, student led	15	24	17
Reading/writing/reflection	3	2	89
Writing reflections in notebooks/journals	72	57	18
Reflecting on activities individually or in groups	6	3	14
Answering textbook/worksheet problems	0	18	15
Reading about science/mathematics	3	0	45
Practicing routine computations	1	5	38
Identifying patterns	7	12	5

When asked how the LSC professional development had affected their classroom teaching, teachers pointed to less reliance on textbooks and more use of student-centered strategies, including cooperative learning groups. Observations confirmed the teacher reports; lessons taught by teachers who had participated in LSC professional development were more likely to receive high ratings from evaluators on lesson design and implementation, as well as on the quality of the content and the classroom culture. (See Figure 8.)

Figure 8

Lessons taught by teachers who had participated in LSC professional development were rated particularly high on the extent to which:

• The discipline was portrayed as a process of inquiry, and the lesson incorporated tasks, roles, and interactions consistent with inquiry;
• Concepts and processes were balanced appropriately;
• The room arrangement encouraged student-centered interaction;
• Students were encouraged to generate ideas, conjectures, and propositions; and
• The degree of closure or resolution of conceptual understanding was appropriate for the purposes of the lesson.

As the final step in completing the observation protocol, evaluators were asked to indicate the "level" that best described each lesson they observed. Table 5 shows the percent of lessons in each Cohort considered effective instruction, beginning stages of effective instruction, and ineffective instruction.³ Note that only 32 percent of Cohort 1 science lessons were rated as effective instruction, quite close to the 27 percent of Cohort 2 science lessons and 29 percent of Cohort 2 mathematics lessons that were rated that highly. The major difference between Cohort 1 and Cohort 2 science lessons is at the lower end of the scale, with lessons of treated teachers more likely to be considered in the beginning stages of effective instruction and less likely to be considered ineffective instruction.

	Percent of Lessons
	Cohort 1 Science	Cohort 2 Science	Cohort 2 Mathematics
Effective Instruction	32	45	29
Beginning stages of effective instruction	27	36	37
Ineffective Instruction	29	54	17

The shaded boxes contain examples of lessons that were assigned to each of these categories.

Lesson #1: Ineffective Science Instruction-Passive "Learning" The teacher stands at the front of a sparsely-equipped urban classroom and leads his third grade class, composed of mostly minority students, through a lesson about the life cycle of a butterfly. Using commercially-prepared packets, the teacher proceeds page-by-page, asking students to locate and hold up the correct page and assigning, round-robin fashion, the task of reading aloud. After reading, the teacher asks questions that require one-word answers, which he writes on the board, and students copy into their notebooks. The students then fill out worksheets. The information presented is limited to the names of the life stages; there is no reference to the processes of change and the lesson seems more like a reading/vocabulary lesson than science. Although desks are arranged in groups, there is no collaboration among students. The lesson appears too simple for the children, although they are eager to participate, to give the "correct" answers, and to fill in the blanks on their worksheets.

Lesson #2: Ineffective Mathematics Instruction-Activity for Activity's Sake As students in this seventh grade mathematics class arrive and take their seats, they open their notebooks and begin correcting their homework based on answers projected on the overhead screen. The teacher then turns off the overhead, and reviews previous work the class has done with measurement, especially alternative methods of measurement (such as arm span, finger length, stride length). She then instructs the class to open their math text to the page containing today's activity: measuring places on campus using alternative methods of measurement. Students are told to form groups of 2-4 and go outside to take measurements. There is little time for students to ask questions; some are clearly bewildered by the assignment, but the teacher is unaware of their reaction. Later, as students return from outside, they seem more focused on socializing than on math. The only closure at the end of the class is to tell students to present their measurements the next day.

Lesson #5: Effective Mathematics Instruction To practice subtraction skills, a first grade teacher reintroduces a previously-played game involving a set number of Unifix cubes in a bowl or tub. Reviewing earlier lessons, the teacher asks for examples and students demonstrate various subtraction strategies including counting cubes, counting with fingers and elbows, and drawing pictures. Then students pair off to play several rounds of "The Tub Game": taking turns, one student hides some of the 12 Unifix cubes, and the partner determines the number of hidden cubes based on the number remaining in the tub. The student then writes a "number sentence" (equation), and checks with the partner to confirm that the equation matches the actual number of cubes taken from the tub. Students are engaged and collaborative, if somewhat competitive, and appear to be making the connection between the symbolic and the concrete. The lesson ends with a group discussion of difficulties, helpful strategies, and a few additional student demonstrations. The teacher's focus is on each student using the strategies that help her/him the most.

A complementary analysis based on questionnaire data looked at classroom practice of teachers who reported varying levels of involvement in science and mathematics professional development. Teachers who had participated in a substantial amount of professional development in the previous year were most likely to feel well prepared to teach science and mathematics, and to use the instructional strategies advocated by national standards. For example, as can be seen in Figure 9, only 37 percent of classes whose teachers had not participated in science professional development engaged in hands-on activities at least once a week, compared to 70 percent whose teachers had had in-depth professional development.

Figure 9

Similarly, teachers in Cohort 1 projects showed an increase between 1995-96 in their feelings of preparedness to manage a class of students using hands-on activities and in the frequency with which they used these activities in their classes. (See Figure 10.)

Cohort 1 Science Teachers Use of Hands-On Activities Figure 10

Supportive Context for Reform

The major emphasis of the LSC is on professional development to enable teachers to effectively implement high quality instructional materials. At the same time, NSF recognized the need for projects to look at the entire system in these districts, not only professional development and instructional materials, but also policies related to student assessment; teacher recruitment, orientation, and evaluation; and systems for purchasing/managing materials and supplies. Clearly, the support of key stakeholders for the reform effort is crucial, not only from district and school-level administrators, but also from parents and the larger community. Finally, NSF hypothesized that reforms would be greatly enhanced if they had the support of business and industry; colleges and universities; and other "science-rich" institutions.

The core evaluation includes information about the supportiveness of the context for reform from a number of sources. In questionnaires and interviews, teachers were asked about the support they receive from parents, principals, and other teachers in their schools. Principals were also asked about their opinions in relation to science and mathematics reform, including how important they considered various reform approaches, and how well prepared they felt to support teachers in the implementation of national standards. In addition, teacher and principal questionnaires asked about the extent to which school and district policies affect science and mathematics instruction. Finally, evaluators were asked to describe the extent to which district policies and resources were aligned in support of the LSC reforms.

Teachers in the LSC districts generally feel supported by other teachers in their schools to try out innovative ideas in science and mathematics teaching, with roughly 3 out of 4 teachers agreeing with statements to that effect. Somewhat fewer, about 2 out of 3, indicated that teachers in their school regularly share ideas and materials. Only about 1 out of 8 teachers indicated they had time during the regular school week to work with their peers on science and mathematics curriculum and instruction.

In both science and mathematics, teachers are most likely to report that principals encourage innovative instructional practices and accept the noise that comes with an active, inquiry-based classroom (roughly 4 out of 5 in each group). Mathematics teachers were more likely than science teachers to report that their principals provided them with the necessary instructional materials and equipment; and encourages them to gear curriculum and instruction to address individual students' learning.

Principals and teachers were also given a list of "factors" that might affect instruction and asked to indicate the extent, if any, of the problem each caused for science and mathematics instruction. Teachers were answering about problems in their own instruction, and principals were answering about the school as a whole. From both the teacher and principal perspective, there is considerable room for improvement in the extent to which the policy environment supports science and mathematics instruction. In both science and mathematics, lack of time to work with other teachers was the most frequently cited problem, with more than half of the Cohort 1 and Cohort 2 science teachers and nearly that many Cohort 2 mathematics teachers labeling it a major problem.

Access to computers ranked second, again with more than half of the teachers in science and nearly half in mathematics indicating that a lack of computers was a major problem. Other frequently cited problems, especially for science instruction, included lack of funds for purchasing equipment and supplies, and lack of time to plan and prepare lessons. Principals generally echoed the concerns expressed by teachers, frequently citing such problems as inadequate funds for purchasing equipment and supplies, and the lack of teacher planning time and opportunities to work together.

Evaluators were also asked to rate the extent to which district policies and practices tended to facilitate reform or act as barriers for reform. Based on their reports, the most frequent facilitators of reform in LSC districts are the professional development systems; the curriculum, scope and sequence; the fact that the various reform efforts are consistent with one another; and the quality of instructional materials.

Cohort 1 districts received higher marks than Cohort 2 districts in a number of areas, including the quality of instructional materials and new teacher orientation. In contrast, Cohort 2 districts tended to be more highly rated in terms of consistency among reforms. In both cohorts, teacher evaluation policies and within-school policies such as time for teachers to plan and prepare lessons were rarely seen as facilitators of reforms.

A number of evaluators spoke of the commitment the districts had made to aligning their policies in support of the LSC reforms. The following are topical comments:

"Districts have given [LSC project] leaders and connected curriculum review groups the responsibility of revising the district science standards and aligning them with the emerging state [frameworks] in science."

"The district's recent decision to adopt new standards-based science and mathematics materials demonstrates a willingness to work in conjunction with, if not in direct support of, the goals of the LSC."

"The district has purchased kits which are already in use in some schools, and the district has made a major commitment to purchase more. Efforts are underway to create a system for purchasing and managing materials and supplies."

In other cases, evaluators noted that there had been major improvements in the availability of materials for hands-on instruction, but the improvements appeared project-based rather than rooted in district commitments.

"Prior to joining [the project], teachers bought their own science materials or, in some cases, districts provided materials. Currently, [the project] assembles, delivers, picks up, and refurbishes materials, greatly reducing the burden on teachers and districts."

Finally, some evaluators in other projects reported that financial difficulties pose serious problems for reform in the districts.

"[Our site] is implementing its LSC within the context of depressing and undermining budget cuts and staff readjustments. The position of teacher expert has disappeared form the system, vice principals have been cut, and much professional development has been eliminated."

"A lack of materials has been prohibitive...since these schools started with no materials, they will need to make future investments for several years."

"Currently, many teachers are forced to supplement their stock of materials with items purchased with their own money. They look forward to the time when all classrooms will be equipped with adequate materials."

Evaluators were asked to summarize their findings by placing each project at the appropriate point on a continuum of "supportiveness of context". As can be seen in Table 6, most projects in each cohort were considered "transitioning toward a supportive context."

	Percent of Projects
	Cohort 1	Cohort 2
Level 1: Predominance of Non-Supportive Context	0	0
Level 2: Minimally Supportive Context	25	0
Level 3: Transitioning Toward Supportive Context	63	72
Level 4: Emerging Supportive Context	13	28
Level 5: Predominance of Supportive Context	0	0

Sustainability of the LSC Reforms

One of the "systemic" aspects of the LSC initiative is the expectation that districts will be able to sustain the reforms after the NSF funding period has ended. Evaluators were asked to rate the extent to which the participating districts have the capacity to implement high-quality professional development, the resources available to support it, and the structures in place to sustain high-quality professional development systems. As several evaluators noted, these ratings are based on a mixture of data and "impressions," but they do provide a broad indication of the status of the districts' professional development systems.

Evaluators were asked to indicate the "continuum level" which best describes the status of the professional development system within the targeted districts. As can be seen in Table 7, most Cohort 1 science projects were judged to be "transitioning toward a high-quality professional development system." Apparently, a number of the Cohort 2 districts had made progress towards institutionalizing high quality professional development systems prior to receiving the LSC awards, with a third of the projects rated as "emerging infrastructure" or even having already institutionalized such a system in the baseline year of their LSC.

	Percent of Projects
	Cohort 1	Cohort 2
Level 1: Predominance of Ineffective Professional Development System	0	6
Level 2: Exploring Components for High-Quality Professional Development System	25	29
Level 3: Transitioning Toward a High-Quality Professional Development System	63	35
Level 4: Emerging Infrastructure for High-Quality Professional Development System	13	24
Level 5: Institutionalization of a High-Quality Professional Development System	0	6

While data from Cohort 1 projects are technically not "baseline," it is still early in the life of these projects to be looking for signs of sustainability. For that reason, and also because the Cohort 2 projects appear to have been further along in institutionalizing high-quality professional development at the time they began their LSC grants, evaluator ratings of individual aspects of sustainability are presented together.

Evaluators report that in roughly two-thirds of the projects, districts have invested in enhancing the expertise and capacity of lead teachers and that these teachers are actively engaged in facilitating professional development activities (based on the percentages responding 4 or 5 on a five-point scale from 1, "not at all" to 5, "to a great extent").

In addition to the capacity to provide high-quality professional development, districts will need resources to support these programs. Figure 11 shows evaluators' ratings of a number of aspects of resource availability. Note that while in more than half of the LSC projects the districts actively seek opportunities to provide on-going, high-quality professional development experiences for their teachers, the use of district funds for these purposes is less common.

Figure 11

Finally, districts will need structures in place to sustain high-quality professional development and provide a supportive context for exemplary instruction in science and mathematics. Evaluators in less than half of the projects reported that the districts provide incentives for teachers to participate in on-going professional development. And while 4 out of 10 have structures in place for continuously assessing and improving their professional development systems, only 1 in 5 has mechanisms in place for monitoring policy alignment, and only 1 in 4 for maintaining support for reform among community stakeholders.

3 "Ineffective instruction" includes lessons the observers categorized as level 1a, passive learning; level 1b, activity-for-activity's sake; and level 2, elements of effective instruction. Lessons rated a low or solid 3 are considered "beginning stages of effective instruction," while those rated a high 3, 4, or 5 are considered "effective instruction."