Highlights of the Local Systemic Change through Teacher Enhancement Year Two Cross-Site Report
Horizon Research, Inc.
In its second year of implementation, the Local Systemic Change Initiative supported 26 projects involving a total of 121 districts throughout the United States. LSC projects are characterized by their focus on adoption and use of exemplary science and mathematics instructional materials. These projects are also distinguished from teacher enhancement projects of the past by their systemic focus, including targeting all teachers in participating schools rather than volunteer teachers, and reviewing district and school policies and practices to determine the extent of alignment with the reform vision. The LSC Initiative is proving to be an effective mechanism for providing high quality professional development to large numbers of teachers. Moreover, by operating in a systemic context, the LSC projects have the potential for greater and more sustained impact than has typically been the case in teacher enhancement projects involving individual teachers from a large number of schools and districts.
Data on the quality and impact of the LSC projects come from teacher and principal questionnaires, teacher interviews, and observation of both professional development activities and science and mathematics classrooms.Quality of Professional Development
Project evaluators observed a number of professional development sessions in each project using a standardized protocol developed by Horizon Research, Inc. These sessions generally received high marks for the quality of their design and implementation, for the appropriateness of the disciplinary and pedagogical content, and their inclusive, collegial culture. Overall, the LSC programs appear to be well-aligned with national science and mathematics standards, and in particular those for professional development; projects generally attend to important science and mathematics content and pedagogy in a spirit of inquiry, reflection, and continuous improvement.
Evaluators noted a number of ways in which the LSC professional development reflects current standards for best practice in professional development. Among these are the relevancy of professional development activities to teachers' work in classrooms, the way in which professional development typically modeled effective pedagogical strategies; the opportunities for teachers to reflect individually and with their colleagues; and the attention to follow-up support as teachers worked on implementing standards-based curriculum materials in their classrooms.
At the same time, evaluators in some projects have noted a need for greater attention to the balance between content and pedagogy when helping teachers learn to implement exemplary instructional materials; "going through the activities" is valuable, but teachers also need opportunities to explore the conceptual underpinnings of these activities in more depth. Also, evaluators note that projects using inexperienced professional development providers need to pay greater attention to ensuring consistently high quality service delivery; this issue was particularly important in projects where university scientists and/or lead teachers were responsible for conducting professional development sessions.Impact on Teachers and Teaching
Teachers who had participated in LSC professional development were generally quite positive about their experiences, indicating that they had become more confident in teaching science as a result. Moreover, teachers who have participated extensively in science and mathematics professional development report higher levels of preparedness and more frequent use of standards-based instructional strategies than do teachers with less professional development involvement.
It is important to note that most teachers in LSC districts reported very positive attitudes towards science and mathematics reform, indicating that they recognize the value of many standards-aligned instructional strategies. However, there is a large disparity between the perceived importance and the preparedness teachers feel for implementing reform strategies in their classrooms. Similarly, sizable proportions of teachers report feeling less than well prepared to teach a number of science and mathematics topics they are expected to teach.
There was a marked consistency in the picture of classroom instruction that emerged from teacher questionnaires and classroom observations. Teacher presentations are quite common in the LSC districts, with most observed lessons including a formal presentation by the teacher. While most observed lessons also included some kind of hands-on/investigative activity, these typically involved students following detailed instructions to answer predetermined questions, rather than allowing the students to pose questions and design their own investigations to address these questions.
Science lessons taught by teachers who had participated in LSC professional development were more likely than baseline science or mathematics lessons to be rated highly on the extent to which the discipline was portrayed as a process of inquiry; students were encouraged to generate ideas, conjectures, and propositions; and the degree of closure was appropriate for the purposes of the lesson.Providing a Supportive Context for Teachers
In keeping with NSF's focus on system-wide reform, the core evaluation questionnaires and interviews asked teachers about the support they receive from parents, principals, and other teachers in their schools. Teachers in the LSC districts generally feel supported by other teachers in their schools to try out innovative ideas in science and mathematics teaching, but they rarely have time during the regular school week to work with one another. In fact, lack of time to work with other teachers, inadequate funds for purchasing equipment and supplies, lack of access to computers, and the need for more planning time headed the list of problems for science and mathematics instruction reported by both teachers and principals.
Evaluators were also asked to describe the extent to which district policies and resources were aligned in support of the LSC reforms and the likelihood that the reforms would be sustained after the NSF funding period had ended. Most projects were rated at Level 3 (out of a possible 5) in terms of district support, indicating that district policies were in transition toward a more supportive context. Many spoke of commitments the districts had made to the LSC reforms, including purchasing kit-based instructional programs and establishing centers for maintaining and refurbishing the kits.
Finally, while many LSC projects have begun the process of developing lead teachers and otherwise increasing internal capacity to provide high quality professional development, many of the LSC districts do not yet have the mechanisms or resources to sustain these systems.
The core evaluation of the Local Systemic Change Initiative requires the energy, efforts, and insights of a very large number of people.
Conrad Katzenmeyer in the National Science Foundation's (NSF) Division of Research, Evaluation, and Communication; Joyce Evans, Susan Snyder, and Diane Spresser in NSF's Division of Elementary, Secondary, and Informal Education (ESIE); and Joy Frechtling of Westat, Inc. were instrumental in the design and implementation of the core evaluation system. Numerous other NSF/ESIE program officers provided valuable assistance throughout the process, as well.
Within HRI, in addition to the authors of this report, Joan Pasley, Ruth Pershing, Ben Kurgat, Eugene Soar, Scott Hanrath, Jarrod Jenzano, Alison Schaff, Carol McDonald, Susan Hudson, Claudia Templeton, and John Supovitz were responsible for various tasks essential to the production of this report, from instrument design, to data processing and analysis, to report production.
Most notably, this report would not have been possible without the efforts of the LSC project lead evaluators, who persevered as we attempted to fine-tune the core evaluation system. And special thanks are due to the thousands of teachers throughout the 121 participating districts who took time from their busy schedules to provide information about their science and mathematics teaching.
Table of Contents
IntroductionIn 1995, the National Science Foundation (NSF) began its Local Systemic Change through Teacher Enhancement Program (LSC). The goal of the program is to improve the teaching of science, mathematics, and technology by focusing on the professional development of teachers within whole schools or school districts. Each targeted teacher is to participate in a minimum of 100 hours of professional development with an emphasis on preparing them to implement exemplary science and mathematics instructional materials in their classrooms.
LSC projects are expected to align policy and practice within the targeted district(s) and to include:
NSF contracted with Horizon Research, Inc. (HRI) to design and implement an evaluation for collecting data that could be aggregated across projects. The core evaluation system designed by HRI in conjunction with NSF and the project PIs and evaluators includes observations of professional development activities and science and mathematics classrooms; teacher and principal questionnaires; and teacher interviews. Project evaluators use these data, as well as any additional, project-specific data they may have collected, to address the following core evaluation questions:
Project data sheets completed by the PIs provide some basic information about the 26 projects included in Cohorts 1 and 2.1
This report presents highlights from the 1995-96 project year, including results from the 8 Cohort 1 projects that were in their second year of funding and the 18 Cohort 2 projects that were funded in 1996.2
Description of LSC Professional Development
Core evaluation data to assess the quality of LSC professional development activities come from observations of the professional development activities, and to a lesser extent, from teacher questionnaires. Cohort 1 projects augmented these data with teacher interviews. Based on these data, evaluators were asked to rate the quality of the project's professional development in eight component areas and to provide an overall rating of the project's professional development system.
Figure 3 shows evaluators' overall "capsule ratings" of the observed sessions. Very few LSC sessions were rated as ineffective professional development (Level 1), and fewer than 10 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary science/mathematics instruction or be leaders in reform. More than 60 percent of the observed professional development sessions received ratings of 4 or 5, with Cohort 1 sessions somewhat more likely than Cohort 2 sessions to be considered "exemplary professional development."
The sample vignettes that follow illustrate two sessions with similar purposes, but judged to be quite different in quality.
Evaluators' overall ratings of the LSC professional development indicate that LSC projects are generally providing high-quality staff development. Each of the component areas of professional development--the design, the implementation, the disciplinary and pedagogical content, and the culture of the sessions--received very high ratings, as well. Mean ratings for each of those areas, as well as the likelihood that the project would enhance teachers' instructional capacity were consistently between 3.5 and 4.0 on the five-point scale. Projects that focused on developing the leadership capacity of teachers in the district were rated somewhat lower in that area, with a mean rating of approximately 3.3.
Each project evaluator was also asked to rate the project's professional development system holistically, considering all of the data available, and place their LSC project on a five-level continuum. All evaluators judged the LSC projects to be at Level 3 or above on a five-level continuum, indicating that the projects are either transitioning toward, or have already achieved, a level of professional development reflective of current standards for best practice.
Continuum Ratings for Quality of LSC Professional Development
Evaluators found that the professional development sessions generally incorporated state-of-the-art strategies, including activities that were relevant to the teachers' work in classrooms, modeling of effective pedagogical strategies, opportunities for teachers to reflect and talk with peers, strong components of follow-up and support, and active involvement of teachers as key players in their own professional development. The sessions were generally led by competent and skilled facilitators and characterized by very positive cultures that encouraged the engagement of teachers as professionals in learning communities.
Areas of strength for some projects were also described as areas in need of improvement for others. For example, some evaluators commented on the need for participants to have more time for reflection, increased attention to issues of classroom application. Others pointed out a need for a better balance in attention to content and pedagogical strategies, or greater emphasis on the content addressed by particular kits or modules. Finally, evaluators noted a need for more consistency in the skills and expertise of facilitators.
Teacher Preparedness in Science and Mathematics Education
The core evaluation design calls for a random sample of teachers in each project to complete questionnaires about their attitudes and instructional practices. In Cohort 2, these teachers are providing baseline data about the status of science and mathematics education. In Cohort 1, the sample includes teachers who have begun to participate in the LSC professional development, but in most projects only a very small proportion of the teachers had received in-depth assistance at the time of questionnaire administration.
Data collected from teachers in the LSC districts suggest that it will take a long and concerted effort to prepare these teachers to teach science and mathematics in a standards-based fashion. While the vast majority of teachers in LSC districts described attitudes in keeping with the standards (e.g., noting that hands-on activities, inquiry-based instruction, providing concrete experiences, and connections with other disciplines are very important for effective science and mathematics instructions), relatively few teachers feel very well prepared to implement these instructional strategies. For example, more than 80 percent of teachers in the LSC districts indicated that it was very important to have students participate in hands-on science activities, but only about 30 percent felt very well prepared in this area. (See Figure 4.)
Comparison of Teachers Responding "Very Important" and "Very Well Prepared" to Use Various Strategies
Teachers are even more likely to recognize inadequacies in their content preparation. While all but a handful of elementary science teachers feel at least somewhat prepared to teach such topics as the human body and ecology, 20 percent or more indicated they were not adequately prepared to teach sound, astronomy, forces and motion, mixtures and solutions, machines, processes of change over time, and electricity, all areas that are integral to the elementary science curriculum.
Elementary mathematics teachers in LSC districts were likely to express confidence in their ability to teach traditional mathematics topics such as computation, numeration and number theory, measurement, and estimation. However, substantial proportions of teachers noted that they were only "somewhat prepared" to teach some of the topics advocated for inclusion in standards-based instruction in the elementary grades, including geometry and spatial sense, data analysis and probability, and algebraic concepts.
Level of Preparedness Reported by
Teachers in Various Science Content Areas
Level of Preparedness Reported by Cohort 2
Teachers in Various Mathematics Content Areas
A number of core evaluation data sources contributed information about classroom practices. The teacher questionnaire data provide self-report information from a sample of all LSC-targeted teachers about the frequency with which they use various instructional strategies in their classes. The classroom observations provide an external look at the quality of classroom instruction, including information related to equity, classroom culture, and student engagement that are difficult to measure with questionnaire items.
As is the case nationally, elementary teachers in LSC districts report that they teach science less frequently than they teach reading/language arts or mathematics, and about as often as they teach social studies. For example, only about 25 percent of teachers reported teaching science (or social studies) on all 5 of the last 5 days instruction, compared to more than 80 percent for mathematics and nearly 90 percent for reading/language arts. Figure 5 shows the purposes of the observed science and mathematics lessons. Note that teachers were most likely to cite developing conceptual understanding as a major purpose of science lessons and reviewing concepts as a major purpose of mathematics lessons.
Observers noted that science lessons were much more likely to focus on life science than on either physical or earth/space science and that mathematics lessons most frequently focused on computation, followed by measurement, numeration and number theory, and mathematical patterns and relationships. (See Figures 6 and 7.)
Most observed lessons included both a formal presentation by the teacher and some kind of hands-on/investigative activity by the students, typically ones in which students followed a detailed set of instructions rather than designing and implementing their own investigations. Cohort 1 science classes taught by teachers who had already participated in LSC professional development were more likely than others to be involved in writing reflections in a notebook or journal and less likely to be working on answering textbook/worksheet questions. (See Table 4).
Instructional Activities in Observed Classes
When asked how the LSC professional development had affected their classroom teaching, teachers pointed to less reliance on textbooks and more use of student-centered strategies, including cooperative learning groups. Observations confirmed the teacher reports; lessons taught by teachers who had participated in LSC professional development were more likely to receive high ratings from evaluators on lesson design and implementation, as well as on the quality of the content and the classroom culture. (See Figure 8.)
Lessons taught by teachers who had participated in LSC professional development were rated particularly high on the extent to which:
As the final step in completing the observation protocol, evaluators were asked to indicate the "level" that best described each lesson they observed. Table 5 shows the percent of lessons in each Cohort considered effective instruction, beginning stages of effective instruction, and ineffective instruction.3 Note that only 32 percent of Cohort 1 science lessons were rated as effective instruction, quite close to the 27 percent of Cohort 2 science lessons and 29 percent of Cohort 2 mathematics lessons that were rated that highly. The major difference between Cohort 1 and Cohort 2 science lessons is at the lower end of the scale, with lessons of treated teachers more likely to be considered in the beginning stages of effective instruction and less likely to be considered ineffective instruction.
Overall Ratings of Observed Classes
The shaded boxes contain examples of lessons that were assigned to each of these categories.
A complementary analysis based on questionnaire data looked at classroom practice of teachers who reported varying levels of involvement in science and mathematics professional development. Teachers who had participated in a substantial amount of professional development in the previous year were most likely to feel well prepared to teach science and mathematics, and to use the instructional strategies advocated by national standards. For example, as can be seen in Figure 9, only 37 percent of classes whose teachers had not participated in science professional development engaged in hands-on activities at least once a week, compared to 70 percent whose teachers had had in-depth professional development.
Supportive Context for Reform
The major emphasis of the LSC is on professional development to enable teachers to effectively implement high quality instructional materials. At the same time, NSF recognized the need for projects to look at the entire system in these districts, not only professional development and instructional materials, but also policies related to student assessment; teacher recruitment, orientation, and evaluation; and systems for purchasing/managing materials and supplies. Clearly, the support of key stakeholders for the reform effort is crucial, not only from district and school-level administrators, but also from parents and the larger community. Finally, NSF hypothesized that reforms would be greatly enhanced if they had the support of business and industry; colleges and universities; and other "science-rich" institutions.
The core evaluation includes information about the supportiveness of the context for reform from a number of sources. In questionnaires and interviews, teachers were asked about the support they receive from parents, principals, and other teachers in their schools. Principals were also asked about their opinions in relation to science and mathematics reform, including how important they considered various reform approaches, and how well prepared they felt to support teachers in the implementation of national standards. In addition, teacher and principal questionnaires asked about the extent to which school and district policies affect science and mathematics instruction. Finally, evaluators were asked to describe the extent to which district policies and resources were aligned in support of the LSC reforms.
Teachers in the LSC districts generally feel supported by other teachers in their schools to try out innovative ideas in science and mathematics teaching, with roughly 3 out of 4 teachers agreeing with statements to that effect. Somewhat fewer, about 2 out of 3, indicated that teachers in their school regularly share ideas and materials. Only about 1 out of 8 teachers indicated they had time during the regular school week to work with their peers on science and mathematics curriculum and instruction.
In both science and mathematics, teachers are most likely to report that principals encourage innovative instructional practices and accept the noise that comes with an active, inquiry-based classroom (roughly 4 out of 5 in each group). Mathematics teachers were more likely than science teachers to report that their principals provided them with the necessary instructional materials and equipment; and encourages them to gear curriculum and instruction to address individual students' learning.
Principals and teachers were also given a list of "factors" that might affect instruction and asked to indicate the extent, if any, of the problem each caused for science and mathematics instruction. Teachers were answering about problems in their own instruction, and principals were answering about the school as a whole. From both the teacher and principal perspective, there is considerable room for improvement in the extent to which the policy environment supports science and mathematics instruction. In both science and mathematics, lack of time to work with other teachers was the most frequently cited problem, with more than half of the Cohort 1 and Cohort 2 science teachers and nearly that many Cohort 2 mathematics teachers labeling it a major problem.
Access to computers ranked second, again with more than half of the teachers in science and nearly half in mathematics indicating that a lack of computers was a major problem. Other frequently cited problems, especially for science instruction, included lack of funds for purchasing equipment and supplies, and lack of time to plan and prepare lessons. Principals generally echoed the concerns expressed by teachers, frequently citing such problems as inadequate funds for purchasing equipment and supplies, and the lack of teacher planning time and opportunities to work together.
Evaluators were also asked to rate the extent to which district policies and practices tended to facilitate reform or act as barriers for reform. Based on their reports, the most frequent facilitators of reform in LSC districts are the professional development systems; the curriculum, scope and sequence; the fact that the various reform efforts are consistent with one another; and the quality of instructional materials.
Cohort 1 districts received higher marks than Cohort 2 districts in a number of areas, including the quality of instructional materials and new teacher orientation. In contrast, Cohort 2 districts tended to be more highly rated in terms of consistency among reforms. In both cohorts, teacher evaluation policies and within-school policies such as time for teachers to plan and prepare lessons were rarely seen as facilitators of reforms.
A number of evaluators spoke of the commitment the districts had made to aligning their policies in support of the LSC reforms. The following are topical comments:
"Districts have given [LSC project] leaders and connected curriculum review groups the responsibility of revising the district science standards and aligning them with the emerging state [frameworks] in science."
"The district's recent decision to adopt new standards-based science and mathematics materials demonstrates a willingness to work in conjunction with, if not in direct support of, the goals of the LSC."
"The district has purchased kits which are already in use in some schools, and the district has made a major commitment to purchase more. Efforts are underway to create a system for purchasing and managing materials and supplies."
In other cases, evaluators noted that there had been major improvements in the availability of materials for hands-on instruction, but the improvements appeared project-based rather than rooted in district commitments.
"Prior to joining [the project], teachers bought their own science materials or, in some cases, districts provided materials. Currently, [the project] assembles, delivers, picks up, and refurbishes materials, greatly reducing the burden on teachers and districts."
Finally, some evaluators in other projects reported that financial difficulties pose serious problems for reform in the districts.
"[Our site] is implementing its LSC within the context of depressing and undermining budget cuts and staff readjustments. The position of teacher expert has disappeared form the system, vice principals have been cut, and much professional development has been eliminated."
"A lack of materials has been prohibitive...since these schools started with no materials, they will need to make future investments for several years."
"Currently, many teachers are forced to supplement their stock of materials with items purchased with their own money. They look forward to the time when all classrooms will be equipped with adequate materials."
Evaluators were asked to summarize their findings by placing each project at the appropriate point on a continuum of "supportiveness of context". As can be seen in Table 6, most projects in each cohort were considered "transitioning toward a supportive context."
Continuum Rating for Supportiveness of Context
Sustainability of the LSC Reforms
One of the "systemic" aspects of the LSC initiative is the expectation that districts will be able to sustain the reforms after the NSF funding period has ended. Evaluators were asked to rate the extent to which the participating districts have the capacity to implement high-quality professional development, the resources available to support it, and the structures in place to sustain high-quality professional development systems. As several evaluators noted, these ratings are based on a mixture of data and "impressions," but they do provide a broad indication of the status of the districts' professional development systems.
Evaluators were asked to indicate the "continuum level" which best describes the status of the professional development system within the targeted districts. As can be seen in Table 7, most Cohort 1 science projects were judged to be "transitioning toward a high-quality professional development system." Apparently, a number of the Cohort 2 districts had made progress towards institutionalizing high quality professional development systems prior to receiving the LSC awards, with a third of the projects rated as "emerging infrastructure" or even having already institutionalized such a system in the baseline year of their LSC.
Continuum Rating for Sustainability
While data from Cohort 1 projects are technically not "baseline," it is still early in the life of these projects to be looking for signs of sustainability. For that reason, and also because the Cohort 2 projects appear to have been further along in institutionalizing high-quality professional development at the time they began their LSC grants, evaluator ratings of individual aspects of sustainability are presented together.
Evaluators report that in roughly two-thirds of the projects, districts have invested in enhancing the expertise and capacity of lead teachers and that these teachers are actively engaged in facilitating professional development activities (based on the percentages responding 4 or 5 on a five-point scale from 1, "not at all" to 5, "to a great extent").
In addition to the capacity to provide high-quality professional development, districts will need resources to support these programs. Figure 11 shows evaluators' ratings of a number of aspects of resource availability. Note that while in more than half of the LSC projects the districts actively seek opportunities to provide on-going, high-quality professional development experiences for their teachers, the use of district funds for these purposes is less common.
Finally, districts will need structures in place to sustain high-quality professional development and provide a supportive context for exemplary instruction in science and mathematics. Evaluators in less than half of the projects reported that the districts provide incentives for teachers to participate in on-going professional development. And while 4 out of 10 have structures in place for continuously assessing and improving their professional development systems, only 1 in 5 has mechanisms in place for monitoring policy alignment, and only 1 in 4 for maintaining support for reform among community stakeholders.
Horizon Research, Inc. 01/06/98 15
1 An abstract of each of these projects is included in the "Local Systemic Change Project Directory" available from the National Science Foundation and can be accessed through the NSF home page at www.nsf.gov/cgi-bin/getpub?nsf 97145.
2 A more detailed treatment of the evaluation results can be found in the Local Systemic Change through Teacher Enhancement 1995-96 Cross-Site Technical Report, available from Horizon Research, Inc.
3 "Ineffective instruction" includes lessons the observers categorized as level 1a, passive learning; level 1b, activity-for-activity's sake; and level 2, elements of effective instruction. Lessons rated a low or solid 3 are considered "beginning stages of effective instruction," while those rated a high 3, 4, or 5 are considered "effective instruction."