Communication Center  Conference  Projects Share  Reports from the Field Resources  Library  LSC Project Websites  NSF Program Notes
 How to Use this site    Contact us  LSC-Net: Local Systemic Change Network
Educational Reform & Policy

Professional Development

Teaching and Learning

LSC Papers and Reports

Cross Site Reports

LSC Case Study Reports

Papers / Presentations authored by LSC Members

About LSC Initiatives and their impact

Bibliographies

Paper

  New!     

Local Systemic Change through Teacher Enhancement Year Two Cross-Site Report

author: Iris R. Weiss, Kathleen A. Rapp, Diana L. Montgomery
description: "In the spring and summer of 1995, the National Science Foundation (NSF) funded the first cohort of eight projects in a new initiative, the Local Systemic Change through Teacher Enhancement Program (LSC). The following year, 18 additional projects were funded, for a total of 26 projects in Cohorts 1 and 2. ...The LSC solicitation indicated NSF's plan to "provide a framework for data collection (including a set of instruments and procedures) that will allow the Foundation to evaluate individual projects, aggregate data and information across projects, and produce a cross-project analysis" (NSF 94-73). NSF contracted with Horizon Research, Inc. (HRI) of Chapel Hill, NC to design the data collection framework, provide technical assistance in its implementation, and prepare a cross-site analysis of the evaluation results."
published in: Horizon Research, Inc.
published: 10/01/1997
posted to site: 02/10/1998
Horizon Research, Inc.
111 Cloister Court - Suite 220
Chapel Hill, NC 27514-2296


Table of Contents
Page
List of Figures iv
List of Tables vi
List of Boxes vii
Acknowledgments viii
Chapter One: Introduction 1
Description of Core Evaluation Data Collection Activities 1
An Overview of Cohort 1 and 2 Projects 4
Chapter Two: Results
Core Evaluation Question I: Quality of Professional Development 9
Teacher Ratings of District-Provided Professional Development9
Description of LSC Professional Development Activities11
Teacher Ratings of the Quality of LSC Professional Development23
Ratings of Individual Professional Development Sessions24
Summary34
Core Evaluation Question II: Involvement in LSC 36
Teacher Participation36
School Involvement in Reform37
Core Evaluation Question III: Teacher Attitudes, Beliefs, and Preparation39
Teacher Attitudes and Beliefs about Science and Mathematics Education40
Teacher Perceptions of Their Preparedness43
Summary 55
Core Evaluation Question IV: Classroom Practice56
Teacher Descriptions of Classroom Practice56
Evaluator Descriptions of Classroom Practice61
An Evaluative Look at Classrooms66
Summary74
A First Look at the Impact of LSC Professional Development76
Teacher Perceptions of the Impact of the LSC76
Changes in Cohort 1 Science Instruction: Year One to Year Two77
Comparison of Results by Level of Involvement in Professional Development79
Summary85
Core Evaluation Question V: Supportive Context for Reform86
Teachers Perceptions of Support for Reform86
Principal Views on Science and Mathematics Reform90
Teacher and Principal Views on the Policy Environment for Reform91
Evaluator Ratings of Support for Reform94
Core Evaluation Question VI: Sustainability of the Professional
Development System
98
Chapter Three: Summary and Conclusions101


List of Figures

  1. Community Type for Cohort 1 and Cohort 2 Schools
  2. Student Race/Ethnicity for Cohort 1 and Cohort 2 Schools
  3. Characteristics of Students in LSC Schools
  4. Involvement of School Personnel in Addition to Classroom Teachers
  5. Outreach to Groups External to the School
  6. Composite: Quality of District Professional Development
  7. Participants in Observed LSC Professional Development Activities
  8. Major Disciplinary Content Areas of Observed Science Professional Development Activities
  9. Major Disciplinary Content Areas of Observed Mathematics Professional Development Activities
  10. Pedagogical Content Areas of Observed LSC Professional Development Activities
  11. Components of Professional Development System
  12. Teachers' Assessment of LSC Professional Development Activities
  13. Professional Development Session Ratings-Design
  14. Professional Development Session Ratings-Implementation
  15. Professional Development Session Ratings-Disciplinary Content
  16. Professional Development Session Ratings-Pedagogical Content
  17. Professional Development Session Ratings-Classroom Culture
  18. Professional Development Session Ratings-Capacity for Exemplary Instruction
  19. Professional Development Session Ratings-Leadership Capacity
  20. Professional Development Capsule Ratings of Observed Sessions
  21. Schools Heavily Involved in Reform in Each Subject
  22. Schools "well-along" in Improving their Program
  23. Composite: Attitudes Toward Teaching
  24. "I enjoy teaching science/mathematics"
  25. "Students learn science/mathematics best with students of similar abilities"
  26. Importance of Various Alternative Assessment Techniques
  27. Level of Preparedness to Teach Science/Mathematics
  28. "I am Well-Informed About National Standards in Science/Mathematics"
  29. Composite: Preparation-Knowledge of Science/Mathematics Topics
  30. Principal Concerns: Effect of Teacher Preparation on Science/Mathematics Instruction
  31. Composite: Preparation-Pedagogical Content
  32. Composite: Preparation-Instructional Strategies
  33. Pedagogical Strategies: Comparison of Teachers Responding "Very Important" and "Very Well Prepared" to Use Various Strategies
  34. Instructional Strategies: Comparison of Teachers Responding "Very Important" and "Very Well Prepared" to Use Various Strategies
  35. Assessment Strategies: Comparison of Teachers Responding "Very Important" and "Very Well Prepared" to Use Various Strategies
  36. Composite: Preparation-Equity and Diversity
  37. Level of Preparedness to Involve Parents
  38. Teachers Reporting Instruction in Each Subject on All 5 of the Last 5 days
  39. Composite: Use of Student Centered Strategies
  40. Composite: Investigative Culture
  41. Composite: Use of Assessment Strategies
  42. Class Size
  43. Minority Enrollment in Observed Classes
  44. Purposes of Science and Mathematics Lessons
  45. Focus of Science Lessons
  46. Focus of Cohort 2 Mathematics Lessons
  47. Lessons with High Ratings For Design
  48. Lessons with High Ratings for Implementation
  49. Lessons with High Ratings for Content
  50. Lessons with High Ratings for Classroom Culture
  51. Lessons with High Ratings for Likely to Enhance Student Ability
  52. Cohort 1 Teachers Feeling At Least Fairly Well Prepared
  53. Cohort 1 Science Classes Engaged in Hands-on Activities At Least Once a Week
  54. 1996 LSC Science Composites by Extent of Teacher Participation in Professional Development
  55. 1996 LSC Mathematics Composites by Extent of Teacher Participation in Professional Development
  56. Teachers Feeling Very Well Prepared to Teach Mathematics Topic
  57. Teachers Feeling Very Well Prepared to Teach Each Science Topic
  58. Teachers Feeling Very Well Prepared For Each Activity-Science
  59. Teachers Feeling Very Well Prepared For Each Activity-Mathematics
  60. Teachers Feeling Very Well Prepared in Aspects of Equity and Diversity-Science
  61. Teachers Feeling Very Well Prepared in Aspects of Equity and Diversity -Mathematics
  62. Instructional Strategies in Science Classes
  63. Instructional Strategies in Mathematics Classes
  64. Teacher Perceptions of Support from Colleagues
  65. Composite: Administrative Support
  66. Composite: Parent Support
  67. Principal Knowledge of National Standards
  68. Composite: Alignment of District Policies
  69. Teachers Labeling Each a Major Problem
  70. Principals Labeling each a Major Problem
  71. Support for LSC Reform
  72. Percent of Projects Where Evaluators Report Each Factor Facilitates Reform
  73. Capacity to Implement High-Quality Professional Development
  74. Resources to Support On-Going Professional Development
  75. Structures for Sustaining High-Quality Professional Development



List of Tables

  1. Professional Development Providers in LSC Projects
  2. Teacher Descriptions of District Professional Development
  3. Major Intended Purposed of Observed LSC Professional Development Sessions
  4. Major Activities at Observed LSC Professional Development Sessions
  5. Continuum Ratings for Quality of LSC Professional Development
  6. Teachers Reporting Participation in Various Types of Professional Development Activities in the Past 12 Months
  7. Teacher Participation in Various Numbers of Hours of Structured Professional Development
  8. Teachers Indicating Each Strategy is "Very Important" for Effective Instruction
  9. Level of Preparedness Reported by Teachers in Various Science Content Areas
  10. Level of Preparedness Reported by Cohort 2 Mathematics Teachers in Various Mathematics Content Areas
  11. Teacher Perceptions of Their Preparation to Implement Standards-Based Instruction
  12. Classes Participating in Each Activity at Least Once a Week
  13. Activities Ever Occurring in Science and Mathematics Classes
  14. Instructional Strategies Used at Least Once a Week
  15. Assessment Activities in Science and Mathematics Classes Using Each Type of Assessment at Least Once a Month
  16. Instructional Activities in Observed Classes
  17. Lessons Rated 4 or 5 on Design Indicators
  18. Lessons Rated 4 or 5 on Implementation Indicators
  19. Lessons Rated 4 or 5 on Content Indicators
  20. Lessons Rated 4 or 5 on Classroom Culture Indicators
  21. Lessons Rated 4 or 5 on Indicators of Enhancing Student Ability
  22. Overall Ratings of Observed Classes
  23. Cohort 1 Science Teachers' Opinions About Science Instruction
  24. Teachers' Perceptions of Principal Support for Science and Mathematics Reform
  25. Principal Ratings of Importance of Various Instructional Strategies for Effective Instruction
  26. Continuum Rating for Supportiveness of Context
  27. Continuum Rating for Sustainability



List of Boxes

Professional Development Session #1: "Learning Through Kit-Based Activities"

Professional Development Session #2: "Study Groups: Classroom Observation and Discussion"

Professional Development Session #3: "Exploring Student Assessments"

Professional Development Session #4: "Learning the Kit"

Lesson #1: Ineffective Science Instruction-Passive "Learning"

Lesson #2: Ineffective Mathematics Instruction-Activity for Activity's Sake

Lesson #3: Beginning Stages of Effective Mathematics Instruction

Lesson #4: Effective Science Instruction

Lesson #5: Effective Mathematics Instruction



Acknowledgments

The core evaluation of the Local Systemic Change Initiative requires the energy, efforts, and insights of a very large number of people.

Conrad Katzenmeyer in the National Science Foundation's (NSF) Division of Research, Evaluation, and Communication; Joyce Evans, Susan Snyder, and Diane Spresser in NSF's Division of Elementary, Secondary, and Informal Education (ESIE); and Joy Frechtling of Westat, Inc. were instrumental in the design and implementation of the core evaluation system. Numerous other NSF/ESIE program officers provided valuable assistance throughout the process, as well.

Within HRI, in addition to the authors of this report, Joan Pasley, Ruth Pershing, Ben Kurgat, Eugene Soar, Scott Hanrath, Jarrod Jenzano, Alison Schaff, Carol McDonald, Susan Hudson, Claudia Templeton, and John Supovitz were responsible for various tasks essential to the production of this report, from instrument design, to data processing and analysis, to report production.

Most notably, this report would not have been possible without the efforts of the LSC project lead evaluators, who persevered as we attempted to fine-tune the core evaluation system. And special thanks are due to the thousands of teachers throughout the 121 participating districts who took time from their busy schedules to provide information about their science and mathematics teaching.



Chapter One


Introduction

In the spring and summer of 1995, the National Science Foundation (NSF) funded the first cohort of eight projects in a new initiative, the Local Systemic Change through Teacher Enhancement Program (LSC). The following year, 18 additional projects were funded, for a total of 26 projects in Cohorts 1 and 2.

The goal of the program is to improve the teaching of science, mathematics, and technology by focusing on the professional development of teachers within whole schools or school districts. Each targeted teacher is to participate in a minimum of 100 hours of professional development with an emphasis on preparing them to implement exemplary science and mathematics instructional materials in their classrooms. Projects are expected to align policy and practice within the targeted district(s) and to include:

  • A shared comprehensive vision of science, mathematics, and technology education;
  • Active partnerships and commitments among stakeholders;
  • A detailed self-study that provides a realistic assessment of the current system's strengths and needs;
  • Strategic planning that incorporates mechanisms for engaging each teacher in intensive professional development activities over the course of the project; and
  • A set of clearly defined, measurable outcomes for teaching, and an evaluation plan that provides on-going feedback for the project.

The LSC solicitation indicated NSF's plan to "provide a framework for data collection (including a set of instruments and procedures) that will allow the Foundation to evaluate individual projects, aggregate data and information across projects, and produce a cross-project analysis" (NSF 94-73). NSF contracted with Horizon Research, Inc. (HRI) of Chapel Hill, NC to design the data collection framework, provide technical assistance in its implementation, and prepare a cross-site analysis of the evaluation results.

Description of Core Evaluation Data Collection Activities

HRI has worked with the National Science Foundation, and PIs and evaluators of the LSC projects, in the design and implementation of a core evaluation system to allow aggregating information across projects. This chapter describes the data collection activities associated with the core evaluation and provides an overview of the first two cohorts of LSC projects. Results organized by core evaluation question are presented in the next chapter, followed by a summary and conclusions chapter.

The core evaluation system includes qualitative and quantitative data collection by the LSC projects to address the following questions:

  1. What is the overall quality of the LSC professional development activities?
  2. What is the extent of school and teacher involvement in LSC activities?
  3. What is the impact of the LSC professional development on teacher preparedness, attitudes, and beliefs about science and mathematics teaching and learning?
  4. What is the impact of the LSC professional development on classroom practices in science and mathematics?
  5. To what extent are the district and school contexts becoming more supportive of the LSC vision for exemplary science and mathematics education?
  6. What is the extent of institutionalization of high quality professional development systems in the LSC districts?

Data collection activities for the projects' 1995-96 Core Evaluation Reports were conducted from September 1, 1995 through August 31, 1996. This was the second year of data collection for Cohort 1 projects; Cohort 2 projects were collecting baseline data for their first year of funding. Activities included the following:

  • Observations of professional development activities;
  • Observations of teachers' classrooms;1
  • A questionnaire to a random sample of all targeted teachers; and
  • A questionnaire to principals of all targeted schools.

In addition, each Cohort 1 project was asked to conduct interviews with ten teachers randomly selected from among those who had participated in the LSC professional development.

The core evaluation calls for projects to conduct 5-8 observations of professional development sessions each year and record their observations on standardized protocols. Evaluators were to consult with PIs on what professional development experiences were planned throughout the data collection year, and select a sample that was representative of the diversity of the project's activities. Program-wide, a total of 180 observations of professional development sessions were conducted.

HRI provided the lead evaluator of each project with a list of 10 randomly selected teachers for each targeted subject to be observed in the spring of 1996. There were a total of 273 classrooms observed, including 76 science classes taught by Cohort 1 teachers who had participated in LSC professional development, and 145 science and 52 mathematics classes as baseline for Cohort 2 projects. In each case, the data were weighted to represent the total population of eligible teachers in the project.

Each project was asked to administer the teacher questionnaire developed for the core evaluation to a sample of 300 teachers per targeted subject; the median response rate was 70 percent. Weights were added to the data file to reflect the probability of each teacher's selection into the sample, adjusted for any non-response in that project. Projects were also asked to administer a questionnaire to the entire population of principals of targeted schools. Return rates on the principal questionnaire were generally higher than for the teacher questionnaire, with a median response rate of 89 percent.

Evaluators of Cohort 1 projects were asked to interview a sample of 10 teachers who had participated in LSC professional development activities. A total of 79 interviews were conducted among the eight projects2. Most interviews (85 percent) were conducted by phone; the remaining 15 percent were conducted in person. Evaluators summarized the interview data by completing an interview summary form that consisted of both close-ended assessments and open-ended descriptions of the information provided by each teacher. Data from each project were weighted to reflect the total number of teachers who had participated in LSC professional development in that project.

Project evaluators were asked to report their findings using guidelines developed for the core evaluation system, including responding to the six core evaluation questions and providing overall ratings of the quality of professional development activities, the supportiveness of the context, and the sustainability of high-quality professional development systems. In some cases, evaluators used additional data in preparing their reports, including expanded use of the core evaluation instruments and use of project-specific data collection activities.

The cross-site results were analyzed separately for science and mathematics projects and for the two cohorts of science projects. Differences noted in this report were statistically significant at the .05 level.

An Overview of Cohort 1 and 2 Projects

Project data sheets completed by the PIs provide some basic information about the 26 projects included in Cohorts 1 and 23.

  • Roughly half of the Cohort 1 and Cohort 2 projects are single-district projects; at the other end of the scale, 4 projects involve more than 10 districts, including 1 with 21 districts.
  • The 8 Cohort 1 projects and 18 Cohort 2 projects plan to involve a total of 28,239 teachers in 1,312 schools in 121 districts across the United States.
  • By the completion of these projects, an estimated 706,000 students will receive instruction from LSC-treated teachers each year.
  • 20 projects target K-8 science; 2 projects target K-8 mathematics; and 4 projects target both K-8 science and mathematics.

In the results presented in this report, the eight Cohort 1 projects, all of which target elementary science teachers, are analyzed in a single group referred to as C1S. Cohort 2 includes project targeting elementary science (C2S) and elementary mathematics (C2M).4

Note that there are clear differences in the populations of schools funded in Cohort 1 and Cohort 2. As can be seen in Figure 1, approximately one-third of Cohort 1 schools and two-thirds of Cohort 2 schools are located in urban areas. Cohort 2 schools tend to be larger, averaging 634 students in schools targeted for LSC science programs and 791 in schools targeted for mathematics programs, compared to 467 students in Cohort 1 schools. Cohort 2 schools also have a higher percentage of students who are members of underrepresented minority groups. (See Figure 2).

Community Type for Cohort 1 and 2 Science and Mathematics Schools

Figure 1

Student Race/Ethnicity for Cohort 1 and 2 Science and Mathematics Schools

Figure 2

On other measures, Cohort 1 and 2 schools are more similar, including the percent of students with limited English Proficiency and the percent receiving free or reduced lunch. (See Figure 3.) The two cohorts are also similar in terms of principal experience; in both cohorts principals have a median of 6-10 years experience as principal and 3-5 years as principal at the current school.

Nearly all of the LSC projects (93 percent) report that they are preparing teacher leaders to serve as mentors within the school. Somewhat fewer (81 percent) include a peer mentoring/teaching component, and nearly that many (73 percent) include teacher study groups as part of their repertoire of professional development activities.

Characteristics of Students in LSC Schools

Figure 3

In terms of professional development providers, all of the projects report that they are using lead teachers in some capacity. Many also involve scientists, mathematicians, and/or engineers from higher education, business/industry and museums or other community organizations. (See Table 1.)

Table 1
Professional Development Providers in LSC Projects

Percent of Projects
Scientists/Mathematicians/ Engineers89
Higher Education77
Business/Industry58
Museums/Community Organizations42
Education Professionals100
Lead Teachers100
Higher Education81
District-Level Personnel54
Museum/Community Organizations50

While the focus of the LSC is on providing high-quality professional development services to science and mathematics classroom teachers, many of the projects are also involving others in the school, district, and larger community. Figure 4 shows the percent of projects reporting that they have a formal component aimed at each of a number of groups within the LSC schools. Note that almost all of the Cohort 1 and Cohort 2 projects have a formal component aimed at principals. Other groups are less frequently targeted; almost half of the projects involve special education teachers and nearly that many involve Title I teachers and media specialists.

Involvement of School Personnel in Addition to Classroom Teachers

Figure 4

Figure 5 shows analogous data for groups outside the individual schools. Roughly 40 percent of the projects include activities for central office staff, parents, and business/industry representatives, while roughly 20 percent report targeting higher education faculty and pre-service teachers. None of the projects reported having a component aimed at the general public.

Outreach to Groups External to the School

Figure 5




  1. Teachers observed in Cohort 2 projects were randomly selected from among all teachers targeted for professional development by the project. Teachers observed in Cohort 1 projects were randomly selected from among those teachers who had already participated in LSC professional development.
  2. Each lead evaluator was provided a list of 10 teachers selected randomly from those who had participated in the LSC professional development. In two cases, however, evaluators substituted other teachers for the randomly selected ones. Therefore, these results may not be representative of the entire population of "treated" teachers.
  3. An abstract of each of these projects is included in the "Local Systemic Change Project Directory" available from the National Science Foundation and can be accessed through the NSF home page at www.nsf.gov/cgi-bin/getpub?nsf 97145.
  4. Four Cohort 2 projects target both science and mathematics. For analysis, observations of professional development sessions that dealt with science topics are included in C2S; observations of professional development sessions that dealt with mathematics topics are included in C2M.

     next page